Editor's Note:
Associate Professor Faisal Nadeem Khan from the Institute of Data and Information has recently published a perspective article entitled Non-technological barriers: the last frontier towards AI-powered intelligent optical networks in Nature Communications. The article highlights key non-technological impediments to the broad deployment of machine learning-based solutions in commercial fiber-optic networks and offers an extensive set of novel solutions.
Machine learning (ML) has long been considered as a promising technology that can fundamentally transform the existing outdated optical networks into next-generation smart and autonomous entities. In the last few years, industry as well as academia has witnessed a significant increase in research endeavors to harness and capitalize on ML across different facets of fiber-optic communications ranging from designing of network components to compensating critical transmission impairments to predicting data traffic flow patterns in networks. However, despite unprecedented interest in this field over the past decade, the developed ML methods have not yet attained anticipated deployment, credibility, and impact in real-world fiber-optic networks. In a recently-published perspective article “Non-technological barriers: the last frontier towards AI-powered intelligent optical networks” in Nature Communications, Faisal Nadeem Khan, associate professor at the Institute of Data and Information, offers solutions to address these issues.
In the article, Khan highlights that the key impediment towards broad deployment of ML-based solutions in commercial fiber-optic networks is the existence of several pending non-technological limiting factors that are crucial from practical networks’ perspective but have been largely ignored by the relevant stakeholders. To this end, he systematically identifies seven major non-technological barriers, as shown in Figure 1, including the prevalence of legacy systems and processes; cost restraints; expert workforce limitations; data accessibility and privacy protection problems; interpretability, transparency, and accountability issues of ML models; lack of standards and regulatory policies for ML-aided methods; and human factors and cognitive biases.
Figure 1. Seven key non-technological challenges faced by ML-aided methods in practical fiber-optic networks
To substantiate his viewpoint, Khan outlines five major ML application areas in optical networks, network failures management, end-to-end (E2E) communication system optimization, lightpaths’ quality of transmission (QoT) estimation, optical performance monitoring (OPM), and network security management, and discusses how the aforementioned seven non-technological barriers greatly diminish the deployment prospects of developed ML-aided methods in practical fiber-optic networks.
He also ranks the non-technological challenges based on the degree of difficulty faced in their resolution, as shown in Figure 2. In his viewpoint, the two biggest challenges to overcome are the legacy issues and cost restraints, respectively, due to the associated financial implications of overhauling the existing optical networks infrastructure. The realization of standardization and regulatory frameworks, which is indispensable for developing universally-operable ML-aided methods for optical networks, is the next big problem to solve. Such frameworks warrant joint initiatives by industry, standardization organizations, and regulatory bodies, which are scarce thus far. The fourth critical challenge is providing global access to relevant sources of data while preserving data privacy and anonymity. However, establishing such data sharing mechanisms and defining clear data-usage terms for proprietary network data are still works in progress. The next two problems are the interpretability and accountability issues of ML models and the presence of intentional/unintentional biases. Although these issues do not fundamentally deter the implementation of ML-aided tools, their solution is vital for actualizing credible decision-making processes in optical networks. Lastly, the expert workforce limitation is expected to be a pressing problem mainly in the short-term as the recent interest shown by industry, academia, and governments around the world in fostering basic ML knowledge and skills will likely be consequential in dealing with the trained workforce scarcity in due course.
Figure 2. Difficulty levels of seven key non-technological barriers hindering vast deployment of ML-enabled solutions in fiber-optic networks
Khan also provides an extensive set of novel solutions that can be instrumental in resolving each of the existing non-technological challenges, thus clearing the path for widespread application of ML-powered solutions for intelligent operations and decision making in future fiber-optic communication networks.
Link to full article: https://www.nature.com/articles/s41467-024-50307-y
Source: Institute of Data & Information
Edited by Alena Shish & Yuan Yang
Reviewed by Zhang Chengping & Chen Chaoqun
