AI-Driven Autonomous Security Protocol for Self-Healing Software Defined Optical Networks Using Deep Reinforcement Learning

Abstract

The increasing deployment of Software-Defined Optical Networks (SDONs) to support 5G/6G 
communications, cloud computing, Internet of Things (IoT), and data-intensive applications 
has significantly enhanced network programmability and resource utilization. However, the 
centralized control architecture and dynamic nature of SDONs expose them to a wide range of 
cyber-physical threats, including distributed denial-of-service attacks, optical jamming, signal 
spoofing, eavesdropping, and control-plane intrusions. Conventional security mechanisms 
primarily rely on static policies and predefined signatures, limiting their effectiveness against 
evolving and previously unseen attack patterns. To address these challenges, this paper 
proposes an AI-driven autonomous security protocol for self-healing software-defined optical 
networks based on Deep Reinforcement Learning (DRL). The proposed framework integrates 
optical performance monitoring, real-time network telemetry, anomaly detection, and 
intelligent decision-making within the SDN control plane to enable proactive threat 
identification and autonomous mitigation. A DRL agent continuously learns optimal security 
policies by interacting with the network environment and dynamically performs actions such 
as route reconfiguration, traffic isolation, resource reallocation, and attack containment while 
minimizing service disruption. The framework further incorporates a self-healing mechanism 
that predicts network vulnerabilities, identifies compromised links or nodes, and restores 
network operations through adaptive path computation and automated recovery procedures. 
Experimental evaluation using realistic optical network traffic and attack scenarios 
demonstrates that the proposed protocol achieves superior threat detection accuracy, reduced 
response latency, enhanced network survivability, and improved service availability compared 
with conventional machine learning and rule-based security approaches. The results indicate 
that the integration of deep reinforcement learning with software-defined optical networking 
provides an effective foundation for building resilient, autonomous, and secure next-generation 
optical communication infrastructures capable of supporting future intelligent networking 
ecosystems.