Towards a Generalisable Cyber Defence Agent for Real-World Computer Networks

Tim Dudman, Martyn Bull
Proceedings of the 2025 Conference on Applied Machine Learning for Information Security, PMLR 299:87-109, 2025.

Abstract

Recent advances in deep reinforcement learning for autonomous cyber defence have resulted in agents that can successfully defend simulated computer networks against cyber-attacks. However, many of these agents would need retraining to defend networks with differing topology or size, making them poorly suited to real-world networks where topology and size can vary over time. In this research we introduce a novel set of Topological Extensions for Reinforcement Learning Agents (TERLA) that provide generalisability for the defence of networks with differing topology and size, without the need for retraining. These extensions can be applied to existing autonomous cyber defence agents and may also help address policy learning challenges inherent with large network action spaces. Our approach involves the use of heterogeneous graph neural network layers to produce a fixed-size latent embedding representing the observed network state. This representation learning stage is coupled with a reduced, fixed-size, semantically meaningful and interpretable action space. We apply TERLA to a standard deep reinforcement learning Proximal Policy Optimisation (PPO) agent model, and to reduce the sim-to-real gap, conduct our research using Cyber Autonomy Gym for Experimentation (CAGE) Challenge 4. This Cyber Operations Research Gym environment has many of the features of a real-world network, such as realistic Intrusion Detection System (IDS) events and multiple agents defending network segments of differing topology and size. TERLA agents have been designed for host-based defence, and defensive actions are resolved to specific hosts using only observable IDS information that would be available in real-world networks. TERLA agents have been evaluated against vanilla PPO agents as well as other approaches including no defence against cyber-attacks and performing random defensive actions. TERLA agents retain the defensive performance of vanilla PPO agents whilst showing improved action efficiency. Generalisability has been demonstrated by showing that all TERLA agents have the same network-agnostic neural network architecture, and by deploying a single TERLA agent multiple times to defend network segments with differing topology and size, showing improved defensive performance and efficiency.

Cite this Paper

BibTeX
@InProceedings{pmlr-v299-dudman25a, title = {Towards a Generalisable Cyber Defence Agent for Real-World Computer Networks}, author = {Dudman, Tim and Bull, Martyn}, booktitle = {Proceedings of the 2025 Conference on Applied Machine Learning for Information Security}, pages = {87--109}, year = {2025}, editor = {Raff, Edward and Rudd, Ethan M.}, volume = {299}, series = {Proceedings of Machine Learning Research}, month = {22--24 Oct}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v299/main/assets/dudman25a/dudman25a.pdf}, url = {https://proceedings.mlr.press/v299/dudman25a.html}, abstract = {Recent advances in deep reinforcement learning for autonomous cyber defence have resulted in agents that can successfully defend simulated computer networks against cyber-attacks. However, many of these agents would need retraining to defend networks with differing topology or size, making them poorly suited to real-world networks where topology and size can vary over time. In this research we introduce a novel set of Topological Extensions for Reinforcement Learning Agents (TERLA) that provide generalisability for the defence of networks with differing topology and size, without the need for retraining. These extensions can be applied to existing autonomous cyber defence agents and may also help address policy learning challenges inherent with large network action spaces. Our approach involves the use of heterogeneous graph neural network layers to produce a fixed-size latent embedding representing the observed network state. This representation learning stage is coupled with a reduced, fixed-size, semantically meaningful and interpretable action space. We apply TERLA to a standard deep reinforcement learning Proximal Policy Optimisation (PPO) agent model, and to reduce the sim-to-real gap, conduct our research using Cyber Autonomy Gym for Experimentation (CAGE) Challenge 4. This Cyber Operations Research Gym environment has many of the features of a real-world network, such as realistic Intrusion Detection System (IDS) events and multiple agents defending network segments of differing topology and size. TERLA agents have been designed for host-based defence, and defensive actions are resolved to specific hosts using only observable IDS information that would be available in real-world networks. TERLA agents have been evaluated against vanilla PPO agents as well as other approaches including no defence against cyber-attacks and performing random defensive actions. TERLA agents retain the defensive performance of vanilla PPO agents whilst showing improved action efficiency. Generalisability has been demonstrated by showing that all TERLA agents have the same network-agnostic neural network architecture, and by deploying a single TERLA agent multiple times to defend network segments with differing topology and size, showing improved defensive performance and efficiency.} }
Endnote
%0 Conference Paper %T Towards a Generalisable Cyber Defence Agent for Real-World Computer Networks %A Tim Dudman %A Martyn Bull %B Proceedings of the 2025 Conference on Applied Machine Learning for Information Security %C Proceedings of Machine Learning Research %D 2025 %E Edward Raff %E Ethan M. Rudd %F pmlr-v299-dudman25a %I PMLR %P 87--109 %U https://proceedings.mlr.press/v299/dudman25a.html %V 299 %X Recent advances in deep reinforcement learning for autonomous cyber defence have resulted in agents that can successfully defend simulated computer networks against cyber-attacks. However, many of these agents would need retraining to defend networks with differing topology or size, making them poorly suited to real-world networks where topology and size can vary over time. In this research we introduce a novel set of Topological Extensions for Reinforcement Learning Agents (TERLA) that provide generalisability for the defence of networks with differing topology and size, without the need for retraining. These extensions can be applied to existing autonomous cyber defence agents and may also help address policy learning challenges inherent with large network action spaces. Our approach involves the use of heterogeneous graph neural network layers to produce a fixed-size latent embedding representing the observed network state. This representation learning stage is coupled with a reduced, fixed-size, semantically meaningful and interpretable action space. We apply TERLA to a standard deep reinforcement learning Proximal Policy Optimisation (PPO) agent model, and to reduce the sim-to-real gap, conduct our research using Cyber Autonomy Gym for Experimentation (CAGE) Challenge 4. This Cyber Operations Research Gym environment has many of the features of a real-world network, such as realistic Intrusion Detection System (IDS) events and multiple agents defending network segments of differing topology and size. TERLA agents have been designed for host-based defence, and defensive actions are resolved to specific hosts using only observable IDS information that would be available in real-world networks. TERLA agents have been evaluated against vanilla PPO agents as well as other approaches including no defence against cyber-attacks and performing random defensive actions. TERLA agents retain the defensive performance of vanilla PPO agents whilst showing improved action efficiency. Generalisability has been demonstrated by showing that all TERLA agents have the same network-agnostic neural network architecture, and by deploying a single TERLA agent multiple times to defend network segments with differing topology and size, showing improved defensive performance and efficiency.
APA
Dudman, T. & Bull, M.. (2025). Towards a Generalisable Cyber Defence Agent for Real-World Computer Networks. Proceedings of the 2025 Conference on Applied Machine Learning for Information Security, in Proceedings of Machine Learning Research 299:87-109 Available from https://proceedings.mlr.press/v299/dudman25a.html.

Related Material