Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

Akindoyin Akinrele, Shreyank N Gowda
Proceedings of the Fourth UK AI Conference 2026, PMLR 348:12-30, 2026.

Abstract

Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code: \url{https://github.com/TimiAkinrele/ai-jailbreak-framework}

Cite this Paper


BibTeX
@InProceedings{pmlr-v348-akinrele26a, title = {Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals}, author = {Akinrele, Akindoyin and Gowda, Shreyank N}, booktitle = {Proceedings of the Fourth UK AI Conference 2026}, pages = {12--30}, year = {2026}, editor = {Benford, Alistair and Büyükateş, Baturalp and Cabrera, Christian and Kiden, Sarah and Salili-James, Arianna and Zakka, Vincent and Zhou, Feng}, volume = {348}, series = {Proceedings of Machine Learning Research}, month = {29--30 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v348/main/assets/akinrele26a/akinrele26a.pdf}, url = {https://proceedings.mlr.press/v348/akinrele26a.html}, abstract = {Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code: \url{https://github.com/TimiAkinrele/ai-jailbreak-framework}} }
Endnote
%0 Conference Paper %T Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals %A Akindoyin Akinrele %A Shreyank N Gowda %B Proceedings of the Fourth UK AI Conference 2026 %C Proceedings of Machine Learning Research %D 2026 %E Alistair Benford %E Baturalp Büyükateş %E Christian Cabrera %E Sarah Kiden %E Arianna Salili-James %E Vincent Zakka %E Feng Zhou %F pmlr-v348-akinrele26a %I PMLR %P 12--30 %U https://proceedings.mlr.press/v348/akinrele26a.html %V 348 %X Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated under limited settings that do not reflect real-world operating constraints. In this work, we present a deployment-aware evaluation of prompt injection detection using a multi-model and multi-regime experimental framework. We compare lexical, semantic, structural, and transformer-based detectors across multiple out-of-distribution settings, repeated data splits, and both ranking and thresholded deployment metrics. We introduce interpretable structural signals that capture hierarchy overrides, system prompt spoofing, role redefinition, and evasion patterns, and assess their contribution both within sparse models and in combination with strong encoder baselines. Our results show that detection performance is highly regime-dependent and sensitive to threshold selection, with no single model dominating across all settings. Transformer-based models achieve the strongest overall performance, while structural signals provide modest but consistent gains in certain regimes and improve low false positive rate behaviour in harder scenarios. These findings highlight the gap between ranking performance and deployment effectiveness and underscore the importance of evaluating prompt injection defences under realistic operational constraints. Code: \url{https://github.com/TimiAkinrele/ai-jailbreak-framework}
APA
Akinrele, A. & Gowda, S.N.. (2026). Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals. Proceedings of the Fourth UK AI Conference 2026, in Proceedings of Machine Learning Research 348:12-30 Available from https://proceedings.mlr.press/v348/akinrele26a.html.

Related Material