The Reliability Gap: A Multi-Dimensional Technical Audit of Memory Safety and Logical Integrity in LLM-Generated C Code

Ricardo Jarrin, Luiza Antonie, Ritu Chaturvedi
Proceedings of the The 39th Canadian Conference on Artificial Intelligence, PMLR 318:968-974, 2026.

Abstract

The integration of Large Language Models (LLMs) into C programming education offers a scalable solution to the systemic enrollment crisis in computer science. However, the non-deterministic nature of these probabilistic engines introduces a critical reliability gap, particularly in C programming - a domain characterized by manual memory management and a high risk of Undefined Behavior (UB). This study presents a multi-dimensional technical audit - integrating static structural analysis with dynamic runtime integrity checks - of four 2026 frontier models: GPT-OSS 120B, Llama 3.3 70B, Moonshot Kimi K2, and Qwen 3 32B. Utilizing an automated evaluation pipeline, 1195 code generations across a gradient of complexity (Pointers, Dynamic Memory, and Data Structures) were subjected to static analysis and dynamic runtime instrumentation. Experimental results reveal a significant "Technical Reliability Gap": while models achieve high Compilation Success Rates (CSR), dynamic analysis identified a frequent incidence of "Definitely Lost" heap memory and "logical hangs". We further identify a correlation between architectural paradigm and safety, finding that sparse Mixture-of-Experts (MoE) architectures exhibit lower Static Error Density (SED) and higher logic density than dense counterparts. We conclude by offering a framework for trust calibration based on these technical breaking points, assisting educators in mitigating the "oracle hazards" of AI-mediated instruction.

Cite this Paper

BibTeX
@InProceedings{pmlr-v318-jarrin26a, title = {The Reliability Gap: A Multi-Dimensional Technical Audit of Memory Safety and Logical Integrity in LLM-Generated C Code}, author = {Jarrin, Ricardo and Antonie, Luiza and Chaturvedi, Ritu}, booktitle = {Proceedings of the The 39th Canadian Conference on Artificial Intelligence}, pages = {968--974}, year = {2026}, editor = {Bouzar-Benlabiod, Lydia and Leung, Carson}, volume = {318}, series = {Proceedings of Machine Learning Research}, month = {25--29 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v318/main/assets/jarrin26a/jarrin26a.pdf}, url = {https://proceedings.mlr.press/v318/jarrin26a.html}, abstract = {The integration of Large Language Models (LLMs) into C programming education offers a scalable solution to the systemic enrollment crisis in computer science. However, the non-deterministic nature of these probabilistic engines introduces a critical reliability gap, particularly in C programming - a domain characterized by manual memory management and a high risk of Undefined Behavior (UB). This study presents a multi-dimensional technical audit - integrating static structural analysis with dynamic runtime integrity checks - of four 2026 frontier models: GPT-OSS 120B, Llama 3.3 70B, Moonshot Kimi K2, and Qwen 3 32B. Utilizing an automated evaluation pipeline, 1195 code generations across a gradient of complexity (Pointers, Dynamic Memory, and Data Structures) were subjected to static analysis and dynamic runtime instrumentation. Experimental results reveal a significant "Technical Reliability Gap": while models achieve high Compilation Success Rates (CSR), dynamic analysis identified a frequent incidence of "Definitely Lost" heap memory and "logical hangs". We further identify a correlation between architectural paradigm and safety, finding that sparse Mixture-of-Experts (MoE) architectures exhibit lower Static Error Density (SED) and higher logic density than dense counterparts. We conclude by offering a framework for trust calibration based on these technical breaking points, assisting educators in mitigating the "oracle hazards" of AI-mediated instruction.} }
Endnote
%0 Conference Paper %T The Reliability Gap: A Multi-Dimensional Technical Audit of Memory Safety and Logical Integrity in LLM-Generated C Code %A Ricardo Jarrin %A Luiza Antonie %A Ritu Chaturvedi %B Proceedings of the The 39th Canadian Conference on Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Lydia Bouzar-Benlabiod %E Carson Leung %F pmlr-v318-jarrin26a %I PMLR %P 968--974 %U https://proceedings.mlr.press/v318/jarrin26a.html %V 318 %X The integration of Large Language Models (LLMs) into C programming education offers a scalable solution to the systemic enrollment crisis in computer science. However, the non-deterministic nature of these probabilistic engines introduces a critical reliability gap, particularly in C programming - a domain characterized by manual memory management and a high risk of Undefined Behavior (UB). This study presents a multi-dimensional technical audit - integrating static structural analysis with dynamic runtime integrity checks - of four 2026 frontier models: GPT-OSS 120B, Llama 3.3 70B, Moonshot Kimi K2, and Qwen 3 32B. Utilizing an automated evaluation pipeline, 1195 code generations across a gradient of complexity (Pointers, Dynamic Memory, and Data Structures) were subjected to static analysis and dynamic runtime instrumentation. Experimental results reveal a significant "Technical Reliability Gap": while models achieve high Compilation Success Rates (CSR), dynamic analysis identified a frequent incidence of "Definitely Lost" heap memory and "logical hangs". We further identify a correlation between architectural paradigm and safety, finding that sparse Mixture-of-Experts (MoE) architectures exhibit lower Static Error Density (SED) and higher logic density than dense counterparts. We conclude by offering a framework for trust calibration based on these technical breaking points, assisting educators in mitigating the "oracle hazards" of AI-mediated instruction.
APA
Jarrin, R., Antonie, L. & Chaturvedi, R.. (2026). The Reliability Gap: A Multi-Dimensional Technical Audit of Memory Safety and Logical Integrity in LLM-Generated C Code. Proceedings of the The 39th Canadian Conference on Artificial Intelligence, in Proceedings of Machine Learning Research 318:968-974 Available from https://proceedings.mlr.press/v318/jarrin26a.html.

Related Material