Concept Probing: Where to Find Human-Defined Concepts

Manuel de Sousa Ribeiro, Afonso Leote, Joao Leite
Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning, PMLR 284:237-251, 2025.

Abstract

Concept probing has recently gained popularity as a way for humans to peek into what is encoded within artificial neural networks. In concept probing, additional classifiers are trained to map the internal representations of a model into human-defined concepts of interest. However, the performance of these probes is highly dependent on the internal representations they probe from, making identifying the appropriate layer to probe an essential task. In this paper, we propose a method to automatically identify which layer’s representations in a neural network model should be considered when probing for a given human-defined concept of interest, based on how informative and regular the representations are with respect to the concept. We validate our findings through an exhaustive empirical analysis over different neural network models and datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v284-ribeiro25a, title = {Concept Probing: Where to Find Human-Defined Concepts}, author = {Ribeiro, Manuel de Sousa and Leote, Afonso and Leite, Joao}, booktitle = {Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning}, pages = {237--251}, year = {2025}, editor = {H. Gilpin, Leilani and Giunchiglia, Eleonora and Hitzler, Pascal and van Krieken, Emile}, volume = {284}, series = {Proceedings of Machine Learning Research}, month = {08--10 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v284/main/assets/ribeiro25a/ribeiro25a.pdf}, url = {https://proceedings.mlr.press/v284/ribeiro25a.html}, abstract = {Concept probing has recently gained popularity as a way for humans to peek into what is encoded within artificial neural networks. In concept probing, additional classifiers are trained to map the internal representations of a model into human-defined concepts of interest. However, the performance of these probes is highly dependent on the internal representations they probe from, making identifying the appropriate layer to probe an essential task. In this paper, we propose a method to automatically identify which layer’s representations in a neural network model should be considered when probing for a given human-defined concept of interest, based on how informative and regular the representations are with respect to the concept. We validate our findings through an exhaustive empirical analysis over different neural network models and datasets.} }
Endnote
%0 Conference Paper %T Concept Probing: Where to Find Human-Defined Concepts %A Manuel de Sousa Ribeiro %A Afonso Leote %A Joao Leite %B Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning %C Proceedings of Machine Learning Research %D 2025 %E Leilani H. Gilpin %E Eleonora Giunchiglia %E Pascal Hitzler %E Emile van Krieken %F pmlr-v284-ribeiro25a %I PMLR %P 237--251 %U https://proceedings.mlr.press/v284/ribeiro25a.html %V 284 %X Concept probing has recently gained popularity as a way for humans to peek into what is encoded within artificial neural networks. In concept probing, additional classifiers are trained to map the internal representations of a model into human-defined concepts of interest. However, the performance of these probes is highly dependent on the internal representations they probe from, making identifying the appropriate layer to probe an essential task. In this paper, we propose a method to automatically identify which layer’s representations in a neural network model should be considered when probing for a given human-defined concept of interest, based on how informative and regular the representations are with respect to the concept. We validate our findings through an exhaustive empirical analysis over different neural network models and datasets.
APA
Ribeiro, M.d.S., Leote, A. & Leite, J.. (2025). Concept Probing: Where to Find Human-Defined Concepts. Proceedings of The 19th International Conference on Neurosymbolic Learning and Reasoning, in Proceedings of Machine Learning Research 284:237-251 Available from https://proceedings.mlr.press/v284/ribeiro25a.html.

Related Material