Probing Graph Representations

Mohammad Sadegh Akhondzadeh, Vijay Lingam, Aleksandar Bojchevski
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:11630-11649, 2023.

Abstract

Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models, and show that Graph Transformers capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating and developing graph-based models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-akhondzadeh23a, title = {Probing Graph Representations}, author = {Akhondzadeh, Mohammad Sadegh and Lingam, Vijay and Bojchevski, Aleksandar}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {11630--11649}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/akhondzadeh23a/akhondzadeh23a.pdf}, url = {https://proceedings.mlr.press/v206/akhondzadeh23a.html}, abstract = {Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models, and show that Graph Transformers capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating and developing graph-based models.} }
Endnote
%0 Conference Paper %T Probing Graph Representations %A Mohammad Sadegh Akhondzadeh %A Vijay Lingam %A Aleksandar Bojchevski %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-akhondzadeh23a %I PMLR %P 11630--11649 %U https://proceedings.mlr.press/v206/akhondzadeh23a.html %V 206 %X Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) isomorphism tests. However, we do not know what is encoded in the learned representations. This is our main question. We answer it using a probing framework to quantify the amount of meaningful information captured in graph representations. Our findings on molecular datasets show the potential of probing for understanding the inductive biases of graph-based models. We compare different families of models, and show that Graph Transformers capture more chemically relevant information compared to models based on message passing. We also study the effect of different design choices such as skip connections and virtual nodes. We advocate for probing as a useful diagnostic tool for evaluating and developing graph-based models.
APA
Akhondzadeh, M.S., Lingam, V. & Bojchevski, A.. (2023). Probing Graph Representations. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:11630-11649 Available from https://proceedings.mlr.press/v206/akhondzadeh23a.html.

Related Material