Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts

Emanuele MarconatoStefano TesoAntonio VergariAndrea Passerini

article2023NeurIPS81 citations

Presents a formal framework to identify why neuro-symbolic models exploit unintended reasoning shortcuts despite high training accuracy, evaluating theoretical and empirical strategies to mitigate these shortcuts and ensure reliable concept learning.

Listen

Neuro-Symbolic artificial intelligence integrates neural networks with symbolic reasoning rules to create transparent, constraint-compliant predictive models. These systems are widely envisioned for high-stakes domains such as medical diagnosis and autonomous driving because they infer outcomes by reasoning over high-level concepts extracted from sensory data. However, recent evidence indicates that these predictors frequently suffer from reasoning shortcuts: unintended solutions where a model attains near-perfect task accuracy while learning concepts with incorrect, unintended semantics. This failure undermines the safety, verification, and real-world generalizability promised by such architectures.

The main objective of the article is to provide a formal mathematical characterization of reasoning shortcuts across neuro-symbolic predictors and systematically evaluate the theoretical and empirical effectiveness of multiple mitigation strategies.

To conduct this evaluation, the article develops a theoretical framework linking training objectives to data generation properties and tests representative neuro-symbolic methods, including DeepProbLog, Semantic Loss, and Logic Tensor Networks. The authors examine these methods across four distinct synthetic and benchmark datasets—ranging from controlled XOR and arithmetic tasks to real-world autonomous driving scenes in BDD-OIA—while testing various interventions such as architectural disentanglement, multi-task learning, unsupervised input reconstruction, entropy regularization, and direct concept supervision.

The analysis yields four central findings. First, reasoning shortcuts represent unintended global optima of the learning objective that arise even when training data is completely exhaustive and unbiased; in baseline XOR and arithmetic tests, 83% to 100% of standard models converged to reasoning shortcuts. Second, enforcing architectural disentanglement successfully eliminated shortcuts in exhaustive settings, dropping shortcut occurrence to 0%, but proved insufficient when data suffered from selection bias. Third, unsupervised techniques such as input reconstruction and entropy regularization failed to resolve shortcuts on their own and frequently degraded overall task performance. Fourth, multi-task learning and direct concept supervision proved to be the most robust remedies, restoring concept accuracy to over 98% in biased arithmetic setups and substantially improving concept quality in complex driving tasks.

These findings demonstrate that high prediction accuracy on a validation set offers a false sense of safety in neuro-symbolic systems, as models can achieve top-tier performance while relying on flawed internal logic. Unsupervised heuristics cannot be trusted alone to enforce correct concept acquisition in high-stakes environments, potentially leading to critical failures when models encounter out-of-distribution scenarios or when learned concepts are transferred to new tasks.

Decision-makers should immediately cease relying solely on downstream task accuracy to validate neuro-symbolic systems. Instead, deployment pipelines must incorporate explicit concept verification alongside multi-task learning or targeted, partial concept supervision during training. Before establishing universal deployment policies, organizations should conduct empirical pilots on domain-specific data, as the efficacy of mitigation strategies depends heavily on model architecture and data distributions, and no universal single-technique solution currently exists.

reasoning reasoning shortcuts neuro-symbolic learning neural probabilistic logic

arXiv: 2305.19951

Paper: Shortcut learning in deep neural networks, Robert Geirhos et al. (2020). Provides the foundational taxonomy and conceptual definition of shortcut learning in neural networks that the source paper directly formalizes and investigates within neuro-symbolic architectures.
Paper: Concept Bottleneck Models, Pang Wei Koh et al. (2020). Establishes the concept bottleneck architecture and concept-based intermediate representation paradigm upon which neuro-symbolic reasoning predictors and their concept learning failures are evaluated.
Paper: Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, Shiori Sagawa et al. (2019). Analyzes how neural networks optimize for spurious correlations under selection biases and why standard loss objectives fail to prevent shortcut behavior.
Paper: Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference, R. Thomas McCoy et al. (2019). Demonstrates how models achieve high task accuracy through unintended heuristic shortcuts rather than intended semantic reasoning, motivating the source paper's study of reasoning shortcuts.
Paper: Annotation Artifacts in Natural Language Inference Data, Suchin Gururangan et al. (2018). Highlights the vulnerability of deep learning models to learning superficial artifacts while maintaining high benchmark accuracy, directly framing the problem of deceptive downstream performance.
Paper: Network Dissection: Quantifying Interpretability of Deep Visual Representations, David Bau et al. (2017). Pioneers quantitative metrics for assessing whether internal neural representations correspond to true human-interpretable concepts rather than entangled or shortcut features.

Paper: Efficient Rectification of Neuro-Symbolic Reasoning Inconsistencies by Abductive Reflection, Wen-Chao Hu et al. (2025). Develops an abductive reflection framework that efficiently detects and rectifies symbolic reasoning inconsistencies in neuro-symbolic architectures.
Paper: Feedback Loops With Language Models Drive In-Context Reward Hacking, Alexander Pan et al. (2024). Extends the study of unintended optimization shortcuts by analyzing how interaction feedback loops drive models to hack proxy reward objectives at inference time.
Paper: The Dead Salmons of AI Interpretability, Maxime Méloux et al. (2025). Critically examines the theoretical non-identifiability and unreliability of internal representations and interpretability methods when verifying intermediate model concepts.
Paper: Rethinking Tabular Data Understanding with Large Language Models, Tianyang Liu et al. (2024). Investigates how structural perturbations and symbolic versus textual representations affect the fidelity of reasoning in structured domains.

Topics

reasoning

Keywords

reasoning shortcuts neuro-symbolic learning neural probabilistic logic Logic Tensor Networks concept supervision concept bottleneck semantic loss knowledge compilation logical constraints systematic generalization

1 Introduction
2 The Family of Neuro-Symbolic Predictors
3 Reasoning Shortcuts as Unintended Optima
4 Properties of Reasoning Shortcuts
5 Analysis of Mitigation Strategies
5.1 Knowledge-based Mitigation
5.2 Data-based Mitigation
5.3 Objective-based Mitigation
5.4 Architecture-based Mitigation
5.5 Other Heuristics based on Entropy Regularization
6 Case Studies
7 Related Work
8 Conclusion
Acknowledgments and Disclosure of Funding
References
A Other NeSy Predictors
A.1 Deterministic Optima are Shared
B Proofs
B.1 Proof of Lemma 1: Upper Bound of the Log-Likelihood
B.2 Proof of Theorem 2: Counting the Deterministic Optima
B.3 Proof of Proposition 3: Link Between Deterministic and Non-deterministic Optima
B.5 Proof of Proposition 5: Concept Supervision
B.6 Proof of Proposition 6: Reconstruction
C Experimental Details and Further Results
C.1 Implementation
C.2 Data sets & Count of the Reasoning Shortcuts
C.2.1 Dataset: XOR
C.2.2 Dataset: MNIST-Addition
C.2.3 Dataset: MNIST-EvenOdd
C.2.4 Dataset: MNIST-AddMul
C.2.5 Dataset: BDD-OIA
C.3 Optimizer and Hyper-parameter Selection
C.4 Architectures
C.5 Confusion Matrices
C.6 XOR
C.7 MNIST-EvenOdd
C.8 MNIST-AddMul

Knowls

Knowl 1 — Reasoning shortcuts are unintended optimal concept representations
definition
A neuro-symbolic predictor receives a sub-symbolic input $X$ , extracts discrete latent concepts $C$ , and predicts labels $Y$ using prior logical knowledge $K$ . In the DeepProbLog-style formulation, the label distribution is
$p_\theta(y\mid x;K)=\sum_{c\in\mathcal C}u_K(y\mid c)\,p_\theta(c\mid x),$
where $\mathcal C$ and $\mathcal Y$ are finite concept and label spaces, $p_\theta(C\mid X)$ is the neural concept extractor, and $u_K(y\mid c)$ is the reasoning-layer distribution over labels compatible with $K$ and concepts $c$ . Given a training set $D=\{(x,y)\}$ , the training objective is the average log-likelihood
$\mathcal L(p_\theta,D,K)=\frac{1}{|D|}\sum_{(x,y)\in D}\log p_\theta(y\mid x;K).$
Let $p^*(G\mid X)$ denote the ground-truth distribution of the semantic concepts $G$ that generate the data. A reasoning shortcut is any concept distribution $p_\theta(C\mid X)$ that both maximizes the training objective and differs from the ground-truth concept distribution:
$\mathcal L(p_\theta,D,K)=\max_{\theta'\in\Theta}\mathcal L(p_{\theta'},D,K) \quad\text{and}\quad p_\theta(C\mid X)\not\equiv p^*(G\mid X).$
Thus, a reasoning shortcut can attain optimal label likelihood while assigning concepts unintended semantics. Such a representation may fail when reused for another task or when inputs involve concept combinations absent from the training distribution, even though label accuracy on the original task is high.
Knowl 2 — Ground-truth process and assumptions for the shortcut analysis
assumption
The analysis assumes that unobserved discrete ground-truth concepts $G=(G_1,\ldots,G_k)$ generate both the observed input $X$ and the label $Y$ , while an independent continuous style variable $S\in\mathbb R^q$ affects $X$ but not $Y$ . The concept space is $\mathcal G=[m_1]\times\cdots\times[m_k]$ . Data are generated by sampling $g$ and $s$ , then sampling $x$ from $p^*(X\mid g,s)$ and $y$ from $p^*(Y\mid g;K)$ ; the ground-truth process never produces a label that violates the prior knowledge $K$ .

The principal theoretical results use two assumptions. A1 states that the input-generation distribution is deterministic, $x=f(g,s)$ , where $f$ is invertible in $(g,s)$ and smooth in $s$ . A2 states that the label is a deterministic function of the ground-truth concepts, $y=\beta_K(g)$ , for a map $\beta_K:\mathcal G\to\mathcal Y$ . The intended concept semantics are recovered exactly when
$p_\theta(C\mid x)\equiv p^*(G\mid x)\qquad\text{for every }x.$
The support $\operatorname{supp}(G)$ is the set of ground-truth concept vectors that occur under the learning distribution. Shortcuts can arise when multiple concept assignments produce the same knowledge-consistent label on this support.
Knowl 3 — Optimal observed-input likelihood is linked to optimal ground-truth-concept likelihood
theoretical result
Under the ground-truth process above, the expected log-likelihood of a neuro-symbolic predictor based on observed inputs is bounded by a ground-truth-concept objective:
$\mathbb E_{(x,y)\sim p^*(X,Y;K)}[\log p_\theta(y\mid x;K)] \leq \mathbb E_{g\sim p^*(G)}\left[ -\operatorname{KL}\!\left(p^*(Y\mid g;K)\,\|\,p_\theta(Y\mid g;K)\right) -H\!\left(p^*(Y\mid g;K)\right) \right].$
Here $p_\theta(Y\mid g;K)$ is the predictor obtained when the ground-truth concepts are supplied directly, $\operatorname{KL}$ is the Kullback–Leibler divergence, and $H$ is Shannon entropy. Under A1 and A2, optimizing the observed-input likelihood is equivalent to optimizing the right-hand-side ground-truth-concept objective: a predictor is optimal as a function of $X$ if and only if its induced predictor as a function of $G$ is optimal. This result permits the shortcut analysis to replace the neural input-to-concept mapping with a mapping directly from ground-truth concepts to learned concepts.
Knowl 4 — The number of deterministic reasoning shortcuts is a model-counting quantity
theoretical result
Assume A1 and A2. Let $\mathcal A$ be the set of all deterministic concept maps $\alpha:\mathcal G\to\mathcal C$ , where each map induces $p_\theta(C\mid G=g)=\mathbf 1\{C=\alpha(g)\}$ . The number of deterministic likelihood optima is
$\sum_{\alpha\in\mathcal A} \mathbf 1\!\left\{ \bigwedge_{g\in\operatorname{supp}(G)} (\beta_K\circ\alpha)(g)=\beta_K(g) \right\}.$
The condition requires every learned concept vector $\alpha(g)$ to imply the correct label for every ground-truth vector $g$ occurring in the data. The ground-truth map is one such optimum, but every additional satisfying map is a deterministic reasoning shortcut. Consequently, the number of shortcuts depends on four factors: the prior knowledge $K$ , the support of the ground-truth data distribution, the likelihood objective used for training, and the set $\mathcal A$ of maps realizable by the concept-extractor architecture. In particular, shortcuts can remain even with exhaustive data when the knowledge maps several distinct concept vectors to the same label.
Knowl 5 — Deterministic shortcuts transfer across several neuro-symbolic predictors and generate non-deterministic optima
theoretical result
Under A2, a deterministic concept assignment that is optimal for the DeepProbLog likelihood is also a deterministic optimum for Semantic Loss and Logic Tensor Networks, because all three methods attain their best objective value whenever the predicted concepts support the correct knowledge-consistent label. Thus, deterministic reasoning shortcuts are principally a property of the concept ambiguity induced by the prior knowledge, not of the specific reasoning relaxation.

For probabilistic-logic predictors including DeepProbLog and Semantic Loss, convex combinations of deterministic likelihood optima are also likelihood optima. Under A1 and A2, every likelihood optimum can be represented as a convex combination of deterministic optima. Therefore, two or more deterministic shortcuts imply infinitely many non-deterministic optima, although factorized concept extractors used by standard DeepProbLog and Semantic Loss cannot necessarily represent every such combination. If A2 is violated, additional non-deterministic optima may exist that are not convex combinations of deterministic optima and may be unaffected by methods targeting only deterministic shortcuts. The convex-combination claim does not generally hold for Logic Tensor Networks, whose fuzzy satisfaction function is nonlinear.
Knowl 6 — Multi-task learning removes maps inconsistent with any shared task
theoretical result
Suppose $T$ neuro-symbolic tasks share the same ground-truth concepts $G$ and data support but use different prior knowledge functions $\beta_{K^{(t)}}$ , for $t\in[T]$ . Under A1 and A2, a deterministic map $\alpha:\mathcal G\to\mathcal C$ is an optimum of the average multi-task likelihood only if it satisfies every task on every observed ground-truth concept:
$\sum_{\alpha\in\mathcal A} \mathbf 1\!\left\{ \bigwedge_{g\in\operatorname{supp}(G)} \bigwedge_{t=1}^{T} (\beta_{K^{(t)}}\circ\alpha)(g)=\beta_{K^{(t)}}(g) \right\}$
counts the deterministic optima. Multi-task learning therefore behaves like a conjunction of the task constraints: any representation that gives an incorrect result for even one task is excluded from the optimum set. This can sharply reduce reasoning shortcuts, but it requires constructing or collecting multiple correlated tasks that share the same underlying concepts.
Knowl 7 — Partial concept supervision constrains shortcut maps on annotated examples and dimensions
theoretical result
Let $I\subseteq[k]$ index the concept dimensions that receive supervision, and let $S\subseteq\operatorname{supp}(G)$ be the subset of ground-truth concept vectors with annotations. Adding a concept cross-entropy objective requires a deterministic map $\alpha:\mathcal G\to\mathcal C$ to satisfy
$\bigwedge_{g\in S}\;\bigwedge_{i\in I}\alpha_i(g)=g_i.$
Under A1, the number of deterministic optima of the concept-supervision objective is
$\sum_{\alpha\in\mathcal A} \mathbf 1\!\left\{ \bigwedge_{g\in S}\;\bigwedge_{i\in I}\alpha_i(g)=g_i \right\}.$
When concept supervision covers every dimension, every ground-truth vector, and a complete support, only the identity semantics remain among deterministic maps. The result explains the trade-off: concept annotations are a direct and powerful remedy, but dense annotations over all examples and concept dimensions may be impractical. When combined with the logical likelihood, both the knowledge-consistency and concept-supervision constraints must hold.
Knowl 8 — Reconstruction discourages many-to-one concept shortcuts under content–style separation
theoretical result
A reconstruction-based mitigation augments the concept extractor with a latent style variable $Z$ and factorizes the encoder as $p_\theta(c,z\mid x)=p_\theta(c\mid x)p_\theta(z\mid x)$ . With decoder $p_\psi(x\mid c,z)$ , the reconstruction penalty is
$\mathcal R(x)=-\mathbb E_{(c,z)\sim p_\theta(C,Z\mid x)}[\log p_\psi(x\mid c,z)].$
In addition to A1, assume A3: the encoder separates content from style, $p_\theta(C,Z\mid G,S)=p_\theta(C\mid G)p_\theta(Z\mid S)$ , and the decoder separates their reconstruction, $p_\psi(G,S\mid C,Z)=p_\psi(G\mid C)p_\psi(S\mid Z)$ . Under A1 and A3, a deterministic map $\alpha:\operatorname{supp}(G)\to\mathcal C$ can minimize reconstruction only if it is injective:
$\bigwedge_{g,g'\in\operatorname{supp}(G):\,g\neq g'}\alpha(g)\neq\alpha(g').$
The number of deterministic reconstruction optima is therefore
$\sum_{\alpha\in\mathcal A} \mathbf 1\!\left\{ \bigwedge_{g,g'\in\operatorname{supp}(G):\,g\neq g'}\alpha(g)\neq\alpha(g') \right\}.$
Reconstruction prevents different ground-truth concepts from collapsing to one learned concept, but it can be difficult to optimize for complex inputs and does not by itself guarantee the intended semantic assignment.
Knowl 9 — Disentangled concept extractors reduce the space of candidate shortcut maps
model/method
A concept extractor is disentangled when its induced ground-truth-to-learned-concept distribution factorizes across corresponding dimensions:
$p_\theta(C\mid G)=\prod_{j=1}^{k}p_\theta(C_j\mid G_j).$
For deterministic representations, this restricts an arbitrary map $\alpha:G\to C$ to independent per-concept maps $\alpha_j:[m_j]\to[m_j]$ . The architecture therefore greatly reduces the number of realizable maps counted by the shortcut theorem. When the underlying concepts are naturally independent, the restriction can be implemented by predicting each concept with a shared network applied separately to its input component. Disentanglement is not a universal guarantee under selection bias, because multiple per-concept maps can still satisfy an incomplete set of logical constraints, but it can eliminate shortcuts in exhaustive tasks such as the XOR and full MNIST-Addition settings.
Knowl 10 — Exhaustive-data experiments show shortcuts across three neuro-symbolic predictors and their removal by disentanglement
empirical result
The authors tested DeepProbLog (DPL), Semantic Loss (SL), and Logic Tensor Networks (LTN) on exhaustive XOR and MNIST-Addition data. Each model was trained from different random seeds until 30 runs reached likelihood at least $0.95$ ; the reported value is the percentage of these near-optimal runs whose concepts were reasoning shortcuts. DIS denotes an architecture enforcing disentanglement, and lower percentages are better.
Could not parse LaTeX table
All three unconstrained predictors acquired shortcuts despite exhaustive coverage of the possible ground-truth concept combinations. Enforcing disentanglement reduced the shortcut frequency to zero in all 30-run evaluations. The result demonstrates that more data alone does not resolve ambiguity created by the reasoning task, whereas an architectural restriction can do so in these two settings.
Knowl 11 — Selection bias and real-world data make mitigation effectiveness task- and model-dependent
empirical result
On MNIST-EvenOdd, only 16 of the 100 possible digit pairs were used for training, and all tested models were disentangled. Results are macro-averaged test-set F1 scores over 10 runs; $F1(Y)$ measures labels and $F1(C)$ measures concepts. Here R is reconstruction, C is concept supervision, and H is Shannon entropy regularization.
Could not parse LaTeX table
No single unsupervised or supervised strategy reliably removed shortcuts for every predictor. Combining strategies improved concepts for DPL and LTN, especially H+C and R+H+C, while SL remained comparatively resistant to the unsupervised remedies. On MNIST-AddMul, where the model must predict digit sums and products from a restricted set of pairs, separate single-task training produced shortcut concepts: for ADD, DPL, SL, and LTN obtained $(F1(Y),F1(C))$ of $(68.1\pm6.7,0.0\pm0.0)$ , $(99.5\pm0.2,0.0\pm0.1)$ , and $(67.4\pm0.1,0.0\pm0.0)$ ; for MULT, the corresponding values were $(100.0\pm0.0,37.6\pm0.2)$ , $(100.0\pm0.0,76.1\pm11.7)$ , and $(98.1\pm0.5,78.1\pm0.4)$ . Joint multi-task training raised concept F1 to $99.8\pm0.1$ , $99.8\pm0.1$ , and $98.3\pm0.2$ for DPL, SL, and LTN, respectively, while retaining label F1 of $100.0\pm0.0$ , $100.0\pm0.0$ , and $98.3\pm0.2$ .

On the real-world BDD-OIA autonomous-driving task, DPL predicted four actions from 21 binary concepts under safety constraints. The following are test-set F1 scores averaged over 10 runs:
Could not parse LaTeX table
DPL achieved high action accuracy while learning substantially poorer concepts unless concept supervision was supplied. Entropy regularization alone did not prevent this mismatch. Concept supervision improved concept F1 but slightly reduced label F1, and it failed for the turn-left concepts because some corresponding annotations were systematically incorrect. These experiments support the paper's conclusion that reasoning shortcuts are pervasive and that no broadly applicable mitigation recipe was established.

Coverage note — Detailed optimizer settings, neural-layer specifications, dataset-specific shortcut counts, and the entropy heuristic's implementation details were omitted because they support the central theory and experiments rather than constituting separate load-bearing contributions.

References

1.Luc De Raedt, Sebastijan Dumanciˇ c, Robin Manhaeve, and Giuseppe Marra. From statistical relational to neural-symbolic artificial intelligence. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 4943–4950, 2021.
2.Artur d’Avila Garcez, Sebastian Bader, Howard Bowman, Luis C Lamb, Leo de Penning, BV Illuminoo, Hoifung Poon, and COPPE Gerson Zaverucha. Neural-symbolic learning and reasoning: A survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art, 342:1, 2022.
3.Eleonora Giunchiglia, Mihaela Catalina Stoian, and Thomas Lukasiewicz. Deep learning with logical constraints. arXiv preprint arXiv:2205.00523, 2022.
4.Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports, 12 (1):1–15, 2022.
5.Michelangelo Diligenti, Marco Gori, and Claudio Sacca. Semantic-based regularization for learning and inference. Artificial Intelligence, 2017.
6.Ivan Donadello, Luciano Serafini, and Artur D’Avila Garcez. Logic tensor networks for semantic image interpretation. In IJCAI, 2017.
7.Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. DeepProbLog: Neural Probabilistic Logic Programming. NeurIPS, 2018.
8.Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck. A semantic loss function for deep learning with symbolic knowledge. In ICML, 2018.
9.Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. NeurIPS, 2020.
10.Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic Probabilistic Layers for Neuro-Symbolic Learning. In NeurIPS, 2022.
11.Zenan Li, Zehua Liu, Yuan Yao, Jingwei Xu, Taolue Chen, Xiaoxing Ma, L Jian, et al. Learning with logical constraints but without shortcut satisfaction. In The Eleventh International Conference on Learning Representations, 2023.
12.Xuan Xie, Kristian Kersting, and Daniel Neider. Neuro-symbolic verification of deep neural networks. 2022.
13.Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215, 2019.
14.Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: Deep learning for interpretable image recognition. NeurIPS, 2019.
15.Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2020.
16.Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, pages 1–10, 2021.
17.Elisa Maiettini, Giulia Pasquale, Vadim Tikhanoff, Lorenzo Rosasco, and Lorenzo Natale. A weakly supervised strategy for learning object detection on a humanoid robot. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages 194–201. IEEE, 2019.
18.Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert Systems with Applications, 165:113816, 2021.
19.Matt Fredrikson, Kaiji Lu, Saranya Vijayakumar, Somesh Jha, Vijay Ganesh, and Zifan Wang. Learning modulo theories. arXiv preprint arXiv:2301.11435, 2023.
20.Robin Manhaeve, Sebastijan Dumanciˇ c, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Neural probabilistic logic programming in deepproblog. Artificial Intelligence, 298:103504, 2021.
21.Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso. Neuro symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal. In ICML, 2023.
22.Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
23.Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev. Dl2: Training and querying neural networks with logic. In International Conference on Machine Learning, pages 1931–1941. PMLR, 2019.
24.Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Neuro-symbolic entropy regularization. In UAI, 2022.
25.Nick Hoernle, Rafael Michael Karampatsis, Vaishak Belle, and Kobi Gal. Multiplexnet: Towards fully satisfied logical constraints in neural networks. In AAAI, 2022.
26.Michelangelo Diligenti, Marco Gori, Marco Maggini, and Leonardo Rigutini. Bridging logic and kernel machines. Machine learning, 86(1):57–88, 2012.
27.Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, and Lise Getoor. Neupsl: Neural probabilistic soft logic. arXiv preprint arXiv:2205.14268, 2022.
28.Luc De Raedt and Angelika Kimmig. Probabilistic (logic) programming concepts. Machine Learning, 2015.
29.Adnan Darwiche and Pierre Marquis. A knowledge compilation map. Journal of Artificial Intelligence Research, 17:229–264, 2002.
30.Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. Advances in Neural Information Processing Systems, 34, 2021.
31.Robin Manhaeve, Giuseppe Marra, and Luc De Raedt. Approximate inference for neural probabilistic logic programming. In KR, 2021.
32.Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si. Scallop: From probabilistic deductive databases to scalable differentiable reasoning. NeurIPS, 2021.
33.Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. DeepStochLog: Neural Stochastic Logic Programming. In AAAI, 2022.
34.Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije. A-nesi: A scalable approximate method for probabilistic neurosymbolic inference. arXiv preprint arXiv:2212.12393, 2022.
35.Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.
36.Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pages 6056–6065. PMLR, 2019.
37.Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.
38.Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
39.Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretabile, leak-proof concept-based models. NeurIPS, 2022.
40.Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances in neural information processing systems, 6, 1993.
41.Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, 2014.
42.Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International conference on machine learning, 2014.
43.Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael J Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. In ICLR, 2020.
44.Eleonora Misino, Giuseppe Marra, and Emanuele Sansone. VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming. NeurIPS, 2022.
45.Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML, 2019.
46.Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348–6359. PMLR, 2020.
47.Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In ICLR, 2019.
48.Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
49.Shuang Li, Yilun Du, Gido Martijn van de Ven, and Igor Mordatch. Energy-based models for continual learning. In Energy Based Models Workshop-ICLR 2021, 2021.
50.Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu. Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning, pages 5884–5894. PMLR, 2020.
51.Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Learning, pages 23631–23644. PMLR, 2022.
52.Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288–15299, 2020.
53.Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regularization. The Journal of Machine Learning Research, 23(1):14632–14662, 2022.
54.Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
55.Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758–41765, 2022.
56.Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673, 2020.
57.Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1–8, 2019.
58.Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR, 2020.
59.Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476–486, 2020.
60.Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez. Right for the right reasons: training differentiable models by constraining their explanations. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 2662–2670, 2017.
61.Giambattista Parascandolo, Alexander Neitz, ANTONIO ORVIETO, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2020.
62.Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly. Leveraging explanations in interactive machine learning: An overview. Frontiers in Artificial Intelligence, 2023.
63.Wolfgang Stammer, Marius Memmel, Patrick Schramowski, and Kristian Kersting. Interactive disentanglement: Learning concepts by interacting with their prototype representations. arXiv preprint arXiv:2112.02290, 2021.
64.Kaifu Wang, Efi Tsamoura, and Dan Roth. On learning latent models with multi-instance weak supervision. arXiv preprint arXiv:2306.13796, 2023.
65.Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
66.Aapo Hyvärinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning. Patterns, 4(10), 2023.
67.Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: A unifying framework. In AISTATS, 2020.
68.Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems, 35:16946–16961, 2022.
69.Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems, 34:28233–28248, 2021.
70.Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International conference on machine learning, pages 372–407. PMLR, 2023.
71.Wendong Liang, Armin Kekic, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. arXiv preprint arXiv:2305.17225, 2023.
72.Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Kekic, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. arXiv preprint arXiv:2306.00542, 2023.
73.Łukasz Maziarka, Aleksandra Nowak, Maciej Wołczyk, and Andrzej Bedychaj. On the relationship between disentanglement and multi-task learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 625–641. Springer, 2022.
74.Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: generalization and identifiability in multi-task learning. In International Conference on Machine Learning, pages 18171–18206. PMLR, 2023.
75.Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. arXiv preprint arXiv:2304.07939, 2023.
76.Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020.
77.Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit B Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. In Proceedings of the 37th International Conference on Machine Learning, pages 7360–7369, 2020.
78.Hao Tang and Kevin Ellis. From perception to programs: regularize, overparameterize, and amortize. In International Conference on Machine Learning, pages 33616–33631. PMLR, 2023.
79.Alessandro Daniele, Tommaso Campari, Sagar Malhotra, and Luciano Serafini. Deep symbolic learning: Discovering symbols and rules from perceptions. arXiv preprint arXiv:2208.11561, 2022.
80.Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338–5348. PMLR, 2020.
81.Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056, 2022.
82.Francesco Giannini, Michelangelo Diligenti, Marco Gori, and Marco Maggini. On a convex logic fragment for learning and reasoning. IEEE Transactions on Fuzzy Systems, 2018.
83.Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
84.Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. A pseudo-semantic loss for deep generative models with logical constraints. In Knowledge and Logical Reasoning in the Era of Data-driven Learning Workshop, July 2023.
85.Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshien, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
86.Tommaso Carraro. LTNtorch: PyTorch implementation of Logic Tensor Networks, mar 2022. URL https://doi.org/10.5281/zenodo.6394282.
87.Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
88.Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.

Citation

MLA

Marconato, E., et al. “Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts”. Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 72507–39, https://proceedings.neurips.cc/paper_files/paper/2023/file/e560202b6e779a82478edb46c6f8f4dd-Paper-Conference.pdf.

APA

Marconato, E., Teso, S., Vergari, A., & Passerini, A. (2023). Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts. Advances in Neural Information Processing Systems, 36, 72507–72539. https://proceedings.neurips.cc/paper_files/paper/2023/file/e560202b6e779a82478edb46c6f8f4dd-Paper-Conference.pdf

Chicago

Marconato, E., S. Teso, A. Vergari, and A. Passerini. 2023. “Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts”. Advances in Neural Information Processing Systems 36: 72507–39. https://proceedings.neurips.cc/paper_files/paper/2023/file/e560202b6e779a82478edb46c6f8f4dd-Paper-Conference.pdf.

Harvard

Marconato, E. et al. (2023) “Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts”, Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 72507–72539. Available at: https://proceedings.neurips.cc/paper_files/paper/2023/file/e560202b6e779a82478edb46c6f8f4dd-Paper-Conference.pdf.

Vancouver

1. Marconato E, Teso S, Vergari A, Passerini A (2023) Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 72507–72539

BibTeX

@inproceedings{marconato2023not,
  title = {Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts},
  author = {Marconato, Emanuele and Teso, Stefano and Vergari, Antonio and Passerini, Andrea},
  year = {2023},
  booktitle = {Advances in Neural Information Processing Systems},
  publisher = {Curran Associates, Inc.},
  volume = {36},
  pages = {72507-72539},
  url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/e560202b6e779a82478edb46c6f8f4dd-Paper-Conference.pdf}
}

Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Reasoning shortcuts are unintended optimal concept representations

Knowl 2 — Ground-truth process and assumptions for the shortcut analysis

Knowl 3 — Optimal observed-input likelihood is linked to optimal ground-truth-concept likelihood

Knowl 4 — The number of deterministic reasoning shortcuts is a model-counting quantity

Knowl 5 — Deterministic shortcuts transfer across several neuro-symbolic predictors and generate non-deterministic optima

Knowl 6 — Multi-task learning removes maps inconsistent with any shared task

Knowl 7 — Partial concept supervision constrains shortcut maps on annotated examples and dimensions

Knowl 8 — Reconstruction discourages many-to-one concept shortcuts under content–style separation

Knowl 9 — Disentangled concept extractors reduce the space of candidate shortcut maps

Knowl 10 — Exhaustive-data experiments show shortcuts across three neuro-symbolic predictors and their removal by disentanglement

Knowl 11 — Selection bias and real-world data make mitigation effectiveness task- and model-dependent

References

Citation

BibTeX

Access the Paper