JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering

Yueqing SunQi ShiLe QiYu Zhang

article2022NAACL94 citations

Proposes JointLK, a model that unites pretrained language models and graph neural networks through fine-grained bidirectional attention and dynamic node pruning to filter irrelevant knowledge and deliver interpretable commonsense question answering.

Harbin Institute of Technology Research Center for Social Computing and Information Retrieval common-sense graph neural networks knowledge graphs

arXiv: 2112.02732 Yueqing-Sun/JointLK

Cover for JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering

Abstract

Existing KG-augmented models for common-sense question answering primarily focus on designing elaborate Graph Neural Networks (GNNs) to model knowledge graphs (KGs). However, they ignore (i) the effectively fusing and reasoning over question context representations and the KG representations, and (ii) automatically selecting relevant nodes from the noisy KGs during reasoning. In this paper, we propose a novel model, JointLK, which solves the above limitations through the joint reasoning of LM and GNN and the dynamic KGs pruning mechanism. Specifically, JointLK performs joint reasoning between LM and GNN through a novel dense bidirectional attention module, in which each question token attends on KG nodes and each KG node attends on question tokens, and the two modal representations fuse and update mutually by multi-step interactions. Then, the dynamic pruning module uses the attention weights generated by joint reasoning to prune irrelevant KG nodes recursively. We evaluate JointLK on the CommonsenseQA and OpenBookQA datasets, and demonstrate its improvements to the existing LM and LM+KG models, as well as its capability to perform interpretable reasoning¹.

Topics

common-sense graph neural networks knowledge graphs language model question answering reasoning bipartite graphs

1 Introduction
2 Related Work
3 Methodology
3.1 Task Definition
3.2 Query Encoder
3.3 GNN Layer
3.5 Dynamic Pruning Module
3.6 Answer Prediction
4 Experimental Setup
4.1 Datasets
4.2 Implementation Details
4.3 Compared Method
5 Results and Analysis
5.1 Main Results
5.2 Ablation Studies
5.3 Quantitative Analysis
5.5 Error Analysis
6 Conclusion
Acknowledgements
Ethical Impact
References
A Extracting subgraph from External KG
B Node Initialization
C Error Types and Examples

Knowls

Knowl 1 — JointLK Framework for Joint Reasoning between Language Models and Knowledge Graphs
model/method
JointLK is a neural architecture for commonsense question answering that performs multi-step joint reasoning and fine-grained cross-modal fusion between a pre-trained language model (LM) and an external knowledge graph (KG). Given a commonsense multiple-choice question $q$ , candidate answer choices $\{a_1, \dots, a_n\}$ , and a retrieved KG subgraph $g=(V, R)$ with entities $V$ and relation types $R$ , JointLK encodes the query (the concatenated text of the question and a candidate choice) with an LM encoder and the subgraph with a graph neural network (GNN).

The core of JointLK consists of $N$ stacked, identical JointLK layers. Each layer $l \in \{1, \dots, N\}$ executes three sequential operations:
1. GNN Layer: Updates entity node representations via relation-aware graph message passing on the current graph topology.
2. Joint Reasoning Module: Computes a dense bidirectional attention matrix between all query token representations and all KG entity node representations, updating query token features with graph context and KG node features with question context.
3. Dynamic Pruning Module: Uses the token-to-node attention weights from the joint reasoning module to prune uninformative nodes and compress the graph adjacency matrix, retaining the top $\lceil K \cdot |V^{(l-1)}| \rceil$ most relevant nodes (where $K \in (0, 1]$ is a retention ratio hyperparameter).
After $N$ stacked layers, the pooled final query vector and pooled final trimmed graph representation are concatenated to predict a plausibility score for each candidate answer.
Knowl 2 — Dense Bidirectional Attention in the JointLK Reasoning Module
equation
In layer $l$ of JointLK, the Joint Reasoning Module receives the query token representations $Q^{l-1} = \{q^{l-1}_i\}_{i=1}^M \in \mathbb{R}^{M \times D}$ and the GNN-updated knowledge graph entity representations $\tilde{X}^l = \{\tilde{x}^l_j\}_{j=1}^{|V|} \in \mathbb{R}^{|V| \times D}$ , where $M$ is the number of query tokens, $|V|$ is the current number of graph nodes, and $D$ is the hidden feature dimension.

First, an affinity matrix $S^l \in \mathbb{R}^{M \times |V|}$ is constructed between each token $i$ and node $j$ : $S^l_{ij} = W_S^\top [q^{l-1}_i; \tilde{x}^l_j; q^{l-1}_i \circ \tilde{x}^l_j]$ where $W_S \in \mathbb{R}^{3D}$ is a learnable parameter vector, $[;]$ denotes vector concatenation, and $\circ$ denotes element-wise multiplication.

Row-wise and column-wise softmax operations derive the KG-to-LM attention maps $S^l_q \in \mathbb{R}^{M \times |V|}$ on query tokens and LM-to-KG attention maps $S^l_x \in \mathbb{R}^{M \times |V|}$ on entity nodes: $S^l_{qi} = \text{softmax}(S^l_{i,:})$ $S^l_{xj} = \text{softmax}((S^l_{:,j})^\top)$

The attended representations $\hat{q}_{ij}$ and $\hat{x}_{ij}$ are computed via matrix multiplication $\otimes$ : $\hat{q}_{ij} = q^{l-1}_i \otimes S^l_{qi}, \quad \hat{x}_{ij} = \tilde{x}^l_j \otimes S^l_{xj}$

The cross-attended features are fused with the original representations and projected back to dimension $D$ using trainable matrices $W_Q, W_X \in \mathbb{R}^{D \times 4D}$ : $q^l_i = W_Q [q^{l-1}_i; \hat{x}_{ij}; q^{l-1}_i \circ \hat{x}_{ij}; q^{l-1}_i \circ \hat{q}_{ij}]$ $\bar{x}^l_j = W_X [\tilde{x}^l_j; \hat{q}_{ij}; \tilde{x}^l_j \circ \hat{q}_{ij}; \tilde{x}^l_j \circ \hat{x}_{ij}]$

The updated query representation $Q^l = \{q^l_i\}_{i=1}^M$ is fed to the next layer's joint reasoning module, and the updated graph representation $\bar{X}^l = \{\bar{x}^l_j\}_{j=1}^{|V|}$ is sent to the dynamic pruning module.
Knowl 3 — Dynamic Graph Pruning Module in JointLK
model/method
To eliminate noisy or irrelevant nodes retrieved from external knowledge graphs during multi-hop reasoning, JointLK applies dynamic graph pruning at each stacked layer $l$ . Pruning is guided by the query-to-node attention weights $S^l_x \in \mathbb{R}^{M \times |V|}$ generated by the dense bidirectional attention module, which reflect the global importance of each KG node for answering the question.

Let $Z \in \mathbb{R}^{|V|}$ denote the importance score vector of the current graph nodes derived from the LM-to-KG attention. Given a retention ratio hyperparameter $K \in (0, 1]$ , the module determines the number of retained nodes as $\lceil K \cdot |V| \rceil$ . The indices of the highest-scoring nodes and their corresponding attention mask are obtained via: $\text{idx} = \text{top-rank}(Z, \lceil K \cdot |V| \rceil)$ $Z_{\text{mask}} = Z_{\text{idx}}$ where $\text{top-rank}(Z, k)$ returns the indices of the $k$ largest values in $Z$ , and $Z_{\text{mask}}$ is the sliced importance vector.

The trimmed node representation matrix $X^l \in \mathbb{R}^{\lceil K \cdot |V| \rceil \times D}$ and the updated adjacency matrix $A^l \in \mathbb{R}^{\lceil K \cdot |V| \rceil \times \lceil K \cdot |V| \rceil}$ for the next layer are defined by: $X^l = \bar{X}^l_{\text{idx}, :} \odot Z_{\text{mask}}$ $A^l = \bar{A}^l_{\text{idx}, \text{idx}}$ where $\bar{X}^l_{\text{idx}, :}$ selects the retained rows of the fused KG representation matrix $\bar{X}^l$ , $\odot$ denotes broadcasted element-wise multiplication with $Z_{\text{mask}}$ , and $\bar{A}^l_{\text{idx}, \text{idx}}$ slices the corresponding rows and columns from the adjacency matrix $\bar{A}^l$ . Over $N$ layers, the graph is pruned recursively, retaining approximately $K^N$ of the original nodes.
Knowl 4 — Relational Graph Attention Network Layer in JointLK
equation
The GNN layer in JointLK updates entity node embeddings using a relational graph attention network (RGAT) that incorporates edge relation types and node types into the attention weights and message passing.

For layer $l$ and node $i \in V$ , the attention weight $\alpha_{ji}$ from neighbor $j \in \mathcal{N}_i \cup \{i\}$ to node $i$ is computed as: $\hat{\alpha}_{ji} = (x^{l-1}_i W_q) (x^{l-1}_j W_k + r_{ji})^\top$ $\alpha_{ji} = \text{softmax}_j\left(\frac{\hat{\alpha}_{ji}}{\sqrt{D}}\right)$ where $x^{l-1}_i, x^{l-1}_j \in \mathbb{R}^D$ are the input node representations from layer $l-1$ , $W_q, W_k \in \mathbb{R}^{D \times D}$ are trainable weight matrices, and $D$ is the hidden dimension.

The relation feature vector $r_{ji} \in \mathbb{R}^D$ is defined by: $r_{ji} = \psi(e_{ji}, u_j, u_i)$ where $e_{ji}$ is a one-hot vector indicating the relation type of edge $(j, i)$ , $u_j, u_i$ are one-hot vectors indicating node types of $j$ and $i$ , and $\psi$ is a linear mapping function.

The aggregated message $\hat{x}^{l-1}_i$ and the updated node representation $\tilde{x}^l_i$ before cross-modal fusion are: $\hat{x}^{l-1}_i = \sum_{j \in \mathcal{N}_i \cup \{i\}} \alpha_{ji} (x^{l-1}_j W_v + r_{ji})$ $\tilde{x}^l_i = \text{LayerNorm}(x^{l-1}_i + \hat{x}^{l-1}_i W_o)$ where $W_v, W_o \in \mathbb{R}^{D \times D}$ are trainable projection matrices, and $\text{LayerNorm}$ denotes layer normalization.
Knowl 5 — Query Encoding and Plausibility Scoring in JointLK
model/method
In JointLK, each candidate answer choice is evaluated via text encoding followed by summary pooling and multi-layer perceptron (MLP) scoring:
1. Query Encoding: A pre-trained language model encodes the token sequence $\{w_i\}_{i=1}^M$ of a question-choice pair into contextual vectors $\{\tilde{q}^0_i\}_{i=1}^M \in \mathbb{R}^T$ , where $T$ is the LM hidden dimension. These are projected into the entity hidden dimension $D$ via: $q^0_i = \sigma(f_s(\tilde{q}^0_i))$ where $f_s: \mathbb{R}^T \to \mathbb{R}^D$ is a linear transformation and $\sigma$ is an activation function, producing initial query token representations $Q^0 = \{q^0_i\}_{i=1}^M \in \mathbb{R}^{M \times D}$ .
2. Answer Scoring: After $N$ stacked JointLK layers of message passing, cross-modal attention, and dynamic pruning, the final query token representations $Q^N \in \mathbb{R}^{M \times D}$ and final entity representations $X^N \in \mathbb{R}^{|V^{(N)}| \times D}$ are pooled into fixed-size summary vectors: $s = \text{MeanPooling}(Q^N)$ $g = \text{AttentivePooling}(X^N)$
The plausibility score $p(a|q)$ of candidate choice $a$ given question $q$ is computed by: $p(a|q) = \text{MLP}([s; g])$ where $[s; g] \in \mathbb{R}^{2D}$ concatenates the pooled textual summary $s$ and graph summary $g$ . The plausibility scores across all choices $\{a_1, \dots, a_n\}$ are normalized with a softmax function to obtain choice probabilities.
Knowl 6 — Knowledge Subgraph Extraction and Node Initialization from ConceptNet
algorithm
To construct external structured commonsense evidence for JointLK from ConceptNet, subgraphs are extracted, filtered, and initialized via the following multi-step procedure:
Input: Question $q$ , candidate choice $a$ , external knowledge graph ConceptNet
Output: Initial node set $V$ , initial node features $X^0$ , relation set $R$ , adjacency matrix $A^0$
Identify concepts in ConceptNet appearing in $q$ as $V_q$ , and in $a$ as $V_a$
Initialize node set $V_{q,a} \leftarrow V_q \cup V_a$
Find all bridging entities on 1-hop and 2-hop paths connecting any entity in $V_q$ to any entity in $V_a$ , adding them to $V$
Score all candidate nodes in $V$ conditioned on $(q + a)$ using a pre-trained LM
Retain only the top 200 highest-scoring nodes in $V$
Add question node $q$ as a special node into $V$
Merge ConceptNet relations into 19 canonical relation types and add reverse relations (yielding 38 relation types total)
Construct relation set $R$ by connecting all edges between nodes in $V$ along with bidirectional edges $(q \leftrightarrow v)$ for $v \in V_q \cup V_a$
for each entity node $v_i \in V$ do
    Convert knowledge triples containing $v_i$ into natural language sentences using templates
    Pass sentences through BERT-Large to extract last-layer token embeddings
    Compute initial node embedding $x^0_i \in \mathbb{R}^D$ by mean pooling across all token occurrences of $v_i$
end for
return $V, X^0 = \{x^0_i\}_{i=1}^{|V|}, R, A^0$

Knowl 7 — Empirical Performance of JointLK on CommonsenseQA and OpenBookQA

empirical result

JointLK was evaluated on two commonsense question answering benchmarks: CommonsenseQA (5-way multiple choice, evaluated on in-house splits IHdev/IHtest and the official test leaderboard) and OpenBookQA (4-way multiple choice, evaluated with RoBERTa-large and AristoRoBERTa). All baseline LM+KG models used the same LM encoders for fair comparison.

Model	CSQA IHdev (%)	CSQA IHtest (%)	OBQA RoBERTa (%)	OBQA AristoRoBERTa (%)
Fine-tuned LM (w/o KG)	$73.07 \pm 0.45$	$68.69 \pm 0.56$	$64.80 \pm 2.37$	$78.40 \pm 1.64$
+ RGCN	$72.69 \pm 0.19$	$68.41 \pm 0.66$	$62.45 \pm 1.57$	$74.60 \pm 2.53$
+ GconAttn	$71.61 \pm 0.39$	$68.59 \pm 0.96$	$64.75 \pm 1.48$	$71.80 \pm 1.21$
+ KagNet	$73.47 \pm 0.22$	$69.01 \pm 0.76$	–	–
+ RN	$74.57 \pm 0.91$	$69.08 \pm 0.21$	$65.20 \pm 1.18$	$75.35 \pm 1.39$
+ MHGRN	$74.45 \pm 0.10$	$71.11 \pm 0.81$	$66.85 \pm 1.19$	$80.60$
+ QA-GNN	$76.54 \pm 0.21$	$73.41 \pm 0.92$	$67.80 \pm 2.75$	$82.77 \pm 1.56$
+ JointLK (Ours)	$\mathbf{77.88 \pm 0.25}$	$\mathbf{74.43 \pm 0.83}$	$\mathbf{70.34 \pm 0.75}$	$\mathbf{84.92 \pm 1.07}$

On CommonsenseQA IHtest, JointLK achieved $74.43\%$ accuracy, outperforming the fine-tuned LM alone by $5.74\%$ and the prior best LM+KG model QA-GNN ( $73.41\%$ ) by $1.02\%$ . On the official CommonsenseQA leaderboard, RoBERTa + JointLK achieved $76.6\%$ test accuracy (compared to $76.1\%$ for RoBERTa + QA-GNN).

On OpenBookQA, AristoRoBERTa + JointLK achieved $84.92\%$ accuracy on the test set and $85.6\%$ on the official leaderboard, outperforming AristoRoBERTa + QA-GNN ( $82.77\%$ test / $82.8\%$ leaderboard) and fine-tuned AristoRoBERTa ( $78.40\%$ ).

Knowl 8 — Ablation Analysis of JointLK Components, Stacking Depth, and Pruning Ratio

empirical result

Ablation experiments conducted on the CommonsenseQA in-house dev (IHdev) split with RoBERTa-large evaluated the necessity of JointLK's core components and hyperparameters:

Module Ablation: Disabling the dynamic pruning module caused a $0.50\%$ drop in accuracy (from $77.88\%$ to $77.38\%$ ). Disabling the joint reasoning module (which also removes dynamic pruning due to reliance on attention weights) caused a performance drop of $1.27\%$ down to $76.61\%$ .
Stacking Depth ( $N$ ): Varying the number of JointLK layers showed continuous accuracy gains up to $N = 5$ . Increasing depth beyond 5 ( $N > 5$ ) caused performance to drop as the model shifted from underfitting to overfitting.
Retention Ratio ( $K$ ): For $N = 5$ layers, setting the per-layer node retention ratio to $K = 0.92$ (leaving approximately $K^5 \approx 66\%$ of initial nodes in the final layer) achieved optimal performance. Setting $K$ too high (e.g., $K = 0.98$ , retaining $90\%$ of nodes) provided insufficient noise reduction, whereas setting $K$ too low eliminated essential reasoning paths.

Knowl 9 — Robustness of JointLK on Negation and Multi-Entity Complex Reasoning

empirical result

To evaluate JointLK's capability in complex reasoning scenarios, accuracy on subsets of the CommonsenseQA IHdev set was compared against QA-GNN:

Metric / Subset	Overall	Questions w/ Negation	Questions w/ $\le 7$ Entities	Questions w/ $> 7$ Entities
Number of Questions	1221	133	723	498
QA-GNN (%)	$76.99$	$72.18$	$76.63$	$77.51$
JointLK (%)	$\mathbf{78.38}$	$\mathbf{75.18}$ ( $\uparrow 3.00$ )	$\mathbf{77.59}$ ( $\uparrow 0.96$ )	$\mathbf{79.52}$ ( $\uparrow 2.01$ )

Questions with Negation: On 133 questions containing negation words (e.g., no, not, nothing, never, unlikely, don't, can't), JointLK achieved a $3.00\%$ improvement over QA-GNN ( $75.18\%$ vs. $72.18\%$ ), indicating that fine-grained token-level cross-attention helps the model capture subtle semantic polarity shifts that pooled representations overlook.
Questions with Many Entities: On complex questions with $> 7$ entities (498 questions), JointLK gained $2.01\%$ over QA-GNN ( $79.52\%$ vs. $77.51\%$ ), compared to a $0.96\%$ gain on questions with $\le 7$ entities ( $77.59\%$ vs. $76.63\%$ ). This demonstrates that recursive dynamic pruning effectively alleviates noise and reasoning difficulty in large multi-entity subgraphs.

Knowl 10 — Error Categorization and Knowledge Graph Limitations in JointLK
limitation
An analysis of 100 randomly sampled error cases of JointLK on the CommonsenseQA validation set revealed three primary failure categories:
1. Missing Important Evidence ( $39/100$ cases): Crucial intermediate facts or relations required to bridge the question and the correct choice were absent from the external KG (ConceptNet) or omitted by the 2-hop/top-200 retrieval heuristic. For example, answering why eating cheese caused indigestion for a lactose-intolerant person requires knowledge that lactose intolerance causes indigestion, which is not present in ConceptNet.
2. Indistinguishable Knowledge ( $25/100$ cases): Multiple candidate choices possessed equivalent, highly plausible relations in the knowledge graph (e.g., both a human and a cat can be at location "bed" or "comfortable chair"). In such ambiguous cases, the model often defaulted to frequency biases learned from pre-training text corpora.
3. Incomprehensible / Complex Narrative Questions ( $23/100$ cases): Long questions involving multi-character scenarios, sequential events, and emotional transitions proved difficult to interpret because ConceptNet encodes static entity-attribute and entity-relation triples rather than complex event-based procedural reasoning.

Coverage note — None was omitted; all key architectural components, equations, algorithms, empirical benchmarks, ablations, and error analyses contributed by the paper are fully covered.

References

1.Pratyay Banerjee and Chitta Baral. 2020. Knowledge fusion and semantic knowledge ranking for open domain question answering. arXiv preprint arXiv:2004.03101.
2.Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral. 2019. Careful selection of knowledge to solve open book question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6120–6129, Florence, Italy. Association for Computational Linguistics.
3.Ning Bian, Xianpei Han, Bo Chen, and Le Sun. 2021. Benchmarking knowledge-enhanced commonsense question answering via knowledge-to-text transformation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12574–12582.
4.Peter Clark, Oren Etzioni, Tushar Khot, Daniel Khashabi, Bhavana Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2020. From ‘f’to ‘a’ on the NY regents science exams: An overview of the aristo project. AI Magazine, 41(4):39–53.
5.Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren. 2020. Scalable multihop relational reasoning for knowledge-aware question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1295–1309, Online. Association for Computational Linguistics.
6.David Gunning. 2018. Machine common sense concept paper. arXiv preprint arXiv:1810.07528.
7.Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Computational Linguistics.
8.Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
9.Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-attention graph pooling. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3734–3743. PMLR.
10.Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829–2839, Hong Kong, China. Association for Computational Linguistics.
11.Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations.
12.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
13.Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. 2020. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8449–8456.
14.Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. 2019. Towards generalizable neuro-symbolic systems for commonsense question answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 22–32, Hong Kong, China. Association for Computational Linguistics.
15.Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
16.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
17.Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
18.Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE transactions on neural networks, 20(1):61–80.
19.Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593–607. Springer.
20.Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-first AAAI conference on artificial intelligence.
21.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958.
22.Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
23.Denny Vrande\v{c}i'c and Markus Kr"otzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78–85.
24.Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020a. Relational graph attention network for aspect-based sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3229–3238, Online. Association for Computational Linguistics.
25.Peifeng Wang, Nanyun Peng, Filip Ilievski, Pedro Szekely, and Xiang Ren. 2020b. Connecting the dots: A knowledgeable path generator for commonsense question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4129–4140, Online. Association for Computational Linguistics.
26.Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, et al. 2019. Improving natural language inference using external knowledge in the science questions domain. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7208–7215.
27.Yichong Xu, Chenguang Zhu, Ruochen Xu, Yang Liu, Michael Zeng, and Xuedong Huang. 2021. Fusing context into knowledge graph for commonsense question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1201–1207, Online. Association for Computational Linguistics.
28.Jun Yan, Mrigank Raman, Aaron Chan, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, and Xiang Ren. 2021. Learning contextualized knowledge structures for commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4038–4051, Online. Association for Computational Linguistics.
29.Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535–546, Online. Association for Computational Linguistics.
30.Avishai Zagoury, Einat Minkov, Idan Szpektor, and William W Cohen. 2021. What’s the best place for an ai conference, vancouver or _: Why completing comparative questions is difficult. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14292–14300.
31.Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations.

Citation

MLA

Sun, Y., et al. “JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering”. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2022, pp. 5049–60, https://doi.org/10.18653/v1/2022.naacl-main.372.

APA

Sun, Y., Shi, Q., Qi, L., & Zhang, Y. (2022). JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 5049–5060. https://doi.org/10.18653/v1/2022.naacl-main.372

Chicago

Sun, Y., Q. Shi, L. Qi, and Y. Zhang. 2022. “JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering”. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 5049–60. https://doi.org/10.18653/v1/2022.naacl-main.372.

Harvard

Sun, Y. et al. (2022) “JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering”, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp. 5049–5060. Available at: https://doi.org/10.18653/v1/2022.naacl-main.372.

Vancouver

1. Sun Y, Shi Q, Qi L, Zhang Y (2022) JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pp 5049–5060

BibTeX

@inproceedings{sun-etal-2022-jointlk,
    title = "{J}oint{LK}: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering",
    author = "Sun, Yueqing  and
      Shi, Qi  and
      Qi, Le  and
      Zhang, Yu",
    editor = "Carpuat, Marine  and
      de Marneffe, Marie-Catherine  and
      Meza Ruiz, Ivan Vladimir",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.372/",
    doi = "10.18653/v1/2022.naacl-main.372",
    pages = "5049--5060"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Configuration	IHdev Accuracy (%)
JointLK ( $N=5$ )	$77.88$
– Dynamic Pruning Module	$77.38$
– Joint Reasoning Module	$76.61$

JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering

Abstract

Topics

Table of Contents

Knowls

Knowl 1 — JointLK Framework for Joint Reasoning between Language Models and Knowledge Graphs

Knowl 2 — Dense Bidirectional Attention in the JointLK Reasoning Module

Knowl 3 — Dynamic Graph Pruning Module in JointLK

Knowl 4 — Relational Graph Attention Network Layer in JointLK

Knowl 5 — Query Encoding and Plausibility Scoring in JointLK

Knowl 6 — Knowledge Subgraph Extraction and Node Initialization from ConceptNet

Knowl 7 — Empirical Performance of JointLK on CommonsenseQA and OpenBookQA

Knowl 8 — Ablation Analysis of JointLK Components, Stacking Depth, and Pruning Ratio

Knowl 9 — Robustness of JointLK on Negation and Multi-Entity Complex Reasoning

Knowl 10 — Error Categorization and Knowledge Graph Limitations in JointLK

References

Citation

BibTeX

Source Code

Access the Paper