Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Peng ZhouWei ShiJun TianZhenyu QiBingchen LiHongwei HaoBo Xu

article2016ACL1,994 citations

Proposes an attention-based bidirectional LSTM architecture that captures key semantic context across entire sentences for relation classification using only raw word vectors, eliminating the need for handcrafted lexical features or external NLP pipelines.

Listen

Identifying semantic relationships between entities in text is a foundational capability for applications like automated question answering and information extraction. Traditional machine learning methods and early deep learning systems have relied heavily on complex linguistic toolkits, external dictionaries, and manually engineered features. These manual pipelines are expensive to build, prone to compounding errors from upstream tools, and struggle to generalize across diverse datasets. The article demonstrates that an attention-based bidirectional recurrent neural network can automatically identify the most decisive words in a sentence, achieving state-of-the-art relation classification performance using only raw text and word vectors.

The authors designed a neural network architecture that combines bidirectional sequential modeling with an attention mechanism. This model reads sentences in both forward and backward directions to capture full context and applies dynamic weights to emphasize the most informative words. To evaluate the approach, the authors benchmarked the system on the standard SemEval-2010 Task 8 benchmark dataset, which consists of 10,717 annotated sentences (8,000 for training and 2,717 for testing) covering nine distinct relationship categories and an "Other" category, using standard macro-averaged accuracy metrics.

The key finding of the article is that the proposed model achieved an overall accuracy score of 84.0%, outperforming traditional feature-engineered models like Support Vector Machines (82.2%) and convolutional neural network baselines (82.7%). When compared under identical 50-dimensional word representations, the attention-based architecture improved classification accuracy by 2.5 percentage points over standard recurrent models (82.5% versus 80.0%). Furthermore, the model achieved performance comparable to the top-performing benchmark system (84.3%), but did so without requiring any syntactic parsing, part-of-speech taggers, or external lexical databases. Adding the attention mechanism consistently boosted the performance of the bidirectional recurrent baseline across multiple vector configurations.

These results demonstrate that organizations can deploy simpler, end-to-end relation extraction pipelines without sacrificing accuracy. Eliminating the dependency on specialized natural language processing tools significantly reduces engineering complexity, operational maintenance overhead, and vulnerability to upstream processing errors. The findings indicate that automated semantic attention provides a viable, cost-effective substitute for labor-intensive feature engineering in text classification workflows.

For operational deployment, engineering teams should consider adopting attention-based bidirectional sequence models to streamline relation extraction systems, especially where external linguistic tools are unavailable or expensive to maintain. However, because the article evaluates the system strictly on a single academic benchmark in English, stakeholders should exercise measured confidence. Before full-scale operational rollout, organizations should conduct pilot evaluations on domain-specific data and real-world noisy text to confirm generalization and robustness.

Institute of Automation, Chinese Academy of Sciences classification dependency parsing named entity recognition Relation Networks Long Short-Term Memory

Zhou et al (2016).pdf

Paper: Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau et al. (2015). Introduces the core soft attention mechanism over bidirectional recurrent hidden states upon which the source's sentence-level attention architecture is built.
Paper: Effective Approaches to Attention-based Neural Machine Translation, Minh-Thang Luong et al. (2015). Formalizes global and local attention scoring functions for recurrent neural networks that provide the theoretical and practical foundation for attention-based sequence modeling.
Paper: Bidirectional LSTM-CRF Models for Sequence Tagging, Zhiheng Huang et al. (2015). Demonstrates the effectiveness of bidirectional LSTM architectures in extracting contextual sequence representations from raw word embeddings for NLP tasks.
Paper: Long Short-Term Memory, Sepp Hochreiter et al. (1997). Establishes the fundamental Long Short-Term Memory cell architecture that enables recurrent networks to learn long-range semantic dependencies.
Paper: Distant supervision for relation extraction without labeled data, Mike D. Mintz et al. (2009). Defines the standard problem formulations and feature-based challenges of relation classification that deep neural approaches seek to overcome.

Paper: A Structured Self-attentive Sentence Embedding, Zhouhan Lin et al. (2017). Extends single-vector attention over bidirectional LSTMs into structured 2D self-attentive matrices to capture multiple semantic aspects of a sentence.
Paper: Deep contextualized word representations, Matthew E. Peters et al. (2018). Generalizes deep bidirectional LSTM representations into contextualized word embeddings (ELMo) that significantly improve downstream relation and sequence modeling.
Paper: Self-Attention with Relative Position Representations, Peter Shaw et al. (2018). Advances beyond recurrent sequential positional encoding by embedding relative positions directly within attention mechanisms.
Paper: Modeling Relational Data with Graph Convolutional Networks, Michael Schlichtkrull et al. (2018). Applies neural relational modeling across multi-relational graph structures using relational graph convolutional networks.

Cover for Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Abstract

Relation classification is an important semantic processing task in the field of natural language processing (NLP). State-of-the-art systems still rely on lexical resources such as WordNet or NLP systems like dependency parser and named entity recognizers (NER) to get high-level features. Another challenge is that important information can appear at any position in the sentence. To tackle these problems, we propose Attention-Based Bidirectional Long Short-Term Memory Networks(Att-BLSTM) to capture the most important semantic information in a sentence. The experimental results on the SemEval-2010 relation classification task show that our method outperforms most of the existing methods, with only word vectors.

Topics

classification dependency parsing named entity recognition random access memory semantic information wordnet classification models natural language understanding syntactics

Keywords

Relation Networks Long Short-Term Memory Relation Classi Cation Short-Term Memory Networks Bidirectional Bidirectional Long Short-Term Model Attention-Based Bidirectional Long Relation Classification

1 Introduction
2 Related Work
3 Model
3.1 Word Embeddings
3.2 Bidirectional Network
3.3 Attention
3.4 Classifying
3.5 Regularization
4 Experiments
4.1 Dataset and Experimental Setup
4.2 Experimental Results
5 Conclusion
Acknowledgments
References

Knowls

Knowl 1 — Att-BLSTM Architecture for Relation Classification
model/method
The Attention-Based Bidirectional Long Short-Term Memory Network (Att-BLSTM) is a neural architecture designed for relation classification between pairs of nominals without relying on handcrafted features, part-of-speech taggers, dependency parsers, or lexical resources like WordNet.

Given a sentence $S = (x_1, x_2, \dots, x_T)$ of $T$ tokens containing two target nominal entities, four special position indicator tokens ( $\langle e_1 \rangle$ , $\langle /e_1 \rangle$ , $\langle e_2 \rangle$ , and $\langle /e_2 \rangle$ ) are inserted directly into the token sequence to mark the beginning and end of the nominals. The model processes this sequence through five stacked layers:
1. Input Layer: Takes the raw word sequence including the four position indicator tokens.
2. Embedding Layer: Projects each token $x_i$ into a continuous $d_w$ -dimensional vector $e_i = W^{\text{wrd}} v^i$ , where $W^{\text{wrd}} \in \mathbb{R}^{d_w \times |V|}$ is a learned embedding matrix and $v^i$ is a one-hot indicator vector over vocabulary $V$ .
3. BLSTM Layer: Processes the sequence of word embeddings $\{e_1, \dots, e_T\}$ using forward and backward LSTM sub-networks with peephole connections. The hidden states at each position are merged via element-wise summation to capture context from both directions.
4. Attention Layer: Computes normalized attention weights over all token hidden states to form a sentence-level semantic representation vector $h^*$ .
5. Output Layer: Feeds $h^*$ into a softmax classifier to predict the semantic relation label between the entity pair.
Knowl 2 — Bidirectional LSTM Layer with Peephole Connections
equation
In the Att-BLSTM network, the LSTM recurrent cells incorporate weighted peephole connections from the cell memory state $c$ directly into the input gate $i_t$ , forget gate $f_t$ , and output gate $o_t$ . At time step $t$ with input word embedding $x_t \in \mathbb{R}^{d_w}$ , previous hidden state $h_{t-1} \in \mathbb{R}^{d_w}$ , and previous cell state $c_{t-1} \in \mathbb{R}^{d_w}$ , the LSTM updates are defined by:

$i_t = \sigma(W_{xi} x_t + W_{hi} h_{t-1} + W_{ci} c_{t-1} + b_i)$

$f_t = \sigma(W_{xf} x_t + W_{hf} h_{t-1} + W_{cf} c_{t-1} + b_f)$

$g_t = \tanh(W_{xc} x_t + W_{hc} h_{t-1} + W_{cc} c_{t-1} + b_c)$

$c_t = i_t \odot g_t + f_t \odot c_{t-1}$

$o_t = \sigma(W_{xo} x_t + W_{ho} h_{t-1} + W_{co} c_t + b_o)$

$h_t = o_t \odot \tanh(c_t)$

where $\sigma(\cdot)$ denotes the sigmoid activation function, $\odot$ represents element-wise multiplication, and $W$ matrices and $b$ vectors are trainable parameters.

To exploit context from both directions, a forward LSTM yields hidden states $\vec{h}_i$ and a backward LSTM yields $\overleftarrow{h}_i$ for each token $i$ . These are combined into a single hidden vector $h_i \in \mathbb{R}^{d_w}$ via element-wise summation:

$h_i = \vec{h}_i \oplus \overleftarrow{h}_i$
Knowl 3 — Sentence-Level Word Attention Mechanism
equation
To automatically focus on words carrying decisive semantic information for relation classification, the model pools the sequence of BLSTM output representations using a word-level attention mechanism.

Let $H = [h_1, h_2, \dots, h_T] \in \mathbb{R}^{d_w \times T}$ be the matrix formed by concatenating the hidden state vectors produced by the BLSTM layer across a sentence of length $T$ , where $d_w$ is the hidden state dimensionality. The sentence representation $r$ and final classification vector $h^*$ are computed as:

$M = \tanh(H)$

$\alpha = \text{softmax}(w^T M)$

$r = H \alpha^T$

$h^* = \tanh(r)$

where $w \in \mathbb{R}^{d_w}$ is a trained attention parameter vector, $M \in \mathbb{R}^{d_w \times T}$ is a non-linear feature projection, $\alpha \in \mathbb{R}^{1 \times T}$ is the normalized attention weight vector across the $T$ tokens, and $h^* \in \mathbb{R}^{d_w}$ is the sentence representation passed to the classifier.
Knowl 4 — Relation Classification Objective and Regularization
equation
Given the sentence representation $h^* \in \mathbb{R}^{d_w}$ , a softmax classification layer predicts the relation category $\hat{y}$ from the set of candidate classes $Y$ :

$\hat{p}(y \mid S) = \text{softmax}(W^{(S)} h^* + b^{(S)})$

$\hat{y} = \arg\max_{y \in Y} \hat{p}(y \mid S)$

where $W^{(S)}$ and $b^{(S)}$ are trainable weight and bias parameters.

The network parameters $\theta$ are optimized by minimizing the regularized negative log-likelihood loss over $m$ target classes:

$J(\theta) = -\frac{1}{m} \sum_{i=1}^m t_i \log(y_i) + \lambda \|\theta\|_F^2$

where $t \in \mathbb{R}^m$ is the one-hot target label vector, $y \in \mathbb{R}^m$ is the predicted class probability distribution, and $\lambda$ is the $L_2$ regularization weight parameter.
Knowl 5 — Training Configuration and Regularization Strategy for Att-BLSTM
experimental setup
Att-BLSTM is trained and evaluated under the following experimental protocol on relation classification:
- Optimization: AdaDelta optimizer with an initial learning rate of $1.0$ and a minibatch size of $10$ .
- Weight Regularization: $L_2$ weight regularization with strength $\lambda = 10^{-5}$ applied per minibatch.
- Weight Vector Rescaling: $L_2$ -norm constraint applied to weight vectors by rescaling $w$ to satisfy $\|w\| = s$ whenever $\|w\| > s$ after a gradient step.
- Dropout: Applied to three locations in the network:
  - Embedding layer: dropout rate = $0.3$
  - LSTM layer: dropout rate = $0.3$
  - Penultimate (sentence representation) layer: dropout rate = $0.5$
- Word Vector Initialization: Evaluated using both 50-dimensional embeddings (Turian et al., 2010) and 100-dimensional GloVe embeddings (Pennington et al., 2014). Position indicators $\langle e_1 \rangle$ , $\langle /e_1 \rangle$ , $\langle e_2 \rangle$ , and $\langle /e_2 \rangle$ are initialized randomly alongside other out-of-vocabulary tokens.
- Hyperparameter Validation: Model hyperparameters are tuned on a validation set of $800$ randomly selected sentences held out from the training split.

Knowl 6 — Empirical Performance on SemEval-2010 Task 8 Relation Classification

empirical result

The Att-BLSTM model was evaluated on the SemEval-2010 Task 8 dataset, which contains $10,717$ annotated sentences ( $8,000$ training and $2,717$ test examples) across 9 directional semantic relation types (yielding 18 directional classes) and one undirected Other class (19 classes total). Evaluation is based on the official macro-averaged $F_1$ -score across the 9 actual relation types (excluding Other), taking directionality into account.

Model	Feature Set	F1 (%)
SVM (Rink and Harabagiu, 2010)	POS, prefixes, morphological, WordNet, dependency parse,	82.2
	Levin classes, PropBank, FrameNet, NomLex-Plus,
	Google n-gram, paraphrases, TextRunner
CNN (Zeng et al., 2014)	WV (Turian, dim=50)	69.7
	+ PF + WordNet	82.7
RNN (Zhang and Wang, 2015)	WV (Turian, dim=50) + PI	80.0
	WV (Mikolov, dim=300) + PI	82.5
SDP-LSTM (Yan et al., 2015)	WV (word2vec, dim=200), syntactic parse	82.4
	+ POS + WordNet + grammar relation embeddings	83.7
BLSTM (Zhang et al., 2015)	WV (GloVe, dim=100)	82.7
	+ PF + POS + NER + WNSYN + DEP	84.3
BLSTM	WV (Turian, dim=50) + PI	80.7
Att-BLSTM	WV (Turian, dim=50) + PI	82.5
BLSTM	WV (GloVe, dim=100) + PI	82.7
Att-BLSTM	WV (GloVe, dim=100) + PI	84.0

(WV: Word Vectors; PF: Position Features; PI: Position Indicators; POS: Part of Speech; NER: Named Entity Recognition; DEP: Dependency Parse; WNSYN: WordNet Synonyms)

Att-BLSTM improves upon baseline BLSTM by $1.8\%$ F1 using 50-dimensional Turian embeddings ( $82.5\%$ vs. $80.7\%$ ) and by $1.3\%$ F1 using 100-dimensional GloVe embeddings ( $84.0\%$ vs. $82.7\%$ ). Operating purely on raw text with position indicator tokens and word embeddings, Att-BLSTM achieves $84.0\%$ F1, outperforming models that rely on dependency parsers, WordNet, POS taggers, and NER.

Coverage note — None was omitted; all architectural designs, mathematical formulations, training hyperparameters, and benchmark results on the SemEval-2010 Task 8 dataset are fully covered.

References

1.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
2.Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In Proceedings of the conference on human language technology and empirical methods in natural language processing, pages 724–731. Association for Computational Linguistics.
3.Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition. In Advances in Neural Information Processing Systems, pages 577–585.
4.Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 6645–6649. IEEE.
5.Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850.
6.Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O´ S´eaghdha, Sebastian Pado´, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 94–99. Association for Computational Linguistics.
7.Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692.
8.Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing coadaptation of feature detectors. arXiv preprint arXiv:1207.0580.
9.Sepp Hochreiter and J¨urgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
10.Tomas Mikolov, Martin Karafia´t, Lukasˇ Burget, Jan Cˇernocky`, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045–1048.
11.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
12.George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.
13.Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1003–1011. Association for Computational Linguistics.
14.Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532–1543.
15.Bryan Rink and Sanda Harabagiu. 2010. Utd: Classifying semantic relations by combining lexical and semantic resources. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 256–259. Association for Computational Linguistics.
16.Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. Association for Computational Linguistics.
17.Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 384–394. Association for Computational Linguistics.
18.Fei Wu and Daniel S Weld. 2010. Open information extraction using wikipedia. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 118–127. Association for Computational Linguistics.
19.Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint arXiv:1502.03044.
20.Xu Yan, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, and Zhi Jin. 2015. Classifying relations via long short term memory networks along shortest dependency path. arXiv preprint arXiv:1508.03720.
21.Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In ACL (1), pages 956–966. Citeseer.
22.Matthew D Zeiler. 2012. Adadelta: An adaptive learning rate method. arXiv preprint arXiv:1212.5701.
23.Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of COLING, pages 2335–2344.
24.Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. arXiv preprint arXiv:1508.01006.
25.Shu Zhang, Dequan Zheng, Xinchen Hu, and Ming Yang. 2015. Bidirectional long short-term memory networks for relation classification.

Citation

MLA

Zhou, P., et al. “Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification”. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2016, pp. 207–12, https://doi.org/10.18653/v1/P16-2034.

APA

Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., & Xu, B. (2016). Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 207–212. https://doi.org/10.18653/v1/P16-2034

Chicago

Zhou, P., W. Shi, J. Tian, et al. 2016. “Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification”. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 207–12. https://doi.org/10.18653/v1/P16-2034.

Harvard

Zhou, P. et al. (2016) “Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification”, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pp. 207–212. Available at: https://doi.org/10.18653/v1/P16-2034.

Vancouver

1. Zhou P, Shi W, Tian J, Qi Z, Li B, Hao H, Xu B (2016) Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pp 207–212

BibTeX

@inproceedings{zhou-etal-2016-attention,
    title = "Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification",
    author = "Zhou, Peng  and
      Shi, Wei  and
      Tian, Jun  and
      Qi, Zhenyu  and
      Li, Bingchen  and
      Hao, Hongwei  and
      Xu, Bo",
    editor = "Erk, Katrin  and
      Smith, Noah A.",
    booktitle = "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)",
    month = aug,
    year = "2016",
    address = "Berlin, Germany",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P16-2034/",
    doi = "10.18653/v1/P16-2034",
    pages = "207--212"
}

Metadata:ACL Anthology

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Att-BLSTM Architecture for Relation Classification

Knowl 2 — Bidirectional LSTM Layer with Peephole Connections

Knowl 3 — Sentence-Level Word Attention Mechanism

Knowl 4 — Relation Classification Objective and Regularization

Knowl 5 — Training Configuration and Regularization Strategy for Att-BLSTM

Knowl 6 — Empirical Performance on SemEval-2010 Task 8 Relation Classification

References

Citation

BibTeX

Access the Paper