Back

End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Xuezhe MaEduard Hovy

article2016ACL2,834 citations

Proposes an end-to-end neural architecture combining CNNs, bidirectional LSTMs, and CRFs that eliminates manual feature engineering while achieving state-of-the-art performance in part-of-speech tagging and named entity recognition.

Listen

Traditional sequence labeling systems for tasks like part-of-speech tagging and named entity recognition rely on hand-crafted features and task-specific resources that are expensive to create and hard to adapt across domains. This limits scalability and performance when moving to new languages or data types.

The article set out to build and test a neural network model that performs sequence labeling in a fully end-to-end manner, using only pre-trained word embeddings and no manual feature engineering or data pre-processing.

The authors combined convolutional networks to capture character-level patterns, bidirectional LSTMs to model word context, and a conditional random field layer to jointly decode label sequences. They trained and evaluated the system on the standard Penn Treebank WSJ split for POS tagging and the CoNLL 2003 English data for NER.

The model reached 97.55 percent accuracy on POS tagging and 91.21 F1 on NER, exceeding prior state-of-the-art results on both benchmarks. Adding the CRF layer produced the largest gains, especially on words absent from both training data and embedding vocabularies. Pre-trained embeddings proved essential, with performance varying noticeably across different embedding sets.

These results show that joint modeling of character and word information plus structured decoding can replace hand-crafted features while improving accuracy. The approach lowers development cost and makes sequence labeling easier to apply to new domains or languages.

Further work should explore multi-task training that combines related labeling tasks and test the model on social media or other out-of-domain text. The main uncertainties stem from dependence on the quality of pre-trained embeddings and the relatively narrow set of two English benchmarks used for evaluation.

Carnegie Mellon Universityconvolutional neural networksengineeringlong short term memory neural networksBi-directional LSTM-CNNsSequence labelingStructured prediction
arXiv: 1603.01354XuezheMax/LasagneNLP
Cover for End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF

Abstract

State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF. Our system is truly end-to-end, requiring no feature engineering or data pre-processing, thus making it applicable to a wide range of sequence labeling tasks. We evaluate our system on two data sets for two sequence labeling tasks --- Penn Treebank WSJ corpus for part-of-speech (POS) tagging and CoNLL 2003 corpus for named entity recognition (NER). We obtain state-of-the-art performance on both the two data --- 97.55% accuracy for POS tagging and 91.21% F1 for NER.

Topics

convolutional neural networksengineeringlong short term memory neural networksnamed entity recognitionnetwork architecturepos taggingsequence labelingtreebanks

Keywords

Bi-directional LSTM-CNNsSequence labelingStructured predictionWord-level representationsJoint decoding

Table of Contents

  • 1 Introduction
  • 2 Neural Network Architecture
  • 2.1 CNN for Character-level Representation
  • 2.2 Bi-directional LSTM
  • 2.2.1 LSTM Unit
  • 2.2.2 BLSTM
  • 2.3 CRF
  • 2.4 BLSTM-CNNs-CRF
  • 3 Network Training
  • 3.1 Parameter Initialization
  • 3.2 Optimization Algorithm
  • 3.3 Tuning Hyper-Parameters
  • 4 Experiments
  • 4.1 Data Sets
  • 4.2 Main Results
  • 4.3 Comparison with Previous Work
  • 4.3.1 POS Tagging
  • 4.3.2 NER
  • 4.4 Word Embeddings
  • 4.5 Effect of Dropout
  • 4.6 OOV Error Analysis
  • 5 Related Work
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — BLSTM-CNNs-CRF End-to-End Sequence Labeling Architecture

    model/method

    The BLSTM-CNNs-CRF architecture is an end-to-end neural network for linguistic sequence labeling tasks (such as POS tagging and Named Entity Recognition) that eliminates the need for handcrafted features, gazetteers, or task-specific data pre-processing.

    Given an input sentence represented as a sequence of words (w1,w2,…,wn)(w_1, w_2, \dots, w_n), the model processes each word via a three-tiered hierarchy:

    1. Character-level representation: For each word wtw_t, character embeddings are passed through a 1D Convolutional Neural Network (CNN) followed by max pooling over the characters of the word, producing a character-level feature vector xtchar\mathbf{x}_t^{\text{char}}.
    2. Word-level contextual representation: The character-level representation xtchar\mathbf{x}_t^{\text{char}} is concatenated with a pre-trained word embedding vector xtword\mathbf{x}_t^{\text{word}} to produce the composite input vector xt=[xtword;xtchar]\mathbf{x}_t = [\mathbf{x}_t^{\text{word}}; \mathbf{x}_t^{\text{char}}]. A Bi-directional LSTM (BLSTM) processes the sequence of input vectors forwards and backwards. The forward hidden state h→t\overrightarrow{\mathbf{h}}_t and backward hidden state h←t\overleftarrow{\mathbf{h}}_t at step tt are concatenated into a context-aware representation zt=[h→t;h←t]\mathbf{z}_t = [\overrightarrow{\mathbf{h}}_t; \overleftarrow{\mathbf{h}}_t].
    3. Joint label decoding: The sequence of hidden states z=(z1,…,zn)\mathbf{z} = (\mathbf{z}_1, \dots, \mathbf{z}_n) is passed into a linear-chain Conditional Random Field (CRF) layer that jointly models label transitions and outputs the globally highest-scoring label sequence y∗=(y1∗,…,yn∗)\mathbf{y}^* = (y_1^*, \dots, y_n^*).

    Dropout regularization is applied at three positions: on character embeddings prior to the CNN, on composite inputs to the BLSTM, and on the output hidden states of the BLSTM.

  2. Knowl 2 — Character-Level Feature Extraction Using Convolutional Neural Networks

    model/method

    To automatically capture word morphological information (such as prefixes and suffixes) without task-specific character-type features or capitalization rules, a 1D Convolutional Neural Network (CNN) is applied directly to character embeddings.

    For a given word consisting of a sequence of characters, each character is mapped to a 30-dimensional character embedding initialized uniformly from [−3/dim,+3/dim][-\sqrt{3/\text{dim}}, +\sqrt{3/\text{dim}}] where dim=30\text{dim} = 30. Character embeddings are passed through a dropout layer with dropout rate p=0.5p = 0.5. The sequence of embeddings is padded and processed by 30 convolutional filters with a window length of 3 characters. A max-pooling operation is then applied across the entire length of the word to extract the most salient feature for each filter, yielding a fixed 30-dimensional character-level representation vector for the word.

  3. Knowl 3 — Sequential Conditional Random Field Layer and Objective

    equation

    For an input word sequence represented by BLSTM output vectors z=(z1,z2,…,zn)\mathbf{z} = (\mathbf{z}_1, \mathbf{z}_2, \dots, \mathbf{z}_n) and a candidate label sequence y=(y1,y2,…,yn)∈Y(z)\mathbf{y} = (y_1, y_2, \dots, y_n) \in \mathcal{Y}(\mathbf{z}), the conditional probability p(y∣z;W,b)p(\mathbf{y} \mid \mathbf{z}; \mathbf{W}, \mathbf{b}) under a linear-chain Conditional Random Field (CRF) is:

    p(y∣z;W,b)=∏i=1nψi(yi−1,yi,z)∑y′∈Y(z)∏i=1nψi(yi−1′,yi′,z)p(\mathbf{y} \mid \mathbf{z}; \mathbf{W}, \mathbf{b}) = \frac{\prod_{i=1}^n \psi_i(y_{i-1}, y_i, \mathbf{z})}{\sum_{\mathbf{y}' \in \mathcal{Y}(\mathbf{z})} \prod_{i=1}^n \psi_i(y'_{i-1}, y'_i, \mathbf{z})}

    where ψi(y′,y,z)=exp⁡(Wy′,y⊤zi+by′,y)\psi_i(y', y, \mathbf{z}) = \exp(\mathbf{W}_{y', y}^\top \mathbf{z}_i + b_{y', y}) denotes the potential function for the transition from label y′y' to label yy at position ii, with weight vector Wy′,y\mathbf{W}_{y', y} and scalar bias by′,yb_{y', y}.

    The model parameters are optimized via maximum conditional likelihood over a training set {(z(j),y(j))}\{(\mathbf{z}^{(j)}, \mathbf{y}^{(j)})\} by maximizing the log-likelihood:

    L(W,b)=∑jlog⁡p(y(j)∣z(j);W,b)\mathcal{L}(\mathbf{W}, \mathbf{b}) = \sum_j \log p(\mathbf{y}^{(j)} \mid \mathbf{z}^{(j)}; \mathbf{W}, \mathbf{b})

    During inference, global sequence decoding searches for the optimal label sequence y∗\mathbf{y}^* using the Viterbi algorithm:

    y∗=arg⁡max⁡y∈Y(z)p(y∣z;W,b)\mathbf{y}^* = \arg\max_{\mathbf{y} \in \mathcal{Y}(\mathbf{z})} p(\mathbf{y} \mid \mathbf{z}; \mathbf{W}, \mathbf{b})

  4. Knowl 4 — Layer Ablation Analysis on POS Tagging and Named Entity Recognition

    data/table

    An ablation study demonstrates the incremental performance contributions of each architectural component across two standard benchmarks: Part-of-Speech (POS) tagging on the Penn Treebank Wall Street Journal (WSJ) corpus and Named Entity Recognition (NER) on the CoNLL-2003 English shared task. All configurations utilized 100-dimensional GloVe word embeddings.

    Model POS (Accuracy %) NER (%)
    Dev Test Dev Prec. Dev Recall Dev F1 Test Prec. Test Recall Test F1
    BRNN 96.56 96.76 92.04 89.13 90.56 87.05 83.88 85.44
    BLSTM 96.88 96.93 92.31 90.85 91.57 87.77 86.23 87.00
    BLSTM-CNN 97.34 97.33 92.52 93.64 93.07 88.53 90.21 89.36
    BLSTM-CNN-CRF 97.46 97.55 94.85 94.63 94.74 91.35 91.06 91.21

    The results establish three findings:

    1. BLSTM outperforms BRNN across all metrics by effectively retaining long-distance sequence context without vanishing/exploding gradients.
    2. Incorporating CNN-extracted character representations into BLSTM yields a substantial gain (+0.40% POS test accuracy, +2.36% NER test F1), confirming the importance of sub-word morphological information.
    3. Adding the CRF layer for structured joint decoding delivers a further boost (+0.22% POS test accuracy, +1.85% NER test F1), demonstrating the necessity of modeling label transition dependencies.

  5. Knowl 5 — Training and Hyperparameter Specifications

    experimental setup

    The BLSTM-CNNs-CRF model is trained using the following settings across tasks:

    • CNN Layer: 30 filters, window length of 3 characters, applied on 30-dimensional character embeddings.
    • BLSTM Layer: Hidden state size of 200 per direction (400 total), initial hidden state 0.0, no peephole connections.
    • Dropout: Dropout rate of 0.5 applied to character embeddings, BLSTM input vectors, and BLSTM output vectors.
    • Parameter Initialization: Character embeddings sampled from [−3/30,+3/30][-\sqrt{3/30}, +\sqrt{3/30}]. Weight matrices sampled uniformly from [−6/(r+c),+6/(r+c)][-\sqrt{6/(r+c)}, +\sqrt{6/(r+c)}] where rr and cc are row and column counts. Biases initialized to zero, except the LSTM forget gate bias bfb_f initialized to 1.0. Word embeddings are fine-tuned via backpropagation.
    • Optimization: Mini-batch Stochastic Gradient Descent (SGD) with batch size 10 and momentum 0.9. Gradient clipping is set to 5.0. Learning rate schedule: ηt=η0/(1+ρt)\eta_t = \eta_0 / (1 + \rho t) with decay rate ρ=0.05\rho = 0.05 per completed epoch tt. Initial learning rate η0=0.01\eta_0 = 0.01 for POS tagging and η0=0.015\eta_0 = 0.015 for NER.
    • Early Stopping: Monitored on development set performance; optimal parameters consistently appear around epoch 50.
    • Corpus Formats: PTB WSJ uses sections 0–18 (train, 38,219 sentences), 19–21 (dev, 5,527 sentences), and 22–24 (test, 5,462 sentences) with 45 POS tags. CoNLL-2003 English NER uses standard train (14,987 sentences), dev (3,466 sentences), and test (3,684 sentences) splits with the BIOES tagging scheme.

  6. Knowl 6 — Benchmark Performance Comparison on WSJ PTB and CoNLL-2003

    data/table

    The end-to-end BLSTM-CNNs-CRF system matches or exceeds all previous state-of-the-art systems on both POS tagging and NER benchmarks without requiring hand-engineered features or external resources (e.g., gazetteers, WordNet, discrete suffix rules).

    POS Tagging (PTB WSJ) NER (CoNLL-2003)
    Model Acc. (%) Model F1 (%)
    Giménez and Màrquez (2004) 97.16 Chieu and Ng (2002) 88.31
    Toutanova et al. (2003) 97.27 Florian et al. (2003) 88.76
    Manning (2011) 97.28 Ando and Zhang (2005) 89.31
    Collobert et al. (2011) 97.29 Collobert et al. (2011) 89.59
    Santos and Zadrozny (2014) 97.32 Huang et al. (2015) 90.10
    Shen et al. (2007) 97.33 Chiu and Nichols (2015) 90.77
    Sun (2014) 97.36 Ratinov and Roth (2009) 90.80
    Søgaard (2011) 97.50 Lin and Wu (2009) 90.90
    – – Passos et al. (2014) 90.90
    – – Lample et al. (2016) 90.94
    – – Luo et al. (2015) 91.20
    BLSTM-CNNs-CRF (This paper) 97.55 BLSTM-CNNs-CRF (This paper) 91.21

    On POS tagging, the model improves upon the previous best result (Søgaard, 2011) by +0.05% and surpasses CharWNN (Santos and Zadrozny, 2014) by +0.23%. On NER, it edges out the joint entity recognition and disambiguation model of Luo et al. (2015) (91.20% F1), which relied heavily on Freebase, Wikipedia, Brown clusters, and POS/chunk tags, whereas BLSTM-CNNs-CRF relies solely on pre-trained word embeddings.

  7. Knowl 7 — Impact of Pre-trained Word Embeddings on Sequence Labeling

    data/table

    Evaluating different initial word embeddings under the identical BLSTM-CNNs-CRF architecture demonstrates the critical role of pre-trained distributed representations, particularly for NER.

    Embedding Dimension POS Accuracy (%) NER F1 (%)
    Random 100 97.13 80.76
    Senna 50 97.44 90.28
    Word2Vec 300 97.40 84.91
    GloVe 100 97.55 91.21

    Key observations:

    • Pre-trained embeddings provide moderate gains on POS tagging (+0.42% accuracy from Random to GloVe) but are crucial for NER (+10.45% F1 from Random to GloVe).
    • 100-dimensional GloVe embeddings (trained on 6B tokens of Wikipedia and web text) outperform 50-dimensional Senna embeddings and 300-dimensional Word2Vec embeddings on both tasks.
    • Word2Vec achieves competitive accuracy on POS tagging (97.40%) but performs poorly on NER (84.91%). This disparity is caused by vocabulary mismatch: Word2Vec embeddings were trained in a case-sensitive manner that omitted punctuation and digits, which degrades performance in a purely end-to-end model that applies no preprocessing to normalize numbers or rare symbols.

  8. Knowl 8 — Out-of-Vocabulary Analysis and CRF Generalization

    data/table

    To evaluate how the architecture handles rare and unseen words, tokens and entities in the evaluation sets are partitioned into four disjoint subsets:

    • IV (In-Vocabulary): Appears in both the training set and the pre-trained embedding vocabulary.
    • OOTV (Out-of-Training-Vocabulary): Appears in the pre-trained embedding vocabulary, but not in the training set.
    • OOEV (Out-of-Embedding-Vocabulary): Appears in the training set, but not in the pre-trained embedding vocabulary.
    • OOBV (Out-of-Both-Vocabulary): Appears in neither the training set nor the embedding vocabulary.
    Model POS Dev / Test Accuracy (%) NER Dev / Test F1 (%)
    IV OOTV OOEV OOBV IV OOTV OOEV OOBV
    Development Set
    LSTM-CNN 97.57 93.75 90.29 80.27 94.83 87.28 96.55 82.90
    LSTM-CNN-CRF 97.68 93.65 91.05 82.71 96.49 88.63 97.67 86.91
    Test Set
    LSTM-CNN 97.55 93.45 90.14 80.07 90.07 89.45 100.00 78.44
    LSTM-CNN-CRF 97.77 93.16 90.65 82.49 92.14 90.73 100.00 80.60

    The largest performance improvement from adding the CRF layer occurs on the OOBV partition (+2.44% dev / +2.42% test accuracy on POS; +4.01% dev / +2.16% test F1 on NER). This shows that joint decoding with structured neighborhood label constraints compensates for the absence of word-level and embedding-level representations.

  9. Knowl 9 — Effect of Multi-Layer Dropout Regularization

    data/table

    Applying dropout (p=0.5p = 0.5) simultaneously to the character embeddings (before the CNN), the concatenated word and character representations (before the BLSTM), and the BLSTM output states is crucial to preventing severe overfitting.

    Dropout POS Accuracy (%) NER F1 (%)
    Train Dev Test Train Dev Test
    No Dropout 98.46 97.06 97.11 99.97 93.51 89.25
    With Dropout 97.86 97.46 97.55 99.63 94.74 91.21

    Without dropout, the network achieves near-perfect training set scores (99.97% F1 on NER) but degrades significantly on unseen evaluation sets (89.25% test F1). Adding multi-layer dropout closes the generalization gap, improving POS test accuracy by +0.44% and NER test F1 by +1.96%.

Coverage note — No substantial contributed material was omitted. The knowls cover the full BLSTM-CNNs-CRF architecture, layer equations, training algorithms, hyperparameter configurations, ablation studies, empirical state-of-the-art results, word embedding analyses, out-of-vocabulary breakdowns, and dropout regularization experiments.

References

  1. 1.Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. The Journal of Machine Learning Research, 6:1817–1853.
  2. 2.Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. Neural Networks, IEEE Transactions on, 5(2):157–166.
  3. 3.James Bergstra, Olivier Breuleux, Fred́ eric Bastien, ́ Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a cpu and gpu math expression compiler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4, page 3. Austin, TX.
  4. 4.Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP-2014, pages 740–750, Doha, Qatar, October.
  5. 5.Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of CoNLL-2003, pages 1–7.
  6. 6.Jason PC Chiu and Eric Nichols. 2015. Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308.
  7. 7.Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Translation, page 103.
  8. 8.Ronan Collobert, Jason Weston, Leon Bottou, Michael ́ Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537.
  9. 9.Hong-Jie Dai, Po-Ting Lai, Yung-Chun Chang, and Richard Tzong-Han Tsai. 2015. Enhancing of chemical compound and drug name recognition using representative tag scheme and fine-grained tokenization. Journal of cheminformatics, 7(S1):1–10.
  10. 10.Yann N Dauphin, Harm de Vries, Junyoung Chung, and Yoshua Bengio. 2015. Rmsprop and equilibrated adaptive learning rates for non-convex optimization. arXiv preprint arXiv:1502.04390.
  11. 11.Cıcero dos Santos, Victor Guimaraes, RJ Niteroi, and ́ Rio de Janeiro. 2015. Boosting named entity recognition with neural character embeddings. In Proceedings of NEWS 2015 The Fifth Named Entities Workshop, page 25.
  12. 12.Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transitionbased dependency parsing with stack long shortterm memory. In Proceedings of ACL-2015 (Volume 1: Long Papers), pages 334–343, Beijing, China, July.
  13. 13.Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of HLT-NAACL-2003, pages 168–171.
  14. 14.Felix A Gers, Jurgen Schmidhuber, and Fred Cummins. ¨ 2000. Learning to forget: Continual prediction with lstm. Neural computation, 12(10):2451–2471.
  15. 15.Felix A Gers, Nicol N Schraudolph, and Jurgen ¨ Schmidhuber. 2003. Learning precise timing with lstm recurrent networks. The Journal of Machine Learning Research, 3:115–143.
  16. 16.Rich Caruana Steve Lawrence Lee Giles. 2001. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, volume 13, page 402. MIT Press.
  17. 17.Jesus Gim ́ enez and Llu ́ ́ıs Marquez. 2004. Svmtool: A ` general pos tagger generator based on support vector machines. In In Proceedings of LREC-2004.
  18. 18.Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, pages 249–256.
  19. 19.Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 347–352. IEEE.
  20. 20.Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP2013, pages 6645–6649. IEEE.
  21. 21.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034.
  22. 22.Sepp Hochreiter and Jurgen Schmidhuber. 1997. ¨ Long short-term memory. Neural computation, 9(8):1735–1780.
  23. 23.Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard H. Hovy, and Eric P. Xing. 2016. Harnessing deep neural networks with logic rules. In Proceedings of ACL-2016, Berlin, Germany, August.
  24. 24.Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
  25. 25.Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2342–2350.
  26. 26.Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  27. 27.Terry Koo and Michael Collins. 2010. Efficient thirdorder dependency parsers. In Proceedings of ACL2010, pages 1–11, Uppsala, Sweden, July.
  28. 28.Matthieu Labeau, Kevin Loser, Alexandre Allauzen, ¨ and Rue John von Neumann. 2015. Non-lexical neural architecture for fine-grained pos tagging. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 232–237.
  29. 29.John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML-2001, volume 951, pages 282–289.
  30. 30.Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-2016, San Diego, California, USA, June.
  31. 31.Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551.
  32. 32.Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of ACL2009, pages 1030–1038.
  33. 33.Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of EMNLP-2015, pages 1520–1530, Lisbon, Portugal, September.
  34. 34.Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of EMNLP-2015, pages 879–888, Lisbon, Portugal, September.
  35. 35.Xuezhe Ma and Eduard Hovy. 2015. Efficient innerto-outer greedy algorithm for higher-order labeled dependency parsing. In Proceedings of the EMNLP2015, pages 1322–1328, Lisbon, Portugal, September.
  36. 36.Xuezhe Ma and Fei Xia. 2014. Unsupervised dependency parsing with transferring distribution via parallel guidance and entropy regularization. In Proceedings of ACL-2014, pages 1337–1348, Baltimore, Maryland, June.
  37. 37.Xuezhe Ma and Hai Zhao. 2012a. Fourth-order dependency parsing. In Proceedings of COLING 2012: Posters, pages 785–796, Mumbai, India, December.
  38. 38.Xuezhe Ma and Hai Zhao. 2012b. Probabilistic models for high-order projective dependency parsing. Technical Report, arXiv:1502.04174.
  39. 39.Xuezhe Ma, Zhengzhong Liu, and Eduard Hovy. 2016. Unsupervised ranking model for entity coreference resolution. In Proceedings of NAACL-2016, San Diego, California, USA, June.
  40. 40.Christopher D Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In Computational Linguistics and Intelligent Text Processing, pages 171–189. Springer.
  41. 41.Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330.
  42. 42.Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL-2005, pages 91–98, Ann Arbor, Michigan, USA, June 2530.
  43. 43.Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119.
  44. 44.Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of ACL-2010, pages 1396–1411, Uppsala, Sweden, July. Association for Computational Linguistics.
  45. 45.Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-2004, pages 64–70, Geneva, Switzerland, August 23-27.
  46. 46.Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063.
  47. 47.Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of CoNLL2014, pages 78–86, Ann Arbor, Michigan, June.
  48. 48.Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In Proceedings of EMNLP2015, pages 548–554, Lisbon, Portugal, September.
  49. 49.Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In Proceedings of ACL-2016, Berlin, Germany, August.
  50. 50.Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP-2014, pages 1532–1543, Doha, Qatar, October.
  51. 51.Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of CoNLL-2009, pages 147–155.
  52. 52.Cicero D Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of ICML-2014, pages 1818–1826.
  53. 53.Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classification. In Proceedings of ACL-2007, volume 7, pages 760–767.
  54. 54.Anders Søgaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 48–52, Portland, Oregon, USA, June.
  55. 55.Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958.
  56. 56.Xu Sun. 2014. Structure regularization for structured prediction. In Advances in Neural Information Processing Systems, pages 2402–2410.
  57. 57.Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003 - Volume 4, pages 142–147, Stroudsburg, PA, USA.
  58. 58.Erik F. Tjong Kim Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of EACL’99, pages 173–179. Bergen, Norway.
  59. 59.Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich partof-speech tagging with a cyclic dependency network. In Proceedings of NAACL-HLT-2003, Volume 1, pages 173–180.
  60. 60.Zhilin Yang, Ruslan Salakhutdinov, and William Cohen. 2016. Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270.
  61. 61.Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.

Citation

MLA
Ma, X., and E. Hovy. “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF”. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 1064–74, https://doi.org/10.18653/v1/P16-1101.
APA
Ma, X., & Hovy, E. (2016). End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1064–1074. https://doi.org/10.18653/v1/P16-1101
Chicago
Ma, X., and E. Hovy. 2016. “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF”. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1064–74. https://doi.org/10.18653/v1/P16-1101.
Harvard
Ma, X. and Hovy, E. (2016) “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF”, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp. 1064–1074. Available at: https://doi.org/10.18653/v1/P16-1101.
Vancouver
1. Ma X, Hovy E (2016) End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp 1064–1074

BibTeX

@inproceedings{ma-hovy-2016-end,
    title = "End-to-end Sequence Labeling via Bi-directional {LSTM}-{CNN}s-{CRF}",
    author = "Ma, Xuezhe  and
      Hovy, Eduard",
    editor = "Erk, Katrin  and
      Smith, Noah A.",
    booktitle = "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2016",
    address = "Berlin, Germany",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/P16-1101/",
    doi = "10.18653/v1/P16-1101",
    pages = "1064--1074"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/