Back

Recurrent Convolutional Neural Networks for Text Classification

Siwei LaiLiheng XuKang LiuJun Zhao

article2015AAAI2,551 citations

Proposes a recurrent convolutional neural network that integrates bidirectional recurrent structures with max-pooling to capture global contextual information with linear computational complexity, outperforming traditional window-based CNNs and tree-based recursive models across text classification tasks.

Listen

The article addresses the challenge of text classification, a core task in applications such as web search, information filtering, and sentiment analysis. Traditional approaches depend on human-designed features like bag-of-words models or tree kernels, which often ignore context and word order or face data sparsity issues. Recent neural network methods, including recursive, recurrent, and convolutional networks, improve semantic capture but still struggle with bias, time complexity, or limited contextual range.

The article sets out to evaluate a recurrent convolutional neural network that learns word representations with broad context and identifies key text components automatically, without relying on manually crafted features. The authors test whether this hybrid model can outperform prior state-of-the-art methods across multiple classification tasks while maintaining linear time complexity.

The approach combines a bidirectional recurrent structure to build contextual word representations with a max-pooling layer that selects the most discriminative features. Experiments were run on four standard datasets covering English and Chinese texts, topic classification, sentiment analysis, and writing style detection, using pre-trained word embeddings and stochastic gradient descent for training. Results were compared against strong baselines such as logistic regression with n-grams, LDA variants, tree kernels, recursive networks, and standard convolutional networks.

The RCNN achieved the highest accuracy on three of the four datasets and competitive performance on the fourth. It reduced error rates by approximately one-third on the 20Newsgroups dataset and by nearly one-fifth on the Fudan set relative to the best prior results. The model captured longer-range context more effectively than fixed-window convolutional networks and required far less training time than recursive networks. It also matched or exceeded hand-crafted feature sets without needing syntactic parsers or language-specific engineering.

These outcomes indicate that the recurrent convolutional design can deliver more accurate and robust text classification for both short and long documents while remaining computationally efficient. The gains matter most for large-scale or low-resource settings where manual feature design is costly or impractical. Organizations relying on text classification may therefore achieve better performance and lower maintenance overhead by adopting similar neural architectures.

Further work should include testing the model on additional languages and tasks, exploring larger-scale pre-training, and conducting ablation studies to isolate the contribution of each component. The main limitations are the restriction to four datasets and dependence on the quality of pre-trained embeddings; results on very different domains or with limited unlabeled data may vary.

Institute of Automation, Chinese Academy of Sciencesconvolutional neural networksknowledge basetext classifiersRecurrent Convolutional Neural Networkstext classificationpre-trained word embeddings
Lai et al (2015).pdf
  • Paper: Hierarchical Attention Networks for Document Classification, Zichao Yang et al. (2016). It advances document-level neural classification beyond recurrent-convolutional pooling by introducing hierarchical bidirectional GRU representations paired with word- and sentence-level attention mechanisms.
  • Paper: Bag of Tricks for Efficient Text Classification, Armand Joulin et al. (2017). It explores an alternative paradigm by demonstrating how computationally lightweight linear models with n-gram features can match deep neural text classification architectures with vastly lower compute requirements.
  • Paper: A Structured Self-attentive Sentence Embedding, Zhouhan Lin et al. (2017). It builds on bidirectional sequence encoding by introducing a multi-hop self-attention mechanism to extract rich, structured sentence embeddings instead of relying on simple max-pooling.
  • Paper: End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF, Xuezhe Ma et al. (2016). It combines character-level CNN feature extractors with bidirectional LSTM sequence encoders to create an end-to-end architecture for sequence labeling.
  • Paper: Recent Trends in Deep Learning Based Natural Language Processing, Tom Young et al. (2017). This survey provides a comprehensive synthesis of how convolutional, recurrent, and hybrid deep architectures evolved to shape modern natural language processing benchmarks.
Cover for Recurrent Convolutional Neural Networks for Text Classification

Abstract

Text classification is a foundational task in many NLP applications. Traditional text classifiers often rely on many human-designed features, such as dictionaries, knowledge bases and special tree kernels. In contrast to traditional methods, we introduce a recurrent convolutional neural network for text classification without human-designed features. In our model, we apply a recurrent structure to capture contextual information as far as possible when learning word representations, which may introduce considerably less noise compared to traditional window-based neural networks. We also employ a max-pooling layer that automatically judges which words play key roles in text classification to capture the key components in texts. We conduct experiments on four commonly used datasets. The experimental results show that the proposed method outperforms the state-of-the-art methods on several datasets, particularly on document-level datasets.

Topics

convolutional neural networksknowledge basetext classifiers

Keywords

Recurrent Convolutional Neural Networkstext classificationpre-trained word embeddingssemi-supervised recursive autoencodersSentiment Treebankparaphrase detectiontopic classificationstochastic gradient descent

Table of Contents

  • Introduction
  • Related Work
  • Text Classification
  • Deep neural networks
  • Model
  • Word Representation Learning
  • Text Representation Learning
  • Training
  • Experiments
  • Datasets
  • Experiment Settings
  • Comparison of Methods
  • Results and Discussion
  • RCNN
  • RNTN
  • Conclusion
  • Acknowledgments
  • References

Knowls

  1. Knowl 1 — Bi-directional Recurrent Word Representation in RCNN

    model/method

    The Recurrent Convolutional Neural Network (RCNN) represents each word in a document by combining the word's own embedding with dense contextual vectors representing its entire left and right contexts within the text. For a document DD consisting of a sequence of words w1,w2,…,wnw_1, w_2, \dots, w_n, let e(wi)∈R∣e∣e(w_i) \in \mathbb{R}^{|e|} denote the word embedding of word wiw_i.

    The left-side context vector cl(wi)∈R∣c∣c_l(w_i) \in \mathbb{R}^{|c|} and right-side context vector cr(wi)∈R∣c∣c_r(w_i) \in \mathbb{R}^{|c|} are computed recurrently via forward and backward passes:

    cl(wi)=f(W(l)cl(wi−1)+W(sl)e(wi−1))c_l(w_i) = f\left(W^{(l)} c_l(w_{i-1}) + W^{(sl)} e(w_{i-1})\right)

    cr(wi)=f(W(r)cr(wi+1)+W(sr)e(wi+1))c_r(w_i) = f\left(W^{(r)} c_r(w_{i+1}) + W^{(sr)} e(w_{i+1})\right)

    where W(l),W(r)∈R∣c∣×∣c∣W^{(l)}, W^{(r)} \in \mathbb{R}^{|c| \times |c|} are transition matrices for the left and right recurrent hidden states, W(sl),W(sr)∈R∣c∣×∣e∣W^{(sl)}, W^{(sr)} \in \mathbb{R}^{|c| \times |e|} are matrices combining word embeddings into the context states, ff is a non-linear activation function, and cl(w1),cr(wn)∈R∣c∣c_l(w_1), c_r(w_n) \in \mathbb{R}^{|c|} are shared initial state parameters for the first and last words of any document.

    The full contextualized representation xi∈R∣e∣+2∣c∣x_i \in \mathbb{R}^{|e| + 2|c|} of word wiw_i is formed by concatenating the left context, the word embedding, and the right context:

    xi=[cl(wi);e(wi);cr(wi)]x_i = [c_l(w_i); e(w_i); c_r(w_i)]

    A linear transformation followed by a hyperbolic tangent non-linearity maps xix_i to a latent semantic vector yi(2)∈RHy_i^{(2)} \in \mathbb{R}^{H}:

    yi(2)=tanh⁡(W(2)xi+b(2))y_i^{(2)} = \tanh\left(W^{(2)} x_i + b^{(2)}\right)

    where W(2)∈RH×(∣e∣+2∣c∣)W^{(2)} \in \mathbb{R}^{H \times (|e| + 2|c|)} is a weight matrix, b(2)∈RHb^{(2)} \in \mathbb{R}^H is a bias vector, and HH is the hidden layer size. This recurrent formulation captures unbounded left and right context in O(n)O(n) time complexity without requiring a fixed window size or a syntactic parse tree.

  2. Knowl 2 — Max-Pooling and Document Classification Layer in RCNN

    model/method

    To aggregate word-level semantic representations into a fixed-length document representation, the Recurrent Convolutional Neural Network (RCNN) applies an element-wise max-pooling operation across the sequence of latent semantic vectors y1(2),y2(2),…,yn(2)∈RHy_1^{(2)}, y_2^{(2)}, \dots, y_n^{(2)} \in \mathbb{R}^H:

    y(3)=max⁡i=1nyi(2)y^{(3)} = \max_{i=1}^{n} y_i^{(2)}

    where the kk-th element of y(3)∈RHy^{(3)} \in \mathbb{R}^H is the maximum among the kk-th elements of all yi(2)y_i^{(2)} (i=1,…,ni = 1, \dots, n). Max-pooling extracts the most salient latent semantic features across the entire text, eliminating positional bias where later words dominate earlier words.

    The pooled vector y(3)y^{(3)} is passed to an output layer followed by a softmax function to generate the class probability distribution:

    y(4)=W(4)y(3)+b(4)y^{(4)} = W^{(4)} y^{(3)} + b^{(4)}

    p(k∣D,θ)=exp⁡(yk(4))∑j=1Oexp⁡(yj(4))p(k \mid D, \theta) = \frac{\exp\left(y^{(4)}_k\right)}{\sum_{j=1}^{O} \exp\left(y^{(4)}_j\right)}

    where W(4)∈RO×HW^{(4)} \in \mathbb{R}^{O \times H} is the output transformation matrix, b(4)∈ROb^{(4)} \in \mathbb{R}^O is the output bias vector, OO is the total number of document classes, and θ\theta denotes the complete parameter set of the network. The computational complexity of the entire cascade (recurrent representation, max-pooling, and output layer) is O(n)O(n), scaling linearly with text length nn.

  3. Knowl 3 — Optimization Objective, Parameter Initialization, and Training of RCNN

    model/method

    The trainable parameter set of the Recurrent Convolutional Neural Network (RCNN) is:

    θ={E,b(2),b(4),cl(w1),cr(wn),W(2),W(4),W(l),W(r),W(sl),W(sr)}\theta = \left\{E, b^{(2)}, b^{(4)}, c_l(w_1), c_r(w_n), W^{(2)}, W^{(4)}, W^{(l)}, W^{(r)}, W^{(sl)}, W^{(sr)}\right\}

    where E∈R∣e∣×∣V∣E \in \mathbb{R}^{|e| \times |V|} is the word embedding matrix for vocabulary VV, b(2)∈RHb^{(2)} \in \mathbb{R}^H, b(4)∈ROb^{(4)} \in \mathbb{R}^O, cl(w1),cr(wn)∈R∣c∣c_l(w_1), c_r(w_n) \in \mathbb{R}^{|c|}, W(2)∈RH×(∣e∣+2∣c∣)W^{(2)} \in \mathbb{R}^{H \times (|e|+2|c|)}, W(4)∈RO×HW^{(4)} \in \mathbb{R}^{O \times H}, W(l),W(r)∈R∣c∣×∣c∣W^{(l)}, W^{(r)} \in \mathbb{R}^{|c| \times |c|}, and W(sl),W(sr)∈R∣c∣×∣e∣W^{(sl)}, W^{(sr)} \in \mathbb{R}^{|c| \times |e|}.

    The training objective is to maximize the log-likelihood over the training set D\mathcal{D}:

    L(θ)=∑D∈Dlog⁡p(classD∣D,θ)\mathcal{L}(\theta) = \sum_{D \in \mathcal{D}} \log p(\text{class}_D \mid D, \theta)

    Parameters are optimized via Stochastic Gradient Descent (SGD) with step update:

    θ←θ+α∂log⁡p(classD∣D,θ)∂θ\theta \leftarrow \theta + \alpha \frac{\partial \log p(\text{class}_D \mid D, \theta)}{\partial \theta}

    where α\alpha is the learning rate (set to α=0.01\alpha = 0.01).

    All parameters are initialized from a uniform distribution with range bounds equal to ±1/fan-in\pm \sqrt{1 / \text{fan-in}}, where fan-in\text{fan-in} is the number of input nodes to the given layer, and the layer's learning rate is divided by fan-in\text{fan-in}. Word embeddings are pre-trained using the Skip-gram algorithm on Wikipedia corpora in the corresponding language.

  4. Knowl 4 — Benchmark Classification Performance of RCNN

    data/table

    The Recurrent Convolutional Neural Network (RCNN) was evaluated across four text classification benchmarks: 20Newsgroups (topic classification, 4 classes, Macro-F1 metric), Fudan Set (Chinese document classification, 20 classes, Accuracy), ACL Anthology Network (native language identification, 5 classes, Accuracy), and Stanford Sentiment Treebank (SST, sentence sentiment classification, 5 classes, Accuracy).

    Model 20News (F1) Fudan (Acc) ACL (Acc) SST (Acc)
    BoW + LR 92.81 92.08 46.67 40.86
    Bigram + LR 93.12 92.97 47.00 36.24
    BoW + SVM 92.43 93.02 45.24 40.70
    Bigram + SVM 92.32 93.03 46.14 36.61
    Average Embedding 89.39 86.89 41.32 32.70
    ClassifyLDA-EM 93.60 - - -
    Labeled-LDA - 90.80 - -
    CFG - - 39.20 -
    CJ - - 49.20 -
    RecursiveNN - - - 43.20
    RNTN - - - 45.70
    Paragraph-Vector - - - 48.70
    CNN 94.79 94.04 47.47 46.35
    RCNN 96.49 95.20 49.19 47.21

    RCNN achieves the highest performance among compared neural and traditional baselines on 20Newsgroups (96.49% Macro-F1, reducing error by 33% relative to ClassifyLDA-EM) and Fudan (95.20% Accuracy, reducing error by 19% relative to Labeled-LDA). On the ACL dataset, RCNN achieves 49.19% without syntactic parse features, matching the heavily engineered Charniak & Johnson (C&J) parse re-ranking baseline (49.20%). On SST, RCNN outperforms recursive models (RecursiveNN at 43.20%, RNTN at 45.70%) and standard window-based CNNs (46.35%).

  5. Knowl 5 — Context Window Sensitivity in CNNs versus Recurrent Context in RCNN

    empirical result

    When comparing standard Convolutional Neural Networks (CNNs) with fixed context window sizes to the Recurrent Convolutional Neural Network (RCNN) on the 20Newsgroups dataset, RCNN strictly outperforms CNN across all odd window sizes from 1 to 19:

    • CNN Macro-F1 by window size: win=1\text{win}=1 (93.26%), win=3\text{win}=3 (94.28%), win=5\text{win}=5 (94.69%), win=7\text{win}=7 (94.75%), win=9\text{win}=9 (94.76%), win=11\text{win}=11 (94.79%), win=13\text{win}=13 (94.76%), win=15\text{win}=15 (94.50%), win=17\text{win}=17 (94.45%), win=19\text{win}=19 (94.42%).
    • Baseline ClassifyLDA-EM: 93.60%.
    • RCNN: 96.49%.

    CNN performance peaks at a window size of 11 (94.79%) and degrades for larger windows due to parameter space expansion and data sparsity, while small windows lose long-range context. RCNN eliminates window size tuning by using a recurrent structure that captures long-range dependencies across the entire sequence while introducing less noise.

  6. Knowl 6 — Saliency and Keyword Extraction via RCNN Max-Pooling

    empirical result

    Analyzing the words and surrounding contexts most frequently selected by the max-pooling layer of RCNN reveals that the network automatically identifies class-discriminative semantic components without requiring a syntactic parser.

    On the Stanford Sentiment Treebank (SST) test set, the most frequently selected contextual trigrams include:

    • Positive indicators: "well worth the", "a wonderful movie", "even stinging at", "and invigorating film", "and ingenious entertainment", "and enjoy .", "'s sweetest movie".
    • Negative indicators: "A dreadful live-action", "Extremely boring .", "is n't a", "'s painful .", "Extremely dumb .", "an awfully derivative", "'s weaker than", "incredibly dull .", "very bad sign".

    Unlike Recursive Neural Tensor Networks (RNTNs), which extract phrase-level constituents constrained by syntactic parse trees (such as "an amazing performance" or "a complete failure"), RCNN extracts salient n-grams driven purely by recurrent context and max-pooling activations.

Coverage note — None was omitted; all core contributions, including the recurrent context and max-pooling architecture, optimization details, benchmark results across four datasets, context window sensitivity analysis, and max-pooling saliency interpretability, are fully covered.

References

  1. 1.Aggarwal, C. C., and Zhai, C. 2012. A survey of text classification algorithms. In Mining text data. Springer. 163–222.
  2. 2.Baroni, M.; Dinu, G.; and Kruszewski, G. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL, 238–247.
  3. 3.Bengio, Y.; Ducharme, R.; Vincent, P.; and Jauvin, C. 2003. A Neural Probabilistic Language Model. JMLR 3:1137–1155.
  4. 4.Bengio, Y.; Courville, A.; and Vincent, P. 2013. Representation learning: A review and new perspectives. IEEE TPAMI 35(8):1798–1828.
  5. 5.Bottou, L. 1991. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nımes, volume 91.
  6. 6.Cai, L., and Hofmann, T. 2003. Text categorization by boosting automatically extracted concepts. In SIGIR, 182–189.
  7. 7.Charniak, E., and Johnson, M. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In ACL, 173–180.
  8. 8.Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natural language processing (almost) from scratch. JMLR 12:2493–2537.
  9. 9.Cover, T. M., and Thomas, J. A. 2012. Elements of information theory. John Wiley & Sons.
  10. 10.Elman, J. L. 1990. Finding structure in time. Cognitive science 14(2):179–211.
  11. 11.Erhan, D.; Bengio, Y.; Courville, A.; Manzagol, P.-A.; Vincent, P.; and Bengio, S. 2010. Why does unsupervised pre-training help deep learning? JMLR 11:625–660.
  12. 12.Hingmire, S.; Chougule, S.; Palshikar, G. K.; and Chakraborti, S. 2013. Document classification by topic labeling. In SIGIR, 877–880.
  13. 13.Hinton, G. E., and Salakhutdinov, R. R. 2006. Reducing the dimensionality of data with neural networks. Science 313(5786):504–507.
  14. 14.Huang, E. H.; Socher, R.; Manning, C. D.; and Ng, A. Y. 2012. Improving word representations via global context and multiple word prototypes. In ACL, 873–882.
  15. 15.Kalchbrenner, N., and Blunsom, P. 2013. Recurrent convolutional neural networks for discourse compositionality. In Workshop on CVSC, 119–126.
  16. 16.Klementiev, A.; Titov, I.; and Bhattarai, B. 2012. Inducing crosslingual distributed representations of words. In Coling, 1459–1474.
  17. 17.Le, Q. V., and Mikolov, T. 2014. Distributed representations of sentences and documents. In ICML.
  18. 18.Lewis, D. D. 1992. An evaluation of phrasal and clustered representations on a text categorization task. In SIGIR, 37–50.
  19. 19.Li, W.; Sun, L.; and Zhang, D. 2008. Text classification based on labeled-lda model. Chinese Journal of Computers 31(4):620–627.
  20. 20.Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In NIPS, 3111–3119.
  21. 21.Mikolov, T.; Yih, W.-t.; and Zweig, G. 2013. Linguistic regularities in continuous space word representations. In NAACL-HLT, 746–751.
  22. 22.Mikolov, T. 2012. Statistical language models based on neural networks. Ph.D. Dissertation, Brno University of Technology.
  23. 23.Mnih, A., and Hinton, G. 2007. Three new graphical models for statistical language modelling. In ICML, 641–648.
  24. 24.Morin, F., and Bengio, Y. 2005. Hierarchical probabilistic neural network language model. In AISTATS, 246–252.
  25. 25.Ng, A. Y. 2004. Feature selection, l1 vs. l2 regularization, and rotational invariance. In ICML, 78.
  26. 26.Petrov, S.; Barrett, L.; Thibaux, R.; and Klein, D. 2006. Learning accurate, compact, and interpretable tree annotation. In Coling-ACL, 433–440.
  27. 27.Plaut, D. C., and Hinton, G. E. 1987. Learning sets of filters using back-propagation. Computer Speech & Language 2(1):35–61.
  28. 28.Post, M., and Bergsma, S. 2013. Explicit and implicit syntactic features for text classification. In ACL, 866–872.
  29. 29.Socher, R.; Huang, E. H.; Pennington, J.; Ng, A. Y.; and Manning, C. D. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS, volume 24, 801–809.
  30. 30.Socher, R.; Pennington, J.; Huang, E. H.; Ng, A. Y.; and Manning, C. D. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In EMNLP, 151–161.
  31. 31.Socher, R.; Perelygin, A.; Wu, J. Y.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 1631–1642.
  32. 32.Turian, J.; Ratinov, L.; and Bengio, Y. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL, 384–394.
  33. 33.Wang, S., and Manning, C. D. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, 90–94.

Citation

MLA
Lai, S., et al. “Recurrent Convolutional Neural Networks for Text Classification”. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, no. 1, 2015, https://doi.org/10.1609/aaai.v29i1.9513.
APA
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015). Recurrent Convolutional Neural Networks for Text Classification. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). https://doi.org/10.1609/aaai.v29i1.9513
Chicago
Lai, S., L. Xu, K. Liu, and J. Zhao. 2015. “Recurrent Convolutional Neural Networks for Text Classification”. Proceedings of the AAAI Conference on Artificial Intelligence 29 (1). https://doi.org/10.1609/aaai.v29i1.9513.
Harvard
Lai, S. et al. (2015) “Recurrent Convolutional Neural Networks for Text Classification”, Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). Available at: https://doi.org/10.1609/aaai.v29i1.9513.
Vancouver
1. Lai S, Xu L, Liu K, Zhao J (2015) Recurrent Convolutional Neural Networks for Text Classification. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v29i1.9513

BibTeX

@article{Lai_2015, title={Recurrent Convolutional Neural Networks for Text Classification}, volume={29}, ISSN={2159-5399}, url={http://dx.doi.org/10.1609/aaai.v29i1.9513}, DOI={10.1609/aaai.v29i1.9513}, number={1}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, publisher={Association for the Advancement of Artificial Intelligence (AAAI)}, author={Lai, Siwei and Xu, Liheng and Liu, Kang and Zhao, Jun}, year={2015}, month=Feb }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF