Skip-Thought Vectors

Ryan KirosYukun ZhuRuslan SalakhutdinovRichard S. ZemelRaquel UrtasunAntonio TorralbaSanja Fidler

article2015NeurIPS2,488 citations

Introduces an unsupervised encoder-decoder model that learns universal sentence embeddings by predicting adjacent text, yielding versatile representations that transfer effectively across diverse language benchmarks.

Listen

The article addresses the challenge of creating sentence representations that capture semantic and syntactic meaning in a way that works across many different language tasks without needing task-specific retraining. This matters because most existing methods tune representations to one narrow problem, limiting their usefulness for broader applications like search, classification, or understanding text at scale.

The article set out to evaluate whether an unsupervised encoder-decoder model, trained to reconstruct surrounding sentences from a given sentence, could produce generic, high-quality sentence vectors that perform well when used off-the-shelf with simple linear classifiers.

The approach involved training recurrent neural network models on a large corpus of over 74 million sentences from free novels, using an objective that predicts the previous and next sentences. After training, the encoder was frozen and tested as a fixed feature extractor on eight tasks, including semantic relatedness, paraphrase detection, image-sentence ranking, and five standard classification benchmarks. A vocabulary expansion technique mapped external word vectors into the model to handle unseen words.

The key findings are that the resulting skip-thought vectors achieved strong results across all tasks, often matching or exceeding prior unsupervised methods and some supervised ones; they outperformed SemEval 2014 submissions on semantic relatedness while remaining competitive with dependency tree-LSTMs; they reached near state-of-the-art on paraphrase detection when combined with basic features; and they performed on par with specialized models on image-sentence retrieval using COCO data. On classification benchmarks, performance was comparable to bag-of-words baselines but did not surpass task-tuned representations.

These results indicate that skip-thought vectors provide robust, reusable sentence features that reduce the need for heavy feature engineering or per-task training, potentially lowering costs and complexity in applications involving text understanding. They also suggest that large-scale unsupervised training on narrative text can yield representations competitive with methods requiring expensive labeled data.

Next steps supported by the work include exploring deeper encoders and decoders, larger context windows, paragraph-level modeling, and alternative architectures such as convolutional networks to further improve representation quality. The authors plan to release the encoder publicly.

The main limitations are that the model underperformed task-specific supervised methods on sentiment classification and that results rely on the BookCorpus domain, so generalization to other text types may vary; confidence is high for the reported tasks given consistent linear-model evaluations but lower for claims about broader applicability without additional validation.

CIFAR Massachusetts Institute of Technology University of Toronto classification encoder-decoder linear models Skip-Thought Vectors Encoder-Decoder Model BookCorpus Dataset

arXiv: 1506.06726 ryankiros/skip-thoughts

Paper: Sequence to Sequence Learning with Neural Networks, Ilya Sutskever et al. (2014). It introduces the fundamental sequence-to-sequence encoder-decoder framework using recurrent networks that Skip-Thought adapts to predict surrounding sentences.
Paper: Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, Kyunghyun Cho et al. (2014). It establishes the gated recurrent unit (GRU) and the RNN encoder-decoder architecture upon which Skip-Thought's sentence encoders and decoders are directly built.
Paper: Distributed Representations of Sentences and Documents, Quoc V. Le et al. (2014). It pioneers unsupervised distributed sentence and document representations, providing the conceptual baseline and inspiration for learning sentence-level embeddings.
Paper: Efficient Estimation of Word Representations in Vector Space, Tomáš Mikolov et al. (2013). It introduces the skip-gram objective of predicting context from a center target, the core learning principle generalized from words to sentences in Skip-Thought.
Paper: Distributed Representations of Words and Phrases and their Compositionality, Tomas Mikolov et al. (2013). It refines skip-gram modeling and vector space properties, establishing foundational techniques for distributed representations that Skip-Thought extends to sentence contexts.
Paper: GloVe: Global Vectors for Word Representation, Jeffrey Pennington et al. (2014). It provides the pre-trained word embedding space utilized by Skip-Thought's vocabulary expansion technique to map unseen words into the sentence encoder.
Paper: Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks, Kai Sheng Tai et al. (2015). It presents Tree-LSTM architectures and semantic relatedness evaluation benchmarks on the SICK dataset that serve as direct points of comparison for Skip-Thought vectors.
Paper: Convolutional Neural Networks for Sentence Classification, Yoon Kim (2014). It defines the standard suite of sentence classification benchmarks used to evaluate the transferability of frozen Skip-Thought representations.

Paper: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data, Alexis Conneau et al. (2017). It evaluates supervised training on natural language inference as an explicit alternative to Skip-Thought's unsupervised objective for producing universal sentence representations.
Paper: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers et al. (2019). It develops siamese Transformer networks to produce fixed-size sentence embeddings, directly advancing universal sentence representation beyond RNN-based skip-thought models.
Paper: Universal Sentence Encoder, Daniel Cer et al. (2018). It extends the paradigm of general-purpose sentence representations by introducing multi-task Transformer and deep averaging encoders evaluated across transfer tasks.
Paper: SimCSE: Simple Contrastive Learning of Sentence Embeddings, Tianyu Gao et al. (2021). It refines sentence embedding quality by replacing reconstructive sequence objectives with contrastive learning frameworks on top of pre-trained encoders.
Paper: A Structured Self-attentive Sentence Embedding, Zhouhan Lin et al. (2017). It builds upon fixed vector sentence embeddings by introducing self-attentive 2D matrix representations to capture multiple semantic components per sentence.
Paper: Generating Sentences from a Continuous Space, Samuel R. Bowman et al. (2016). It advances continuous sentence-space modeling by applying variational autoencoders with recurrent decoders to the same BookCorpus domain introduced alongside Skip-Thought.
Paper: Text Embeddings by Weakly-Supervised Contrastive Pre-training, Liang Wang et al. (2022). It advances universal text representation learning through weakly-supervised contrastive pre-training across broad retrieval and semantic benchmarks.

Abstract

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

Topics

classification encoder-decoder linear models semantic similarity supervised machine learning syntactic structure unsupervised learning vocabulary expansion methods

Keywords

Skip-Thought Vectors Encoder-Decoder Model BookCorpus Dataset Vocabulary Expansion GRU Linear Classifier Semantic Relatedness Paraphrase Detection Image‑Sentence Ranking t‑SNE Visualization

1 Introduction
2 Approach
2.1 Inducing skip-thought vectors
2.2 Vocabulary expansion
3 Experiments
3.1 Details of training
3.2 Semantic relatedness
3.3 Paraphrase detection
3.4 Image-sentence ranking
3.5 Classification benchmarks
3.6 Visualizing skip-thoughts and generating stories
4 Conclusion
References

Knowls

Knowl 1 — Skip-Thoughts Framework and Training Objective
model/method
The skip-thoughts model abstracts the word-level skip-gram objective to the sentence level. Given a contiguous sequence of three sentences $(s_{i-1}, s_i, s_{i+1})$ extracted from continuous text, the model uses an encoder to map the center sentence $s_i$ to a fixed-dimensional vector representation $h_i$ . Conditioned on $h_i$ , two separate decoders reconstruct the surrounding context: one generates the previous sentence $s_{i-1}$ and the other generates the next sentence $s_{i+1}$ .

Let $w_i^t$ denote the $t$ -th word in sentence $s_i$ , and let $w_i^{<t} = (w_i^1, \dots, w_i^{t-1})$ denote the prefix of words prior to time step $t$ . The training objective optimizes the sum of the log-probabilities of predicting the tokens in both the preceding and following sentences conditioned on the sentence embedding $h_i$ :

$\sum_t \log P(w_{i+1}^t \mid w_{i+1}^{<t}, h_i) + \sum_t \log P(w_{i-1}^t \mid w_{i-1}^{<t}, h_i)$

This objective encourages the encoder to map sentences with similar semantic and syntactic context into proximate regions in the embedding space without requiring supervised task labels.
Knowl 2 — Recurrent Architecture for Skip-Thought Encoder and Decoder
model/method
The skip-thoughts model employs a Gated Recurrent Unit (GRU) encoder and two conditional GRU decoders.

Encoder

For a sentence $s_i$ containing $N$ words $(w_i^1, \dots, w_i^N)$ with corresponding word embedding vectors $(x_i^1, \dots, x_i^N)$ , the encoder hidden state $h^t$ is updated at each step $t$ according to:

$r^t = \sigma(W_r x^t + U_r h^{t-1})$

$z^t = \sigma(W_z x^t + U_z h^{t-1})$

$\bar{h}^t = \tanh(W x^t + U(r^t \odot h^{t-1}))$

$h^t = (1 - z^t) \odot h^{t-1} + z^t \odot \bar{h}^t$

where $\sigma(\cdot)$ is the logistic sigmoid activation function, $\odot$ denotes component-wise multiplication, $r^t$ is the reset gate, $z^t$ is the update gate, $\bar{h}^t$ is the candidate hidden state, and $W_r, W_z, W, U_r, U_z, U$ are learned weight matrices. The final hidden state $h_i = h^N$ serves as the representation of the entire sentence $s_i$ .

Decoder

The decoder is a neural language model conditioned on the encoder vector $h_i$ . At step $t$ , the decoder hidden state $h_{i+1}^t$ for generating sentence $s_{i+1}$ is computed via:

$r^t = \sigma(W_r^d x^{t-1} + U_r^d h^{t-1} + C_r h_i)$

$z^t = \sigma(W_z^d x^{t-1} + U_z^d h^{t-1} + C_z h_i)$

$\bar{h}^t = \tanh(W^d x^{t-1} + U^d (r^t \odot h^{t-1}) + C h_i)$

$h_{i+1}^t = (1 - z^t) \odot h^{t-1} + z^t \odot \bar{h}^t$

where $C_r, C_z, C$ are projection matrices that bias the reset gate, update gate, and hidden state candidate with the sentence representation $h_i$ . Given hidden state $h_{i+1}^t$ , the unnormalized probability distribution over the vocabulary is given by:

$P(w_{i+1}^t \mid w_{i+1}^{<t}, h_i) \propto \exp(v_{w_{i+1}^t}^\top h_{i+1}^t)$

where $v_{w}$ denotes the row corresponding to word $w$ in the vocabulary output matrix $V$ . An analogous decoder with separate parameters (sharing only the output vocabulary matrix $V$ ) is used to generate the preceding sentence $s_{i-1}$ .
Knowl 3 — Cross-Space Vocabulary Expansion via Linear Regression
model/method
To handle words absent from the encoder's limited training vocabulary $\mathcal{V}_{\text{rnn}}$ , skip-thoughts expands its vocabulary using an external, large pre-trained word embedding space $\mathcal{V}_{\text{w2v}}$ (such as continuous bag-of-words word2vec embeddings).

A linear mapping $f: \mathcal{V}_{\text{w2v}} \to \mathcal{V}_{\text{rnn}}$ parameterized by a matrix $W$ is defined such that for an external word vector $v \in \mathcal{V}_{\text{w2v}}$ , its mapped representation in the RNN word embedding space is:

$v' = W v$

The matrix $W$ is learned by solving an unregularized $L_2$ linear regression objective using all words shared between the pre-trained embedding vocabulary and the RNN training vocabulary:

$\min_W \sum_{w \in \mathcal{V}_{\text{rnn}} \cap \mathcal{V}_{\text{w2v}}} \|v_w^{\text{rnn}} - W v_w^{\text{w2v}}\|_2^2$

After $W$ is fitted, any out-of-vocabulary word appearing in $\mathcal{V}_{\text{w2v}}$ can be mapped directly into the encoder's input space $\mathcal{V}_{\text{rnn}}$ , expanding the usable vocabulary from 20,000 words to over 930,000 words without retraining the recurrent neural network.
Knowl 4 — Model Variants and Training Setup on BookCorpus
experimental setup
Skip-thought models are trained on the BookCorpus dataset, comprising 11,038 books across 16 genres (74,004,228 sentences, 984,846,357 words, 1,316,420 unique words, and an average sentence length of 13 words).

Three encoder configurations are defined:
1. uni-skip: A unidirectional GRU encoder with a hidden dimension of 2400.
2. bi-skip: A bidirectional GRU encoder containing two separate 1200-dimensional GRU encoders (one processing the sentence in forward order and the other in reverse order), whose hidden outputs are concatenated into a 2400-dimensional vector.
3. combine-skip: A 4800-dimensional vector formed post-training by concatenating the representations from uni-skip and bi-skip.
Optimization Hyperparameters
- Vocabulary size during training: 20,000 words (expanded to 930,911 via linear mapping to CBOW word2vec).
- Mini-batch size: 128.
- Parameter initialization: Recurrent weight matrices are initialized with orthogonal initialization; non-recurrent weights are sampled uniformly from $[-0.1, 0.1]$ .
- Gradient clipping: Gradients are clipped if the parameter norm exceeds 10.
- Optimizer: Adam optimizer, training for roughly two weeks per model.
Knowl 5 — Sentence Pair Representation and Continuous Target Discretization
model/method
For downstream tasks involving sentence pairs $(s_1, s_2)$ with sentence vectors $u, v \in \mathbb{R}^d$ , feature representations are formed by concatenating the element-wise (component-wise) product and absolute difference:

$f(u, v) = [u \odot v, |u - v|] \in \mathbb{R}^{2d}$

For continuous score prediction bounded in $[1, K]$ (such as SICK semantic relatedness with $K=5$ ), continuous ground truth scores $y \in [1, K]$ are transformed into a probability distribution $p = [p_1, \dots, p_K]^\top$ over discrete integer classes $i \in \{1, \dots, K\}$ :

$p_i = \begin{cases} y - \lfloor y \rfloor & \text{if } i = \lfloor y \rfloor + 1 \\ \lfloor y \rfloor - y + 1 & \text{if } i = \lfloor y \rfloor \\ 0 & \text{otherwise} \end{cases}$

A multi-class logistic regression model is trained on $f(u, v)$ to predict class probabilities $\hat{p} \in \mathbb{R}^K$ . At inference time, the final scalar relatedness score $\hat{y}$ is computed as the expected value under the integer vector $r = [1, 2, \dots, K]^\top$ :

$\hat{y} = r^\top \hat{p}$

Knowl 6 — Semantic Relatedness Results on SICK Benchmark

empirical result

On the SemEval 2014 SICK semantic relatedness subtask (4500 train, 500 dev, 4927 test pairs), sentence pairs are represented as $[u \odot v, |u - v|]$ and evaluated using linear logistic regression on top of fixed skip-thought features without fine-tuning.

Method	$r$	$\rho$	MSE
Illinois-LH	0.7993	0.7538	0.3692
UNAL-NLP	0.8070	0.7489	0.3550
Meaning Factory	0.8268	0.7721	0.3224
ECNU	0.8414	–	–
Mean vectors	0.7577	0.6738	0.4557
DT-RNN	0.7923	0.7319	0.3822
SDT-RNN	0.7900	0.7304	0.3848
LSTM	0.8528	0.7911	0.2831
Bidirectional LSTM	0.8567	0.7966	0.2736
Dependency Tree-LSTM	0.8676	0.8083	0.2532
uni-skip	0.8477	0.7780	0.2872
bi-skip	0.8405	0.7696	0.2995
combine-skip	0.8584	0.7916	0.2687
combine-skip+COCO	0.8655	0.7995	0.2561

Evaluation metrics are Pearson's correlation coefficient $r$ , Spearman's rank correlation $\rho$ , and mean squared error (MSE). combine-skip outperforms all SemEval 2014 submissions as well as task-trained LSTMs ( $r = 0.8584$ vs. $0.8528$ ), falling behind only Dependency Tree-LSTM. When augmented with image-sentence features learned on MS COCO (combine-skip+COCO), the representation achieves $r = 0.8655$ and $\text{MSE} = 0.2561$ , performing on par with Dependency Tree-LSTM ( $r = 0.8676$ , $\text{MSE} = 0.2532$ ) without requiring syntactic parsers.

Knowl 7 — Paraphrase Detection Results on Microsoft Research Paraphrase Corpus

empirical result

On the Microsoft Research Paraphrase (MSRP) benchmark (4,076 training pairs, 1,725 test pairs), sentence pairs are represented by the feature vector $[u \odot v, |u - v|]$ and classified using $L_2$ -regularized logistic regression.

Method	Accuracy (%)	$F_1$ (%)
feats	73.2	–
RAE+DP	72.6	–
RAE+feats	74.2	–
RAE+DP+feats	76.8	83.6
FHS	75.0	82.7
PE	76.1	82.7
WDDP	75.6	83.0
MTMETRICS	77.4	84.1
uni-skip	73.0	81.9
bi-skip	71.2	81.2
combine-skip	73.0	82.0
combine-skip + feats	75.8	83.0

Without task-specific feature engineering, combine-skip alone attains 73.0% accuracy and 82.0% $F_1$ , outperforming recursive autoencoders with dynamic pooling (RAE+DP, 72.6% accuracy). When combined with basic pairwise statistics (feats), combine-skip + feats achieves 75.8% accuracy and 83.0% $F_1$ , matching heavily engineered paraphrase detection systems.

Knowl 8 — Cross-Modal Image-Sentence Retrieval on MS COCO

empirical result

Cross-modal retrieval on the Microsoft COCO dataset (80,000+ training images, 1,000 dev / 1,000 test images each with 5 captions) evaluates fixed sentence embeddings for image annotation (image query $\to$ rank captions) and image search (caption query $\to$ rank images).

Images are represented by 4096-dimensional OxfordNet features $x$ , and sentences by skip-thought vectors $y$ . Linear projection matrices $U$ and $V$ map images and sentences into a joint 1000-dimensional space. The models are trained for 15 epochs using a pairwise contrastive ranking loss:

$\sum_x \sum_k \max\{0, \alpha - s(Ux, Vy) + s(Ux, Vy_k)\} + \sum_y \sum_k \max\{0, \alpha - s(Vy, Ux) + s(Vy, Ux_k)\}$

where $s(a, b) = \frac{a^\top b}{\|a\| \|b\|}$ is cosine similarity, margin $\alpha = 0.2$ , and $k = 50$ contrastive (incorrect) items per instance.

	Image Annotation				Image Search
Model	R@1	R@5	R@10	Med $r$	R@1	R@5	R@10	Med $r$
Random	0.1	0.6	1.1	631	0.1	0.5	1.0	500
DVSA	38.4	69.6	80.5	1	27.4	60.2	74.8	3
GMM+HGLMM	39.4	67.9	80.9	2	25.1	59.8	76.6	4
m-RNN	41.0	73.0	83.5	2	29.0	42.2	77.0	3
uni-skip	30.6	64.5	79.8	3	22.7	56.4	71.7	4
bi-skip	32.7	67.3	79.6	3	24.2	57.1	73.2	4
combine-skip	33.8	67.7	82.1	3	25.9	60.0	74.6	4

Metrics are Recall@K (R@K, higher is better) and Median Rank (Med $r$ , lower is better). Linear embeddings of combine-skip achieve 82.1% Annotation R@10 and 74.6% Search R@10, which is on par with jointly trained recurrent neural models (DVSA, GMM+HGLMM) without requiring recurrent language model fine-tuning during cross-modal training.

Knowl 9 — Evaluation on Text Classification Benchmarks

empirical result

Fixed skip-thought representations are evaluated across five standard classification benchmarks using linear logistic regression with $L_2$ regularization tuned via cross-validation:

MR: Movie review sentiment (10-fold CV)
CR: Customer product reviews (10-fold CV)
SUBJ: Subjectivity/objectivity classification (10-fold CV)
MPQA: Opinion polarity classification (10-fold CV)
TREC: Question-type classification (predefined train/test split)

Method	MR	CR	SUBJ	MPQA	TREC
NB-SVM	79.4	81.8	93.2	86.3	–
MNB	79.0	80.0	93.6	86.3	–
cBoW	77.2	79.9	91.3	86.4	87.3
GrConv	76.3	81.3	89.5	84.5	88.4
RNN	77.2	82.3	93.7	90.1	90.2
BRNN	82.3	82.6	94.2	90.3	91.0
CNN	81.5	85.0	93.4	89.6	93.6
AdaSent	83.1	86.3	95.5	93.3	92.4
Paragraph-Vector	74.8	78.1	90.5	74.2	91.8
uni-skip	75.5	79.3	92.1	86.9	91.4
bi-skip	73.9	77.9	92.5	83.3	89.4
combine-skip	76.5	80.1	93.6	87.1	92.2
combine-skip + NB	80.4	81.3	93.6	87.5	–

combine-skip consistently outperforms the unsupervised Paragraph-Vector baseline across all datasets (e.g., 76.5 vs. 74.8 on MR, 93.6 vs. 90.5 on SUBJ, 87.1 vs. 74.2 on MPQA). When concatenated with bigram Naive Bayes features (combine-skip + NB), accuracy on MR increases to 80.4%, establishing a competitive linear classification baseline.

Coverage note — Qualitative t-SNE embedding plots and the sample 20-sentence story generated by the decoder were omitted as standalone knowls because they represent illustrative qualitative demonstrations rather than core quantitative results or methodological specifications.

References

1.Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
2.Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
3.Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. ACL, 2014.
4.Yoon Kim. Convolutional neural networks for sentence classification. EMNLP, 2014.
5.Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. SSST-8, 2014.
6.Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. IJCAI, 2015.
7.Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. ICML, 2014.
8.Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. ICLR, 2013.
9.Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Arxiv, 2015.
10.Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In EMNLP, pages 1700–1709, 2013.
11.Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. EMNLP, 2014.
12.Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
13.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
14.Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS Deep Learning Workshop, 2014.
15.Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013.
16.Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR, 2014.
17.Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
18.Alice Lai and Julia Hockenmaier. Illinois-lh: A denotational and distributional approach to semantics. SemEval 2014, 2014.
19.Sergio Jimenez, George Duenas, Julia Baquero, Alexander Gelbukh, Av Juan Dios Bátiz, and Av Mendizábal. Unal-nlp: Combining soft cardinality features for semantic textual similarity, relatedness and entailment. SemEval 2014, 2014.
20.Johannes Bjerva, Johan Bos, Rob van der Goot, and Malvina Nissim. The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity. SemEval 2014, page 642, 2014.
21.Jiang Zhao, Tian Tian Zhu, and Man Lan. Ecnu: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. SemEval 2014, 2014.
22.Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. ACL, 2015.
23.Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional semantics for finding and describing images with sentences. TACL, 2014.
24.Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS, 2011.
25.Andrew Finch, Young-Sook Hwang, and Eiichiro Sumita. Using machine translation evaluation techniques to determine sentence-level semantic equivalence. In IWP, 2005.
26.Dipanjan Das and Noah A Smith. Paraphrase identification as probabilistic quasi-synchronous recognition. In ACL, 2009.
27.Stephen Wan, Mark Dras, Robert Dale, and Cécile Paris. Using dependency-based features to take the “para-farce” out of paraphrase. In Proceedings of the Australasian Language Technology Workshop, 2006.
28.Nitin Madnani, Joel Tetreault, and Martin Chodorow. Re-examining machine translation metrics for paraphrase identification. In NAACL, 2012.
29.Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. SemEval-2014, 2014.
30.Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, 2004.
31.A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
32.Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, 2015.
33.Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). ICLR, 2015.
34.Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. 2014.
35.Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
36.Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In ACL, pages 115–124, 2005.
37.Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177, 2004.
38.Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In ACL, 2004.
39.Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 2005.
40.Xin Li and Dan Roth. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics, 2002.
41.Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, 2012.
42.Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.

Citation

MLA

Kiros, R., et al. “Skip-Thought Vectors”. arXiv, 2015, http://arxiv.org/abs/1506.06726v1.

APA

Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015). Skip-Thought Vectors. arXiv. http://arxiv.org/abs/1506.06726v1

Chicago

Kiros, R., Y. Zhu, R. Salakhutdinov, et al. 2015. “Skip-Thought Vectors”. arXiv. http://arxiv.org/abs/1506.06726v1.

Harvard

Kiros, R. et al. (2015) “Skip-Thought Vectors”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1506.06726v1.

Vancouver

1. Kiros R, Zhu Y, Salakhutdinov R, Zemel RS, Torralba A, Urtasun R, Fidler S (2015) Skip-Thought Vectors. arXiv

BibTeX

@article{kiros2015skip,
  title = {Skip-Thought Vectors},
  author = {Kiros, Ryan and Zhu, Yukun and Salakhutdinov, Ruslan and Zemel, Richard S. and Torralba, Antonio and Urtasun, Raquel and Fidler, Sanja},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1506.06726v1},
  eprint = {1506.06726}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors

Skip-Thought Vectors

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Skip-Thoughts Framework and Training Objective

Knowl 2 — Recurrent Architecture for Skip-Thought Encoder and Decoder

Encoder

Decoder

Knowl 3 — Cross-Space Vocabulary Expansion via Linear Regression

Knowl 4 — Model Variants and Training Setup on BookCorpus

Optimization Hyperparameters

Knowl 5 — Sentence Pair Representation and Continuous Target Discretization

Knowl 6 — Semantic Relatedness Results on SICK Benchmark

Knowl 7 — Paraphrase Detection Results on Microsoft Research Paraphrase Corpus

Knowl 8 — Cross-Modal Image-Sentence Retrieval on MS COCO

Knowl 9 — Evaluation on Text Classification Benchmarks

References

Citation

BibTeX

Source Code

Access the Paper