Towards VQA Models That Can Read

Amanpreet SinghVivek NatarajanMeet ShahYu JiangXinlei ChenDhruv BatraDevi ParikhMarcus Rohrbach

article2019CVPR2,297 citations

Introduces the TextVQA dataset and the LoRRA model to enable visual question answering systems to read and reason about text embedded in everyday images.

Listen

Visual Question Answering models frequently assist users by interpreting scene imagery, yet they routinely fail when required to read and reason about embedded text. This capability is critical for practical assistive technology, where visually impaired users frequently ask text-dependent questions about their daily environment, such as reading appliance settings, signs, or currency. Standard benchmark datasets contain negligible text-focused tasks, while existing assistive datasets remain too small or noisy to train effective models, leaving models unequipped to handle text in everyday scenes.

To address this gap, the article introduces a dedicated benchmark and a specialized model architecture designed to evaluate and improve how artificial intelligence systems read visual text and reason over it to answer questions.

The researchers established TextVQA, a dataset comprising 45,336 questions across 28,408 natural images from Open Images, where answering requires reading embedded text. In parallel, they developed the Look, Read, Reason & Answer (LoRRA) model architecture. LoRRA incorporates an optical character recognition module to read text tokens in an image and utilizes a dynamic copy mechanism. This mechanism allows the model to either deduce an answer from a standard, fixed vocabulary or directly copy extracted text strings to produce out-of-vocabulary answers.

The findings show that leading standard models struggle significantly with text-focused queries, as state-of-the-art models like Pythia achieved only 13.04% validation accuracy on TextVQA. LoRRA substantially improved performance, reaching 26.56% validation accuracy and 27.63% test accuracy on TextVQA, more than doubling baseline results, while also improving single-model accuracy on the broader VQA 2.0 benchmark to 69.21%. Despite these advances, machine performance remains far below human capability, which achieved 85.01% validation accuracy and 86.79% test accuracy, highlighting a 58-percentage-point performance gap.

These results demonstrate that monolithic neural networks cannot organically master complex reading and multimodal reasoning from distant supervision alone; integrating specialized reading modules and dynamic answer mechanisms is essential. The findings imply that assistive tools and visual document systems face severe reliability risks in reading-heavy deployments unless purpose-built reading pipelines are integrated. Current top-performing models remain experimental and are not yet sufficiently accurate for mission-critical or safety-sensitive autonomous assistive applications.

Organizations developing computer vision and assistive applications should integrate optical character recognition modules and dynamic copy mechanisms into their system pipelines rather than relying solely on fixed-vocabulary visual models. Future research and development should focus on enabling systems to copy multi-word phrases (n-grams), improving scene text detection under rotation and occlusion, and expanding external reasoning capabilities before deploying systems in high-stakes environments.

The conclusions are limited by the performance of the underlying optical character recognition system, which struggles with rotated, unstructured, or partially occluded text. Additionally, the model architecture is currently constrained to copying a single text token at a time, preventing it from directly resolving approximately 8.9% of TextVQA questions that require multi-word outputs. Confidence in the relative benchmark improvements is high, but practitioners should exercise caution given the remaining performance gap relative to human accuracy.

Georgia Institute of Technology Meta color images digital image reasoning Answer OCR VQA

arXiv: 1904.08920 facebookresearch/pythia

Paper: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, Yash Goyal et al. (2016). Reading the VQA v2.0 dataset paper is essential because it established the balanced benchmark framework that the TextVQA task builds upon to reduce language bias.
Paper: VQA: Visual Question Answering, Stanislaw Antol et al. (2015). Understanding the foundational Visual Question Answering task definition provides the core evaluation methodology and baseline models assumed by the TextVQA framework.

Paper: LLaVA-OneVision: Easy Visual Task Transfer, Bo Li et al. (2024). This work directly continues the source paper's push for document visual understanding by scaling up unified architectures to achieve state-of-the-art text reading on benchmarks like DocVQA.
Paper: Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, Jinze Bai et al. (2023). Building directly upon TextVQA's focus on optical character recognition in images, this paper extends generalist multimodal models with dedicated text reading and fine-grained localization capabilities.
Paper: Improved Baselines with Visual Instruction Tuning, Haotian Liu et al. (2024). This paper applies improved visual instruction tuning and targeted text-reading datasets to significantly boost the performance of models first designed for tasks like TextVQA.

Cover for Towards VQA Models That Can Read

Abstract

Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today's VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new "TextVQA" dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA). We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0.

Topics

color images digital image reasoning reference image visually-impaired users

Keywords

Answer OCR VQA Text OCR Tokens Model Question Questions Image Models

1 Introduction
2 Related work
3 LoRRA: Look, Read, Reason & Answer
3.1 VQA Component
3.2 Reading Component
3.3 Answer Module
3.4 Implementation Details
4
4.1 Images
4.2 Questions and Answers
4.3 Statistics and Analysis
5 Experiments
6 Conclusion
References
A OCR and Answer Space Analysis
B TextVQA Examples and LoRRA Predictions
C Interface Screenshots

Knowls

Knowl 1 — TextVQA Dataset Specification and Collection Pipeline
definition
The TextVQA dataset is designed to benchmark visual question answering models on questions that require reading and reasoning about text present in natural images. It consists of 45,336 questions asked by human annotators on 28,408 images sourced from the Open Images v3 dataset, with each question-image pair annotated with 10 ground-truth answers from distinct human annotators.

Image Selection and Curation

Images are sampled from Open Images v3 categories that frequently contain text (e.g., "billboard", "traffic sign", "whiteboard"). Category sampling weights are determined by running the Rosetta Optical Character Recognition (OCR) system on a 100-image sample per category and computing the normalized average number of detected OCR bounding boxes. Images confirmed by annotators to contain readable text are retained.

Dataset Splits

TextVQA is partitioned into three disjoint splits without image overlap:
- Train split: 34,602 questions on images from the Open Images training set.
- Validation split: 5,000 questions on images from the Open Images training set.
- Test split: 5,734 questions on images from the Open Images test set.
Key Statistics and Linguistic Diversity
- Unique questions: 37,912 unique questions out of 45,336 total (83.6%), with an average question length of 7.18 words (minimum 3 words).
- Unique majority answers: 26,263 unique answers out of 53,354 majority answers (49.2%), with an average answer length of 1.58 words.
- Yes/No frequency: The single most frequent majority answer ("yes") accounts for only 4.71% of the dataset, and all yes/no questions combined constitute 5.55% of the dataset.
- Annotator agreement: All 10 annotators agree on the most frequent answer for 22.8% of questions, while $\ge 3$ annotators agree for 97.9% of questions.
- OCR token presence: Images contain an average of 3.14 OCR tokens, and approximately 81% of validation images contain $\ge 2$ OCR tokens.
Knowl 2 — Look, Read, Reason & Answer (LoRRA) Model Architecture
model/method
The Look, Read, Reason & Answer (LoRRA) architecture extends traditional visual question answering (VQA) models by integrating an explicit Optical Character Recognition (OCR) reading component and a dynamic pointer-based copying mechanism alongside visual and linguistic reasoning modules.

Core Components
1. Question Encoder ( $f_Q$ ): Given question words $q = (w_1, w_2, \dots, w_L)$ , words are embedded using pre-trained GloVe embeddings (vocabulary size $\approx 77\text{k}$ ) and processed through a Long Short-Term Memory (LSTM) network with self-attention to generate a question representation $f_Q(q)$ .
2. Visual Feature Pipeline ( $f_I$ ): Image visual features $f_I(v)$ are formed by extracting:
  - Grid-based features: 2048-dimensional features average-pooled from the res-5c block of a pre-trained ResNet-152 network.
  - Region-based features: Bounding box proposal features from the fc6 layer of a Faster R-CNN model trained on Visual Genome objects and attributes, fine-tuning the fc7 weights during training. Spatial attention $f_A(f_I(v), f_Q(q))$ produces a question-guided weighted average of the visual features, which is concatenated and fused with $f_Q(q)$ using an element-wise (Hadamard) product $\otimes$ via $f_{\text{comb}}$ to form the VQA feature vector $f_{VQA}(v, q)$ .
3. Reading Pipeline ( $f_O$ ): An external OCR model (Rosetta) extracts $M$ word tokens $s = (s_1, s_2, \dots, s_M)$ from the image. Each token is mapped to an embedding $f_O(s_i)$ using pre-trained FastText embeddings (which handle out-of-vocabulary tokens). Top-down attention $f_A(f_O(s), f_Q(q))$ computes attention weights across the $M$ tokens. To prevent loss of spatial and reading order, the raw attention weights are concatenated to the weighted-average token embedding. The result is combined with $f_Q(q)$ via Hadamard product $\otimes$ through $f_{\text{comb}}$ to produce the OCR-question feature vector $f_{OCR}(s, q)$ .
4. Answer Module and Copy Mechanism ( $f_{\text{MLP}}$ ): The representation $[f_{VQA}(v, q); f_{OCR}(s, q)]$ is fed into a two-layer feed-forward multi-layer perceptron (MLP) to output unnormalized prediction logits over an augmented answer space of size $N + M$ , where $N$ corresponds to a fixed answer vocabulary and $M$ corresponds to the dynamic slots for the image's OCR tokens.
Knowl 3 — LoRRA Feature Fusion and Dynamic Answer Logit Formulation
equation
The mathematical formulation of feature extraction, cross-modal attention fusion, and dynamic logit prediction in the Look, Read, Reason & Answer (LoRRA) model is defined as follows:
$f_{VQA}(v, q) = f_{\text{comb}}(f_A(f_I(v), f_Q(q)), f_Q(q))$ $f_{OCR}(s, q) = f_{\text{comb}}(f_A(f_O(s), f_Q(q)), f_Q(q))$ $f_{\text{LoRRA}}(v, s, q) = f_{\text{MLP}}\big([f_{VQA}(v, q);\, f_{OCR}(s, q)]\big)$
Variable and Operator Definitions
- $v$ : The input image.
- $q = (w_1, w_2, \dots, w_L)$ : The question composed of $L$ words.
- $s = (s_1, s_2, \dots, s_M)$ : The sequence of $M$ OCR tokens detected in image $v$ .
- $f_I(v)$ : Image visual representations combining grid-level convolutional features and region proposal features.
- $f_Q(q)$ : Question embedding generated via GloVe word embeddings, an LSTM, and self-attention.
- $f_O(s)$ : FastText word embeddings of the OCR tokens $s$ .
- $f_A(\cdot, f_Q(q))$ : Top-down question-guided attention mechanism computing normalized attention weights over spatial visual regions or OCR tokens, outputting weighted-average feature vectors (for OCR tokens, attention weights are concatenated with the weighted feature average to preserve token order).
- $f_{\text{comb}}(x, y) = x \otimes y$ : Multimodal combination operator using element-wise (Hadamard) product $\otimes$ .
- $[\cdot ; \cdot]$ : Vector concatenation operator along feature dimensions.
- $f_{\text{MLP}}$ : A two-layer feed-forward network with parameters $\Theta$ .
- $f_{\text{LoRRA}}(v, s, q) \in \mathbb{R}^{N + M}$ : Vector of output logits over an answer space containing $N$ fixed vocabulary tokens $(a_1, \dots, a_N)$ and $M$ dynamic OCR candidate tokens $(s_1, \dots, s_M)$ .
Prediction and Loss Function

The predicted answer index $i^*$ is chosen by $i^* = \arg\max_{1 \le i \le N+M} [f_{\text{LoRRA}}(v, s, q)]_i$ . If $i^* > N$ , the model copies the OCR token $s_{i^* - N}$ as the answer. Training optimizes the binary cross-entropy loss over independent sigmoid probabilities derived from the logits, allowing the ground-truth answer to match entries in both the fixed vocabulary and the dynamic OCR token set without penalty.

Knowl 4 — TextVQA Benchmark Evaluation and Ablation Results

data/table

The performance of heuristic baselines, upper bounds, ablations, and the full Look, Read, Reason & Answer (LoRRA) model on the TextVQA dataset is detailed below. Accuracy is measured using the standard VQA soft-accuracy metric over 10 ground-truth answers.

Model	Vocabulary	Val Accuracy (%)	Test Accuracy (%)
Human	–	85.01	86.79
OCR UB (up to 4-grams)	–	37.12	36.52
LA UB (top 8000 answers)	LA	48.46	48.16
LA + OCR UB	LA	67.56	68.24
Random 100	–	0.22	0.20
Weighted Random 100	–	0.27	0.26
Majority Answer (“yes”)	–	4.48	2.63
Random OCR Token	–	7.72	9.12
OCR Max (most frequent OCR token)	–	9.76	11.60
Question Only ( $Q$ )	LA	8.09	8.70
Image Only ( $I$ )	LA	6.29	5.58
Pythia ( $I+Q$ )	LA	13.04	14.00
Pythia + OCR ( $I+Q+O$ )	LA	18.35	–
Pythia + OCR + Copy ( $I+Q+O+C$ )	n/a	20.06	–
BAN ( $I+Q$ )	LA	12.30	–
BAN + LoRRA	SA	18.41	–
Pythia + LoRRA	LA	26.23	–
Pythia + LoRRA (Ours)	SA	26.56	27.63

Key Takeaways

Standard VQA Model Inability to Read: State-of-the-art VQA models without OCR components (Pythia $I+Q$ at 13.04% and Bilinear Attention Networks (BAN) $I+Q$ at 12.30%) perform only marginally better than Question-Only or OCR-heuristic baselines, falling far short of human performance (85.01% val).
Impact of OCR Features and Copy Mechanism: Adding OCR features to Pythia ( $+O$ ) raises validation accuracy from 13.04% to 18.35%. Predicting solely from OCR tokens via the copy mechanism ( $+O+C$ ) achieves 20.06%.
Vocabulary Trade-off (SA vs. LA): Using a smaller fixed vocabulary (SA, size 3,996; answers appearing $\ge 2$ times in training) allows LoRRA to predict from the dynamic OCR token space more frequently, improving validation accuracy from 26.23% (with large vocabulary LA of 8,000 words) to 26.56%.

Knowl 5 — LoRRA Generalization on Standard VQA 2.0 and VizWiz Benchmarks

empirical result

Integrating Look, Read, Reason & Answer (LoRRA) into standard VQA models preserves and improves performance on general VQA benchmarks that are not exclusively focused on scene text.

VQA 2.0 Benchmark Model	test-dev Accuracy (%)
BUTD	65.32
Counter	68.09
BAN	69.08
Pythia v0.1	68.49
Pythia v0.3 (Baseline)	68.71
Pythia v0.3 + LoRRA	69.21
VizWiz Benchmark Model	test Accuracy (%)
BAN	51.40
Pythia v0.3	54.72

On the VQA 2.0 dataset, single-model Pythia v0.3 achieves 68.71% accuracy on test-dev. Adding LoRRA increases the accuracy to 69.21% (+0.50%), establishing state-of-the-art single-model performance and demonstrating that the model can leverage reading and copying abilities in broader multimodal contexts without harming general reasoning.

Knowl 6 — LoRRA Training and Implementation Hyperparameters
experimental setup
LoRRA is implemented in PyTorch and trained end-to-end (except for fixed pre-trained visual backbones and offline OCR extraction) under the following protocol:
- Optimization: AdaMax optimizer with batch size 128 distributed across 8 GPUs for 24,000 total iterations.
- Learning Rate Schedule: Initial learning rate of $5 \times 10^{-2}$ for all core layers and $5 \times 10^{-3}$ for fine-tuning the Faster R-CNN fc7 layers. The learning rate is uniformly decayed by a factor of 10 to $5 \times 10^{-4}$ after 14,000 iterations.
- Input Dimensions: Maximum question length is padded or truncated to $L = 14$ tokens; maximum number of OCR tokens is padded or truncated to $M = 50$ tokens per image.
- Visual Features: ResNet-152 res-5c grid features pooled to 2048 dimensions combined with Faster R-CNN object proposal features from Visual Genome.
- Embeddings: GloVe embeddings for question words (top $\approx 77\text{k}$ words from VQA 2.0); FastText embeddings for OCR tokens.
- Vocabularies: Small Answer vocabulary (SA) contains 3,996 answers (occurring $\ge 2$ times in training); Large Answer vocabulary (LA) contains the 8,000 most frequent training answers.
- Model Selection: Validation accuracy evaluated every 1,000 iterations using the standard VQA soft-accuracy metric; the checkpoint with the highest validation score is selected for test evaluation. Validation accuracies are averaged over 5 random seeds.
Knowl 7 — LoRRA Prediction Distribution and OCR Copying Behavior
empirical result
Analysis of Look, Read, Reason & Answer (LoRRA) predictions on the TextVQA validation set reveals the empirical distribution between the dynamic OCR copy mechanism and the fixed vocabulary:
- Prediction Source Ratio: LoRRA predicts an answer by copying an OCR token in 44.9% of cases, and predicts from the fixed Short Answer (SA) vocabulary in 55.1% of cases.
- Copying Accuracy: When the model predicts from OCR tokens, it produces the exact ground-truth answer string 27% of the time and a partially correct answer (matching at least one constituent word) 11% of the time.
- Vocabulary Accuracy: When predicting from the SA fixed vocabulary, it is exactly correct 22.4% of the time.
- Target Location Alignment:
  - For the 30.6% of questions whose ground-truth answers are present in the detected OCR tokens, LoRRA chooses to predict from OCR tokens 68% of the time and achieves 57.5% accuracy on those predictions.
  - For the 48.0% of questions whose ground-truth answers exist in the SA vocabulary, LoRRA predicts from the fixed vocabulary 66.75% of the time and achieves 38% accuracy on those predictions.
- Multi-Token Presence: Among the 4,645 validation questions where images contain $\ge 2$ OCR tokens (81% of the validation split), LoRRA copies an OCR token 49.7% of the time and achieves 24.3% accuracy, demonstrating learned context-guided selection rather than random token copying.
Knowl 8 — Limitations of LoRRA and Scene Text VQA Systems
limitation
The LoRRA architecture and current scene-text visual question answering pipelines are subject to two primary limitations:
1. Single-Token Copy Constraint: LoRRA's copy mechanism is structurally restricted to pointing to and outputting a single OCR token $s_i$ . It cannot generate multi-token phrases or compose strings by copying multiple OCR tokens in sequence. In the TextVQA dataset, 8.9% of questions require answers formed by concatenating multiple OCR tokens (e.g., compound names like "fly emirates" or dates like "may 19").
2. Sensitivity to Upstream OCR Quality: LoRRA relies on an offline, non-jointly-trained OCR detector (Rosetta). If the OCR module fails to detect or correctly recognize text—such as rotated text, handwritten scribbles, complex typography, or partially occluded characters—the model cannot recover the correct text answer through the copy mechanism. On TextVQA, the theoretical upper bound of correctly answering questions exclusively from Rosetta OCR tokens (up to 4-grams) is 37.12% on the validation set, imposing an upper limit on reading-dependent accuracy.

Coverage note — None was omitted; all primary contributions—including the TextVQA dataset specification, LoRRA architecture, mathematical formulation, TextVQA and VQA 2.0 empirical evaluations, analysis of the copy mechanism, and stated limitations—are fully captured.

References

1.Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2018.
2.Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Computer Vision and Pattern Recognition (CVPR), 2016.
3.Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
4.Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
5.Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342. ACM, 2010.
6.Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar. Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 71–79. ACM, 2018.
7.Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP, 2016.
8.Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. Detectron. https://github.com/facebookresearch/detectron, 2018.
9.Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: training imagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017.
10.Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
11.J Gu, Z Lu, H Li, and VOK Li. Incorporating copying mechanism in sequence-to-sequence learning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2016. Association for Computational Linguistics., 2016.
12.Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. Pointing the unknown words. In ACL, 2016.
13.Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Conference on Computer Vision and PatternRecognition (CVPR), 2017.
14.Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
15.Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
16.Lu Jiang, Junwei Liang, Liangliang Cao, Yannis Kalantidis, Sachin Farfade, and Alexander G Hauptmann. Memexqa: Visual memex question answering. arXiv:1708.01336, 2017.
17.Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0. 1: the winning entry to the vqa challenge 2018. arXiv preprint arXiv:1807.09956, 2018.
18.Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 1988–1997. IEEE, 2017.
19.Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In European Chapter of the Association for Computational Linguistics, 2017.
20.Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5648–5656, 2018.
21.Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. In ICLR workshop track, 2018.
22.Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In Document Analysis and Recognition (ICDAR), 2015 13th International Conference on, pages 1156–1160. IEEE, 2015.
23.Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision, pages 235–251. Springer, 2016.
24.Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In Computer Vision and Pattern Recognition (CVPR), volume 2, page 3, 2017.
25.Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Neural Information Processing Systems, 2018.
26.Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
27.Ivan Krasin, Tom Duerig, Neil Alldrin, Andreas Veit, Sami Abu-El-Haija, Serge Belongie, David Cai, Zheyun Feng, Vittorio Ferrari, Victor Gomes, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github.com/openimages, 2(6):7, 2016.
28.Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
29.Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541–551, 1989.
30.Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pages 289–297, 2016.
31.Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in neural information processing systems, pages 1682–1690, 2014.
32.Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), 2017.
33.Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC-British Machine Vision Conference. BMVA, 2012.
34.Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. In The SIGNLL Conference on Computational Natural Language Learning (CoNLL), 2016.
35.Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. NIPS AutoDiff Workshop, 2017.
36.Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
37.Dinesh Raghu, Nikhil Gupta, et al. Hierarchical pointer memory network for task oriented dialogue. arXiv preprint arXiv:1805.01216, 2018.
38.Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In Advances in neural information processing systems, pages 2953–2961, 2015.
39.Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Association for Computational Linguistics, 2017.
40.Amanpreet Singh, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia-a platform for vision & language research. SysML Workshop, NeurIPS 2019, 2018.
41.Ray Smith. An overview of the tesseract ocr engine. In Document Analysis and Recognition, 2007. ICDAR 2007. Ninth International Conference on, volume 2, pages 629–633. IEEE, 2007.
42.Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 217–223, 2017.
43.Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016.
44.Kai Wang and Serge Belongie. Word spotting in the wild. In European Conference on Computer Vision, pages 591–604. Springer, 2010.
45.Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 2018.
46.Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. In International Conference on Learning Representations (ICLR), 2016.
47.Huijuan Xu and Kate Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In European Conference on Computer Vision, pages 451–466. Springer, 2016.
48.Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In CVPR, 2016.
49.Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE Transactions on Neural Networks and Learning Systems, 2018.
50.Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Learning to count objects in natural images for visual question answering. In International Conference on Learning Representations (ICLR), 2018.
51.Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4995–5004, 2016.

Citation

MLA

Singh, A., et al. “Towards VQA Models That Can Read”. arXiv, 2019, http://arxiv.org/abs/1904.08920v2.

APA

Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019). Towards VQA Models That Can Read. arXiv. http://arxiv.org/abs/1904.08920v2

Chicago

Singh, A., V. Natarajan, M. Shah, et al. 2019. “Towards VQA Models That Can Read”. arXiv. http://arxiv.org/abs/1904.08920v2.

Harvard

Singh, A. et al. (2019) “Towards VQA Models That Can Read”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1904.08920v2.

Vancouver

1. Singh A, Natarajan V, Shah M, Jiang Y, Chen X, Batra D, Parikh D, Rohrbach M (2019) Towards VQA Models That Can Read. arXiv

BibTeX

@article{singh2019towards,
  title = {Towards VQA Models That Can Read},
  author = {Singh, Amanpreet and Natarajan, Vivek and Shah, Meet and Jiang, Yu and Chen, Xinlei and Batra, Dhruv and Parikh, Devi and Rohrbach, Marcus},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1904.08920v2},
  eprint = {1904.08920}
}

Metadata:arXiv

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: IEEE

Towards VQA Models That Can Read

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — TextVQA Dataset Specification and Collection Pipeline

Image Selection and Curation

Dataset Splits

Key Statistics and Linguistic Diversity

Knowl 2 — Look, Read, Reason & Answer (LoRRA) Model Architecture

Core Components

Knowl 3 — LoRRA Feature Fusion and Dynamic Answer Logit Formulation

Variable and Operator Definitions

Prediction and Loss Function