Semantic Parsing on Freebase from Question-Answer Pairs

Jonathan BerantAndrew ChouRoy FrostigPercy Liang

article2013EMNLP2,266 citations

Proposes a scalable semantic parser that learns to query large-scale knowledge bases like Freebase using only question-answer pairs by combining text-to-predicate alignment with a novel bridging operation over neighboring predicates.

Listen

Building automated systems that answer natural language questions by querying massive databases is essential for modern search and information retrieval. Traditional semantic parsing methods translate questions into executable computer queries but rely on expensive, expert-annotated formal logic and are restricted to narrow domains. Scaling these systems to large-scale knowledge repositories like Freebase, which contains tens of millions of entities and thousands of properties, creates a major bottleneck because the space of potential query interpretations expands exponentially.

The article demonstrates that a semantic parser can successfully scale to large-scale databases by training exclusively on question-answer pairs rather than expensive logical annotations. It aims to establish that combining text alignment with automated logical bridging allows systems to resolve complex natural language queries accurately across open domains.

To achieve this, the authors developed a system named SEMPRE. The approach generates candidate query operations using two mechanisms: a coarse mapping built by aligning text patterns from 15 million web assertions to database properties, and a novel bridging technique that infers missing relations based on compatibility with adjacent elements. The system filters candidate queries using a statistical model powered by syntactic cues and answer characteristics. The framework was evaluated on the established FREE917 benchmark of 917 questions and on WEBQUESTIONS, a new and realistic open-domain dataset of 5,810 web-mined questions collected via crowdsourcing.

The findings show that training without expert logic annotations can match or surpass supervised methods. On the FREE917 benchmark, the proposed system achieved 62% accuracy, outperforming the previous state of the art at 59% despite having no access to annotated logical structures during training. On the challenging WEBQUESTIONS dataset, the system achieved 31.4% accuracy, representing an absolute improvement of 4.5 percentage points over the natural baseline of 26.9%. Feature analysis confirmed that combining text-based lexicon alignment with bridging was crucial, as bridging captured rare and implicit relations that text alignment alone missed, while denotation and part-of-speech features substantially reduced parsing errors.

These results indicate that automated question-answering systems can be developed and scaled at significantly lower costs by crowdsourcing simple question-answer pairs instead of employing scarce linguistic experts. By eliminating rigid formal rule construction, organizations can deploy broader search capabilities over complex business and public databases with reduced annotation overhead.

To build on this framework, future engineering efforts should focus on improving entity disambiguation across large inventories and handling complex, multi-clause linguistic structures. Incorporating methods that automatically learn composite database predicates from data is recommended to capture more nuanced language. Decision-makers should maintain moderate confidence in these findings, noting that while performance on curated benchmarks is strong, accuracy on noisy, open-ended web questions remains around 31.4% due to entity recognition ambiguity and partial crowdsourced answers.

Stanford University knowledge base parsing algorithm syntactics semantic parsing question-answer pairs compositional semantics

percyliang/sempre Berant et al (2013).pdf

Paper: Distant supervision for relation extraction without labeled data, Mike D. Mintz et al. (2009). Reading this distant supervision work first clarifies how large knowledge bases like Freebase can be leveraged automatically to extract training data for semantic parsers.
Paper: Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars, Luke S. Zettlemoyer et al. (2005). This foundational paper on mapping sentences to logical forms without manual tree annotations provides the primary semantic parsing methodology built upon by the source.
Paper: YAGO: A Core of Semantic Knowledge Unifying WordNet and Wikipedia, Fabian M. Suchanek et al. (2007). Understanding the structure and integration of WordNet and Wikipedia in YAGO provides vital background on the core semantic knowledge bases utilized in large-scale parsing.

Paper: TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, Mandar Joshi et al. (2017). This benchmark extends open-domain question-answering research by introducing distantly supervised challenge datasets built from natural questions and web search results.
Paper: Reading Wikipedia to Answer Open-Domain Questions, Danqi Chen et al. (2017). This work builds directly upon open-domain question answering techniques by presenting DrQA for reading Wikipedia to answer factual queries.
Paper: Dense Passage Retrieval for Open-Domain Question Answering, Vladimir Karpukhin et al. (2020). This paper advances open-domain question answering by replacing sparse retrieval with dense passage retrieval methods trained on question-answer pairs.

Cover for Semantic Parsing on Freebase from Question-Answer Pairs

Abstract

In this paper, we train a semantic parser that scales up to Freebase. Instead of relying on annotated logical forms, which is especially expensive to obtain at large scale, we learn from question-answer pairs. The main challenge in this setting is narrowing down the huge number of possible logical predicates for a given question. We tackle this problem in two ways: First, we build a coarse mapping from phrases to predicates using a knowledge base and a large text corpus. Second, we use a bridging operation to generate additional predicates based on neighboring predicates. On the dataset of Cai and Yates (2013), despite not having annotated logical forms, our system outperforms their state-of-the-art parser. Additionally, we collected a more realistic and challenging dataset of question-answer pairs and improves over a natural baseline.

Topics

knowledge base parsing algorithm syntactics

Keywords

semantic parsing question-answer pairs compositional semantics logical forms relation extraction entity lexicon alignment lexicon open information extraction breadth-first search

1 Introduction
2 Setup
2.1 Knowledge base
2.2 Logical forms
2.3 Framework
3 Approach
3.1 Alignment
3.2 Bridging
3.3 Composition
4 Experiments
4.1 Main results
4.1.1 FREE917
4.1.2 WEBQUESTIONS
4.2 Detailed analysis
5 Discussion
Acknowledgments
References

Knowls

Knowl 1 — Simple Lambda-DCS Formalism for Knowledge Base Semantic Parsing
definition
Simple $\lambda$ -DCS is a variable-free logical formalism for querying a knowledge base directed graph $\mathcal{K} = \{(e_1, p, e_2) \in \mathcal{E} \times \mathcal{P} \times \mathcal{E}\}$ , where $\mathcal{E}$ is a set of entities and $\mathcal{P}$ is a set of binary properties. Every logical form $z$ in simple $\lambda$ -DCS evaluates recursively to a denotation $\llbracket z \rrbracket_{\mathcal{K}}$ (either a subset of $\mathcal{E}$ for unaries or a subset of $\mathcal{E} \times \mathcal{E}$ for binaries):
- Unary base case: An entity $e \in \mathcal{E}$ denotes a singleton subset: $\llbracket e \rrbracket_{\mathcal{K}} = \{e\}$
- Binary base case: A property $p \in \mathcal{P}$ denotes a binary relation: $\llbracket p \rrbracket_{\mathcal{K}} = \{(e_1, e_2) \in \mathcal{E} \times \mathcal{E} : (e_1, p, e_2) \in \mathcal{K}\}$
- Join: For a binary $b$ and a unary $u$ , the expression $b.u$ denotes a join followed by existential projection: $\llbracket b.u \rrbracket_{\mathcal{K}} = \{e_1 \in \mathcal{E} : \exists e_2 \in \mathcal{E}.\, (e_1, e_2) \in \llbracket b \rrbracket_{\mathcal{K}} \wedge e_2 \in \llbracket u \rrbracket_{\mathcal{K}}\}$
- Intersection: For unaries $u_1$ and $u_2$ , the intersection $u_1 \sqcap u_2$ denotes set intersection: $\llbracket u_1 \sqcap u_2 \rrbracket_{\mathcal{K}} = \llbracket u_1 \rrbracket_{\mathcal{K}} \cap \llbracket u_2 \rrbracket_{\mathcal{K}}$
- Aggregation (Cardinality): For a unary $u$ , the aggregation $\text{count}(u)$ computes set cardinality: $\llbracket \text{count}(u) \rrbracket_{\mathcal{K}} = \{|\llbracket u \rrbracket_{\mathcal{K}}|\}$
Simple $\lambda$ -DCS eliminates explicit lambda variables and existential quantifiers for composition operations, rendering logical forms as tree-like graph patterns over the knowledge base.
Knowl 2 — Discriminative Log-Linear Derivation Model and Marginal Likelihood Objective
model/method
Given an utterance $x$ and a knowledge base $\mathcal{K}$ , a semantic parser constructs a set of candidate derivations $D(x)$ , where each derivation $d \in D(x)$ is a derivation tree culminating in a root logical form $d.z$ .

The probability distribution over derivations $d \in D(x)$ is defined by a log-linear model: $p_\theta(d \mid x) = \frac{\exp\{\phi(x, d)^\top \theta\}}{\sum_{d' \in D(x)} \exp\{\phi(x, d')^\top \theta\}}$ where $\phi(x, d) \in \mathbb{R}^b$ is a feature vector extracted from the question and derivation, and $\theta \in \mathbb{R}^b$ is the parameter vector.

When training from question-answer pairs $\{(x_i, y_i)\}_{i=1}^n$ without annotated logical forms, parameters are learned by maximizing the marginal log-likelihood of generating the correct denotation $y_i$ ( $y_i = \llbracket d.z \rrbracket_{\mathcal{K}}$ ): $\mathcal{O}(\theta) = \sum_{i=1}^n \log \sum_{d \in D(x_i): \llbracket d.z \rrbracket_{\mathcal{K}} = y_i} p_\theta(d \mid x_i)$

Because $D(x)$ is combinatorially large, $D(x_i)$ is approximated during training by a beam search candidate set $\tilde{D}(x_i; \tilde{\theta})$ generated under current model parameters $\tilde{\theta}$ . Optimization is performed via stochastic gradient ascent using AdaGrad per-feature adaptive step sizes: $\theta_{t+1} = \theta_t + \eta_t \left. \frac{\partial \mathcal{O}(\theta; \theta_t)}{\partial \theta} \right|_{\theta = \theta_t}$ initializing $\theta_0 = \mathbf{0}$ .
Knowl 3 — Lexicon Construction via Distant Supervision Alignment with Open IE Triples
model/method
To map natural language phrases to knowledge base predicates over large schemas (such as Freebase with over 19,000 properties), a lexicon $\mathcal{L}$ mapping text phrases $w$ to candidate predicates $z$ and associated alignment features is built by aligning an Open IE text corpus with the knowledge base $\mathcal{K}$ :
1. Typed Phrases: From 15 million ReVerb Open IE triples $(e_1, r, e_2)$ extracted from ClueWeb09 and linked to Freebase entities, text relations $r$ are lemmatized and annotated with Freebase argument types $[t_1, t_2]$ (e.g., "born in" $[\text{Person}, \text{Location}]$ vs. "born in" $[\text{Person}, \text{Date}]$ ) to produce typed binary phrases $\mathcal{R}_1$ . Unary text phrases are extracted using Hearst patterns of the form (is|was a|the) x IN.
2. Extension Generation: For each text phrase $r_1 \in \mathcal{R}_1$ , its extension $\mathcal{F}(r_1)$ is the set of entity pairs $(e_1, e_2)$ co-occurring with $r_1$ in the corpus matching type signature $[t_1, t_2]$ . For each knowledge base predicate $r_2 \in \mathcal{R}_2$ (including atomic properties, composite event chains $p_1.p_2$ , and unaries $\text{Type}.t$ / $\text{Profession}.t$ ), its extension $\mathcal{F}(r_2)$ is its denotation $\llbracket r_2 \rrbracket_{\mathcal{K}}$ .
3. Bipartite Overlap Matching: A bipartite graph is formed with edges between phrase $r_1$ and predicate $r_2$ if their type signatures match and their extensions share at least one entity pair: $\mathcal{F}(r_1) \cap \mathcal{F}(r_2) \neq \emptyset$
Rather than applying hard filtering thresholds on noisy overlap, all candidate edges are retained in the lexicon along with unlexicalized alignment features (co-occurrence counts and overlap ratios) for downstream ranking by the log-linear model.
Knowl 4 — Bridging Operation for Implicit, Light-Verb, and Rare Predicates
model/method
The bridging operation generates binary predicates dynamically based on type compatibility between neighboring logical forms in a derivation tree rather than explicit lexical alignment. It addresses predicates that are unmentioned, expressed via light verbs (e.g., "have") or prepositions, or absent from the alignment lexicon due to lexical rarity:
1. Two-Unary Bridging: Given two unary logical forms $z_1$ (of type $t_1$ ) and $z_2$ (of type $t_2$ ), bridging generates candidate logical forms: $z_1 \sqcap b.z_2$ for all binary predicates $b \in \mathcal{P}$ having type signature $(t_1, t_2)$ . For example, combining $z_1 = \text{Type.FormOfGovernment}$ and $z_2 = \text{Chile}$ generates $\text{Type.FormOfGovernment} \sqcap \text{GovernmentTypeOf.Chile}$ .
2. Single-Unary Injection: Given a single unary $z$ of type $t$ , bridging generates candidate logical forms: $b.z$ for any binary predicate $b \in \mathcal{P}$ with type signature $(*, t)$ . For example, for $z = \text{X-Men}$ , bridging generates $\text{ComicBookCoverPriceOf.X-Men}$ .
3. Intermediate Event Modification (Neo-Davidsonian Bridging): Given a composite logical form $p_1.p_2.z'$ representing an intermediate event variable where $p_2$ has type signature $(t_1, *)$ , and a unary modifier $z$ of type $t$ , bridging injects $z$ directly into the intermediate event: $p_1.(p_2.z' \sqcap b.z)$ for every binary predicate $b \in \mathcal{P}$ with type signature $(t_1, t)$ . For example, combining $z_1 = \text{Marriage.Spouse.TomCruise}$ and $z_2 = 2006$ produces $\text{Marriage.(Spouse.TomCruise} \sqcap \text{StartDate.2006)}$ .
Bridged predicates are discriminated by features including predicate co-occurrence counts, bridging type indicators, and string similarity between the predicate's schema name and words in the question.
Knowl 5 — Feature Representations for Scoring Semantic Derivations
model/method
In the log-linear semantic parsing framework, derivation candidates $d$ for utterance $x$ are scored by extracting features across five categories:
1. Alignment Features (Unlexicalized):
  - $\log |\mathcal{F}(r_1)|$ : Log count of entity pairs co-occurring with phrase $r_1$ .
  - $\log |\mathcal{F}(r_2)|$ : Log count of entity pairs belonging to predicate $r_2$ .
  - $\log |\mathcal{F}(r_1) \cap \mathcal{F}(r_2)|$ : Log count of overlapping entity pairs.
  - Indicator of whether predicate $r_2 = \arg\max_r |\mathcal{F}(r_1) \cap \mathcal{F}(r)|$ (best match for phrase $r_1$ ).
2. Lexicalized Features:
  - Conjunction indicators of the text phrase $w$ and the assigned logical predicate $z$ .
3. Text Similarity Features:
  - Exact match, prefix, and suffix indicators comparing the lemmatized phrase $r_1$ to the Freebase schema name $s_2$ of predicate $r_2$ .
  - Token overlap metrics between $r_1$ and $s_2$ .
4. Bridging Features:
  - Log predicate entity-pair count $\log |\mathcal{F}(b)|$ for the bridged binary predicate $b$ .
  - Categorical indicators for the bridging category (two unaries, single unary, or event modification).
  - Indicator of the specific binary predicate $b$ when applied via injection.
  - Text similarity features between the Freebase name of $b$ and content words in question $x$ .
5. Composition Features:
  - Indicator features on the exact counts of intersection, join, and bridging operations within $d$ .
  - Part-of-Speech (POS) tag features indicating when a word of a specific POS tag is skipped.
  - POS tag conjunctions of the head words of sub-derivations combined via join or bridging.
  - Denotation size indicator features: categorical indicators for whether $|\llbracket d.z \rrbracket_{\mathcal{K}}|$ equals $0, 1, 2,$ or $\ge 3$ .
Knowl 6 — Construction and Structure of the WebQuestions Dataset
experimental setup
The WEBQUESTIONS benchmark is a question-answering dataset consisting of 5,810 question-answer pairs designed to evaluate open-domain semantic parsing over Freebase without relying on expert-annotated logical forms.
- Question Generation: Seeded with the question "Where was Barack Obama born?", a breadth-first search was performed over Google Suggest queries. Nodes represented questions, and edges were queried by stripping the entity, the phrase before the entity, or the phrase after the entity (each returning 5 candidate completions). 1,000,000 questions containing a $wh$ -word and exactly one entity were crawled, from which 100,000 random questions were selected.
- Crowdsourced Answers: Amazon Mechanical Turk workers identified answers by selecting entities, values, or lists of entities exclusively from the Freebase page corresponding to the entity in the question, or marked the question as unanswerable on Freebase. Questions were retained if annotated identically by at least two independent workers, yielding 6,642 consensus questions.
- Statistics and Splits: The cleaned dataset comprises 5,810 questions with 4,525 distinct word types (compared to 917 questions and 2,036 word types in FREE917). A 35% random split (2,032 examples) was held out for testing, while the remaining 65% was divided into an 80%-20% train/validation split.
- Evaluation Metric: Average $F_1$ score comparing the predicted set of answer entities $\llbracket d.z \rrbracket_{\mathcal{K}}$ against the crowdsourced reference set to accommodate partial answer lists on Freebase pages.
Knowl 7 — Test Performance on FREE917 and WEBQUESTIONS Benchmarks
empirical result
The SEMPRE semantic parser, trained from question-answer pairs via simple $\lambda$ -DCS, distant supervision alignment, and bridging, was evaluated on held-out test splits for FREE917 and WEBQUESTIONS:
- FREE917 Benchmark (Evaluated on exact match accuracy on the 30% held-out test split of 275 examples, using a 1,100-entry entity lexicon):
  - Supervised CCG semantic parser trained with full logical form annotations (Cai and Yates, 2013): $59.0\%$
  - SEMPRE (Trained only on question-answer pairs without logical forms): $62.0\%$
- WEBQUESTIONS Benchmark (Evaluated on average $F_1$ score on the 35% held-out test split of 2,032 examples, using a Lucene entity index over 41M Freebase entities):
  - Baseline parser (without bridging, alignment features, or denotation features): $26.9\%$
  - SEMPRE (Full model with alignment, bridging, denotation, and lexicalized features): $31.4\%$

Knowl 8 — Ablation of Predicate Generation: Alignment vs. Bridging across Datasets

data/table

The performance of the semantic parser under different binary predicate generation schemes was measured on the development sets of FREE917 (exact match accuracy, %) and WEBQUESTIONS (average $F_1$ , %) across 3 random splits:

On FREE917, bridging alone ( $66.9\%$ ) outperforms alignment alone ( $38.0\%$ ) because the dataset was generated from Freebase properties, containing rare predicates absent from the Open IE alignment lexicon that are resolved through type compatibility with adjacent unaries. On WEBQUESTIONS, alignment alone ( $30.6\%$ ) outperforms bridging alone ( $21.2\%$ ) because real search queries focus on frequent relations exhibiting high lexical variation. Combining both mechanisms achieves optimal performance on both datasets ( $71.3\%$ on FREE917, $32.9\%$ on WEBQUESTIONS).

Knowl 9 — Ablation of Denotation, Part-of-Speech, and Lexicalized Features

data/table

Ablation experiments evaluated on the development sets of FREE917 (exact match accuracy, %) and WEBQUESTIONS (average $F_1$ , %) demonstrate the impact of individual feature groups:

Feature Configuration	FREE917 (% Acc)	WebQuestions (% $F_1$ )
Full Model	71.3	32.9
$-$ POS Tag Features	70.5	28.9
$-$ Denotation Features	58.6	28.0
Alignment Only	71.3	32.9
Lexicalized Only	68.5	34.2
Lexicalized + Alignment	69.0	36.4

Denotation Features: Removing denotation size indicators causes the largest drop ( $-12.7\%$ on FREE917, $-4.9\%$ on WEBQUESTIONS) by failing to penalize logical forms that evaluate to empty sets $\emptyset$ .
Part-of-Speech Features: Removing POS features causes a $-4.0\%$ drop on WEBQUESTIONS by allowing the parser to skip essential proper nouns and content words or misidentify argument directions.
Lexicalization: Conjunction features between phrases and predicates improve performance on the large WEBQUESTIONS dataset ( $+3.5\%$ over alignment alone), but cause overfitting on FREE917 due to its smaller training size ( $69.0\%$ vs. $71.3\%$ ).

Knowl 10 — Beam-Based Bottom-Up Derivation Parsing Algorithm
algorithm
The semantic parser constructs derivations bottom-up over utterance spans using a chart parsing algorithm that maintains the top- $k$ scoring derivations per span:
Input: Utterance tokens $x = (w_1, \dots, w_m)$ , Lexicon $\mathcal{L}$ , KB properties $\mathcal{P}$ , Beam capacity $k$ , Parameters $\theta$
Output: Candidate derivations $\tilde{D}(x)$ at the root span
Initialize chart spans $C[i, j] \leftarrow \emptyset$ for $1 \le i \le j \le m$
for span_length = 1 to $m$ do
    for $i = 1$ to $m - \text{span\_length} + 1$ do
         $j \leftarrow i + \text{span\_length} - 1$
        phrase $\leftarrow x[i:j]$
        if phrase matches entity/unary/binary POS filters then
            for $(z, s) \in \mathcal{L}(\text{phrase})$ do
                 $C[i, j] \leftarrow C[i, j] \cup \{\text{Derivation}(z)\}$
        for $i' = i$ to $j - 1$ do
            for $d_1 \in C[i, i']$ and $d_2 \in C[i'+1, j]$ do
                 $z_1 \leftarrow d_1.z, \quad z_2 \leftarrow d_2.z$
                if type_compatible( $z_1, z_2$ ) then
                    add $\text{Derivation}(z_1 \sqcap z_2)$ to $C[i, j]$
                    add $\text{Derivation}(z_1.z_2)$ to $C[i, j]$
                    if $z_1 \in \{\text{count}\}$ then
                        add $\text{Derivation}(z_1(z_2))$ to $C[i, j]$
                    for $b \in \mathcal{P}$ type-compatible with $(z_1, z_2)$ do
                        add $\text{Derivation}(z_1 \sqcap b.z_2)$ to $C[i, j]$
         $C[i, j] \leftarrow \text{keep\_top\_k}(C[i, j], k, \text{score}_\theta)$
return $C[1, m]$
Hyperparameters: Beam size $k = 500$ on FREE917 and $k = 200$ on WEBQUESTIONS (generating on average 197 derivations at the root beam). Candidate logical forms that violate Freebase type constraints are pruned. Logical forms are converted to SPARQL queries and executed against a Virtuoso Freebase engine.

Coverage note — None was omitted; all key contributions including semantic formalisms, distant supervision alignment, bridging operations, feature design, the WebQuestions dataset, empirical evaluations, and ablation studies are covered.

References

1.Y. Artzi and L. Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Empirical Methods in Natural Language Processing (EMNLP), pages 421–432.
2.Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL), 1:49–62.
3.M. Banko, M. J. Cafarella, S. Soderland, M. Broadhead, and O. Etzioni. 2007. Open information extraction from the web. In International Joint Conference on Artificial Intelligence (IJCAI), pages 2670–2676.
4.S. Branavan, L. Zettlemoyer, and R. Barzilay. 2010. Reading between the lines: Learning to map high-level instructions to commands. In Association for Computational Linguistics (ACL), pages 1268–1277.
5.S. Branavan, D. Silver, and R. Barzilay. 2011. Learning to win by reading manuals in a Monte-Carlo framework. In Association for Computational Linguistics (ACL), pages 268–277.
6.S. Branavan, N. Kushman, T. Lei, and R. Barzilay. 2012. Learning high-level planning from text. In Association for Computational Linguistics (ACL), pages 126–135.
7.Q. Cai and A. Yates. 2013. Large-scale semantic parsing via schema matching and lexicon extension. In Association for Computational Linguistics (ACL).
8.A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. H. Jr, and T. M. Mitchell. 2010. Toward an architecture for never-ending language learning. In Association for the Advancement of Artificial Intelligence (AAAI).
9.A. X. Chang and C. Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In Language Resources and Evaluation (LREC), pages 3735–3740.
10.D. Chen. 2012. Fast online lexicon learning for grounded language acquisition. In Association for Computational Linguistics (ACL).
11.H. H. Clark. 1975. Bridging. In Workshop on theoretical issues in natural language processing, pages 169–174.
12.J. Clarke, D. Goldwasser, M. Chang, and D. Roth. 2010. Driving semantic parsing from the world’s response. In Computational Natural Language Learning (CoNLL), pages 18–27.
13.J. Duchi, E. Hazan, and Y. Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT).
14.A. Fader, S. Soderland, and O. Etzioni. 2011. Identifying relations for open information extraction. In Empirical Methods in Natural Language Processing (EMNLP).
15.A. Fader, L. Zettlemoyer, and O. Etzioni. 2013. Paraphrase-driven learning for open question answering. In Association for Computational Linguistics (ACL).
16.D. Goldwasser, R. Reichart, J. Clarke, and D. Roth. 2011. Confidence driven unsupervised semantic parsing. In Association for Computational Linguistics (ACL), pages 1486–1495.
17.Google. 2013. Freebase data dumps (2013-06-09). https://developers.google.com/freebase/data.
18.M. A. Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Interational Conference on Computational linguistics, pages 539–545.
19.R. Hoffmann, C. Zhang, X. Ling, L. S. Zettlemoyer, and D. S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Association for Computational Linguistics (ACL), pages 541–550.
20.J. Krishnamurthy and T. Kollar. 2013. Jointly learning to parse and perceive: Connecting natural language to the physical world. Transactions of the Association for Computational Linguistics (TACL), 1:193–206.
21.J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 754–765.
22.N. Kushman and R. Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Human Language Technology and North American Association for Computational Linguistics (HLT/NAACL), pages 826–836.
23.T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. 2010. Inducing probabilistic CCG grammars from logical form with higher-order unification. In Empirical Methods in Natural Language Processing (EMNLP), pages 1223–1233.
24.T. Kwiatkowski, L. Zettlemoyer, S. Goldwater, and M. Steedman. 2011. Lexical generalization in CCG grammar induction for semantic parsing. In Empirical Methods in Natural Language Processing (EMNLP), pages 1512–1523.
25.N. Lao, T. Mitchell, and W. W. Cohen. 2011. Random walk inference and learning in a large scale knowledge base. In Empirical Methods in Natural Language Processing (EMNLP).
26.T. Lei, F. Long, R. Barzilay, and M. Rinard. 2013. From natural language specifications to program input parsers. In Association for Computational Linguistics (ACL).
27.P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL), pages 590–599.
28.P. Liang. 2013. Lambda dependency-based compositional semantics. Technical report, ArXiv.
29.T. Lin, Mausam, and O. Etzioni. 2012. Entity linking at web scale. In Knowledge Extraction Workshop (AKBC-WEKEX).
30.Masaum, M. Schmitz, R. Bart, S. Soderland, and O. Etzioni. 2012. Open language learning for information extraction. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 523–534.
31.C. Matuszek, N. FitzGerald, L. Zettlemoyer, L. Bo, and D. Fox. 2012. A joint model of language and perception for grounded attribute learning. In International Conference on Machine Learning (ICML).
32.R. McDonald, K. Crammer, and F. Pereira. 2005. Online large-margin training of dependency parsers. In Association for Computational Linguistics (ACL), pages 91–98.
33.H. Poon. 2013. Grounded unsupervised semantic parsing. In Association for Computational Linguistics (ACL).
34.A. Popescu, O. Etzioni, and H. Kautz. 2003. Towards a theory of natural language interfaces to databases. In International Conference on Intelligent User Interfaces (IUI), pages 149–157.
35.S. Riedel, L. Yao, and A. McCallum. 2010. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pages 148–163.
36.M. Surdeanu, J. Tibshirani, R. Nallapati, and C. D. Manning. 2012. Multi-instance multi-label learning for relation extraction. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 455–465.
37.S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. J. Teller, and N. Roy. 2011. Understanding natural language commands for robotic navigation and mobile manipulation. In Association for the Advancement of Artificial Intelligence (AAAI).
38.C. Unger, L. Bhmann, J. Lehmann, A. Ngonga, D. Gerber, and P. Cimiano. 2012. Template-based question answering over RDF data. In World Wide Web (WWW), pages 639–648.
39.Y. W. Wong and R. J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Association for Computational Linguistics (ACL), pages 960–967.
40.M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath, V. Tresp, and G. Weikum. 2012. Natural language questions for the web of data. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL), pages 379–390.
41.M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic proramming. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050–1055.
42.L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI), pages 658–666.
43.C. Zhang, R. Hoffmann, and D. S. Weld. 2012. Ontological smoothing for relation extraction with minimal supervision. In Association for the Advancement of Artificial Intelligence (AAAI).

Citation

MLA

Berant, J., et al. “Semantic Parsing on Freebase from Question-Answer Pairs”. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013, pp. 1533–44, https://aclanthology.org/D13-1160/.

APA

Berant, J., Chou, A., Frostig, R., & Liang, P. (2013). Semantic Parsing on Freebase from Question-Answer Pairs. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1533–1544. https://aclanthology.org/D13-1160/

Chicago

Berant, J., A. Chou, R. Frostig, and P. Liang. 2013. “Semantic Parsing on Freebase from Question-Answer Pairs”. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1533–44. https://aclanthology.org/D13-1160/.

Harvard

Berant, J. et al. (2013) “Semantic Parsing on Freebase from Question-Answer Pairs”, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 1533–1544. Available at: https://aclanthology.org/D13-1160/.

Vancouver

1. Berant J, Chou A, Frostig R, Liang P (2013) Semantic Parsing on Freebase from Question-Answer Pairs. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 1533–1544

BibTeX

@inproceedings{berant-etal-2013-semantic,
    title = "Semantic Parsing on {F}reebase from Question-Answer Pairs",
    author = "Berant, Jonathan  and
      Chou, Andrew  and
      Frostig, Roy  and
      Liang, Percy",
    editor = "Yarowsky, David  and
      Baldwin, Timothy  and
      Korhonen, Anna  and
      Livescu, Karen  and
      Bethard, Steven",
    booktitle = "Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2013",
    address = "Seattle, Washington, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D13-1160/",
    pages = "1533--1544"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by-nc-sa/4.0/

System Configuration	FREE917 (% Acc)	WebQuestions (% $F_1$ )
Alignment	38.0	30.6
Bridging	66.9	21.2
Alignment + Bridging	71.3	32.9

Semantic Parsing on Freebase from Question-Answer Pairs

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Simple Lambda-DCS Formalism for Knowledge Base Semantic Parsing

Knowl 2 — Discriminative Log-Linear Derivation Model and Marginal Likelihood Objective

Knowl 3 — Lexicon Construction via Distant Supervision Alignment with Open IE Triples

Knowl 4 — Bridging Operation for Implicit, Light-Verb, and Rare Predicates

Knowl 5 — Feature Representations for Scoring Semantic Derivations

Knowl 6 — Construction and Structure of the WebQuestions Dataset

Knowl 7 — Test Performance on FREE917 and WEBQUESTIONS Benchmarks