Using Syntactic Distributional Patterns for
Data-Driven Answer Extraction from the Web
Alejandro Figueroa1 and John Atkinson2,⋆
1
Deutsches Forschungszentrum für Künstliche Intelligenz - DFKI,
Stuhlsatzenhausweg 3, D - 66123, Saarbrücken, Germany
2
Department of Computer Sciences, Universidad de Concepción, Concepción, Chile
,
Abstract. In this work, a data-driven approach for extracting answers
from web-snippets is presented. Answers are identified by matching con-
textual distributional patterns of the expected answer type(EAT) and
answer candidates. These distributional patterns are directly learnt from
previously annotated tuples {question, sentence, answer}, and the learn-
ing mechanism is based on the principles language acquisition. Results
shows that this linguistic motivated data-driven approach is encouraging.
Keywords: Natural Language Processing, Question Answering.
1 Introduction
The increase of the amount of information on the Web has led search engines
to deal with a huge amount of data as users have become retrievers of all sorts.
Nowadays, search engines are not only focusing on retrieving relevant documents
for a user’s particular request. They also provide other services (i.e., Group
Search, News Search, Glossary), hence the complexity of the request of the users
has addressed the research to Question Answering (QA) systems. These aim to
answer natural language (NL) questions prompted by users, by searching the
answer in a set of available documents on the Web. QA is a challenging task due
to the ambiguity of language and the complexity of the linguistic phenomena
that can be found in NL documents.
Typical questions to answer are those that look for name entities as answers
(i.e., locations, persons, dates, organizations). Nevertheless, QA systems are not
restricted to these kinds of questions. They also try to deal with more complex
ones that may require demanding reasoning tasks while the system is looking
for the answer [11].
Usually, QA systems start by analyzing the query [4,7] in order to determine
the EAT. The EAT allows the QA system to narrow the search space [8], while
it is ranking documents, sentences or sequences of words in which the answer is
⋆
This research is sponsored by FONDECYT, Chile under grant number 1040469 “Un
Modelo Evolucionario de Descubrimiento de Conocimiento Explicativo desde Textos
con Base Semantica con Implicaciones para el Analisis de Inteligencia.”
A. Gelbukh and C.A. Reyes-Garcia (Eds.): MICAI 2006, LNAI 4293, pp. 985–995, 2006.
c Springer-Verlag Berlin Heidelberg 2006
, 986 A. Figueroa and J. Atkinson
supposed to be. This set of likely answers is called answer candidates. In this
last step of the zooming process, the QA system must decide which are the most
suitable answers for the triggering query. This extraction and ranking of answer
candidates is traditionally based on [6,7,8] frequency counting, pattern match-
ing and detecting different orderings of query words, called paraphrases. Answer
extraction modules attempt to take advantage of the redundancy provided by
different information sources. This redundancy increases significantly the prob-
ability of finding a paraphrase, in which the answer can be readily identified.
Normally, QA systems extract these paraphrases at the sentence level [10]. The
rules for identifying paraphrases can manually be written or automatically learnt
[6,10], and they can consist of pre-parsed trees [10], or simple string based ma-
nipulations [6]. In general, paraphrases are learnt by retrieving sentences that
contain preciously annotated question-answer pairs. For example in [10], anchor
terms (i.e., “Lennon 1980”) are sent to the web, in order to retrieve sentences
that contain query and answer terms. Then, patterns are extracted from this
set of sentences with their likelihood being proportional to their redundancy on
the Web[7]. In most cases, the new set of retrieved sentences is matched with
paraphrases in order to extract new answers. At the same time, a huge set of
paraphrases [6] decreases considerably the need of deep linguistic processing like:
anaphora or synonym resolution. In some cases, it reduces the extraction to a
pattern matching by means of regular expressions[10]. As a result, strategies
based on paraphrases tend to perform better when questions aim for a name
entity as an answer: Locations, Names, Organizations. But, they perform poorly
when they aim for Noun Phrases[10].
Due to the huge amount of paraphrases, statistical methods are also used for
extracting answers. In [5], a strategy for answering questions is learnt directly
from data. This strategy conceives the answer extraction problem as a binary
classification problem in which text snippets are labelled as correct or incorrect.
The classifier is based on a set of features from lexical n-grams to parse trees.
The major problem of statistical-based approaches is that, frequently, they get
inexact answers, which usually consist of substrings of the answer, the answer
surrounded by some context words, or strings highly closed to answers.
Nevertheless, it is still unclear how each different technique contributes to deal
with the linguistic phenomena that QA systems face while searching for the an-
swer. One solution for this may involve a trade-off between the implementation
of rule-based and easy re-trainable data-driven systems. In [10], a strategy for
combining the output of different kinds of answer extractors is introduced. This
re-ranker is based on a Maximum Entropy Linear Classifier, which was trained
on a set of 48 different types of features such as ranking in the answer extraction
modules, redundancy, negative feedback, etc. Results show that a good strategy
for combing answer extractors, based mainly on different strategies, can signifi-
cantly improve the overall performance of QA systems [11].
Strategies based on paraphrases aim to find a re-writing of the query within
the text where the answer is easily identified. Their main drawback is that when-
ever the answer is in an context, which do not match any re-writing rule, it will
Data-Driven Answer Extraction from the Web
Alejandro Figueroa1 and John Atkinson2,⋆
1
Deutsches Forschungszentrum für Künstliche Intelligenz - DFKI,
Stuhlsatzenhausweg 3, D - 66123, Saarbrücken, Germany
2
Department of Computer Sciences, Universidad de Concepción, Concepción, Chile
,
Abstract. In this work, a data-driven approach for extracting answers
from web-snippets is presented. Answers are identified by matching con-
textual distributional patterns of the expected answer type(EAT) and
answer candidates. These distributional patterns are directly learnt from
previously annotated tuples {question, sentence, answer}, and the learn-
ing mechanism is based on the principles language acquisition. Results
shows that this linguistic motivated data-driven approach is encouraging.
Keywords: Natural Language Processing, Question Answering.
1 Introduction
The increase of the amount of information on the Web has led search engines
to deal with a huge amount of data as users have become retrievers of all sorts.
Nowadays, search engines are not only focusing on retrieving relevant documents
for a user’s particular request. They also provide other services (i.e., Group
Search, News Search, Glossary), hence the complexity of the request of the users
has addressed the research to Question Answering (QA) systems. These aim to
answer natural language (NL) questions prompted by users, by searching the
answer in a set of available documents on the Web. QA is a challenging task due
to the ambiguity of language and the complexity of the linguistic phenomena
that can be found in NL documents.
Typical questions to answer are those that look for name entities as answers
(i.e., locations, persons, dates, organizations). Nevertheless, QA systems are not
restricted to these kinds of questions. They also try to deal with more complex
ones that may require demanding reasoning tasks while the system is looking
for the answer [11].
Usually, QA systems start by analyzing the query [4,7] in order to determine
the EAT. The EAT allows the QA system to narrow the search space [8], while
it is ranking documents, sentences or sequences of words in which the answer is
⋆
This research is sponsored by FONDECYT, Chile under grant number 1040469 “Un
Modelo Evolucionario de Descubrimiento de Conocimiento Explicativo desde Textos
con Base Semantica con Implicaciones para el Analisis de Inteligencia.”
A. Gelbukh and C.A. Reyes-Garcia (Eds.): MICAI 2006, LNAI 4293, pp. 985–995, 2006.
c Springer-Verlag Berlin Heidelberg 2006
, 986 A. Figueroa and J. Atkinson
supposed to be. This set of likely answers is called answer candidates. In this
last step of the zooming process, the QA system must decide which are the most
suitable answers for the triggering query. This extraction and ranking of answer
candidates is traditionally based on [6,7,8] frequency counting, pattern match-
ing and detecting different orderings of query words, called paraphrases. Answer
extraction modules attempt to take advantage of the redundancy provided by
different information sources. This redundancy increases significantly the prob-
ability of finding a paraphrase, in which the answer can be readily identified.
Normally, QA systems extract these paraphrases at the sentence level [10]. The
rules for identifying paraphrases can manually be written or automatically learnt
[6,10], and they can consist of pre-parsed trees [10], or simple string based ma-
nipulations [6]. In general, paraphrases are learnt by retrieving sentences that
contain preciously annotated question-answer pairs. For example in [10], anchor
terms (i.e., “Lennon 1980”) are sent to the web, in order to retrieve sentences
that contain query and answer terms. Then, patterns are extracted from this
set of sentences with their likelihood being proportional to their redundancy on
the Web[7]. In most cases, the new set of retrieved sentences is matched with
paraphrases in order to extract new answers. At the same time, a huge set of
paraphrases [6] decreases considerably the need of deep linguistic processing like:
anaphora or synonym resolution. In some cases, it reduces the extraction to a
pattern matching by means of regular expressions[10]. As a result, strategies
based on paraphrases tend to perform better when questions aim for a name
entity as an answer: Locations, Names, Organizations. But, they perform poorly
when they aim for Noun Phrases[10].
Due to the huge amount of paraphrases, statistical methods are also used for
extracting answers. In [5], a strategy for answering questions is learnt directly
from data. This strategy conceives the answer extraction problem as a binary
classification problem in which text snippets are labelled as correct or incorrect.
The classifier is based on a set of features from lexical n-grams to parse trees.
The major problem of statistical-based approaches is that, frequently, they get
inexact answers, which usually consist of substrings of the answer, the answer
surrounded by some context words, or strings highly closed to answers.
Nevertheless, it is still unclear how each different technique contributes to deal
with the linguistic phenomena that QA systems face while searching for the an-
swer. One solution for this may involve a trade-off between the implementation
of rule-based and easy re-trainable data-driven systems. In [10], a strategy for
combining the output of different kinds of answer extractors is introduced. This
re-ranker is based on a Maximum Entropy Linear Classifier, which was trained
on a set of 48 different types of features such as ranking in the answer extraction
modules, redundancy, negative feedback, etc. Results show that a good strategy
for combing answer extractors, based mainly on different strategies, can signifi-
cantly improve the overall performance of QA systems [11].
Strategies based on paraphrases aim to find a re-writing of the query within
the text where the answer is easily identified. Their main drawback is that when-
ever the answer is in an context, which do not match any re-writing rule, it will
