Automatic Generation and Reranking of SQL-Derived Answers to NL Questions - Trustworthy Eternal Systems via Evolving, Software Data and Knowledge - page 73

Fig. 7. Learning curve comparison between simple answer generator and the reranking
model using the STK × STK kernel

6.1 Setup
To learn the reranker, we used SVM-Light-TK 6 , which extends the SVM-Light
optimizer [6] with tree kernels. i.e. Syntactic Tree Kernel (STK) as described
in Section 5. We modeled many different combinations described in the next
section. We used the default parameters, i.e. the cost and trade-off parameters
= 1 (for normalized kernels) and λ =0 . 4 (see Sec. 4).
To generate the set of possible SQL queries we applied our algorithm de-
scribed in Section 3 to GeoQueries 7 corpus. We started from a set of 700 NL
questions 8 . Thanks to our generative algorithm we discovered and fixed all er-
rors and inconsistencies in SQL queries, except for 3 cases that still lead to a
MySQL error. Indeed, since we can't test the correctness of our generated query
(without a result set to compare with) we considered a subset of 697 pairs.

6.2 Generative Results

Given a question from GeoQuery, our algorithm was able to generate a correct
SQL query in the first 25 in 95.3% of the cases. This also means that our system
cannot answer to 33 questions. This is due to (1) empty clauses set

S

and/or

W

, for example, “ How many square kilometers in the us? ” does not contain
any useful stem; and (2) from mismatching nested queries, for example, “ Count
the states which have elevations lower than what alabama has ”containsanim-
plicit reference to a missing piece of question. In addition there are ambiguous
questions like “ Which states does the colorado? ” from which we retrieve an in-
complete dependency set.
For all remaining questions from which we succeed in generating an ordered
list of possible queries, we find that the query on top of the list retrieves the

6 http://disi.unitn.it/ ~ moschitt/Tree-Kernel.htm
7 Available at http://www.cs.utexas.edu/ ml/geo.html
8 This are the first 700 questions of the 880 ones that Mooney's group [14] paired with
logical formulas in Prolog and that Popescu et al. [10] manually converted into SQL.