Elamur Research⌕
☰
home / papers / 2014 / Semantic approaches to software component retrieval with English queries

Semantic approaches to software component retrieval with English queries

Paper · 2014 · Huijing Deng and Grzegorz Chrupała

Summary

The paper tackles code search: letting a programmer describe needed functionality in plain English and get back a ranked list of library methods, even when those methods have no prose documentation. The authors note that the Java Standard Library offers a naturally parallel resource. Each method has a formal signature and an English Javadoc description. From 7183 such pairs, they learn which English words tend to go with which signature terms. Retrieval is framed as query likelihood: each method signature is scored by how probably it generates the query words. Three variants are compared: a smoothed term-matching baseline, Polylingual LDA interpolated with that baseline, and IBM model 1 treated as a word-to-code translation table. Test queries are simulated from the first sentence of held-out descriptions. Each query is ranked against several thousand signatures. IBM model 1 clearly wins, with MRR near 0.49 and the correct method in the top 10 for almost 80% of queries. This suggests grounding words in code through parallel documentation is practical for real API search tools.

Key points

  • Data: six packages of the Java SE 6 API (io, lang, math, net, text, util), 7183 signature–description pairs, split 60/20/20 into train, validation and test. Queries shorter than three words were dropped.

  • Preprocessing:

    • camelCase identifiers were split and lowercased, and punctuation and HTML were removed;
    • signatures were enriched with their package, class and superclass.

  • Test results (MRR / Acc@1 / Acc@10):

    ModelMRRAcc@1Acc@10
    Baseline (λ=0.7)0.3320.2230.530
    Interpolated PLDA (α=0.1)0.3520.2420.562
    IBM model 1 (λ=0.3)0.4930.3390.793

  • On validation, pure PLDA did best with k=500 topics (MRR 0.291). IBM model 1 stopped improving after about 15 EM iterations.

  • The translation model rescues queries with little lexical overlap. The correct method for the "tangent of an angle" query moved from rank 1998 to 2, and an equals method moved from 1718 to 1. Data and code are released in the codeine repository on Bitbucket.

Interesting ideas

  • I1. Existing documentation that pairs prose with a formal object is free parallel data, so a word-alignment model can learn word-to-symbol links without manual annotation.
  • I2. Realistic test queries can be built by taking the first, summary-like sentence of held-out documents. This yields a large evaluation set with exactly one known correct answer per query.
  • I3. Candidates are ranked against the whole collection (training plus test items) rather than only the test pool, which makes the task harder and closer to real use.
  • I4. A learned translation score is blended with a literal-match estimate and collection-level smoothing, so the system falls back on surface overlap when learned associations are weak.
  • I5. The error analysis shows that shared but generic tokens like "object" carry no signal, while topic models blur fine distinctions compared with word-level links.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Glosses from WordNet, FrameNet definitions or Wikidata descriptions could be aligned with their formal entries. This would induce lexicon links from words to ontology items, and each link would stay inspectable as an explicit translation probability.
  • I2 (answer): QA readers could be tested automatically by turning the lead sentence of held-out fable or textbook items into questions, each with a single gold target.
  • I3 (learn): Evaluations could rank gold answers among large distractor pools, acting as a negative control so that scores are not inflated by small candidate sets.
  • I4 (frames): When mapping verbs to VerbNet classes, a learned association score could be interpolated with lemma-level matching. Unseen verbs would still get a traceable default.
  • I5 (world): When matching text entities to ontology nodes, very general shared labels should be down-weighted, and precise alignments should be preferred over coarse topic clusters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tilburg UniversityTilburg · Netherlands50 works · 6,429 citations🎓ETH ZurichZurich · Switzerland72 works · 5,176 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksInformation retrieval & search · 473 works

Cite

Cite as Huijing Deng and Grzegorz Chrupała. 2014. Semantic approaches to software component retrieval with English queries. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{deng2014semantic,
  title = {{Semantic approaches to software component retrieval with English queries}},
  author = {Huijing Deng and Grzegorz Chrupała},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5jtrwudtkmf9},
  url = {https://doi.org/10.63317/5jtrwudtkmf9}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bajracharya, S., Ngo, T., Linstead, E., Dou, Y., Rigor, P., Baldi, P., and Lopes, C. (2006). Sourcerer: a search engine for open source code supporting structure-based search. In Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications, pages 681–682. ACM. find
  2. Berger, A. L. and Lafferty, J. D. (1999). Information Retrieval as Statistical Translation. In SIGIR, pages 222– 229. find
  3. Branavan, S., Zettlemoyer, L. S., and Barzilay, R. (2010). Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1268–1277. Association for Computational Linguistics. find
  4. Brown, P. F., Pietra, S. D., Pietra, V. J. D., Mercer, R. L., and Mercer, R. L. (1993). The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, pages 263–311. find
  5. Fazly, A., Alishahi, A., and Stevenson, S. (2010). A probabilistic computational model of cross-situational word learning. Cognitive Science, 34(6):1017–1063. ↗
  6. Lafferty, J. and Zhai, C. (2001). Document language models, query models, and risk minimization for information retrieval. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 111–119. ACM. find
  7. Liang, P., Jordan, M. I., and Klein, D. (2011). Learning dependency-based compositional semantics. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 590–599. Association for Computational Linguistics. ↗
  8. Mandelin, D., Xu, L., Bodı́k, R., and Kimelman, D. (2005). Jungloid mining: helping to navigate the API jungle. In ACM SIGPLAN Notices, volume 40, pages 48–61. ACM. find
  9. Matuszek, C., Fox, D., and Koscher, K. (2010). Following directions using statistical machine translation. In Proceedings of the 5th ACM/IEEE international conference on Human-robot interaction, pages 251–258. IEEE Press. ↗
  10. McMillan, C., Grechanik, M., Poshyvanyk, D., Xie, Q., and Fu, C. (2011). Portfolio: finding relevant functions and their usage. In Software Engineering (ICSE), 2011 33rd International Conference on, pages 111–120. IEEE. find
  11. Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., and McCallum, A. (2009). Polylingual topic models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 880–889. Association for Computational Linguistics.
  12. Momtazi, S. and Klakow, D. (2011). Trained trigger language model for sentence retrieval in QA: bridging the vocabulary gap. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 2005–2008. ACM. find
  13. Song, F. and Croft, W. B. (1999). A general language model for information retrieval. In Proceedings of the eighth international conference on Information and knowledge management, pages 316–321. ACM. find
  14. Stylos, J. and Myers, B. A. (2006). Mica: A web-search tool for finding API components and examples. In Visual Languages and Human-Centric Computing, 2006. VL/HCC 2006. IEEE Symposium on, pages 195–202. IEEE. find
  15. Wei, X. and Croft, W. B. (2006). Lda-based document models for ad-hoc retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 178– 185. ACM. find
  16. Zhai, C. and Lafferty, J. (2001). A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 334–342. ACM. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deng-2014-semantic-approaches-software-component-retrieval/