Elamur Research⌕
☰
home / papers / 2024 / ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering

ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering

Paper · 2024 · Xiuying Chen, Tairan Wang, Taicheng Guo et al.

Summary

Chemistry has lacked QA benchmarks built from real research. Existing chemical sets such as KGQA rely on templates over a knowledge graph and mostly test basic concepts. The authors address this gap with ScholarChemQA. They mined chemistry papers whose titles are phrased as questions and treated each abstract as the evidence for a yes/no/maybe answer. Of roughly 40k such pairs, 1,050 were manually labelled for training, validation and testing. The rest remain unlabelled, and the labelled part is heavily skewed toward "yes".

Strong LLMs handle the task poorly; GPT-3.5 reaches only about 54% accuracy. The authors therefore propose QAMatch, a compact semi-supervised model. It counters class imbalance by weighting each example's loss by inverse class frequency. It exploits unlabelled pairs through pseudo-labels, which are calibrated toward a target label distribution. A SoftMix operation creates augmented variants of both question and context in representation space rather than in the raw text. QAMatch beats similar-sized baselines and LLMs on ScholarChemQA and on four other benchmarks. This shows that small, specialised models can be effective for scientific QA.

Key points

  • Data source: over a million chemistry titles and abstracts were collected and filtered down to about 40k question–abstract pairs. Another ~4k declarative titles were rewritten as yes/no questions to add data.
  • Labelled subset: 1,050 manually labelled instances, split 65.8% yes, 21.2% no and 13.0% maybe. "Maybe" marks mixed or inconclusive evidence in the abstract.
  • LLM evaluation: Llama2-70B, GPT-3.5 and GPT-4 were tested. GPT-3.5 scored around 54% accuracy, which shows how hard terminology-dense papers are for these models.
  • QAMatch components: inverse-frequency loss re-weighting, pseudo-label calibration toward a desired class distribution, and dual (question- and context-side) SoftMix augmentation in embedding space.
  • Results and availability: QAMatch is described as open-source and computationally efficient. It outperformed GPT-3.5, GPT-4 and comparable-scale baselines on ScholarChemQA and four additional benchmark datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMaterials Science · 26 worksMaterials Chemistry · 26 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 works

Cite

Cite as Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Mingchen Zhuge, Jurgen Schmidhuber, Xin Gao and Xiangliang Zhang. 2024. ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering. arXiv:2407.16931.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.16931 · arXiv 2407.16931

@misc{chen2024scholarchemqa,
  title = {{ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering}},
  author = {Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Mingchen Zhuge, Jurgen Schmidhuber, Xin Gao and Xiangliang Zhang},
  year = {2024},
  doi = {10.48550/arxiv.2407.16931},
  eprint = {2407.16931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.16931}
}

References

71 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Christmann, R. Saha Roy, and G. Weikum, “Conversational question answering on heterogeneous sources,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  2. C. Qu, L. Yang, C. Chen, M. Qiu, W. B. Croft, and M. Iyyer, “Open-retrieval conversational question answering,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2020. find
  3. V. Bolotova, V. Blinov, F. Scholer, W. B. Croft, and M. Sanderson, “A non-factoid question-answering taxonomy,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  4. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” in Proceeding of of North American Chapter of the Association for Computational Linguistics, 2019.
  5. A. Ghoshal, S. Iyer, B. Paranjape, K. Lakhotia, S. W.-t. Yih, and Y. Mehdad, “Quaser: Question answering with scalable extractive rationalization,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  6. A. Garcia-Silva, C. Berrio, J. M. Gomez-Perez, J. A. Martı́nez-Heras, A. Donati, and I. Roma, “Spaceqa: Answering questions about the design of space missions and space craft concepts,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  7. G. Peretz, M. Arraf, and K. Radinsky, “What if: Generating code to answer simulation questions in chemistry texts,” in Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 1335–1344. find
  8. Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” in Proceeding of Empirical Methods in Natural Language Processing, 2019.
  9. D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits, “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, 2021.
  10. H. Wang, L. Zhou, W. Zhang, and X. Wang, “Literatureqa: A qestion answering corpus with graph knowledge on academic literature,” in Proceeding of CIKM, 2021. find
  11. X. Zhou, D. Nurkowski, S. Mosbach, J. Akroyd, and M. Kraft, “Question answering system for chemistry,” Journal of Chemical Information and Modeling, 2021. find
  12. C. Huang, Y. Li, C. C. Loy, and X. Tang, “Learning deep representation for imbalanced classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5375– 5384. find
  13. J. Allan, J. Aslam, N. Belkin, C. Buckley, J. Callan, B. Croft, S. Dumais, N. Fuhr, D. Harman, D. J. Harper et al., “Challenges in information retrieval and language modeling: report of a workshop held at the center for intelligent information retrieval, university of massachusetts amherst, septembe… find
  14. P. J. Smith, S. J. Shute, B. Galdes, and M. H. Chignell, “Knowledge-based search tactics for an intelligent intermediary system,” ACM Transactions on Information Systems (TOIS), vol. 7, no. 3, pp. 246– 270, 1989. find
  15. B. Sun, P. Mitra, C. Lee Giles, and K. T. Mueller, “Identifying, indexing, and ranking chemical formulae and chemical names in digital documents,” ACM Transactions on Information Systems (TOIS), vol. 29, no. 2, pp. 1–38, 2011. find
  16. R. Entlich, J. Olsen, L. Garson, M. Lesk, L. Normore, and S. Weibel, “Making a digital library: the contents of the core project,” ACM Transactions on Information Systems (TOIS), vol. 15, no. 2, pp. 103– 123, 1997. find
  17. M. Lupu, J. Huang, J. Zhu, and J. Tait, “Trec-chem: large scale chemical information retrieval evaluation at trec,” in Acm Sigir Forum, vol. 43, no. 2. ACM New York, NY, USA, 2009, pp. 63–70. find
  18. N. Pang, L. Qian, W. Lyu, and J.-D. Yang, “Transfer learning for scientific data chain extraction in small chemical corpus with joint bert-crf model.” in BIRNDL@ SIGIR, 2019, pp. 28–41. find
  19. N. Krdzavac, S. Mosbach, D. Nurkowski, P. Buerger, J. Akroyd, J. Martin, A. Menon, and M. Kraft, “An ontology and semantic web service for quantum chemistry calculations,” Journal of chemical information and modeling, 2019. find
  20. F. Farazi, J. Akroyd, S. Mosbach, P. Buerger, D. Nurkowski, M. Salamanca, and M. Kraft, “Ontokin: An ontology for chemical kinetic reaction mechanisms,” Journal of Chemical Information and Modeling, 2019. find
  21. Z. Wei, W. Ji, X. Geng, Y. Chen, B. Chen, T. Qin, and D. Jiang, “Chemistryqa: A complex question answering dataset from chemistry,” 2020. find
  22. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” ICLR, 2021.
  23. L. Yang, C.-K. Hsieh, H. Yang, J. P. Pollak, N. Dell, S. Belongie, C. Cole, and D. Estrin, “Yum-me: a personalized nutrient-based meal recommender system,” ACM Transactions on Information Systems (TOIS), vol. 36, no. 1, pp. 1–31, 2017. find
  24. H. Huang, X. Wei, L. Nie, X. Mao, and X.-S. Xu, “From question to text: Question-oriented feature attention for answer selection,” ACM Transactions on Information Systems (TOIS), vol. 37, no. 1, pp. 1–33, 2018. find
  25. S. Liang, Y. Luo, and Z. Meng, “Profiling users for question answering communities via flow-based constrained co-embedding model,” ACM Transactions on Information Systems (TOIS), vol. 40, no. 2, pp. 1–38, 2021. find
  26. V. Yadav, R. Sharp, and M. Surdeanu, “Sanity check: A strong alignment and information retrieval baseline for question answering,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018, pp. 1217–1220. find
  27. A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in Conference on Health, Inference, and Learning. PMLR, 2022, pp. 248–260. — MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
  28. J. Li, S. Zhong, and K. Chen, “Mlec-qa: A chinese multi-choice biomedical question answering dataset,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8862–8874. find
  29. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in Proceeding of Empirical Methods in Natural Language Processing, 2018.
  30. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee et al., “Natural questions: A benchmark for question answering research,” Transactions of the Association for Computational Linguistics, 2019.
  31. M. Saeidi, M. Bartolo, P. Lewis, S. Singh, T. Rocktäschel, M. Sheldon, G. Bouchard, and S. Riedel, “Interpretation of natural language rules in conversational machine reading,” in Proceeding of Empirical Methods in Natural Language Processing, 2018.
  32. G. Tsatsaronis, G. Balikas, P. Malakasiotis, I. Partalas, M. Zschunke, M. R. Alvers, D. Weissenborn, A. Krithara, S. Petridis, D. Polychronopoulos et al., “An overview of the bioasq large-scale biomedical semantic indexing and question answering competition,” BMC bioinformatics, 2015. find
  33. S. Ertekin, J. Huang, and C. L. Giles, “Active learning for class imbalance problem,” in Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, 2007, pp. 823–824. find
  34. A. Moreo, A. Esuli, and F. Sebastiani, “Distributional random oversampling for imbalanced text classification,” in Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, 2016, pp. 805–808. find
  35. M. Naseri and H. Zamani, “Analyzing and predicting news popularity in an instant messaging service,” in Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 1053–1056. find
  36. Y. Zhu, X. Wang, Q. Li, T. Yao, and S. Liang, “Botspot++: A hierarchical deep ensemble model for bots install fraud detection in mobile advertising,” ACM Transactions on Information Systems (TOIS), vol. 40, no. 3, pp. 1–28, 2021. find
  37. M. Lukasik, K. Bontcheva, T. Cohn, A. Zubiaga, M. Liakata, and R. Procter, “Gaussian processes for rumour stance classification in social media,” ACM Transactions on Information Systems (TOIS), vol. 37, no. 2, pp. 1–24, 2019. find
  38. A. Frummet, D. Elsweiler, and B. Ludwig, ““what can i cook with these ingredients?”-understanding cooking-related information needs in conversational search,” ACM Transactions on Information Systems (TOIS), vol. 40, no. 4, pp. 1–32, 2022. find
  39. X. Quan, Q. Wang, Y. Zhang, L. Si, and L. Wenyin, “Latent discriminative models for social emotion detection with emotional dependency,” ACM Transactions on Information Systems (TOIS), vol. 34, no. 1, pp. 1–19, 2015. find
  40. P. Wang, Z. Yang, S. Niu, Y. Zhang, L. Zhang, and S. Niu, “Modeling dynamic pairwise attention for crime classification over legal articles,” in the 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 485–494. find
  41. B. Xu, J. Huang, L. Hou, H. Shen, J. Gao, and X. Cheng, “Label-consistency based graph neural networks for semi-supervised node classification,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2020. find
  42. J. Lee, Y. Oh, Y. In, N. Lee, D. Hyun, and C. Park, “Grafn: Semi-supervised node classification on graph with few labels via non-parametric distribution assignment,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  43. S. Li, M. Yang, C. Li, and R. Xu, “Dual pseudo supervision for semi-supervised text classification with a reliable teacher,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2022. find
  44. Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” in Proceeding of International Conference on Computer Vision and Pattern Recogintion, 2019. find
  45. K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning imbalanced datasets with label-distribution-aware margin loss,” Proceeding of Neural Information Processing Systems, 2019. find
  46. J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization loss for long-tailed object recognition,” in Proceeding of International Conference on Computer Vision and Pattern Recogintion, 2020. find
  47. J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi et al., “Balanced meta-softmax for long-tailed visual recognition,” Proceeding of Neural Information Processing Systems, 2020. find
  48. D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring,” Proceeding of International Conference on Learning Representations, 2020. find
  49. H. Chen, R. Tao, Y. Fan, Y. Wang, J. Wang, B. Schiele, X. Xie, B. Raj, and M. Savvides, “Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning,” in Proceeding of International Conference on Learning Representations, 2023. find
  50. G. Faggioli, L. Dietz, C. L. Clarke, G. Demartini, M. Hagen, C. Hauff, N. Kando, E. Kanoulas, M. Potthast, B. Stein et al., “Perspectives on large language models for relevance judgment,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2023. find
  51. S. Tian, Q. Jin, L. Yeganova, P.-T. Lai, Q. Zhu, X. Chen, Y. Yang, Q. Chen, W. Kim, D. C. Comeau et al., “Opportunities and challenges for chatgpt and large language models in biomedicine and health,” Briefings in Bioinformatics, vol. 25, no. 1, p. bbad493, 2024. find
  52. J. Zhou and X. Gao, “Skingpt: A dermatology diagnostic system with vision large language model,” arXiv preprint arXiv:2304.10691, 2023. — [arXiv:2304.10691](https://arxiv.org/abs/2304.10691) find
  53. K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit, “Is gpt-3 all you need for low-data discovery in chemistry?” 2023. find
  54. C. M. Castro Nascimento and A. S. Pimentel, “Do large language models understand chemistry? a conversation with chatgpt,” Journal of Chemical Information and Modeling, 2023. find
  55. T. Guo, K. Guo, Z. Liang, Z. Guo, N. V. Chawla, O. Wiest, X. Zhang et al., “What indeed can gpt models do in chemistry? a comprehensive benchmark on eight tasks,” arXiv preprint arXiv:2305.18365, 2023. — [arXiv:2305.18365](https://arxiv.org/abs/2305.18365) find
  56. O. A. Jefferson, A. Jaffe, D. Ashton, B. Warren, D. Koellhofer, U. Dulleck, A. Ballagh, J. Moe, M. DiCuccio, K. Ward et al., “Mapping the global influence of published research on industry and innovation,” Nature biotechnology, 2018. find
  57. K. Toutanvoa and C. D. Manning, “Enriching the knowledge sources used in a maximum entropy part-of-speech tagger,” in Proceeding of Empirical Methods in Natural Language Processing, 2000.
  58. Y. Wang, H. Chen, Q. Heng, W. Hou, M. Savvides, T. Shinozaki, B. Raj, Z. Wu, and J. Wang, “Freematch: Self-adaptive thresholding for semi-supervised learning,” Proceeding of International Conference on Learning Representations, 2023. find
  59. Y. Gan, X. Chen, Q. Huang, M. Purver, J. R. Woodward, J. Xie, and P. Huang, “Towards robustness of text-to-sql models against synonym substitution,” in Proceeding of Association for Computational Linguistics, 2021, pp. 2505–2515.
  60. M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in Proceeding of ECCV, 2014.
  61. V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in Proceeding of International Conference on Machine Learning, 2019. find
  62. X. Chen, G. Long, C. Tao, M. Li, X. Gao, C. Zhang, and X. Zhang, “Improving the robustness of summarization systems with dual augmentation,” Proceeding of Association for Computational Linguistics, 2023. find
  63. J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceeding of AAssociation for Computational Linguistics, 2019.
  64. K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Proceeding of Neural Information Processing Systems, 2020. find
  65. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
  66. X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” Proceeding of Neural Information Processing Systems, 2015.
  67. H. Tzaban, I. Guy, A. Greenstein-Messica, A. Dagan, L. Rokach, and B. Shapira, “Product bundle identification using semi-supervised learning,” in Proceeding of International Conference on Research on Development in Information Retrieval, 2020. find
  68. J. Kim, Y. Hur, S. Park, E. Yang, S. J. Hwang, and J. Shin, “Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning,” Proceeding of Neural Information Processing Systems, 2020. find
  69. H. Lee, S. Shin, and H. Kim, “Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning,” Proceeding of Neural Information Processing Systems, 2021. find
  70. C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, “Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning,” in Proceeding of International Conference on Computer Vision and Pattern Recogintion, 2021. find
  71. J. R. Hershey and P. A. Olsen, “Approximating the kullback leibler divergence between gaussian mixture models,” in 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP’07, vol. 4. IEEE, 2007, pp. IV–317. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2024-scholarchemqa-unveiling-power-language-models/