基于正负例思维链的表格-文本混合金融数据自动问答方法

Summary

Answering questions over financial reports that mix tables and prose requires multi-step numerical reasoning, and LLMs still struggle with it. Program-of-Thoughts (PoT) prompting, for example, reaches only 64.5% accuracy on FinQA. Its demonstrations are also hand-crafted, which is costly and hard to justify. The authors instead select demonstrations automatically, drawing on how teachers use both worked examples and typical mistakes. GPT-3.5-Turbo first answers FinQA training questions zero-shot. Correctly solved items become a positive set, and failed items become a negative set. Questions in each set are embedded with Sentence-BERT and clustered with k-means, and the centroid items form the sample pools. Every prompt then pairs a fixed best-performing positive example with two retrieved negatives that resemble the current question. The example Python code also carries line-by-line comments, which keep generated programs concise and prevent garbled variable names. Finally, the authors corrected annotation errors in FinQA, producing FinQA-fix. The method improves accuracy by 3.6% on FinQA and 12.73% on FinQA-fix. These results show that both demonstration choice and data quality strongly affect financial numerical reasoning.

Key points

• Positive and negative pools come from LLM outputs. The number of clusters was chosen by silhouette coefficient: k=9 for positives and k=10 for negatives. Selection is static for the best positive and dynamic for the two nearest negatives.
• On FinQA (8,281 expert-written QA pairs; 6,251/883/1,147 train/dev/test), accuracy rises by 3.6% over the original PoT setup with 4 demonstrations.
• The authors estimate that nearly 20% of FinQA answers are wrong. They sort these errors into seven types, including wrong formulas, sign errors, unit mistakes, and Unicode noise. The revised FinQA-fix dataset yields a 12.73% accuracy gain.
• The method was also tested on TAT-QA with a 15-sample negative pool grouped by answer type (7 Span, 4 Multi-spans, 4 Arithmetic). It used 6 demonstrations (4 positive, 2 similar negative) against PoT's 8.
• Commented code in the prompts fixed failures such as undefined, truncated variable names, giving a small improvement over the PoT baseline.