Elamur Research⌕
☰
home / papers / 2026 / Logic-Level Evaluation of Logical Table-to-Text Generation

Logic-Level Evaluation of Logical Table-to-Text Generation

Summary

Aggregate fidelity scores for Logical Table-to-Text (LT2T) generation suggest that LLMs perform well. However, a single score can hide failures on specific operations and does not show whether an error comes from grounding, operator choice or wording. The authors propose a diagnostic framework that splits LT2T into four tasks:

  • executing a logical form (LF) with an evidence-grounded trace;
  • turning a given LF into an entailed sentence;
  • naming the logic type of a statement;
  • writing a new entailed claim without any LF.

They test frontier and small/distilled LLMs on 700 Logic2Text instances in a one-shot setting. Results are broken down by seven logic types and scored with NLI-Acc, TAPAS-Acc, TAPEX-Acc and SP-Acc, plus a small human study.

The verifiers often disagree, so model rankings depend on which verifier is used. The authors also find a "meta-logical gap": models write statements judged faithful yet cannot reliably say which operation they expressed. Ordinal and aggregation reasoning are the weakest areas. The authors argue that per-operation evaluation is needed to judge whether LT2T systems are logically reliable.

Key points

  • Logic types: seven categories make up the taxonomy, namely Only, Majority, Comparative, Superlative, Ordinal, Aggregation and Count. Table 1 lists typical execution errors for each type, such as miscalculated averages, wrong row filtering, and fuzzy versus strict equality.
  • Removing the LF hurts fidelity:
    • For frontier models, macro TAPAS-Acc drops from 74 to 69, TAPEX-Acc from 79 to 73, and SP-Acc from 61 to 56.
    • Small/distilled models lose about 10–11 TAPAS/TAPEX points.
    • Majority suffers most for frontier models (TAPAS 75 → 46).
    • Ordinal scores sometimes rise without the LF, apparently because models fall back on easier claims.
  • Logic-type identification:
    • Frontier models average 76% overall, ranging from GPT5.1 (60%) to Deepseek-v3.2 (93%).
    • Ordinal is the hardest type, at 48% for frontier models and 21% for small models.
    • Small models average 52%; Deepseek-R1-Distill-Llama70B scores best among them at 83%.
  • Verifier disagreement:
    • NLI-Acc disagrees with TAPEX-Acc at a rate of 0.73 and with TAPAS-Acc at 0.67, compared with 0.50 between TAPAS-Acc and TAPEX-Acc.
    • Disagreement is highest for Superlative (up to 0.82) and Comparative.
  • Limitations: the study uses a single one-shot prompt. The human study involved one annotator, a small sample and one model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.41

@inproceedings{trigg2026logic,
  title = {{Logic-Level Evaluation of Logical Table-to-Text Generation}},
  author = {Trigg, Lena and Hougen, Dean F. and Bilal, Ahsan},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.41},
  url = {https://doi.org/10.18653/v1/2026.conll-main.41}
}

References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929– 7942, Online. Association for Computational Linguistics. find our card
  2. Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020b. Logic2Text: High-fidelity natural language generation from logical forms. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2096–2111, Online. Association for… find
  3. Ao Liu, Haoyu Dong, Naoaki Okazaki, Shi Han, and Dongmei Zhang. 2022. PLOG: Table-to-logic pre-training for logical table-to-text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5531– 5546, Abu Dhabi, United Arab Emirates. Association for… find
  4. Lena Trigg and Dean F. Hougen. 2025. Logical table-to-text generation: Challenges, methods, and reasoning. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pag… find
  5. Bohao Yang, Chen Tang, Kun Zhao, Chenghao Xiao, and Chenghua Lin. 2024. Effective distillation of table-based reasoning ability from LLMs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5538– 5550, T… find
  6. Xueliang Zhao, Tingchen Fu, Lemao Liu, Lingpeng Kong, Shuming Shi, and Rui Yan. 2023a. SORTIE: Dependency-aware symbolic reasoning for logical data-to-text generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11247–11266, Toronto, Canada. Association for Computat… find
  7. Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023b. Investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processin… find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/trigg-2026-logic-level-evaluation-logical-table/