Yejin Choi · page 3
Papers
- NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics (2022 · 140 citations)
Left-to-right decoders such as beam search and top-k sampling pick each token based only on the prefix generated so far. - GO FIGURE: A Meta Evaluation of Factuality in Summarization (2021 · 130 citations)
Neural summarizers often produce fluent text that misstates or invents facts. Many new automatic metrics aim to detect such errors, but there has been no systematic way to check whether these metrics actually work. GO FIGURE is a meta-evaluation framework that tests factuality metrics against five requirements… - Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest (2023 · 112 citations)
The paper asks whether today's AI systems grasp humor or merely produce joke-like text. To probe this, the authors build three benchmarks from 14 years of the weekly New Yorker Cartoon Caption Contest, each harder than the last. The first is matching : pick the caption actually written for a cartoon out of five… - SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks (2023 · 110 citations)
Agents in text-based simulated environments struggle with long, multi-step tasks. They must plan over many steps, remember context, break goals into subgoals and recover when something goes wrong. Imitation-trained models pick up action patterns well but plan and recover poorly. Prompted LLMs plan well but have… - We're Afraid Language Models Aren't Modeling Ambiguity (2023 · 110 citations)
The paper asks whether pretrained language models can notice when a sentence has more than one plausible meaning and separate those meanings. - NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints (2021 · 110 citations)
Large pretrained language models finetuned on task data still often ignore lexical constraints. A recipe generator, for example, may skip listed ingredients or invent new ones, and adding much more finetuning data barely helps. The authors introduce NeuroLogic Decoding, an inference-time algorithm that works with… - WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (2024 · 105 citations)
Existing automatic LLM benchmarks poorly reflect what real users actually ask. MT-bench is tiny, AlpacaEval is dominated by easy information-seeking prompts, and ArenaHard leans heavily toward coding. WildBench addresses this by drawing hard, naturally distributed tasks from WildChat, a log of roughly one million… - WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models (2024 · 92 citations)
Frontier LLMs can still be jailbroken, and open safety-training data is scarce, since even open-weight models rarely publish theirs. - It's not Rocket Science: Interpreting Figurative Language in Narratives (2022 · 91 citations)
Transformer-based language models handle literal text well but often fail on figurative expressions. Idioms and similes are hard because their meaning cannot be built from the meanings of their parts. The authors frame this as a story-continuation problem, following the Story Cloze setup. They took short passages… - Do Membership Inference Attacks Work on Large Language Models? (2024 · 87 citations)
This paper asks whether standard membership inference attacks (MIAs) can tell if a given text was in an LLM's pre-training data. - Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020 · 86 citations)
Abductive reasoning asks what plausibly happened between an observed past and an observed future. Counterfactual reasoning asks how a story's later events should change when an earlier event is altered. Both require a model to condition on context on both sides of the text it generates. Standard left-to-right… - Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (2022 · 82 citations)
Commonsense question answering often improves when a model receives extra background knowledge alongside the question. - OpinionFinder: A System for Subjectivity Analysis (2005 · 78 citations)
OpinionFinder is a demonstration system that automatically detects subjective language in English text, meaning opinions, emotions, speculations and other private states. - Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker (2023 · 76 citations)
Large language models still handle theory of mind (ToM) poorly. They struggle to track what each character in a story believes, especially when a belief is false, and what one character assumes about another's beliefs. Earlier fixes trained models on templated benchmarks, and those models tend to exploit dataset… - Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaboration (2024 · 68 citations)
Standard alignment tends to push LLMs toward one averaged human preference. This leaves the diverse values of different cultures, demographics and communities poorly represented. Existing fixes usually require retraining or weight access, so they cannot be used with proprietary black-box models. Patching a newly… - Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens (2024 · 62 citations)
Classical n-gram language models are usually limited to small n (often 5) and modest data, because count tables grow explosively with n. - Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (2024 · 50 citations)
Preference-based training is now a standard final stage for language models. Published setups, however, combine different datasets, algorithms and evaluation suites, so it is unclear which ingredient actually drives the gains. The authors divide the pipeline into four components: preference data, the learning… - Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding (2023 · 46 citations)
Language models aligned with PPO are usually deployed using only the policy network. The value network that PPO trains alongside it is typically thrown away. The authors argue that this byproduct is well suited to scoring candidates during guided decoding. It is trained to estimate the expected return of unfinished… - CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting (2024 · 40 citations)
The paper asks whether large language models know enough about the world's cultures, and whether they portray them fairly when generating free text. - CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs' Cultural Knowledge Through Human-AI Red-Teaming (2025 · 38 citations)
Measuring how well language models understand everyday cultural practice worldwide is difficult. Static hand-written test sets soon become too easy. Benchmarks built from web pages, value surveys or synthetic generation may overlap with training data or carry over model biases. - Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing (2023 · 36 citations)
Small but capable models for paraphrase generation and sentence summarization usually need one of three things: human-labeled data, noisy surrogates such as back-translation, or costly distillation from huge LLMs like GPT3 or ChatGPT. - PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning (2023 · 25 citations)
The paper asks whether compact language models can handle procedural planning, meaning turning a high-level goal into an ordered list of steps. - Language Model Alignment in Multilingual Trolley Problems (2024 · 23 citations)
Most moral-alignment benchmarks for LLMs are English-only and reflect mainly US respondents, so it is unclear whether models' moral choices match the preferences of speakers of other languages. - HALoGEN: Fantastic LLM Hallucinations and Where to Find Them (2025 · 22 citations)
Large language models often produce fluent but false content, and checking their outputs by hand is too slow and costly to do at scale. - Can Language Models Reason about Individualistic Human Values and Preferences? (2025 · 21 citations)
The paper argues that pluralistic alignment usually sorts people into predefined groups such as demographics, personality, or culture.
This page in Markdown → https://md.research.elamur.ai/free/en/authors/yejin-choi/page/3/