Furu Wei · page 2
Papers
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting (2023 · 224 citations)
Large language models handle languages unevenly: English and other well-resourced languages get strong results, while low-resource languages lag. The authors propose cross-lingual-thought prompting (XLT), a training-free fix. It is a single English-language prompt template that works across tasks. - Visualizing and Understanding the Effectiveness of BERT (2019 · 222 citations)
Pre-training followed by fine-tuning works well across NLP tasks, but it has not been clear why it trains more easily and generalizes better. - Neural Latent Extractive Document Summarization (2018 · 220 citations)
Extractive summarizers are usually trained on sentence-level True/False labels. Most datasets supply only document–summary pairs, so these labels must be derived heuristically, for example by choosing the sentence subset with the highest ROUGE against the reference, and they may be suboptimal. Zhang, Lapata, Wei and… - Neural Open Information Extraction (2018 · 188 citations)
Most Open IE systems derive relation tuples from hand-written patterns applied to syntactic parser output. - Generative Representational Instruction Tuning (2024 · 172 citations)
LLMs are usually specialised. A model tuned for text generation produces weak embeddings, and a model tuned for embeddings loses its ability to generate. This paper introduces GRIT (generative representational instruction tuning), which trains one LLM on both kinds of data. The instruction format tells the model… - Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models (2024 · 156 citations)
Large language models handle many languages well even though they are trained mostly on English and without curated parallel data, and it is unclear how they do this internally. - MiniLLM: On-Policy Distillation of Large Language Models (2023 · 144 citations)
Large language models can be shrunk with knowledge distillation, but most existing objectives train a small student to match the teacher's whole output distribution, which is forward KL. - Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (2023 · 140 citations)
Large language models still make factual mistakes on knowledge-heavy tasks and stumble on problems that need slow, multi-step reasoning. - Controllable Natural Language Generation with Contrastive Prefixes (2022 · 130 citations)
The paper addresses steering a frozen GPT2 toward chosen text attributes, such as sentiment, topic or non-toxicity, without costly fine-tuning or slow decoding. - XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2022 · 127 citations)
Cross-lingual encoders such as XLM-R are typically trained with masked language modeling, sometimes extended with parallel-data objectives like TLM, and this demands very large compute budgets. - A Length-Extrapolatable Transformer (2023 · 118 citations)
Transformers trained on sequences of a fixed maximum length usually degrade sharply when evaluated on longer inputs, and retraining large pre-trained models for longer contexts is costly. - Learning to Retrieve In-Context Examples for Large Language Models (2024 · 106 citations)
Few-shot prompting of large language models works well only when the demonstrations are good, and picking them with BM25 or generic sentence embeddings is a rough heuristic. - SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval (2023 · 106 citations)
Dense retrievers usually start from general-purpose language models. However, models that score better on GLUE do not necessarily retrieve better. On MS-MARCO, BERT outperforms RoBERTa and ELECTRA. The authors attribute this to the [CLS] vector not being trained to capture a whole passage. SimLM is a self-supervised… - Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation (2023 · 103 citations)
Autoregressive (AR) decoding produces one token per step. This leaves GPUs underused and makes Transformer inference slow and costly. The authors revisit the draft-then-verify paradigm, borrowing the concept of speculative execution from computer architecture. A drafter proposes several future tokens cheaply, and the… - Chain-of-Dictionary Prompting Elicits Translation in Large Language Models (2024 · 81 citations)
Large language models translate fairly well without dedicated training, but they still handle low-resource languages poorly, often because they mistranslate rare words. - Calibrating LLM-Based Evaluator (2024 · 71 citations)
LLMs are increasingly used as reference-free judges of generated text, but their scores often diverge from human experts. - Language Models as Inductive Reasoners (2024 · 70 citations)
Classical computational induction, such as Inductive Logic Programming, works over symbolic first-order logic. - Instruction Pre-Training: Language Models are Supervised Multitask Learners (2024 · 59 citations)
Modern language models are usually pre-trained on raw text with next-token prediction, while supervised multitask learning is saved for post-training. - PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (2023 · 52 citations)
Pre-trained LLMs work reliably only up to the context length used during their training. The standard fix, fine-tuning on sequences of the desired target length (usually combined with position interpolation), is costly because attention cost grows quadratically with input size. For example, earlier work needed 32… - Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (2025 · 43 citations)
The paper asks whether making LLMs reason with ever-longer Chains of Thought can actually hurt accuracy, beyond simply wasting compute. - LongEmbed: Extending Embedding Models for Long Context Retrieval (2024 · 37 citations)
Most text embedding models handle only about 512 tokens, which limits retrieval over long sources such as books, meeting transcripts or long Wikipedia pages. - Autoregressive Speech Synthesis without Vector Quantization (2025 · 25 citations)
Codec-based zero-shot TTS systems such as VALL-E represent speech as vector-quantized tokens. These tokens lose fidelity compared with continuous features. Sampling from them can also produce silences or noise loops, and the systems need a two-pass AR+NAR decoder. MELLE removes quantization entirely: a single… - Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective (2025 · 23 citations)
Current LLM math solvers are usually tuned for a single reasoning style. Some use natural-language chains of thought, others use executable code, and others use formal proofs. As a result, a model strong at arithmetic word problems often fails at theorem proving, and the reverse. The authors propose… - MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark (2025 · 21 citations)
Popular multiple-choice benchmarks such as MMLU are public, and LLM training corpora are huge. As a result, models may have already seen the test items, which makes their scores unreliable. The authors show this directly: given only an MMLU question, some models reproduce the original answer options word for word.… - Preference Optimization for Reasoning with Pseudo Feedback (2024 · 15 citations)
Preference learning methods such as DPO and PPO help LLMs reason better in math and coding. However, they need reliable labels to separate good solutions from bad ones, and expert-verified reasoning data is scarce and costly to scale. The authors propose PFPO (Pseudo-Feedback Preference Optimization), which treats…
This page in Markdown → https://md.research.elamur.ai/free/en/authors/furu-wei/page/2/