📄 POaaS: Minimal-Edit Prompt Optimization as a Service to Lift Accuracy and Cut Hallucinations on On-Device sLLMs

Summary

Small language models running on-device often receive flawed prompts with typos, vague wording or missing context, and these flaws can push them toward wrong answers and hallucinations. Automatic prompt optimization (APO) methods such as OPRO, PromptWizard and EvoPrompt were built for large cloud LLMs. They run expensive searches and produce long prompts, which can hurt a small model that must act as both optimizer and solver.

The authors propose POaaS, a service layer placed in front of a frozen sLLM. Cheap CPU heuristics score each prompt. Well-formed prompts pass through unchanged, and the rest are routed to three LoRA specialists: Cleaner, Paraphraser and Fact-Adder. A merger keeps only edits that stay within drift and length limits.

The approach was tested on Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct across reasoning benchmarks (BBH, GSM8K, CommonsenseQA) and factuality benchmarks (HaluEval, HalluLens, FActScore), using both clean and corrupted inputs. POaaS raised accuracy and factuality, while the APO baselines lowered them. This suggests that light, per-query conservative editing suits edge deployment better than heavy search.

Key points

• Specialists and training data: Cleaner is tuned on JFLEG, Paraphraser on PAWS and QQP, and Fact-Adder on Wikipedia/Wikidata-derived corpora. Fact-Adder adds at most 3 facts within 120 tokens, and guard models reject edits that are unfaithful.
• Robustness to corrupted inputs: With 5–15% token deletion or mixup, POaaS recovers up to +7.4 points over No Optimization. For example, at 15% mixup on the 3B model, accuracy rises from 48.8% to 54.6%. PromptWizard degrades the most, losing about 20% at 15% deletion.
• Cost per query: POaaS adds about 0.95 s, about 1.4 specialist calls and about 48 prompt tokens. The APO baselines need 336–602 s of offline search, 96–172 LLM calls and 2,840–4,180 extra tokens.
• Ablation: Averaged over six benchmarks, full POaaS on Llama-3.2-3B scores 66.0 vs 63.7 on clean inputs and 52.8 vs 45.8 at 10% deletion. The Cleaner drives most of the robustness gain, and the Fact-Adder mainly helps on clean queries.
• Limitations: The system works only in English, its thresholds were tuned empirically, and it uses a small set of three agents.