Elamur Research⌕
☰
home / papers / 2026 / Optimizing What We Trust: Reliability-Guided QUBO Selection of Multi-Agent Weak Framing Signals for Arabic Sentiment Prediction

Optimizing What We Trust: Reliability-Guided QUBO Selection of Multi-Agent Weak Framing Signals for Arabic Sentiment Prediction

Paper · 2026 · Rabab Alkhalifa

Summary

The paper asks how LLM-generated weak labels for Arabic framing can be made trustworthy enough to train on, without forcing every disagreement into one "true" label. Framing in Arabic social media, such as debates over women driving, mixes moral, religious, legal and security arguments, and expert frame annotations are scarce. The author builds a small multi-agent pipeline. Two LLM labelers each output a frame, a confidence score and an evidence-based rationale. A critic LLM compares the two rationales, picks a frame and gives a rubric score. A discriminator then turns these signals into a per-instance reliability value. Reliability does not change labels or weight the loss. It only drives a QUBO subset selection that rewards reliable items, penalizes near-duplicates and fixes a budget per frame. Compared with samples chosen to match the label distribution, the chosen subsets score higher on reliability and contain fewer near-duplicates. On an out-of-domain, gold-labeled women-driving sentiment test, the resulting framing features hold up against text-only baselines and beat noise and shuffle controls. The work is framed as a methodological case study, not a benchmark win.

Key points

  • Agents: Labeler A is Qwen-2.5 (3B), Labeler B is Mistral-7B, and the Critic is Gemma-2 (9B). All are instruction-tuned and 4-bit quantized. The critic's rubric score ranges from 0 to 8.
  • The QUBO objective combines a reliability reward (weight λ_conf), a TF–IDF cosine redundancy penalty (weight λ_red) and fixed per-frame budgets. Frames with too few candidates for their budget leave only one feasible selection.
  • Downstream tests use a BoW logistic regression throughout. The text-only baseline (S0) reaches macro-F1 0.624, and text plus QUBO framing features (SQ) scores slightly higher, though not significantly. Gaussian noise features (SN) drop to 0.604 and shuffled QUBO features (SQshuf) to 0.616, giving the ordering SQ > SQshuf > SN.
  • Framing-only models score below the text models but above chance. The QUBO-based variant (FQ) beats the DistMatch variant (FD).
  • The datasets, model versions and prompts are released at github.com/Rababalkhalifa/OptimizingWhatWeTrust. One stated limitation is that the QUBO cost grows quadratically with the candidate pool.

Interesting ideas

  • I1. Reliability scores are kept out of the labels and the loss; they act purely as a filter for choosing which silver examples enter training, so the trust estimate never contaminates the targets.
  • I2. A critic agent judges which of two rationales is better grounded in the text, instead of counting votes, which turns disagreement into an evidence-based decision.
  • I3. Subset selection is posed as a QUBO with a per-item reward, a pairwise similarity penalty and per-class budgets, because redundancy is a pairwise property that pointwise filters cannot control.
  • I4. The evaluation pairs the real features with Gaussian-noise and shuffled-feature controls, so even a modest gain can be shown to come from genuine structure rather than extra dimensions.
  • I5. Disagreement between annotators is treated as a sign of real interpretive ambiguity, not as an error to average away.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Agreement among witness parsers could score each silver tree for inclusion in a training set, while the trees themselves stay untouched.
  • I2 (learn): An LLM teacher tier could see two competing parses, each with its evidence, and pick the better-supported one with a rubric score, giving more than a tie-break.
  • I3 (learn): Selecting silver sentences under per-construction or per-relation quotas would stop the set from filling up with near-duplicate easy sentences.
  • I4 (learn): Any new frame- or tree-derived feature fed into a reader should be checked against noise and shuffle variants before it is kept.
  • I5 (frames): Frame or speech-act assignments where analyses persistently diverge, such as possible irony, could be flagged as ambiguous instead of being forced into a single reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Imam Abdulrahman Bin Faisal UniversityDammam · Saudi Arabia7 works · 95 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Rabab Alkhalifa. 2026. Optimizing What We Trust: Reliability-Guided QUBO Selection of Multi-Agent Weak Framing Signals for Arabic Sentiment Prediction. Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.abjadnlp-1.3

@inproceedings{alkhalifa2026optimizing,
  title = {{Optimizing What We Trust: Reliability-Guided QUBO Selection of Multi-Agent Weak Framing Signals for Arabic Sentiment Prediction}},
  author = {Rabab Alkhalifa},
  booktitle = {Proceedings of the 2nd Workshop on NLP for Languages Using Arabic Script},
  year = {2026},
  doi = {10.18653/v1/2026.abjadnlp-1.3},
  url = {https://doi.org/10.18653/v1/2026.abjadnlp-1.3}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nan-Jiang Jiang and Marie-Catherine De Marneffe. 2022. Investigating Reasons for Disagreement in Natural Language Inference. Transactions of the Association for Computational Linguistics, 10:1357– 1374. ↗
  2. Aseel Addawood, Amirah Alshamrani, Amal Alqahtani, Jana Diesner, and David Broniatowski. 2018. Women’s driving in saudi arabia–analyzing the discussion of a controversial topic on twitter. In 2018 International Conference on Social Computing, Behavioral-Cultural Modeling, and Prediction and Behavior Representation in Modeling and Simulation, BRiMS 2018. find
  3. Miaofeng Liu, Yan Song, Hongbin Zou, and Tong Zhang. 2019. Reinforced Training Data Selection for Domain Adaptation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1957–1968, Florence, Italy. Association for Computational Linguistics. ↗
  4. Maliheh Aramon, Gili Rosenberg, Elisabetta Valiante, Toshiyuki Miyazawa, Hirotaka Tamura, and Helmut G. Katzgraber. 2019. Physics-Inspired Optimization for Quadratic Unconstrained Problems Using a Digital Annealer. Frontiers in Physics, 7:48. ↗
  5. Riccardo Nembrini, Maurizio Ferrari Dacrema, and Paolo Cremonesi. 2021. Feature selection for recommender systems with quantum computing. Entropy, 23(8). ↗
  6. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Mag., 36(1):15–24. ↗
  7. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning - ICML ’05, pages 625–632, Bonn, Germany. ACM Press. find
  8. Stephen H. Bach, Bryan He, Alexander Ratner, and Christopher Ré. 2017. Learning the structure of generative models without labeled data. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 273– 282. JMLR.org. Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active Learning is a Strong Baseline for Data Subset Selection. ↗
  9. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. We Need to Consider Disagreement in Evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics.
  10. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics, 6:571–585. ↗
  11. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. ↗
  12. De Cheng, Tongliang Liu, Yixiong Ning, Nannan Wang, Bo Han, Gang Niu, Xinbo Gao, and Masashi Sugiyama. 2022. Instance-Dependent Label-Noise Learning with Manifold-Regularized Transition Matrix Estimation . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16609–16618, Los Alamitos, CA, USA. IEEE Computer Society. find
  13. Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings of the VLDB Endowment, 11(3):269–282. find
  14. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. ↗
  15. Ozan Sener and Silvio Savarese. 2018. ACTIVE LEARNING FOR CONVOLUTIONAL NEURAL NETWORKS: A CORE-SET APPROACH. ↗
  16. Burr Settles. 2009. Active learning literature survey.
  17. A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 28(1):20. ↗
  18. Muhammad Talha Shaikh, Muhammad Hamza, Syed Bilal Ali, Muhammad Rafi, and Sumaiyah Zahid. 2025. Feature Selection Using Quantum Annealing: A Mutual Information Based QUBO Approach. Working Notes of CLEF. find
  19. Benoit Frenay and Michel Verleysen. 2014. Classification in the presence of label noise: A survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5):845–869. ↗
  20. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 614–622, Las Vegas Nevada USA. ACM. find
  21. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org. The generation process avoids the reuse of user-authored content and is intended to capture realistic framing patterns rather than reproduce original tweets. After deduplication, the resulting corpus contains 2,733 sentences. ↗
  22. Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. 2023. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 34(11):8135–8153. ↗
  23. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. 2019. An Empirical Study of Example Forgetting during Deep Neural Network Learning. Preprint, arXiv:1812.05159. Framing Taxonomy Discovery. To identify a stable framing taxonomy, we sample approximately 80 sentences from the synthetic corpus and annotate them using the proposed multi-agent framework. The agents consistently converged on seven framing categories, which are fixed and enforced throughout the full weak supervision pipeline. ↗
  24. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72:1385– 1470. ↗
  25. Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. 2024. Diversity Measurement and Subset Selection for Instruction Tuning Datasets. Preprint, arXiv:2402.02318. Label Distribution. Applying the multi-agent framework to the full corpus yields a highly imbalanced label distribution, summarized in Table 4. The distribution is dominated by Identity/Group and Moral/Religious frames, with several minority categories below 4%. This imbalance motivates the need for frame-balanced subset selection. ↗
  26. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508. ↗
  27. Frame Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. Train/Test Split. We perform an 80/20 stratified train/test split, preserving both temporal and label proportions. The split statistics are reported in Table 5. Dataset Construction and Statistics This appendix provides additional details on dataset construction, label distributions, and data splits, omitted from the main paper for space reasons. A.1 50.2 29.0 11.5 3.3 2.9 2.5 0.6 *Table 4: Label distribution of the synthetic weak framing dataset.* Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jinghan Jia, Jiancheng Liu, Gaowen Liu, Mingyi Hong, Shiyu Chang, and Sijia Liu. 2023. Selectivity drives productivity: Efficient dataset pruning for enhanced transfer learning. In Thirty-seventh Conference on Neural Information Processing Systems.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alkhalifa-2026-optimizing-what-we-trust-reliability/