Elamur Research⌕
☰
home / papers / 2026 / AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation

AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation

Paper · 2026 · Rui Qian, Chuanhang Deng, Qiang Huang et al.

Summary

Reasoning segmentation asks a model to produce a pixel mask for a target described only indirectly, for example by its function rather than its name. Earlier systems in the LISA line pack all of this into one <SEG> token embedding. That single vector has to carry both the meaning of the target and its location. AnchorSeg separates these two roles.

The underlying LMM (LLaVA, fine-tuned with LoRA) generates an ordered "query bank":

  • several latent reasoning tokens that hold intermediate semantic states;
  • a final <SEG> anchor token that handles localization.

The anchor is compared against image tokens to form a spatial prior, which is added to SAM's visual features. The contextual queries then modulate the result. SAM's decoder receives the whole bank as its prompt. A bidirectional loss, Token–Mask Cycle Consistency (TMCC), aligns the coarse token-level responses with the full-resolution ground-truth masks.

The method sets a new state of the art on the ReasonSeg test set and is also evaluated on RefCOCO(+/g). This suggests that making the reasoning-versus-grounding structure explicit improves language-driven segmentation, at a modest compute cost.

Key points

  • Main result: On the ReasonSeg test set, AnchorSeg reaches 67.7% gIoU and 68.1% cIoU. Code and models are released at github.com/rui-qian/AnchorSeg.
  • Component ablation (cIoU):
    • adding the anchor-derived spatial prior alone gives 71.6%;
    • adding TMCC on top gives 74.0%;
    • also feeding the contextual queries to the decoder gives 75.9%.
  • TMCC ablation:
    • token-to-mask loss only: 67.4% gIoU / 73.8% cIoU;
    • mask-to-token loss only: drops to 64.4% gIoU;
    • both directions combined: 68.3% gIoU / 75.9% cIoU.
  • Cost (single A100-40GB):
    • training takes 2.94 s per iteration and 29.89 GB, versus 1.26 s and 23.68 GB for LISA;
    • inference runs at 4.00 FPS, versus 4.68 FPS for LISA and 9.24 FPS for PixelLM;
    • 306.29M of 7.77B parameters are trainable (3.94%).
  • Stated limitations: the number K of latent tokens is fixed regardless of query complexity, performance depends on the quality of the LMM, and the approach has not been tested on tasks other than segmentation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Rui Qian, Chuanhang Deng, Qiang Huang, Jian Xiong, Mingxuan Li, Yingbo Zhou, Wei Zhai, Jintao Chen and Dejing Dou. 2026. AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.938

@inproceedings{qian2026anchorseg,
  title = {{AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation}},
  author = {Rui Qian, Chuanhang Deng, Qiang Huang, Jian Xiong, Mingxuan Li, Yingbo Zhou, Wei Zhai, Jintao Chen and Dejing Dou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.938},
  url = {https://doi.org/10.18653/v1/2026.acl-long.938}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy… — Flamingo: a Visual Language Model for Few-Shot Learning
  2. Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. 2024. One token to seg them all: Language instructed reasoning segmentation in videos. In NeurIPS. find
  3. Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2018. Coco-stuff: Thing and stuff classes in context. In CVPR. find
  4. Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. 2014. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR. find
  5. Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-next: What else influences visual instruction tuning beyond data? find
  6. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML.
  7. Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR. find
  8. Tianming Liang, Qirui Du, Jian-Fang Hu, Haichao Jiang, Zicheng Lin, and Wei-Shi Zheng. 2026. Seg-ReSearch: Segmentation with interleaved reasoning and external search. In arXiv. find
  9. Chang Liu, Henghui Ding, and Xudong Jiang. 2023a. GRES: Generalized referring expression segmentation. In CVPR. find
  10. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. In NeurIPS.
  11. Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. 2021. Vision-language transformer and query generation for referring segmentation. In ICCV. find
  12. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, and et al Su, Hang. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV.
  13. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In ICLR.
  14. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. find
  15. Yutao Hu, Qixiong Wang, Wenqi Shao, Enze Xie, Zhenguo Li, Jungong Han, and Ping Luo. 2023. Beyond one-to-one: Rethinking the referring image segmentation. In ICCV. find
  16. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o system card. In arXiv.
  17. Yi Lu, Jiawang Cao, Yongliang Wu, Bozheng Li, Licheng Tang, Yangguang Ji, Chong Wu, Jay Wu, and Wenbo Zhu. 2025. RSVP: Reasoning segmentation via visual prompting and multi-modal chain-of-thought. In ACL. find
  18. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, and et al. Yuille, Alan L. 2016. Generation and comprehension of unambiguous object descriptions. In CVPR.
  19. Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, and Tieniu Tan. 2021. Locate then segment: A strong pipeline for referring image segmentation. In CVPR. find
  20. Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. 2017. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV. find
  21. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP.
  22. OpenAI. 2025. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. ↗
  23. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, et al. 2023. Segment anything. In ICCV.
  24. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In CVPR. find
  25. Rui Qian, Xin Yin, and Dejing Dou. 2025. Reasoning to attend: Try to understand how \<SEG\> token works. In CVPR. find
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
  27. Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. 2023. PACO: Parts and attributes of common objects. In CVPR. find
  28. Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. GlaMM: Pixel grounding large multimodal model. In CVPR. find
  29. Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2025. SAM 2: Segment anything in images and videos. In ICML. find
  30. Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. 2024. PixelLM: Pixel reasoning with large multimodal model. In CVPR. find
  31. Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2023. VisionLLM: large language model is also an open-ended decoder for vision-centric tasks. In NeurIPS. find
  32. Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. 2023. LISA++: An improved baseline for reasoning segmentation with large language model. In arXiv. find
  33. Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, and et al. Zhao, Hengshuang. 2022. LAVT: Language-aware vision transformer for referring image segmentation. In CVPR. find
  34. Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018. MAttNet: Modular attention network for referring expression comprehension. In CVPR. find
  35. Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, et al. 2024. GPT4RoI: Instruction tuning large language model on region-of-interest. In ECCV. find
  36. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In CVPR. find
  37. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In ICLR.
  38. Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. CRIS: Clip-driven referring image segmentation. In CVPR. find
  39. Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. 2023. Generalized decoding for pixel, image, and language. In CVPR. find
  40. Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Jie Hu, Dengjie Li, Zheng Zhao, and Yujiu Yang. 2025. HyperSeg: Hybrid segmentation assistant with fine-grained visual perceiver. In CVPR. find
  41. Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, and et al. Gao, Jianfeng. 2024. Segment everything everywhere all at once. In NeurIPS. find
  42. Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, and Dacheng Tao. 2024a. Towards robust referring image segmentation. In TIP. find
  43. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2024b. Next-GPT: Any-to-any multimodal llm. In ICML. find
  44. Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph E Gonzalez, and Trevor Darrell. 2024c. See, say, and segment: Teaching lmms to overcome false premises. In CVPR. find
  45. Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. 2024. GSVA: Generalized segmentation via multimodal large language models. In CVPR. find
  46. Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. 2024. u-LLaVA: Unifying multi-modal tasks via large language model. In ECAI. find
  47. Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. 2024. VISA: Reasoning video object segmentation via large language models. In ECCV. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2026-anchorseg-language-grounded-query-banks/