Elamur Research⌕
☰
home / papers / 2026 / Know What You See: Grounded localization of product components

Know What You See: Grounded localization of product components

Paper · 2026 · Manan Soni, Abinesh Kanagarajan and Shyam Mohan

Summary

Many practical product-related tasks require locating individual product parts rather than the whole item. Examples include checking whether a listing really includes a charger, searching by component, and spotting missing safety guards. Zero-shot open-vocabulary detectors treat parts as an unrelated flat list. As a result, they often place a part on background objects or report parts the product does not have. The authors (Amazon) propose KWYS (Know What You See), which works in three steps:

  1. An LLM turns product documentation (PDF manuals or web descriptions) into a containment-based component hierarchy.
  2. An off-the-shelf detector proposes boxes for every component.
  3. A top-down verification step keeps a child box only if it falls largely inside its parent's box.

No retraining or annotation is needed. Evaluation uses 1,000 manually annotated images of 250 products. Hierarchical verification improves mAP@50 for every detector backbone tested, and structured extraction yields fewer hallucinated components in the taxonomy. The approach is detector-agnostic, which makes component grounding feasible across large catalogs.

Key points

  • Dataset: It pairs manuals from the e-manuals corpus with Flipkart product images. It covers 5 categories (Household Appliances, Multimedia, Furniture, Fitness, Tools), with up to 4 images per product. Ground-truth boxes were annotated by experts.
  • Backbones: Taxonomies are built with Claude Haiku. Detection is tested with Grounding DINO (Tiny/Base) and OWL-ViT v2. Verification adds 18.6% on average, and the default Grounding DINO (Base) gains 11%.
  • Verification rule: A child box is retained if its Intersection-over-Child score with the parent box is at least δ. Performance peaks at δ = 0.8 (mAP@50 0.56). If an intermediate part is not detected, its children are searched within the parent's region instead.
  • Hallucination: Structured extraction lowers the Taxonomy Hallucination Rate to about 5%, compared with roughly 30% for a standalone LLM and 20% for RAG. This corresponds to the reported 25% reduction.
  • Knowledge source: User manuals yield much richer taxonomies than web descriptions: 51.9 vs 2.4 components on average, with depth 2.5 vs 1.3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Manan Soni, Abinesh Kanagarajan and Shyam Mohan. 2026. Know What You See: Grounded localization of product components. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.75

@inproceedings{soni2026know,
  title = {{Know What You See: Grounded localization of product components}},
  author = {Manan Soni, Abinesh Kanagarajan and Shyam Mohan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.75},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.75}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Prakash Chandra Chhipa, Kanjar De, Meenakshi Subhash Chippa, Rajkumar Saini, and Marcus Liwicki. 2024. Open-vocabulary object detectors: Robustness challenges under distribution shifts. arXiv preprint arXiv:2405.14874. — [arXiv:2405.14874](https://arxiv.org/abs/2405.14874) find
  2. Jiahui Cui, Zhongcheng Wu, and Xinkuang Wang. 2024. Vehicle appearance parts identification based on instance segmentation algorithms. In Proceedings of the International Conference on Algorithms, Software Engineering, and Network Security, ASENS ’24, page 306–311, New York, NY, USA. Association for… find
  3. Jing Li, Jinan Gu, Zedong Huang, and Jia Wen. 2019. Application research of improved yolo v3 algorithm in pcb electronic component detection. Applied Sciences, 9:3750. find
  4. Abhilash Nandy, Soumya Sharma, Shubham Maddhashiya, Kapil Sachdeva, Pawan Goyal, and NIloy Ganguly. 2021. Question answering over electronic devices: A new benchmark dataset and a multi-task learning based QA framework. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages… find
  5. Kitsuchart Pasupa, Phongsathorn Kittiworapanya, Na-pasin Hongngern, and Kuntpong Woraratpanya. 2021. Evaluation of deep learning algorithms for semantic segmentation of car parts. Complex Intelligent Systems, 8. find
  6. PromptCloud. 2017. Flipkart products. https://www.kaggle.com/datasets/PromptCloudHQ/ flipkart-products. Accessed: 2026; Dataset with 20k Flipkart product listings including descriptions and images. ↗
  7. Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, and 1 others. 2024. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300. — [arXiv:2405.10300](https://arxiv.org/abs/2405.10300) find
  8. Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. 2022. Glipv2: unifying localization and vl understanding. In 36th Conf. Neural Inf. Process. Syst. NeurIPS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soni-2026-know-what-you-see-grounded/