Elamur Research⌕
☰
home / papers / 2026 / EdgeFormer: Latency-Aware Collaborative Multi-Head Attention of Transformer Inference in Edge Networks

EdgeFormer: Latency-Aware Collaborative Multi-Head Attention of Transformer Inference in Edge Networks

Paper · 2026 · Yiming Yao, Jianwei Niu, Bin Dai et al.

Summary

Cloud-hosted Transformer models require users to send potentially sensitive data to third parties. Splitting inference across trusted edge devices keeps data local, but earlier systems partition the model statically and treat multi-head attention as one indivisible unit. EdgeFormer addresses both problems. First, it assigns contiguous ranges of Transformer blocks to devices, ranking devices by compute per unit of storage and filling them greedily. Second, its collaborative Multi-Head Attention (cMHA) spreads individual attention heads across local and neighbouring devices and drops redundant heads according to current resources. Heads are scored with LiScore, which combines attention-weight variance (a proxy for semantic value) with estimated local or remote latency. The resulting NP-hard allocation problem is approximated by a two-stage heuristic: greedy prioritization followed by iterative pruning. A similarity-based lookup reuses previously computed scores. On SuperGLUE tasks with GPT2 and OPT models, EdgeFormer is reported to be up to 2.01× faster than state-of-the-art baselines, with at most 1.06% accuracy loss. This makes private LLM inference on modest hardware more feasible.

Key points

  • A motivating analysis of GPT2-Small on ReCoRD (SuperGLUE) found that about 27.78% of heads were prunable for a short input and 31.94% for a long one. Most of the redundancy sat in lower layers.
  • The evaluation covers ReCoRD, BoolQ, MultiRC and WiC, using GPT2-L and OPT-1.3B/6.7B/13B. Robustness was tested under changing bandwidth and device capacity.
  • The OPT-1.3B ablation on ReCoRD reports these latencies:
    • full system: 2.8 s;
    • without head pruning: 3.8 s;
    • without dynamic scheduling: 4.4 s;
    • without both: 9.5 s, which the paper describes as a 3.4× speedup for the full system.
    • Accuracy dropped from 40.3 to 38.2 in the same comparison.
  • In the visualization, roughly 32% of heads are pruned in shallow blocks 1–6, while over 80% are kept in blocks 19–24. Retained heads in deeper layers go roughly 70% / 20% / 10% to the 20 / 13 / 7 TFLOPS devices.
  • Stated limitations are reliance on fast inter-device links, an assumption of trusted devices, and coordination overhead in volatile networks. Code is available at github.com/yymmine/EdgeFormer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🎓Beihang UniversityBeijing · China86 works · 9,097 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations🔬Institute of SoftwareBeijing · China27 works · 4,980 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksComputer Networks and Communications · 100 works

Cite

Cite as Yiming Yao, Jianwei Niu, Bin Dai and Tao Ren. 2026. EdgeFormer: Latency-Aware Collaborative Multi-Head Attention of Transformer Inference in Edge Networks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2007

@inproceedings{yao2026edgeformer,
  title = {{EdgeFormer: Latency-Aware Collaborative Multi-Head Attention of Transformer Inference in Edge Networks}},
  author = {Yiming Yao, Jianwei Niu, Bin Dai and Tao Ren},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2007},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2007}
}

References

41 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Fenglong Cai, Dong Yuan, Mengwei Xie, Wei He, Lanju Kong, Wei Guo, Yali Jiang, and Lizhen Cui. 2023. Paratra: A parallel transformer inference framework for concurrent service provision in edge computing. In 2023 IEEE International Conference on Web Services (ICWS), pages 258–268. find
  3. Huaiguang Cai, Zhi Zhou, and Qianyi Huang. 2024. Online resource allocation for edge intelligence with colocated model retraining and inference. In IEEE Conference on Computer Communications, pages 1900–1909. IEEE. find
  4. Quan Chen, Song Guo, Kaijia Wang, Wenchao Xu, Jing Li, Zhipeng Cai, Hong Gao, and Albert Y. Zomaya. 2024a. Towards real-time inference offloading with distributed edge computing: The framework and algorithms. IEEE Transactions on Mobile Computing, 23(7):7552–7571. find
  5. Yanming Chen, Tong Luo, and Weiwei Fang. 2024b. Edgeci: Distributed workload assignment and model partitioning for cnn inference on edge clusters. ACM Transactions on Internet Technology, 24(2):1–24. find
  6. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39.
  7. Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. 2023. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16091–16101. find
  8. Fateneh Golpayegani, Nanxi Chen, Nima Afraz, Eric Gyamfi, Abdollah Malekjafarian, Dominik Schäfer, and Christian Krupitzer. 2024. Adaptation in edge computing: a review on design principles and research challenges. ACM Transactions on Autonomous and Adaptive Systems, 19(3):1–43. find
  9. Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. find
  10. Daya Guo, Dejian Yang, Haowei Zhang, and Junxiao Song. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Xiaotian Guo, Quan Jiang, and Yixian Shen. 2024. Easter: Learning to split transformers at the edge robustly. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(11):3626– 3637. find
  12. Jiahui Hou, Huiqi Liu, Yunxin Liu, Yu Wang, Peng-Jun Wan, and Xiang-Yang Li. 2021. Model protection: Real-time privacy-preserving inference service for model privacy at the edge. IEEE Transactions on Dependable and Secure Computing, 19(6):4270–4284. find
  13. Xueyu Hou, Yongjie Guan, Tao Han, and Ning Zhang. 2022. Distredge: Speeding up convolutional neural network inference on distributed edge devices. In IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1097–1107. find
  14. Chenghao Hu and Baochun Li. 2024. When the edge meets transformers: Distributed inference with transformer models. In 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS), pages 82–92. find
  15. Hao Kong, Di Liu, and Shuo Huai. 2023. Edgecom-press: Coupling multidimensional model compression and dynamic inference for edgeai. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(12):4657–4670. find
  16. François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. 2021. Block pruning for faster transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10619–10629.
  17. Malgorzata Lazuka, Andreea Anghel, and Thomas Parnell. 2024. Llm-pilot: Characterize and optimize performance of your llm inference services. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1– 18. IEEE. find
  18. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In 26th ACM International Conference on Knowledge Discovery & Data Mining (SIGKDD), pages 3505–3506. find
  19. Juhyeon Lee, Insung Bahk, Hoseung Kim, Sinjin Jeong, Suyeon Lee, and Donghyun Min. 2024. An autonomous parallelization of transformer model inference on heterogeneous edge devices. In Proceedings of the 38th ACM International Conference on Supercomputing, ICS ’24, page 50–61, New York, NY, USA. Asso… find
  20. Reent Schlegel, Siddhartha Kumar, and Eirik Rosnes. 2022. Privacy-preserving coded mobile edge computing for low-latency distributed inference. IEEE Journal on Selected Areas in Communications, 40(3):788– 799. find
  21. Hui Li, Xiuhua Li, Qilin Fan, Qiang He, Xiaofei Wang, and Victor CM Leung. 2024. Distributed dnn inference with fine-grained model partitioning in mobile edge computing networks. IEEE Transactions on Mobile Computing, 23(10):9060–9074. find
  22. Jing Li, Weifa Liang, Yuchen Li, Zichuan Xu, Xiaohua Jia, and Song Guo. 2021. Throughput maximization of delay-aware dnn inference in edge computing by exploring dnn model partitioning and inference parallelism. IEEE Transactions on Mobile Computing, 22(5):3017–3030. find
  23. Nan Li, Alexandros Iosifidis, and Qi Zhang. 2022. Collaborative edge computing for distributed cnn inference acceleration using receptive field-based segmentation. Computer Networks, 214:109150. find
  24. Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. 2020. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1529–1538. find
  25. Zhiyan Liu, Qiao Lan, and Kaibin Huang. 2023a. Resource allocation for multiuser edge inference with batching and early exiting. IEEE Journal on Selected Areas in Communications, 41(4):1186–1200. find
  26. Ziming Liu, Shenggan Cheng, Haotian Zhou, and Yang You. 2023b. Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–13. find
  27. May Malka, Erez Farhan, Hai Morgenstern, and Nir Shlezinger. 2024. Decentralized low-latency collaborative inference via ensembles on the edge. IEEE Transactions on Wireless Communications. find
  28. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
  29. Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. IEEE. find
  30. Hugo Touvron, Louis Martin, and Kevin Stone. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  31. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808. ACL Anthology.
  32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32.
  33. Chaoqi Wang, Guodong Zhang, and Roger Grosse. 2020. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations. find
  34. Li Wang, Liang Li, Lianming Xu, Xian Peng, and Aiguo Fei. 2024. Failure-resilient distributed inference with model compression over heterogeneous edge devices. IEEE Transactions on Mobile Computing. find
  35. Yuanxin Wei, Shengyuan Ye, Jiazhi Jiang, Xu Chen, Dan Huang, Jiangsu Du, and Yutong Lu. 2024. Communication-efficient model parallelism for distributed in-situ transformer inference. In 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1–6. find
  36. Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate models. In 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1513–1528.
  37. Zichuan Xu, Liqian Zhao, Weifa Liang, Omer F. Rana, Pan Zhou, Qiufen Xia, Wenzheng Xu, and Guowei Wu. 2021. Energy-aware inference offloading for dnn-driven applications in mobile edge clouds. IEEE Transactions on Parallel and Distributed Systems, 32(4):799–814. find
  38. Shusen Yang, Zhanhua Zhang, Cong Zhao, Xin Song, Siyan Guo, and Hailiang Li. 2022. Cnnpc: End-edge-cloud collaborative cnn inference with joint model partition and compression. IEEE Transactions on Parallel and Distributed Systems, 33(12):4039–4056. find
  39. Shengyuan Ye, Jiangsu Du, and Liekang Zeng. 2024. Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference. In IEEE Conference on Computer Communications, pages 1001–1010. find
  40. Jeffrey Yu, Kartik Prabhu, Yonatan Urman, Robert M. Radway, Eric Han, and Priyanka Raina. 2024. 8-bit transformer inference and fine-tuning for edge accelerators. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, page 5–21, New York, NY, U… find
  41. Liekang Zeng and Xu Chen. 2020. Coedge: Cooperative dnn inference with adaptive workload partitioning over heterogeneous edge devices. IEEE/ACM Transactions on Networking, 29(2):595–608. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2026-edgeformer-latency-aware-collaborative-multi/