Elamur Research⌕
☰
home / papers / 2024 / TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers

TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers

Paper · 2024 · Savitha Viswanadh Kandala, Pramuka Medaranga and Ambuj Varshney

Summary

Large language models usually need too much memory and compute to run on mobile or edge hardware. Applications therefore call remote models over the network, which adds latency, depends on connectivity, and exposes sensitive sensor data. The authors go against the scaling trend and argue that very small models (roughly 30–120M parameters) can compete on narrow tasks if their pre-training and fine-tuning data are chosen carefully.

Focusing on embedded sensing, they build TinyLLM, an end-to-end pipeline. A user supplies a domain dataset, and the framework:

  • mixes it with curated general conversational text,
  • normalizes sensor readings into timestamped tabular form and tokenizes them,
  • pre-trains a GPT-2-style model,
  • adapts it with LoRA,
  • deploys it on an edge device.

Experiments on in-house and external IoT sensor datasets show that these compact models match, and sometimes beat, models with billions of parameters. They need only hours on a single GPU and generate tokens quickly on single-board computers. This makes private, offline, on-device language model inference practical for sensing applications.

Key points

  • Five custom models of 30M–124M parameters were trained from scratch on the GPT-2 architecture. For comparison, the framework was also used to fine-tune Phi 2, Phi 3, Llama 2 and Llama 3.
  • Pre-training a small model took only a few hours on one Nvidia H100. The authors contrast this with Llama 3.1 70B, which needed about 7 million H100 GPU hours.
  • LoRA fine-tuning needs few samples: about 440 examples were enough for high accuracy on the hand-gesture sensing task.
  • Deployment was tested on the Latte Panda Sigma, Orange Pi 5 and Orange Pi Zero 2W single-board computers.
    • The models ran on all three boards, and several instances could run concurrently with acceptable token rates.
    • Token throughput fell sharply on the weakest hardware.
  • The framework and resources are published at tinyllm.org (arXiv:2412.15304).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore221 works · 39,953 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksComputer Networks and Communications · 115 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Savitha Viswanadh Kandala, Pramuka Medaranga and Ambuj Varshney. 2024. TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers. arXiv:2412.15304.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.15304 · arXiv 2412.15304

@misc{kandala2024tinyllm,
  title = {{TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computers}},
  author = {Savitha Viswanadh Kandala, Pramuka Medaranga and Ambuj Varshney},
  year = {2024},
  doi = {10.48550/arxiv.2412.15304},
  eprint = {2412.15304},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.15304}
}

References

9 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, 2024.
  2. Anthropic AI. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2023. Accessed: 2024-06-24. find
  3. Beelink. Beelink EQ13 N100, 2024. Accessed: 2024-10-23. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  5. Gino Brunner, Darya Melnyk, Birkir Sigfússon, and Roger Wattenhofer. Swimming style recognition and lap counting using a smartwatch and deep learning. In Proceedings of the 2019 ACM International Symposium on Wearable Computers, ISWC ’19, page 23–31, New York, NY, USA, 2019. Association for Computin… find
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, 2023.
  7. DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  9. Linhao Dong, Shuang Xu, and Bo Xu. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5884–5888, 2018. [52] Gemini Team, Rohan Anil, et al. Gemini: A Family of Highly…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kandala-2024-tinyllm-framework-training-deploying-language/