Elamur Research⌕
☰
home / papers / 2023 / Consonant is all you need: a compact representation of English text for efficient NLP

Consonant is all you need: a compact representation of English text for efficient NLP

Paper · 2023 · Maged S. Al-shaibani and Irfan Ahmad

Summary

The authors ask whether English NLP models need vowels at all. Abjad scripts write only consonants, so they test two reduced views of English. One deletes every vowel letter (consonants-only). The other replaces each vowel with a single placeholder symbol, '#' (masked-vowels). They compare both views with normal text in several ways:

  • corpus statistics: vocabulary size and entropy;
  • language models: KenLM n-grams, an LSTM and a transformer, trained on Wikitext-2;
  • downstream tasks: classification (IMDB, AGNews), POS tagging (EWT), NER (ConLLpp) and NMT (Multi30K).

They also train a model that puts the vowels back, evaluated on AGNews. Both reduced views perform close to standard text while shrinking the vocabulary, which shrinks embedding layers, total parameters and training time. Restoring vowels works better from the masked form than from the consonants-only form. The work matters because it changes the input before embedding, rather than compressing the model afterwards. That makes it compatible with other efficiency methods and largely independent of model architecture.

Key points

  • Two schemes. Consonants-only drops A, E, I, O, U in either case, so vowel-only words such as "a" and "I" disappear. Masked-vowels keeps word positions intact, which sequence labeling tasks like POS tagging and NER require.
  • Model size (Wikitext LM). The parameter count falls from 25.5M (standard) to 21.5M (consonants-only) and 23.8M (masked-vowels). On disk this is 101 MB vs 86 MB vs 95 MB.
  • Training speed. One LM epoch takes 82.14 s on standard text, 67.55 s with consonants-only and 77.07 s with masked-vowels.
  • AGNews vocabulary. It shrinks from 79,037 types to 54,596 (consonants-only) and 67,982 (masked). The share of single-occurrence types drops from 34.60% to 31.66% and 33.33%.
  • Vowel restoration. It is scored with WER, CER and a dedicated vowel-error-rate (VER). Most errors come from rarely seen nouns, and the consonants-only variant also struggles with abbreviations, letter case and placing vowels correctly.

Interesting ideas

  • I1. A lossy view that keeps one placeholder per removed character still aligns token-for-token with the original, so per-word labels can be carried over unchanged.
  • I2. The authors compute entropy of a candidate representation as a cheap, training-free signal of whether it will be about as learnable as the original.
  • I3. They measure restoration with an extra metric that counts errors only on the deleted part, here vowels, alongside overall WER/CER. This shows exactly where reconstruction fails.
  • I4. Efficiency is gained by changing the input before embedding rather than by shrinking the model. The authors note this can be stacked with quantization or distillation.
  • I5. Their error analysis traces restoration failures to rare nouns, showing that the losses concentrate in the low-frequency tail.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a vowel-masked "skeleton" of an unknown word could act as a backoff key for the tagger and lemmatizer, grouping spelling variants and typos with known forms.
  • I2 (learn): before retraining parsers, compare the entropy of alternative tokenizations or UD dialect label sets to discard options that lose too much information.
  • I3 (back): the text → graph → text compressor could report an error rate restricted to content the graph omitted, separating generation mistakes from compression loss.
  • I4 (tree): the gparse transformer parser could be trained on a reduced-vocabulary input to cut embedding size, with LAS checked against the standard-text model to quantify any loss.
  • I5 (learn): evaluation of any normalization step should be broken down by word frequency and POS, so that negative controls catch regressions hidden in rare items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia52 works · 4,666 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Maged S. Al-shaibani and Irfan Ahmad. 2023. Consonant is all you need: a compact representation of English text for efficient NLP. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.775

@inproceedings{alshaibani2023consonant,
  title = {{Consonant is all you need: a compact representation of English text for efficient NLP}},
  author = {Maged S. Al-shaibani and Irfan Ahmad},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.775},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.775}
}

References

35 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zaid Alyafeai, Maged S Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2022. Evaluating various tokenizers for arabic text classification. Neural Processing Letters, pages 1–23. ↗
  2. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166. ↗
  3. Bentrevett. bentrevett/multi30k · datasets at hugging face. https://huggingface.co/datasets/ bentrevett/multi30k. (Accessed on 10/16/2023). ↗
  4. James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2017. Quasi-recurrent neural networks. In International Conference on Learning Representations. ↗
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Alexis Conneau, Holger Schwenk, Yann Le Cun, and Löc Barrault. 2017. Very deep convolutional networks for text classification. In 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, pages 1107–1116. Association for Computational Linguistics (ACL).
  7. Peter T Daniels. 2013. The arabic writing system. In Jonathan Owens, editor, The Oxford handbook of Arabic linguistics, chapter 18, pages 422–431. Oxford University Press, New York. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  9. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. Characterbert: Reconciling elmo and bert for word-level open-vocabulary representations from characters. In International Conference on Computational Linguistics, pages 6903–6915. ↗
  10. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74. Association for Computational Linguistics.
  11. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
  12. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29.
  13. Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630. ↗
  14. Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197.
  15. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696.
  16. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations. ↗
  17. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  18. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  19. Edward Loper and Steven Bird. 2002. Nltk: the natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1, pages 63–70. ↗
  20. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. CharBERT: Character-aware pre-trained language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 39–50, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  21. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  22. Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70–74, Online. Association for Computational Linguistics. find
  23. Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in nlp. arXiv preprint arXiv:2112.10508. ↗
  24. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. ↗
  25. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. ↗
  26. Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. 2019. Crossweigh: Training named entity tagger from imperfect annotations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5157–5166. ↗
  27. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. ↗
  28. George Kingsley Zipf. 2016. Human behavior and the principle of least effort: An introduction to human ecology. Ravenio Books. find
  29. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. ↗
  30. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163, Valencia, Spain. Association for Computational Linguistics.
  31. Vikas Raunak, Vivek Gupta, and Florian Metze. 2019. Effective dimensionality reduction for word embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 235–243. find
  32. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  33. Claude E Shannon. 1951. Prediction and entropy of printed english. Bell system technical journal, 30(1):50–64. ↗
  34. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. find
  35. Merity Stephen, Xiong Caiming, Bradbury James, and Richard Socher. 2017. Pointer sentinel mixture models. Proceedings of ICLR. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shaibani-2023-consonant-is-all-you-need/