🧑‍🔬 Arun Raja

Arun Raja works in natural language processing, and their papers here centre on large language models. One question is how such models can cope with tasks they were never directly trained on. The other is which units of text a model should read and produce in the first place. The methods range from assembling large collections of prompted training data to surveying long lines of research on tokenization and segmentation.

The most cited of these works is Multitask Prompted Training Enables Zero-Shot Task Generalization. It asks whether the zero-shot abilities seen in very large models can be produced on purpose by explicitly training on many tasks at once, rather than left to whatever task-like text a pretraining corpus happens to contain. To do this, the authors created PromptSource, which combines a templating language with an openly available crowdsourcing tool. They used it to recast many English datasets as natural-language inputs paired with targets, each dataset receiving several prompts phrased in different ways. The datasets were then sorted by their format into task families, so that entire families could be kept aside for evaluation. The second work, Between words and characters, is a survey written by the tokenization working group of the BigScience Workshop. It draws pre-neural and neural research into one account. It untangles related notions such as typographic tokens, word forms and subwords, and covers hybrid word–character models, subword methods like BPE, WordPiece and Unigram LM, and approaches built on characters, bytes or even pixels.

Both papers here date from 2021, so they mark a single moment in the work. That moment joins large-scale prompted training with a close look at how text is divided into units.

Raja is affiliated with the Institute for Infocomm Research in Singapore. Frequent co-authors include Colin Raffel, Manan Dey and Zaid Alyafeai. Other collaborators on these papers include Sabrina J. Mielke and Benoît Sagot.

Arun Raja has 2 papers in our library, in 2021, most of them on Tokenization & segmentation; often with Colin Raffel, Manan Dey, Zaid Alyafeai.