📄 BELA: Bot for English Language Acquisition

Summary

The paper describes BELA, a chatbot that acts as an English mentor for underprivileged Hindi-speaking students at the Indian non-profit Udayan Care, where volunteer mentors were in short supply. Users can write in English, Hindi or Hinglish. Non-English input goes through three steps: language identification with XLM-RoBERTa, transliteration into Devanagari, and Hindi–English machine translation from Salesken.ai. A BERT-based binary mode classifier then routes each message. Learning-related queries go to the retrieval-based Tutor Bot, which uses a BERT intent classifier with ten intents and per-intent helper functions. All other messages go to the Buddy Bot, a GPT-3 chit-chat partner restricted by its prompt to four topics. To personalise content, the author trained a TF-IDF CEFR level predictor and used it to label scraped reading passages and TED talk transcripts by difficulty. Both classifiers score above 98% test accuracy on small, partly augmented datasets. The work shows how off-the-shelf components can be assembled into a low-cost language tutor for a code-switching learner population. It also documents concrete failure modes.

Key points

• The mode classifier was trained on roughly 3040/380/380 utterances and reached 0.998 train and 0.987 test accuracy. Its negative examples were sampled from subreddits such as r/Food, r/Movies and r/AskReddit.
• The intent classifier (roughly 1520/190/190 utterances) reached 0.997 train and 0.995 test accuracy. It covers ten intents, including getWordMeaning, getTranslation, getCorrection and getAdvice. Its data was enlarged with back translation, Parrot Paraphraser and deliberately misspelled utterances.
• The CEFR predictor was trained on Adam Montgomerie's dataset of 1500 texts across 6 levels. It beats Montgomerie's baseline by 27.6% accuracy.
• The predictor was used to build levelled resources: cefr-levelled-passages.csv (from an ESL site) and cefr-levelled-tedtalks.csv (from the TED – Ultimate Dataset). A levelled word list was scraped from English Vocabulary Profile.
• Other components:
  • Word meanings and synonyms/antonyms come from the Oxford API.
  • Pronunciation uses Google TTS with the "co.in" domain to produce an Indian accent.
  • Reading-comprehension MCQs are generated by GPT-3.
• Known failure: the brand name "Jio" in a Hinglish query was translated as the Hindi verb for "live".

Interesting ideas

• I1. A coarse in-scope/out-of-scope gate runs before the fine-grained intent classifier, so anything off-task falls back to a separate handler instead of being forced into the wrong intent.
• I2. Negative training examples were cheaply harvested from topically unrelated public forums rather than hand-written.
• I3. Training data was augmented with the spelling errors learners typically make, making the classifier tolerant of noisy input.
• I4. A small difficulty predictor was trained once and then used to auto-label large unlabelled collections (passages, video transcripts) into a levelled resource.
• I5. The pipeline translated a brand name as an ordinary Hindi word, showing that named entities must be protected before translation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): A reader could first decide whether a question fits a supported task (test item, word problem, fable QA) and otherwise give an explicit, explained "outside my scope" path.
• I2 (learn): Unrelated text from clearly different genres could serve as negative controls when evaluating whether a frame or answer module fires only on the inputs it should.
• I3 (tokens): The tagger and lemmatizer could be stress-tested and trained on synthetically misspelled tokens so learner-style text still yields correct lemmas for the UD tree.
• I4 (back): Generated text could be scored by a similar level estimator so the generator can target, and verify, a requested reading difficulty.
• I5 (back): Translation through the meta-world should lock entities identified in the world of the text as names, so they pass through unchanged instead of being treated as lexical items.