📄 MTurk Crowdsourcing: A Viable Method for Rapid Discovery of Arabic Nicknames?

Summary

The paper asks whether Amazon Mechanical Turk (MTurk) can cheaply and quickly supply Arabic nicknames that existing Named Entity (NE) lexicons lack. Building such lists through native speakers, engineers and media mining is slow and expensive. The authors posted simple bilingual (Arabic/English) HITs asking workers to type a nickname they had heard, in Arabic script, along with where they heard it and their country. Each submission then passed three checks: Likert ratings from five MTurk workers and three experts, a Google search confirming the name appears online, and an exact-match lookup against the Database of Arabic Names (DAN) to keep only new entries. From 108 checked submissions, 39 valid nicknames absent from DAN survived. The study also varied pay. Higher rates sharply sped up collection, and posting a better-paid batch also sped up completion of an older, cheaper batch. The bilingual design drew over a third of workers from Arabic-speaking countries. MTurk participation from that region had previously been minimal.

Key points

• 332 nicknames were collected in total, but only the first 108 went through validation. 51 passed the majority vote (at least 3 of 5 workers and 2 of 3 experts rating ≥3 on a 5-point scale). 50 were found via Google, and 11 of those were already in DAN, leaving 39 new names.
• The reference lexicon was DAN Version 1.1, with 147,739 lines of names. It has no explicit nickname field but contains some common ones.
• Collection speed by pay per HIT:
  • $0.03 gave about 9.8 names/day.
  • $0.05 gave 25 names/day.
  • $0.25 gave 100 names in one day.
• Validation HITs were run twice, at $0.01 and $0.05. Once the higher-paid batch appeared, the cheaper batch also finished faster, which the authors call an advertising effect.
• Self-reported worker origin:
  • about 35% from Arabic-speaking countries (Morocco, Egypt, Lebanon, Jordan, UAE, Dubai);
  • 46% from India;
  • 13% from the U.S.;
  • 5% elsewhere.

Interesting ideas

• I1. Forcing answers into a specific script quietly filters out contributors who lack real familiarity with the language, with no explicit screening test.
• I2. Validation stacks independent signals: a crowd majority, a separate expert majority, and an external existence check (web occurrence), each with its own threshold.
• I3. Candidates are kept only if they are both valid and absent from the reference lexicon, so effort goes into genuinely new knowledge.
• I4. Throughput rose steeply with payment, and the authors suggest mixing price levels to balance speed against cost.
• I5. Bilingual, short-sentence instructions that include a definition of the target concept brought in the right contributors and guided non-experts.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver-data or LLM-teacher prompts could require a strict structured output format (e.g., valid CoNLL-U), so that careless or off-task outputs fail automatically before voting.
• I2 (learn): Agreement among witness parsers could be complemented by an independent external check, such as corpus attestation of a lemma or construction, before an item enters silver data.
• I3 (world): When expanding compiled knowledge from WordNet/Wikidata, harvested entries (names, aliases, nicknames for entity linking) should be diffed against existing resources, so only true additions are reviewed.
• I4 (learn): LLM teacher tiers could be scheduled the same way, with cheap tiers running continuously and expensive tiers added in bursts when a data batch must be finished quickly.
• I5 (learn): Annotation or teacher prompts for frames and roles should carry a brief plain definition of each label, which reduces inconsistent labelling by non-expert or model annotators.