Elamur Research⌕
☰
home / papers / 2017 / Identifying dialects with textual and acoustic cues

Identifying dialects with textual and acoustic cues

Paper · 2017 · Abualsoud Hanani, Aziz Qaroush and Stephen Taylor

Summary

This paper describes the Birzeit/Fitchburg team's (ahaqst) entries in two VarDial 2017 DSL shared tasks. The Arabic task (ADI) had five dialect classes. The Swiss-German task (GDI) had four. For Arabic, the organizers supplied ASR transcripts and 400-dimensional I-vectors, and the team also downloaded the raw .wav files. The Swiss-German data was text only, transcribed phonetically with the Dieth guidelines.

The team built several kinds of classifiers:

  • text classifiers, including per-dialect word n-gram language models scored by cross-entropy, a model that tiles the input with character strings seen in training, and Weka-based voting;
  • acoustic classifiers, including GMM-UBM, GMM tokenization and I-vector systems.

Their scores were merged with Focal multiclass logistic regression. On Arabic, combining text and audio gave the best result, a weighted F1 of 0.628. Audio alone did much better than text alone. On Swiss-German, the best run reached about 63% accuracy.

The authors also found a large gap between scores on held-out training lines and on the official test set. They attribute it to topic bias or to an inhomogeneous corpus, which is a useful warning about how evaluation splits are built.

Key points

  • ADI data: 14,000 training lines (1.7MB), 1524 development lines and 1492 test lines. The team ran three runs:

    RunInputAccuracyWeighted F1
    run2text only0.3230.314
    run3acoustic only0.5930.586
    run1text + acoustic0.6290.628

  • Team progress: the best ADI F1 rose from 0.426 in 2016 to 0.628 in 2017. The top team in 2017 (unibuckernel) reached 0.763.

  • GDI data: 14,478 lines in total, split into 13,032 for training and two 723-line development files, plus 3638 test lines.

    • The character-string entropy run (run2) scored 0.629 accuracy and 0.614 weighted F1.
    • The word-entropy run (run1) scored 0.562 accuracy.
    • On reserved training data the same two classifiers had scored 79% and 84%.

  • Character-string classifier: it collects all byte strings of up to 25 bytes from training. For each dialect, it covers the test segment with the longest known strings first and adds up their log-frequencies.

  • Acoustic fusion: the final acoustic system combines four subsystems with Focal linear logistic regression:

    • GMM-UBM with 2048 components;
    • GMM-tokenization bigram with 256 components;
    • unigram with 2048 components;
    • I-vector.

  • Baseline n-gram order: in the SVM baseline (svm multiclass), word n-grams longer than 3 made accuracy worse.

Interesting ideas

  • I1. Classify text by covering it greedily with the longest substrings seen in training per class and summing their log-frequencies. This ignores word boundaries entirely, so it can pick up spelling and pronunciation variants.
  • I2. To train fusion weights on the development set without leakage, cut it into two halves, fit on one, test on the other, then swap.
  • I3. When samples come from coherent conversations, rare topic words (such as a city name) can look like class markers even when they say nothing about the class. A random line-level holdout then overstates accuracy, here 84% on held-out lines versus 56% on test.
  • I4. Combine heterogeneous evidence at the level of scores, using a light logistic-regression layer over each subsystem's per-class scores. This let weak text cues still add to strong acoustic ones.
  • I5. Note how a transcription convention shapes the data: Arabic transcripts were normalized to standard spelling, while Swiss-German ones were written as pronounced. The same word therefore shows up as one form in one corpus and many forms in the other.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a similar longest-known-piece covering could score how to segment or normalize unknown and variant-spelled words before tagging, and the chosen pieces would show why.
  • I2 (learn): the same swap-halves scheme could calibrate how much each witness parser's vote counts in silver-data agreement without grading the weights on the data used to fit them.
  • I3 (learn): split evaluation by document or story rather than by sentence, and add negative controls that swap topic words, so that topical shortcuts are not counted as real skill.
  • I4 (learn): a small, inspectable fusion over the per-arc scores of the perceptron parser and gparse could replace plain majority voting and still show each witness's contribution.
  • I5 (tree): when converting or mixing treebanks of different UD dialects or registers, record each source's spelling-normalization policy so that vocabulary differences are not mistaken for grammatical ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations🎓Fitchburg State UniversityFitchburg · United States2 works · 39 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Abualsoud Hanani, Aziz Qaroush and Stephen Taylor. 2017. Identifying dialects with textual and acoustic cues. Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1211

@inproceedings{hanani2017identifying,
  title = {{Identifying dialects with textual and acoustic cues}},
  author = {Abualsoud Hanani, Aziz Qaroush and Stephen Taylor},
  booktitle = {Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)},
  year = {2017},
  doi = {10.18653/v1/w17-1211},
  url = {https://doi.org/10.18653/v1/w17-1211}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1). ↗
  2. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals. 2016. Automatic dialect detection in Arabic broadcast speech. In Proceedings of Interspeech 2016, pages 2934–2938.
  3. Abualsoud Hanani, Hanna Basha, Yasmeen Sharaf, and Stephen Taylor. 2015. Palestinian Arabic regional accent recognition. In The 8th International Conference on Speech Technology and Human-Computer Dialogue. find
  4. Fadi Biadsy, Julia Hirschberg, and Nizar Habash. 2009. Spoken arabic dialect identification using phonotactic modeling. In Proceedings of the EACL Workshop on Computational Approaches to Semitic Languages, pages 53–61.
  5. Abualsoud Hanani, Aziz Qaroush, and Stephen Taylor. 2016. Classifying ASR Transcriptions According to Arabic Dialect. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 126–134, Osaka, Japan.
  6. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC14). European Language Resources Association (ELRA), May.
  7. Radu Tudor Ionescu and Marius Popescu. 2016. UnibucKernel: An Approach for Arabic Dialect Identification Based on Multiple String Kernels. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages 135–144, Osaka, Japan.
  8. Pierre-Michel Bousquet, Anthony Larcher, Driss Matrouf, Jean-François Bonastre, and Oldrich Plchot. 2012. Variance-spectra based normalization for ivector standard and probabilistic linear discriminant analysis. In Odyssey: The Speaker and Language Recognition Workshop, pages 157–164. find
  9. Thorsten Joachims. 2008. Multi-class support vector machine. downloaded in January 2017 from http://www.cs.cornell.edu/people/ tj/svm_light/svm_multiclass.html. ↗
  10. Niko Brümmer. 2007. Focal multi-class: Toolkit for evaluation, fusion and calibration of multi-Shervin Malmasi and Marcos Zampieri. 2016. Arabic class recognition scores tutorial and user Dialect Identification in Speech Transcripts. In Pro-manual. downloaded in January 2017 from https://sites.google.com/site/nikobrummer/focalmulticlass. ceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3), pages Çağri Çöltekin and Taraka Rama. 2016. Discriminat-106–113, Osaka, Japan. ing Similar Languages with Linear SVMs and Neu-Shervin Malmasi, Eshrag Refaee, and Mark Dras. ral Networks. In Proceedings of the Third Work-2015. Arabic dialect identification using a parallel shop on NLP for Similar Languages, Varieties and multidialectal corpus. In Proceedings of the 14th Dialects (VarDial3), pages 15–24, Osaka, Japan. Conference of the Pacific Association for Computational Linguistics (PACLING 2015), pages 209–217, Najim Dehak, Patrick J. Kenny, Réda Dehak, Pierre Bali, Indonesia, May. ↗
  11. Dumouchel, and Pierre Ouellet. 2011. Front end factor analysis for speaker verification. IEEE Trans-Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, actions on Audio, Speech and Language Processing, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 19:788–798. 2016. Discriminating between similar languages and Arabic dialect identification: A report on the Eugen Dieth. 1986. Schwyzertütschi Dialäktschrift: third DSL shared task. In Proceedings of the 3rd Dieth-Schreibung. Sauerländer Verlage, Aarau, Workshop on Language Technology for Closely Re-Switzerland. lated Languages, Varieties and Dialects (VarDial), John W. Eaton et al. 2012. Gnu octave program for Osaka, Japan. scientific calculation. current version available from David Martı́nez, Oldrich Plchot, Lukás Burget, On-http://www.octave.org.drej Glembek, and Pavel Matejka. 2011. Language recognition in i-vectors space. In Interspeech, pages Heba ElFardy and Mona Diab. 2013. Sentence level 861–864. dialect identification in Arabic. In Proceedings of Maryam Najafian, Andrea DeMarco, Stephen Cox, and Martin Russell. 2014. Unsupervised model selection for recognition of regional accented speech. In Proceedings of Interspeech 2014. ↗
  12. Fred S. Richardson and William M. Campbell. 2008. Language recognition with discriminative keyword selection. In ICASSP’08, pages 4145–4148. find
  13. Tanja Samardzic, Yves Scherrer, and Elvira Glaser. 2016. ArchiMob–A corpus of spoken Swiss German. In Proceedings of the Language Resources and Evaluation (LREC), pages 4061–4066, Portoroz, Slovenia). find
  14. Ioannis Tsochantaridis, Thomas Hofmann, Thorsten Joachims, and Yasemin Altun. 2004. Support vector machine learning for interdependent and structured output spaces. In Proceedings of the Twenty-first International Conference on Machine Learning, ICML ’04, pages 104–, New York, NY, USA. ACM. find
  15. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic Online Commentary dataset: An annotated dataset of informal Arabic with high dialectal content. In Proceedings of ACL, pages 37–41.
  16. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  17. Marcos Zampieri, Shervin Malmasi, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, Jörg Tiedemann, Yves Scherrer, and Noëmi Aepli. 2017. Findings of the VarDial Evaluation Campaign 2017. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial), Valencia, Spain.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hanani-2017-identifying-dialects-textual-acoustic-cues/