Elamur Research⌕
☰
home / papers / 2026 / Re-evaluating the Word Token for Bilingual Speech Processing: The Case for Intonation Units

Re-evaluating the Word Token for Bilingual Speech Processing: The Case for Intonation Units

Paper · 2026 · Rebecca Pattichis, Dora LaCasse and Rena Torres Cacoullos

Summary

Code-switching (CS) metrics in NLP usually count every gap between two words as a possible switch site. The authors argue that this assumption does not fit spoken bilingual data. In speech, multi-word switches cluster at the edges of Intonation Units (IUs) and rarely occur inside them. IUs are prosodic chunks marked by cues such as pitch reset, final lengthening and pauses.

Building on their earlier IU-based work, the authors define a single IU-level version of the I-Index (Guzmán et al. 2017) that combines CS rates at IU boundaries with CS rates within IUs. They compute word-based and IU-based variants for ten bilingual datasets with multi-word CS, which range from balanced to heavily skewed language proportions. They then compare how each variant groups the datasets using hierarchical clustering.

Treating all words as eligible switch points inflates the denominator. This squeezes word-based scores into a narrow band that the usual two-significant-figure reporting cannot separate. The IU variant spreads the values out and achieves better cluster similarity. For data without prosodic transcription, the authors suggest dividing word counts by mean IU length.

Key points

  • The metric adapted is the I-Index (CS probability). Its IU version merges boundary and within-IU switches. Most switches fall at boundaries: in one dataset (recording 23 of the NMSEB corpus), 490 switches occurred at IU boundaries versus 76 within IUs.
  • Across the ten datasets, IU-based I-Index values span about 0.033–0.221, while word-based values span only about 0.010–0.058. At two significant figures, several word-based scores become indistinguishable, for example two datasets that both score 0.0228.
  • The proposed interim normalization uses mean IU length, which is roughly 3.3 words for the Spanish data and 3.9 for the English data (35,113 words / 10,634 IUs and 42,835 / 11,032).
  • All examples are drawn from NMSEB, a Spanish-English bilingual corpus recorded in New Mexico, whose transcription divides speech into IUs according to the Du Bois et al. (1993) conventions. Lone other-language items are tagged separately from multi-word CS.
  • The paper motivates the work with inconsistent CS counts reported for the same corpus (5.8% vs. 9.2% of utterances). Stated limitations include transitional continuity effects, burstiness of switches, and the need to test datasets from more diverse bilingual communities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations

Also on the byline: University of Montana.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksSpeech & spoken language · 480 works

Cite

Cite as Rebecca Pattichis, Dora LaCasse and Rena Torres Cacoullos. 2026. Re-evaluating the Word Token for Bilingual Speech Processing: The Case for Intonation Units. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.580

@article{pattichis2026re,
  title = {{Re-evaluating the Word Token for Bilingual Speech Processing: The Case for Intonation Units}},
  author = {Rebecca Pattichis, Dora LaCasse and Rena Torres Cacoullos},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.580},
  url = {https://doi.org/10.1162/coli.a.580}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aaron, Jessi Elana. 2004. “So respetamos un tradición del uno al otro”: So and entonces in New Mexican bilingual discourse. Spanish in Context, 1(2):161–179. https://doi.org/10.1075/sic.1.2.02aar Aaron, Jessi Elana. 2015. Lone English-origin nouns in Spanish: The precedence of community norms. Int… ↗
  2. Bybee, Joan. 2010. Language, Usage and Cognition. Cambridge University Press. https://doi.org/10.1017/CBO9780511750526 Chafe, Wallace. 1994. Discourse, Consciousness, and Time: The Flow and Displacement of Conscious Experience in Speaking and Writing. University of Chicago Press. ↗
  3. Croft, William. 1995. Intonation units and grammatical structure. Linguistics, 33(5):839–882. https://doi.org/10 .1515/ling.1995.33.5.839 Davies, David L. and Donald W. Bouldin. 1979. A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2):224–227. htt…
  4. Ellison, T. Mark and Aung Si. 2021. A quantitative analysis of age-related differences in Hindi–English code-switching. International Journal of Bilingualism, 25(6):1510–1528. https://doi.org/10.1177/13670069211028311 Fricke, Melinda, Judith F. Kroll, and Paola E. Dussias. 2016. Phonetic variation… — [doi:10.1177/13670069211028311](https://doi.org/10.1177/13670069211028311) ↗
  5. Gambäck, Björn and Amitava Das. 2016. Comparing the level of code-switching in corpora. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1850–1855. Goh, K.-I. and A.-L. Barabási. 2008. Burstiness and memory in complex systems. Europhysics… — [doi:10.1209/0295-5075](https://doi.org/10.1209/0295-5075) find
  6. Labov, William. 1984. Field methods of the project on linguistic change and variation. In John Baugh and Joel Sherzer, editors, Language in Use: Readings in Sociolinguistics. Prentice Hall, pages 28–53. find
  7. Lipski, John M. 1978. Code-switching and the problem of bilingual competence. In M. Paradis, editor, Aspects of Bilingualism. Hornbeam Press, pages 250–264. find
  8. Mettouchi, Amina. 2008. Kabyle/French codeswitching: A case study. In M. Lafkioui and V. Brugnatelli, editors, Berber in Contact: Linguistic and Sociolinguistic Perspectives. Rüdiger Köppe Verlag, pages 187–198. find
  9. Muysken, Pieter. 2000. Bilingual Speech: A Typology of Code-mixing. find
  10. Cambridge University Press. Muysken, Pieter. 2015. Dé jà voodoo or new trails ahead? Re-evaluating the mixing typology model. In R. Torres Cacoullos, N. Dion, and A. Lapierre, editors, Linguistic Variation: Confronting Fact and Theory. Routledge, pages 242–261. find
  11. Myers-Scotton, Carol. 1997. Duelling Languages: Grammatical Structure in Codeswitching. find
  12. Oxford University Press. Ono, Tsuyoshi and Sandra A Thompson. 2020. The ‘Noun Phrase’Across Languages: An Emergent Unit in Interaction, volume 128. John Benjamins Publishing Company. https://doi.org/10.1075/tsl.128 Pattichis, Rebecca, Dora LaCasse, Sonya Trawick, and Rena Torres Cacoullos. 2023. Co… — [doi:10.18653/v1/2023.emnlp-main](https://doi.org/10.18653/v1/2023.emnlp-main) ↗
  13. Poplack, Shana. 2017. Borrowing: Loanwords in the Speech Community and in the Grammar. Oxford University Press. https://doi.org/10.1093/oso/9780190256388.003.0004 Poplack, Shana. 2022. 16 Data management at the uOttawa Sociolinguistics Laboratory. The Open Handbook of Linguistic Data Management, pa… — [doi:10.1093/oso](https://doi.org/10.1093/oso) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pattichis-2026-re-evaluating-word-token-bilingual/