Elamur Research⌕
☰
home / papers / 2026 / MazzikaAI: A knowledge-based performance-to-prompt compiler for real-time Arabic maqam accompaniment with a streaming text-to-music model

MazzikaAI: A knowledge-based performance-to-prompt compiler for real-time Arabic maqam accompaniment with a streaming text-to-music model

Summary

Streaming text-to-music generators can take new prompts while they play, but nothing decides what those prompts should say as a musician performs. Their training is also mostly Western, so they handle microtonal Arabic maqam poorly. MazzikaAI fills this gap with a classical knowledge-based system. It has three parts: a hand-written knowledge base, a working memory that estimates the live performance state from MIDI, gestures and inferred harmony, and deterministic rules. Its unusual output channel is natural-language prompts sent to an unmodified Google Lyria RealTime model, with no fine-tuning. The knowledge base covers six maqamat with quarter-tone spellings, ornaments, resting tones and "avoid" guidance, plus a takht ensemble. A four-state call-and-response policy and a change-detection gate decide when to re-steer the stream. The symbolic layer costs under a millisecond, and median key-to-audible-update latency is 263 ms. Explicit maqam grounding significantly raises quarter-tone content. The paper suggests that much of an interactive partner's intelligence can sit in a transparent compiler rather than in model weights.

Key points

  • The compiler is a deterministic mapping from estimated performance state and policy state to a prompt. It steers two Lyria RealTime streams running in parallel: an ensemble melody line that responds to the soloist, and a percussion layer that stays locked to the tempo.
  • Measured costs:
    • The rule layer takes under 1 ms.
    • Median key-to-audible-update latency is 263 ms.
    • The stream handled 179 re-prompts per minute without failure, and 202 per minute with the gate switched off.
  • Ablations used a replay harness that feeds identical input to each variant:
    • Maqam grounding shifted output toward the tonic and gave significantly more quarter-tone melody.
    • With a static prompt, trigger-to-audio latency grew to tens of seconds.
    • Prompt-level instrument prohibition was a poor replacement for stem-level mixing.
  • Coverage beyond the Arabic core: nine further Western modes and eight further genres, each added only by writing configuration fragments. Their output quality is not evaluated.
  • Two expert musicians judged the system in playing sessions. They found it controls what is played much better than when, with beat-level entrainment the main remaining gap.

Interesting ideas

  • I1. Treat a frozen generative model as an actuator: a deterministic compiler turns an explicit state into precise prose, so the expertise stays inspectable outside the weights. — For Mova (learn): LLM teacher tiers could receive prompts compiled deterministically from the current UD tree and frame state, making each teacher query reproducible and auditable.
  • I2. A coarse "control signature" gate re-issues a prompt only when the summarized state changes meaningfully, which saves API calls without hurting stability. — For Mova (learn): compute a coarse signature of a sentence's analysis (tree shape, frame types) and call an expensive teacher or re-parse only when that signature differs from cached cases.
  • I3. The knowledge base stores negative guidance (what an idiom must avoid) alongside positive traits, and the paper measures how far prohibition alone actually works. — For Mova (world): ontology and verb-class entries could carry explicit "does not license" constraints, with evaluation of whether these constraints actually block wrong frame assignments.
  • I4. Replaying input-identical sessions through ablated variants, with a static-prompt arm as a degenerate baseline, attributes behaviour to individual design choices. — For Mova (learn): run fixed input sets through pipelines with single components disabled (for example, frames frozen to a trivial default) as negative controls, so each stage's real contribution is measured.
  • I5. New idioms are added purely as configuration records (sections, role wording, parameters) while the estimator and controller stay untouched, which makes adaptation cost explicit. — For Mova (tree): UD dialect conversions could be written as declarative per-treebank configuration over one shared converter, so adding a treebank needs no code change.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Institutions

Also on the byline: Grand Valley State University.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.10360 · arXiv 2608.10360

@misc{du2026mazzikaai,
  title = {{MazzikaAI: A knowledge-based performance-to-prompt compiler for real-time Arabic maqam accompaniment with a streaming text-to-music model}},
  author = {Du, Jiaxin and Abdeljaouad, Boulbaba and Zhuang, Yong and Li, Haoyu},
  year = {2026},
  doi = {10.48550/arxiv.2608.10360},
  eprint = {2608.10360},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.10360}
}

References

26 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agostinelli, A., Denk, T.I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., Frank, C., 2023. MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325 . ↗
  2. Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., Zeghidour, N., 2023. AudioLM: A language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, 2523–2533. doi:10.1109/TASLP.2023.3288409. ↗ our card
  3. Bozkurt, B., Ayangil, R., Holzapfel, A., 2014. Computational analysis of Turkish makam music: Review of state-of-the-art and challenges. Journal of New Music Research 43, 3–23. doi:10.1080/09298215.2013.865760. ↗
  4. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020. Language models are few-shot learners, in: Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 1877–1901. find our card
  5. Cope, D., 1992. Computer modeling of musical intelligence in EMI. Computer Music Journal 16, 69–83. find
  6. Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., Défossez, A., 2023. Simple and controllable music generation, in: Advances in Neural Information Processing Systems 36 (NeurIPS 2023). ArXiv:2306.05284. ↗
  7. Défossez, A., Copet, J., Synnaeve, G., Adi, Y., 2023. High fidelity neural audio compression. Transactions on Machine Learning Research ArXiv:2210.13438. ↗ our card
  8. Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., Sutskever, I., 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 . ↗
  9. Ebcioğlu, K., 1988. An expert system for harmonizing four-part chorales. Computer Music Journal 12, 43–51. Lead Controller nay harmonica saxophone lead guitar oboe vocal aah flamenco gtr piano horn section bespoke (takht) bespoke generic generic generic generic generic generic generic Evans, Z., Carr, C., Taylor, J., Hawley, S.H., Pons, J., 2024. Fast timing-conditioned latent audio diffusion, in: Proceedings of the 41st International Conference on Machine Learning (ICML). ArXiv:2402.04825. ↗
  10. Forsgren, S., Martiros, H., 2022. Riffusion: Stable diffusion for real-time music generation. https://riffusion.com/about. ↗
  11. Google DeepMind, 2025. Lyria RealTime: Live music generation via the Gemini API. https://magenta.withgoogle.com/lyria-realtime. Accessed via the Gemini API live music interface; technical report: arXiv:2508.04651. ↗
  12. Hadjeres, G., Pachet, F., Nielsen, F., 2017. DeepBach: A steerable model for Bach chorales generation, in: Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1362–1371. find
  13. Huang, C.Z.A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A.M., Hoffman, M.D., Dinculescu, M., Eck, D., 2019. Music transformer: Generating music with long-term structure, in: International Conference on Learning Representations (ICLR). find
  14. Lewis, G.E., 2000. Too many notes: Computers, complexity and culture in Voyager. Leonardo Music Journal 10, 33–39. find
  15. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G., 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys 55, 1–35. doi:10. 1145/3560815. find our card
  16. Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., Lee, J., Chang, W.T., Hua, W., Georg, M., Grundmann, M., 2019. MediaPipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172 . ↗
  17. Lyria Team, Google DeepMind, 2025. arXiv:2508.04651 . ↗
  18. Monson, I., 1996. Saying Something: Jazz Improvisation and Interaction. Chicago Studies in Ethnomusicology, University of Chicago Press, Chicago, IL. find
  19. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., Kavukcuoglu, K., 2016. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 . ↗ our card
  20. Pachet, F., 2003. The continuator: Musical interaction with style. Journal of New Music Research 32, 333–341. find
  21. Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X., 2024. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering 36, 3580–3599. doi:10.1109/TKDE.2024. 3352100. Live music models. arXiv preprint Roberts, A., Engel, J., Raffel, C., Hawthorne, C., Eck, D., 2018. A hierarchical latent vector model for learning long-term structure in music, in: Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 4364– 4373. ↗ our card
  22. Rowe, R., 1993. Interactive Music Systems: Machine Listening and Composing. MIT Press, Cambridge, MA. find
  23. Scarlatos, A., Wu, Y., Simon, I., Roberts, A., Cooijmans, T., Jaques, N., Tarakajian, C., Huang, C.Z.A., 2025. ReaLJam: Real-time human-AI music jamming with reinforcement learning-tuned transformers, in: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25), ACM. doi:10.1145/3706599.3720227. ↗
  24. Shahriar, S., Tariq, U., 2021. Classifying maqams of Qur’anic recitations using deep learning. IEEE Access 9, 117271–117281. doi:10.1109/ACCESS.2021. 3098415. ↗
  25. Thom, B., 2000. BoB: An interactive improvisational music companion, in: Proceedings of the Fourth International Conference on Autonomous Agents (Agents 2000), ACM, Barcelona, Spain. pp. 309–316. doi:10.1145/336595. 337510. ↗
  26. W3C, 2015. Web MIDI API. https://www.w3.org/TR/webmidi/. W3C Working Draft. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/du-2026-mazzikaai-knowledge-based-performance-prompt/