🧑‍🔬 Yunfeng Liu

Yunfeng Liu works on neural language models, and their papers here deal with how transformer encoders represent text. One strand concerns the internal design of the transformer, in particular how a model built on self-attention, which has no built-in sense of word order, can be given information about where each token sits in a sequence. The other strand concerns pre-training data and vocabulary, asking how a model can be adapted to a language variety that dominant training corpora largely ignore.

The most cited of these works is RoFormer: Enhanced Transformer with Rotary Position Embedding (2021). It introduces Rotary Position Embedding (RoPE), which turns query and key vectors through an angle tied to each token's absolute position. As a result, their dot product reflects only how far apart two tokens are. The paper motivates the idea with complex numbers in two dimensions and extends it to higher dimensions by treating the vector as pairs of coordinates. The method also suits inputs of varying length and fits more easily with linear self-attention than additive position vectors do.

The more recent paper, AlcLaM: Arabic Dialectal Language Model (2024), turns to Semitic languages and to everyday written Arabic. The authors gathered dialectal posts and comments from social media and used a classifier to filter out Modern Standard Arabic passages such as quoted religious text. They combined this material with subtitle text, built a vocabulary from the dialectal data, and trained a BERT-base encoder from scratch on far less text than earlier Arabic models used. The work also derives coarser regional and binary benchmarks from a fine-grained dialect label set, which shows at what level of detail a model's errors appear. Both papers sit within language models and pretraining.

Liu's current affiliations include Bohai University. Co-authors on the site include Jianlin Su, with whom the RoFormer paper was written, and Bo Wen.

Yunfeng Liu has 2 papers in our library, in 2021–2024, most of them on Language models & pretraining, Semitic languages; often with Bo Wen, Jianlin Su.