GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
Paper · 2026 · Jiaying Zhang, Lei Shi, Jiguo Li et al.
Summary
Parameter-efficient fine-tuning methods are mostly designed for SFT, but RLVR changes model weights in a different way. Its useful updates tend to avoid the dominant principal directions of pre-trained matrices and concentrate in a small, low-magnitude region. Adapters like PiSSA therefore train the wrong subspace. Sparse fine-tuning matches RLVR patterns better but gains little real speed on current hardware.
GeoRA addresses both problems. For each weight matrix, it builds a geometry-constrained matrix W_Geo by masking. A spectral prior keeps the lowest-magnitude entries of a rank-r approximation, a Euclidean prior keeps the smallest original weights, and the two masks are combined as a union. The top-r SVD components of W_Geo initialize the LoRA factors. The scaled adapter product is subtracted from W to form a frozen residual matrix, so the model's output is unchanged at the start and computation stays dense.
Across RLVR in math, medicine and coding on Qwen and Llama models (1.5B–32B), GeoRA outperforms low-rank baselines and forgets less on out-of-domain tasks. This makes RLVR adaptation cheaper without eroding pre-trained structure.
Key points
- Baselines are LoRA, PiSSA and MiLoRA. Models are Qwen and Llama families from 1.5B to 32B parameters, including Qwen3-8B and Llama-3.1-8B.
- Both masks use the same sparsity ratio ρ. On Qwen3-8B with ρ = 0.2, each mask selects 20% of parameters, but they overlap on only 4.55% (Jaccard 0.128). This suggests the two priors are complementary, and ablations confirm that removing either one lowers performance.
- In spectral analysis, sparse random noise is as spectrally flat as dense noise. By contrast, W_Geo and the actual full fine-tuning update ΔW_FullFT both show heavy-tailed, compressible spectra, which supports a low-rank form for RLVR updates.
- On Normalized Spectral Shift (NSS), PiSSA distorts pre-trained spectra strongly (>0.39). GeoRA stays at about 0.09, compared with roughly 0.21–0.24 for LoRA and 0.13 for MiLoRA.
- Ablations show that simply training the lowest-energy ("Tail-r") directions performs poorly. Gains require the full design: geometry-constrained subspace, structured initialization and frozen residual anchor.
Institutions
ChinaFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works
🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works
Cite
Cite as Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao and Renqing He. 2026. GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
@inproceedings{zhang2026geora,
title = {{GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR}},
author = {Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao and Renqing He},
booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2026},
doi = {10.18653/v1/2026.acl-long.1110},
url = {https://doi.org/10.18653/v1/2026.acl-long.1110}
}References
- Md Tanvirul Alam and Nidhi Rastogi. 2025. Limits of generalization in rlvr: Two case studies in mathematical reasoning. Preprint, arXiv:2510.27044. high-energy principal subspace (S Head ≤ 0.02)— enhancing stability—and efficiently adapts the geometry-constrained tail (S Tail ≥ 0.96). In contrast, P… find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-geora-geometry-aware-low-rank/