UniXcoder: Unified Cross-Modal Pre-training for Code Representation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.acl-long.499.pdf

Summary

Pre-trained code models usually fall into one of three types, and each has a weakness. Encoder-only models need a decoder trained from scratch to generate text or code. Decoder-only models are weaker at understanding tasks. Encoder-decoder models are inefficient for code completion, which in practice should run decoder-only. UniXcoder addresses this with one multi-layer Transformer. Mask attention matrices combined with prefix adapters switch the model between encoder, decoder and encoder-decoder modes.

Besides source code, the model takes code comments and the AST as input. The AST is flattened by a reversible (one-to-one) mapping that keeps the full tree structure. Pre-training mixes three language-modeling objectives with two representation tasks: multi-modal contrastive learning and cross-modal generation, which uses comments to align embeddings across programming languages.

The authors evaluated UniXcoder on five task types over nine datasets, plus a new zero-shot code-to-code search benchmark built from CodeNet. It reached state-of-the-art results on most tasks, and ablations show that both comments and AST help. This makes UniXcoder a practical single backbone for both understanding and generation in code intelligence.

Key points

Pre-training objectives are masked language modeling, unidirectional language modeling and a denoising objective. Two further tasks target code-fragment embeddings: contrastive learning and cross-modal generation.
The AST mapping wraps each non-terminal node's children with left and right boundary tokens. Swapping it for BFS or DFS traversal lowered scores, sometimes below the variant with no AST at all.
The new zero-shot code-to-code search dataset contains 11,744 Ruby, 15,594 Python and 23,530 Java functions from CodeNet, covering 4,053 problems. It is scored with MAP.
On this benchmark UniXcoder reaches an overall MAP of 20.45%, about 11 points above GraphCodeBERT. Without contrastive learning the score drops to 13.73%.
For code completion, accuracy on the Java corpus rose by an absolute 2.3%, and UniXcoder outperformed PLBART and CodeT5. Code and data are released at github.com/microsoft/CodeBERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

132 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 130 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.499 · arXiv 2203.03850

@inproceedings{guo2022unixcoder,
  title = {{UniXcoder: Unified Cross-Modal Pre-training for Code Representation}},
  author = {Guo, Daya and Lu, Shuai and Duan, Nan and Wang, Yanlin and Zhou, Ming and Yin, Jian},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.499},
  eprint = {2203.03850},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.499}
}

References

5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668. find our card
Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 207– 216. IEEE. find
GraphCodeBERT: def ranks(self, key, value): return [normalize\_rank(el) for el in force\_list(value.get('a'))] PLBART: def zrank(self, name, value): with self.pipe as pipe: value = self.valueparse.encode(value) return pipe.zrank(self.redis\_key(name), value) find
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
CodeT5: def sort\_key(x): name, (r, u) = x return - len(u) + u.count('}') * 100 UniXcoder: def revrank\_dict(dict, key=lambda t: t[1], as\_tuple=False): sorted\_list = sorted(dict.items(), key=key, reverse=True) return OrderedDict(sorted\_list) if not as\_tuple else tuple(sorted\_list) *Figure 4: An exampl… find