🖥️ Computer Science

A survey · about 17 minutes to read · every paper named here has its own page in this library

The Computer Science shelf holds every paper of this library that OpenAlex places in computer science. Most of them are language technology, surveyed on the Artificial Intelligence sub-shelf; this survey is about the rest, the papers whose main contribution lies in computer science proper rather than in the study of language: algorithms for search and ranking on the web, systems that train and serve large neural networks on GPUs, compilers for tensor programs, tools for organising and clustering large text collections, and models that read and write source code. Many of them were written by language researchers, and most of them handle text in some form, but their questions are about efficiency, scale, data structures and software.

The shelf is unusually spread out in time. Some of its most cited works date from the late 1990s, when the web was young and the problem of the day was how to rank millions of pages; others are from the last few years, when the problem became how to fit a model with billions of parameters into GPU memory and answer thousands of requests a second. Between them lie two decades of probabilistic topic models, learning-to-rank, dense retrieval and code generation.

What links these strands is infrastructure. The ranking methods here decide what search engines return, the retrieval libraries and embedding models feed retrieval-augmented systems, the serving and quantisation papers determine what it costs to run a language model, and the code benchmarks define how progress in programming assistants is measured. The shelf is a map of the machinery beneath modern language technology.

What the field covers

The shelf groups several areas of computer science that meet language processing.

Web search and information retrieval. Link analysis (PageRank, hubs and authorities), probabilistic and language-model ranking functions, learning to rank, and, since 2019, neural retrievers that encode queries and documents as vectors. This strand overlaps heavily with the topic Information retrieval & search.

Text mining, topic models and clustering. Latent-variable models that discover themes in document collections, metrics for judging whether those themes make sense to people, classic text classification, and general-purpose tools for clustering and dimension reduction.

Systems for large models. Distributed training, memory management for inference, quantisation, long-context attention kernels, speculative and parallel decoding, and compilers that turn tensor programs into fast GPU code. The topics GPU & parallel computing and Efficient models & compression cover much of this.

Code and programming languages. Pretrained models of source code, program synthesis from natural language, text-to-SQL, and benchmarks that execute generated programs against tests. This is the core of Code & programming languages.

Language-processing infrastructure. Tokenisers, language identifiers, efficient parsing algorithms and grammar formalisms written with complexity in mind, and tree transducers for syntax-based translation.

It differs from Artificial Intelligence, its largest neighbour, in that the papers here usually treat language as data to be indexed, compressed or served rather than as a phenomenon to be explained. It differs from Artificial Intelligence in its focus on engineering: throughput, memory, latency, and the software that ships the algorithms.

History and milestones

Probabilistic retrieval and efficient parsing (1994–2001)

The oldest paper on the shelf, Okapi at TREC-3, reports the system in which the BM25 weighting function took its familiar form; BM25 remained the default first-stage ranker for a quarter of a century. A Language Modeling Approach to Information Retrieval proposed ranking documents by the probability that their language model generated the query, a framework that shaped academic IR for years. In parsing, Three New Probabilistic Models for Dependency Parsing and Efficient Parsing for Bilexical Context-Free Grammars and Head Automaton Grammars introduced the cubic-time algorithms still used for projective dependency parsing, while An Efficient Probabilistic Context-Free Parsing Algorithm that Computes Prefix Probabilities and Fast Context-Free Grammar Parsing Requires Fast Boolean Matrix Multiplication tied parsing to deeper questions of complexity.

The link-analysis moment (1998–1999)

In 1998 and 1999 three papers changed web search. The Anatomy of a Large-Scale Hypertextual Web Search Engine described the Google prototype built at Stanford; the companion technical report on PageRank defined page importance through the link graph and a random surfer; and Authoritative Sources in a Hyperlinked Environment introduced hubs and authorities. Around the same time, machine learning for text matured: Text Categorization with Support Vector Machines established SVMs for document classification, and Combining Labeled and Unlabeled Data with Co-Training founded multi-view semi-supervised learning using web pages as its example. The papers of this period gather a large share of all citations on the shelf.

Topic models and learning to rank (1999–2015)

Probabilistic Latent Semantic Analysis recast latent semantic indexing as a mixture model. Latent Dirichlet Allocation, the most cited paper on the shelf, made it fully generative with Dirichlet priors and launched a decade of topic modelling, including Dynamic Topic Models. Researchers then asked whether topics are interpretable: Reading Tea Leaves measured it with people, and Automatic Evaluation of Topic Coherence, Optimizing Semantic Coherence in Topic Models, Machine Reading Tea Leaves and Exploring the Space of Topic Coherence Measures automated it. In ranking, Learning to Rank using Gradient Descent (RankNet) brought a pairwise probabilistic loss to commercial search, and Reciprocal Rank Fusion showed that a very simple formula merges ranked lists better than elaborate methods. Benchmarks such as RCV1 supported reproducible text classification.

Neural text, GPUs and tooling (2015–2019)

Deep learning moved the shelf towards GPUs. Character-level Convolutional Networks for Text Classification treated text as a raw signal; Deep Speech 2 showed what a high-performance training stack could do for speech recognition; Billion-scale similarity search with GPUs produced the Faiss library. Compilers appeared for tensor code, with TVM and Triton, and model parallelism with Mesh-TensorFlow. Widely used tools from this time include SentencePiece, UMAP, hdbscan and Sentence-BERT.

Dense retrieval, code models and LLM systems (2020–2025)

Most of the shelf's papers are recent. Dense Passage Retrieval for Open-Domain Question Answering and ColBERT made learned retrieval practical; CodeBERT and Evaluating Large Language Models Trained on Code opened the era of code models; and with large language models in production, systems work turned to serving, as in Efficient Memory Management for Large Language Model Serving with PagedAttention. By 2024–2025 the emphasis had shifted to agents that edit real repositories (SWE-agent), contamination-free code evaluation, and systems that discover algorithms (AlphaEvolve).

Main problems and tasks

• Ranking. Given a query, order a very large collection so that the most useful documents come first. Signals range from term statistics (BM25) and link structure (PageRank) to learned scoring functions and neural encoders.
• First-stage retrieval at scale. Finding candidates among millions or billions of items fast enough for interactive use, via inverted indexes, approximate nearest-neighbour search, or learned sparse representations such as SPLADE.
• Text classification and clustering. Assigning documents to categories, with or without labels, and grouping them without supervision; evaluation measures such as V-Measure belong here.
• Topic discovery. Finding interpretable themes in a corpus and tracking how they change.
• Code understanding and generation. Searching code with natural language, writing functions from docstrings, translating between programming languages, generating SQL from questions, and resolving real software issues.
• Efficient training. Fitting models that exceed one GPU's memory through data, tensor, pipeline and sequence parallelism, optimizer-state offloading and activation recomputation.
• Efficient inference. Reducing the memory, latency and energy cost of serving language models through batching, caching, quantisation, sparsity and faster decoding.
• Language identification. Telling which language, or which closely related variety, a text is in, a prerequisite for building multilingual web corpora.

Methods and schools

Probabilistic and link-based ranking

The first school treats retrieval as estimation. BM25 derives term weights from a probabilistic relevance model; the language-modelling approach scores documents by query likelihood; PageRank and HITS compute principal eigenvectors of link matrices. These methods are unsupervised, cheap and hard to beat on unfamiliar data, which explains BM25's survival as a baseline in nearly every neural retrieval paper here.

Learning to rank

From the mid-2000s, ranking functions were trained on relevance judgements. RankNet's pairwise logistic loss became the basis of a family of later methods. Reciprocal Rank Fusion is the counterpoint: an untrained combination of ranked lists that often beats trained ones, and that is again widely used in hybrid search.

Dense, late-interaction and learned sparse retrieval

The neural era brought three designs. Bi-encoders such as DPR and Sentence-BERT compress each text into one vector and rely on fast vector search such as Faiss. Late-interaction models such as ColBERT and ColBERTv2 keep a vector per token and compare them at query time. Learned sparse models such as SPLADE, Document Expansion by Query Prediction and Query2doc keep the inverted index but let a neural model choose the terms. The debate between these designs is largely one about the trade-off between accuracy, index size and latency, and about how well each generalises beyond its training data, which BEIR was built to measure.

Probabilistic topic models and their successors

pLSA and LDA model each document as a mixture of word distributions and are fitted by EM, variational inference or sampling. Later work added time, correlation and embeddings: Topic Modeling in Embedding Spaces places words and topics in one vector space, and BERTopic abandons the generative model altogether, clustering sentence embeddings with UMAP and HDBSCAN and labelling clusters with a class-based TF-IDF.

Systems for large models

Training at scale relies on partitioning. ZeRO-Offload moves optimizer state to CPU memory; Reducing Activation Recomputation in Large Transformer Models and DeepSpeed Ulysses split activations and sequences across devices; Ring Attention passes blocks of keys and values around a ring of GPUs to handle very long contexts; DeepSpeed-MoE addresses mixture-of-experts models; PyTorch Distributed documents the data-parallel machinery most practitioners use.

For inference, vLLM's PagedAttention borrows virtual-memory paging from operating systems to remove waste in the key–value cache, and SGLang reuses cached prefixes across structured programs of calls. FlexGen and LLM in a flash run large models with limited GPU or device memory. Quantisation methods compress weights, activations or the cache to a few bits: ZeroQuant, I-BERT, SqueezeLLM, QuaRot, QuIP#, KVQuant, KIVI and 8-bit Optimizers via Block-wise Quantization. Decoding is accelerated by drafting several tokens at once, as in Medusa, EAGLE and Lookahead Decoding, while DuoAttention and Native Sparse Attention cut the cost of long contexts. From Words to Watts measures the energy these systems consume.

Compilers for tensor programs

TVM searches over schedules to generate fast kernels for many hardware back ends; Triton gives programmers a tile-based language that compiles to efficient GPU code without hand-written CUDA. KernelBench closes the loop by asking whether language models can write such kernels themselves.

Learning from code

Code models evolved from syntax-aware generators such as A Syntactic Neural Model for General-Purpose Code Generation, through encoder pretraining (CodeBERT, GraphCodeBERT, UniXcoder, PLBART), to large generative models (Codex, AlphaCode, Code Llama). Training tricks such as fill-in-the-middle and reinforcement from unit tests (CodeRL) and agent loops (Reflexion, SWE-agent) followed. Unsupervised Translation of Programming Languages carried back-translation from machine translation to code.

Datasets, resources and benchmarks

Key software described by papers here includes Faiss, SentencePiece, sentence-transformers, vLLM, TVM, Triton, langid.py, UMAP, hdbscan and the TAGME entity annotator. Web-scale corpora such as The FineWeb Datasets and DataComp-LM link data engineering to pretraining.

Landmark papers in this library

Latent Dirichlet Allocation. Blei, Ng and Jordan gave each document a Dirichlet-distributed mixture of topics and fitted the model with variational inference, fixing pLSA's lack of a document-level generative story. LDA became one of the most widely used models in all of text analysis and spawned dynamic, correlated and supervised variants.

The Anatomy of a Large-Scale Hypertextual Web Search Engine. A rare full description of a search engine's crawler, indexer and ranker, built around link analysis and anchor text. It describes the prototype that grew into Google.

The PageRank Citation Ranking: Bringing Order to the Web. Defines a page's importance recursively by the importance of pages that link to it, computed by power iteration with a random-jump term. Personalised variants of PageRank are now used far beyond the web, in graph-based ranking of all kinds.

Authoritative Sources in a Hyperlinked Environment. Kleinberg's HITS algorithm separates hubs, which point to good pages, from authorities, which good hubs point to, and obtains both as eigenvectors of matrices built from a query-focused subgraph. It is a foundation of web-graph analysis.

Text Categorization with Support Vector Machines. Joachims argued from the properties of text, with its many relevant but sparse features, that large-margin classifiers fit the task, and showed them beating the standard alternatives on Reuters and Ohsumed. Linear SVMs remained the default text classifier until neural methods took over.

UMAP. A manifold-learning method grounded in topology that produces layouts comparable to t-SNE while running faster and preserving more global structure. It is now a standard step for visualising embeddings, including in topic-modelling pipelines.

Dense Passage Retrieval for Open-Domain Question Answering. Two BERT encoders trained with in-batch negatives outperformed BM25 on several question-answering datasets without special pretraining. DPR made dense retrieval the usual first stage of retrieval-augmented generation.

Sentence-BERT. Fine-tuning BERT in a siamese network yields sentence vectors comparable by cosine similarity, reducing large similarity searches from hours to seconds. The accompanying library became a standard toolkit for embeddings.

Billion-scale similarity search with GPUs. A fast GPU k-selection routine and product-quantised indexes made nearest-neighbour search over a billion vectors feasible on a single machine. The resulting Faiss library underlies much of today's vector search.

SentencePiece. A language-independent subword tokeniser that learns directly from raw text and treats whitespace as an ordinary symbol, making tokenisation reversible. Many multilingual and large language models use it; it connects the shelf to Tokenization & segmentation.

Evaluating Large Language Models Trained on Code. Introduced Codex and the HumanEval benchmark, and made functional correctness, measured by running unit tests with the pass@k metric, the standard for code generation.

Efficient Memory Management for Large Language Model Serving with PagedAttention. By storing the key–value cache in non-contiguous fixed-size blocks, vLLM nearly eliminates memory fragmentation and allows much larger batches. It became one of the most widely deployed open serving engines.

CodeBERT. An early bimodal model pretrained on natural language and code from several programming languages, strong on code search and documentation generation. It led to a series of code encoders from the same group.

Learning to Rank using Gradient Descent. RankNet models the probability that one document should outrank another with a logistic function of their score difference and trains neural networks on that loss, an approach that underpinned later commercial ranking systems.

SWE-bench. Collected real issues and pull requests from popular Python repositories and asked models to produce patches that pass the project's tests. It moved code evaluation from isolated functions to whole codebases and became the main yardstick for coding agents.

People

• Marcos Zampieri (George Mason University) is the most prolific author on the shelf, through the shared tasks on discriminating similar languages and dialects and the survey of automatic language identification. His co-organisers include Jörg Tiedemann (University of Helsinki), Nikola Ljubešić (Jožef Stefan Institute and University of Ljubljana) and Preslav Nakov (Mohamed bin Zayed University of Artificial Intelligence).
• David M. Blei (Columbia University) is behind the shelf's most cited line of work: LDA, dynamic topic models, the human study of topic interpretability and embedded topic models.
• Lawrence Page and Sergey Brin wrote the two foundational papers on PageRank and the Google prototype at Stanford.
• Thorsten Joachims established SVMs for text classification and later shaped learning to rank from user behaviour.
• Thomas Hofmann introduced probabilistic latent semantic analysis, the direct precursor of LDA.
• Timothy Baldwin (University of Melbourne and MBZUAI) contributed both to topic-coherence evaluation and to language identification with langid.py, alongside Jey Han Lau and David Newman.
• Lianmin Zheng, Ying Sheng, Joseph E. Gonzalez and Ion Stoica (University of California, Berkeley) built vLLM, SGLang and FlexGen; Zheng also co-authored TVM with Tianqi Chen.
• Minjia Zhang, Yuxiong He, Samyam Rajbhandari, Zhewei Yao and Reza Yazdani Aminabadi form the DeepSpeed group behind ZeRO-Offload, ZeroQuant, DeepSpeed-MoE and Ulysses.
• Kurt Keutzer, Michael W. Mahoney, Amir Gholami and Sehoon Kim (Berkeley) developed the integer-only and low-bit quantisation papers I-BERT, SqueezeLLM and KVQuant.
• Nils Reimers (now at Cohere) created Sentence-BERT with Iryna Gurevych and co-authored BEIR and MTEB, defining how text embeddings are built and evaluated.
• Matei Zaharia co-authored ColBERT and ColBERTv2 with Omar Khattab, as well as Ring Attention.
• Daya Guo and Nan Duan (Microsoft Research Asia) led the CodeBERT, GraphCodeBERT and UniXcoder series.
• Karthik Narasimhan (Princeton University) and Shunyu Yao co-authored Reflexion, SWE-bench and SWE-agent, central works on language agents for software.
• Baptiste Rozière worked on unsupervised translation between programming languages, Code Llama and CRUXEval.
• Kevin Knight and Jason M. Eisner represent the formal side: syntax-based translation models, tree transducers and efficient parsing algorithms.
• Francisco J. R. Ruiz (Google DeepMind) links embedded topic models to program search for mathematical discovery in FunSearch and AlphaEvolve.

Institutions and places

Stanford University, the birthplace of PageRank, is also the most frequent current home of authors on the shelf, together with Tsinghua University. The University of California, Berkeley anchors the serving and quantisation work, and Microsoft the DeepSpeed and code-model work. Other strong centres include Carnegie Mellon University, the University of Illinois Urbana-Champaign, Columbia University, Princeton University, the University of Washington, Cornell University, the University of California, Santa Cruz, Amazon, Google and the Allen Institute for Artificial Intelligence.

Outside the United States, Chinese institutions such as Peking University, the Harbin Institute of Technology, the Chinese Academy of Sciences and the University of Chinese Academy of Sciences are prominent. Mohamed bin Zayed University of Artificial Intelligence in the United Arab Emirates and the University of Helsinki in Finland stand out through the language-identification work.

Venues are unusually diverse. Retrieval papers appear at SIGIR and CIKM, machine-learning classics at ICML, COLT and ECML and in JMLR, web papers at WWW and in the Journal of the ACM, systems papers at SOSP and MAPL, and software in the Journal of Open Source Software. The language-oriented work appears at EMNLP, ACL, NAACL and COLING. Many of the newest and most cited papers circulated first as preprints.

Open questions

• Which retrieval architecture wins? Dense, late-interaction, learned sparse and classical lexical methods each lead on some benchmarks; hybrid systems fused with simple methods such as reciprocal rank fusion often do best. How well learned retrievers transfer to new domains and languages, as tested by BEIR and MIRACL, remains unsettled.
• Evaluating code honestly. HumanEval-style tests can be too weak (EvalPlus found many accepted solutions wrong), and public problems leak into training data, which LiveCodeBench addresses by using only problems published after a model's cutoff. Repository-level tasks such as SWE-bench are more realistic but costlier to run.
• The cost of inference. Serving rather than training now dominates the bill for many models. How far quantisation can go before quality drops, how to handle contexts of millions of tokens, and how to measure energy use fairly are all active questions.
• Hardware-aware model design. Work such as Native Sparse Attention and KernelBench asks whether architectures should be designed together with the kernels that run them, and whether models can write those kernels.
• Topic models in the embedding era. Whether neural clustering approaches such as BERTopic replace probabilistic models or complement them, and how to evaluate topics reliably, are still debated.
• Language identification for thousands of languages. As Language ID in the Wild, GlotLID and An Open Dataset and Model for Language Identification show, identifiers that work well on clean benchmarks fail on noisy web text for low-resource languages, which affects every multilingual corpus built from the web.
• Safety filters as infrastructure. Classifiers such as Llama Guard, WildGuard and A Holistic Approach to Undesired Content Detection now sit around deployed models; how to make them accurate without over-blocking ties the shelf to Bias, fairness & safety.

Neighbouring topics

The largest overlap is with Artificial Intelligence, followed by Language and Linguistics, Artificial Intelligence, Machine translation and Mathematics. The machine-translation overlap comes from syntax-based translation models such as A Syntax-based Statistical Translation Model, What's in a translation rule?, Training Tree Transducers and Synchronous Grammars as Tree Transducers, and from engineering studies such as Training Tips for the Transformer Model and Deep Encoder, Shallow Decoder.

Among topics, Language models & pretraining is the most common, followed by Code & programming languages, Information retrieval & search, Evaluation & benchmarks and Efficient models & compression. Smaller links reach Machine translation, GPU & parallel computing, Syntax & grammar formalisms, Semantic parsing & meaning representation (through text-to-SQL work such as RAT-SQL), Dependency parsing, Morphology (through CRF-based Japanese morphological analysis), Part-of-speech tagging, Semantic role labeling, Named entity recognition, Question answering, Reasoning and Multilingual & cross-lingual NLP.

Key works in this library

• Okapi at TREC-3 — the system that gave BM25 its standard form.
• Three New Probabilistic Models for Dependency Parsing: An Exploration — efficient probabilistic dependency parsing.
• The Anatomy of a Large-Scale Hypertextual Web Search Engine — the Google prototype.
• A Language Modeling Approach to Information Retrieval — ranking by query likelihood.
• Text Categorization with Support Vector Machines: Learning with Many Relevant Features — SVMs for text.
• Combining Labeled and Unlabeled Data with Co-Training — two views, few labels.
• The PageRank Citation Ranking: Bringing Order to the Web — importance from links.
• Authoritative Sources in a Hyperlinked Environment — hubs and authorities.
• Probabilistic Latent Semantic Analysis — LSA as a mixture model.
• Latent Dirichlet Allocation — the generative topic model.
• Learning to Rank using Gradient Descent — RankNet.
• Reading Tea Leaves: How Humans Interpret Topic Models — likelihood versus interpretability.
• Billion-scale similarity search with GPUs — Faiss.
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction — fast manifold embedding.
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing — tokenisation from raw text.
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — compiling tensor programs.
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — sentence vectors for search.
• Dense Passage Retrieval for Open-Domain Question Answering — learned first-stage retrieval.
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — token-level late interaction.
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages — pretraining on code and text.
• Evaluating Large Language Models Trained on Code — Codex and HumanEval.
• Competition-Level Code Generation with AlphaCode — sampling and filtering at scale for contest problems.
• Efficient Memory Management for Large Language Model Serving with PagedAttention — vLLM.
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — repository-level evaluation.
• AlphaEvolve: A coding agent for scientific and algorithmic discovery — evolving programs with language models.