📄 Data structures for statistical computing in Python

Summary

The paper addresses why Python has been slow to win over statisticians compared with R. A key reason is the lack of convenient in-memory structures for labeled, mixed-type tabular data. NumPy structured arrays can hold such data, but they fit poorly with the rest of NumPy and are awkward next to R's data.frame. McKinney presents pandas, a library first built at AQR Capital Management for quantitative finance. It provides labeled 1-, 2- and 3-dimensional containers, including Series and DataFrame. The design choices are illustrated with stock price and volume examples for GOOG and AAPL and with a table of country and industry categories. Central ideas include:

• axes tied to Index objects that act as ordered sets backed by hash maps;
• automatic label-based alignment when combining objects;
• NaN-based handling of missing values;
• joins on the index or on key columns;
• reshaping with pivot;
• calendar-aware date ranges and frequency conversion.

The main contribution is a practical, R-like data model that fits the scientific Python stack. It is offered as a foundation for statistical modeling tools in Python.

Key points

• The name pandas comes from "panel data", the econometric term for 3-dimensional data sets. Higher dimensions were deliberately left out as rare in practice.
• Index keeps labels both as an ndarray and as a dict, which makes lookups and membership checks constant-time. Arithmetic between objects aligns on labels and returns their union, filling gaps with NaN. Core routines were sped up with Cython.
• NaN was chosen over NumPy MaskedArrays for performance. The trade-off is that it does not work in integer arrays. Helpers such as isnull/notnull, valid and fillna handle missing values, and reductions like sum and count skip them.
• DateRange (an Index subclass) and DateOffset subclasses support custom frequencies, asfreq and shift. For example, business month-ends from 2000 to 2010 give 120 dates. Relying on Python's built-in datetime may limit performance on high-frequency data.
• Related packages include la/larry, tabular and pydataframe. Closer integration with scikits.statsmodels is planned.