python package to run stratified/saturated regressions out-of-memory with duckdb. The package is a wrapper around the duckdb package and provides a simple interface to run regressions on very large datasets that do not fit in memory by reducing the data to a set of summary statistics and runs weighted least squares with frequency weights. Robust standard errors are computed from sufficient statistics, while clustered standard errors are computed using the cluster bootstrap.

See examples in notebooks/introduction.ipynb.

install (preferably in a venv) with

(uv) pip install git+https://github.com/apoorvalal/duckreg.git

or git clone this repository and install in editable mode.

Currently supports the following regression specifications:

• DuckRegression: general linear regression, which compresses the data to y averages stratified by all unique values of the x variables
• DuckMundlak: Mundlak regression, which compresses the data to y averages stratified by $1, w, \bar{w}{i, .}, \bar{w}{., t}$ where $w$ is a covariate (typically treatment)
• DuckDoubleDemeaning: Double demeaning regression, which compresses the data to y averages by all values of $w$ after demeaning by $\bar{w}{i, .}, \bar{w}{., t}, \bar{w}$ .

references:

methods:

• Arkhangelsky and Imbens (2023)
• Wooldridge 2021
• Wong et al 2021

libraries:

• Grant McDermott's duckdb lecture