Back to NumFOCUS
GSoC 2026

PyMC Streaming Variational Inference for Large Datasets

PyMC's variational inference stack — including ADVI and Pathfinder — assumes all data fits in memory. Real-world applications such as financial tick data, sensor streams, and large panel datasets regularly exceed RAM. This project extends PyMC's variational inference to support true streaming data via arbitrary Python iterables delivering batches, with ELBO estimation from running streams. Deliverables include a StreamingDataset abstraction wrapping iterators with the PyMC minibatch system, streaming-compatible ADVI and Pathfinder training loops using stochastic L-BFGS, online convergence monitoring callbacks, and tutorial demonstrations on financial time series and panel data exceeding memory limits.

Project details

Contributor

Yicheng Yang

Mentors

Not available

Technologies

Not listed in the archive