Lumen + Xarray Integration
Lumen handles tabular data through DuckDB but cannot work with N-dimensional scientific formats like NetCDF, Zarr, and HDF5. This project brings first-class xarray support to Lumen by building on my already-merged XArraySQLSource (PR #1741), wiring it into Lumen AI for natural language queries over scientific data, and extending the Pipeline to natively handle xarray.Dataset via narwhals. After this integration, a climate scientist can run "lumen-ai serve data.nc" and ask "show temperature anomalies over Europe" - the AI generates DataFusion SQL, queries the data, and renders the visualisation. Deliverables: 1. Source enhancements: cloud data support (S3/GCS/HTTP via fsspec), multi-file datasets, Dask chunked lazy loading 2. AI integration: coordinate-aware context builder, Jinja2 prompt templates, scientific analysis subclasses 3. Pipeline support: Narwhal compatibility layer, native xarray.Dataset operations without flattening 4. Documentation: user guide, API reference, 3 tutorials, 5 gallery examples Xarray has 16M+ monthly downloads, but no framework connects it to an AI-queryable dashboard today. This project fills that gap.
Project details
Technologies
Not listed in the archive