Back to CERN-HSF
GSoC 2026

Apache Arrow Interface for PODIO

This project aims to implement a high-performance Apache Arrow backend for PODIO. Currently, PODIO data is primarily persistent in ROOT and SIO formats, which can introduce significant I/O overhead for modern, language-independent analytics. The Problem In high-energy physics, data analysis often requires processing only specific subsets of complex event structures. Traditional file-based backends often require loading entire object hierarchies, leading to memory and I/O bottlenecks. The Solution The proposed solution implements a dual-path ingestion strategy within PODIO's code generation engine In-Memory Zero-Copy Access: Utilizing arrow::ListView to map existing PODIO memory buffers directly into Arrow tables for real-time, language-independent analysis. Serialized Normalization: Implementing a normalization layer that packs scattered memory blocks into contiguous arrow::ListArray structures to satisfy the strict requirements of Parquet storage. Relational Mapping: Resolving C++ pointers into a Composite Key (Collection ID + Index) to maintain complex object relationships across different platforms and memory layouts. Key Deliverables Core Arrow Backend: A C++ implementation integrated into PODIO's generator to translate YAML-defined models into Arrow’s columnar format. Parquet Support: Out-of-the-box serialization to industry-standard Parquet files. Round-Trip Validation: A robust testing suite ensuring 100% data parity when converting between PODIO Frames and Arrow Tables. Performance Benchmarking: A side-by-side comparison of throughput and disk footprints between ROOT TTree and the new Arrow/Parquet backend.

Project details

Contributor

Arnav Dham

Mentors

Not available

Technologies

Not listed in the archive