Integrate IRx with Apache Arrow (C Data Interface)
This proposal integrates IRx with the Apache Arrow C Data Interface (CDI) to enable zero-copy interoperability with systems such as PyArrow and DuckDB. Currently, IRx lacks a complete Arrow export path, supports only int32, and contains a critical issue where finish() hardcodes type metadata, leading to silent data corruption for other types. The project introduces a full Arrow lowering pipeline covering all fixed-width types and UTF-8 strings, implements a correct ArrowArray export mechanism, and resolves type handling through a generalized type_id-based runtime design. This ensures that future type additions remain localized and scalable without modifying shared logic. The outcome is a production-ready integration that allows IRx programs to exchange data directly with Arrow-compatible systems without copying or serialization, enabling its use in real-world data processing pipelines.
Project details
Technologies
Not listed in the archive