PyArrow Improvements for Astronomy

The PyArrow project currently faces performance bottlenecks when used in astronomy computational work on massive datasets, such as the HATS catalog light curves. Specifically, the Apache Arrow C++ and PyArrow implementations struggle with parsing multi-level nested Parquet files due to underutilized multi-threading, lack the ability to selectively load specific fields from deeply nested structures, and feature compute kernels that do not support nested datatypes. This project aims to resolve these core issues to drastically improve load times and resource efficiency for the astronomy community.