GSoC 2026
Parallel Scan Task Execution in Apache Iceberg Spark Readers
Apache Iceberg's Spark readers process scan tasks sequentially, causing significant overhead for small-file workloads on object storage where file open latency dominates. This idea introduces an optional parallel execution path in BaseReader by decoupling task opening from record consumption using a producer-consumer model built on Iceberg's existing ParallelIterable. A working prototype (PR #15341) benchmarked on AWS S3 shows 76.4% reduction in execution time for 1000 small files. Deliverables include a configurable async execution path, unit and integration tests, and documentation.
Project details
Technologies
Not listed in the archive