Back to Apache Software Foundation
GSoC 2026

Parallel Scan Task Execution in Apache Iceberg Spark Readers

Apache Iceberg's Spark readers process scan tasks sequentially, causing significant overhead for small-file workloads on object storage where file open latency dominates. This idea introduces an optional parallel execution path in BaseReader by decoupling task opening from record consumption using a producer-consumer model built on Iceberg's existing ParallelIterable. A working prototype (PR #15341) benchmarked on AWS S3 shows 76.4% reduction in execution time for 1000 small files. Deliverables include a configurable async execution path, unit and integration tests, and documentation.

Project details

Contributor

Varun Lakhyani

Mentors

Not available

Technologies

Not listed in the archive