Back to Apache Software Foundation
GSoC 2026

Make Wayang More Datalake-Friendly

Apache Wayang currently supports batch engines (Spark, Flink) and OLTP databases via JDBC, but lacks integration with modern data lake engines — Trino, BigQuery, Dremio, and AWS Athena — where most analytical workloads now run. This project integrates these engines into Wayang's optimizer and executor through a shared JDBC Template abstraction layer, so each new engine inherits common infrastructure (connection management, SQL assembly, channel conversions) and only overrides engine-specific behavior. The deliverables are: (1) platform, executor, and plugin implementations for Trino, BigQuery, Dremio, and Athena; (2) operator mappings that translate Wayang logical operators (filter, projection, aggregate, join) into engine-native SQL; (3) per-engine cost models calibrated via TPC-H benchmarks so the optimizer can score data lake engines against Spark/Postgres alternatives; (4) end-to-end demo pipelines and contributor documentation. The approach has already been verified with working Trino, BigQuery, and Dremio demonstrations for table scan, filter, and projection operators.

Project details

Contributor

Jun Wang

Mentors

Not available

Technologies

Not listed in the archive