Make Wayang More Datalake-Friendly
Apache Wayang currently supports batch engines (Spark, Flink) and OLTP databases via JDBC, but lacks integration with modern data lake engines — Trino, BigQuery, Dremio, and AWS Athena — where most analytical workloads now run. This project integrates these engines into Wayang's optimizer and executor through a shared JDBC Template abstraction layer, so each new engine inherits common infrastructure (connection management, SQL assembly, channel conversions) and only overrides engine-specific behavior. The deliverables are: (1) platform, executor, and plugin implementations for Trino, BigQuery, Dremio, and Athena; (2) operator mappings that translate Wayang logical operators (filter, projection, aggregate, join) into engine-native SQL; (3) per-engine cost models calibrated via TPC-H benchmarks so the optimizer can score data lake engines against Spark/Postgres alternatives; (4) end-to-end demo pipelines and contributor documentation. The approach has already been verified with working Trino, BigQuery, and Dremio demonstrations for table scan, filter, and projection operators.
Project details
Technologies
Not listed in the archive