Back to Apache Software Foundation
GSoC 2026

Apache Beam Add Kafka Streams Runner

This project will build a portable Kafka Streams runner for Apache Beam so that Beam pipelines can run on Kafka Streams using the Beam Fn API. The runner will translate portable Beam pipeline protos into Kafka Streams topologies and support a working skeleton for core transforms: Read, stateless ParDo, GroupByKey, Combine, non-merging windowing, and Flatten. The implementation will focus on correct watermark propagation, commit-based bundle boundaries, exactly-once semantics, and integration with the existing Beam portability stack. Deliverables include a new runners/kafka-streams module, runner entry points, a pipeline translator with transform-specific translators, watermark and bundle management, tests using TopologyTestDriver and real Kafka-based integration coverage, and documentation for the Beam community.

Project details

Contributor

Muhammad Junaid Shaukat

Mentors

Not available

Technologies

Not listed in the archive