Native Streaming Transforms for the Apache Beam Python SDK
Beam's Python SDK is increasingly the first choice for ML and data-intensive pipelines, making robust native streaming APIs more important than ever. However, two essential streaming primitives, UnboundedSource (Issue #19137) and Watch (Issue #21521), remain unavailable in Python despite being long established in the Java SDK. Today, Python developers who need these capabilities either pull in cross-language transforms that add Java dependencies and gRPC overhead, or wrestle with low-level RestrictionTracker internals. This project solves the problem by porting both primitives natively to Python on top of Beam's existing SDF framework. The UnboundedSource wrapper adapts the legacy reader API into a Splittable DoFn that handles checkpointing, watermark progression, deduplication, and splitting. The Watch transform adds periodic polling with composable termination conditions and stable dedup behavior. Porting both primitives natively abstracts away that complexity behind clean and Pythonic interfaces. It also unlocks native enhancements, such as updating fileio.MatchContinuously, without leaving the Python ecosystem. The planned deliverables are: (D1) a Python UnboundedSource API with its SDF-based wrapper, (D2) a native Watch transform with PollFn and TerminationConditions, (D3) a test suite spanning DirectRunner and Dataflow, and (D4) supporting documentation including docstrings, programming-guide updates, and migration notes.
Project details
Technologies
Not listed in the archive