Debezium Source Connector for Milvus
Milvus currently lacks a Debezium source connector, meaning data can flow into Milvus (via sink connectors) but not out of it into the broader data ecosystem. This critical gap forces users into manual workarounds for data replication, auditing, and prevents the creation of hybrid pipelines that join vector search results with relational data. The plan is to implement a Hybrid strategy focused exclusively on the future-proof Milvus 2.6+ StreamingNode architecture. This approach will leverage two separate sources to guarantee a complete and ordered change stream: Woodpecker Streaming Service (WAL): Used as the only source for all Data Manipulation Language (DML) events (inserts and deletes). Etcd Watch: Used as the cleanest source for all Data Definition Language (DDL) events (schema changes), which simplifies parsing and aligns with Debezium's schema history mechanism.
Project details
Technologies
Not listed in the archive