OWASP OpenCRE Module A: Incremental Information Harvesting Pipeline

OpenCRE currently lacks an automated mechanism to continuously monitor upstream security knowledge sources such as OWASP repositories and detect meaningful documentation updates as they occur. Important changes are often introduced incrementally across commits, making manual tracking inefficient and prone to delays. This project proposes an incremental information harvesting pipeline that runs on a scheduled basis using GitHub Actions. The system will monitor configured repositories and detect recent commits using time-based Git queries, filter out non-relevant files through a noise reduction layer, and extract meaningful textual changes from diffs. These changes will be reconstructed into clean, structured artifacts and stored in an append-only raw change bucket for downstream processing. The key deliverables include a configurable repository registry, an incremental diff extraction engine, a noise filtered text parsing pipeline, structured change artifact generation, and a production ready automated workflow with logging, retry handling, and test coverage. The output of this module will serve as a reliable foundation for downstream systems such as RAG based knowledge processing pipelines.