OpenCRE Noise & Relevance Filtering Pipeline

This project addresses the challenge of distinguishing meaningful security knowledge from noisy or irrelevant changes in OWASP repositories. Currently, large volumes of repository updates include formatting changes, test files, and administrative edits, which dilute the quality of extracted knowledge and reduce the efficiency of downstream systems. To solve this, I propose building a two-stage intelligent filtering pipeline within the OpenCRE Scraper & Indexer framework. The first stage uses regex-based rule filtering to eliminate obvious noise (e.g., non-informative files and structural changes), ensuring low-cost, high-speed preprocessing. The second stage applies LLM-based semantic classification (via models like GPT-4o-mini or Gemini Flash) to determine whether a change introduces genuine security knowledge, such as new vulnerabilities, attack techniques, or mitigation strategies. This hybrid approach balances efficiency with deep contextual understanding. The system will be integrated into the OpenCRE pipeline, forwarding only relevant outputs into a knowledge queue for further processing. It will include structured logging, prompt optimization, cost-efficient API usage, and robust evaluation based on precision, recall, and accuracy. Deliverables include: 1. A production-ready two-stage filtering pipeline (regex + LLM) 2. Integration with the OpenCRE knowledge ingestion workflow 3. A labeled evaluation dataset and performance benchmarks (targeting high accuracy and recall) 4. Unit-tested, scalable, and well-documented codebase 5. Detailed documentation on prompt engineering, system design, and evaluation methodology This project will significantly improve the quality, reliability, and scalability of security knowledge extraction, enabling the OWASP community to focus on high-value insights while reducing noise in the ecosystem.