Enhancing Nirjas & Atarashi for Accurate, Scalable License Intelligence

This project will enhance FOSSology's license intelligence stack by revamping both Nirjas and Atarashi into a faster, ML-driven, and language-aware pipeline. Nirjas will move from regex-oriented parsing to a Tree-sitter architecture with language packs across 248 languages, improving comment extraction fidelity in real repositories. Atarashi will be upgraded from broad full-file matching toward retrieval-plus-classification that emphasizes high-signal fragments, because keywords matter for license intent (for example, phrases like permission grant, redistribution, warranty disclaimer, copyleft, and patent terms are often decisive). To support both components, I have already upgraded the Minerva dataset pipeline to generate train/validation/test datasets for Nirjas and Atarashi using merged inputs from ScanCode LicenseDB, FOSSology licenseRef.json endpoint, and The Stack Smol comment corpus, followed by augmentation, LLM-generated hard-negative synthesis, and near-dedup. A sample dataset for benchmarking and integration testing is ready, while the full-scale dataset build is in progress. For modeling, Nirjas will use a model2vec classifier distilled from strong sentence-transformer teachers selected using MTEB/RTEB task performance together with our internal evaluations; the same model2vec distillation path is also planned for Atarashi, with a fine-tuned quantized embedding baseline retained as a fallback depending on speed/accuracy trade-offs. The expected outcome is a production-ready path to more accurate, scalable, and maintainable license detection across diverse codebases.