Back to OpenVINO Toolkit
GSoC 2026

GGUF Reader V2 in OpenVINO GenAI for Direct GGUF Execution via Dynamic Graph Translation

The current GGUF reader in OpenVINO GenAI manually reconstructs model architectures by parsing metadata and building the OpenVINO model layer by layer — requiring explicit C++ implementations for every supported model topology. This project implements GGUFReaderV2, a new GGUF reader that uses dynamic graph translation instead. It leverages llama.cpp APIs to produce a GGML computation graph (ggml_cgraph), passes it through the existing GgmlOvDecoder and ov::frontend::ggml::FrontEnd infrastructure to produce an ov::Model, and integrates the result into OpenVINO GenAI's read_model() pipeline. A working prototype has already been validated with TinyLlama.gguf — successfully capturing 5,141 OpenVINO operations and executing them through the translation pipeline. Key architectural challenges around tensor naming, capture state safety, and test design have been identified and documented. Deliverables: 1. Functional GGUFReaderV2 class with correct RAII resource management and passing math equivalence tests 2. Stable GGML to OpenVINO graph translation with collision-free node mapping 3. Full integration into GenAI's read_model() API with dynamic shape support 4. Regression test suite validating accuracy across quantization formats (Q4_K, Q6_K, Q8_0) and architectures (TinyLlama, Phi-3, Llama-3, Qwen2.5) 5. Comprehensive documentation and developer guide for extending GGML operator support

Project details

Contributor

KARNAV SHAH

Mentors

Not available

Technologies

Not listed in the archive