Handwritten Text Recognition for early modern Spanish documents with Vision Language Model creation

This project builds an end-to-end Handwritten Text Recognition (HTR) pipeline for early modern Spanish manuscripts by placing a Vision-Language Model (VLM) at the center of every processing stage, not just as a late-stage corrector. The pipeline uses Qwen2.5-VL with LoRA-based multi-task fine-tuning across four stages: document analysis, literal reading, line-level OCR via a fine-tuned TrOCR model, and reconciliation/correction, where the VLM reconciles multiple reading sources using visual and contextual evidence. As a pre-GSoC contribution, I have built and validated this pipeline, achieving a median CER of 0.107 (~89.3% character accuracy) on the Rodrigo dataset with perfect transcription on well-preserved pages. The proposed GSoC work will transform this into a production-ready, multi-model application by abstracting the VLM backend to support multiple open-source models (Qwen2.5-VL 3B/7B/72B, InternVL), adding batch processing for multi-page manuscripts with PDF support, building a locally-run desktop application with a Gradio interface, packaging the pipeline as a pip-installable CLI tool, conducting comprehensive benchmarks and ablation studies across all model backends, releasing fine-tuned model weights on HuggingFace Hub, and providing a training pipeline for researchers to fine-tune on their own manuscript collections. The fully local execution model ensures data privacy for sensitive archival material, while the multi-model architecture allows users to select models based on their hardware constraints and accuracy requirements.