Build a GUI Agent with Local LLM/VLM using OpenVINO for Desktop Automation

A lot of everyday computer tasks—like navigating applications, managing files, and configuring settings—are repetitive and require multiple manual steps. Existing automation solutions rely on brittle scripts or cloud-based AI systems, which introduce latency, privacy concerns, and limited adaptability. This project aims to build a native desktop GUI agent that can understand natural language instructions and autonomously operate the computer interface. The system will combine Vision-Language Models (VLMs) for screen understanding and Large Language Models (LLMs) for reasoning and task planning, all running locally using OpenVINO for efficient, low-latency inference. The proposed solution uses a multi-agent architecture consisting of a Router, Planning, UI Grounding, Action Execution, and Reflection agents. The VLM (Phi-3.5 Vision) will interpret screenshots and map UI elements to coordinates, while the LLM (DeepSeek-R1 Distill Qwen 7B) will decompose tasks and generate action sequences. A PyQt-based desktop interface will allow users to interact with the agent, monitor execution, and configure settings.