Behavioral evals, Quality, and the OSS Community

This proposal focuses on improving the behavioral evaluation (eval) infrastructure of Gemini CLI to ensure reliable and safe AI tool usage. The current system has significant gaps, including missing eval coverage for ~37.5% of tools, a large number of non-CI-blocking (flaky) tests, and lack of tooling for debugging and contributing to evals. To address this, the proposal introduces a three-part approach: Expand and Stabilize Behavioral Evals Write 15–20 new evals covering uncovered tools and missing behaviors (error recovery, multi-step reasoning, safety, ambiguity handling). Stabilize flaky evals and promote reliable ones to CI-blocking status. Focus on testing tool-call decisions instead of text output for robustness. Enable Contributor Participation via Skills & Subagents Introduce an eval-writer skill to help generate evals. Build subagents like eval_investigator (debugging failures) and quality_analyst (coverage analysis). Make eval development accessible to open-source contributors. Fix Ecosystem Gaps (Process, Tooling, Documentation) Define clear eval review, promotion, and regression-handling processes. Improve logging with readable tool-call traces. Create the Eval Development Kit (EDK) with tools for: Eval inventory & coverage analysis Validation checks Auto-generating evals from real chat logs Add documentation, tutorials, and GitHub automation for easier community contributions. Overall, the goal is to transform evals from a limited internal system into a scalable, contributor-friendly quality framework, enabling the OSS community to actively improve AI behavior, reliability, and performance.