Generative-AI Assisted Testing of Complex Stacks of Spack Packages

The Problem: Spack’s combinatorial flexibility makes exhaustive CI testing computationally impossible. Current CI focuses almost exclusively on "leading-edge" configurations (latest packages with latest dependencies). This structurally excludes off-leading-edge scenarios where older packages meet newer, unbounded dependencies, leaving the ecosystem highly vulnerable to silent API/ABI breakages that users only discover in production. The Solution: This project introduces a Generative AI-assisted testing system to autonomously hunt for high-risk configuration breakages. The architecture treats configuration testing as a feedback-driven sequential decision-making pipeline. First, a data-driven heuristic risk model filters dependency pairs based on structural risk and historical failures. Next, an LLM acts as an iterative hypothesis generator to propose structurally diverse, high-risk abstract specs. Finally, a strict deterministic validation layer ensures mathematical correctness and bounds CI compute costs using Spack's Clingo ASP concretizer and semantic deduplication. Key Deliverables: Metadata Extraction Module: Parses package repositories to identify unbounded dependencies and analyze version gaps. Adaptive Risk Scoring Engine: Classifies package pairs into risk tiers using a multiplicative formula driven by structural signals and historical failure data. LLM Hypothesis Generator: An iterative prompt engine utilizing selective context retrieval and within-run adaptation to propose risky testing configurations. Validation Pipeline: A strict gating system utilizing spack.concretize_one() and 3-layered semantic deduplication to prevent redundant or hallucinated tests. State Management Registry: A unified SQLite database to map historical CI data, track tested configurations, and feed failure patterns back into the LLM context. Integration & Diagnostics: A working spack audit ai CLI subcommand and a heuristic log-triage pipeline.