Characterizing I/O Performance for ML Data Loaders at Scale using Darshan

Modern machine learning workflows in High Energy Physics increasingly rely on large-scale datasets, where the data-loading stage can become a primary bottleneck for performance and scalability, particularly due to the lack of visibility into the I/O behavior of data-loading pipelines. While model performance is extensively studied, data ingestion remains comparatively under-characterized and often limits scalability due to inefficient access patterns, filesystem contention, and poor interaction between data formats and loading strategies. This project addresses this gap by developing a reproducible benchmarking and analysis framework for characterizing data-loading performance in ML pipelines. The system will integrate the Darshan I/O characterization tool to capture system-level access patterns and combine them with application-level metrics such as throughput and runtime. It will support multiple data formats and compare different loading approaches, including the native PyTorch DataLoader and a ROOT-based loader. The methodology consists of controlled experiments that vary data formats, loader configurations, and execution parameters. Results will be processed through an analysis layer to identify recurring patterns and bottlenecks, and a diagnostic component will detect inefficiencies and provide actionable recommendations. The main deliverables of the project are: - A modular CLI-based benchmarking system for dataset generation and ingestion, configurable experiment execution, and structured result collection. It will enable reproducible experiments across data formats (NumPy .npz, HDF5, CSV, ROOT RNTuple), loader implementations, and execution configurations. - Integration of Darshan for automated I/O profiling and trace collection - A curated set of benchmark experiments covering representative combinations of data formats, loader implementations, and configuration parameters - Documentation and a final report summarizing findings and best practices