PROJECT 08 · REPRODUCIBILITY
ML Reproducibility Auditor
Updated: May 2026
Built a CLI-based system that uses GitHub API analysis to evaluate ML repositories for reproducibility, engineering quality, and systems-design patterns.
Problem
Many ML repositories lack reproducible environments, explicit seed control, and engineering discipline. Assessing their reliability otherwise requires manual inspection.
Technical Highlights
Python · CLI · GitHub API · Static Analysis · ML Systems
- Reproducibility scoring system
- GitHub API-based analysis
- ML systems pattern detection
System Design
- Repository inspection through the GitHub API without cloning
- Structure analysis for CI/CD, benchmarks, datasets, and packaging
- Code-quality and determinism checks
- Pattern detection for PyTorch, distributed training, and all-reduce
- Scoring with reproducibility and risk classification
Architecture
The tool fetches repository metadata, analyzes structure and code signals, scores reproducibility, classifies risk, and generates actionable findings.
Results and Insights
- Identified missing reproducibility signals such as CI/CD and seed control.
- Detected system patterns across real-world ML repositories.
- Automated evaluation of engineering maturity.
- Enabled comparison of infrastructure practices across projects.
Takeaway
Reproducibility in ML systems depends on engineering practices, not only model design.
Example Output
Reproducibility Score: 7.5/10 Risk Level: MEDIUM Missing CI/CD → Not automatically validated Missing seed control → Not reproducible