PROJECT 06 · DISTRIBUTED SYSTEMS
Distributed Training Simulator
Updated: May 2026
Built a distributed training simulator to analyze scaling efficiency, communication overhead, and the point where synchronization cost overtakes useful computation.
Problem
Adding workers does not guarantee faster training: model size, collective strategy, and network bandwidth determine when communication dominates each step.
Technical Highlights
Python · Distributed Systems · All-Reduce · Performance Modeling
- ~0.77 efficiency for small models
- Compute-to-communication transition near eight workers
- Negative scaling for 1000 MB models
System Design
- Data-parallel training with worker-level computation
- Ring and tree all-reduce communication strategies
- Step-time, speedup, and efficiency modeling
- Communication ratio and bottleneck detection
Architecture
Compute and collective-communication models combine worker count, model size, and network bandwidth to estimate step time, speedup, efficiency, and bottleneck regime.
Results and Insights
- Small models achieved near-linear scaling at ~0.77 efficiency.
- Workloads became communication-bound near eight workers.
- 1000 MB models showed negative scaling from communication cost.
- Bandwidth materially changed scalability at 1 GB/s versus 100 GB/s.
- Ring all-reduce outperformed tree at higher worker counts.
Takeaway
Distributed training stops scaling efficiently when communication overhead begins to dominate computation.
Communication Ratio
Scaling Behavior
