PROJECT 07 · AUTOMATIC DIFFERENTIATION
Autograd Engine
Updated: May 2026
Built a reverse-mode autodiff engine from scratch to understand computation-graph construction and execution, with emphasis on correctness, scalability, and gradient-system trade-offs.
Problem
ML frameworks hide gradient computation and graph execution. Rebuilding autograd from first principles makes their correctness and systems trade-offs observable.
Technical Highlights
Python · NumPy · Autograd · Backpropagation · Performance Analysis
- ~1e-10 gradient error
- Scales to 10,000+ nodes
- Iterative DFS avoids recursion limits
System Design
- Dynamic define-by-run computation graph
- Reverse-mode autodiff with topological traversal
- Gradient accumulation across dependency paths
- Iterative traversal for deep graphs
Architecture
Forward execution builds a dynamic graph; reverse-mode differentiation then traverses its topological order and accumulates gradients through every dependency path.
Results and Insights
- Gradient correctness validated at approximately 1e-10 error.
- Backward execution scaled approximately linearly with graph size.
- Recursive traversal reached depth limits near 1,000 nodes.
- Iterative DFS extended execution beyond 10,000 nodes.
- Flexibility trades against memory use and execution overhead.
Takeaway
Autograd performance depends on graph traversal, memory behavior, and execution overhead as well as mathematical correctness.
Backward-Pass Benchmark
Backward-pass time increases with graph size, demonstrating near-linear scaling behavior.
