Stochastic Rounding Improves BF16 Training Accuracy
TL;DR. New research shows stochastic rounding, rather than round-to-nearest, prevents errors from compounding in AI model training, matching FP32 precision with less memory. - Round-to-nearest rounding causes errors to accumulate, hindering model convergence over long training runs. - Stochastic rounding introduces zero-mean errors that largely cancel out, leading to more accurate results. - This method allows BF16 precision to achieve FP32 accuracy in optimizer states, saving memory and bandwidth. - The technique was validated using a small MLP on a teacher-student regression task with AdamW optimizer.
- Round-to-nearest in BF16 causes consistent errors that compound during AI model training.
- Stochastic rounding introduces varying, zero-mean errors that cancel over time, preventing error accumulation.
- Using stochastic rounding with BF16 for optimizer states achieves accuracy comparable to FP32, reducing memory and bandwidth.
- Experimental results on an MLP demonstrate improved training outcomes and loss reduction with stochastic rounding.
Sources
- Bias Compounds, Variance Washes Out — convergentthinking.sh