New Research Reimagines MoE Scaling Laws for Cluster Efficiency

TL;DR. Researchers introduce MOSAIC, a framework that optimizes Mixture-of-Experts models for cluster-specific throughput, moving beyond traditional FLOP-based scaling laws. - The paper highlights that compute-optimal designs often differ significantly from cluster-optimal designs for large model training. - MOSAIC integrates system-level factors like parallel layouts and cluster efficiency into the model architecture optimization process. - This approach means ideal MoE sparsity and token budgets vary based on the specific hardware infrastructure used for training.

• Traditional scaling laws optimize for FLOPs, but clusters bill for GPU-hours, leading to inefficiencies.
• The MOSAIC framework folds system-level optimization into scaling laws, considering cluster-specific throughput.
• Mixture-of-Experts (MoEs) are particularly affected, as their sparsity and expert split need cluster-aware tuning.
• Two models with identical loss-per-FLOP can have vastly different loss-per-GPU-hour performance.

Back to QLANKR News