Dharma-AI Boosts Cluster Utilization 33 Points With Order Change
TL;DR. Dharma-AI improved GPU cluster utilization by 33 percentage points for its Dharma-OCR-LITE model by optimizing the order of inference requests. - Engineers discovered that processing image groups by size, rather than random order, significantly reduced resource waste. - The change addressed underutilization issues caused by variability in image processing times impacting batching efficiency. - This optimization allowed the same hardware to handle a substantially higher volume of image processing tasks.
- Dharma-AI achieved a 33% increase in GPU cluster utilization for its OCR model.
- The improvement stemmed from reordering inference requests based on image size.
- This simple change led to more efficient batching and reduced idle compute resources.
Sources
- Same Cluster, 33 Points More Utilization: What Changed Was the Order — huggingface.co