Nvidia X-Token Outperforms GOLD in Cross-Tokenizer KD on Llama-3.2-1B
TL;DR. Nvidia introduced X-Token, a new projection-guided cross-tokenizer knowledge distillation method, significantly improving Llama-3.2-1B performance. - X-Token provides a drop-in replacement for standard KD loss without requiring architectural changes or auxiliary trainable components. - It addresses limitations in prior cross-tokenizer KD methods like GOLD, particularly with uncommon token fragmentation. - The method utilizes P-KL and H-KL loss formulations to achieve superior accuracy in various benchmarks.
- NVIDIA developed X-Token, a projection-guided cross-tokenizer knowledge distillation (KD) method.
- X-Token improves upon previous state-of-the-art method GOLD by +3.82 average points on Llama-3.2-1B.
- It resolves issues with token fragmentation and over-conservative matching in existing cross-tokenizer KD approaches.
- The method requires no auxiliary trainable components or architectural changes, operating as a direct replacement for standard KD loss.