Verifier Exposes High Failure Rate in LLM-Generated GPU Kernels
TL;DR. Researchers developed a new verifier that found significant correctness issues in GPU kernels generated by large language models, indicating current testing methods are insufficient. - The 'contract-grade' verifier identifies silent failures in nearly two-thirds of previously accepted LLM-generated kernels. - It uses twelve adversarial gates to test properties, including tolerance-free checks for accuracy and robustness. - This research suggests current correctness metrics for AI-generated code are weaker than reported, impacting AI compute reliability.
- A new 'contract-grade' verifier found 62.1% of LLM-generated GPU kernels had at least one correctness violation.
- The verifier employs 12 adversarial gates, several of which are tolerance-free, to rigorously test kernel properties.
- Current standard testing methods accepted 1,487 kernels that the new verifier rejected, highlighting a significant gap in reliability assessment.
- The findings suggest reported progress in AI-driven kernel generation relies on flawed correctness metrics.