New API Rates AI Model Hallucination Scores
TL;DR. A new API offers per-token hallucination scores for frontier language models by using a white-box proxy model to infer internal confusion. - The API works by routing black-box LLM queries through an observable open-weight model. - This proxy model is trained against human-judged GPT-5.4 Nano outputs to identify hallucinated claims. - Scores indicate the likelihood of the proxy model being confused, agreeing with human judges over 95% of the time.
- API provides per-token hallucination scores for LLM outputs.
- Utilizes a white-box proxy model (e.g., Gemma-4) to infer internal states of black-box models.
- Proxy model trained on GPT-5.4 Nano outputs with human-labeled hallucinations.
- Agreement between probe and judge LLM exceeded 95% on validation sets.
- Allows real-time highlighting of suspect spans during model generation.