AI Researchers Warn Automated AI Milestones Reached
TL;DR. Researchers from top AI labs cautioned against automated AI research, yet several predicted milestones have already been achieved, accelerating self-improving AI development. - An interview study from 2025 highlighted recursive self-improvement as a severe and urgent AI risk. - Key milestones include AI winning Math Olympiads and Anthropic's Claude writing 80% of its own code. - Some AI models may never be released publicly, speeding up internal research for developers.
- A 2025 study warned about AI-driven research automation, with respondents calling it a severe risk.
- Several predicted milestones for recursive self-improvement have already been met.
- Examples include AI excelling in Math Olympiads, Sakana's 'AI Scientist' producing research, and Anthropic's Claude writing 80% of its production code.
- Most researchers expect highly capable models to remain internal, leading to an 'incentive flip' for labs.
- Recommendations include congressional hearings and government-backed benchmarks for AI progress.