AI Harnesses Yield Varied Code from Same LLM
TL;DR. A hands-on experiment showed five AI coding harnesses produced wildly different outputs despite using an identical prompt and the same underlying LLM. - The experiment tested Codex, PI agent, Jcode, OpenCode, and Reasonix, all powered by DeepSeek V4, on a simple browser toy prompt. - The quality and availability of feedback loops, like headless browser testing, proved more critical than raw code generation for successful output. - Each harness demonstrated unique approaches and varying degrees of functionality and complexity in their generated code for the jelly blob.
- AI coding harnesses produce distinct results even with identical prompts and LLMs.
- Feedback mechanisms like headless browsers significantly influence code quality.
- The experiment highlights the 'eyes and hands' role of harnesses for LLMs.