Study Audits AI Assistant Restaurant Recommendations in Bali
TL;DR. A new study evaluated four AI assistants' venue recommendations against a complete census of restaurants, cafes, and bars in two Bali regions. - Researchers found 85.6% of venues were never recommended by any system, even established ones. - Documentation, not star ratings, determined whether a venue appeared in recommendations; ratings influenced rank. - AI systems rarely fabricated venues but frequently recommended permanently closed establishments, highlighting data staleness.
- A comprehensive audit found AI assistants recommended 14.4% of venues in Bali.
- Venue visibility in AI recommendations was linked to having a website, listed prices, and third-party web mentions, not star ratings.
- Among recommended venues, higher star ratings did predict a better rank in results.
- AI systems primarily failed by recommending closed venues due to outdated data, rather than outright fabrication.
- Cross-system agreement on recommendations was low, and results showed significant sampling stochasticity.