ShieldFont Counters AI Scrapers Disregarding Robots.txt
TL;DR. A new method, ShieldFont, uses ligatures to make scraped text nonsensical for AI models while appearing normal to human readers. - ShieldFont transforms about a quarter of words into ligature-based versions, grammatically valid but semantically broken for LLM scraping. - This technique aims to increase the cost and complexity for AI scrapers that ignore standard robots.txt directives. - It highlights ongoing efforts to protect website content from unauthorized use for AI training data.
- ShieldFont uses ligatures to alter website text, making it nonsensical for AI scrapers but readable for humans.
- The method targets LLM training data scrapers that disregard robots.txt protocols.
- It works by substituting about 25% of words with ligature-based versions, which are grammatically correct but semantically broken.
- This increases the cost and effort for AI scrapers by forcing them to parse rendered text.
- ShieldFont is one of several evolving tools, alongside maze traps, designed to combat unwanted AI data harvesting.