OpenAI tests AI governance

OpenAI said its ChatGPT fetch bot may not be governed by robots.txt restrictions, challenging one of the web’s long-standing conventions for automated access. Robots.txt is not a legal access-control system, but publishers, search engines and crawlers have used it for decades as a practical boundary-setting mechanism. If AI agents claim different treatment than search crawlers, website owners may have to rely more on authentication, paywalls or litigation to control data use. The issue matters because AI assistants increasingly fetch live pages on behalf of users, blurring the line between browsing, scraping and model-driven data extraction.

Sources from the briefing

• OpenAI Says Robots.txt May Not Apply To ChatGPT’s Fetch Bot - Search Engine JournalSearch Engine Journal