Major AI firms support new standard for opting out of scraping

OpenAI, Google, Microsoft, and Anthropic have announced they will adopt a new standard allowing publishers to opt out of being scraped by AI systems. This initiative, reported by Search Engine Roundtable, enables website operators to apply their AI training preferences across multiple AI companies rather than setting them individually. The shared 'policy token' signal, proposed by Kevin Kelley, a technical lead at Anthropic, is part of the Internet Engineering Task Force (IETF). It will be integrated into a website's robots.txt file, which traditionally signals which web crawlers can scrape a site. This development aims to provide greater control over how content is used for AI training. The move is seen as a significant step toward standardizing AI ethics and data usage practices across the industry. The new standard is designed to simplify the process for publishers to communicate their preferences to participating AI companies. By leveraging the existing robots.txt framework, the policy token offers a centralized method for managing scraping permissions. This approach is expected to reduce the complexity for website operators who previously had to manage multiple settings for different AI systems. The collaboration among major AI firms highlights a growing consensus on the need for transparency and control in AI data practices. As the technology continues to evolve, such standards are likely to play a crucial role in shaping the ethical landscape of AI development.