AI crawlers are outpacing publishers' defenses

The latest State of the Bots report from TollBit highlights the growing challenge publishers face in blocking unauthorized AI crawlers. Jonathan Roberts, Chief Innovation Officer at People Inc., explains that the company actively blocks unauthorized bots while allowing access to legitimate crawlers, typically through licensing agreements. However, Roberts acknowledges that identifying and blocking these unauthorized bots has become increasingly difficult. The report notes that some malicious bots attempt to mimic legitimate ones, such as Google’s, often rotating IP addresses to evade detection. Some industrial-scale scraping operations use networks of devices in people’s homes to mimic real user traffic, making it harder to distinguish between genuine and malicious activity. Despite these efforts, People Inc. has not been entirely successful in blocking all unauthorized crawlers. The report emphasizes that the battle over bot access should not define the future of publishing in the AI era. Instead, the focus should shift from content scraping to how that content is used. Publishers should consider how content surfaces for end users and what measures can be taken to preserve value and enforce good behavior. The media-AI conflict has moved away from training data, as the real threat lies in information retrieval—people using AI as a discovery tool. Training large language models is costly and primarily aimed at improving predictive capabilities, not providing accurate information. Early AI chatbots, for example, provided directionally correct answers but often struggled with specific details like names and dates.