Cloudflare has publicly accused AI search startup Perplexity of deliberately bypassing website restrictions meant to block its web crawlers. In a detailed report, the internet infrastructure company claimed that Perplexity’s bots actively attempted to circumvent protections once they encountered blocks.
Website Owners Raised Concerns
According to Cloudflare, many of its customers complained that Perplexity’s bots continued to access their sites even after blocking them via robots.txt files and additional Web Application Firewall (WAF) rules. To verify the issue, Cloudflare created test domains with similar restrictions and monitored the behavior of Perplexity’s crawlers.
Perplexity Allegedly Used Stealth Tactics
Initially, Perplexity identified its crawlers using user agents like PerplexityBot” or “Perplexity-User.However, once blocked, the company allegedly began disguising its bots as Google Chrome on macOS. Cloudflare also reported that Perplexity rotated IP addresses and even changed its autonomous system numbers (ASN) to evade detection and maintain access.
Cloudflare observed that this stealth crawling affected tens of thousands of domains and generated millions of requests per day.As a result, the company removed Perplexity from its list of verified bots and introduced new tools for blocking the startup’s crawlers.
Perplexity Responds to the Accusations
In response, Perplexity spokesperson Jesse Dwyer labeled Cloudflare’s claims a “publicity stunt.” He said the blog post was full of “misunderstandings” about how the startup operates.
This disagreement highlights the growing tension between AI companies and content providers over data scraping practices. Many publishers are becoming increasingly protective of their digital content as AI tools rapidly expand their web crawling efforts.
A Broader AI Scraping Problem
Cloudflare CEO Matthew Prince emphasized the broader concern, warning that unregulated AI scraping could become an “existential threat” to publishers and digital content creators. The company has taken a strong stance, recently enabling default AI crawler blocking across its services. Cloudflare also introduced a feature allowing website owners to require payment before allowing bots to access their content.
The Ongoing Debate
This clash is part of a larger debate about the ethics and legality of AI-driven data collection. As more AI startups like Perplexity rely on publicly available web content to train their models and provide answers, the question arises: how much of that content can be used without consent?
For now, Cloudflare has responded firmly by tightening access controls and warning other companies about similar activities. Meanwhile, Perplexity continues to defend its operations and deny any intentional wrongdoing.
The dispute between Cloudflare and Perplexity highlights the tension between innovation and digital rights. While AI companies push for more data access to improve their services, infrastructure providers and content owners are demanding greater transparency and control. As the AI industry grows, these conflicts are likely to become more common—and more heated.
