Thursday, 3 April 2025
26.2 C
Singapore
29.3 C
Thailand
21 C
Indonesia
26.9 C
Philippines

AI startup Anthropic is accused of bypassing anti-scraping rules

Websites accuse AI startup Anthropic of bypassing anti-scraping protocols, causing disruptions and sparking debates over compliance and licensing.

In recent news, AI startup Anthropic, known for developing the Claude large language models, has accused multiple websites of disregarding their anti-scraping protocols. Freelancer and iFixit have raised concerns over Anthropic’s alleged behaviour, claiming that the company’s web crawler has been excessively active on their sites.

Freelancer’s complaints

Matt Barrie, CEO of Freelancer, has stated that Anthropicโ€™s ClaudeBot is “the most aggressive scraper by far.” Barrie said the crawler visited Freelancer’s website 3.5 million times within four hours, causing significant disruption. This traffic volume is reportedly “about five times the volume of the number two” AI crawler. Barrie noted that this aggressive scraping has negatively impacted their site’s performance and revenue. Despite initially trying to refuse access requests, Freelancer blocked Anthropicโ€™s crawler to prevent further issues.

iFixit’s experience

Kyle Wiens, CEO of iFixit, echoed similar concerns. Wiens mentioned on social media platform X (formerly Twitter) that Anthropic’s bot hit iFixit’s servers one million times within 24 hours. This high volume of requests led to considerable strain on iFixitโ€™s resources, prompting the team to set alarms for high traffic that woke them up at 3 AM due to Anthropic’s activities. The situation improved only after iFixit specifically disallowed Anthropicโ€™s bot in its robots.txt file.

This isn’t the first time an AI company has been accused of ignoring the Robots Exclusion Protocol, or robots.txt. Back in June, Wired reported that AI firm Perplexity had been crawling its website despite the presence of a robots.txt file, which typically instructs web crawlers on which pages they can and cannot access. Although adherence to robots.txt is voluntary, bad bots often need to pay more attention to it. After Wiredโ€™s report, startup TollBit revealed that other AI firms, including OpenAI and Anthropic, have also bypassed robots.txt signals.

Anthropic’s response and ongoing issues

Anthropic has responded to these accusations, telling The Information that it respects robots.txt and that its crawler “respected that signal when iFixit implemented it.” The company strives for minimal disruption by being thoughtful about how quickly it crawls the exact domains and is currently investigating the issue to ensure compliance.

AI firms frequently use web crawlers to collect content to train their generative AI technologies. However, this practice has led to multiple lawsuits from publishers accusing these firms of copyright infringement. Companies like OpenAI have started forming partnerships with content providers to mitigate the risk of further legal action. OpenAI’s content partners include News Corp., Vox Media, the Financial Times, and Reddit.

Wiens from iFixit is willing to discuss a potential licensing agreement with Anthropic, suggesting that a formal deal could benefit both parties. This approach could pave the way for a more collaborative relationship between content providers and AI developers, reducing the friction caused by unauthorised scraping activities.

Hot this week

The Tesla Model Y now comes in a 110kW version that qualifies for Cat A COE

Tesla launches the Model Y 110 in Singapore, a budget-friendly EV with Cat A COE. Discover its features, price, and availability.

Apple may launch an AI-powered Health app with a coaching feature next year

Apple may introduce an AI-powered Health app with coaching, food tracking, and fitness guidance in 2026, possibly as a new subscription service.

Samsung Galaxy A06 5G offers modern features at an affordable S$228

The Samsung Galaxy A06 5G, with a 50MP camera and 5,000mAh battery, launches in Singapore on March 21, 2025, for S$228.

Google’s Gemini 2.5 Pro AI model is now available for all users

Google's Gemini 2.5 Pro AI model is now available for all users, offering advanced coding and reasoning abilities with a free trial for Gemini Advanced.

Intelโ€™s future in the GPU market looks uncertain

Intel may not release a high-end Battlemage GPU, and Arc Celestialโ€™s future is unclear, leaving gamers with limited options in a challenging market.

Qualcomm expands AI research with MovianAI acquisition

Qualcomm has acquired Vietnamese AI research firm MovianAI to boost its AI development in smartphones, PCs, and software-defined vehicles.

Roblox introduces new parental controls to enhance child safety

Roblox introduces new parental controls, allowing parents to block games, restrict friends, and monitor their childโ€™s activity for better safety.

Anthropic introduces Claude for Education, a new AI chatbot plan for universities

Anthropic launches Claude for Education, an AI chatbot plan for universities that offers advanced learning tools and administration support.

Exabeam introduces Nova, an agentic AI that boosts cybersecurity operations

Exabeam unveils Nova, a proactive AI agent that boosts security team productivity and reduces incident investigation time by over 50%.

Related Articles