Thursday, 3 April 2025
30 C
Singapore
34.3 C
Thailand
28.3 C
Indonesia
28.3 C
Philippines

AI startup Anthropic is accused of bypassing anti-scraping rules

Websites accuse AI startup Anthropic of bypassing anti-scraping protocols, causing disruptions and sparking debates over compliance and licensing.

In recent news, AI startup Anthropic, known for developing the Claude large language models, has accused multiple websites of disregarding their anti-scraping protocols. Freelancer and iFixit have raised concerns over Anthropic’s alleged behaviour, claiming that the company’s web crawler has been excessively active on their sites.

Freelancer’s complaints

Matt Barrie, CEO of Freelancer, has stated that Anthropicโ€™s ClaudeBot is “the most aggressive scraper by far.” Barrie said the crawler visited Freelancer’s website 3.5 million times within four hours, causing significant disruption. This traffic volume is reportedly “about five times the volume of the number two” AI crawler. Barrie noted that this aggressive scraping has negatively impacted their site’s performance and revenue. Despite initially trying to refuse access requests, Freelancer blocked Anthropicโ€™s crawler to prevent further issues.

iFixit’s experience

Kyle Wiens, CEO of iFixit, echoed similar concerns. Wiens mentioned on social media platform X (formerly Twitter) that Anthropic’s bot hit iFixit’s servers one million times within 24 hours. This high volume of requests led to considerable strain on iFixitโ€™s resources, prompting the team to set alarms for high traffic that woke them up at 3 AM due to Anthropic’s activities. The situation improved only after iFixit specifically disallowed Anthropicโ€™s bot in its robots.txt file.

This isn’t the first time an AI company has been accused of ignoring the Robots Exclusion Protocol, or robots.txt. Back in June, Wired reported that AI firm Perplexity had been crawling its website despite the presence of a robots.txt file, which typically instructs web crawlers on which pages they can and cannot access. Although adherence to robots.txt is voluntary, bad bots often need to pay more attention to it. After Wiredโ€™s report, startup TollBit revealed that other AI firms, including OpenAI and Anthropic, have also bypassed robots.txt signals.

Anthropic’s response and ongoing issues

Anthropic has responded to these accusations, telling The Information that it respects robots.txt and that its crawler “respected that signal when iFixit implemented it.” The company strives for minimal disruption by being thoughtful about how quickly it crawls the exact domains and is currently investigating the issue to ensure compliance.

AI firms frequently use web crawlers to collect content to train their generative AI technologies. However, this practice has led to multiple lawsuits from publishers accusing these firms of copyright infringement. Companies like OpenAI have started forming partnerships with content providers to mitigate the risk of further legal action. OpenAI’s content partners include News Corp., Vox Media, the Financial Times, and Reddit.

Wiens from iFixit is willing to discuss a potential licensing agreement with Anthropic, suggesting that a formal deal could benefit both parties. This approach could pave the way for a more collaborative relationship between content providers and AI developers, reducing the friction caused by unauthorised scraping activities.

Hot this week

Garmin launches premium Connect+ plan to boost health and fitness tracking

Garmin introduces Connect+ with AI insights, advanced training tools, and social features to help users reach their health and fitness goals.

Android Auto beta now supports full-screen gaming

Android Autoโ€™s latest beta introduces full-screen gaming, allowing you to play Candy Crush Soda Saga and Angry Birds 2 while parked.

ESET uncovers RansomHub links to rival gangs and highlights emerging EDR killer threats

ESET uncovers RansomHubโ€™s links to rival gangs and reveals its custom EDR killer, signalling new threats in the ransomware landscape.

Intelโ€™s future in the GPU market looks uncertain

Intel may not release a high-end Battlemage GPU, and Arc Celestialโ€™s future is unclear, leaving gamers with limited options in a challenging market.

Anthropic introduces Claude for Education, a new AI chatbot plan for universities

Anthropic launches Claude for Education, an AI chatbot plan for universities that offers advanced learning tools and administration support.

YouTube expands shopping affiliate programme in Singapore through Shopee partnership

YouTube teams up with Shopee to launch its Shopping affiliate programme in Singapore, giving creators new ways to monetise their content.

Misconceptions about STEM careers continue to deter young women in Singapore

New research shows stereotypes and lack of support are deterring young women from STEM careers, posing a risk to Singaporeโ€™s innovation goals.

Synagie and HKT launch ShopHK to help Hong Kong brands expand into Southeast Asia

Synagie and HKT launch ShopHK, helping Hong Kong SMEs tap into Southeast Asia's booming US$600 billion e-commerce market.

Informatica introduces new AI features to boost cloud data integration and management

Informatica adds AI tools to simplify data integration and improve enterprise access to AI-ready data across its cloud platform.

Related Articles