Reddit's Upcoming Changes Aim to Protect Platform from AI Crawlers

Cover Image

Reddit app icon

Reddit has announced an update to its Robots Exclusion Protocol (robots.txt file), which informs automated web bots whether they are allowed to crawl a site. The move is designed to safeguard the platform against AI crawlers.

Historically, the robots.txt file was used to permit search engines to scrape a site and direct users to the content. However, with the rise of AI, websites are being scraped and used to train models without acknowledging the original source of the content.

In addition to the updated robots.txt file, Reddit will continue to rate-limit and block unknown bots and crawlers from accessing its platform. The company stated that bots and crawlers will be rate-limited or blocked if they don't comply with Reddit's Public Content Policy and don't have an agreement with the platform.

Reddit claims the update shouldn't affect the majority of users or good faith actors, such as researchers and organizations like the Internet Archive. Instead, the update is designed to deter AI companies from training their large language models on Reddit content. However, AI crawlers could ignore Reddit's robots.txt file.

The announcement comes after a recent investigation found that AI-powered search startup Perplexity has been stealing and scraping content, ignoring requests not to scrape its website. Perplexity's CEO responded to the claims, stating that the robots.txt file is not a legal framework.

Reddit's upcoming changes won't affect companies that have an agreement with the platform. For instance, Reddit has a $60 million deal with Google that allows the search giant to train its AI models on the social platform's content. With these changes, Reddit is signaling to other companies that want to use Reddit's data for AI training that they will have to pay.

"Anyone accessing Reddit content must abide by our policies, including those in place to protect redditors," Reddit said in a blog post. "We are selective about who we work with and trust with large-scale access to Reddit content."