Reddit Takes a Stand Against AI Crawlers
Reddit recently announced changes to its Robots Exclusion Protocol, also known as its robots.txt file. This update is part of a larger negotiation between AI companies and content owners.
The robots.txt file communicates to third parties how a website can be crawled, such as allowing Google to crawl for search results. However, when it comes to AI, the value exchange is less clear. Websites attract clicks and eyeballs, but AI companies hoover up content without sending traffic or compensating the owners.
By changing its robots.txt file and continuing to rate limit and block unknown bots and crawlers, Reddit appears to be preventing practices criticized in the case of Perplexity AI.
Watch the video to learn more and share your thoughts in the comments!
Get All The Latest Updates Delivered Straight To Your Inbox For Free!