Reddit Blocks Search Engines Except Google

In July 2024, users and reporters noticed that searching 'site:reddit.com' on any engine other than Google returned almost no recent results. The cause, first reported by 404 Media, was a robots.txt change Reddit had made around late June 2024 that blocked the search crawlers it had not reached an agreement with. DuckDuckGo, Bing, Mojeek, Qwant, and others were shut out from indexing fresh Reddit content; on DuckDuckGo, Reddit links appeared without descriptions, accompanied by a notice that the site would not allow the preview.

The one exception was Google. Earlier in 2024, Reddit and Google had signed an agreement reported by Bloomberg at roughly $60 million per year, giving Google access to Reddit content — including for AI training — and prominent placement of Reddit results in Google Search. With the new robots.txt in force, the practical result was that Google became effectively the only major search engine able to surface fresh Reddit discussions, turning one of the internet's largest user-generated knowledge bases into something close to a Google exclusive. Reddit reportedly differentiated crawlers by serving a permissive robots.txt to Google while presenting a restrictive one to others.

Reddit framed the blocks as a refusal to let crawlers access its data without enforceable commitments not to use it for AI. A spokesperson said Reddit was open to access for engines willing to commit to using the data 'without training, without summarization, and without selling it to others,' and denied that the move was financially motivated. Microsoft confirmed that Bing had been blocked and argued the arrangement was harming competition, given that the one engine retaining access was the one that had paid.

Critics found the company's denial difficult to square with the facts on the ground. The obvious pattern was that the search engine which had signed a lucrative licensing deal kept its access, while those that had not were cut off, effectively turning Google into a paid gatekeeper for discovering Reddit content. The change was part of the same 2024 robots.txt overhaul through which Reddit moved to block unauthorized AI crawlers and scrapers generally, but the collateral effect on ordinary search was distinct and far-reaching: it was not just AI bots that lost access, but the search tools millions of people use to find Reddit answers.

The consequences for information access and the open web were significant. A large share of the practical, experience-based answers people seek online live in Reddit threads, and Reddit results had become a staple of general web search precisely because users often append 'reddit' to queries to find human responses. By making that content far harder to discover outside Google, Reddit concentrated the discoverability of a major public knowledge resource behind a single commercial relationship, raising concerns that monetizing data access can quietly reduce the openness of the web for everyone who is not party to the deal.