Reddit Blocks Search Engines Except Google
July 2024
By July 2024 Reddit's robots.txt changes had blocked Bing, DuckDuckGo, Mojeek, Qwant and other engines from indexing recent Reddit content, while Google retained access under a reported $60 million annual AI-data deal — making Google effectively the only search engine able to surface fresh Reddit results.
What happened
In July 2024, users and reporters noticed that searching 'site:reddit.com' on any engine other than Google returned almost no recent results. The cause was a robots.txt change that blocked search crawlers Reddit had not reached an agreement with. DuckDuckGo, Bing, Mojeek, Qwant, and others were shut out of indexing fresh Reddit content; on DuckDuckGo, Reddit links appeared without descriptions, accompanied by a notice that the site would not allow the preview.
The one exception was Google, which earlier in 2024 had signed a reported $60-million-per-year agreement giving it access to Reddit content, including for AI training. Reddit framed the blocks as a refusal to let crawlers access its data without enforceable commitments not to use it for AI, and a spokesperson denied the move was financially motivated. Critics noted the obvious tension: the engine that paid retained access, while those that did not were cut off, effectively turning Google into a paid gatekeeper for discovering Reddit discussions. The change is part of the same 2024 robots.txt overhaul through which Reddit moved to block unauthorized AI crawlers and scrapers generally.
The change is significant for information access and for the open web: a large share of useful Reddit answers became far harder to find outside Google, concentrating discoverability of one of the internet's largest user-generated knowledge bases behind a single commercial relationship.
Impact
Non-Google search engines lost the ability to index recent Reddit content, sharply reducing discoverability of Reddit discussions for users of Bing, DuckDuckGo, and others, and concentrating access to a major public knowledge base behind Google's paid deal. The move drew criticism as an anti-competitive, pay-to-index arrangement.