samber/the-great-gpt-firewall
🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs
This project provides a monthly-updated list of websites that use their 'robots.txt' file to prevent AI agents and crawlers, like those used by ChatGPT or Google's Bard, from accessing their content. It details which major media, video, and music sites block AI access. Website owners, content creators, and digital rights managers can use this to understand how various platforms protect their content from AI scraping.
Use this if you are a website owner, content creator, or digital rights professional looking to understand which major platforms restrict AI access and how to implement similar protections for your own site.
Not ideal if you are looking for an active tool to automatically block AI bots on your own website, as this is primarily an informational resource.
Stars
91
Forks
7
Language
Python
License
MIT
Category
Last pushed
Mar 01, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/samber/the-great-gpt-firewall"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
Haidra-Org/AI-Horde
A crowdsourced distributed cluster for AI art and text generation
meetpateltech/AI-Infinity
A set of AI tools that will help you explore the infinite possibilities of AI.
lianxhcn/research_with_AI
Empirical research with AI tools. 你可以点击 Fork 按钮把本仓库复制到你的账号下。我的小红书号: 连玉君 (ID: 95085566173).
alvi-se/ai-ublock-blacklist
Websites I personally found that are completely generated by AI. Pull requests welcome.
Diraw/AI-Screenshot-Translator
🚀全新重构！论文阅读工具，一键截图AI翻译，支持数学公式，贴片截图，窗口锁定，归档管理