Ryaang/gpt-web-crawler
A web crawler for GPTs to build knowledge bases 用于GPT构建知识库的网站爬虫
This tool helps you gather information from websites to create a knowledge base for AI agents or other data analysis. You input a starting website URL, and it outputs a JSON file containing the website's text, titles, keywords, and descriptions. It's designed for data analysts, AI developers, or anyone building AI agents who needs structured web content without complex coding.
No commits in the last 6 months.
Use this if you need to quickly extract content from specific websites to feed into an AI agent, database, or perform content analysis.
Not ideal if you require advanced custom crawling behaviors, very deep and complex site navigation, or extensive interaction with dynamic web elements beyond basic content extraction.
Stars
25
Forks
7
Language
Python
License
MIT
Category
Last pushed
Sep 02, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/Ryaang/gpt-web-crawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
jamesturk/scrapeghost
👻 Experimental library for scraping websites using OpenAI's GPT API.
Priyanshu-hawk/ChatGPT-unofficial-api-selenium
This is unofficial ChatGPT API using selenium for prompt testing and flow testing purposes
3281448091/easyChatGPT
An unofficial yet elegant interface of the ChatGPT API using browser automation that bypasses...
ryuseisan/auto-chatgpt
Automate interaction with the browser version of ChatGPT.
djb-gt/gpt-automated-web-scraper
The GPT-based Universal Web Scraper MVP is a solution that leverages GPT models and web scraping...