obeone/crawler-to-md
Convert web content to Markdown & JSON files to fuel your GPTs !
This tool helps you gather information from websites by converting their content into structured Markdown and JSON files. You input a list of website addresses or a starting URL, and it produces ready-to-use files that capture the text and metadata from those pages. It's designed for anyone who needs to collect web content for AI model training, data analysis, or building custom GPTs.
Available on PyPI.
Use this if you need to quickly and easily collect web page content into a structured format for AI training, analysis, or uploading to GPT models.
Not ideal if you need a sophisticated web automation tool for complex interactions, form filling, or bypassing advanced anti-scraping measures.
Stars
26
Forks
6
Language
Python
License
—
Category
Last pushed
Nov 19, 2025
Commits (30d)
0
Dependencies
11
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/obeone/crawler-to-md"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
mohamed-chs/convoviz
Convert your ChatGPT export (ZIP) into clean Markdown text files with inline media, and generate...
RICHQAQ/PasteMD
一键将 Markdown 和网页 AI 对话（ChatGPT/DeepSeek等）完美粘贴到 Word、WPS 和 Excel 的效率工具 | One-click paste Markdown...
sugurutakahashi-1234/ai-chat-md-export
Command-line tool for converting ChatGPT and Claude chat history to readable Markdown files
Hugo-COLLIN/SaveMyPhind-conversation-exporter
🚀 Save my Chatbot: Download your ChatGPT, Claude, Perplexity and Phind conversations into...
pionxzh/chatgpt-exporter
Export and Share your ChatGPT conversation history