oxylabs/ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
This tool helps businesses and researchers gather specific information from websites by simply describing what they need in plain English. You provide a starting web address and a prompt (like "find all pricing pages"), and it returns the relevant data as structured JSON or readable Markdown. It's ideal for market analysts, competitive intelligence professionals, or anyone who needs to quickly collect organized information from many web pages.
2,764 stars. No commits in the last 6 months.
Use this if you need to quickly gather structured data from websites without writing complex code or maintaining custom web scraping scripts.
Not ideal if you require real-time data feeds from highly dynamic sites that change frequently or need to interact with web elements (like filling forms) beyond simple data extraction.
Stars
2,764
Forks
12
Language
—
License
—
Category
Last pushed
Oct 13, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/agents/oxylabs/ai-crawler-py"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
vakra-dev/reader
Open-source, production-grade web scraping engine built for LLMs. Scrape and crawl the entire...
joaobenedetmachado/scrapit
A (really) easy way to web scrape
firecrawl/open-scouts
🔥 AI-powered web monitoring platform. Create automated scouts that search the web and send email...
BrowserCash/teracrawl
High-performance web crawler API optimized for LLMs. Turn any search or website into clean...
memvid/maw
Crawl any website into a single searchable file. Query it forever, offline.