apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
This tool helps developers reliably gather information from websites, even complex ones that use a lot of JavaScript. It takes a list of URLs and, after visiting each page, extracts specific data, like text, links, or even entire files (PDF, images), which it then stores in a structured, machine-readable format. This is ideal for developers building applications that need large amounts of web data for things like AI model training or content analysis.
8,682 stars. Used by 2 other packages. Actively maintained with 50 commits in the last 30 days. Available on PyPI.
Use this if you need to programmatically collect data from many web pages, including those that adapt to user interaction or have anti-bot measures.
Not ideal if you're a non-developer seeking a no-code solution for basic data extraction, or if you only need to process static files locally.
Stars
8,682
Forks
706
Language
Python
License
Apache-2.0
Category
Last pushed
Mar 28, 2026
Commits (30d)
50
Dependencies
13
Reverse dependents
2
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/apify/crawlee-python"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.
orangecoding/fredy
❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments,...