Tyrone-Zhao/crawlerUtils
Utils for programming web crawler
This project helps developers gather data from websites, including those with dynamic content or CAPTCHAs. It simplifies the process of sending requests, handling cookies, parsing HTML or JSON content, and managing asynchronous or multiprocessing tasks. It’s designed for anyone building automated data collection tools or web scrapers.
No commits in the last 6 months. Available on PyPI.
Use this if you need to programmatically extract information from many web pages and require advanced features like CAPTCHA recognition or efficient handling of concurrent requests.
Not ideal if you're looking for a no-code solution or a graphical interface for web scraping, as this tool requires coding knowledge.
Stars
11
Forks
2
Language
Python
License
MIT
Category
Last pushed
May 16, 2019
Commits (30d)
0
Dependencies
9
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/Tyrone-Zhao/crawlerUtils"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.