ulixee/secret-agent
The web scraper that's nearly impossible to block - now called @ulixee/hero
This project helps web developers reliably extract data from websites, even those with strong anti-bot measures. It takes a target website URL as input and outputs structured data, text, or images from that site. It's designed for web developers who need to collect information from the public web for various applications.
728 stars. No commits in the last 6 months. Available on npm.
Use this if you are a web developer who needs to build robust web scrapers that can bypass common detection mechanisms and gather data from complex, dynamic websites.
Not ideal if you are looking for a simple, no-code scraping solution or if your primary goal is automated website testing rather than data extraction.
Stars
728
Forks
48
Language
TypeScript
License
MIT
Category
Last pushed
Mar 07, 2023
Commits (30d)
0
Dependencies
5
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/ulixee/secret-agent"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.