lobstrio/shadow-useragent
Pick the most common user-agents on the Internet 👻
When scraping websites, you often need to appear as a regular web browser to avoid being blocked. This tool provides up-to-date and commonly used browser identification strings (User-Agents) so your automated requests blend in better. You get a list of current User-Agents, or can select them by browser type or commonality. It's for anyone building automated web scrapers or bots who needs to emulate diverse web browsers.
172 stars. No commits in the last 6 months. Available on PyPI.
Use this if your web scraping or automated browsing tasks are getting blocked because websites detect you're a bot, and you need a reliable way to rotate realistic browser identities.
Not ideal if you're not doing web scraping or automated requests that require mimicking real web browsers, or if you need to control every byte of your HTTP headers manually.
Stars
172
Forks
11
Language
Python
License
MIT
Category
Last pushed
Jan 09, 2021
Commits (30d)
0
Dependencies
3
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/lobstrio/shadow-useragent"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.