tokahuke/lopez
Crawling and scraping the Web for fun and profit
This tool helps you gather specific information from websites automatically, enabling you to collect data on a large scale. You provide it with instructions on what websites to visit and what content to extract, and it delivers the scraped data into a PostgreSQL database. It's designed for anyone who needs to systematically collect web data for research, analysis, or competitive intelligence.
No commits in the last 6 months.
Use this if you need to extract structured data from many web pages and store it efficiently for later analysis.
Not ideal if you need to interact with websites that heavily rely on JavaScript for content display or if you require a distributed scraping solution.
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/tokahuke/lopez"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.