pavlovtech/WebReaper
Web scraper, crawler and parser in C#. Designed as simple, declarative and scalable web scraping solution.
This tool helps you automatically collect specific information from websites, such as product prices, news articles, or competitor reviews. You provide it with website addresses and tell it what data to look for (like product titles or image links), and it outputs that structured information into a file or database. It's designed for market researchers, analysts, or anyone who needs to gather large amounts of publicly available web data.
135 stars. No commits in the last 6 months.
Use this if you need to regularly or rapidly extract structured data from multiple web pages or entire websites to power your analysis or data-driven projects.
Not ideal if you only need to grab a few pieces of information manually, or if the website explicitly forbids automated scraping through its terms of service.
Stars
135
Forks
33
Language
C#
License
GPL-3.0
Category
Last pushed
Oct 29, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/pavlovtech/WebReaper"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.