scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.
This tool helps web scraping engineers ensure their data collection jobs run smoothly and produce accurate results. It takes the output of your web scraping processes and verifies its quality and consistency, then provides clear reports or alerts. It's designed for developers and operations engineers who manage web scrapers.
552 stars. Actively maintained with 6 commits in the last 30 days. Available on PyPI.
Use this if you need automated checks for your web scraping jobs to prevent bad data from being collected or to catch failures early.
Not ideal if you're looking for a general-purpose data validation tool not specific to web scraping or if you don't use Scrapy for your data extraction.
Stars
552
Forks
101
Language
Python
License
BSD-3-Clause
Category
Last pushed
Mar 20, 2026
Commits (30d)
6
Dependencies
2
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/scrapinghub/spidermon"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
soxoj/maigret
🕵️♂️ Collect a dossier on a person by username from 3000+ sites