pragmar/mcp-server-webcrawl
MCP server tailored to connecting web crawler data and archives
This project helps anyone working with large collections of website data or archives to search and retrieve specific information efficiently. It takes data from various web crawling tools like ArchiveBox or HTTrack, and lets you use advanced search queries to find exactly what you need. The output is filtered web content and analyses, which an AI client can then process. This tool is for web archivists, SEO specialists, digital marketers, or researchers who need to make sense of extensive crawled web data.
Use this if you need to perform detailed searches, audits, or analyses on large datasets collected by web crawlers, enabling an AI to efficiently process and understand the content.
Not ideal if you are looking for a web crawler itself, or if you only need to perform basic keyword searches on small, static website archives.
Stars
37
Forks
14
Language
Python
License
—
Category
Last pushed
Dec 08, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/mcp/pragmar/mcp-server-webcrawl"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
apify/apify-mcp-server
The Apify MCP server enables your AI agents to extract data from social media, search engines,...
bartholomej/node-csfd-api
ČSFD API in JavaScript. Amazing NPM library for scrapping csfd.cz. Now with MCP server
brightdata/brightdata-mcp
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public...
ScrapeGraphAI/scrapegraph-mcp
ScapeGraph MCP Server
firecrawl/firecrawl-mcp-server
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and...