N0taN3rd/Squidwarc
Squidwarc is a high fidelity, user scriptable, archival crawler that uses Chrome or Chromium with or without a head
This tool helps personal archivists and digital preservationists capture web pages exactly as they appear in a browser, including interactive elements and JavaScript. You provide a list of web addresses, and it produces a high-fidelity archive of those pages and their associated content. It's designed for anyone needing to preserve the dynamic web for future reference or research.
174 stars. No commits in the last 6 months.
Use this if you need to reliably save a complete and interactive version of a web page, including all its embedded content and dynamic behaviors.
Not ideal if you primarily need to perform broad, plain-text crawls of static web content without needing to execute JavaScript.
Stars
174
Forks
25
Language
JavaScript
License
Apache-2.0
Category
Last pushed
May 19, 2020
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/N0taN3rd/Squidwarc"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.