crwlrsoft/crawler
Library for Rapid (Web) Crawler and Scraper Development
This library helps developers quickly build tools to automatically browse websites and collect specific information from them. It takes URLs as input and can extract various data types like text, links, or structured data (e.g., product details, prices) from HTML, XML, or JSON, outputting it in a usable format. Web developers or data engineers who need to programmatically gather information from the internet for analysis or integration would use this.
369 stars.
Use this if you need to programmatically collect data from many web pages in a structured way, respecting website rules and handling complex navigation like pagination or JavaScript-heavy sites.
Not ideal if you're a non-developer looking for a no-code solution to extract data, or if you only need to scrape a few pages manually.
Stars
369
Forks
13
Language
PHP
License
MIT
Category
Last pushed
Jan 05, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/crwlrsoft/crawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.