lizongying/go-crawler
A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middleware and supports various parsing and storage methods. Additionally, it supports distributed deployment. 基于golang实现的爬虫框架，编写简单，性能强劲。内置了丰富的实用中间件，支持多种解析、保存方式，支持分布式部署。
This is a web crawling framework that helps developers collect data from websites. You define what information you need from web pages, and the framework fetches it, handles complexities like cookies or redirects, and then provides the extracted data in various formats like JSON, CSV, or directly into databases. It's designed for software developers who need to build high-performance, custom web scrapers or data collection systems.
150 stars. No commits in the last 6 months.
Use this if you are a developer building a custom, high-performance web data extraction solution and need fine-grained control over the crawling process, including handling complex website interactions or distributing the workload.
Not ideal if you need a simple, no-code, or low-code tool for basic data scraping, or if you're not comfortable with programming in Golang.
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/lizongying/go-crawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.