Kimdonghyeon7645/Crawling-Book
๐งพ๐ ๋๋ด์ฃผ๋ ํฌ๋กค๋ง&๋ฉํฌ๋ก ์คํฌ๋ฆฝํธ๋ฅผ ์์ฑํ๋ ๋ฐฉ๋ฒ (with Python)
This helps you gather information from websites automatically, similar to how you might manually copy-paste details but much faster. You'll input website addresses and define what specific data you need, and it will output that collected data, potentially into a database or a file. Anyone who needs to collect data from many web pages for research, analysis, or competitive intelligence would find this useful.
No commits in the last 6 months.
Use this if you need to systematically collect specific text, numbers, or other content from numerous web pages for analysis or storage.
Not ideal if you only need to grab a small amount of information from one or two pages, as setting it up for simple tasks might take more effort than doing it manually.
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/Kimdonghyeon7645/Crawling-Book"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.