peterdalle/mechanicalnews
Web server app that crawls and saves news articles, provides article API for research
This tool helps social science researchers gather and analyze news articles. It automatically collects full-text articles and their associated metadata from various online news sources, then makes them available through a structured API. Researchers can then use their preferred R or Python tools to access and analyze this clean, organized news data.
No commits in the last 6 months.
Use this if you need to systematically collect, store, and access the full text and rich metadata of news articles for social science research, without having to build your own scraping infrastructure from scratch.
Not ideal if you only need occasional data from a few articles, or if you prefer a pre-hosted service where you don't manage your own server.
Stars
8
Forks
—
Language
Python
License
GPL-3.0
Category
Last pushed
Sep 11, 2020
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/peterdalle/mechanicalnews"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.