chiphuyen/lazynlp
Library to scrape and clean web pages to create massive datasets.
This tool helps researchers and data scientists gather vast amounts of text from the web for training large language models or other natural language processing tasks. You provide lists of web page links, and it systematically downloads, cleans, and deduplicates the content, outputting organized files of pure, human-readable text. It's designed for anyone needing to create very large, clean text datasets from web sources.
2,242 stars. No commits in the last 6 months.
Use this if you need to build a massive dataset of high-quality, deduplicated text content scraped from numerous web pages, such as for training a machine learning model or linguistic research.
Not ideal if you only need a small amount of text from a few specific websites, or if you require advanced web scraping features beyond basic page download and cleaning.
Stars
2,242
Forks
321
Language
Python
License
—
Category
Last pushed
Nov 11, 2020
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/chiphuyen/lazynlp"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.