yohasebe/wp2txt
A command-line tool to extract plain text from Wikipedia dumps with category and section filtering
This tool helps researchers, linguists, and data scientists extract clean text and category data from Wikipedia's vast archives. You can specify a language and get plain text, or filter by specific articles or categories. The output is clean text or JSON, ready for analysis in fields like corpus linguistics or text mining.
191 stars.
Use this if you need to systematically collect large amounts of Wikipedia article content, stripped of complex formatting, for research or data analysis purposes.
Not ideal if you're looking for an interactive Wikipedia browser or need to extract images and complex page layouts, as it focuses on plain text and metadata.
Stars
191
Forks
37
Language
Ruby
License
MIT
Category
Last pushed
Feb 23, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/yohasebe/wp2txt"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
DerwenAI/pytextrank
Python implementation of TextRank algorithms ("textgraphs") for phrase extraction
Tiiiger/bert_score
BERT score for text generation
BrikerMan/Kashgari
Kashgari is a production-level NLP Transfer learning framework built on top of tf.keras for...
asyml/texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. ...
neuralmind-ai/portuguese-bert
Portuguese pre-trained BERT models