soskek/bookcorpus
Crawl BookCorpus
This project helps natural language processing (NLP) researchers and practitioners create their own large-scale text dataset from books. It takes URLs from Smashwords.com as input and outputs a collection of text files, each containing a book's content. The output can then be further processed into a single file with one sentence per line, or even tokenized for use in training language models or sentence encoders.
853 stars. No commits in the last 6 months.
Use this if you need to build a substantial collection of free book texts for unsupervised learning in natural language processing and prefer to control the data acquisition process yourself.
Not ideal if you need an exact, bit-for-bit replica of the original BookCorpus dataset, as this tool creates a similar but potentially larger and slightly different collection.
Stars
853
Forks
112
Language
Python
License
MIT
Category
Last pushed
Jul 14, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/soskek/bookcorpus"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
chrismattmann/tika-python
Tika-Python is a Python binding to the Apache Tika™ REST services allowing Tika to be called...
sloria/TextBlob
Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase...
cltk/cltk
The Classical Language Toolkit
allenai/scispacy
A full spaCy pipeline and models for scientific/biomedical documents.
wi2trier/cbrkit
Customizable Case-Based Reasoning (CBR) toolkit for Python with a built-in API and CLI.