taabishhh/LLM_Preprocessing
This project implements a Byte Pair Encoding (BPE) tokenization approach along with a Word2Vec model to generate word embeddings from a text corpus. The implementation leverages Apache Hadoop for distributed processing and includes evaluation metrics for optimal dimensionality of embeddings.
19
/ 100
Experimental
No commits in the last 6 months.
Stale 6m
No Package
No Dependents
Maintenance
2 / 25
Adoption
1 / 25
Maturity
16 / 25
Community
0 / 25
Stars
1
Forks
—
Language
Scala
License
Apache-2.0
Category
Last pushed
Aug 26, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/embeddings/taabishhh/LLM_Preprocessing"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.