stephantul/piecelearn
Learning BPE embeddings by first learning a segmentation model and then training word2vec
This tool helps natural language processing (NLP) practitioners create robust text representations, even for words they haven't seen before. You provide a collection of raw text documents, and it generates a specialized BPE encoder and word embeddings. This makes your text analysis models more resilient to variations and misspellings in text data.
No commits in the last 6 months.
Use this if you need to generate high-quality word embeddings for text analysis in domains where new or unusual words frequently appear, or when dealing with noisy text data.
Not ideal if your text data is perfectly clean, has a very limited and stable vocabulary, or if you strictly need traditional word-based embeddings without subword segmentation.
Stars
19
Forks
1
Language
Python
License
MIT
Category
Last pushed
Dec 18, 2022
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/embeddings/stephantul/piecelearn"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
MilaNLProc/contextualized-topic-models
A python package to run contextualized topic modeling. CTMs combine contextualized embeddings...
vinid/cade
Compass-aligned Distributional Embeddings. Align embeddings from different corpora
spcl/ncc
Neural Code Comprehension: A Learnable Representation of Code Semantics
criteo-research/CausE
Code for the Recsys 2018 paper entitled Causal Embeddings for Recommandation.
vintasoftware/entity-embed
PyTorch library for transforming entities like companies, products, etc. into vectors to support...