technion-cs-nlp/BiologicalTokenizers
Effect of tokenization on transformers for biological sequence
This project helps bioinformaticians and computational biologists improve the accuracy and efficiency of deep learning models when working with long biological sequences like DNA or protein data. It takes raw biological sequences as input and outputs optimized 'tokenized' versions, significantly reducing sequence length while boosting model performance. This is for researchers and scientists who use transformer models for tasks like protein function prediction or sequence alignment.
Use this if you are building or training deep learning models on biological sequence data and need to optimize input representation for better accuracy and faster processing.
Not ideal if you are not working with deep learning models, particularly transformer architectures, or if your biological sequence analysis doesn't involve complex prediction or classification tasks.
Stars
22
Forks
—
Language
Python
License
MIT
Category
Last pushed
Dec 31, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/technion-cs-nlp/BiologicalTokenizers"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
huggingface/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
megagonlabs/ginza-transformers
Use custom tokenizers in spacy-transformers
Kaleidophon/token2index
A lightweight but powerful library to build token indices for NLP tasks, compatible with major...
Hugging-Face-Supporter/tftokenizers
Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels
NVIDIA/Cosmos-Tokenizer
A suite of image and video neural tokenizers