Scientific-Computing-Lab/Tokompiler
Scope is all you need: Transforming LLMs for HPC Code
This project helps machine learning engineers and researchers preprocess high-performance computing (HPC) source code for training large language models (LLMs). It takes raw source code in various programming languages and transforms it into a standardized, tokenized format that enhances code representation and understanding for AI models. This process results in a structured sequence of tokens and numerical IDs ready for model ingestion, making it easier to build specialized LLMs for HPC tasks.
No commits in the last 6 months.
Use this if you are an AI/ML engineer or researcher working on pretraining large language models specifically for understanding and generating high-performance computing code.
Not ideal if you are looking for a general-purpose code tokenizer for static analysis, syntax highlighting, or general natural language processing tasks outside of large language model pretraining for HPC.
Stars
10
Forks
3
Language
Python
License
—
Category
Last pushed
Oct 14, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Scientific-Computing-Lab/Tokompiler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ModelTC/LightCompress
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs,...
p-e-w/heretic
Fully automatic censorship removal for language models
Orion-zhen/abliteration
Make abliterated models with transformers, easy and fast
YerbaPage/LongCodeZip
LongCodeZip: Compress Long Context for Code Language Models [ASE2025]
locuslab/wanda
A simple and effective LLM pruning approach.