Whisper Fine-Tuning Voice AI Tools

Tools and frameworks for fine-tuning Whisper models on custom datasets, including language-specific adaptation, accent conditioning, and model distillation. Does NOT include pre-built Whisper applications, deployment wrappers, or inference optimization without training components.

There are 33 whisper fine-tuning tools tracked. 1 score above 50 (established tier). The highest-rated is YuanGongND/whisper-at at 50/100 with 412 stars.

Get all 33 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=voice-ai&subcategory=whisper-fine-tuning&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	YuanGongND/whisper-at Code and Pretrained Models for Interspeech 2023 Paper "Whisper-AT:...	50	Established	412	Python
2	adi-gov-tw/Taiwan-Tongues-ASR-CE Taiwan Tongues ASR CE 是一個開源語音辨識（Automatic Speech Recognition,...	46	Emerging	29	Python
3	huggingface/distil-whisper Distilled variant of Whisper for speech recognition. 6x faster, 50% smaller,...	45	Emerging	4,056	Python
4	phineas-pta/fine-tune-whisper-vi jupyter notebooks to fine tune whisper models on Vietnamese using Colab...	39	Emerging	19	Jupyter Notebook
5	KevKibe/African-Whisper 🚀 Framework for seamless fine-tuning of Whisper model on a multi-lingual...	37	Emerging	37	Python
6	huuquyet/PhoWhisper-next Demo using PhoWhisper models of VinAI built with Transformers.js + Next.js	35	Emerging	7	TypeScript
7	LianjiaTech/bella-whisper bella-whisper是一系列基于OpenAI...	35	Emerging	3	Python
8	sandy1990418/ChineseTaiwaneseWhisper This repository focuses on leveraging OpenAI's Whisper model for speech...	34	Emerging	70	Python
9	EtienneAb3d/WhisperHallu Experimental code: sound file preprocessing to optimize Whisper...	32	Emerging	348	Python
10	ga642381/Taiwanese-Whisper fine-tune Whipser model for Taiwanese speech recognition	32	Emerging	37	Python
11	tonywu71/distilling-and-forgetting-in-large-pre-trained-models Code for my dissertation on "Distilling and Forgetting in Large Pre-Trained...	31	Emerging	2	Jupyter Notebook
12	HiMeditator/wfts-chinese-tool 使用中文游玩《群星低语》游戏。Playing the game "Whisper from the Stars" in Chinese.	29	Experimental	8	Python
13	fengredrum/finetune-whisper-lora Fine-Tune Whisper with Transformers and PEFT	27	Experimental	58	Python
14	sovse/base_rus_whisper_stt Fine tuning of the base model from OpenAI Whisper in Russian language on the...	23	Experimental	40	Jupyter Notebook
15	sonhm3029/Realtime-Vietnamese-ASR-React-Native-and-Whisper This project implement end to end realtime vietnamese speech recognition...	23	Experimental	4	JavaScript
16	my-north-ai/semantic_audio_filtering Synthetic data augmentation technique via LLM for Automatic Speech...	23	Experimental	10	Python
17	naver/multilingual-distilwhisper This repository contains all the code necessary for running the multilingual...	21	Experimental	33	Python
18	HKAB/vietnamese-rnnt-tutorial A tutorial on how to train RNN-T from scratch with Whisper encoder	21	Experimental	12	Python
19	petitwhito/Speech_to_text_project Complete Speech-to-Text pipeline: from-scratch architectures (MLP, CNN, RNN,...	20	Experimental	1	Python
20	thomas-ferraz/Whisper-Robustness Fine-tuning SOTA Speech Foundation Model (Whisper) for Speech Transcription	19	Experimental	3	Python
21	innerNULL/simpler-distil-whisper Simpler Distil-Whisper	19	Experimental	3	Python
22	navalnica/whisper-finetuning-be Finetuning Whisper ASR model for Belarusian language	19	Experimental	17	Jupyter Notebook
23	10809104/taigi-speech-to-text 台語語音轉文字訓練資料集，資料來源：教育部《臺灣閩南語常用詞辭典》。	18	Experimental	1	Python
24	Bilel-Eljaamii/Whisper-Arabic-Poetry-Performance Benchmarking OpenAI Whisper models (tiny→turbo) for classical Arabic poetry...	18	Experimental	2	Python
25	mavleo96/whisper-accent Conditioning via Adaptive Layer Norm for accented speech recognition	18	Experimental	1	Python
26	runze123/cantonese-asr-evaluation This project presents a systematic evaluation of two state-of-the-art...	14	Experimental	1	Python
27	egorsmkv/whisper-ukrainian Trainer and Evaluation scripts for fine-tuning Whisper models for the...	14	Experimental	19	Python
28	namphung134/ASR-Vietnamese Fine-tuning the openai/whisper-small model on the 250h dataset for...	13	Experimental	3	Jupyter Notebook
29	backspacetg/distilAlhubert code for our paper DistilALHuBERT: A Distilled Parameter Sharing Audio...	12	Experimental	7	Python
30	Avinraj01/SHL-Grammar-Scoring-Engine-for-Voice-Samples This model predicts grammar scores (1–5) from audio files. It uses Whisper...	12	Experimental	2	Jupyter Notebook
31	2003HARSH/OpenAI-Whisper-Automated-Hindi-Speech-Recognition This project adapts OpenAI's Whisper model to create an automated speech...	11	Experimental	3	Jupyter Notebook
32	bivex/whisper-large-v3-turbo Whisper Large V3 Turbo - fast speech-to-text model implementation with...	11	Experimental	—	Python
33	sanket-poojary-03/Fine-tuning-Whisper Fine tuning Whisper-Small LLM for Hinglish Audio dataset	11	Experimental	3	Python

Comparisons in this category

ChineseTaiwaneseWhisper and Taiwanese-Whisper (34 vs 32)