waikato-llm/llm-dataset-converter
For converting LLM datasets from one format into another.
This tool helps AI practitioners prepare large language model (LLM) training data. It takes datasets in various common formats like CSV, JSON Lines, or Parquet, and converts them into other formats suitable for different LLM tasks, like classification, pre-training, or translation. You would use this if you're an AI developer, researcher, or data scientist working with LLMs.
Available on PyPI.
Use this if you need to transform your raw LLM datasets into specific formats required by different training frameworks or models, potentially cleaning or filtering the data in the process.
Not ideal if you are not working with large language model datasets or do not need to convert between their specific formats.
Stars
22
Forks
2
Language
Python
License
MIT
Category
Last pushed
Nov 12, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/waikato-llm/llm-dataset-converter"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
allenai/dolma
Data and tools for generating and inspecting OLMo pre-training data.
refuel-ai/autolabel
Label, clean and enrich text datasets with LLMs.
niclasgriesshaber/llm_patent_pipeline
LLMs for Historical Dataset Construction from Archival Image Scans
cgxjdzz/FeatureForge-LLM
FeatureForge LLM is a Python package that leverages large language models (LLMs) to automate and...
codeastra2/llm-feat
Automated feature engineering using Large Language Models (LLMs) for tabular data