lamroger/waffie
LLM API automated testing tool
This tool helps software developers ensure the quality and consistency of their applications that use large language model (LLM) APIs. You provide a Waffiefile configuring your LLM prompt and test data (like customer feedback and its expected sentiment), and the tool runs tests against different LLM providers and models. It then gives you a report on how accurately the LLM classified your test data.
No commits in the last 6 months. Available on npm.
Use this if you are a software developer building applications with LLM APIs and need to regularly check if your LLM responses are still accurate and consistent over time or across different models.
Not ideal if you are looking for a tool to train or fine-tune LLMs, or if you don't work with LLM APIs in your software.
Stars
10
Forks
3
Language
TypeScript
License
—
Category
Last pushed
Sep 30, 2023
Commits (30d)
0
Dependencies
7
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/lamroger/waffie"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
lmstudio-ai/lmstudio-js
LM Studio TypeScript SDK
lmstudio-ai/lms
LM Studio CLI
samestrin/llm-interface
A simple NPM interface for seamlessly interacting with 36 Large Language Model (LLM) providers,...
nbonamy/multi-llm-ts
A Typescript library to use LLM providers APIs in a unified way.
token-js/token.js
Integrate 200+ LLMs with one TypeScript SDK using OpenAI's format.