artas728/spelltest
AI-to-AI Testing | Simulation framework for LLM-based applications
Building an application powered by large language models (LLMs) means ensuring it consistently provides accurate and relevant responses. This tool helps you automatically test your LLM-based application by simulating interactions with different types of synthetic users. You provide descriptions of your users, quality expectations, and application prompts, and it outputs a quality score for your application's responses, highlighting areas for improvement.
136 stars. No commits in the last 6 months. Available on PyPI.
Use this if you are developing an LLM-powered application and need to systematically test its responses from various user perspectives to ensure high quality before release.
Not ideal if you are looking for a free testing solution, as running simulations with this tool incurs costs for LLM API usage.
Stars
136
Forks
7
Language
Python
License
MIT
Category
Last pushed
Nov 07, 2023
Commits (30d)
0
Dependencies
19
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/artas728/spelltest"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
open-compass/opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral,...
IBM/unitxt
🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the...
lean-dojo/LeanDojo
Tool for data extraction and interacting with Lean programmatically.
GoodStartLabs/AI_Diplomacy
Frontier Models playing the board game Diplomacy.
google/litmus
Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application...