geminimir/promptproof-action
Deterministic LLM contract checks for CI. Replays recorded fixtures, enforces schema/regex/budget rules, generates HTML/JUnit/JSON reports, and comments on PRs. Zero live model calls in CI.
This tool helps developers working with large language models (LLMs) ensure that their model outputs meet specific quality, safety, and cost requirements. It takes recorded LLM responses and a set of defined rules, then checks for violations like incorrect formats, PII leakage, or exceeding cost budgets. The result is a detailed report and immediate feedback on pull requests, allowing developers to catch issues before deploying new LLM features or updates.
No commits in the last 6 months.
Use this if you need a reliable way to automatically test LLM outputs in your development pipeline, ensuring they always adhere to predefined contracts without making live API calls during testing.
Not ideal if you need to test live LLM interactions in real-time or if your primary concern is evaluating subjective model quality rather than objective contract compliance.
Stars
13
Forks
2
Language
HTML
License
MIT
Category
Last pushed
Sep 02, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/geminimir/promptproof-action"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
open-compass/opencompass
OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral,...
IBM/unitxt
🦄 Unitxt is a Python library for enterprise-grade evaluation of AI performance, offering the...
lean-dojo/LeanDojo
Tool for data extraction and interacting with Lean programmatically.
GoodStartLabs/AI_Diplomacy
Frontier Models playing the board game Diplomacy.
google/litmus
Litmus is a comprehensive LLM testing and evaluation tool designed for GenAI Application...