IlyaGusev/ping_pong_bench
A benchmark for role-playing language models
This helps evaluate how well different large language models (LLMs) can role-play in conversations. It takes a description of a character and a conversational scenario, then uses another LLM to pretend to be a user interacting with the model being tested. The output is a detailed rating of the tested model's performance on criteria like staying in character, being entertaining, and fluency. It's for researchers and developers who need to assess the conversational abilities of LLMs without extensive human involvement.
116 stars. No commits in the last 6 months.
Use this if you need an automated way to test and compare how well large language models maintain a specific persona and engage in multi-turn role-playing conversations.
Not ideal if you want to evaluate general conversational abilities that don't involve specific character roles or complex scenarios.
Stars
116
Forks
11
Language
Python
License
Apache-2.0
Category
Last pushed
May 25, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/IlyaGusev/ping_pong_bench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)