ai8hyf/llm_split_recall_test
Split and Recall: A simple and efficient benchmark to evaluate in-context recall performance of Large Language Models (LLMs)
This project helps evaluate how well large language models (LLMs) can find and recall specific sentences from a given text, especially within longer documents. It takes a document (like a research paper) and outputs a performance score indicating the model's accuracy in identifying and reproducing sentences. This is useful for AI developers, researchers, or data scientists working on or comparing different LLMs.
No commits in the last 6 months.
Use this if you need to benchmark and compare the 'in-context recall' ability of various large language models, particularly their precision in extracting specific sentences from a paragraph or a longer document.
Not ideal if you need a benchmark for aspects of LLM performance other than sentence-level recall, or if your evaluation data is significantly different from academic paper abstracts.
Stars
9
Forks
3
Language
Python
License
MIT
Category
Last pushed
Mar 31, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/ai8hyf/llm_split_recall_test"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
jncraton/languagemodels
Explore large language models in 512MB of RAM
microsoft/unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
haizelabs/verdict
Inference-time scaling for LLMs-as-a-judge.
albertan017/LLM4Decompile
Reverse Engineering: Decompiling Binary Code with Large Language Models
bytedance/Sa2VA
Official Repo For Pixel-LLM Codebase