s2e-lab/SecurityEval
Repository for "SecurityEval Dataset: Mining Vulnerability Examples to Evaluate Machine Learning-Based Code Generation Techniques" published in MSR4P&S'22.
This project helps security researchers and software engineers evaluate how prone AI code generation tools are to producing insecure code. It provides a dataset of programming prompts and corresponding examples of vulnerable Python code. You can use this to test code generation models like GitHub Copilot or InCoder, feeding them prompts and then analyzing their output for security flaws.
No commits in the last 6 months.
Use this if you are a security researcher or software engineer who needs to assess the security quality of AI-generated code, particularly for identifying common vulnerabilities in Python.
Not ideal if you are looking for a tool to fix vulnerabilities in existing code or for evaluating code generation models for languages other than Python.
Stars
85
Forks
15
Language
Python
License
—
Category
Last pushed
Nov 04, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ai-coding/s2e-lab/SecurityEval"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
k4black/codebleu
Pip compatible CodeBLEU metric implementation available for linux/macos/win
LiveCodeBench/LiveCodeBench
Official repository for the paper "LiveCodeBench: Holistic and Contamination Free Evaluation of...
EdinburghNLP/code-docstring-corpus
Preprocessed Python functions and docstrings for automated code documentation (code2doc) and...
hendrycks/apps
APPS: Automated Programming Progress Standard (NeurIPS 2021)
solis-team/Hydra
[FSE 2026] Do Not Treat Code as Natural Language: Implications for Repository-Level Code...