CrowdTruth/CrowdTruth-core
CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems
When building an AI system, you often need to gather labeled data from many people to train it. This tool helps researchers and AI practitioners process the results from crowdsourcing platforms like Amazon Mechanical Turk and CrowdFlower. It takes the raw crowdsourcing data and applies a specialized methodology to determine the most reliable "ground truth" labels, even when crowd workers disagree.
No commits in the last 6 months.
Use this if you are developing or evaluating AI systems and need to derive high-quality, reliable labeled datasets from potentially noisy or inconsistent crowdsourced annotations.
Not ideal if you are looking for a platform to run crowdsourcing tasks or a general-purpose data labeling tool, as this focuses specifically on processing and evaluating existing crowdsourcing results.
Stars
64
Forks
11
Language
Jupyter Notebook
License
Apache-2.0
Category
Last pushed
Apr 08, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/CrowdTruth/CrowdTruth-core"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
cvat-ai/cvat
Annotate better with CVAT, the industry-leading data engine for machine learning. Used and...
HumanSignal/label-studio
Label Studio is a multi-type data labeling and annotation tool with standardized output format
wkentaro/labelme
Image annotation with Python. Supports polygon, rectangle, circle, line, point, and AI-assisted...
CVHub520/X-AnyLabeling
Effortless data labeling with AI support from Segment Anything and other awesome models.
doccano/doccano
Open source annotation tool for machine learning practitioners.