aplmikex/deduplication_mnbvc
文本去重
This tool helps you clean up large collections of text files by identifying and removing exact duplicate files based on their content and size. It takes various text files, primarily TXT, and outputs them as structured, queryable data in JSONL format, making it easier to manage and analyze. This is ideal for anyone dealing with extensive archives of documents and needing to ensure data uniqueness.
No commits in the last 6 months.
Use this if you have millions of text files on your personal computer and need a fast way to remove duplicate content and organize them into a structured format.
Not ideal if you need to deduplicate file types other than text files or require cluster-based processing for extremely large, distributed datasets right now.
Stars
78
Forks
12
Language
Python
License
MIT
Category
Last pushed
May 23, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/aplmikex/deduplication_mnbvc"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.