gustavoespindola/chunkerizer
Split and analyze text files using langchain and streamlit
This tool helps you precisely divide large text files or code into smaller, manageable pieces, often called 'chunks.' You upload a text file, choose how you want to split it (by characters, tokens, or programming language structure), and it outputs a detailed breakdown of the resulting chunks. It's for anyone working with large text documents or codebases who needs to segment them for further processing or analysis, such as preparing data for AI models or auditing code sections.
No commits in the last 6 months.
Use this if you need to systematically break down long text documents or code into smaller, well-defined segments while controlling chunk size and overlap.
Not ideal if you only need to perform simple line-by-line file splitting or if your primary goal is general text editing rather than structured segmentation.
Stars
50
Forks
10
Language
Python
License
—
Category
Last pushed
May 22, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/gustavoespindola/chunkerizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.