SiluPanda/agent-crawl
High performance, lightweight and typesafe library to crawl and scrape web, built for LLM agents.
This tool helps developers working with AI agents to efficiently gather information from websites. It takes URLs as input, intelligently scrapes web content, and delivers clean, token-optimized Markdown or structured JSON output. AI agent developers can use this to provide their agents with relevant, noise-free web data for tasks like summarization, question answering, or data extraction.
Available on npm.
Use this if you are building AI agents and need a reliable, high-performance way to fetch web content and process it into a clean, token-efficient format ready for an LLM's context window.
Not ideal if you are looking for a simple browser automation tool for personal use or a general-purpose web scraping solution that doesn't prioritize LLM agent integration.
Stars
8
Forks
—
Language
TypeScript
License
MIT
Category
Last pushed
Feb 10, 2026
Commits (30d)
0
Dependencies
3
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/agents/SiluPanda/agent-crawl"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.