scrapfly/python-scrapfly
Scrapfly Python SDK for headless browsers and proxy rotation
This helps data scientists and AI engineers gather specific information from websites for training large language models (LLMs). It takes URLs as input, scrapes the webpage content (even from complex, JavaScript-heavy sites), and outputs clean text or Markdown that can be used to build question-answering chatbots, enhance document understanding, or power autonomous research agents.
Use this if you need to reliably extract data from many web pages, especially those with anti-bot measures or dynamic content, to train or augment your AI models.
Not ideal if you're looking for a simple, no-code web scraping tool or if your primary need isn't integrating web data with AI frameworks like LlamaIndex or LangChain.
Stars
53
Forks
15
Language
Python
License
—
Category
Last pushed
Feb 26, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/scrapfly/python-scrapfly"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.