oxylabs/ai-map-py
AI Map is an AI-powered website mapping tool by Oxylabs AI Studio that uses natural language prompts to intelligently discover and extract relevant URLs from any website.
This tool helps you quickly get a structured list of specific pages from any website, like all blog posts or product pages. You tell it what kind of pages you're looking for using everyday language, and it gives you a clear list of URLs. It's ideal for market researchers, SEO specialists, or data analysts who need to quickly identify and categorize web content.
1,108 stars. Actively maintained with 1 commit in the last 30 days.
Use this if you need to rapidly map out specific types of content on a website, such as all job postings, product categories, or documentation articles, using simple natural language prompts.
Not ideal if you need to extract specific data fields from within those pages rather than just finding the URLs, or if the website requires authentication.
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/oxylabs/ai-map-py"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.