The leading open-source tool for extracting business leads from Google Maps at production scale. The Go-based engine processes approximately 120 places per minute with optimized concurrency, extracting 33+ data points per listing including business name, address, phone number, website URL, rating, review count, latitude and longitude, opening hours, price level, and optionally crawling business websites for email addresses. Three interfaces serve different workflows: the CLI accepts query files for cron jobs and CI/CD pipelines with output to CSV, JSON, PostgreSQL, S3, or LeadsDB; the Web UI provides a browser-based dashboard with real-time job monitoring, a map view of scraped places, and interactive query submission; and the REST API at /api/v1 enables programmatic integration with full Swagger documentation at /api/docs. Built-in proxy rotation supports SOCKS5, HTTP, and HTTPS with authentication for large-scale runs, while the architecture scales from a laptop to Kubernetes clusters with queue-based worker distribution. The SaaS edition adds multi-user access with API key management, admin UI with 2FA, job queue orchestration, and one-command cloud deployment via an interactive wizard. An AI Agent Skill enables coding agents to run scrapes programmatically. Deploy via Docker or build from source requiring Go 1.26.5+. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

With over 164,000 GitHub stars and one of the fastest-growing open-source projects in the AI tooling ecosystem, Firecrawl is the web context API that turns any website into clean markdown, structured JSON, or screenshots optimized for large language models. The Scrape endpoint converts a single URL into LLM-ready output with approximately 67% fewer tokens than raw HTML, handling JavaScript rendering, rotating proxies, anti-bot bypasses, and dynamic content extraction with zero configuration. The Crawl endpoint recursively scrapes entire websites from a single request with configurable depth, URL filters, and concurrent page limits. The Map endpoint discovers all URLs on a domain instantly for sitemap generation. The Search endpoint performs web searches and returns full page content from results. The Interact endpoint scrapes a page then continues working with it — clicking buttons, filling forms, and extracting dynamic content using AI prompts or code. The Agent endpoint provides autonomous web data gathering where users describe what they need in plain English. SDKs are available for Python, Node.js, Go, Rust, Ruby, PHP, Java, C#/.NET, and Elixir, with an MCP server for connecting to any AI agent or MCP client. Self-hosting deploys via Docker Compose and requires Redis and a Playwright-based browser service for JavaScript rendering. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Pagesift converts messy web pages, PDFs, and Office documents into clean Markdown and structured JSON that language models can actually use, from one self-hosted binary you run yourself. Call it the way you already call Firecrawl: the HTTP API speaks the Firecrawl v1 and v2 formats, so existing scrape, crawl, and batch jobs point at your server instead, and a Jina Reader style endpoint answers simple /r/ URL requests. Scrape a single page, submit batches, or crawl whole sites with sitemap support, focused crawling by query, robots.txt respect, and per-host rate limits by default. Headless rendering kicks in only when a page needs JavaScript, keeping plain pages fast. Documents work offline too: Word, Excel, PowerPoint, EPUB, and CSV files become Markdown with titles, links, images, and publication metadata attached. Jobs persist so long crawls survive restarts, webhooks report completions, and per-API-key limits and quotas keep multi-tenant use civilized, with Prometheus metrics and a health endpoint for operators. AI agents can drive the same engine directly through the bundled MCP server over stdio or HTTP, while developers get Python, Node, and Rust libraries. Memory stays bounded under load by design, sized from container limits rather than hope. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Webclaw turns messy web pages into clean content your AI tools can actually use, extracting the meaningful text and leaving the navigation, ads, and boilerplate behind. You point it at a URL and get back Markdown, plain text, structured JSON, raw HTML, or a format trimmed for language model input, with images stripped, links deduplicated, and whitespace cleaned. Crawl a whole site by following same-origin links page after page, map a domain first to discover every URL from its sitemap before deciding what to fetch, or batch many URLs at once and stream the results as they finish. CSS selectors let you include only the part of a page you care about or exclude the parts you never want, and main-content filtering keeps the article while dropping the chrome around it. Compare today's page against an earlier snapshot to see exactly what changed, or pull a site's colors, fonts, and logo into a brand profile. With your own language model connected, extraction follows a JSON schema or a plain prompt, and summaries come back at the length you ask for. Agents reach the same engine through an MCP server, scripts through the command line, and applications through the REST API. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Obscura is the headless browser you point Puppeteer or Playwright at when Chromium feels like renting a truck to fetch a letter. Start the server and your existing automation connects over the usual DevTools protocol, opens pages, clicks, types, waits for selectors, and reads the fully rendered result, JavaScript included. Scrape one page or fan out across many in parallel, evaluating an expression on each and getting structured output back. Capture viewport or full-page screenshots, export PDFs, intercept requests to block or rewrite them, and manage cookies and headers per session. An MCP mode lets AI agents drive the browser as a tool instead of shelling out to scripts. Stealth mode randomizes fingerprints per session, presents realistic browser identity signals, and blocks thousands of known tracker domains before they load. Proxies, persistent storage for cookies and local storage, and robots handling round out the controls for longer crawls. The whole thing starts instantly and sips memory compared with a conventional headless Chrome, so a modest server can keep many more pages in flight. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

With 2,000 GitHub stars and 10 releases since its September 2025 launch, HeadlessX delivers a self-hosted browser automation platform that replaces Chromium-based scraping with Camoufox — a Firefox fork performing kernel-level fingerprint spoofing to achieve 0% detection across Cloudflare, DataDome, PerimeterX, and other anti-bot systems where Puppeteer and Playwright regularly fail. The web dashboard provides workspace-based job organization with a visual interface for configuring scrape targets, managing browser profiles, monitoring queue status, and viewing extracted results in real time. The protected REST API accepts requests with API key authentication for programmatic access, supporting HTML extraction, screenshot capture, PDF generation, and structured data parsing with configurable stealth parameters. Profile-based scraping maintains persistent browser contexts with cookie jars, localStorage, and fingerprint configurations that survive between requests — reducing cold-start latency from 25 seconds to under 2 seconds on subsequent requests. Queue-backed workflows enable batch processing of URLs with configurable concurrency, retry logic, and webhook notifications on completion. The Google AI Search integration provides AI-assisted web research workflows through dedicated endpoints. Remote MCP support exposes automation capabilities as tool endpoints for AI agent integration. Deploy via the official CLI with `headlessx init` and `headlessx start` commands, scaffolding a Docker Compose stack with Caddy reverse proxy for automatic HTTPS. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Lightpanda is a headless browser built for AI agents and automation that loads and runs web pages without ever drawing them, so crawling, scraping, and agent browsing finish far faster and lighter than in Chrome. You drive it with the tooling you already have: Puppeteer, Playwright, and Chromedp connect over the Chrome DevTools Protocol, Selenium works over WebDriver BiDi, and existing Chrome automation scripts keep running against the new endpoint. Scripts can execute JavaScript on real pages, wait for network activity, read and mutate the DOM, submit forms, and pull structured data out of dynamic sites that plain HTTP clients cannot see. An MCP server mode hands the browser to AI applications as a callable tool, so a model can open pages, follow links, and read content in the middle of an agent workflow. The built-in agent accepts plain-language browsing tasks and can export what it did as PandaScript, a replayable JavaScript file that runs later with no model call at runtime. A fetch command loads one or more URLs and prints the result as HTML or Markdown, which turns page extraction into a single shell step inside data pipelines. In the project's published crawl benchmark it processed pages roughly nine times faster than headless Chrome at a fraction of the memory, so one server can hold many more concurrent browsing sessions. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

With 3,100+ GitHub stars, CyberScraper 2077 replaces brittle CSS selectors and XPath queries with natural language data extraction powered by Large Language Models. Users paste a URL, describe the data they want in plain English, and the AI extracts structured results from any website — no HTML parsing knowledge required. The tool supports three LLM backends: OpenAI GPT models for maximum accuracy, Google Gemini for cost-effective extraction, and local Ollama instances for fully private scraping where URLs and data never leave your server. Built on Python asyncio with Playwright browser automation, it handles concurrent page fetching with content-based and query-based LRU caching to minimize redundant API calls. The Streamlit web interface runs on port 8501 and provides one-click export to JSON, CSV, HTML, SQL, Excel, and direct Google Sheets upload. Tor network integration routes requests through onion routing for anonymous scraping of both clearnet and .onion hidden service sites with automatic circuit management and stream isolation. Stealth mode randomizes user agents, manages cookies, and controls JavaScript execution timing to bypass bot detection systems. Multi-page scraping navigates through paginated results with automatic URL pattern detection. Docker deployment packages all dependencies including Playwright browsers into a single container. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

With over 77,000 GitHub stars, Crawl4AI is the most-starred open-source web crawler on GitHub and the go-to tool for converting the web into AI-ready data. Built on Python and Playwright, it transforms any website into clean Markdown with headings, tables, code blocks, and citation hints optimized for LLM ingestion, or extracts structured JSON via CSS selectors, XPath expressions, or direct LLM-based schema extraction through OpenAI, Anthropic, and Ollama providers. The self-hosted Docker server exposes a REST API on port 11235 with endpoints for crawling, streaming results, screenshots, PDF generation, JavaScript execution, and LLM-powered extraction. Version 0.9.x introduced secure-by-default operation with mandatory JWT authentication, strict request validation, declarative hooks replacing inline code, and bounded job queues. Adaptive crawling uses information foraging algorithms to determine when sufficient data has been gathered, while deep crawl mode traverses link graphs intelligently. The async browser pool manages concurrent sessions with stealth plugins, proxy rotation, custom headers, and session persistence for authenticated scraping. A built-in MCP server enables direct integration with Claude, ChatGPT, and Cursor for AI-driven web research workflows. Content filtering applies BM25 and TF-IDF relevance scoring to extract only pertinent sections from noisy pages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

CRW gives your apps and AI agents a private web data API that scrapes pages, crawls sites, maps links, and searches the web without sending every request to a hosted scraping service. Post a URL to the scrape endpoint and receive clean Markdown, HTML, screenshots, links, or structured JSON shaped by a schema you supply. Start a crawl when a whole section matters, then poll for progress while CRW walks pages asynchronously and returns the collected content in one job. Use map first to discover the URLs on a site before spending effort extracting pages you do not need. Connect agent tools through the MCP endpoint so assistants can search, scrape, and crawl from the same self-hosted service your applications call. Add browser rendering when JavaScript-heavy pages hide content from a plain HTTP fetch, and keep simpler pages on the fast lightweight path. Protect a public deployment with Bearer token authentication, your own reverse proxy, logging, and rate limits, or keep the service inside a private network for internal tools. Run one Docker container for the core API, or use the Compose stack when you want bundled search and rendering sidecars beside it. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

With over 16,800 GitHub stars and growing rapidly, Maxun has become the go-to open-source platform for teams who need structured web data without writing scrapers. The TypeScript-based platform provides a no-code visual recorder that captures point-and-click interactions in real-time browser sync, automatically generating reusable extraction robots that handle pagination, infinite scrolling, and dynamic content. LLM-powered extraction accepts natural language prompts like "Extract 10 companies from the Y Combinator website" without requiring a URL — Maxun identifies the source and performs the extraction autonomously. The platform handles authentication-protected pages, adapts automatically to website layout changes through self-healing selectors, and exports directly to Google Sheets, Airtable, or any destination via webhooks. Robots run on configurable schedules with cron-based timing, turning any website into a perpetually fresh RESTful API endpoint. The crawl engine discovers and processes linked pages across entire domains with configurable depth and URL filtering, while the search capability runs automated queries across multiple engines. Official Node.js and Python SDKs provide programmatic control over robot creation, execution, and data retrieval, with MCP integration enabling direct connection to AI tools like Claude. The n8n community node enables workflow automation without custom code. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPLv3 licensed.

Scoring 64.4 on the WebBench benchmark — state-of-the-art among browser automation platforms — Skyvern replaces brittle XPath-based scripts with Vision LLM reasoning that adapts when websites change their layouts. The platform extends Playwright with AI-powered page methods including page.act(), page.extract(), and page.validate() that accept natural language prompts while still supporting traditional CSS selectors as fallback. The drag-and-drop Workflow Studio offers 17+ block types including navigation, extraction, login, loops, conditionals, code blocks, file download, and file upload — enabling non-technical users to build complex multi-step automations without writing code. Self-hosted deployments support bring-your-own-LLM with OpenAI, Anthropic, Gemini, and Ollama, while the multi-engine architecture allows swapping between Skyvern 2.0, OpenAI CUA, Anthropic CUA, or UI-TARS per task with a single parameter. Built-in infrastructure handles persistent browser sessions preserving cookies and localStorage across runs, automatic CAPTCHA solving for reCAPTCHA and hCaptcha, anti-bot bypass for Cloudflare and DataDome, residential proxy rotation across 20+ countries, and a credential vault integrating with Bitwarden and 1Password for secure 2FA management. Real-time session livestreaming via WebRTC enables visual debugging, while step-by-step action logs with screenshots and full LLM diagnostic traces provide production observability. The MCP server integration exposes Skyvern as a tool for Claude, Cursor, Windsurf, and any MCP-compatible AI agent. Connect to 6,000+ apps through Zapier, Make.com, or self-hosted N8N workflows. Deploy via Docker Compose or pip install with a two-command setup. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Every account you run deserves its own browser identity, and FoxProfile keeps those identities permanently separate instead of letting them bleed into one another. Create a profile once and it remembers its device: screen, GPU, CPU cores, fonts, and canvas noise are generated on first launch and reused on every later one, so the fingerprint a site saw yesterday is the fingerprint it sees tomorrow. Sessions stick too: history, cookies, and logins live in the profile directory, and the tabs you left open reopen where you left them. Each profile carries its own proxy (HTTP, HTTPS, SOCKS4, or SOCKS5, with a built-in checker) while timezone and locale follow the exit IP automatically or stay pinned to your choice, and the Check IP button compares the exit address across three services and flags mismatches or datacenter addresses. Cookies move in and out as standard editor JSON or Netscape files, whole profiles travel as ZIP archives, and bulk actions launch, stop, or delete fleets at once. On a server, the web panel runs profiles headlessly so you can manage them and view their screens from anywhere, the REST API scripts everything, and the MCP server lets AI agents drive the browsers directly. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

With over 7,400 GitHub stars and benchmarked at 0.89 seconds average session lifecycle — 1.7x to 9x faster than competing browser automation platforms — Steel Browser delivers production-grade headless Chrome infrastructure purpose-built for AI agents that need to interact with the modern web. The TypeScript-based server exposes a REST API providing on-demand browser sessions with full CDP (Chrome DevTools Protocol) access, allowing connections from Puppeteer, Playwright, or Selenium through standard WebSocket endpoints without framework lock-in. Each session maintains persistent state including cookies, localStorage, IndexedDB, and authentication credentials across requests, enabling stateful multi-step agent workflows that survive session restarts. Built-in anti-detection includes stealth plugins, browser fingerprint randomization, and configurable user-agent rotation, while the proxy chain manager handles IP rotation through residential, datacenter, or custom proxy pools. CAPTCHA solving integrates natively so agents encounter fewer blocking interrupts during autonomous navigation. The Session Viewer provides real-time WebRTC-streamed visual debugging of live sessions and playback of recorded sessions with full network request logging. Browser Tools APIs convert any page to clean Markdown, readability-optimized text, PDF documents, or high-resolution screenshots with a single API call. The MCP Server integration exposes Steel sessions as tools accessible to Claude, Cursor, and other Model Context Protocol-compatible AI agents. Deploy via Docker with a single container or use Docker Compose for production configurations with automatic resource cleanup and session lifecycle management. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

More than 70 automated checks sniff out broken links, redirect chains, and missing meta tags across your whole site, and SEOnaut, the self-hosted SEO audit crawler, sorts every finding into critical, alert, or warning so you know what to fix first. Add a website as a project, press Crawl Now, and watch URLs stream in with their status codes; a Stop button ends any run that wanders off. Reports also cover duplicate titles and descriptions, heading order, hreflang, canonicals, image alt text, thin content, orphan pages, and non-HTTPS URLs, while a passed-checks list shows what is already healthy. Dashboard charts follow crawl history, issue trends, status codes, link types, and page depth. Crawl options are set per project: ignore robots.txt, follow nofollow links, include noindex pages, read the sitemap, allow subdomains, check external links, or authenticate with HTTP Basic to audit a password-protected staging site. Page Details opens any URL with its inlinks, outgoing links, redirections, and images, and URL Explorer searches the crawled pages by term. Export sitemap.xml, CSV lists, or a replayable WACZ archive. Each person picks a light or dark theme and one of five interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

RankMe puts the whole SEO workbench on your own server: audits, rankings, keywords, backlinks, and AI visibility without a per-seat subscription deciding what you may check. Crawl a site and it runs 26 automated checks, returning a plain-language report sorted into Fix now, Watch, and Passed, where each finding names the page, explains in one sentence why it matters, and tells you how to fix it, with an optional AI summary on top. Rank tracking follows Google organic positions over time plus Bing, YouTube, and Amazon, watches SERP features, fires confirmed-drop alerts, and maps local rankings on a geogrid. Keyword research covers volume, difficulty, trends, and SERP-overlap clustering, while Search Console data feeds cannibalization reports and connects to GA4, PageSpeed Insights, and CrUX for performance context. Backlink profiles, link-gap analysis, toxic-link review, and competitor discovery show who links to rivals and which content earns it. A dedicated AI visibility module tracks prompts in AI Overviews and LLM answers and measures share of voice in a weekly digest. Agencies get PDF and CSV exports, scheduled white-label client reports, read-only share links, a REST API, and an MCP server, with alerts delivered by email, Slack, or signed webhook across seven interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Unlimited domains, unlimited keywords, daily Google position checks with stored history and trend charts: SerpBear is an open-source search engine rank tracker. Retrieval works through your choice of third-party SERP APIs - ScrapingAnt, ScrapingRobot, SerpApi, SearchApi, HasData - or your own proxy IP pool, and a flexible scrape-strategy system (Basic, Custom, or Smart, set globally or per domain) works around Google's removal of the 100-results-per-page parameter by choosing how many pages to check per keyword. Google Search Console integration adds real visit counts, impressions, and click-through rates per tracked keyword and surfaces top-performing pages and countries; Google Ads integration supplies monthly search volumes and auto-generates keyword ideas from your site's content. Email notifications report position changes daily, weekly, or monthly, a built-in REST API feeds dashboards and reporting tools, and data exports to CSV. Built with Next.js on SQLite, deployed via Docker, installable as a PWA on mobile - with no per-keyword or monthly SaaS fees.

Instead of trusting one search company with every query, degoog is a self-hosted search aggregator that fans each one out to the engines you choose, from your own server, and merges the answers into one deduplicated, ranked list. It installs with no engines at all; a first-run tour then walks you through picking Google, Bing, DuckDuckGo, Brave, Startpage, Reddit, or Wikipedia from the built-in Store. Pages found by several engines collapse into a single row that lists every source, while a sidebar times each engine, explains failures like rate limits or CAPTCHAs, and retries only the one that failed. Tabs appear for each search type your engines serve, and the Images tab adds size, color, layout, and safe-mode filters beside a zoomable preview panel. Bangs target one engine or run commands (!w for Wikipedia, !ip, !speedtest, !uuid), and Store plugins add weather, definitions, QR codes, and search across your own Jellyfin or Meilisearch libraries. Owners can block, rewrite, or score domains, serve a script-free page to text browsers, and turn on an indexer that remembers seen results for later recall. A JSON search API, a SearXNG-compatible response mode for Open WebUI, and an MCP companion feed the same results to scripts and LLM clients. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Up to 280 search services - Google, Bing, DuckDuckGo, Brave, Qwant, Startpage - aggregated without tracking or profiling: SearXNG is a privacy-respecting metasearch engine (AGPL-3.0, successor to Searx). Your instance queries the upstream engines on your behalf: your IP address, cookies, and search history never reach them, tracker parameters are stripped from result URLs, and an optional image proxy fetches thumbnails server-side so result pages leak nothing. It can even route outbound queries through Tor for full anonymity. Search is organized into categories - general, images, videos, news, maps, music, IT, science, files - with bang shortcuts for targeting specific engines, and every source can be enabled, disabled, or weighted per category in settings.yml. A plugin system adds calculators, hash tools, tracker removal, and unit conversions inline, and preferences (themes, safe search, languages, engine selection) persist in cookies rather than server-side accounts. The real argument for running your own instance rather than trusting a public one is control: you decide the logging policy (none), the engine mix, rate limiting, and who gets access - making it the default search backend for browsers, families, and teams that want Google-quality results without the profile.

Google's search results without Google's surveillance: Whoogle is a self-hosted proxy that strips the tracking and keeps the results. Your query goes from browser to your Whoogle instance, which fetches results from Google with a randomly generated User Agent and strips everything hostile before returning them: no ads or sponsored content, no third-party JavaScript or cookies, no AMP links, no URL tracking tags like utm_source, no referrer header - and Google sees your server's IP, never yours. Unlike metasearch engines that blend sources, Whoogle proxies Google exclusively, so result quality is exactly what you'd get logged out and incognito, minus the noise. A lightweight Flask app configured entirely through environment variables, it supports DuckDuckGo-style bang shortcuts, autocomplete suggestions, safe search, per-country and per-language filtering, site blocklists, and automatic rewriting of social links to privacy front-ends like Nitter and Invidious. Privacy hardening goes further: built-in Tor routing makes Google see an exit node instead of your server, HTTP/SOCKS proxy support covers other setups, and POST-based queries keep search terms out of logs. Light, dark, and fully custom CSS themes plus browser search-engine registration make it a drop-in default on desktop and mobile. Stateless, tiny, and trivial to run.