More than 70 automated checks sniff out broken links, redirect chains, and missing meta tags across your whole site, and SEOnaut, the self-hosted SEO audit crawler, sorts every finding into critical, alert, or warning so you know what to fix first. Add a website as a project, press Crawl Now, and watch URLs stream in with their status codes; a Stop button ends any run that wanders off. Reports also cover duplicate titles and descriptions, heading order, hreflang, canonicals, image alt text, thin content, orphan pages, and non-HTTPS URLs, while a passed-checks list shows what is already healthy. Dashboard charts follow crawl history, issue trends, status codes, link types, and page depth. Crawl options are set per project: ignore robots.txt, follow nofollow links, include noindex pages, read the sitemap, allow subdomains, check external links, or authenticate with HTTP Basic to audit a password-protected staging site. Page Details opens any URL with its inlinks, outgoing links, redirections, and images, and URL Explorer searches the crawled pages by term. Export sitemap.xml, CSV lists, or a replayable WACZ archive. Each person picks a light or dark theme and one of five interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

With over 164,000 GitHub stars and one of the fastest-growing open-source projects in the AI tooling ecosystem, Firecrawl is the web context API that turns any website into clean markdown, structured JSON, or screenshots optimized for large language models. The Scrape endpoint converts a single URL into LLM-ready output with approximately 67% fewer tokens than raw HTML, handling JavaScript rendering, rotating proxies, anti-bot bypasses, and dynamic content extraction with zero configuration. The Crawl endpoint recursively scrapes entire websites from a single request with configurable depth, URL filters, and concurrent page limits. The Map endpoint discovers all URLs on a domain instantly for sitemap generation. The Search endpoint performs web searches and returns full page content from results. The Interact endpoint scrapes a page then continues working with it — clicking buttons, filling forms, and extracting dynamic content using AI prompts or code. The Agent endpoint provides autonomous web data gathering where users describe what they need in plain English. SDKs are available for Python, Node.js, Go, Rust, Ruby, PHP, Java, C#/.NET, and Elixir, with an MCP server for connecting to any AI agent or MCP client. Self-hosting deploys via Docker Compose and requires Redis and a Playwright-based browser service for JavaScript rendering. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

CRW gives your apps and AI agents a private web data API that scrapes pages, crawls sites, maps links, and searches the web without sending every request to a hosted scraping service. Post a URL to the scrape endpoint and receive clean Markdown, HTML, screenshots, links, or structured JSON shaped by a schema you supply. Start a crawl when a whole section matters, then poll for progress while CRW walks pages asynchronously and returns the collected content in one job. Use map first to discover the URLs on a site before spending effort extracting pages you do not need. Connect agent tools through the MCP endpoint so assistants can search, scrape, and crawl from the same self-hosted service your applications call. Add browser rendering when JavaScript-heavy pages hide content from a plain HTTP fetch, and keep simpler pages on the fast lightweight path. Protect a public deployment with Bearer token authentication, your own reverse proxy, logging, and rate limits, or keep the service inside a private network for internal tools. Run one Docker container for the core API, or use the Compose stack when you want bundled search and rendering sidecars beside it. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Pagesift converts messy web pages, PDFs, and Office documents into clean Markdown and structured JSON that language models can actually use, from one self-hosted binary you run yourself. Call it the way you already call Firecrawl: the HTTP API speaks the Firecrawl v1 and v2 formats, so existing scrape, crawl, and batch jobs point at your server instead, and a Jina Reader style endpoint answers simple /r/ URL requests. Scrape a single page, submit batches, or crawl whole sites with sitemap support, focused crawling by query, robots.txt respect, and per-host rate limits by default. Headless rendering kicks in only when a page needs JavaScript, keeping plain pages fast. Documents work offline too: Word, Excel, PowerPoint, EPUB, and CSV files become Markdown with titles, links, images, and publication metadata attached. Jobs persist so long crawls survive restarts, webhooks report completions, and per-API-key limits and quotas keep multi-tenant use civilized, with Prometheus metrics and a health endpoint for operators. AI agents can drive the same engine directly through the bundled MCP server over stdio or HTTP, while developers get Python, Node, and Rust libraries. Memory stays bounded under load by design, sized from container limits rather than hope. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Lightpanda is a headless browser built for AI agents and automation that loads and runs web pages without ever drawing them, so crawling, scraping, and agent browsing finish far faster and lighter than in Chrome. You drive it with the tooling you already have: Puppeteer, Playwright, and Chromedp connect over the Chrome DevTools Protocol, Selenium works over WebDriver BiDi, and existing Chrome automation scripts keep running against the new endpoint. Scripts can execute JavaScript on real pages, wait for network activity, read and mutate the DOM, submit forms, and pull structured data out of dynamic sites that plain HTTP clients cannot see. An MCP server mode hands the browser to AI applications as a callable tool, so a model can open pages, follow links, and read content in the middle of an agent workflow. The built-in agent accepts plain-language browsing tasks and can export what it did as PandaScript, a replayable JavaScript file that runs later with no model call at runtime. A fetch command loads one or more URLs and prints the result as HTML or Markdown, which turns page extraction into a single shell step inside data pipelines. In the project's published crawl benchmark it processed pages roughly nine times faster than headless Chrome at a fraction of the memory, so one server can hold many more concurrent browsing sessions. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Webclaw turns messy web pages into clean content your AI tools can actually use, extracting the meaningful text and leaving the navigation, ads, and boilerplate behind. You point it at a URL and get back Markdown, plain text, structured JSON, raw HTML, or a format trimmed for language model input, with images stripped, links deduplicated, and whitespace cleaned. Crawl a whole site by following same-origin links page after page, map a domain first to discover every URL from its sitemap before deciding what to fetch, or batch many URLs at once and stream the results as they finish. CSS selectors let you include only the part of a page you care about or exclude the parts you never want, and main-content filtering keeps the article while dropping the chrome around it. Compare today's page against an earlier snapshot to see exactly what changed, or pull a site's colors, fonts, and logo into a brand profile. With your own language model connected, extraction follows a JSON schema or a plain prompt, and summaries come back at the length you ask for. Agents reach the same engine through an MCP server, scripts through the command line, and applications through the REST API. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

With over 77,000 GitHub stars, Crawl4AI is the most-starred open-source web crawler on GitHub and the go-to tool for converting the web into AI-ready data. Built on Python and Playwright, it transforms any website into clean Markdown with headings, tables, code blocks, and citation hints optimized for LLM ingestion, or extracts structured JSON via CSS selectors, XPath expressions, or direct LLM-based schema extraction through OpenAI, Anthropic, and Ollama providers. The self-hosted Docker server exposes a REST API on port 11235 with endpoints for crawling, streaming results, screenshots, PDF generation, JavaScript execution, and LLM-powered extraction. Version 0.9.x introduced secure-by-default operation with mandatory JWT authentication, strict request validation, declarative hooks replacing inline code, and bounded job queues. Adaptive crawling uses information foraging algorithms to determine when sufficient data has been gathered, while deep crawl mode traverses link graphs intelligently. The async browser pool manages concurrent sessions with stealth plugins, proxy rotation, custom headers, and session persistence for authenticated scraping. A built-in MCP server enables direct integration with Claude, ChatGPT, and Cursor for AI-driven web research workflows. Content filtering applies BM25 and TF-IDF relevance scoring to extract only pertinent sections from noisy pages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

With 3,100+ GitHub stars, CyberScraper 2077 replaces brittle CSS selectors and XPath queries with natural language data extraction powered by Large Language Models. Users paste a URL, describe the data they want in plain English, and the AI extracts structured results from any website — no HTML parsing knowledge required. The tool supports three LLM backends: OpenAI GPT models for maximum accuracy, Google Gemini for cost-effective extraction, and local Ollama instances for fully private scraping where URLs and data never leave your server. Built on Python asyncio with Playwright browser automation, it handles concurrent page fetching with content-based and query-based LRU caching to minimize redundant API calls. The Streamlit web interface runs on port 8501 and provides one-click export to JSON, CSV, HTML, SQL, Excel, and direct Google Sheets upload. Tor network integration routes requests through onion routing for anonymous scraping of both clearnet and .onion hidden service sites with automatic circuit management and stream isolation. Stealth mode randomizes user agents, manages cookies, and controls JavaScript execution timing to bypass bot detection systems. Multi-page scraping navigates through paginated results with automatic URL pattern detection. Docker deployment packages all dependencies including Playwright browsers into a single container. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

RankMe puts the whole SEO workbench on your own server: audits, rankings, keywords, backlinks, and AI visibility without a per-seat subscription deciding what you may check. Crawl a site and it runs 26 automated checks, returning a plain-language report sorted into Fix now, Watch, and Passed, where each finding names the page, explains in one sentence why it matters, and tells you how to fix it, with an optional AI summary on top. Rank tracking follows Google organic positions over time plus Bing, YouTube, and Amazon, watches SERP features, fires confirmed-drop alerts, and maps local rankings on a geogrid. Keyword research covers volume, difficulty, trends, and SERP-overlap clustering, while Search Console data feeds cannibalization reports and connects to GA4, PageSpeed Insights, and CrUX for performance context. Backlink profiles, link-gap analysis, toxic-link review, and competitor discovery show who links to rivals and which content earns it. A dedicated AI visibility module tracks prompts in AI Overviews and LLM answers and measures share of voice in a weekly digest. Agencies get PDF and CSV exports, scheduled white-label client reports, read-only share links, a REST API, and an MCP server, with alerts delivered by email, Slack, or signed webhook across seven interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

OpenSEO provides an alternative to the $100+/month SEO suite monopoly by connecting directly to DataForSEO APIs on a pay-as-you-go basis — you bring your own API key and pay only for the queries you actually make. The platform delivers focused workflows for keyword research with volume, difficulty, CPC, intent, and live SERP inspection; rank tracking with historical position monitoring; backlink analysis with referring domain quality scoring; technical site audits that crawl pages and surface issues; competitor insights comparing SERP positions across keyword sets; and AI visibility monitoring that tracks brand mentions, citations, and prompts across AI models. The integrated MCP server exposes all tools to AI coding agents including Claude Code, OpenClaw, Hermes, and Cursor, while Agent Skills provide reusable SKILL.md workflows that guide agents through complex SEO research tasks. Google Search Console integration reads first-party performance data including clicks, impressions, CTR, and position metrics with URL-level index status inspection. The Prompt Explorer compares how different AI models respond to SEO-relevant queries. Self-host via Docker for personal use or deploy to Cloudflare for team access with automatic database backups. The hosted version at openseo.so adds a 28% markup on DataForSEO requests. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Unlimited domains, unlimited keywords, daily Google position checks with stored history and trend charts: SerpBear is an open-source search engine rank tracker. Retrieval works through your choice of third-party SERP APIs - ScrapingAnt, ScrapingRobot, SerpApi, SearchApi, HasData - or your own proxy IP pool, and a flexible scrape-strategy system (Basic, Custom, or Smart, set globally or per domain) works around Google's removal of the 100-results-per-page parameter by choosing how many pages to check per keyword. Google Search Console integration adds real visit counts, impressions, and click-through rates per tracked keyword and surfaces top-performing pages and countries; Google Ads integration supplies monthly search volumes and auto-generates keyword ideas from your site's content. Email notifications report position changes daily, weekly, or monthly, a built-in REST API feeds dashboards and reporting tools, and data exports to CSV. Built with Next.js on SQLite, deployed via Docker, installable as a PWA on mobile - with no per-keyword or monthly SaaS fees.

Bisibility puts your Google rankings, keyword research, and backlink profile in one application that runs on your server instead of behind a per-seat subscription. Track every keyword by market and device, watch positions move over time, and open the stored snapshot of the actual results page behind any check to see exactly what Google showed. Alerts flag the movements that matter, while competitor share of voice shows whose visibility is growing at your expense. The keyword explorer surfaces suggestions with search volume, trends, cost per click, difficulty, and intent, and shows the estimated price before a lookup runs, so research never surprises your wallet. Backlink and domain views break down referring domains, anchors, and new and lost links next to estimated traffic and top pages for any site you type in. Search Console clicks, impressions, and queries are kept beyond the usual retention window, and a recurring prompt tracker follows how AI answer engines cite your brand over time. Results flow outward through a versioned API, MCP access, signed webhooks, and CSV exports. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Several EU data protection authorities have ruled Google Analytics deployments unlawful; Matomo (formerly Piwik) is the most complete open-source replacement - a full analytics platform with 30+ report types across visitors, actions, referrers, goals, and ecommerce. The self-hosted PHP/MySQL edition is free and keeps every byte of visitor data on your infrastructure, which matters more each year: several EU data protection authorities have ruled Google Analytics deployments unlawful, while Matomo configured for cookieless tracking is approved by France's CNIL for use without a consent banner. All reporting runs on 100% unsampled data - no extrapolation at high traffic volumes. The GDPR Manager handles data subject requests and deletion, with IP anonymization, retention controls, and Do Not Track support built in. A dedicated importer pulls your historical Google Analytics data so years of trends survive the migration. Core analytics cover campaigns, custom variables and dimensions, entry/exit pages, downloads, site search, and full ecommerce tracking with a comprehensive HTTP API for reporting and ingestion. Premium plugins extend the platform into Hotjar-class behavioral tooling - click and scroll heatmaps, session recordings, conversion funnels, form analytics, A/B testing - plus a tag manager and SAML SSO. For teams that need GA-equivalent depth with actual data ownership, Matomo is the realistic drop-in replacement.

Built as a direct rejection of the adtech model, Plausible is the best-known privacy-first web analytics tool - lightweight, cookie-free, and open-source. It sets no cookies and stores no personal data: unique visitors are counted via a hash of IP plus User-Agent that rotates every 24 hours and is never stored raw, so no consent banner is required and GDPR compliance is structural rather than contractual. The tracking script is under 1 KB - orders of magnitude lighter than GA - and the dashboard is a deliberate contrast to GA4's sprawl: one fast-loading page with visitors, sources, top pages, countries, devices, and UTM breakdowns, filterable by any dimension. Custom events and goals track signups and clicks, Google Search Console integration pulls in search queries, scheduled email reports keep stakeholders updated, and the Stats API (v2) plus CSV export feed data anywhere. This is the AGPL-licensed Community Edition, the same Elixir codebase that powers Plausible's cloud service, running as three containers: the web app, PostgreSQL for accounts, and ClickHouse for event storage - which means self-hosters get direct SQL access to raw analytics data the cloud version never exposes. Traffic data stays entirely on your server, with no visitor caps or per-pageview pricing.

Search is being answered by AI crawlers first and humans second, yet most site owners cannot name a single bot that read their pages last week; Geo-Monitor answers that from the access logs you already have. A small collector sits beside Nginx or OpenResty, mounts the log read-only, and streams only new lines in batches, resuming after outages and surviving log rotation without ever touching your site's code. The console separates today, the last week, and the last month into identified requests, crawled pages, active crawlers, and trend lines, then drills into every visit with time, path, method, status code, claimed identity, and verification state. Identification and verification are deliberately different things here: a user agent only proves what a request claims to be, while a verified badge requires trusted log addresses and official IP evidence to agree. Twenty built-in identities cover OpenAI, Anthropic, Perplexity, Kimi, and Bytespider crawlers, and you can add, edit, or disable rules as new bots appear. Your sitemap syncs or imports as the content catalog, so coverage views show which pages crawlers actually reached, and collector health exposes heartbeat, backlog, and parsing problems. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.