RustXNG gives your AI agents a private web search API that needs no keys, no accounts, and no per-search fees: ask once and get ranked results plus the text of the top pages, ready to ground an answer in real sources. A single request fans out to 180 search engines in parallel, drawn from 226 engine types covering general web, images, video, academic literature, package registries, and news, then merges what comes back with its own ranking that rewards term coverage and canonical-site authority instead of trusting any one engine's order. Engines that time out or keep answering nothing are suspended with backoff and rested automatically, so one dead source never stalls a task, and a repeated query returns from cache in about a hundredth of a second. Call it over HTTP with the familiar SearXNG-shaped JSON, CSV, RSS, or HTML formats, narrow results by domain, engine, or time range, or run the built-in MCP server and hand your agent native search and read_page tools with bounded, truncation-aware responses. Page fetches are hardened against untrusted URLs, an experimental local index can crawl your own corpus and merge it into results, and health, metrics, and per-engine status stay visible through the ops endpoints and the browser UI. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Google Browser Scraper is a self-hosted SERP API that gets Google results from a real Chrome browser and your own residential proxy, because plain HTTP scraping is refused and a fresh browser on a fresh IP is refused almost as fast. It warms a sticky proxy session with ordinary browsing first, lets the browser pass Google's checks, and then switches to a hybrid mode that fetches results pages directly over HTTPS with the browser's own cookies, at about 0.4 MB and 1.4 seconds per search on a warm IP. Every page comes back as SERP-style JSON with organic results, ads, related questions, related searches, pagination, the AI Overview, and local results. If a proxy IP starts serving poisoned pages for some other query, the scraper catches it within a few pages, drops that IP, and asks the query again on a fresh one. Captchas and blocks are reported as captchas and blocks, never disguised as an empty result set, so your pipeline knows when to retry or rotate. Run it as a CLI for keyword files, import it as a Python library, serve it as a local HTTP API, or expose the MCP server so AI agents get live Google search. Every run ends with a traffic and cost summary per thousand pages, and any sticky residential proxy works, with NodeMaven sessions handled for you. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Instead of trusting one search company with every query, degoog is a self-hosted search aggregator that fans each one out to the engines you choose, from your own server, and merges the answers into one deduplicated, ranked list. It installs with no engines at all; a first-run tour then walks you through picking Google, Bing, DuckDuckGo, Brave, Startpage, Reddit, or Wikipedia from the built-in Store. Pages found by several engines collapse into a single row that lists every source, while a sidebar times each engine, explains failures like rate limits or CAPTCHAs, and retries only the one that failed. Tabs appear for each search type your engines serve, and the Images tab adds size, color, layout, and safe-mode filters beside a zoomable preview panel. Bangs target one engine or run commands (!w for Wikipedia, !ip, !speedtest, !uuid), and Store plugins add weather, definitions, QR codes, and search across your own Jellyfin or Meilisearch libraries. Owners can block, rewrite, or score domains, serve a script-free page to text browsers, and turn on an indexer that remembers seen results for later recall. A JSON search API, a SearXNG-compatible response mode for Open WebUI, and an MCP companion feed the same results to scripts and LLM clients. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

CRW gives your apps and AI agents a private web data API that scrapes pages, crawls sites, maps links, and searches the web without sending every request to a hosted scraping service. Post a URL to the scrape endpoint and receive clean Markdown, HTML, screenshots, links, or structured JSON shaped by a schema you supply. Start a crawl when a whole section matters, then poll for progress while CRW walks pages asynchronously and returns the collected content in one job. Use map first to discover the URLs on a site before spending effort extracting pages you do not need. Connect agent tools through the MCP endpoint so assistants can search, scrape, and crawl from the same self-hosted service your applications call. Add browser rendering when JavaScript-heavy pages hide content from a plain HTTP fetch, and keep simpler pages on the fast lightweight path. Protect a public deployment with Bearer token authentication, your own reverse proxy, logging, and rate limits, or keep the service inside a private network for internal tools. Run one Docker container for the core API, or use the Compose stack when you want bundled search and rendering sidecars beside it. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

With over 164,000 GitHub stars and one of the fastest-growing open-source projects in the AI tooling ecosystem, Firecrawl is the web context API that turns any website into clean markdown, structured JSON, or screenshots optimized for large language models. The Scrape endpoint converts a single URL into LLM-ready output with approximately 67% fewer tokens than raw HTML, handling JavaScript rendering, rotating proxies, anti-bot bypasses, and dynamic content extraction with zero configuration. The Crawl endpoint recursively scrapes entire websites from a single request with configurable depth, URL filters, and concurrent page limits. The Map endpoint discovers all URLs on a domain instantly for sitemap generation. The Search endpoint performs web searches and returns full page content from results. The Interact endpoint scrapes a page then continues working with it — clicking buttons, filling forms, and extracting dynamic content using AI prompts or code. The Agent endpoint provides autonomous web data gathering where users describe what they need in plain English. SDKs are available for Python, Node.js, Go, Rust, Ruby, PHP, Java, C#/.NET, and Elixir, with an MCP server for connecting to any AI agent or MCP client. Self-hosting deploys via Docker Compose and requires Redis and a Playwright-based browser service for JavaScript rendering. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Pagesift converts messy web pages, PDFs, and Office documents into clean Markdown and structured JSON that language models can actually use, from one self-hosted binary you run yourself. Call it the way you already call Firecrawl: the HTTP API speaks the Firecrawl v1 and v2 formats, so existing scrape, crawl, and batch jobs point at your server instead, and a Jina Reader style endpoint answers simple /r/ URL requests. Scrape a single page, submit batches, or crawl whole sites with sitemap support, focused crawling by query, robots.txt respect, and per-host rate limits by default. Headless rendering kicks in only when a page needs JavaScript, keeping plain pages fast. Documents work offline too: Word, Excel, PowerPoint, EPUB, and CSV files become Markdown with titles, links, images, and publication metadata attached. Jobs persist so long crawls survive restarts, webhooks report completions, and per-API-key limits and quotas keep multi-tenant use civilized, with Prometheus metrics and a health endpoint for operators. AI agents can drive the same engine directly through the bundled MCP server over stdio or HTTP, while developers get Python, Node, and Rust libraries. Memory stays bounded under load by design, sized from container limits rather than hope. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

Webclaw turns messy web pages into clean content your AI tools can actually use, extracting the meaningful text and leaving the navigation, ads, and boilerplate behind. You point it at a URL and get back Markdown, plain text, structured JSON, raw HTML, or a format trimmed for language model input, with images stripped, links deduplicated, and whitespace cleaned. Crawl a whole site by following same-origin links page after page, map a domain first to discover every URL from its sitemap before deciding what to fetch, or batch many URLs at once and stream the results as they finish. CSS selectors let you include only the part of a page you care about or exclude the parts you never want, and main-content filtering keeps the article while dropping the chrome around it. Compare today's page against an earlier snapshot to see exactly what changed, or pull a site's colors, fonts, and logo into a brand profile. With your own language model connected, extraction follows a JSON schema or a plain prompt, and summaries come back at the length you ask for. Agents reach the same engine through an MCP server, scripts through the command line, and applications through the REST API. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

With over 77,000 GitHub stars, Crawl4AI is the most-starred open-source web crawler on GitHub and the go-to tool for converting the web into AI-ready data. Built on Python and Playwright, it transforms any website into clean Markdown with headings, tables, code blocks, and citation hints optimized for LLM ingestion, or extracts structured JSON via CSS selectors, XPath expressions, or direct LLM-based schema extraction through OpenAI, Anthropic, and Ollama providers. The self-hosted Docker server exposes a REST API on port 11235 with endpoints for crawling, streaming results, screenshots, PDF generation, JavaScript execution, and LLM-powered extraction. Version 0.9.x introduced secure-by-default operation with mandatory JWT authentication, strict request validation, declarative hooks replacing inline code, and bounded job queues. Adaptive crawling uses information foraging algorithms to determine when sufficient data has been gathered, while deep crawl mode traverses link graphs intelligently. The async browser pool manages concurrent sessions with stealth plugins, proxy rotation, custom headers, and session persistence for authenticated scraping. A built-in MCP server enables direct integration with Claude, ChatGPT, and Cursor for AI-driven web research workflows. Content filtering applies BM25 and TF-IDF relevance scoring to extract only pertinent sections from noisy pages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

With 3,100+ GitHub stars, CyberScraper 2077 replaces brittle CSS selectors and XPath queries with natural language data extraction powered by Large Language Models. Users paste a URL, describe the data they want in plain English, and the AI extracts structured results from any website — no HTML parsing knowledge required. The tool supports three LLM backends: OpenAI GPT models for maximum accuracy, Google Gemini for cost-effective extraction, and local Ollama instances for fully private scraping where URLs and data never leave your server. Built on Python asyncio with Playwright browser automation, it handles concurrent page fetching with content-based and query-based LRU caching to minimize redundant API calls. The Streamlit web interface runs on port 8501 and provides one-click export to JSON, CSV, HTML, SQL, Excel, and direct Google Sheets upload. Tor network integration routes requests through onion routing for anonymous scraping of both clearnet and .onion hidden service sites with automatic circuit management and stream isolation. Stealth mode randomizes user agents, manages cookies, and controls JavaScript execution timing to bypass bot detection systems. Multi-page scraping navigates through paginated results with automatic URL pattern detection. Docker deployment packages all dependencies including Playwright browsers into a single container. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Google's search results without Google's surveillance: Whoogle is a self-hosted proxy that strips the tracking and keeps the results. Your query goes from browser to your Whoogle instance, which fetches results from Google with a randomly generated User Agent and strips everything hostile before returning them: no ads or sponsored content, no third-party JavaScript or cookies, no AMP links, no URL tracking tags like utm_source, no referrer header - and Google sees your server's IP, never yours. Unlike metasearch engines that blend sources, Whoogle proxies Google exclusively, so result quality is exactly what you'd get logged out and incognito, minus the noise. A lightweight Flask app configured entirely through environment variables, it supports DuckDuckGo-style bang shortcuts, autocomplete suggestions, safe search, per-country and per-language filtering, site blocklists, and automatic rewriting of social links to privacy front-ends like Nitter and Invidious. Privacy hardening goes further: built-in Tor routing makes Google see an exit node instead of your server, HTTP/SOCKS proxy support covers other setups, and POST-based queries keep search terms out of logs. Light, dark, and fully custom CSS themes plus browser search-engine registration make it a drop-in default on desktop and mobile. Stateless, tiny, and trivial to run.

Up to 280 search services - Google, Bing, DuckDuckGo, Brave, Qwant, Startpage - aggregated without tracking or profiling: SearXNG is a privacy-respecting metasearch engine (AGPL-3.0, successor to Searx). Your instance queries the upstream engines on your behalf: your IP address, cookies, and search history never reach them, tracker parameters are stripped from result URLs, and an optional image proxy fetches thumbnails server-side so result pages leak nothing. It can even route outbound queries through Tor for full anonymity. Search is organized into categories - general, images, videos, news, maps, music, IT, science, files - with bang shortcuts for targeting specific engines, and every source can be enabled, disabled, or weighted per category in settings.yml. A plugin system adds calculators, hash tools, tracker removal, and unit conversions inline, and preferences (themes, safe search, languages, engine selection) persist in cookies rather than server-side accounts. The real argument for running your own instance rather than trusting a public one is control: you decide the logging policy (none), the engine mix, rate limiting, and who gets access - making it the default search backend for browsers, families, and teams that want Google-quality results without the profile.

Websurfx aggregates results from multiple search engines into one private, ad-free interface that never tracks your queries. Written in Rust, the backend provides memory safety against buffer overflows and data races at the language level while handling concurrent requests across upstream providers like DuckDuckGo and Searx. A built-in re-ranking algorithm scores and merges results by relevance, producing a unified page that draws from many sources while appearing as one coherent search. Safe-search filtering operates across four levels from unrestricted to complete content blocking, with regex-based filter lists for parental controls in schools and organizations. Upstream requests route through configurable proxy chains or Tor, preventing search providers from correlating your queries with a specific IP address. Four Docker image variants target different caching strategies: hybrid Redis plus in-memory for maximum throughput, Redis-only, in-memory-only, or zero-persistence for environments where cached queries must never touch disk. The Lua configuration file controls engine selection, request timeouts, rate limiting thresholds, and production-mode delays that throttle upstream traffic to prevent abuse. Quick-result widgets handle calculations, currency conversions, and unit transformations inline without external redirects. Image search capabilities serve visual discovery for designers and researchers who need media results without surveillance. Runs on a RepoCloud VPS with dedicated resources and full root access. AGPL-3.0 licensed.

With nearly 12,000 GitHub stars and a lineage tracing back to Sphinx Search, Manticore Search is the C++ search database that delivers the full-text, vector, and hybrid search capabilities of Elasticsearch at a fraction of the resource cost — starting in under a second and consuming just 40MB RAM for an empty instance. The SQL-first interface speaks the MySQL wire protocol, meaning mysql client, MySQL Workbench, and any MySQL-compatible driver connects natively without adapters, while the HTTP JSON API provides RESTful access for modern applications. Over 20 full-text operators handle proximity search, quorum matching, field-start and field-end constraints, MAYBE operators, and regex patterns, backed by stemming, lemmatization, stopwords, synonyms, wordforms, and advanced morphology in 70+ languages. Vector search with HNSW indexing enables semantic similarity queries, and hybrid mode combines keyword relevance with vector distance in a single ranked result set using a cost-based query optimizer. Real-time indexing delivers sub-second document availability after insert, sharded tables distribute data across nodes, and Galera-based synchronous replication ensures high availability. Conversational search via CREATE CHAT MODEL and CALL CHAT integrates LLM-backed responses with KNN retrieval and conversation history directly inside the database. Client libraries ship for PHP, Python, JavaScript, TypeScript, Java, Go, Rust, and Elixir. Columnar storage via the Manticore Columnar Library handles analytical workloads on large datasets. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. GPLv3 licensed.

More than 70 automated checks sniff out broken links, redirect chains, and missing meta tags across your whole site, and SEOnaut, the self-hosted SEO audit crawler, sorts every finding into critical, alert, or warning so you know what to fix first. Add a website as a project, press Crawl Now, and watch URLs stream in with their status codes; a Stop button ends any run that wanders off. Reports also cover duplicate titles and descriptions, heading order, hreflang, canonicals, image alt text, thin content, orphan pages, and non-HTTPS URLs, while a passed-checks list shows what is already healthy. Dashboard charts follow crawl history, issue trends, status codes, link types, and page depth. Crawl options are set per project: ignore robots.txt, follow nofollow links, include noindex pages, read the sitemap, allow subdomains, check external links, or authenticate with HTTP Basic to audit a password-protected staging site. Page Details opens any URL with its inlinks, outgoing links, redirections, and images, and URL Explorer searches the crawled pages by term. Export sitemap.xml, CSV lists, or a replayable WACZ archive. Each person picks a light or dark theme and one of five interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

RankMe puts the whole SEO workbench on your own server: audits, rankings, keywords, backlinks, and AI visibility without a per-seat subscription deciding what you may check. Crawl a site and it runs 26 automated checks, returning a plain-language report sorted into Fix now, Watch, and Passed, where each finding names the page, explains in one sentence why it matters, and tells you how to fix it, with an optional AI summary on top. Rank tracking follows Google organic positions over time plus Bing, YouTube, and Amazon, watches SERP features, fires confirmed-drop alerts, and maps local rankings on a geogrid. Keyword research covers volume, difficulty, trends, and SERP-overlap clustering, while Search Console data feeds cannibalization reports and connects to GA4, PageSpeed Insights, and CrUX for performance context. Backlink profiles, link-gap analysis, toxic-link review, and competitor discovery show who links to rivals and which content earns it. A dedicated AI visibility module tracks prompts in AI Overviews and LLM answers and measures share of voice in a weekly digest. Agencies get PDF and CSV exports, scheduled white-label client reports, read-only share links, a REST API, and an MCP server, with alerts delivered by email, Slack, or signed webhook across seven interface languages. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.

OpenSEO provides an alternative to the $100+/month SEO suite monopoly by connecting directly to DataForSEO APIs on a pay-as-you-go basis — you bring your own API key and pay only for the queries you actually make. The platform delivers focused workflows for keyword research with volume, difficulty, CPC, intent, and live SERP inspection; rank tracking with historical position monitoring; backlink analysis with referring domain quality scoring; technical site audits that crawl pages and surface issues; competitor insights comparing SERP positions across keyword sets; and AI visibility monitoring that tracks brand mentions, citations, and prompts across AI models. The integrated MCP server exposes all tools to AI coding agents including Claude Code, OpenClaw, Hermes, and Cursor, while Agent Skills provide reusable SKILL.md workflows that guide agents through complex SEO research tasks. Google Search Console integration reads first-party performance data including clicks, impressions, CTR, and position metrics with URL-level index status inspection. The Prompt Explorer compares how different AI models respond to SEO-relevant queries. Self-host via Docker for personal use or deploy to Cloudflare for team access with automatic database backups. The hosted version at openseo.so adds a 28% markup on DataForSEO requests. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

A question goes in; a cited, long-form report comes out - GPT Researcher is an open-source autonomous research agent. A planner agent decomposes the query into sub-questions, execution agents crawl 20+ web sources in parallel with JavaScript-enabled scraping, and a publisher aggregates findings into a 2,000+ word report with inline citations, exportable to PDF, Word, and Markdown. The Deep Research mode extends this recursively: each result yields follow-up questions that are explored to configurable breadth and depth in a tree pattern, while accumulated learnings, citations, and visited URLs are shared across branches. It also researches local documents (PDF, CSV, Word) alongside the web. LLM and search providers are pluggable, including OpenAI, Anthropic, Google, DeepSeek, and Ollama for models, and Tavily, Google, Bing, DuckDuckGo, and SearXNG for retrieval. It ships as a Python package, a FastAPI server with web frontend, a Docker image, and an MCP server for use inside Claude or Cursor. MIT-licensed.

Live web search plus an LLM of your choice: Farfalle is an open-source, self-hosted answer engine in the Perplexity mold. Queries route through one of several search providers - self-hosted SearXNG for a fully independent stack, or Tavily, Serper, and Bing APIs - and the model composes a cited answer from the retrieved results. Model flexibility is the core design: run llama3, mistral, gemma, or phi3 locally through Ollama for zero per-query cost and full privacy, use cloud models like GPT-4o or Groq-hosted Llama 3 for speed, or route to any provider via LiteLLM. An Expert Search mode uses an agent that plans a multi-step search strategy and executes it for harder questions, and chat history keeps prior research sessions available. The stack is a Next.js and shadcn/ui frontend over a FastAPI backend with Redis rate limiting, shipped as a pre-built Docker image. A browser search-engine entry pointing at your instance makes it the default search from the address bar. Paired with SearXNG and Ollama, the whole pipeline runs with no external API at all.