Stars
Forks
Watchers
Developer links
Searcharvester
Searcharvester gives your AI agents a private search and research backend that runs entirely on your server, with a web UI where you watch every research job unfold. The search endpoint speaks the Tavily schema, so existing agent code points at your host and queries more than a hundred SearXNG engines without API keys or quotas. Send a URL to the extract endpoint and it returns clean markdown with navigation, footers, and ads stripped, sized in presets from five thousand characters to fully paginated long documents. Three extractors compete on each page and a quality gate keeps the best result, so hard pages still yield readable text. For bigger questions, the research endpoint runs a team of agents: a lead plans sub-questions, researchers gather in parallel, a critic tries to disprove the findings, and a fact-checker re-verifies numbers and dates before the final report arrives with clickable citations. You assign a different model and reasoning level to each role, follow agent branches, the reading graph, sources, and media live, then read the finished report in the same UI. Search settings let you toggle engines per category, test proxies, and apply changes with a targeted restart. A loop guard enforces budgets for calls, tokens, searches, and page reads across the whole job. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.
Benefits
- Search without keys or quotas
- SearXNG aggregates more than a hundred engines behind a Tavily-compatible endpoint, so agents search freely with no provider accounts, billing meters, or rate ceilings to manage.
- Pages return as clean markdown
- Trafilatura, Readability, and Defuddle compete per page under a quality gate, and size presets plus pagination deliver exactly the amount of text your context window needs.
- Deep research with receipts
- Parallel researchers, a critic, and a fact-checker produce a cited markdown report, and every source, picture, and agent step stays visible for inspection afterwards.
- Budgets that stop runaway jobs
- A loop guard watches calls, tokens, searches, and reads across all agents, warning early and cancelling stalls or repeated queries before they burn your model budget.
Features
- Tavily-compatible search
- The search endpoint follows the Tavily request and response schema across web, image, and video categories, with per-category engine selection.
- Competitive extraction
- Auto mode runs trafilatura, Readability, and Defuddle, keeps the best-rated text, caches by URL, and paginates full documents at 25000 characters.
- Multi-agent research
- Quick mode uses one agent for factual questions; deep mode plans, delegates, critiques, and fact-checks before writing a cited report.
- Model per role
- Lead, researcher, critic, and fact-checker each take their own model and reasoning effort, validated against the gateway model list.
- Live research UI
- Branches, agent graph, sources, media, and per-agent chats stream events live, and finished jobs survive restarts on disk.
- Search settings page
- Toggle SearXNG engines by category, test rotating proxies, set timeouts, and apply changes with a short targeted restart.
