Categories
Self-Hosted API Knowledge Management Web Scraping MCP AI Research Competitive IntelligenceStars
Forks
Watchers
Developer links
SurfSense
Positioned as the open-source NotebookLM alternative for AI agents, SurfSense delivers a live web research platform where your agents access structured data from Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, Indeed, and any page on the open web through one REST API or MCP server. Scheduled and event-triggered agents transform findings into briefs, alerts, podcasts, and presentations, while a built-in knowledge base keeps every discovery searchable with Perplexity-style cited answers using hybrid semantic and full-text search powered by PostgreSQL with pgvector. Upload PDFs, Office documents, images, and audio files, or sync Google Drive, OneDrive, and Dropbox — 50+ file formats supported with AI file sorting that auto-organizes documents by source, date, and topic. The MCP server exposes scrapers, knowledge base, and workspaces as native tools for Claude, Cursor, and any MCP-compatible agent. Cross-country proxy rotation handles Reddit, TikTok, and Google Search scraping with geo-aware sticky sessions and captcha-aware anti-bot handling. The platform features collaborative chats, multi-format document export, git-native knowledge base with Open Knowledge Format export, and a desktop quick-ask panel with global shortcut. Docker Compose deployment manages nine services including Caddy proxy, PostgreSQL, Redis, FastAPI backend, Celery workers, zero-cache real-time sync, and Next.js frontend with automatic Watchtower updates. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.
Benefits
- Live Web Data for Agents
- Structured connectors for Reddit, YouTube, Instagram, TikTok, Amazon, Walmart, Google Maps, Google Search, and Indeed deliver fresh data through one API with anti-bot handling.
- MCP Server and REST API
- Expose scrapers, knowledge base, and workspaces as native tools for Claude, Cursor, and any MCP-compatible agent with typed schemas and full CRUD access to research data.
- Hybrid Search with Citations
- PostgreSQL with pgvector powers both semantic vector search and full-text keyword search delivering Perplexity-style cited answers grounded in your indexed sources and documents.
- Scheduled Agent Automations
- Event-triggered and cron-scheduled agents monitor topics, track competitors, and generate briefs, alerts, podcasts, and presentations without manual intervention or oversight.
Features
- Knowledge Base Management
- Git-native knowledge store with Open Knowledge Format export, AI file sorting, 50+ format support, and Google Drive, OneDrive, and Dropbox sync.
- Multi Source Scraping
- Dedicated AI subagents handle each platform with cross-country proxy rotation, geo-aware sticky sessions, and captcha-aware anti-bot handling for reliable data extraction.
- Real Time Sync Engine
- Rocicorp Zero replicates PostgreSQL changes to all connected browsers via logical replication for live notifications, indexing progress, and collaborative features.
- Docker Compose Deployment
- Nine-service architecture with Caddy proxy, PostgreSQL, Redis, FastAPI, Celery workers, scheduler, zero-cache, and Next.js managed by one-line install scripts.
- Podcast and Report Generation
- Transform research findings into audio podcasts, slide presentations, and structured reports with citations using configurable LLM providers and templates.