CRW
CRW gives your apps and AI agents a private web data API that scrapes pages, crawls sites, maps links, and searches the web without sending every request to a hosted scraping service. Post a URL to the scrape endpoint and receive clean Markdown, HTML, screenshots, links, or structured JSON shaped by a schema you supply. Start a crawl when a whole section matters, then poll for progress while CRW walks pages asynchronously and returns the collected content in one job. Use map first to discover the URLs on a site before spending effort extracting pages you do not need. Connect agent tools through the MCP endpoint so assistants can search, scrape, and crawl from the same self-hosted service your applications call. Add browser rendering when JavaScript-heavy pages hide content from a plain HTTP fetch, and keep simpler pages on the fast lightweight path. Protect a public deployment with Bearer token authentication, your own reverse proxy, logging, and rate limits, or keep the service inside a private network for internal tools. Run one Docker container for the core API, or use the Compose stack when you want bundled search and rendering sidecars beside it. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. AGPL-3.0 licensed.
Deploy