Deploy Now

Stars

109

Forks

93

Watchers

0

Developer links

FastLLM Proxy

Local vLLM and llama.cpp boxes and hosted providers like OpenRouter, Anthropic, Bedrock, and Gemini all answer to one OpenAI-compatible address once FastLLM Proxy sits in front of them. Clients ask for a frontend model by name, and ordered rules pick the target by caller, prompt size, headers, budget spent, time of day, or how busy a backend is. Prompts that share a prefix return to the node holding their cache, and semantic routing sends coding questions one way and chat another using example prompts you write. API keys carry per-model grants, rate limits, and token or spend budgets, and the management console shows traffic, usage and spend, an append-only audit log, and a fleet topology. Failures fall through to the next target before a byte reaches the caller, and an admission gate turns a flooded engine into a bounded queue. An MCP gateway and an A2A agent endpoint put tool servers and agents behind the same address. Existing LiteLLM configs import unchanged. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

FastLLM Proxy
FastLLM Proxy
FastLLM Proxy
FastLLM Proxy
FastLLM Proxy
FastLLM Proxy

Benefits

  • One Endpoint For Every Backend
  • Point every client at a single OpenAI-compatible address in front of vLLM, SGLang, llama.cpp, and 81 providers, and swap models without changing client configuration.
  • Routing That Follows Your Rules
  • Frontend models resolve to ordered targets by caller, prompt size, headers, budget used, time window, or backend load, with failover on 5xx and 429 before the client sees anything.
  • Keys With Budgets Attached
  • Each API key carries per-model grants, rate limits, and token or spend budgets, so a runaway script hits its cap instead of the whole month's invoice.
  • Drop-In For LiteLLM Setups
  • An importer reads existing LiteLLM config files, bringing models, backends, keys, and per-key grants across, and re-running it converges instead of duplicating records.

Features

  • Cache-Affinity Routing
  • Requests sharing a prompt prefix return to the vLLM or SGLang node holding its KV cache, unless that node runs hotter than the least-loaded one.
  • Semantic Prompt Classes
  • Define a class with example prompts and the proxy embeds each request, routing coding, maths, or legal questions to the model you chose, with an evaluation for collisions.
  • Usage And Spend Console
  • A web console charts traffic, latency, tokens, and spend by principal, model, or day, with an audit log of every admin change.
  • Admission Gate
  • A per-backend gate turns a flooded self-hosted engine's endless queue into a bounded one, answering overflow with 503 and a Retry-After header.
  • MCP And A2A Gateway
  • Register tool servers and agents once, then reach them through the same address with namespaced tools, authorization, and pinned agent versions.
  • Fleet Topology View
  • See the control plane, proxy replicas, and engine hosts with per-node health, in-flight counts, and cache figures on one page.

Apps Similar to FastLLM Proxy