Logo
Deploy Now

Categories

Developer Tools Machine Learning Cloud Native Kubernetes API Gateway AI Gateway LLM Proxy

Stars

232

Forks

72

Watchers

1

Developer links

Last Updated: 08-22-2026 10:48 AM (UTC)

Inference Gateway

Inference Gateway puts a single OpenAI-compatible API endpoint in front of OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Google, Mistral, MiniMax, Moonshot, Nvidia, and llama.cpp, so your application code never changes when you switch models or providers. The Go binary starts on port 8080 and normalizes authentication, streaming protocols, and response formats across all backends transparently. Native Model Context Protocol support auto-discovers tools from connected MCP servers and injects them into LLM requests without client-side management, enabling server-side tool execution across any provider that supports function calling. Agent-to-Agent protocol integration allows distributed agent communication through a declarative Agent Definition Language that generates production-ready Go or Rust servers from a single YAML manifest. The dedicated Kubernetes Operator manages Gateway, Agent, MCP, and Orchestrator custom resources with automatic HPA scaling, OIDC authentication, and service discovery that rebuilds MCP configurations when the discovered server set changes. Prometheus metrics and OpenTelemetry tracing provide full request-level observability across the entire inference pipeline. Middleware controls enable per-request provider selection, model routing, and fallback strategies. Official SDKs in Go, Python, TypeScript, and Rust provide typed client interfaces with streaming support. Docker Compose deployment requires only environment variables for API keys. A CNCF Sandbox applicant. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Inference Gateway
Inference Gateway

Benefits

  • Universal LLM Provider Access
  • Single OpenAI-compatible API endpoint routes requests to 14+ providers including OpenAI, Anthropic, Groq, Ollama, and DeepSeek with normalized authentication and response formats.
  • Native MCP Tool Integration
  • Auto-discovers tools from connected Model Context Protocol servers and injects them into LLM requests transparently, enabling server-side tool execution without client configuration.
  • Kubernetes Native Operator Management
  • Dedicated Operator manages Gateway, Agent, and MCP custom resources with declarative CRDs, automatic horizontal pod autoscaling, OIDC auth, and GitOps-friendly configuration.
  • Full Observability Pipeline
  • Built-in Prometheus metrics and OpenTelemetry distributed tracing provide request-level visibility across the entire inference chain from client through gateway to provider.

Features

  • Multi-Provider Proxy
  • Routes requests to OpenAI, Anthropic, Groq, Cohere, Ollama, DeepSeek, Google, Mistral, Nvidia, and llama.cpp through a single endpoint with unified streaming support.
  • MCP Server Integration
  • Connects to multiple MCP servers simultaneously with automatic tool discovery, dynamic injection into LLM requests, and configurable include/exclude tool filtering.
  • Agent Definition Language
  • Declarative YAML manifests define A2A agents with provider, model, tools, and deployment specifications that generate enterprise-ready Go or Rust server projects.
  • Kubernetes Operator
  • CRDs for Gateway, Agent, MCP, and Orchestrator resources with HPA autoscaling, OIDC authentication, TLS termination, and automatic MCP service discovery.
  • Multi-Language SDKs
  • Official client SDKs in Go, Python, TypeScript, and Rust provide typed interfaces for gateway interaction with streaming support and tool-use capabilities.