Categories
Machine Learning MLOps Natural Language Processing AI Infrastructure LLM Serving Model Deployment API ServerStars
Forks
Watchers
Developer links
OpenLLM
OpenLLM serves any large language model as an OpenAI-compatible API endpoint from a single CLI command, handling model download, backend selection, quantization, and port binding automatically. It supports the full spectrum of popular models including Llama 3.3, Qwen2.5, DeepSeek, Mistral, and Phi3, choosing between vLLM and PyTorch inference backends based on hardware capabilities. When vLLM is available, continuous batching with PagedAttention achieves up to 23x throughput improvement over naive serving, while GPTQ and bitsandbytes quantization reduces memory requirements for GPU-constrained deployments. The server exposes a RESTful API on port 3000 with full OpenAI client library compatibility, enabling drop-in replacement for commercial providers in any application using the standard chat completions format. A built-in web chat UI at the /chat endpoint provides immediate interactive testing without external clients. Custom model repositories allow teams to maintain private catalogs of fine-tuned models alongside the default repository that tracks the latest releases. Deployment workflows generate production-ready Docker images automatically, with Kubernetes manifest support for orchestrated scaling. Native integration with LangChain and LlamaIndex supports RAG pipelines, Transformers Agents enables tool-calling workflows, and HuggingFace Hub handles model discovery. Server-Sent Events enable real-time token streaming across all API endpoints. Backed by BentoML's production ML infrastructure. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.
Benefits
- Single-Command Model Deployment
- Launch any supported open-source LLM as a production API server with one CLI command, automatically handling model download, backend selection, quantization configuration, and port binding without manual setup.
- vLLM Continuous Batching Engine
- Achieves up to 23x throughput improvement over sequential inference using PagedAttention memory management and continuous batching, maximizing GPU utilization across concurrent requests on CUDA-capable hardware.
- OpenAI API Drop-In Compatibility
- Exposes chat completions, embeddings, and model listing endpoints matching the OpenAI API specification, enabling seamless migration of existing applications from commercial providers without code changes.
- Custom Model Repository System
- Maintain private catalogs of fine-tuned or proprietary models alongside the default open-source repository, with version tracking and automatic dependency resolution for team-specific model management workflows.
Features
- Built-In Chat UI
- Web-based chat interface served at /chat endpoint on port 3000 for interactive model testing, conversation management, and real-time streaming response preview.
- Dual Inference Backends
- Automatically selects between vLLM for CUDA-enabled GPU serving and PyTorch for CPU or unsupported architectures, with GPTQ and bitsandbytes quantization support.
- Docker Image Generation
- Automatically generates production-ready Docker container images for any configured model, with Kubernetes deployment manifests and BentoCloud serverless integration.
- LangChain Integration
- Native LangChain and LlamaIndex connectors for building RAG pipelines, multi-model chains, and Transformers Agents tool-calling workflows with local model serving.
- gRPC and REST APIs
- Dual-protocol API server supporting RESTful HTTP endpoints and gRPC for high-performance inter-service communication with Server-Sent Events streaming on all routes.