Logo
Deploy Now

Categories

Developer Tools Observability AI/ML LLMOps AI Evaluation Prompt Engineering

Stars

4,513

Forks

636

Watchers

30

Developer links

Last Updated: 08-22-2026 07:34 PM (UTC)

Agenta

Agenta delivers a comprehensive open-source LLMOps workspace that covers the full lifecycle of AI application development — from prompt engineering through production monitoring. The platform supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure, AWS Bedrock, and self-hosted models via Ollama, enabling teams to switch between providers without code changes. The prompt playground allows side-by-side comparison of different configurations, while the evaluation system offers LLM-as-a-Judge assessment, 20+ pre-built evaluators covering semantic similarity, regex matching, and factual accuracy, plus custom Python evaluators for domain-specific requirements. Teams run evaluations through both the web UI for subject matter experts and the Evaluation SDK for programmatic CI/CD integration. The observability layer captures full trace visibility across complex agentic workflows, flagging quality issues like hallucinations and off-topic responses in real time. Human annotation workflows let domain experts review and annotate LLM outputs, feeding corrections back into the evaluation loop. The architecture supports Chain of Prompts, RAG pipelines, and multi-step agent workflows, integrating with frameworks like LangChain and LlamaIndex. Self-hosting deploys via Docker Compose with Traefik for routing, requiring only a clone, environment configuration, and a single docker compose command. On RepoCloud, deploy Agenta on a dedicated VPS with root SSH access, persistent storage for evaluation datasets and traces, and complete control over model provider credentials, all under the MIT license with no usage restrictions.

Agenta
Agenta
Agenta
Agenta
Agenta

Benefits

  • Complete LLMOps Development Lifecycle
  • Covers prompt management, playground experimentation, automated evaluation, human annotation, A/B testing, and production observability in a single unified self-hosted platform.
  • Provider-Agnostic Model Support
  • Supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Azure, AWS Bedrock, and self-hosted Ollama, with configuration-level provider switching.
  • Systematic Evaluation Framework
  • 20+ pre-built evaluators including LLM-as-a-Judge, semantic similarity, and regex matching, plus custom Python evaluators and SDK for CI/CD pipeline integration.
  • Human-in-the-Loop Annotation Workflows
  • Domain experts review, annotate, and score LLM outputs through the web UI, feeding structured feedback into the evaluation loop for continuous quality improvement.

Features

  • Prompt Playground
  • Side-by-side prompt comparison interface with support for loading production traces, test sets, and real-time model output streaming across providers.
  • Evaluation System
  • Run automated evaluations using LLM-as-a-Judge, 20+ built-in evaluators, or custom Python evaluators via the web UI or programmatic SDK.
  • Observability Dashboard
  • Full trace visibility across agentic workflows with latency, cost, and quality metrics, supporting LangChain, LlamaIndex, and custom framework integrations.
  • Docker Compose Deployment
  • Self-host with Docker Compose and Traefik routing, supporting environment customization for ports, authentication, and remote deployment configurations.
  • Evaluation SDK
  • Python SDK for running programmatic evaluations in CI/CD pipelines with full test data control and results synced to the Agenta dashboard.