Agenta delivers a comprehensive open-source LLMOps workspace that covers the full lifecycle of AI application development — from prompt engineering through production monitoring. The platform supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure, AWS Bedrock, and self-hosted models via Ollama, enabling teams to switch between providers without code changes. The prompt playground allows side-by-side comparison of different configurations, while the evaluation system offers LLM-as-a-Judge assessment, 20+ pre-built evaluators covering semantic similarity, regex matching, and factual accuracy, plus custom Python evaluators for domain-specific requirements. Teams run evaluations through both the web UI for subject matter experts and the Evaluation SDK for programmatic CI/CD integration. The observability layer captures full trace visibility across complex agentic workflows, flagging quality issues like hallucinations and off-topic responses in real time. Human annotation workflows let domain experts review and annotate LLM outputs, feeding corrections back into the evaluation loop. The architecture supports Chain of Prompts, RAG pipelines, and multi-step agent workflows, integrating with frameworks like LangChain and LlamaIndex. Self-hosting deploys via Docker Compose with Traefik for routing, requiring only a clone, environment configuration, and a single docker compose command. On RepoCloud, deploy Agenta on a dedicated VPS with root SSH access, persistent storage for evaluation datasets and traces, and complete control over model provider credentials, all under the MIT license with no usage restrictions.

Comparing prompts and models by eye stops working fast, so ChainForge lays every experiment out on one visual canvas. Drop a table of inputs into a Prompt Node, add a handful of models, and the app sends every combination of template variable and model, then collects the answers in an Inspect Node that groups responses by any variable. Evaluator nodes score outputs with a few lines of code, and visualization nodes plot pass rates and numeric scores as histograms or box plots. Chat Turn nodes test follow-up questions, to see if a model caves when a user pushes back. Sparkle buttons draft synthetic table rows, test questions, and starter evaluator code with a model of your choice, and an LLM scorer can judge answers against a rubric you write. Example Flows open ready-made experiments, from a no-key starter with tiny in-browser models to prompt template comparisons, sycophancy checks, gender bias audits, red-teaming runs, and 188 flows converted from OpenAI Evals. Documents, Chunking, Retrieval, and Rerank nodes cover retrieval experiments. Finished flows export as cforge files, and results export to spreadsheets or JSON for the rest of the team. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.