When an AI agent goes wrong at step seven of twelve, Opik, an open-source LLM observability platform, shows the exact trace, tool call, and prompt behind it, then scores your fix against test datasets. Full trace trees capture every LLM call, retrieval step, and tool execution, and you can annotate traces and spans with feedback scores in the UI or through the SDK. Native integrations for OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, Google ADK, and DSPy, plus OpenTelemetry for any language with an OpenTelemetry SDK, feed traces in, or add the @track decorator to your own functions. Datasets and experiments compare prompt and model versions, scored by LLM-as-a-judge metrics for hallucination, moderation, and RAG relevance, with a PyTest integration to run evaluations in CI. A Prompt Playground tests prompts across models, and the Agent Optimizer SDK helps improve prompts and agents. Production dashboards track feedback scores, trace counts, and token usage, online evaluation rules score live traces, and optional guardrails add safety checks. An MCP server lets Claude Code, Cursor, VS Code Copilot, Codex, or opencode read traces and run evaluations from chat. Run it yourself with the ./opik.sh script on Docker Compose, or a Helm chart on Kubernetes, so trace data stays in your infrastructure. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Agenta delivers a comprehensive open-source LLMOps workspace that covers the full lifecycle of AI application development — from prompt engineering through production monitoring. The platform supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure, AWS Bedrock, and self-hosted models via Ollama, enabling teams to switch between providers without code changes. The prompt playground allows side-by-side comparison of different configurations, while the evaluation system offers LLM-as-a-Judge assessment, 20+ pre-built evaluators covering semantic similarity, regex matching, and factual accuracy, plus custom Python evaluators for domain-specific requirements. Teams run evaluations through both the web UI for subject matter experts and the Evaluation SDK for programmatic CI/CD integration. The observability layer captures full trace visibility across complex agentic workflows, flagging quality issues like hallucinations and off-topic responses in real time. Human annotation workflows let domain experts review and annotate LLM outputs, feeding corrections back into the evaluation loop. The architecture supports Chain of Prompts, RAG pipelines, and multi-step agent workflows, integrating with frameworks like LangChain and LlamaIndex. Self-hosting deploys via Docker Compose with Traefik for routing, requiring only a clone, environment configuration, and a single docker compose command. On RepoCloud, deploy Agenta on a dedicated VPS with root SSH access, persistent storage for evaluation datasets and traces, and complete control over model provider credentials, all under the MIT license with no usage restrictions.

Every call your AI agents make becomes a searchable trace in LangWatch, and the same platform tests and scores those agents before a new prompt or model reaches users. The Trace Explorer filters by origin, user, model, cost, and evaluator, reads each run as a waterfall, flame graph, topology, or sequence diagram, and takes traces from the Python, TypeScript, and Go SDKs or any OpenTelemetry source. Agent Testing sends simulated users, in text or beta voice calls, through your agent while a judge passes or fails every criterion with its reasoning, adds multi-turn red teaming, and compares two agents side by side. The Experiments workbench scores prompts and agents on datasets with LLM judges, exact match, and RAG checks, then compares runs on cost, latency, and pass rate, while online evaluations monitor live traffic and guardrails block unsafe replies. Prompts keep version history and staging or production tags, Workflows wire prompts, Python code, and HTTP calls on a visual canvas, and annotation queues route traces to human reviewers. Alerts fire when a graph crosses a threshold, automations post to Slack, email, or webhooks, and scheduled reports go out by email or Slack. Claude Code, Codex, Copilot, Gemini CLI, and opencode report session cost, pull request cost, and terminal replays, and an MCP server exposes trace search and prompt management to coding assistants. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

When an AI agent breaks in production, Tracely freezes that exact run into a replayable regression test and blocks the pull request that would ship the bug again. Span soup hides the failure. OTLP/HTTP traces group into conversation threads, and a waterfall marks the failing span in red with its input and output. Nobody knows if an answer was good. Evaluators sit as columns on the trace table, mixing model-free checks with LLM judges at conversation, run, or span level. Thirty-one broken runs, one issue. Clustering turns them into a single counted issue and can suggest the evaluator that would have caught it. A fix nobody can prove. One click promotes a failing trace into a hermetic case with recorded fixtures and a fail-to-pass contract, and scenarios send a red-team model after an adversarial goal. Model bills on every CI run. The suite replays offline with no API keys and no model spend, and tracely gate posts a commit status, comments on the pull request, and blocks regressions. No warning when it breaks again. Alert flows drawn on a canvas react to failed gates, broken conversations, unseen failure modes, or rising failure rates by messaging Slack, sending email, calling a webhook, or prompting an LLM. Coding agents can read traces and write evaluators through an MCP endpoint. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

A looping agent burns money quietly until someone can see where it drifted, and PandaProbe is an observability and evaluation platform for AI agents that makes that drift visible step by step. Capture complete executions as sessions, traces, and spans spanning model calls, tool calls, multi-agent handoffs, and custom logic, then follow a looping or failing run back to the decision where it drifted. Score traces and sessions with agent-specific metrics, run asynchronous and sampled evaluations so judge models do not cost more than inference, and use lightweight trajectory signals to catch stalls, bad tool use, and regressions over time. Bring agents built with LangGraph, CrewAI, the Claude Agent SDK, the OpenAI Agents SDK, and other integrations, with MCP tool calls captured automatically where supported. Organize work by user, organization, and project, send data through an SDK or HTTP API with project keys, and query filtered, paginated traces when you need the raw record. Keep sensitive workflows local when that matters: prompts, traces, and evaluation results stay inside the deployment you run, instead of leaving for a hosted console. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Production AI agents fail in quiet ways, and Latitude records every trace, session, and tool call so each failure can be found, grouped, and fixed. One line of telemetry from the TypeScript or Python SDK, or any OpenTelemetry exporter, sends traces from OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, and dozens of other providers and frameworks. Failing traces are clustered into signals with status, size, and trend, and flaggers and annotation queues let reviewers score outputs by hand. Evaluations are generated from those judgments and tracked with an alignment score. Semantic and exact-text search covers every trace, and saved searches, filters, and behavior groupings keep recurring questions one click away. Dashboards report cost per session, tool failure rates, and agent score across outcome, reliability, speed, and safety. Memory views show how an agent's stored memory changes over time, with diffs and the trace behind each edit. Monitors raise alerts, regression datasets replay fixes against the traces that failed, and Agent Dispatch hands the evidence to a coding agent. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Backed by Y Combinator and trusted by over 2,300 companies processing billions of observations monthly, Langfuse is the most widely adopted open-source platform for building, monitoring, evaluating, and debugging LLM applications. The hierarchical tracing engine captures every LLM call, tool invocation, retrieval step, and agent action as nested spans based on OpenTelemetry, with automatic cost calculation, latency tracking, and token usage attribution across sessions and users. Prompt Management separates prompts from code with versioned artifacts, label-based deployments, one-click rollbacks, and runtime SDK fetching with server-side caching, while linking every generation back to its exact prompt version for attribution analytics. The evaluation system supports LLM-as-a-judge scoring, heuristic code evaluators, user feedback collection, and manual annotation workflows that run automatically on production traces or against curated datasets. The Playground enables interactive prompt testing on real production inputs with side-by-side model comparison across providers. Datasets and Experiments define test cases for systematic benchmarking with comparative result visualization. Native SDKs for Python and TypeScript provide decorator-based instrumentation, while 100+ integrations cover LangChain, LlamaIndex, OpenAI SDK, LiteLLM, Vercel AI SDK, and any OpenTelemetry-instrumented framework. The analytics dashboard surfaces cost breakdowns, quality scores, latency percentiles, and usage trends across models and prompt versions. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

An agent run can finish without errors and still give the wrong answer, so TraceRoot traces every model and tool call, flags the production runs that missed the mark, and helps your coding agent ship verified fixes. Each run becomes a browsable call tree with inputs, outputs, latency, and cost, searchable by session or filter. Instrument with the Python or TypeScript SDK, the agent-native CLI, or any OpenTelemetry exporter, with ready integrations for LangChain, LangGraph, OpenAI Agents SDK, CrewAI, Vercel AI SDK, Mastra, and Pydantic AI. Detectors use LLM judges to score live traffic against the behavior you define and can post alerts to Slack. Judge outputs cluster into signals, then a background coding agent reads the traces and your connected GitHub code in a sandbox and recommends changes down to the commit, file, and line. Datasets, test cases, and scored eval runs compare versions so regressions surface before a release, and dashboards of widgets chart your trace data. The CLI hands your coding agent, such as Claude Code or Codex, the traces, signals, detectors, and evals to work from. Self-hosting keeps all of that trace data on infrastructure you control. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed (core edition).

Backed by Y Combinator (S24) and processing traces from thousands of AI agents in production, Laminar is the open-source observability platform that treats agent debugging as a first-class engineering discipline rather than an afterthought. Its OpenTelemetry-native SDK auto-instruments Vercel AI SDK, LangChain, OpenAI, Anthropic, Gemini, Browser Use, Stagehand, Mastra, Pydantic AI, and the OpenAI Agents SDK with a single line of code, capturing every LLM turn, tool call, and sub-agent delegation as nested spans with full input/output data and token costs. The Signals engine lets you describe failures in plain language — "agent is stuck in a loop" or "tool returned empty results" — then reads every trace and alerts via Slack when it detects a match. A built-in debugger records runs and replays them from cache so each iteration takes seconds, designed for Claude Code, Cursor, or Codex to drive the repair loop via the MCP server or CLI. Run code-first evaluations in Python or TypeScript locally or in CI/CD pipelines, build datasets from production traces, and query everything with raw SQL through custom dashboards, the in-app editor, or your coding agent. The Rust backend delivers 20x trace compression, a custom real-time streaming engine, ultra-fast full-text search, and gRPC ingestion, while ClickHouse powers columnar analytics and PostgreSQL stores application state. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Stop guessing which model writes better and watch them compete on your own prompts: LMRing runs two to five language models side by side on the same input, streams every answer in real time, and lets you vote for the winner. Each comparison feeds a leaderboard you can read as a ranked table, a bar chart, or a scatter plot, so patterns in cost, speed, and quality stop hiding inside individual chats. The arena goes beyond text, with dedicated modes for comparing generated images, generated video across providers such as Sora and Veo, and live-rendered web development output where model code runs in front of you. More than fifty providers connect through their APIs, including OpenAI, Anthropic, Google, DeepSeek, Mistral, Groq, OpenRouter, and any OpenAI-compatible endpoint you point it at. Your comparison history is saved, shareable, and revisitable, which turns one-off impressions into an evaluation record your team can argue from. Sign-in flexes to your setup with GitHub, Google, or Linux.do OAuth, or email one-time codes, and the interface speaks English, French, and Chinese. Because the arena is yours, the prompts, votes, and provider keys stay on infrastructure you control instead of feeding someone else's public ranking. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

To pull a team's skills, MCP servers, hooks, prompts, and agent files out of scattered Git repositories, Observal keeps them in one reviewed registry and traces real sessions to show which ones help. Build an agent visually, with a live config preview for every harness. Submissions wait in an admin review queue, where reviewers inspect side-by-side version diffs before approving a release. Teamspaces give each team its own namespace, and approved releases can be forked into independent drafts. One pull command then writes the right config for Claude Code, Cursor, Kiro, Codex, Copilot, Pi, OpenCode, Antigravity, or Goose. Every approved resource is discoverable through the open Agentic Resource Discovery standard, and a running agent can hand part of its task to another approved agent or a remote Agent2Agent service. Hooks record each session as a trace of prompts, tool calls, MCP round trips, tokens, and errors, with turn-by-turn replay for debugging and audits. Insight reports, generated through providers such as Anthropic, OpenAI, or Ollama, list what is working, what is hindering, and quick wins. A dashboard follows adoption, cost, and department usage, a leaderboard ranks agents by downloads, and audit logs, SAML single sign-on, and SCIM provisioning are included. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Put a checkpoint in front of every tool call your AI agents make, so a poisoned web page or file cannot quietly move money or send private data. Each call is classified and policy-checked before it executes, then allowed, blocked, or held for a person. Held calls land in an approvals queue that shows what the user asked, which passage the agent read, and the exact recipient or amount in question, with one-click block or allow. Instruction provenance flags destinations that appeared only in untrusted content, like an IBAN hidden in a file. The cockpit lists every run as a timeline of steps with verdicts, groups violations by the protection that caught them, and scores ten attack surfaces on a Security 360 page. An agent registry tracks owners, scope, and call counts. Budgets, anomaly alerts, and a kill switch cap runaway behavior. Sensitive values are redacted from traces, signed receipts prove a call was allowed, and exports produce PDF and CSV audit evidence. Python and TypeScript SDKs plug in with one line, or an LLM proxy catches calls with an environment variable. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Prismor sits between your AI coding agents and the tools they want to use, judging every shell command, file read, and API call before it runs so a helpful agent cannot quietly become a destructive one. Run it in observe mode to watch what agents actually do across every workspace, then switch chosen rules to enforce and block secret leaks, prompt-injection pivots, privilege escalation, and risky package installs before execution. When a call is sensitive rather than forbidden, Prismor can pause it for human approval, letting a person decide instead of the model. Plain-language prompt guardrails such as never push to main are added to the agent context per agent and per session, and edits are picked up on the next prompt. Every action lands in a hash-chained, Ed25519-signed audit trail you can verify later, while the web dashboard breaks down sessions, findings, friction, and token spend so you can see where runs went wrong. Agents that cannot be hooked are still governed through the LLM proxy or the MCP gateway, which evaluates each tool call and scans responses for injected instructions before the model sees them. Honeytoken canary files trip a critical finding the moment an agent reads them, catching reconnaissance early. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Your AI coding agent has your API keys in its environment and unrestricted network access, which means one prompt injection away from sending those secrets anywhere. Pipelock closes that gap by sitting as a proxy between your agents and every outbound connection, scanning the actual content of HTTP, WebSocket, MCP, and Agent-to-Agent traffic before it leaves your server. An 11-layer scanner pipeline checks every request against 62 credential patterns covering AWS, GCP, Azure, GitHub, OpenAI, Anthropic, SSH keys, and database URLs, then inspects every response for prompt injection using 29 detection patterns with six-pass normalization that catches base64-encoded, leetspeak, and whitespace-obfuscated payloads. The MCP proxy wraps any Model Context Protocol server (stdio, HTTP, or WebSocket) with bidirectional scanning that detects tool description poisoning and mid-session rug-pull changes via SHA-256 fingerprinting. Every scanning decision produces a cryptographically signed action receipt that third parties can verify offline without trusting the agent or the vendor. The Operator Console provides a web dashboard for reviewing evidence scorecards, receipt timelines, agent sessions, enforcement decisions, and fleet posture at a glance. Cross-request taint tracking catches slow-drip exfiltration attempts that spread a secret across multiple calls. Canary tokens plant synthetic secrets that trip alerts the moment an agent tries to exfiltrate them. Pre-built Prometheus metrics and a Grafana dashboard provide real-time visibility into traffic volumes and block rates. Deploy on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Before a language model learns from your data, someone has to label it well; Argilla gives AI engineers and domain experts a shared workspace to label, review, and refine the datasets that fine-tune language models and train NLP pipelines. You build each dataset from the Python SDK, mixing text, chat, and image fields with the exact questions your task needs: single-label classification for intent detection, span questions for named entities, ratings for response quality, rankings for preference tuning, or free text for corrections. Annotators then work through records in a focused labeling interface where model predictions already sit attached as suggestions, turning most reviews into a fast confirm-or-correct decision instead of a blank-page exercise. Full-text search, metadata filters, and status filters slice a hundred thousand records down to the ones needing attention, while vector-based semantic search finds lookalike examples, near-duplicates, and the disagreement hotspots where guidelines need sharpening. Dataset owners watch completion progress, manage who has responded to what, and split work across workspaces with owner, admin, and annotator roles for each team or client. Finished datasets export with responses, suggestions, and metadata intact, locally or straight to the Hugging Face Hub, ready for fine-tuning and evaluation runs. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Images, text, audio, video, HTML, PDFs, and time series, labeled in one tool with a standardized output format: Label Studio is the open-source data labeling platform for building training datasets. Computer vision tasks cover classification, object detection (boxes, polygons, ellipses, keypoints), and semantic segmentation; audio work spans transcription, speaker diarization, and emotion recognition; NLP handles named entity recognition and document classification with taxonomies up to 10,000 classes; and GenAI workflows support LLM fine-tuning data and RLHF response ranking. Labeling interfaces are fully configurable with an XML-like templating language, so the UI matches the task instead of the reverse. The ML backend SDK turns any model into a connected web server for pre-annotation (model predicts, humans verify), interactive labeling (real-time predictions as annotators draw regions or highlight text), and model evaluation - cutting annotation time dramatically on large datasets. Data imports from S3, GCS, or file uploads; the Data Manager filters and explores tasks; exports convert to the format your ML library expects via label-studio-converter. Multi-user accounts tie every annotation to its author, and webhooks, a Python SDK, and REST API embed labeling into any pipeline. Self-hosting keeps proprietary training data - often a company's most sensitive asset - entirely on your infrastructure.