Promptfoo replaces vibes-based prompt tweaking with measurable tests, then attacks your AI app the way a hostile user would. An evaluation matrix lines up every prompt and model side by side, with pass or fail badges, assertion messages, latency, token counts, and cost in each cell. A red team wizard walks through the target, application details, plugins, and strategies, then probes for prompt injection, data exposure, broken access control, and harmful content. Preset bundles map plugins to the OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, GDPR, ISO 42001, and the EU AI Act. Meta Agent and Hydra Multi-Turn strategies adapt attacks across a whole conversation instead of firing one-shot prompts. Providers span OpenAI, Anthropic, Gemini, Bedrock, Azure, and local Ollama models, and any HTTP endpoint, Python script, or JavaScript function can be tested as a target. The history page lists every run with pass rates and raw scores, and prompt and dataset pages show which templates and test sets each run used. A provider allowlist file narrows the models offered in the web editor for cost control. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Agenta delivers a comprehensive open-source LLMOps workspace that covers the full lifecycle of AI application development — from prompt engineering through production monitoring. The platform supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure, AWS Bedrock, and self-hosted models via Ollama, enabling teams to switch between providers without code changes. The prompt playground allows side-by-side comparison of different configurations, while the evaluation system offers LLM-as-a-Judge assessment, 20+ pre-built evaluators covering semantic similarity, regex matching, and factual accuracy, plus custom Python evaluators for domain-specific requirements. Teams run evaluations through both the web UI for subject matter experts and the Evaluation SDK for programmatic CI/CD integration. The observability layer captures full trace visibility across complex agentic workflows, flagging quality issues like hallucinations and off-topic responses in real time. Human annotation workflows let domain experts review and annotate LLM outputs, feeding corrections back into the evaluation loop. The architecture supports Chain of Prompts, RAG pipelines, and multi-step agent workflows, integrating with frameworks like LangChain and LlamaIndex. Self-hosting deploys via Docker Compose with Traefik for routing, requiring only a clone, environment configuration, and a single docker compose command. On RepoCloud, deploy Agenta on a dedicated VPS with root SSH access, persistent storage for evaluation datasets and traces, and complete control over model provider credentials, all under the MIT license with no usage restrictions.

Backed by Y Combinator and trusted by over 2,300 companies processing billions of observations monthly, Langfuse is the most widely adopted open-source platform for building, monitoring, evaluating, and debugging LLM applications. The hierarchical tracing engine captures every LLM call, tool invocation, retrieval step, and agent action as nested spans based on OpenTelemetry, with automatic cost calculation, latency tracking, and token usage attribution across sessions and users. Prompt Management separates prompts from code with versioned artifacts, label-based deployments, one-click rollbacks, and runtime SDK fetching with server-side caching, while linking every generation back to its exact prompt version for attribution analytics. The evaluation system supports LLM-as-a-judge scoring, heuristic code evaluators, user feedback collection, and manual annotation workflows that run automatically on production traces or against curated datasets. The Playground enables interactive prompt testing on real production inputs with side-by-side model comparison across providers. Datasets and Experiments define test cases for systematic benchmarking with comparative result visualization. Native SDKs for Python and TypeScript provide decorator-based instrumentation, while 100+ integrations cover LangChain, LlamaIndex, OpenAI SDK, LiteLLM, Vercel AI SDK, and any OpenTelemetry-instrumented framework. The analytics dashboard surfaces cost breakdowns, quality scores, latency percentiles, and usage trends across models and prompt versions. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Every call your AI agents make becomes a searchable trace in LangWatch, and the same platform tests and scores those agents before a new prompt or model reaches users. The Trace Explorer filters by origin, user, model, cost, and evaluator, reads each run as a waterfall, flame graph, topology, or sequence diagram, and takes traces from the Python, TypeScript, and Go SDKs or any OpenTelemetry source. Agent Testing sends simulated users, in text or beta voice calls, through your agent while a judge passes or fails every criterion with its reasoning, adds multi-turn red teaming, and compares two agents side by side. The Experiments workbench scores prompts and agents on datasets with LLM judges, exact match, and RAG checks, then compares runs on cost, latency, and pass rate, while online evaluations monitor live traffic and guardrails block unsafe replies. Prompts keep version history and staging or production tags, Workflows wire prompts, Python code, and HTTP calls on a visual canvas, and annotation queues route traces to human reviewers. Alerts fire when a graph crosses a threshold, automations post to Slack, email, or webhooks, and scheduled reports go out by email or Slack. Claude Code, Codex, Copilot, Gemini CLI, and opencode report session cost, pull request cost, and terminal replays, and an MCP server exposes trace search and prompt management to coding assistants. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

When an AI agent breaks in production, Tracely freezes that exact run into a replayable regression test and blocks the pull request that would ship the bug again. Span soup hides the failure. OTLP/HTTP traces group into conversation threads, and a waterfall marks the failing span in red with its input and output. Nobody knows if an answer was good. Evaluators sit as columns on the trace table, mixing model-free checks with LLM judges at conversation, run, or span level. Thirty-one broken runs, one issue. Clustering turns them into a single counted issue and can suggest the evaluator that would have caught it. A fix nobody can prove. One click promotes a failing trace into a hermetic case with recorded fixtures and a fail-to-pass contract, and scenarios send a red-team model after an adversarial goal. Model bills on every CI run. The suite replays offline with no API keys and no model spend, and tracely gate posts a commit status, comments on the pull request, and blocks regressions. No warning when it breaks again. Alert flows drawn on a canvas react to failed gates, broken conversations, unseen failure modes, or rising failure rates by messaging Slack, sending email, calling a webhook, or prompting an LLM. Coding agents can read traces and write evaluators through an MCP endpoint. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

When an AI agent goes wrong at step seven of twelve, Opik, an open-source LLM observability platform, shows the exact trace, tool call, and prompt behind it, then scores your fix against test datasets. Full trace trees capture every LLM call, retrieval step, and tool execution, and you can annotate traces and spans with feedback scores in the UI or through the SDK. Native integrations for OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, Google ADK, and DSPy, plus OpenTelemetry for any language with an OpenTelemetry SDK, feed traces in, or add the @track decorator to your own functions. Datasets and experiments compare prompt and model versions, scored by LLM-as-a-judge metrics for hallucination, moderation, and RAG relevance, with a PyTest integration to run evaluations in CI. A Prompt Playground tests prompts across models, and the Agent Optimizer SDK helps improve prompts and agents. Production dashboards track feedback scores, trace counts, and token usage, online evaluation rules score live traces, and optional guardrails add safety checks. An MCP server lets Claude Code, Cursor, VS Code Copilot, Codex, or opencode read traces and run evaluations from chat. Run it yourself with the ./opik.sh script on Docker Compose, or a Helm chart on Kubernetes, so trace data stays in your infrastructure. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Backed by Y Combinator (S24) and processing traces from thousands of AI agents in production, Laminar is the open-source observability platform that treats agent debugging as a first-class engineering discipline rather than an afterthought. Its OpenTelemetry-native SDK auto-instruments Vercel AI SDK, LangChain, OpenAI, Anthropic, Gemini, Browser Use, Stagehand, Mastra, Pydantic AI, and the OpenAI Agents SDK with a single line of code, capturing every LLM turn, tool call, and sub-agent delegation as nested spans with full input/output data and token costs. The Signals engine lets you describe failures in plain language — "agent is stuck in a loop" or "tool returned empty results" — then reads every trace and alerts via Slack when it detects a match. A built-in debugger records runs and replays them from cache so each iteration takes seconds, designed for Claude Code, Cursor, or Codex to drive the repair loop via the MCP server or CLI. Run code-first evaluations in Python or TypeScript locally or in CI/CD pipelines, build datasets from production traces, and query everything with raw SQL through custom dashboards, the in-app editor, or your coding agent. The Rust backend delivers 20x trace compression, a custom real-time streaming engine, ultra-fast full-text search, and gRPC ingestion, while ClickHouse powers columnar analytics and PostgreSQL stores application state. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Evidently gives your models and LLM apps a proper check-up and keeps every result on one shared dashboard. Project dashboards chart predictions against targets, data drift, and quality scores over time, with counters for model calls and the share of drifted features. A reports page lists every evaluation snapshot with tags, metadata, timestamps, and view or download buttons, and a search box filters them by metadata. Over 100 built-in metrics cover data quality, classification, regression, ranking, recommendations, text descriptors, and LLM judges. Test suites add pass or fail conditions, so a nightly job can flag a drifting feature before customers notice a bad prediction. Datasets, traces, and prompts have their own tabs, which keeps retrieval and generation experiments beside the scores they produced. Reports export as HTML or JSON for sharing outside the dashboard. Results live in a local folder, a SQL database, or S3-compatible storage, and a remote workspace accepts snapshots sent from jobs running anywhere. Built-in demo projects with bike demand forecasting and e-commerce review classification show how a finished dashboard looks before you wire in your own data. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

Building LangGraph agents in code means imagining the graph; LangGraph GUI turns that graph into something you draw, wire, and run from the browser instead. You place a start node, add step nodes, and drag edges between ports to define the flow, with cycles created by simply connecting nodes back toward each other. Condition nodes split execution into green true paths and red false paths, info nodes append context to stage output, and subgraph nodes run a whole named graph inside another one. Tool steps call real Python functions that you define in tool nodes with the familiar decorator style, so file saving, lookups, and custom logic sit inside the visual flow. Graphs save to JSON on your computer, load back for editing, and upload to the server as workflow files ready to execute. You configure which LLM to use and its API key in the panel, run against local models through Ollama or hosted APIs, then download every file the run produced in the workspace. Files upload to the server workspace as working environment, and a cache clean clears it between runs. Example graphs for role-playing loops, lottery branch switches, and tool-using flows show patterns you can copy and refine with your own prompts. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

Drag nodes onto a canvas and ship an LLM app: Flowise is an open-source visual builder for AI agents and LLM applications, written in Node.js on LangChain.js and licensed Apache-2.0. You assemble flows by dragging nodes onto a canvas: models, prompts, memory, vector stores, retrievers, and tools, then wire them together and test in the built-in chat panel. Three builder types cover increasing complexity: Assistant for simple RAG chat over uploaded files, Chatflow for single-agent systems with techniques like rerankers and Graph RAG, and Agentflow for multi-agent orchestration with branching, looping, shared flow state, and human-in-the-loop checkpoints. Over 100 integrations connect data sources, vector databases, and both proprietary and open-source models, plus MCP client and server nodes for standard tool interop. Finished flows are exposed as REST APIs, embedded chat widgets, or via JS and Python SDKs - each flow gets an endpoint the moment it is saved, removing the deployment gap between a working prototype and something your application can call. Execution logs, visual step debugging, and external log streaming trace behavior, while input moderation and rate limiting act as guardrails; RBAC, SSO, and workspaces cover team deployments. Self-hosting keeps prompts, encrypted credentials, and conversation data on your own instance, which matters when flows handle internal documents or customer data - and wiring a model, prompt, memory, and vector store on the canvas replaces the boilerplate a hand-coded LangChain project would need.

Langflow turns LLM application development into a visual canvas where every node maps to a real LangChain primitive (chains, agents, retrievers, memory, vector stores, and tools) that teams wire together without writing integration boilerplate. The platform supports 15+ LLM provider components including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Cohere, Azure OpenAI, HuggingFace, and Ollama for fully local inference, with LiteLLM proxying to over 100 additional providers through a single OpenAI-compatible endpoint. Vector database integrations cover Pinecone, Weaviate, Chroma, Qdrant, Astra DB, OpenSearch, FAISS, and Milvus, while built-in Knowledge Bases introduced in version 1.8 allow RAG pipelines without any external vector service. Multi-agent orchestration enables agent-to-agent communication with conversation management, persistent Memory Bases for cross-session context retrieval, and step-by-step reasoning visibility in the interactive Playground. Every flow automatically becomes a callable REST API endpoint via the /run route and an MCP server exposable to Claude Desktop, Cursor, or any MCP-compatible client. Every component is a real Python class that developers can customize, extend, or replace, while Extension Bundles package third-party integrations as independent pip packages for modular installation. Tool integrations include web search, Slack, Gmail, Google Drive, GitHub, and custom REST API calls. Docker deployment with PostgreSQL persistence runs on port 7860. Backed by 153,000+ stars and DataStax. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.