Trusted by thousands of organizations with over 30 million monthly downloads and 20,000+ GitHub stars, MLflow is the largest open-source AI engineering platform providing end-to-end lifecycle management for traditional ML models, LLMs, and AI agents. The OpenTelemetry-based tracing system captures complete request flows through any LLM provider or agent framework — including OpenAI, LangChain, DSPy, Vercel AI, PydanticAI, and smolagents — with one-line auto-instrumentation that tracks inputs, outputs, token usage, and costs at every intermediate step. MLflow's evaluation engine offers 50+ built-in metrics and LLM judges for systematic quality assessment, detecting issues across correctness, latency, adherence, relevance, and safety dimensions before code reaches production. The Prompt Registry versions, tests, and deploys prompts with full lineage tracking while automated optimization algorithms improve prompt performance using evaluation feedback. The AI Gateway provides a unified API endpoint for all LLM providers, enforcing rate limits, cost controls, and access policies across the organization. MLflow 3.0 introduces the LoggedModel abstraction linking traces, metrics, and prompts to specific model versions across Python, TypeScript, Java, and R SDKs. The model registry manages deployment workflows with automated quality gates, while experiment tracking records parameters, metrics, and artifacts across training runs. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache License 2.0 licensed.

Agenta delivers a comprehensive open-source LLMOps workspace that covers the full lifecycle of AI application development — from prompt engineering through production monitoring. The platform supports 15+ model providers including OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure, AWS Bedrock, and self-hosted models via Ollama, enabling teams to switch between providers without code changes. The prompt playground allows side-by-side comparison of different configurations, while the evaluation system offers LLM-as-a-Judge assessment, 20+ pre-built evaluators covering semantic similarity, regex matching, and factual accuracy, plus custom Python evaluators for domain-specific requirements. Teams run evaluations through both the web UI for subject matter experts and the Evaluation SDK for programmatic CI/CD integration. The observability layer captures full trace visibility across complex agentic workflows, flagging quality issues like hallucinations and off-topic responses in real time. Human annotation workflows let domain experts review and annotate LLM outputs, feeding corrections back into the evaluation loop. The architecture supports Chain of Prompts, RAG pipelines, and multi-step agent workflows, integrating with frameworks like LangChain and LlamaIndex. Self-hosting deploys via Docker Compose with Traefik for routing, requiring only a clone, environment configuration, and a single docker compose command. On RepoCloud, deploy Agenta on a dedicated VPS with root SSH access, persistent storage for evaluation datasets and traces, and complete control over model provider credentials, all under the MIT license with no usage restrictions.

Backed by Y Combinator and trusted by over 2,300 companies processing billions of observations monthly, Langfuse is the most widely adopted open-source platform for building, monitoring, evaluating, and debugging LLM applications. The hierarchical tracing engine captures every LLM call, tool invocation, retrieval step, and agent action as nested spans based on OpenTelemetry, with automatic cost calculation, latency tracking, and token usage attribution across sessions and users. Prompt Management separates prompts from code with versioned artifacts, label-based deployments, one-click rollbacks, and runtime SDK fetching with server-side caching, while linking every generation back to its exact prompt version for attribution analytics. The evaluation system supports LLM-as-a-judge scoring, heuristic code evaluators, user feedback collection, and manual annotation workflows that run automatically on production traces or against curated datasets. The Playground enables interactive prompt testing on real production inputs with side-by-side model comparison across providers. Datasets and Experiments define test cases for systematic benchmarking with comparative result visualization. Native SDKs for Python and TypeScript provide decorator-based instrumentation, while 100+ integrations cover LangChain, LlamaIndex, OpenAI SDK, LiteLLM, Vercel AI SDK, and any OpenTelemetry-instrumented framework. The analytics dashboard surfaces cost breakdowns, quality scores, latency percentiles, and usage trends across models and prompt versions. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. MIT licensed.

When an AI agent goes wrong at step seven of twelve, Opik, an open-source LLM observability platform, shows the exact trace, tool call, and prompt behind it, then scores your fix against test datasets. Full trace trees capture every LLM call, retrieval step, and tool execution, and you can annotate traces and spans with feedback scores in the UI or through the SDK. Native integrations for OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, Google ADK, and DSPy, plus OpenTelemetry for any language with an OpenTelemetry SDK, feed traces in, or add the @track decorator to your own functions. Datasets and experiments compare prompt and model versions, scored by LLM-as-a-judge metrics for hallucination, moderation, and RAG relevance, with a PyTest integration to run evaluations in CI. A Prompt Playground tests prompts across models, and the Agent Optimizer SDK helps improve prompts and agents. Production dashboards track feedback scores, trace counts, and token usage, online evaluation rules score live traces, and optional guardrails add safety checks. An MCP server lets Claude Code, Cursor, VS Code Copilot, Codex, or opencode read traces and run evaluations from chat. Run it yourself with the ./opik.sh script on Docker Compose, or a Helm chart on Kubernetes, so trace data stays in your infrastructure. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Every call your AI agents make becomes a searchable trace in LangWatch, and the same platform tests and scores those agents before a new prompt or model reaches users. The Trace Explorer filters by origin, user, model, cost, and evaluator, reads each run as a waterfall, flame graph, topology, or sequence diagram, and takes traces from the Python, TypeScript, and Go SDKs or any OpenTelemetry source. Agent Testing sends simulated users, in text or beta voice calls, through your agent while a judge passes or fails every criterion with its reasoning, adds multi-turn red teaming, and compares two agents side by side. The Experiments workbench scores prompts and agents on datasets with LLM judges, exact match, and RAG checks, then compares runs on cost, latency, and pass rate, while online evaluations monitor live traffic and guardrails block unsafe replies. Prompts keep version history and staging or production tags, Workflows wire prompts, Python code, and HTTP calls on a visual canvas, and annotation queues route traces to human reviewers. Alerts fire when a graph crosses a threshold, automations post to Slack, email, or webhooks, and scheduled reports go out by email or Slack. Claude Code, Codex, Copilot, Gemini CLI, and opencode report session cost, pull request cost, and terminal replays, and an MCP server exposes trace search and prompt management to coding assistants. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

A looping agent burns money quietly until someone can see where it drifted, and PandaProbe is an observability and evaluation platform for AI agents that makes that drift visible step by step. Capture complete executions as sessions, traces, and spans spanning model calls, tool calls, multi-agent handoffs, and custom logic, then follow a looping or failing run back to the decision where it drifted. Score traces and sessions with agent-specific metrics, run asynchronous and sampled evaluations so judge models do not cost more than inference, and use lightweight trajectory signals to catch stalls, bad tool use, and regressions over time. Bring agents built with LangGraph, CrewAI, the Claude Agent SDK, the OpenAI Agents SDK, and other integrations, with MCP tool calls captured automatically where supported. Organize work by user, organization, and project, send data through an SDK or HTTP API with project keys, and query filtered, paginated traces when you need the raw record. Keep sensitive workflows local when that matters: prompts, traces, and evaluation results stay inside the deployment you run, instead of leaving for a hosted console. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

Backed by Y Combinator (S24) and processing traces from thousands of AI agents in production, Laminar is the open-source observability platform that treats agent debugging as a first-class engineering discipline rather than an afterthought. Its OpenTelemetry-native SDK auto-instruments Vercel AI SDK, LangChain, OpenAI, Anthropic, Gemini, Browser Use, Stagehand, Mastra, Pydantic AI, and the OpenAI Agents SDK with a single line of code, capturing every LLM turn, tool call, and sub-agent delegation as nested spans with full input/output data and token costs. The Signals engine lets you describe failures in plain language — "agent is stuck in a loop" or "tool returned empty results" — then reads every trace and alerts via Slack when it detects a match. A built-in debugger records runs and replays them from cache so each iteration takes seconds, designed for Claude Code, Cursor, or Codex to drive the repair loop via the MCP server or CLI. Run code-first evaluations in Python or TypeScript locally or in CI/CD pipelines, build datasets from production traces, and query everything with raw SQL through custom dashboards, the in-app editor, or your coding agent. The Rust backend delivers 20x trace compression, a custom real-time streaming engine, ultra-fast full-text search, and gRPC ingestion, while ClickHouse powers columnar analytics and PostgreSQL stores application state. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

An agent run can finish without errors and still give the wrong answer, so TraceRoot traces every model and tool call, flags the production runs that missed the mark, and helps your coding agent ship verified fixes. Each run becomes a browsable call tree with inputs, outputs, latency, and cost, searchable by session or filter. Instrument with the Python or TypeScript SDK, the agent-native CLI, or any OpenTelemetry exporter, with ready integrations for LangChain, LangGraph, OpenAI Agents SDK, CrewAI, Vercel AI SDK, Mastra, and Pydantic AI. Detectors use LLM judges to score live traffic against the behavior you define and can post alerts to Slack. Judge outputs cluster into signals, then a background coding agent reads the traces and your connected GitHub code in a sandbox and recommends changes down to the commit, file, and line. Datasets, test cases, and scored eval runs compare versions so regressions surface before a release, and dashboards of widgets chart your trace data. The CLI hands your coding agent, such as Claude Code or Codex, the traces, signals, detectors, and evals to work from. Self-hosting keeps all of that trace data on infrastructure you control. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed (core edition).

Exploratory data work, teaching material, and research code all live in the same browser tab with JupyterLab, the web workspace of Project Jupyter. Notebooks combine runnable code cells, Markdown, and rich output such as images, JSON, and Vega charts, and they use the same notebook format as the classic Jupyter Notebook. Any Jupyter kernel can execute the code, so Python, R, and other languages share one interface, and code inside plain text files such as Markdown or LaTeX can be run interactively as well. Tabs and splitters arrange documents side by side, and a notebook's cell outputs can be mirrored into their own tab to build a simple dashboard with interactive controls. Code consoles serve as scratchpads linked to a notebook kernel, while full terminals run system shells such as bash on the server. A visual debugger offers breakpoints, a call stack, and a variable explorer for kernels that support the Jupyter debug protocol. Markdown, CSV, and Vega documents show a live preview as they are edited, and built-in viewers handle images, JSON, and PDF files. The Extension Manager installs themes, editors, and renderers from PyPI, and real-time collaboration becomes available by adding the jupyter-collaboration extension. Access is protected by a generated token unless a password is set. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. BSD-3-Clause licensed.

Marimo is a reactive Python notebook that treats cells like spreadsheet formulas: change one cell or interact with a UI widget and every dependent cell automatically re-executes, eliminating the hidden state bugs that make traditional notebooks unreliable. Backed by over 22,000 GitHub stars, notebooks are stored as pure Python files with PEP 723 inline metadata, making them Git-diffable, importable as modules, executable as CLI scripts with parameterized arguments, and testable with PyTest. Built-in SQL cells query Polars, Pandas, PyArrow, DuckDB, SQLite, PostgreSQL, and MySQL databases, with results automatically flowing into the reactive dependency graph. The AI-native editor provides GitHub Copilot autocomplete, context-aware assistants that access live runtime variables, inline code edits powered by configurable models from OpenAI, Anthropic, or local Ollama instances, and a pair mode that lets external AI agents connect over WebSocket. Notebooks become read-only interactive web applications with marimo run, collaborative authoring environments with marimo edit, or embedded flows inside existing FastAPI applications through ASGI middleware. Gallery mode serves multiple notebooks from a single instance with an auto-generated index page. The Docker image ships with SQL support, token-based authentication, health check endpoints at /health and /api/status, and configurable WebSocket or SSE kernel transport for proxy compatibility. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache 2.0 licensed.

One server, a login for everyone, and a private JupyterLab in every browser tab: that is the whole pitch, and for classrooms and research teams it is transformative. JupyterHub authenticates each user against your existing identity system (campus OAuth, GitHub, Google, LDAP, or local accounts), then spawns a dedicated single-user server just for them, with their files, their kernels, and their compute allocation. Users open notebooks, terminals, and extensions in JupyterLab without installing anything, and their environment is identical whether they log in from a lab workstation or a laptop at home. The Spawner decides where those servers live: plain processes on one machine for a small class, Docker containers for isolation, or Kubernetes pods that scale across a cluster for a whole university. Administrators manage users, groups, and named servers from the admin panel, grant precise permissions through roles and scopes, and watch start times and profiles for every running server. External services plug into the Hub API, so dashboards, autograders, and portals can authenticate through the same OAuth machinery. When a user disappears for the semester break, culling stops their idle server and frees the resources. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. BSD-3-Clause licensed.

Images, text, audio, video, HTML, PDFs, and time series, labeled in one tool with a standardized output format: Label Studio is the open-source data labeling platform for building training datasets. Computer vision tasks cover classification, object detection (boxes, polygons, ellipses, keypoints), and semantic segmentation; audio work spans transcription, speaker diarization, and emotion recognition; NLP handles named entity recognition and document classification with taxonomies up to 10,000 classes; and GenAI workflows support LLM fine-tuning data and RLHF response ranking. Labeling interfaces are fully configurable with an XML-like templating language, so the UI matches the task instead of the reverse. The ML backend SDK turns any model into a connected web server for pre-annotation (model predicts, humans verify), interactive labeling (real-time predictions as annotators draw regions or highlight text), and model evaluation - cutting annotation time dramatically on large datasets. Data imports from S3, GCS, or file uploads; the Data Manager filters and explores tasks; exports convert to the format your ML library expects via label-studio-converter. Multi-user accounts tie every annotation to its author, and webhooks, a Python SDK, and REST API embed labeling into any pipeline. Self-hosting keeps proprietary training data - often a company's most sensitive asset - entirely on your infrastructure.

Before a language model learns from your data, someone has to label it well; Argilla gives AI engineers and domain experts a shared workspace to label, review, and refine the datasets that fine-tune language models and train NLP pipelines. You build each dataset from the Python SDK, mixing text, chat, and image fields with the exact questions your task needs: single-label classification for intent detection, span questions for named entities, ratings for response quality, rankings for preference tuning, or free text for corrections. Annotators then work through records in a focused labeling interface where model predictions already sit attached as suggestions, turning most reviews into a fast confirm-or-correct decision instead of a blank-page exercise. Full-text search, metadata filters, and status filters slice a hundred thousand records down to the ones needing attention, while vector-based semantic search finds lookalike examples, near-duplicates, and the disagreement hotspots where guidelines need sharpening. Dataset owners watch completion progress, manage who has responded to what, and split work across workspaces with owner, admin, and annotator roles for each team or client. Finished datasets export with responses, suggestions, and metadata intact, locally or straight to the Hugging Face Hub, ready for fine-tuning and evaluation runs. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

With 23,600 GitHub stars, 13 million monthly PyPI downloads, and 425+ contributors automating over 200 million data tasks monthly for Fortune 50 companies like Progressive Insurance and disruptors like Cash App, Prefect is the most widely deployed open-source workflow orchestration framework for Python — turning any script into a resilient production pipeline with a single @flow decorator while eliminating rigid DAG structures entirely. The durable execution engine persists task results and automatically resumes from failures without replaying expensive upstream work, guaranteeing exactly-once execution for any Python function. Event-driven automation triggers workflows from webhooks, cloud events, or state changes through a real-time event bus that detects what happens or fails to happen across your entire data platform. Work pools decouple workflow code from infrastructure, enabling seamless switching between Docker, Kubernetes, AWS ECS, Azure Container Instances, GCP Cloud Run, and serverless environments without modifying pipeline logic. Native Ray and Dask task runners extend execution across clusters for compute-intensive workloads. The self-hosted server provides a monitoring dashboard with flow run timelines, task state visualization, scheduling, and automation configuration. The third-generation engine reduces overhead by over 90 percent compared to Prefect 2, supporting batch, event-driven, interactive, and background task workflows. Deploy via Docker Compose with PostgreSQL, Redis, server, background services, and worker containers, or use official Helm charts for production Kubernetes. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

With nearly 16,000 GitHub stars, 5.7 million monthly PyPI downloads, and 400+ contributors, Dagster is the most widely adopted asset-centric data orchestration platform — replacing task-oriented schedulers like Apache Airflow with a declarative model where every pipeline is defined as Python functions producing data assets such as tables, datasets, machine learning models, and reports. The built-in asset graph provides automatic lineage tracking across your entire data platform, showing exactly how data flows from ingestion through transformation to downstream consumption in a single unified view. Declarative Automation goes beyond cron scheduling with event-driven conditions that intelligently trigger materializations based on upstream freshness, data quality signals, and dependency state. The integrated data catalog auto-generates documentation from asset metadata, ensuring it never drifts out of sync with production. Native first-class integrations connect dbt, Snowflake, BigQuery, Databricks, Fivetran, Airbyte, Spark, Great Expectations, Tableau, Power BI, AWS, GCP, and Azure without custom glue code. The web UI visualizes asset graphs, run history, schedules, sensors, and partitioned materializations with built-in alerting via Slack and PagerDuty. Dagster Pipes enables executing arbitrary code in external environments including Spark clusters, Kubernetes Jobs, and cloud functions. Deploy via Docker Compose on a single VM with separate containers for the webserver, daemon, and code locations, or use official Helm charts for production Kubernetes with K8sRunLauncher scaling each run as an independent Job. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. Apache-2.0 licensed.

The de facto dashboard of observability: Grafana is the open-source frontend that turns the data stores you already run into interactive graphs. It does not store metrics itself; it connects to the data stores you already run and turns their contents into interactive dashboards. Supported sources number over 150 via plugins: Prometheus, Loki, Tempo, InfluxDB, Elasticsearch, MySQL, PostgreSQL, Microsoft SQL Server, AWS CloudWatch, Azure Monitor, Google Cloud Monitoring, and many more. Dashboards are built from a large library of panel types (time series, heatmaps, tables, gauges, logs) with template variables for reusable, parameterized views. Unified alerting evaluates rules against any connected data source, not just Prometheus, and routes notifications to Slack, PagerDuty, email, and other channels with grouping and silencing - unlike Prometheus Alertmanager, a single rule can combine a Loki log pattern, a PostgreSQL query result, and a CloudWatch metric. Dashboards serialize to JSON and data sources configure via provisioning files, so the entire observability setup can live in Git and deploy repeatably across environments. Explore mode adds ad-hoc querying outside dashboards, with split view for correlating a metric spike against the matching log lines, and access control spans organizations, teams, folder permissions, and OAuth, LDAP, and SAML integration. Written in Go and TypeScript, AGPL-licensed. Self-hosting gives you unlimited users, dashboards, and queries at flat hosting cost, without Grafana Cloud's usage-based pricing.