Stars
Forks
Watchers
Developer links
Kestra
Data, AI, and infrastructure workflows, orchestrated from declarative YAML: Kestra is an open-source, event-driven orchestration platform. Flows are declared in YAML - no DSL rewrites or Python decorators - and the definition stays the single source of truth even when edited through the UI, API, CI/CD, or Terraform, which makes pull-request review, versioning, and rollback natural. Tasks run in any language: Python, Node.js, Go, Rust, R, SQL, or Bash scripts executed in containers, and a plugin ecosystem of 1,000+ integrations covers ingestion, dbt, Airbyte, Spark, cloud storage, databases, and messaging systems. Scheduling supports cron triggers, event triggers, backfills, and conditional branching, with retries, timeouts, error handling, and typed inputs and outputs that surface artifacts in the UI. Namespaces, labels, and subflows organize workflows at scale, and the embedded code editor includes Git integration. Common uses span ETL/ELT pipelines, dbt runs, microservice coordination, infrastructure provisioning, and human-in-the-loop approvals. Java-based, Apache 2.0 licensed, deployed via Docker or Kubernetes.
Benefits
- Orchestration Logic as Reviewable Code
- A whole pipeline lives in one YAML file, so changes go through pull requests, diffs stay readable, and rollbacks are a git revert - without writing framework-specific Python.
- Keep Your Existing Scripts
- Kestra wraps the code you already have. Python, R, Node.js, Go, or Bash tasks run as-is in containers, with no DSL rewrite or decorator retrofit required.
- One Engine for Batch and Events
- Cron schedules, event triggers, and backfills share the same declarative model, replacing the separate schedulers and glue scripts teams usually stitch together.
- Accessible Beyond Data Engineers
- The UI editor with autocomplete and no-code task forms lets analysts and ops staff modify flows safely, while everything remains governed, versioned code underneath.
Features
- Declarative YAML Flows
- Workflows defined as configuration with structured tasks, inline documentation, variables, and typed inputs/outputs; UI edits sync back to the YAML definition.
- Scheduled and Event-Driven Triggers
- Cron scheduling, event and webhook triggers, backfills for historical runs, and conditional execution in one trigger model.
- 1,000+ Plugins
- Integrations for dbt, Airbyte, Spark, cloud storage, warehouses, databases, and messaging, plus a framework for building custom plugins.
- Any-Language Task Runners
- Run Python, Node.js, Go, Rust, R, SQL, or Shell tasks in Docker containers with per-task dependency management.
- Resilience Built In
- Retries, timeouts, error branches, SLAs, and parallel or sequential execution with dynamic tasks generated at runtime.
- Governance and Observability
- Namespaces, labels, and subflows for organization; execution history, logs, artifacts, and instance dashboards for runtime health.