Logo
Deploy Now

Categories

Data Pipeline ETL Data Engineering Data Integration AI Platform ELT Connectors

Developer links

Last Updated: 01-01-2023 12:00 AM (UTC)

Airbyte

Backed by over 21,800 GitHub stars and more than 1,000 community contributors, Airbyte has become the standard open-source data movement platform, powering ELT pipelines for organizations ranging from startups to Fortune 500 enterprises. The platform provides 600+ pre-built connectors covering PostgreSQL, MySQL, MongoDB, Snowflake, BigQuery, Redshift, S3, Salesforce, HubSpot, Stripe, Shopify, Google Analytics, and hundreds of additional APIs, databases, and SaaS applications. The no-code Connector Builder lets practitioners create new source connectors in minutes by pointing at an API documentation URL, while the Python CDK enables custom connectors with full programmatic control for complex authentication flows and pagination strategies. Airbyte's AI agent capabilities include the MCP Gateway for Model Context Protocol integration, the open-source Agent SDK compatible with pydantic-ai, LangChain, OpenAI Agents, and FastMCP, and a Context Store that lets AI agents query business data across connected systems without runtime API stitching. Change Data Capture streams incremental updates from PostgreSQL, MySQL, and SQL Server using Debezium, while dbt integration handles post-load transformations within the pipeline. Self-hosted deployment uses Kubernetes via the abctl CLI tool, which bootstraps a local kind cluster with a single command, or Helm charts for production clusters with Keycloak OIDC authentication and secrets management through AWS Secrets Manager, Google Secrets Manager, or HashiCorp Vault. Running on a dedicated VPS on RepoCloud with guaranteed CPU, RAM, and SSD, full root SSH access, and a browser serial console. ELv2 licensed with MIT-licensed connectors.

Airbyte
Airbyte
Airbyte
Airbyte
Airbyte

Benefits

  • 600+ Pre-Built Data Connectors
  • Connect to PostgreSQL, MySQL, MongoDB, Snowflake, BigQuery, Salesforce, Stripe, HubSpot, and hundreds more sources and destinations with maintained, community-tested connectors out of the box.
  • AI Agent Context Layer
  • Provide LLMs real-time business data access via MCP Gateway and the open-source Agent SDK, compatible with pydantic-ai, LangChain, OpenAI Agents, and FastMCP for production-grade AI workflows.
  • No-Code Connector Builder
  • Create new source connectors without writing code using the visual Connector Builder. Point it at API documentation and configure authentication, pagination, and schema mapping through the UI.
  • Incremental Change Data Capture
  • Stream incremental updates from PostgreSQL, MySQL, and SQL Server using Debezium-based CDC, reducing data transfer volumes and enabling near-real-time synchronization with warehouse destinations.

Features

  • ELT Pipeline Orchestration
  • Schedule and monitor data sync jobs with configurable frequency, full and incremental modes, schema change handling, and integrated dbt transformations for post-load processing.
  • Connector Development Kit
  • Build custom connectors using the Python CDK with declarative YAML manifests or full programmatic control for complex APIs requiring custom authentication and pagination.
  • MCP Gateway Integration
  • Expose connected data sources as Model Context Protocol tools for AI agents, enabling LLMs to query CRMs, databases, and SaaS APIs through a unified protocol.
  • Kubernetes-Native Deployment
  • Deploy via abctl for single-command local clusters or Helm charts for production Kubernetes with Keycloak OIDC, secrets management, and horizontal workload scaling.
  • Schema Management
  • Automatically detect schema changes in source systems with configurable propagation policies that add, remove, or ignore column changes in destination tables.