Categories
PostgreSQL ETL Data Engineering Database Replication Change Data Capture Data Warehousing Real-Time StreamingStars
Forks
Watchers
Developer links
PeerDB
Replicate PostgreSQL to data warehouses 10x faster than conventional CDC tools, proven across 400+ companies including AutoNation and LC Waikiki collectively moving 200 TB monthly. The architecture pairs a Rust-based Nexus query layer implementing the PGWire protocol with Go-based Flow workers orchestrated by Temporal. Because Nexus speaks native Postgres wire protocol, any client tool (pgAdmin, psql, Grafana, Tableau, Flyway) can manage replication through standard SQL commands like CREATE MIRROR. Three streaming modes serve different needs: log-based CDC via logical replication slots, cursor-based streaming through timestamp or integer columns, and XMIN-based capture for tables lacking logical replication. Parallel initial load achieves consistent snapshots through transaction snapshotting and CTID range scans, reducing 100+ GB migrations from days to minutes. Native TOAST column handling processes large JSONB payloads and IoT data efficiently without row expansion penalties. Destinations include ClickHouse, Snowflake, BigQuery, Kafka, Azure Event Hubs, Google PubSub, S3, and PostgreSQL with in-flight SQL transformations. Schema change propagation, partitioned table support, and slot growth alerts ensure production reliability. Docker Compose bundles Temporal, catalog Postgres, Flow API, workers, and the Next.js monitoring UI. Deployable on RepoCloud with dedicated VPS resources under AGPL-3.0.
Benefits
- 10x Faster Postgres Replication
- Parallel initial load via CTID scans and transaction snapshotting reduces 100+ GB table migrations from days to minutes with sub-30-second CDC lag at 5000 transactions per second.
- Postgres-Compatible SQL Interface
- Rust-based Nexus implements PGWire protocol, enabling replication management through pgAdmin, psql, Grafana, Tableau, Flyway, or any Postgres-compatible client without proprietary APIs.
- Native TOAST Column Handling
- Efficiently streams large JSONB payloads, arrays, geospatial data, and custom Postgres types without row expansion penalties that slow down competing CDC tools.
- Multi-Destination Warehouse Support
- Replicates simultaneously to ClickHouse, Snowflake, BigQuery, Kafka, Azure Event Hubs, PubSub, and S3 with warehouse-specific query optimizations that reduce compute costs.
Features
- Real-Time CDC Streaming
- Log-based change data capture using PostgreSQL logical replication slots delivers 10-second latency with schema change propagation and partitioned table support.
- Temporal Workflow Orchestration
- Temporal engine manages data flow reliability with automatic retries, checkpointing, and horizontal scaling of Go-based Flow Workers across multiple nodes.
- Next.js Monitoring Dashboard
- Web UI provides real-time visibility into mirror status, throughput metrics, slot growth alerts, and connection health for all configured replication peers.
- SQL Transformations In-Flight
- Apply SQL-based transformations during replication including column mapping, filtering, and type coercion without intermediate staging or additional ETL tooling.
- Parallel Initial Load
- CTID range scans with transaction snapshot isolation enable consistent parallel loading of large tables across multiple workers simultaneously.