PostgreSQL CDC with Redpanda Connect¶
Redpanda Connect is an ecosystem of high-performance streaming connectors that serves as a simplified and powerful alternative to Kafka Connect.
Tinybird is the ideal complement to Postgres for handling OLAP workloads. The following guide shows you how to use Redpanda Connect to ingest data from a Postgres database into Tinybird.
Before you start¶
Before you connect Postgres to Redpanda, ensure:
- You have a Redpanda cluster and Redpanda Connect installed with version 4.43.0 or higher. The following instructions use Redpanda Serverless, but you can use Redpanda Cloud Dedicated or self-hosted.
- You have a PostgreSQL database with logical replication enabled.
Connect Postgres to Redpanda¶
In the Redpanda Cloud console, select Connect from the navigation menu, then select Create Pipeline.
Add the pipeline configuration. You need the following information:
- Postgres connection string (
dsn
) - Redpanda brokers (
seed_brokers
) - SASL mechanism (
mechanism
) - Username (
username
) - Password (
password
)
Use the following YAML template:
input: label: "postgres_cdc" postgres_cdc: dsn: <<postgresql://user:pass@host:port/db>> include_transaction_markers: false slot_name: test_slot_native_decoder snapshot_batch_size: 100000 stream_snapshot: true temporary_slot: true schema: public tables: - <<Table name>> output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: <<Topic name>> tls: enabled: false sasl: - mechanism: SCRAM-SHA-512 password: <<Password>> username: <<Username>>
See the Redpanda Connect docs for more information on the redpanda
output and postgres_cdc
input.
- Start the Redpanda Connect pipeline
Select Create to save and create the pipeline. This takes you back to the pipeline screen, where you can find your new pipeline. Open the new pipeline to view the logs and confirm that the pipeline is running.
Select the Topics page from the navigation menu and confirm that the topic exists and that messages are being produced.
- Connect Redpanda to Tinybird
In Tinybird, add a new Data Source and select the Redpanda connector. Enter your Redpanda connection details, then select the topic used in the Redpanda Connect pipeline. Confirm your schema and select Create Data Source.
Redpanda Connect continuosly consumes changes from Postgres and pushes them to your Redpanda topic. Tinybird consumes the changes from Redpanda in real time, making them available to query with minimal latency.