Connections¶

Connections store the credentials and settings Tinybird uses to connect to external services such as Kafka, Amazon S3, and Google Cloud Storage. Data Sources reference connections to stream or batch import data into Tinybird.

Connections are managed by Tinybird and are deployed as part of your project. They handle the integration details for:

• Authentication and secure connections
• Schema detection and mapping
• Incremental updates and change data capture
• Error handling and monitoring
• Scheduling and orchestration

Connections are independent from Data Sources. Deleting a Data Source that uses a connection does not delete the connection, so you can reuse the same connection in other Data Sources.

Define a connection¶

You can define connections with .connection files, in .ts files with the TypeScript SDK, or in .py files with the Python SDK. Use tb_secret() in datafiles, or secret() in the SDKs, to avoid storing credentials in plain text. See Templating language for how secrets are rendered across environments.

TYPE kafka
KAFKA_BOOTSTRAP_SERVERS {{ tb_secret("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092") }}
KAFKA_SECURITY_PROTOCOL SASL_SSL
KAFKA_SASL_MECHANISM PLAIN
KAFKA_KEY {{ tb_secret("KAFKA_KEY") }}
KAFKA_SECRET {{ tb_secret("KAFKA_SECRET") }}

See all syntax options in the Connection files reference, TypeScript SDK reference, and Python SDK reference.

Create a connection¶

To create a connection, add the .connection file to the project, or the connection definition to .ts or .py files and deploy.

For datafile projects, you can scaffold a new connection with tb connection create:

tb connection create <type>

Where <type> is one of the available connection types:

• kafka - Create a Kafka connection
• s3 - Create an Amazon S3 connection
• gcs - Create a Google Cloud Storage connection

The CLI will guide you through the configuration process, prompting you for the required credentials and settings. Once created, the connection will be saved as a .connection file in your project's connections/ folder.

To convert existing .connection files to SDK definitions, use tinybird migrate for TypeScript projects or tinybird migrate for Python projects.

Use a connection in a Data Source¶

After you define a connection, reference it from a Data Source. Use KAFKA_CONNECTION_NAME, S3_CONNECTION_NAME, or GCS_CONNECTION_NAME in .datasource files, or pass the connection object in the TypeScript and Python SDKs.

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME events_kafka
KAFKA_TOPIC events
KAFKA_GROUP_ID {{ tb_secret("KAFKA_GROUP_ID") }}

Delete a connection¶

To delete a connection, remove its .connection file or SDK definition, make sure no Data Sources reference it, and deploy your changes.

tb deploy --allow-destructive-operations

The Tinybird CLI and TypeScript SDK CLI require the --allow-destructive-operations flag to confirm the removal. The Python SDK CLI doesn't currently support that flag.

Available connection types¶

Tinybird supports the following connection types:

• Kafka
• Amazon S3
• Google Cloud Storage