S3 connector¶

You can set up an S3 connector to load your CSV, NDJSON, or Parquet files into Tinybird from any S3 bucket. Tinybird can detect new files in your buckets and ingest them automatically.

Setting up the S3 connector requires:

1. Configuring AWS permissions using IAM roles.
2. Creating a connection file in Tinybird.
3. Creating a data source that uses this connection

Environment considerations¶

Before setting up the connector, understand how it works in different environments. See Environments.

Cloud environment¶

In the Tinybird Cloud environment, Tinybird uses its own AWS account to assume the IAM role you create, allowing it to access your S3 bucket.

Local environment¶

When using the S3 connector in the Tinybird Local environment, which runs in a container, you need to pass your local AWS credentials to the container. These credentials must have the permissions described in the AWS permissions section, including access to S3 operations like GetObject, ListBucket, etc. This allows Tinybird Local to assume the IAM role you specify in your connection.

To pass your AWS credentials, use the --use-aws-creds flag when starting Tinybird Local:

tb local start --use-aws-creds

» Starting Tinybird Local...
✓ AWS credentials found and will be passed to Tinybird Local (region: us-east-1)
* Waiting for Tinybird Local to be ready...
✓ Tinybird Local is ready!

If you're using a specific AWS profile, you can specify it using the AWS_PROFILE environment variable:

AWS_PROFILE=my-profile tb local start --use-aws-creds

Set up the connector¶

tb connection create s3

TYPE s3
S3_REGION "<S3_REGION>"
S3_ARN "<IAM_ROLE_ARN>"

DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_CONNECTION_NAME s3sample
IMPORT_BUCKET_URI s3://my-bucket/*.csv
IMPORT_SCHEDULE @auto

tb --cloud deploy --check

.connection settings¶

The S3 connector use the following settings in .connection files:

.datasource settings¶

The S3 connector uses the following settings in .datasource files:

S3 file URI¶

The S3 connector supports the following wildcard patterns:

• Single asterisk or *: matches zero or more characters within a single directory level, excluding /. It doesn't cross directory boundaries. For example, s3://bucket-name/*.ndjson matches all .ndjson files in the root of your bucket but doesn't match files in subdirectories.
• Double asterisk or **: matches zero or more characters across multiple directory levels, including /. It can cross directory boundaries recursively. For example: s3://bucket-name/**/*.ndjson matches all .ndjson files in the bucket, regardless of their directory depth.

Use the full S3 file URI and wildcards to select multiple files. The file extension is required to accurately match the desired files in your pattern.

Examples¶

The following are examples of patterns you can use and whether they'd match the example file path:

Considerations¶

When using patterns:

• Use specific directory names or even specific file URIs to limit the scope of your search. The more specific your pattern, the narrower the search.
• Combine wildcards: you can combine ** with other patterns to match files in subdirectories selectively. For example, s3://bucket-name/**/logs/*.ndjson matches .ndjson files within any logs directory at any depth.
• Avoid unintended matches: be cautious with ** as it can match many files, which might impact performance and return partial matches.

Supported file types¶

The S3 connector supports the following file types:

You can upload files with .json extension, provided they follow the Newline Delimited JSON (NDJSON) format. Each line must be a valid JSON object and every line has to end with a \n character.

Parquet schemas use the same format as NDJSON schemas, using JSONPath syntax.

AWS permissions¶

The S3 connector requires an IAM Role with specific permissions to access objects in your Amazon S3 bucket:

• s3:GetObject
• s3:ListBucket
• s3:GetBucketNotification
• s3:PutBucketNotification
• s3:GetBucketLocation

You need to create both an access policy and a trust policy in AWS:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetObject",
          "s3:ListBucket",
          "s3:GetBucketNotification",
          "s3:PutBucketNotification",
          "s3:GetBucketLocation"
        ],
        "Resource": [
          "arn:aws:s3:::{bucket-name}",
          "arn:aws:s3:::{bucket-name}/*"
        ]
      }
    ]
}