S3 connector¶

You can set up a S3 connector to load your CSV, NDJSON, or Parquet files into Tinybird from any S3 bucket. Tinybird detects new files in your buckets and ingests them automatically.

Setting up the S3 connector requires creating and enabling a data source and its connection as separate files.

Supported file types¶

The S3 connector supports the following file types:

You can upload files with .json extension, provided they follow the Newline Delimited JSON (NDJSON) format. Each line must be a valid JSON object and every line has to end with a \n character.

Parquet schemas use the same format as NDJSON schemas, using JSONPath syntax.

Set up the connector¶

To set up the S3 connector, follow these steps.

Prerequisites¶

The S3 connector requires permissions to access objects in your Amazon S3 bucket. The IAM Role needs the following permissions:

• s3:GetObject
• s3:ListBucket
• s3:GetBucketNotification
• s3:PutBucketNotification
• s3:GetBucketLocation
• s3:PutObject
• s3:PutObjectAcl

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:GetObject",
          "s3:ListBucket",
          "s3:GetBucketNotification",
          "s3:PutBucketNotification",
          "s3:GetBucketLocation",
          "s3:PutObject",
          "s3:PutObjectAcl"
        ],
        "Resource": [
          "arn:aws:s3:::{bucket-name}",
          "arn:aws:s3:::{bucket-name}/*"
        ]
      }
    ]
}

TYPE s3
S3_REGION "<S3_REGION>"
S3_ARN "<IAM_ROLE_ARN>"

DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_CONNECTION_NAME s3sample
IMPORT_BUCKET_URI s3://my-bucket/*.csv
IMPORT_SCHEDULE @auto
IMPORT_STRATEGY APPEND

.datasource settings¶

The S3 connector use the following settings in .datasource files:

.connection settings¶

The S3 connector use the following settings in .connection files:

S3 file URI¶

The S3 connector supports the following wildcard patterns:

• Single asterisk or *: matches zero or more characters within a single directory level, excluding /. It doesn't cross directory boundaries. For example, s3://bucket-name/*.ndjson matches all .ndjson files in the root of your bucket but doesn't match files in subdirectories.
• Double asterisk or **: matches zero or more characters across multiple directory levels, including /. It can cross directory boundaries recursively. For example: s3://bucket-name/**/*.ndjson matches all .ndjson files in the bucket, regardless of their directory depth.

Use the full S3 file URI and wildcards to select multiple files. The file extension is required to accurately match the desired files in your pattern.

Examples¶

The following are examples of patterns you can use and whether they'd match the example file path:

Considerations¶

When using patterns:

• Use specific directory names or even specific file URIs to limit the scope of your search. The more specific your pattern, the narrower the search.
• Combine wildcards: you can combine ** with other patterns to match files in subdirectories selectively. For example, s3://bucket-name/**/logs/*.ndjson matches .ndjson files within any logs directory at any depth.
• Avoid unintended matches: be cautious with ** as it can match many files, which might impact performance and return partial matches.