Connect Amazon DynamoDB to Tinybird¶
This guide covers how to send DynamoDB data to Tinybird.
A native DynamoDB connector is being developed. Reach out to support@tinybird.co if you're interested in this feature.
This integration uses DynamoDB Streams to stream changes via Amazon Kinesis Data Streams out to Tinybird's Events API.
1. Configure a DynamoDB Stream¶
Configure a DynamoDB Stream to push events to Kinesis:
- Navigate to your DynamoDB dashboard.
- Select the relevant table.
- Go to Exports and streams.
- Select Turn on for Amazon Kinesis stream details.
- Choose an existing Kinesis data stream or create a new one.
You should now see a stream with an On status.
2. Set up your Tinybird Data Source¶
You'll need the URL for the Tinybird Events API in Step 3. The URL looks like this:
https://api.tinybird.co/v0/events/<your-datasource-name>?wait=true&token=<your-token>
Where <your-datasource-name>
is the name of your Tinybird Data Source, and your-token
is either the Workspace admin token
(testing) or a new Token with limited DATASOURCES:APPEND:datasource_name
and DATASOURCES:CREATE
scopes (production).
Replace the Tinybird API hostname/region with the right API URL region that matches your Workspace. Your Token lives in the Workspace under "Tokens".
You can either create a new Data Source manually now, or use this URL in the next step. If you don't create a Data Source manually, one will be created automatically when data is sent using this request, so replace <your-datasource-name>
with the desired name of the auto-created Data Source.
3. Create the Amazon Firehose stream¶
Create an Amazon Data Firehose that takes the events from the Kinesis data stream and sends them to Tinybird via HTTP POST request:
- Navigate to the “Amazon Data Firehose” AWS console.
- Select Create Firehose stream.
- Select Kinesis Data streams as the source and HTTP Endpoint as the destination.
- In Source Settings select Browse and select the Kinesis stream created in the previous step.
- Name your Kinesis stream.
- In Destination Settings, configure the HTTP endpoint URL with the Tinybird Events API (see previous step for base URL).
- Leave all other settings as default and select Create Delivery Stream.
4. Finished! Verify data in Tinybird¶
Once you have the Firehose stream created, you should start seeing data in your Tinybird Data Source. Go to the Tinybird UI and check that your Data Source has data.
Data from Kinesis is encoded. Working with this data is explained in the Kinesis guide.
Next steps¶
- Read more about working with Amazon Kinesis streams in Tinybird
- If you need to backfill historical data, use the Amazon S3 Connector to ingest files from Amazon S3.
- Want scheduled exports from DynamoDB? Check out this GitHub repo that uses Amazon S3 and AWS Lambda to push files to Tinybird.