insertion_date deprecation¶

We're deprecating the automatically-generated insertion_date column. Here's why:

When creating a new Data Source in Tinybird, the Sorting Key is a required setting. Users can specify it with the optional ENGINE_SORTING_KEY setting. If it's not specified, Tinybird chooses a safe default from the defined columns. Only non-nullable columns can be used for the Sorting Key, so if all columns in the schema are nullable, Tinybird created a new non-nullable column called insertion_date and used that as the Sorting Key.

This automatically-created column, which isn't defined by the user, was causing issues when iterating a Data Source. It's not explicitly listed in the .datasource file but still exists, creating an inconsistency. To address this, we've deprecated this behavior. Now, ENGINE_SORTING_KEY will default to tuple() if there are no suitable columns in the schema definition.

This new behavior ensures that the .datasources files in your data project are 100% consistent with your Workspace resources.

How to prepare for the deprecation¶

Update your Tinybird CLI to > 5.0.0 and run tb pull --force. This will make insertion_date explicit in your datafiles.

If you use Branches and version control, we have added an example in our Use Case repository that demonstrates how to integrate these changes.

You can use a CLI version prior to 5.0.0 if you need to continue using the insertion_date column implicitly.