Common use cases¶
The following uses cases illustrate how Tinybird CLI solve common situations using available commands.
Download Pipes and data sources from your account¶
There are two ways you can start working with the CLI. You can either start a new data project from scratch, or if you already have some data and API Endpoints in your Tinybird account, pull it to your local disk to continue working from there.
For this second option, use the --match
flag to filter Pipes or data sources containing the string passed as parameter.
For instance, to pull all the files named project
:
Pull all the project files
tb pull --match project [D] writing project.datasource(demo) [D] writing project_geoindex.datasource(demo) [D] writing project_geoindex_pipe.pipe(demo) [D] writing project_agg.pipe(demo) [D] writing project_agg_API_endpoint_request_log_pipe_3379.pipe(demo) [D] writing project_exploration.pipe(demo) [D] writing project_moving_avg.pipe(demo)
The pull command doesn't preserve the directory structure, so all your datafiles are downloaded to your current directory.
Once the files are pulled, you can diff
or push
the changes to your source control repository and continue working from the command line.
When you pull data sources or Pipes, your data isn't downloaded, just the data source schemas and Pipes definition, so they can be replicated easily.
Push the entire data project¶
Push the whole project
tb push --push-deps
Push a Pipe with all its dependencies¶
Push dependencies
tb push pipes/mypipe.pipe --push-deps
Adding a new column to a Data Source¶
Data Source schemas are mostly immutable, but you have the possibility to append new columns at the end of an existing Data Source with an Engine from the MergeTree Family or Null Engine.
If you want to change columns, add columns in other positions, or modify the engine, you must first create a new version of the Data Source with the modified schema. Then ingest the data and finally point the Pipes to this new API Endpoint. To force a Pipe replacement use the --force
flag when pushing it.
If you create a new column with a DEFAULT
o MATERIALIZED
value, only the rows inserted after adding the column will write the data to disk. Data already in a Data Source when the column is added will not be modified, and the value will be computed at query time. That's not problematic when the expression is constant, for example a specific date or number, but for dynamic expression like now()
or now64()
, the returned value might change every time a select query is performed.
Append new columns to an existing Data Source¶
As an example, imagine you have the following Data Source defined, and it has been already pushed to Tinybird:
Appending a new column to a Data Source
SCHEMA > `test` Int16, `local_date` Date, `test3` Int64
If you want to append a new column, you must change the *.datasource
file to add the new column new_column
. You can append as many columns as you need at the same time:
Appending a new column to a Data Source
SCHEMA > `test` Int16, `local_date` Date, `test3` Int64, `new_column` Int64
Remember that when appending or deleting columns to an existing Data Source, the engine of that Data Source must be of the MergeTree family.
After appending the new column, execute tb push my_datasource.datasource --force
and confirm the addition of the column(s). The --force
parameter is required for this kind of operation.
Existing imports will continue working once the new columns are added, even if those imports don't carry values for the added columns. In those cases, the new columns contain empty values like 0
for numeric values or ''
for Strings, or if defined, the default values in the schema.
Create a new version of the Data Source to make additional add/change column operations¶
To create a new version of a Data Source, create a separate datafile with a different name. You can choose a helpful naming convention such as adding a _version
suffix (e.g. my_ds_1.datasource
).
Debug mode¶
When you work with Pipes that use several versions of different data sources, you might need to double check which version of which Data Source the Pipe is pointing at before you push it to your Tinybird account.
To do so, use the --dry-run --debug
flags like this:
Debug mode
tb push my_pipe.pipe --dry-run --debug
After you've validated the content of the Pipe, push your Pipe as normal.
Automatic regression tests for your API Endpoints¶
Any time you --force
push a Pipe which has a public API Endpoint that has received requests, some automatic regression tests are executed.
If the previous version of the API Endpoint returns the same data as the version you are pushing, the CLI checks for the top ten requests. This can help you validate whether you are introducing a regression in your API.
Other times, you are consciously --force
pushing a new version which returns different data. In that case you can avoid the regression tests with the --no-check
flag:
Avoid regression tests
tb push my_may_view_pipe.pipe --force --no-check
When pushing a Pipe with a public API Endpoint, the API Endpoint will be maintained based on the node name. If the existing API Endpoint node is renamed, the last node of the Pipe will be recreated as an API Endpoint. The latter option isn't an atomic operation: The API Endpoint will be down for a few moments while the new API Endpoint is created.