Jan 31, 2022

Add data from CSVs with different column orders

Add data to a Data Source from a CSV, even if the columns are in a different order to the one used when the Data Source was created.
Pablo Marcos Oltra
Backend Engineer

Some of our users explained that sometimes the column order in CSV data files they wanted to add to a Data Source was not consistent with the column order in the Data Source. This resulted in data going into quarantine.

We improved Tinybird so that even if the columns are in a different order to the one used when the Data Source was created, you can add data to a Data Source from a CSV, provided, of course, that all the column names are in the CSV header.

Let’s look at an example of appending data to a exiting Data Source of movie likes and dislikes. The original Data Source was created from a CSV with this column order:

Depending on the source of the data and how the CSVs are generated, the order of the columns might not be guaranteed. You could very well find a CSV with the same columns in different order:

This is not a problem since the data will be appended correctly.

Append from the UI

In the UI, to add data the new data to the movies Data Source just select add Data Source and upload the file named movies.csv, where the CSV header contains the column names.

SCHEMA > Evolution
Get 10 links weekly to the Data and AI articles the Tinybird team is reading.
Appending data with reordered columns
Appending data with reordered columns

The new data is appended to the movies Data Source.

Append from the API

From the API, to append data into the existing Data Source movies, you specify the mode and the name. The order of the columns doesn’t matter, provided the CSV header contains the column names.

Append from the CLI

From the CLI, append the new data to the movies Data Source without worrying about the column order, provided the CSV header contains the column names.

Login to your account to check this out and tell us what you think in our Slack.

Do you like this post?

Related posts

A big performance boost, adding columns and more

Tinybird

Team

May 31, 2021
Changelog #18: High-frequency ingestion, handling NDJSON files and more product enhancements

Tinybird

Team

Feb 01, 2022
ClickHouse tips #10: Null behavior with LowCardinality columns
The era of JSON data analytics
Improved Support for Replacing or Appending Data
Changelog #20: Data Source descriptions and beta testing of Parquet ingestion

Tinybird

Team

Apr 13, 2022
Tiny Updates: Drag and Drop to Rearrange Nodes in a Pipe
Querying large CSVs online with SQL
Changelog: BigQuery connector, CLI improvements, Pagination on endpoints and more

Tinybird

Team

Mar 04, 2021
Iterating terabyte-sized ClickHouse®️ tables in production

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.