Quarantine data sources

Every data source in your workspace has an associated quarantine data source that stores data that doesn't fit the schema. If you send rows that don't match the data source schema, they're automatically sent to the quarantine table so that the ingest process doesn't fail.

By convention, quarantine data sources follow the naming pattern {datasource_name}_quarantine. You can review quarantined rows at any time or perform operations on them using Pipes. This is a useful source of information when fixing issues in the origin source or applying changes during ingest.

Review quarantined data

To check your quarantine data sources, run the tb sql command. For example:

tb sql "select * from <datasource_name>_quarantine limit 10"

A sample output of the tb sql command is the following:

──────────────────────────────────────────────────────────────────
c__error_column: ['abslevel']
c__error: ["value '' on column 'abslevel' is not Float32"]
c__import_id: 01JKQPWT8GVXAN5GJ1VBD4XM27
day: 2014-07-30
station: Embassament de Siurana (Cornudella de Montsant)
volume: 11.57
insertion_date: 2025-02-10 10:36:20
──────────────────────────────────────────────────────────────────

The quarantine data source schema contains the columns of the original row and the following columns with information about the issues that caused the quarantine:

  • c__error_column Array(String) contains an array of all the columns that contain an invalid value.
  • c__error Array(String) contains an array of all the errors that caused the ingestion to fail and lead to store the values in quarantine. This column along the c__error_column allows you to easily identify which is the columns that has problems and which is the error.
  • c__import_id Nullable(String) contains the job's identifier in case the column was imported through a job.
  • insertion_date (DateTime) contains the timestamp in which the ingestion was done.
Updated