Jobs API¶

Certain operations can start a job in Tinybird:

Import data via URL.
Populate a Data Source from a materialized Node.
Run background queries.

When any of these operations start, the response contains a job_id field.

With the Jobs API, you can list the jobs for the last 48 hours or the last 100 jobs.

You can also get the details for a job. Depending on the job kind (import, query, or populate), it will return certain information related with the specific job, along with the status of the job.

The Job status can be one of the following:

waiting: The initial status of a job. When creating a job, it has to wait if there're other jobs running.
working: Once the job operation has started.
done: The job has finished successfully.
error: The job has finished with an error.

The created_at field reflects when the job was created, while updated_at shows when the job changed its state. Once the status of the job is done or error the difference between created_at and updated_at is the total execution time, including the time spent on the job queue.

The statistics fields shows: - row_count: For import and populate jobs, it shows the number of rows of the Data Source. In case of a query job, it shows the number of rows returned by the query. - bytes: For import and populate jobs, it shows the amount of bytes of the target Data Source. In case of a query job, the size, in bytes, returned by the query.

The quarantine_rows field shows the number of rows in the quarantine Data Source. Those that do not match the schema but can still be parsed, while the invalid_lines field shows the number of rows that cannot be properly parsed.

GET /v0/jobs/?¶

We can get a list of the last 100 jobs in the last 48 hours, with the possibility of filtering them by kind, status, pipe_id, pipe_name, created_after, and created_before.

Request parameters¶
Key	Type	Description
kind	String	This will return only the jobs with that particular kind. Example: `kind=populateview` or `kind=copy` or `kind=import`
status	String	This will return only the jobs with the status provided. Example: `status=done` or `status=waiting` or `status=working` or `status=error`
pipe_id	String	This will return only the jobs associated with the provided pipe id. Example: `pipe_id=t_31a0ff508c9843b59c32f7f81a156968`
pipe_name	String	This will return only the jobs associated with the provided pipe name. Example: `pipe_name=test_pipe`
created_after	String	This will return jobs that were created after the provided date in the ISO 8601 standard date format. Example: `created_after=2023-06-15T18:13:25.855Z`
created_before	String	This will return jobs that were created before the provided date in the ISO 8601 standard date format. Example: `created_before=2023-06-19T18:13:25.855Z`

Getting the latest jobs¶

curl \
-H "Authorization: Bearer <token>" \
"https://api.tinybird.co/v0/jobs" \

You will get a list of jobs with the kind, status, id, and the url to access the specific information about that job.

Jobs list¶

{
    "jobs": [
        {
            "id": "c8ae13ef-e739-40b6-8bd5-b1e07c8671c2",
            "kind": "import",
            "status": "done",
            "created_at": "2020-12-04 15:08:33.214377",
            "updated_at": "2020-12-04 15:08:33.396286",
            "job_url": "https://api.tinybird.co/v0/jobs/c8ae13ef-e739-40b6-8bd5-b1e07c8671c2",
            "datasource": {
                "id": "t_31a0ff508c9843b59c32f7f81a156968",
                "name": "my_datasource_1"
            }
        },
        {
            "id": "1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb",
            "kind": "import",
            "status": "error",
            "created_at": "2020-12-04 15:08:09.051310",
            "updated_at": "2020-12-04 15:08:09.263055",
            "job_url": "https://api.tinybird.co/v0/jobs/1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb",
            "datasource": {
                "id": "t_49806938714f4b72a225599cdee6d3ab",
                "name": "my_datasource_2"
            }
        }
    ]
}

Job details in job_url will be available for 48h after its creation.

POST /v0/jobs/(.+)/cancel¶

With this endpoint you can try to cancel an existing Job. All jobs can be cancelled if they are in the “waiting” status, but you can’t cancel a Job in “done” or “error” status.

In the case of the job of type “populate”, you can cancel it in the “working” state.

After successfully starting the cancellation process, you will see two different status in the job: - “cancelling”: The Job can’t be immediately cancelled as it’s doing some work, but the cancellation will eventually happen. - “cancelled”: The Job has been completely cancelled.

A Job cancellation doesn’t guarantee a complete rollback of the changes being made by it, sometimes you will need to delete new inserted rows or datasources created.

The fastest way to know if a job is cancellable, is just reading the “is_cancellable” key inside the job JSON description.

Depending on the Job and its status, when you try to cancel it you may get different responses: - HTTP Code 200: The Job has successfully started the cancellation process. Remember that if the Job has now a “cancelling” status, it may need some time to completely cancel itself. This request will return the status of the job. - HTTP Code 404: Job not found. - HTTP Code 403: The token provided doesn’t have access to this Job. - HTTP Code 400: Job is not in a cancellable status or you are trying to cancel a job which is already in the “cancelling” state.

Try to cancel a Job¶

curl \
-H "Authorization: Bearer <token>" \
-X POST "https://api.tinybird.co/v0/jobs/:job_id/cancel"

Populate Job in cancelling state right after the cancellation request.¶

{
    "kind": "populateview",
    "id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c",
    "job_id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c",
    "status": "cancelling",
    "created_at": "2021-03-17 18:56:23.939380",
    "updated_at": "2021-03-17 18:56:44.343245",
    "is_cancellable": false,
    "datasource": {
        "id": "t_02043945875b4070ae975f3812444b76",
        "name": "your_datasource_name",
        "cluster": null,
        "tags": {},
        "created_at": "2020-07-15 10:55:12.427269",
        "updated_at": "2020-07-15 10:55:12.427270",
        "statistics": null,
        "replicated": false,
        "version": 0,
        "project": null,
        "used_by": []
    },
    "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2",
    "pipe_id": "t_7fa8009023a245b696b4f2f7195b23c3",
    "pipe_name": "top_product_per_day",
    "queries": [
        {
            "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2",
            "status": "done"
        },
        {
            "query_id": "01HSZ9WY6QS6XAMBHZMSNB1G75",
            "status": "done"
        },
        {
            "query_id": "01HSZ9X8YVEQ0PXA6T2HZQFFPX",
            "status": "working"
        },
        {
            "query_id": "01HSZQ5YX0517X81JBF9G9HB2P",
            "status": "waiting"
        },
        {
            "query_id": "01HSZQ6PZJA3P81RC6Q6EF6HMK",
            "status": "waiting"
        },
        {
            "query_id": "01HSZQ76D7YYFB16TFT32KXMCY",
            "status": "waiting"
        }
    ],
    "progress_percentage": 50.0
}

GET /v0/jobs/(.+)¶

Get the details of a specific Job. You can get the details of a Job by using its ID.

Get the details of a Job¶

curl \
-H "Authorization: Bearer <token>" \
"https://api.tinybird.co/v0/jobs/:job_id"

You will get a JSON response with the details of the Job, including the kind, status, id, created_at, updated_at, and the datasource associated with the Job. This is available for 48h after the Job creation. After that, you can consult the Job details in the Service Data Source jobs_log.

Job details¶

{
"kind": "import",
"id": "d5b869ed-3a74-45f9-af54-57350aae4cef",
"job_id": "d5b869ed-3a74-45f9-af54-57350aae4cef",
"status": "done",
"created_at": "2024-07-22 11:47:58.207606",
"updated_at": "2024-07-22 11:48:52.971327",
"started_at": "2024-07-22 11:47:58.351734",
"is_cancellable": false,
"mode": "append",
"datasource": {
    "id": "t_caf95c54174e48f488ea65d181eb5b75",
    "name": "events",
    "cluster": "default",
    "tags": {

    },
    "created_at": "2024-07-22 11:47:51.807384",
    "updated_at": "2024-07-22 11:48:52.726243",
    "replicated": true,
    "version": 0,
    "project": null,
    "headers": {
    "cached_delimiter": ","
    },
    "shared_with": [

    ],
    "engine": {
    "engine": "MergeTree",
    "partition_key": "toYear(date)",
    "sorting_key": "date, user_id, event, extra_data"
    },
    "description": "",
    "used_by": [

    ],
    "last_commit": {
    "content_sha": "",
    "status": "changed",
    "path": ""
    },
    "errors_discarded_at": null,
    "type": "csv"
},
"import_id": "d5b869ed-3a74-45f9-af54-57350aae4cef",
"url": "https://storage.googleapis.com/tinybird-assets/datasets/guides/events_50M_1.csv",
"statistics": {
    "bytes": 1592720209,
    "row_count": 50000000
},
"quarantine_rows": 0,
"invalid_lines": 0
}