Scheduled Copy Pipes can now run on on-demand compute. Add ON_DEMAND_COMPUTE true to a Copy Pipe datafile and each scheduled run uses temporary, dedicated ClickHouse® compute instead of the primary compute in your Tinybird workspace.
Here is a Copy Pipe that builds an hourly rollup every night:
NODE hourly_api_usage
SQL >
SELECT
toStartOfHour(timestamp) AS hour,
workspace_id,
endpoint,
count() AS requests,
quantile(0.95)(response_time_ms) AS response_time_p95
FROM api_events
WHERE timestamp >= now() - INTERVAL 1 DAY
GROUP BY hour, workspace_id, endpoint
TYPE COPY
TARGET_DATASOURCE api_usage_hourly
COPY_SCHEDULE 0 2 * * *
COPY_MODE append
ON_DEMAND_COMPUTE true
The configuration lives with the query, target, mode, and schedule. Deploy the datafile as usual, and Tinybird uses on-demand compute when the schedule triggers the Copy Pipe.
Isolate recurring copy work from primary compute
Copy Pipes run a query and write its result to a target Data Source. They are useful when you need to periodically reshape a large dataset rather than calculate the same expensive result for every request.
That work can be substantial. Common examples include:
- Creating snapshots from change data capture (CDC) streams, keeping the latest version of each record.
- Deduplicating event data before downstream queries consume it.
- Calculating hourly or daily aggregates for dashboards and APIs.
Without separate compute, a large scheduled copy query uses the same CPU and memory as other work in the workspace. Choosing a quiet hour can help, but it does not isolate the job. Data volumes, job duration, and application traffic all change over time.
With ON_DEMAND_COMPUTE true, Tinybird provisions temporary dedicated compute for the scheduled job and removes it afterward. The copy query is separated from primary workspace compute while it runs. This is useful for recurring work that is resource-intensive or whose timing overlaps with production traffic.
Scheduled and manual runs use different controls
The datafile property controls scheduled executions:
ON_DEMAND_COMPUTE true
It is an explicit opt-in. Copy Pipes without the property continue to use their existing execution behavior.
For a manual run in Forward, pass the CLI flag:
tb copy run my_copy_pipe --on-demand-compute
For a manual run in Classic, use:
tb pipe copy run my_copy_pipe --on-demand-compute
The datafile setting does not replace the flag for a manual run. Update your Tinybird CLI before deploying the datafile or triggering a manual job so it recognizes the current Copy Pipe syntax and options.
Both append and replace Copy Pipe modes can use on-demand compute. Choose the mode based on the result you need. append adds each result to the target, which suits incremental rollups. replace replaces the target contents, which suits a current-state snapshot built from a CDC stream.
A batch alternative to Materialized Views
Materialized Views are usually the first choice for preparing data as it arrives. They process each inserted block incrementally. Copy Pipes are the batch alternative when a transformation needs to see a broader slice of the data, such as a deduplicated snapshot, a window calculation, or an aggregation over a ReplacingMergeTree Data Source.
That distinction also makes Copy Pipes useful in a lambda architecture. A scheduled Copy Pipe periodically builds the batch view from historical data. An Endpoint can combine that snapshot with rows ingested since the last copy, keeping results fresh between runs. On-demand compute isolates the heavier batch step without changing the real-time path.
Where on-demand compute fits
On-demand compute is most useful when the cost of a copy query is large enough to affect other workloads or make capacity planning awkward.
Consider a CDC stream containing multiple versions of every customer record. A nightly Copy Pipe can group or order those versions to produce a deduplicated current-state Data Source. As the history grows, the snapshot query must scan and process more rows. Running it on temporary dedicated compute keeps that recurring batch workload separate from primary workspace resources.
Periodic aggregations have a similar profile. An hourly or daily job might scan raw events, group by customer and endpoint, and calculate counts or latency quantiles. The resulting Data Source makes downstream API queries cheaper, but building it can require much more CPU and memory than reading it. On-demand compute separates those two workloads.
It is also useful when several scheduled copies overlap. Moving the heaviest job to dedicated compute can reduce pressure on the workspace without changing its SQL or target schema.
Cost and provisioning tradeoffs
On-demand compute is available on paid plans. The default on-demand instance has 64 cores. Billing covers the actual compute time, charged in credits per core per minute. The credit price varies by region, so use the current pricing documentation when estimating a job rather than relying on a fixed example.
There is provisioning overhead before the query starts. For a large deduplication pass or aggregation, that overhead may be a reasonable trade for dedicated resources. A small Copy Pipe that finishes quickly on primary compute may cost less and complete sooner without provisioning another instance.
Measure representative runs before enabling the setting across every schedule. Look at job duration, frequency, overlap with other workloads, and the effect the current job has on primary compute. The goal is to isolate the copy jobs that warrant it, not to move every scheduled query by default.
Get started
Add ON_DEMAND_COMPUTE true to a scheduled Copy Pipe that benefits from dedicated resources, then deploy it with an up-to-date CLI. Keep using the corresponding --on-demand-compute flag when you trigger a manual run.
Read the Copy Pipe documentation for the full workflow. See on-demand CPU pricing for regional credit rates before enabling it on a recurring schedule.