Process and copy data¶

Tinybird provides several ways to process and copy data between Data Sources:

• Copy Pipes allow you to capture the result of a Pipe at a specific point in time and write it to a target Data Source. They can run on a schedule or be executed on demand, making them ideal for event-sourced snapshots, data experimentation, and deduplication with snapshots.

• Materialized Views continuously re-evaluate a query as new events are inserted, maintaining an always up-to-date derived dataset. Unlike Copy Pipes which create point-in-time snapshots, Materialized Views provide real-time transformations of your data.

• Populate operations let you copy data between Data Sources by processing it partition by partition. This approach provides progress tracking and allows retrying individual steps, making it safer and more reliable for large data transfers.

Each approach has its own strengths and use cases:

• Use Copy Pipes when you need scheduled or on-demand snapshots of your data
• Use Materialized Views when you need continuous, real-time transformations
• Use Populate operations when you need controlled, partition-by-partition data transfers with progress tracking

The following sections detail how to implement each of these approaches effectively, including best practices and important considerations for maintaining data integrity.