# Tinybird Documentation Generated on: 2025-02-18T16:02:48.971Z URL: https://www.tinybird.co/docs/work-with-data Content: --- title: "Work with data · Tinybird Docs" theme-color: "#171612" description: "Learn how to work with data in Tinybird." --- # Work with data [¶](https://www.tinybird.co/docs/about:blank#work-with-data) Tinybird provides comprehensive tools and strategies for working with your data throughout its lifecycle - from querying and processing to organizing and deploying data projects. Here's what you can do: ## Query your data [¶](https://www.tinybird.co/docs/about:blank#query-your-data) Explore and manipulate your ingested data through powerful tools like Data Flow visualization, SQL Playgrounds for testing queries, and Time Series analysis. Use query parameters and follow SQL best practices to build efficient data pipelines. ## Process and copy data [¶](https://www.tinybird.co/docs/about:blank#process-and-copy-data) Transform and move your data using Copy Pipes for scheduled snapshots, Materialized Views for real-time transformations, and Populate operations for controlled partition-by-partition transfers. Each approach serves different use cases while maintaining data integrity. ## Organize your work [¶](https://www.tinybird.co/docs/about:blank#organize-your-work) Leverage Git integration and version control to manage your data projects like software. Use branches for isolated environments, implement CI/CD pipelines, and follow established development practices. This allows teams to enforce standardized processes for code reviews, testing, and deployment. ## Implement strategies [¶](https://www.tinybird.co/docs/about:blank#implement-strategies) Develop clear approaches for testing data pipelines, deploying changes safely, managing backfills and migrations, and integrating external data. These proven strategies help ensure data quality and smooth operations when working with high-volume, real-time data processing. --- URL: https://www.tinybird.co/docs/use-cases Last update: 2024-10-03T12:47:27.000Z Content: --- title: "Use Cases · Tinybird Docs" theme-color: "#171612" description: "Build a user-facing analytics project with these end-to-end tutorials, videos, and blog posts." --- # Use Case Hub [¶](https://www.tinybird.co/docs/about:blank#use-case-hub) Tinybird is the data platform for building real-time, user-facing analytics. This Use Case Hub provides centralized resources to help you understand and build specific solutions for your project. ## Examples [¶](https://www.tinybird.co/docs/about:blank#examples) The top 5 most popular ways people are using Tinybird to build user-facing analytics features: - Want to provide a real-time, beautiful dashboard for your users? -->[ User-facing dashboards](https://www.tinybird.co/docs/docs/use-cases/user-facing-dashboards) . - Need to build a change data capture (CDC) system for your database? -->[ Real-time Change Data Capture (CDC)](https://www.tinybird.co/docs/docs/use-cases/realtime-cdc) . - Looking to level up your game with leaderboards, match-making, and personalized in-game ads? -->[ Gaming analytics](https://www.tinybird.co/docs/docs/use-cases/gaming-analytics) . - Want to build a way to monitor and analyze your web traffic? -->[ Web analytics](https://www.tinybird.co/docs/docs/use-cases/web-analytics) . - Already know that the secret to amazing UX in your app is personalization? -->[ Real-time personalization](https://www.tinybird.co/docs/docs/use-cases/realtime-personalization) . - Need a way for your content creators to analyze their content performance? -->[ User-generated content (UGC) analytics](https://www.tinybird.co/docs/docs/use-cases/ugc-analytics) . - Need to build a content recommendation system? -->[ Content recommendation systems](https://www.tinybird.co/docs/docs/use-cases/content-recommendation) . - Want to learn how to build a vector search system using real-time data? -->[ Vector search](https://www.tinybird.co/docs/docs/use-cases/vector-search) . Explore the nav for more. ## Not sure where to start? [¶](https://www.tinybird.co/docs/about:blank#not-sure-where-to-start) Read the [Tinybird "Definitive Guide To Real-Time User-Facing Analytics" blog post](https://www.tinybird.co/blog-posts/user-facing-analytics). ## Customer stories [¶](https://www.tinybird.co/docs/about:blank#customer-stories) Learn how Tinybird customers have built user-facing analytics. - [ ](https://www.tinybird.co/case-studies/audiense) Audiense evolves its social media timeline with Tinybird - [ ](https://www.tinybird.co/case-studies/canva) Canva designs brilliant user experiences with Tinybird - [ ](https://www.tinybird.co/case-studies/dub) Dub shortens time to market for user-facing analytics with Tinybird - [ ](https://www.tinybird.co/case-studies/factorial-builds-real-time-data-products-with-tinybird) Factorial builds real-time data products with Tinybird - [ ](https://www.tinybird.co/case-studies/fanduel) FanDuel harnesses the power of real-time data using Tinybird - [ ](https://www.tinybird.co/case-studies/genially-uses-tinybird-to-display-realtime-metrics) Genially uses Tinybird to display real-time content interaction metrics for their customers - [ ](https://www.tinybird.co/case-studies/smartme-analytics) Smartme Analytics builds complete consumer insights with Tinybird - [ ](https://www.tinybird.co/case-studies/vercel-relies-on-tinybird-to-power-their-realtime-user-facing-analytics) Vercel uses Tinybird to help developers ship code faster --- URL: https://www.tinybird.co/docs/sql-reference Last update: 2025-01-08T13:04:07.000Z Content: --- title: "SQL reference · Tinybird Docs" theme-color: "#171612" description: "SQL reference for Tinybird" --- # SQL reference [¶](https://www.tinybird.co/docs/about:blank#sql-reference) Tinybird supports the following statements, data types, and functions in queries. ## SQL statements [¶](https://www.tinybird.co/docs/about:blank#sql-statements) The only statement you can use in Tinybird's queries is `SELECT` . The SQL clauses for `SELECT` are fully supported. All other SQL statements are handled by Tinybird's features. ## Data types [¶](https://www.tinybird.co/docs/about:blank#data-types) Tinybird supports a variety of data types to store and process different kinds of information efficiently. Data types define the kind of values that can be stored in a column and determine how those values can be used in queries and operations. See [Data types](https://www.tinybird.co/docs/docs/sql-reference/data-types). The following data types are supported at ingest: - `Int8` , `Int16` , `Int32` , `Int64` , `Int128` , `Int256` - `UInt8` , `UInt16` , `UInt32` , `UInt64` , `UInt128` , `UInt256` - `Float32` , `Float64` - `Decimal` , `Decimal(P, S)` , `Decimal32(S)` , `Decimal64(S)` , `Decimal128(S)` , `Decimal256(S)` - `String` - `FixedString(N)` - `UUID` - `Date` , `Date32` - `DateTime([TZ])` , `DateTime64(P, [TZ])` - `Bool` - `Array(T)` - `Map(K, V)` - `LowCardinality` - `Nullable` - `JSON` If you are ingesting using the NDJSON format and would like to store `Decimal` values containing 15 or more digits, send the values as string instead of numbers to avoid precision issues. In the following example, the first value has a high chance of losing accuracy during ingestion, while the second one is stored correctly: {"decimal_value": 1234567890.123456789} # Last digits might change during ingestion {"decimal_value": "1234567890.123456789"} # Will be stored correctly The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). ## Table engines [¶](https://www.tinybird.co/docs/about:blank#table-engines) Table engines are a crucial component of Tinybird's data sources, defining how data is stored, indexed, and accessed. Each table engine is optimized for specific use cases, such as handling large volumes of data, providing high-speed read and write operations, or supporting complex queries and transactions. Tinybird supports a variety of table engines, including: - [ MergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/mergetree) : A general-purpose engine for storing and querying large datasets. - [ AggregatingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/aggregatingmergetree) : Suitable for aggregating data and reducing storage volume. - [ ReplacingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/replacingmergetree) : Ideal for deduplicating rows and removing duplicate entries. - [ SummingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/summingmergetree) : Optimized for summarizing rows and reducing storage volume. - [ CollapsingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/collapsingmergetree) : Designed for collapsing rows and deleting old object states in the background. - [ VersionedCollapsingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/versionedcollapsingmergetree) : Allows for collapsing rows and deleting old object states in the background, with support for versioning. - [ Null](https://www.tinybird.co/docs/docs/sql-reference/engines/null) : A special engine for not storing values. Choosing the right table engine for your data source is essential for optimal performance, data integrity, and query efficiency. ## Functions [¶](https://www.tinybird.co/docs/about:blank#functions) Tinybird provides a comprehensive set of built-in functions to help you transform and analyze your data effectively. These functions can be broadly categorized into: - Aggregate functions: Perform calculations across rows and return a single value, like `count()` , `sum()` , `avg()` . - String functions: Manipulate and analyze text data with operations like substring, concatenation, pattern matching. - Date and time functions: Work with temporal data through date arithmetic, formatting, and time window operations. - Mathematical functions: Handle numerical computations and transformations. - Type conversion functions: Convert between different data types safely. - Array functions: Operate on array columns with filtering, mapping, and reduction operations. - Conditional functions: Implement if-then-else logic and case statements. - Window functions: Perform calculations across a set of rows related to the current row. See [Functions](https://www.tinybird.co/docs/docs/sql-reference/functions). ### Private beta [¶](https://www.tinybird.co/docs/about:blank#private-beta) Tinybird supports the following table functions upon request: - `mysql` - `url` ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) Tinybird supports the following settings: - `aggregate_functions_null_for_empty` - `join_use_nulls` - `group_by_use_nulls` - `join_algorithm` - `date_time_output_format` You can use the settings by adding `SETTINGS=` to the final node of your Pipe. For example: SELECT id, country_id, name as country_name FROM events e LEFT JOIN country c ON e.country_id = c.id SETTINGS join_use_nulls = 1 --- URL: https://www.tinybird.co/docs/publish Content: --- title: "Publish data · Tinybird Docs" theme-color: "#171612" description: "Overview of publishing data using Tinybird" --- # Publish your data [¶](https://www.tinybird.co/docs/about:blank#publish-your-data) Whatever you need from your data, you can achieve it using Tinybird. Publish it as a queryable [API Endpoint](https://www.tinybird.co/docs/docs/publish/api-endpoints) or a [Sink Pipe](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink#sink-pipes) ). --- URL: https://www.tinybird.co/docs/monitoring Content: --- title: "Monitoring · Tinybird Docs" theme-color: "#171612" description: "Learn how to monitor your Tinybird data platform." --- # Monitoring [¶](https://www.tinybird.co/docs/about:blank#monitoring) Tinybird is built around the idea of data that changes or grows continuously. Use the built-in Tinybird tools to monitor your data ingestion and API Endpoint processes. - [ Analyze endpoint performance](https://www.tinybird.co/docs/docs/monitoring/analyze-endpoints-performance) - [ Health checks](https://www.tinybird.co/docs/docs/monitoring/health-checks) - [ Measure endpoint latency](https://www.tinybird.co/docs/docs/monitoring/latency) - [ Monitor ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) - [ Monitor Workspace jobs](https://www.tinybird.co/docs/docs/monitoring/jobs) - [ Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) --- URL: https://www.tinybird.co/docs/index Last update: 2024-12-30T16:52:42.000Z Content: --- title: "Overview of Tinybird · Tinybird Docs" theme-color: "#171612" description: "Tinybird is data infrastructure for software teams. Ship analytics features in days, not months. Tinybird abstracts data ingestion, storage, compute, and API development into a single workflow. Fewer dependencies means faster development." --- # Welcome to Tinybird [¶](https://www.tinybird.co/docs/about:blank#welcome-to-tinybird) Tinybird is data infrastructure for software teams, giving you the tooling and infra you need to ship analytics features in your application while minimizing external dependencies. Use Tinybird to capture, store, query, and serve analytics data in your application. ## Create an account [¶](https://www.tinybird.co/docs/about:blank#create-an-account) Tinybird has a time-unlimited free tier with limits generous enough for most proofs of concept (and some production apps!), so you can start building today and scale at your own pace. [Create a free account](https://www.tinybird.co/signup) using a Google, Microsoft, or GitHub account (or your email address). After you create your account, pick the cloud region that works best for you, then create a [Workspace](https://www.tinybird.co/docs/docs/get-started/administration/workspaces). ## Try out your Workspace [¶](https://www.tinybird.co/docs/about:blank#try-out-your-workspace) Follow the [Quick start](https://www.tinybird.co/docs/docs/get-started/quick-start) to ship your first analytics in a few minutes. Learn the basics of ingesting data into Tinybird, writing SQL, publishing APIs, and integrating them in your application, from empty Workspace to working prototype. ## Use a template [¶](https://www.tinybird.co/docs/about:blank#use-a-template) Use one of the existing templates to get started quickly. See the [Templates](https://www.tinybird.co/templates) page for more information. ## Watch the videos [¶](https://www.tinybird.co/docs/about:blank#watch-the-videos) Watch the following videos to get familiar with Tinybird's user interface and CLI. - [ The Tinybird basics in 3 minutes (UI)](https://www.youtube.com/watch?v=cvay_LW685w) - [ Get started with the CLI](https://www.youtube.com/watch?v=OOEe84ly7Cs) - [ Ingest data from a file (UI vs CLI)](https://www.youtube.com/watch?v=1R0G1EolSEM) ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Browse the[ Use Case Hub](https://www.tinybird.co/docs/docs/use-cases) and see how to boost your project with real-time, user-facing analytics. - Learn the Tinybird[ core concepts](https://www.tinybird.co/docs/docs/get-started/quick-start/core-concepts) . - Start building using the[ quick start](https://www.tinybird.co/docs/docs/get-started/quick-start) . - Read how to build a[ user-facing web analytics dashboard](https://www.tinybird.co/docs/docs/get-started/quick-start/user-facing-web-analytics) . --- URL: https://www.tinybird.co/docs/get-started Content: --- title: "Get started · Tinybird Docs" theme-color: "#171612" description: "Get started with Tinybird, the data platform for building real-time data applications." --- # Get started [¶](https://www.tinybird.co/docs/about:blank#get-started) The following resources will help you get started with Tinybird: - Follow the Quick start guide. See[ Quick start](https://www.tinybird.co/docs/docs/get-started/quick-start) . - Learn about the different plans and pricing options. See[ Plans](https://www.tinybird.co/docs/docs/get-started/plans) . - Discover all the integrations available in Tinybird. See[ Integrations](https://www.tinybird.co/docs/docs/get-started/integrations) . - Familiarize yourself with how organizations and workspaces work in Tinybird. See[ Administration](https://www.tinybird.co/docs/docs/get-started/administration) . --- URL: https://www.tinybird.co/docs/get-data-in Content: --- title: "Get data in · Tinybird Docs" theme-color: "#171612" description: "Tinybird can get your data in from a variety of sources, then create Data Sources that can be queried, published, materialized, and more." --- # Get your data into Tinybird [¶](https://www.tinybird.co/docs/about:blank#get-your-data-into-tinybird) You can bring your data into Tinybird from a variety of sources, then create Tinybird Data Sources that you can query, publish, materialize, and more. To get started, you only need to select your Data Source and get connected. ## Create your schema [¶](https://www.tinybird.co/docs/about:blank#create-your-schema) You define schemas using a specific file type: .datasource files. See [.datasource](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files). You can either define the schema first or send data directly and let Tinybird infer the schema. ### Define the schema yourself [¶](https://www.tinybird.co/docs/about:blank#define-the-schema-yourself) If you want to define your schema first, use one of the following methods: #### Option 1: Create an empty Data Source in the UI [¶](https://www.tinybird.co/docs/about:blank#option-1-create-an-empty-data-source-in-the-ui) Follow these steps to create an empty Data Source: 1. Select the** +** icon then "Data Source". 2. Select the** Write schema** option. 3. Review the .datasource file that appears. Update the generated parameters to match your desired schema. 4. Select** Next** . 5. Review your column names and types and rename the Data Source. 6. Select** Create Data Source** . Make sure you understand the definition and syntax for the schema, JSONPaths, engine, partition key, sorting keys, and TTL. They are all essential for efficient data operations later down the line. #### Option 2: Upload a .datasource file with the desired schema [¶](https://www.tinybird.co/docs/about:blank#option-2-upload-a-datasource-file-with-the-desired-schema) You can define your schema locally in a .datasource file. Drag the .datasource file to the UI and import the resource. You can also select the **+** icon followed by **Write schema** and drag the file onto the dialog. #### Other options [¶](https://www.tinybird.co/docs/about:blank#other-options) Among other options, you can create a data source in the following ways: - Use the[ Data Sources API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/datasource-api) . - Use the[ Tinybird CLI](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files) . ### Send data and use inferred schema [¶](https://www.tinybird.co/docs/about:blank#send-data-and-use-inferred-schema) If you want to send data directly and let Tinybird infer the schema, use one of the Connectors, upload a CSV, NDJSON, or Parquet file, or send data to the [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api#send-individual-json-events). To use the Events API to infer the schema, send a single event over the API and let Tinybird create the schema automatically. If the schema is incorrect, go to the **Data Source schema** screen and download the schema as a .datasource file. Edit the file to adjust the schema if required, then drag the file back. You can't override an existing Data Source. Delete the existing one or rename the new Data Source. ## Supported data types, file types, and compression formats [¶](https://www.tinybird.co/docs/about:blank#supported-data-types-file-types-and-compression-formats) See [Concepts > Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources#supported-data-types) for more information on supported types and formats. ## Update your schema [¶](https://www.tinybird.co/docs/about:blank#update-your-schema) If your data doesn't match the schema, it ends up in the quarantine Data Source. See [Quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine). You might also want to change your schema for optimization purposes. See [Iterate a Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source). ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. --- URL: https://www.tinybird.co/docs/cli Last update: 2024-12-18T11:12:31.000Z Content: --- title: "Tinybird CLI · Tinybird Docs" theme-color: "#171612" description: "Use the Tinybird CLI to access all the Tinybird features from the command line." --- # Tinybird command-line interface (CLI) [¶](https://www.tinybird.co/docs/about:blank#tinybird-command-line-interface-cli) Use the Tinybird CLI to access all the Tinybird features directly from the command line. You can test and run commands from the terminal or integrate the CLI in your pipelines. - Read the Quick start guide. See[ Quick start](https://www.tinybird.co/docs/docs/cli/quick-start) . - Install Tinybird CLI on your machine. See[ Install](https://www.tinybird.co/docs/docs/cli/install) . - Learn about Tinybird datafiles and their format. See[ Datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) . - Organize your CLI projects. See[ Data projects](https://www.tinybird.co/docs/docs/cli/data-projects) . --- URL: https://www.tinybird.co/docs/changelog Content: --- title: "Changelog · Tinybird" theme-color: "#171612" description: "Tinybird helps data teams build real-time Data Products at scale through SQL-based API endpoints." --- # We're in love with Tinybird Forward! [¶](https://www.tinybird.co/docs/about:blank#were-in-love-with-tinybird-forward) We’ve designed a Tinybird CLI to make managing state changes in your data infrastructure effortless. No more manual steps or error-prone deployments: just make your changes locally and run `tb deploy` . Think of it as a compiler, a VM, and a deployment tool all in one. This new local-first approach simplifies your workflow, letting you focus on shipping faster. We can't wait for you to try it out in a new workspace. Try the beta now! - [ Read the blog post](https://www.tinybird.co/blog-posts/tinybird-forward) - [ Get started with the docs](https://www.tinybird.co/docs/docs/v2) ## New job overview [¶](https://www.tinybird.co/docs/about:blank#new-job-overview) You can now see your complete job history in the **Dashboard overview** . Jobs run various operations in the background, some of them periodically, such as copy jobs, writing data to a sink. Some of them on demand, like populate operations. Previously, the dashboard visibility was limited to the last 48 hours. With this update, you have a more comprehensive view of your job activity. You can now access job history for the past 30 days and filter the list by both job status and type. ![Video]() Controls: true With this update, you can now: - Identify performance bottlenecks by sorting jobs by duration to spot which ones take the longest. - Track issues more effectively by easily checking if any jobs encountered problems in recent days - Monitor queue status to see if jobs are waiting to start or if operations like your latest populate are still running In summary, we hope this new view helps you maintain a healthier data project by keeping you better informed about your job activity. ## Copy operations are now safer to retry [¶](https://www.tinybird.co/docs/about:blank#copy-operations-are-now-safer-to-retry) Copy operations transfer data between two data sources and might include downstream materialized views, with each step copying data separately. Previously, data was inserted into the destination immediately at each step, which could lead to inconsistency if later steps failed due to memory issues or timeouts. With this update, data is only inserted into destination data sources after all steps are successfully completed. This means: - If any step fails, no data is inserted anywhere. - Jobs can be safely retried without risk of data inconsistency. ## S3 Connector performance and usability improvements [¶](https://www.tinybird.co/docs/about:blank#s3-connector-performance-and-usability-improvements) We've made improvements to the S3 Connector under the hood. Now, new files upload to the bucket that match the specific file expression are detected through SQS events, instead of having to scan the whole bucket every few minutes. The initial backfill is also more efficient now, and you can expect much better throughput - remember you can always use the `tinybird.datasources_ops_log` Service Data Source to check the appended files, and the time they take. From now on, when you create a new S3 Connection the generated AWS IAM Policy includes these new permissions about bucket notifications: `s3:GetBucketNotification` and `s3:PutBucketNotification` . If you're trying to reuse an existing S3 Connection, you have two options: - You can add these permissions to the IAM role you have in AWS. - You can create a new S3 Connection from the UI We recommend the first option, and this is an example policy with the permission you need: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket", "s3:GetBucketNotification", "s3:PutBucketNotification", "s3:GetBucketLocation", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": [ "arn:aws:s3:::{bucket-name}", "arn:aws:s3:::{bucket-name}/*" ] } ] } Also notice that we removed the `s3:ListAllMyBuckets` permission, which concerning for some. ## Burst mode for QPS at no extra cost [¶](https://www.tinybird.co/docs/about:blank#burst-mode-for-qps-at-no-extra-cost) At Tinybird we learn and iterate fast based on your feedback. That's why we've introduced burst mode for queries per second (QPS) at no extra cost. New limits are also more generous now. Your operations can now take x2 QPS per second allowed in your plan for API endpoint requests or SQL queries. See [Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode) for details and examples. ## New doc on billing concepts [¶](https://www.tinybird.co/docs/about:blank#new-doc-on-billing-concepts) What are active minutes? And queries per second? How does burst mode work? To answer all these questions following the launch of new plans, we've added a new doc on [billing concepts](https://www.tinybird.co/docs/docs/get-started/plans/concepts). ## Improvements and bug fixes [¶](https://www.tinybird.co/docs/about:blank#improvements-and-bug-fixes) - Improved the way we estimate plan size when migrating or upgrading to a new plan. - Improved error messages when the cluster is under heavy load. --- URL: https://www.tinybird.co/docs/api-reference Last update: 2025-01-09T09:46:35.000Z Content: --- title: "API Overview · Tinybird Docs" theme-color: "#171612" description: "Tinybird's API Endpoints, such as the Data Sources API to import Data, the Pipes API to transform data and publish the results through API Endpoints, and the Query API to run arbitrary queries." --- # API Overview [¶](https://www.tinybird.co/docs/about:blank#api-overview) You can control Tinybird services using the API as an alternative to the UI and CLI. The following APIs are available: | API name | Description | | --- | --- | | [ Analyze API](https://www.tinybird.co/docs/docs/api-reference/analyze-api) | Analyze a given NDJSON, CSV, or Parquet file to generate a Tinybird Data Source schema. | | [ Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) | List, create, update, or delete your Tinybird Data Sources, and insert or delete data from Data Sources. | | [ Events API](https://www.tinybird.co/docs/docs/api-reference/events-api) | Ingest NDJSON events with a simple HTTP POST request. | | [ Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api) | Get details on Tinybird jobs, and list the jobs for the last 48 hours or the last 100 jobs. | | [ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) | Interact with your Pipes, including Pipes themselves, API Endpoints, Materialized Views, and managing Copy jobs. | | [ Query API](https://www.tinybird.co/docs/docs/api-reference/query-api) | Query your Pipes and Data Sources inside Tinybird as if you were running SQL statements against a regular database. | | [ Environment Variables API](https://www.tinybird.co/docs/docs/api-reference/environment-variables-api) | Create, update, delete, and list variables that can be used in Pipes in a Workspace. | | [ Sink Pipes API](https://www.tinybird.co/docs/docs/api-reference/sink-pipes-api) | Create, delete, schedule, and trigger Sink Pipes. | | [ Tokens API](https://www.tinybird.co/docs/docs/api-reference/token-api) | List, create, update, or delete your Tinybird Static Tokens. | Make all requests to Tinybird's API Endpoints over TLS (HTTPS). All response bodies, including errors, are encoded as JSON. You can get information on several Workspace operations by [monitoring your jobs](https://www.tinybird.co/docs/docs/monitoring/jobs) , using either the APIs or the built-in Tinybird Service Data Sources. ## Regions and endpoints [¶](https://www.tinybird.co/docs/about:blank#regions-and-endpoints) A Workspace belongs to one region. The API for each region has a specific API base URL that you use to make API requests. The following table lists the current regions and their corresponding API base URLs: ### Current Tinybird regions [¶](https://www.tinybird.co/docs/about:blank#current-tinybird-regions) | Region | Provider | Provider region | API base URL | | --- | --- | --- | --- | | Europe | GCP | europe-west3 | [ https://api.tinybird.co](https://api.tinybird.co/) | | US East | GCP | us-east4 | [ https://api.us-east.tinybird.co](https://api.us-east.tinybird.co/) | | Europe | AWS | eu-central-1 | [ https://api.eu-central-1.aws.tinybird.co](https://api.eu-central-1.aws.tinybird.co/) | | Europe | AWS | eu-west-1 | [ https://api.eu-west-1.aws.tinybird.co](https://api.eu-west-1.aws.tinybird.co/) | | US East | AWS | us-east-1 | [ https://api.us-east.aws.tinybird.co](https://api.us-east.aws.tinybird.co/) | | US West | AWS | us-west-2 | [ https://api.us-west-2.aws.tinybird.co](https://api.us-west-2.aws.tinybird.co/) | ## Authentication [¶](https://www.tinybird.co/docs/about:blank#authentication) Tinybird makes use of Tokens for every API call. This ensures that each user or application can only access data that they are authorized to access. See [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens). You must make all API requests over HTTPS. Don't make calls over plain HTTP or send API requests without authentication. Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace. There are two ways to authenticate your requests in the Tinybird API: using an authorization header, or using a URL parameter. ### Authorization header [¶](https://www.tinybird.co/docs/about:blank#authorization-header) You can send a Bearer authorization header to authenticate API calls. With cURL, use `-H "Authorization: Bearer "`. If you have a valid Token with read access to the particular Data Source, you can get a successful response by sending the following request: ##### Authorization header Authenticated request curl \ -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/sql?q=SELECT+*+FROM+" ### URL parameter [¶](https://www.tinybird.co/docs/about:blank#url-parameter) You can also specify the Token using a parameter in the URL, using `token=` . For example: ##### URL parameter authenticated request curl -X GET \ "https://api.tinybird.co/v0/sql?q=SELECT+*+FROM+&token=" ## Compression [¶](https://www.tinybird.co/docs/about:blank#compression) To compress API responses, add `Accept-Encoding: gzip` to your requests. For example: ##### Request with compressed response curl \ -X GET \ -H "Authorization: Bearer " \ -H "Accept-Encoding: gzip" \ "https://api.tinybird.co/v0/sql?q=SELECT+*+FROM+" ## Errors [¶](https://www.tinybird.co/docs/about:blank#errors) Tinybird 's API returns standard HTTP success or error status codes. In case of errors, responses include additional information in JSON format. The following table lists the error status codes. ### Response codes | Code | Description | | --- | --- | | 400 | Bad request. This could be due to a missing parameter in a request, for instance | | 403 | Forbidden. Provided auth token doesn't have the right scope or the Data Source isn't available | | 404 | Not found | | 405 | HTTP Method not allowed | | 408 | Request timeout (e.g. query execution time was exceeded) | | 409 | You need to resubmit the request due to a conflict with the current state of the target source (e.g.: you need to delete a Materialized View) | | 411 | No valid Content-Length header containing the length of the message-body | | 413 | The message body is too large | | 429 | Too many requests. When over the rate limits of your account | | 500 | Unexpected error | ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. ## Versioning [¶](https://www.tinybird.co/docs/about:blank#versioning) All Tinybird APIs are versioned with a version string specified in the base URL. Always use the latest API available. When versioning services, Tinybird adheres to [semantic versioning](https://semver.org/) rules. ## Reserved words [¶](https://www.tinybird.co/docs/about:blank#reserved-words) The following keywords are reserved. You can't use them to name Data Sources, Pipes, nodes or Workspaces. Case is ignored. - `Array` - `Boolean` - `Date` - `Date32` - `DateTime` - `DateTime32` - `DateTime64` - `Decimal` - `Decimal128` - `Decimal256` - `Decimal32` - `Decimal64` - `Enum` - `Enum16` - `Enum8` - `FixedString` - `Float32` - `Float64` - `IPv4` - `IPv6` - `Int128` - `Int16` - `Int256` - `Int32` - `Int64` - `Int8` - `MultiPolygon` - `Point` - `Polygon` - `Ring` - `String` - `TABLE` - `UInt128` - `UInt16` - `UInt256` - `UInt32` - `UInt64` - `UInt8` - `UUID` - `_temporary_and_external_tables` - `add` - `after` - `all` - `and` - `anti` - `any` - `array` - `as` - `asc` - `asof` - `between` - `by` - `case` - `collate` - `column` - `columns` - `cross` - `cube` - `custom_error` - `day_diff` - `default` - `defined` - `desc` - `distinct` - `else` - `end` - `enumerate_with_last` - `error` - `exists` - `from` - `full` - `functions` - `generateRandom` - `global` - `group` - `having` - `if` - `ilike` - `in` - `inner` - `insert` - `interval` - `into` - `join` - `left` - `like` - `limit` - `limits` - `max` - `min` - `not` - `null` - `numbers_mt` - `on` - `one` - `or` - `order` - `outer` - `prewhere` - `public` - `right` - `sample` - `select` - `semi` - `split_to_array` - `sql_and` - `sql_unescape` - `system` - `table` - `then` - `tinybird` - `to` - `union` - `using` - `where` - `with` - `zeros_mt` Pipe, Data Source and node names are globally unique. You can't use an alias for a column that matches a globally unique name. --- URL: https://www.tinybird.co/docs/work-with-data/strategies Content: --- title: "Strategies · Tinybird Docs" theme-color: "#171612" description: "Learn how to implement strategies for testing, organizing, and managing your data in Tinybird." --- # Strategies [¶](https://www.tinybird.co/docs/about:blank#strategies) When working with data in Tinybird, having clear strategies for testing, deployment, and data management is crucial for maintaining reliable and efficient data operations. This section covers key strategies for: - Testing your data pipelines and API endpoints - Deploying changes safely to production - Managing data backfills and migrations - Bringing external data into Tinybird These strategies help ensure data quality, maintain backward compatibility, and enable smooth transitions when making changes to your data infrastructure. They provide proven approaches developed from real-world experience working with high-volume, real-time data processing. --- URL: https://www.tinybird.co/docs/work-with-data/query Content: --- title: "Query overview · Tinybird Docs" theme-color: "#171612" description: "Explore and manipulate your data in Tinybird to make it more useful and relevant." --- # Query your ingested data [¶](https://www.tinybird.co/docs/about:blank#query-your-ingested-data) After you've brought your data to Tinybird, you can explore and manipulate it to make it more useful and relevant. ## Data Flow [¶](https://www.tinybird.co/docs/about:blank#data-flow) Data Flow visualizes how your Data Sources, API Endpoints, Materialized Views, and Pipes connect and relate to each other. You can filter the view by keyword, resource type, or tag. Select an item to see its details in the side panel. <-figure-> ![Data Flow demonstration](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fdataflow.gif&w=3840&q=75) ## Playground [¶](https://www.tinybird.co/docs/about:blank#playground) Playgrounds are sandbox environments where you can test your queries using ingested data. For example, you can use playgrounds to quickly query real-time production data, debug existing queries, or prototype new Pipes. You can download any playground by selecting **Download** . You can then add the .pipe file to your project. To share your Playground contents with other users of the Workspace, select **Share**. For more information on the statements, functions, and settings you can use in queries, see [SQL reference](https://www.tinybird.co/docs/docs/sql-reference). <-figure-> ![Playground demonstration](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fplayground.png&w=3840&q=75) ## Time Series [¶](https://www.tinybird.co/docs/about:blank#time-series) Time series help analyze a sequence of data points collected over an interval of time. Use the Time Series feature to visualize any time series Data Source in your Workspace, including [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources). When you create a new Time Series in your Workspace, your team can view it. You can also generate a public URL to share the visualization outside of your team. Viewers using the public URL can explore the chart but can't change the original data source, filters, or groupings. <-figure-> ![Time Series example](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftimeseries.png&w=3840&q=75) ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn how to[ use query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) . - Read the[ SQL best practices](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) . --- URL: https://www.tinybird.co/docs/work-with-data/process-and-copy Content: --- title: "Process and copy · Tinybird Docs" theme-color: "#171612" description: "Learn how to process and copy data in Tinybird." --- # Process and copy data [¶](https://www.tinybird.co/docs/about:blank#process-and-copy-data) Tinybird provides several ways to process and copy data between Data Sources: - ** Copy Pipes** allow you to capture the result of a Pipe at a specific point in time and write it to a target Data Source. They can run on a schedule or be executed on demand, making them ideal for event-sourced snapshots, data experimentation, and deduplication with snapshots. - ** Materialized Views** continuously re-evaluate a query as new events are inserted, maintaining an always up-to-date derived dataset. Unlike Copy Pipes which create point-in-time snapshots, Materialized Views provide real-time transformations of your data. - ** Populate operations** let you copy data between Data Sources by processing it partition by partition. This approach provides progress tracking and allows retrying individual steps, making it safer and more reliable for large data transfers. Each approach has its own strengths and use cases: - Use Copy Pipes when you need scheduled or on-demand snapshots of your data - Use Materialized Views when you need continuous, real-time transformations - Use Populate operations when you need controlled, partition-by-partition data transfers with progress tracking The following sections detail how to implement each of these approaches effectively, including best practices and important considerations for maintaining data integrity. --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work Content: --- title: "Continuous Integration Introduction · Tinybird Docs" theme-color: "#171612" description: "Learn about the importance of CI/CD in Tinybird, how it integrates with Git, and the advantages it offers for managing real-time data projects." --- # Prepare your data project for production [¶](https://www.tinybird.co/docs/about:blank#prepare-your-data-project-for-production) Tinybird's Git integration transforms data pipeline management, aligning it with established software development practices. This ensures each Tinybird Workspace action relates to a specific Git commit, offering a robust, version-controlled environment for your real-time data deployments. In a nutshell, it brings your workflow closer to industry best practices. ## Why use version control? [¶](https://www.tinybird.co/docs/about:blank#why-use-version-control) Version control, a standard in software development, is now integral to building real-time data products with Tinybird. If your Workspace uses Tinybird's integration with Git, it means you can build real-time data products like you build any software - not just benefiting from version control, but also isolated environments, CI/CD, and testing too. This approach allows you to treat and manage your real-time data in the **same way you manage your code** . You can take your existing software engineering knowledge and apply the same principles to real-time data products. When managing your Tinybird projects with version control, you can also: - Sync Tinybird actions with your Git commits. - Deploy semantically-versioned Data Sources, Pipes, and API Endpoints as code. - Use Branches to attach production data to non-production Branches, and test your data pipelines safely with real data. If you're familiar with version control already, it should make testing, merging, and deploying your Tinybird data projects a familiar process. Data teams can use Tinybird in the same way that software engineering teams work: To enforce standardized agreements to use version control, code reviews, quality assurance, testing strategies, and continuous deployment. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Get familiar with the core concepts:[ Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches) and[ Workspaces](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) . - Follow the version control tutorial to connect your Tinybird Workspace to version control:[ Working with version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) . - Explore our[ repository of common use cases for iterating using version control](https://github.com/tinybirdco/use-case-examples) to explore and try out how to iterate Tinybird projects with version control. --- URL: https://www.tinybird.co/docs/work-with-data/optimization Content: --- title: "Optimizations guides · Tinybird Docs" theme-color: "#171612" description: "In this set of guides, you'll learn where to look when looking to optimize your Tinybird project, what to edit, and how to monitor changes." --- # Optimizations [¶](https://www.tinybird.co/docs/about:blank#optimizations) This compilation of guides assumes some familiarity with using Tinybird (ingesting data, building query Pipes, publishing API Endpoints), particularly with using [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources). The good news is that Tinybird is so fast that even for un-optimized projects, response times are excellent. Many projects, especially younger or smaller ones, won't necessarily or immediately *need* to optimize right now. However, it's never too soon, especially because data moves fast. It's better to have 30ms latency than 300ms, and better to process less data so your [bills are smaller](https://www.tinybird.co/docs/docs/get-started/plans/billing#reduce-your-bill). ## About this section [¶](https://www.tinybird.co/docs/about:blank#about-this-section) The guides in this Optimizations section curate the best applied knowledge across Tinybird's docs ( [Best practices for faster SQL](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) ), videos ( [Materialization process saves $40K](https://youtu.be/rfckIMbLWBg?si=dY1dP_ctx5tOSyhq), [Tips & Tricks to Keep Your Queries under 100ms](https://www.youtube.com/watch?v=MN2M6HAoO64) ), blog posts ( [Thinking in Tinybird](https://www.tinybird.co/blog-posts/thinking-in-tinybird) ), and the deep expertise of our Data Engineering and Customer Support teams. It gives you both practical examples *and* a framework of questions you can ask in your own unique scenario. This combination should empower you with the tools, tips, tricks, and approach to build the best-optimized projects. So, if you want to feel like [Marc](https://x.com/mfts0/status/1797651962692767801) or [Thibault](https://x.com/thibaultleouay/status/1699492486488498270) , start digging in to the fascinating world of optimizing Tinybird projects. ## Optimizations mantra [¶](https://www.tinybird.co/docs/about:blank#optimizations-mantra) Tinybird gives you the platform to manage your real-time data. Measure what matters, detect inefficiencies, fix and eliminate common (or unusual!) mistakes, and move faster. Remember, speed wins! ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Dive in with[ Optimizations 101: Detecting inefficient resources](https://www.tinybird.co/docs/docs/work-with-data/optimization/opt101-detect-inefficiencies) . - Improve the efficiency of project with[ Optimizations 201: Fixing common mistakes](https://www.tinybird.co/docs/docs/work-with-data/optimization/opt201-fix-mistakes) . - Understand how to[ monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) . --- URL: https://www.tinybird.co/docs/use-cases/web-analytics Last update: 2024-05-13T16:10:11.000Z Content: --- title: "Web analytics · Tinybird Docs" theme-color: "#171612" description: "Monitor and analyze web traffic." --- # Web analytics [¶](https://www.tinybird.co/docs/about:blank#web-analytics) Get the data you need about your users, while satisfying data privacy laws, and without compromising on speed or scale. Track, analyze, and report on website visits, marketing conversions, ad-generated revenue, and more. Tinybird can help you monitor and analyze your web traffic, faster. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/get-started/quick-start/user-facing-web-analytics) Build a user-facing web analytics dashboard - [ ](https://www.tinybird.co/docs/docs/publish/charts/guides/bigquery-dashboard) Build user-facing dashboard with BigQuery - [ ](https://www.tinybird.co/docs/docs/publish/charts/guides/analytics-with-confluent) Build user-facing analytics apps with Confluent ## Videos [¶](https://www.tinybird.co/docs/about:blank#videos) [](/docs/live/google-analytics-free)![Build a GDPR-compliant alternative to Google Analytics](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fgoogle-analytics-free.png&w=750&q=75) Build a GDPR-compliant alternative to Google Analytics ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) - [ Multi-tenant analytics for SaaS applications](https://www.tinybird.co/blog-posts/multi-tenant-saas-options) - [ Building privacy-first native app telemetry with Guilherme Oenning](https://www.tinybird.co/blog-posts/dev-qa-guilherme-oenning) - [ Developer Q&A: Monitoring global API latency with chronark](https://www.tinybird.co/blog-posts/dev-qa-global-api-latency-chronark) - [ Developer Q&A with JR the Builder, co-creator of Beam Analytics](https://www.tinybird.co/blog-posts/qa-with-jr-the-builder-beam-analytics) - [ Looking for an open source Google Analytics alternative? Set one up in 3 minutes.](https://www.tinybird.co/blog-posts/google-analytics-alternative-in-3-minutes) - [ How I replaced Google Analytics with Retool and Tinybird, Part 2](https://www.tinybird.co/blog-posts/how-i-replaced-google-analytics-with-retool-and-tinybird-part-2) - [ How an eCommerce giant replaced Google Analytics for privacy and speed](https://www.tinybird.co/blog-posts/ecommerce-google-analytics-alternative) - [ How I replaced Google Analytics with Retool and Tinybird, Part 1](https://www.tinybird.co/blog-posts/how-i-replaced-google-analytics-with-retool-and-tinybird-part-1) - [ A privacy-first approach to building a Google Analytics alternative](https://www.tinybird.co/blog-posts/privacy-first-google-analytics-alternative) --- URL: https://www.tinybird.co/docs/use-cases/vector-search Last update: 2024-09-30T12:19:50.000Z Content: --- title: "Vector search · Tinybird Docs" theme-color: "#171612" description: "Use SQL functions to calculate vector distances and search large content tables." --- # Vector search [¶](https://www.tinybird.co/docs/about:blank#vector-search) Vector search allows you to search through multi-modal content based on calculated embeddings. It is a popular approach to search when keyword matching is insufficient. Tinybird can help you search largescale vector embeddings in real-time and apply real-time analytics principles to make vector search more performant. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/get-started/quick-start/vector-search-recommendation) Build a content recommendation API using vector search --- URL: https://www.tinybird.co/docs/use-cases/user-facing-dashboards Last update: 2024-05-22T09:05:57.000Z Content: --- title: "User-facing dashboards · Tinybird Docs" theme-color: "#171612" description: "Build user-facing dashboards." --- # User-facing dashboards [¶](https://www.tinybird.co/docs/about:blank#user-facing-dashboards) Users love insights. Give your users a real-time data analytics dashboard so they can monitor what's happening (plus how, when, and where), as it happens. Tinybird can help you implement user-facing dashboards, faster. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/publish/charts/guides/real-time-dashboard) Build a real-time dashboard with Tremor & Next.js - [ ](https://www.tinybird.co/docs/docs/publish/charts/guides/bigquery-dashboard) Build user-facing dashboard with BigQuery - [ ](https://www.tinybird.co/docs/docs/publish/charts/guides/analytics-with-confluent) Build user-facing analytics apps with Confluent ## Videos [¶](https://www.tinybird.co/docs/about:blank#videos) Watch screencasts and free training workshops that build user-facing dashboards. [](/docs/live/build-a-real-time-dashboard-apr-16)![Build User-Facing Analytics with Kafka, Tinybird, and React](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fbuild-a-real-time-dashboard-apr-16.png&w=750&q=75) Build User-Facing Analytics with Kafka, Tinybird, and React [](/docs/live/kafka-real-time-dashboard)![Build a Real-Time Dashboard over Kafka](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fkafka-real-time-dashboard.png&w=750&q=75) Build a Real-Time Dashboard over Kafka [](/docs/live/ugc-analytics-mux-tinybird)![Building UGC Analytics with Mux and Tinybird](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fugc-analytics-mux-tinybird.png&w=750&q=75) Building UGC Analytics with Mux and Tinybird ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) Learn more about user-facing analytics on the Tinybird Blog. - [ Building real-time leaderboards with Tinybird](https://www.tinybird.co/blog-posts/building-real-time-leaderboards-with-tinybird) - [ JWTs for API Endpoints now in public beta!](https://www.tinybird.co/blog-posts/jwt-api-endpoints-public-beta) - [ 7 tips to make your dashboards faster](https://www.tinybird.co/blog-posts/7-tips-to-make-your-dashboards-faster) - [ Build a real-time dashboard over BigQuery](https://www.tinybird.co/blog-posts/bigquery-real-time-dashboard) - [ Real-time dashboards: Are they worth it?](https://www.tinybird.co/blog-posts/real-time-dashboards-are-they-worth-it) - [ Real-time Data Visualization: How to build faster dashboards](https://www.tinybird.co/blog-posts/real-time-data-visualization) - [ How Typeform Built a Fully Functional User Dash With Tinybird](https://www.tinybird.co/blog-posts/typeform-utm-realtime-analytics) --- URL: https://www.tinybird.co/docs/use-cases/ugc-analytics Last update: 2024-05-22T09:34:43.000Z Content: --- title: "User-generated content · Tinybird Docs" theme-color: "#171612" description: "Provide your content creators with insights into their content performance." --- # User-generated content (UGC) analytics [¶](https://www.tinybird.co/docs/about:blank#user-generated-content-ugc-analytics) For user-generated content platforms, creators are everything. Creators need to know how their content is performing, so they can make better decisions about what to create next. Tinybird can help you ship creator insights, faster. ## Demo [¶](https://www.tinybird.co/docs/about:blank#demo) Want to see it in action? Check out our Tiny Flappybird demo: - [ Multi-tenant user-facing analytics with Tinybird and Mux](https://github.com/tinybirdco/demo-user-generated-content-analytics) ## Videos [¶](https://www.tinybird.co/docs/about:blank#videos) Watch screencasts and free training workshops that build user-facing content analytics. [](/docs/live/ugc-analytics-mux-tinybird)![Building UGC Analytics with Mux and Tinybird](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fugc-analytics-mux-tinybird.png&w=750&q=75) Building UGC Analytics with Mux and Tinybird ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) - [ Upgrading short link analytics by 100x with Steven Tey](https://www.tinybird.co/blog-posts/upgrading-short-link-analytics-by-100x-with-steven-tey) - [ Low-code analytics with James Devonport of UserLoop](https://www.tinybird.co/blog-posts/low-code-analytics-with-james-devonport-of-userloop) --- URL: https://www.tinybird.co/docs/use-cases/realtime-personalization Last update: 2024-05-22T09:05:57.000Z Content: --- title: "Real-time personalization · Tinybird Docs" theme-color: "#171612" description: "Personalize your user experience in real time." --- # Real-time personalization [¶](https://www.tinybird.co/docs/about:blank#real-time-personalization) Real-time personalization is the pathway to better user experiences. This user-centered approach powers a vast number of modern customer experiences, and is fast becoming the standard of digital engagement. Not only can you provide meaningfully tailored recommendations, ads, or products, you can also leverage real-time personalization to analyze transaction streams for digital fraud, and more. ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) Learn more about user-facing analytics on the Tinybird Blog. - [ Using Tinybird for real-time marketing at Tinybird](https://www.tinybird.co/blog-posts/tinybird-for-real-time-marketing) - [ Real-time Personalization: Choosing the right tools](https://www.tinybird.co/blog-posts/real-time-personalization) - [ How to build a real-time fraud detection system](https://www.tinybird.co/blog-posts/how-to-build-a-real-time-fraud-detection-system) - [ Building real-time solutions with Snowflake at a fraction of the cost](https://www.tinybird.co/blog-posts/real-time-solutions-with-snowflake) - [ Designing a faster data model to personalize browsing in real time](https://www.tinybird.co/blog-posts/clickhouse-query-optimization) --- URL: https://www.tinybird.co/docs/use-cases/realtime-cdc Last update: 2024-10-03T12:50:26.000Z Content: --- title: "Real-time Change Data Capture · Tinybird Docs" theme-color: "#171612" description: "Build real-time CDC systems using fresh, accurate data from your favorite databases." --- # Change data capture [¶](https://www.tinybird.co/docs/about:blank#change-data-capture) Change Data Capture (CDC) is a process used in databases to track and capture any changes made to data, such as inserts, updates, or deletes. It enables the identification and recording of modifications in real-time or near real-time, ensuring that downstream systems can be kept in sync with the primary database without having to constantly query the entire dataset. This is achieved by monitoring the transaction logs of the database, allowing CDC to capture only the changes rather than reprocessing all data. Tinybird helps you quickly build applications on top of your CDC data. When you send your change data to Tinybird, you are able to unify your CDC data with other data sources and transform it using SQL, and with one click or command line instruction, convert your SQL queries into high-concurrency, low-latency REST APIs. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies) Deduplicate data in your Data Source - [ ](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-example-cdc) Change Data Capture project with lambda architecture ## Videos [¶](https://www.tinybird.co/docs/about:blank#videos) Watch screencasts and free training workshops for building CDC systems. [](/docs/live/mongodb-cdc-confluent-tinybird-sep-2024)![Real-time MongoDB CDC with Confluent and Tinybird](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fmongodb-cdc-confluent-tinybird-sep-2024.png&w=750&q=75) Real-time MongoDB CDC with Confluent and Tinybird [](/docs/live/dynamodb-cdc-july-2024)![Capturing DynamoDB Change Streams for Real-Time Analytics](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fdynamodb-cdc-july-2024.png&w=750&q=75) Capturing DynamoDB Change Streams for Real-Time Analytics ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) Learn more about CDC on the Tinybird Blog. - [ From CDC to real-time analytics with Tinybird and Estuary](https://www.tinybird.co/blog-posts/cdc-real-time-analytics-estuary-dekaf) - [ A practical guide to real-time CDC with MySQL](https://www.tinybird.co/blog-posts/mysql-cdc) - [ A practical guide to real-time CDC with Postgres](https://www.tinybird.co/blog-posts/postgres-cdc) - [ A practical guide to real-time CDC with MongoDB](https://www.tinybird.co/blog-posts/mongodb-cdc) --- URL: https://www.tinybird.co/docs/use-cases/gaming-analytics Last update: 2024-09-18T17:42:03.000Z Content: --- title: "Gaming analytics · Tinybird Docs" theme-color: "#171612" description: "Embed analytics in games." --- # Gaming analytics [¶](https://www.tinybird.co/docs/about:blank#gaming-analytics) From leaderboards, to match-making, to serving personalized in-game ads, data is at the core of modern gaming. Tinybird can help you ship user-facing analytics features in your games, faster. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/get-started/quick-start/leaderboard) Build a real-time game leaderboard ## Demo [¶](https://www.tinybird.co/docs/about:blank#demo) Want to see it in action? Check out our Tiny Flappybird demo: - [ Tiny Flappybird Demo](https://flappy.tinybird.co/) - [ Demo code](https://github.com/tinybirdco/flappy-tinybird) ## Videos [¶](https://www.tinybird.co/docs/about:blank#videos) Watch screencasts and free training workshops that build user-facing analytics. [](/docs/live/build-real-time-leaderboards-june-2024)![Free Workshop: Build Real-Time Leaderboards](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fflappy-bird.jpeg&w=750&q=75) Free Workshop: Build Real-Time Leaderboards [](/docs/live/user-segmentation-at-1m-events-per-second)![User segmentation at 1M events per second](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Flive%2Fuser-segmentation-at-1m-events-per-second.png&w=750&q=75) User segmentation at 1M events per second --- URL: https://www.tinybird.co/docs/use-cases/content-recommendation Last update: 2024-09-30T12:19:50.000Z Content: --- title: "Content recommendation · Tinybird Docs" theme-color: "#171612" description: "Recommend useful or related content in real-time using SQL." --- # Content recommendation [¶](https://www.tinybird.co/docs/about:blank#content-recommendation) Content recommendation systems can use real-time analytics or vector search approaches to show a web visitor or platform user content their likely to enjoy based on similar content. Learn more about content recommendation systems and how to build them with Tinybird. ## Tutorials [¶](https://www.tinybird.co/docs/about:blank#tutorials) - [ ](https://www.tinybird.co/docs/docs/get-started/quick-start/vector-search-recommendation) Build a content recommendation API using vector search ## Blog posts [¶](https://www.tinybird.co/docs/about:blank#blog-posts) Learn more about content recommendation on the Tinybird Blog. --- URL: https://www.tinybird.co/docs/sql-reference/functions Content: --- title: "Functions · Tinybird Docs" theme-color: "#171612" description: "Functions supported by Tinybird" --- # Functions [¶](https://www.tinybird.co/docs/about:blank#functions) Functions in SQL are predefined operations that perform specific tasks on data. They accept input parameters, process them according to defined rules, and return results. Functions help you transform, calculate, or analyze data within your queries. - [ Aggregate functions](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions) - [ Arithmetic functions](https://www.tinybird.co/docs/docs/sql-reference/functions/arithmetic-functions) - [ Array functions](https://www.tinybird.co/docs/docs/sql-reference/functions/array-functions) - [ ArrayJoin functions](https://www.tinybird.co/docs/docs/sql-reference/functions/array-join) - [ Bit functions](https://www.tinybird.co/docs/docs/sql-reference/functions/bit-functions) - [ Bitmap functions](https://www.tinybird.co/docs/docs/sql-reference/functions/bitmap-functions) - [ Comparison functions](https://www.tinybird.co/docs/docs/sql-reference/functions/comparison-functions) - [ Conditional functions](https://www.tinybird.co/docs/docs/sql-reference/functions/conditional-functions) - [ Date and time functions](https://www.tinybird.co/docs/docs/sql-reference/functions/date-time-functions) - [ Distance functions](https://www.tinybird.co/docs/docs/sql-reference/functions/distance-functions) - [ Encoding functions](https://www.tinybird.co/docs/docs/sql-reference/functions/encoding-functions) - [ Encryption functions](https://www.tinybird.co/docs/docs/sql-reference/functions/encryption-functions) - [ Functions for nulls](https://www.tinybird.co/docs/docs/sql-reference/functions/functions-for-nulls) - [ Geo functions](https://www.tinybird.co/docs/docs/sql-reference/functions/geo-functions) - [ Hash functions](https://www.tinybird.co/docs/docs/sql-reference/functions/hash-functions) - [ IP addresses functions](https://www.tinybird.co/docs/docs/sql-reference/functions/ip-address-functions) - [ JSON functions](https://www.tinybird.co/docs/docs/sql-reference/functions/json-functions) - [ Logical functions](https://www.tinybird.co/docs/docs/sql-reference/functions/logical-functions) - [ Machine learning functions](https://www.tinybird.co/docs/docs/sql-reference/functions/machine-learning-functions) - [ Math functions](https://www.tinybird.co/docs/docs/sql-reference/functions/math-functions) - [ Other functions](https://www.tinybird.co/docs/docs/sql-reference/functions/other-functions) - [ Parametric functions](https://www.tinybird.co/docs/docs/sql-reference/functions/parametric-functions) - [ Random functions](https://www.tinybird.co/docs/docs/sql-reference/functions/random-functions) - [ Rounding functions](https://www.tinybird.co/docs/docs/sql-reference/functions/rounding-functions) - [ Splitting and merging functions](https://www.tinybird.co/docs/docs/sql-reference/functions/splitting-merging-functions) - [ String functions](https://www.tinybird.co/docs/docs/sql-reference/functions/string-functions) - [ String replace functions](https://www.tinybird.co/docs/docs/sql-reference/functions/string-replace-functions) - [ String search functions](https://www.tinybird.co/docs/docs/sql-reference/functions/string-search-functions) - [ Time series functions](https://www.tinybird.co/docs/docs/sql-reference/functions/time-series-functions) - [ Time window functions](https://www.tinybird.co/docs/docs/sql-reference/functions/time-window-functions) - [ Tuple functions](https://www.tinybird.co/docs/docs/sql-reference/functions/tuple-functions) - [ Tuple map functions](https://www.tinybird.co/docs/docs/sql-reference/functions/tuple-map-functions) - [ Type conversion functions](https://www.tinybird.co/docs/docs/sql-reference/functions/type-conversion-functions) - [ ULID functions](https://www.tinybird.co/docs/docs/sql-reference/functions/ulid-functions) - [ UniqTheta functions](https://www.tinybird.co/docs/docs/sql-reference/functions/uniqtheta-functions) - [ URL functions](https://www.tinybird.co/docs/docs/sql-reference/functions/url-functions) - [ UUID functions](https://www.tinybird.co/docs/docs/sql-reference/functions/uuid-functions) - [ Window functions](https://www.tinybird.co/docs/docs/sql-reference/functions/window-functions) --- URL: https://www.tinybird.co/docs/sql-reference/engines Content: --- title: "Table engines · Tinybird Docs" theme-color: "#171612" description: "Use the following table engines in your Data Sources." --- # Table engines [¶](https://www.tinybird.co/docs/about:blank#table-engines) Tinybird features different strategies to store data, which define where and how the data is stored and also what kind of data access, queries, and availability your data has. A Tinybird Data Source uses a table engine that determines those factors. With Tinybird you can select the Table Engine for your Data Source. Tinybird supports the following engines: - [ MergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/mergetree) - [ AggregatingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/aggregatingmergetree) - [ ReplacingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/replacingmergetree) - [ SummingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/summingmergetree) - [ CollapsingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/collapsingmergetree) - [ VersionedCollapsingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/versionedcollapsingmergetree) - [ Null](https://www.tinybird.co/docs/docs/sql-reference/engines/null) You can use the `engine` parameter in the [Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) to specify the name of any of the available engines, for example `engine=ReplacingMergeTree` . To set the engine parameters and the engine options, use as many `engine_*` request parameters as needed. You can also configure settings in .datasource files. The supported parameters for each engine are: | ENGINE | SIGNATURE | PARAMETER | DESCRIPTION | | --- | --- | --- | --- | | ReplacingMergeTree | `(ver, is_deleted)` | `engine_ver` | Optional. The column with the version. If not set, the last row is kept during a merge. If set, the maximum version is kept. | | | | `engine_is_deleted` | Active only when `ver` is used. The name of the column used to determine whether the row is to be deleted; `1` is a `deleted` row, `0` is a `state` row. | | SummingMergeTree | `([columns])` | `engine_columns` | Optional. The names of columns where values are summarized. | | CollapsingMergeTree | `(sign)` | `engine_sign` | Name of the column for computing the state. | | VersionedCollapsingMergeTree | `(sign, version)` | `engine_sign` | Name of the column for computing the state. | | | | `engine_version` | Name of the column with the version of the object state. | See the [Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) for examples of creating Data Sources with custom engine settings using the Tinybird REST API. --- URL: https://www.tinybird.co/docs/sql-reference/data-types Content: --- title: "Data types · Tinybird Docs" theme-color: "#171612" description: "Data types supported by Tinybird" --- # Data types [¶](https://www.tinybird.co/docs/about:blank#data-types) Data types define how values are stored and processed in a database. They determine what kind of data can be stored in a column (like numbers, text, dates, etc.), how much storage space the data will use, and what operations can be performed on the values. Choosing the right data type is important for both data integrity and query performance. Each column in a table must have a specified data type that matches the kind of data you plan to store. For example, you would use numeric types for calculations, string types for text, and date/time types for temporal data. The following data types are supported at ingest: - `Int8` , `Int16` , `Int32` , `Int64` , `Int128` , `Int256` - `UInt8` , `UInt16` , `UInt32` , `UInt64` , `UInt128` , `UInt256` - `Float32` , `Float64` - `Decimal` , `Decimal(P, S)` , `Decimal32(S)` , `Decimal64(S)` , `Decimal128(S)` , `Decimal256(S)` - `String` - `FixedString(N)` - `UUID` - `Date` , `Date32` - `DateTime([TZ])` , `DateTime64(P, [TZ])` - `Bool` - `Array(T)` - `Map(K, V)` - `LowCardinality` - `Nullable` - `JSON` If you are ingesting using the NDJSON format and would like to store `Decimal` values containing 15 or more digits, send the values as string instead of numbers to avoid precision issues. In the following example, the first value has a high chance of losing accuracy during ingestion, while the second one is stored correctly: {"decimal_value": 1234567890.123456789} # Last digits might change during ingestion {"decimal_value": "1234567890.123456789"} # Will be stored correctly The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). --- URL: https://www.tinybird.co/docs/publish/sinks Content: --- title: "Sinks · Tinybird Docs" theme-color: "#171612" description: "Sinks are the destinations for your data. They are the places where you can store your data after it has been transformed." --- # Sinks [¶](https://www.tinybird.co/docs/about:blank#sinks) Tinybird Sinks allow you to export data from your Tinybird Workspace to external systems on a scheduled or on-demand basis. Sinks are built on top of [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) and provide a fully managed way to push data to various destinations. ## Available Sinks [¶](https://www.tinybird.co/docs/about:blank#available-sinks) Tinybird supports the following Sink destinations: - [ Kafka sink](https://www.tinybird.co/docs/docs/publish/sinks/kafka-sink) - [ S3 sink](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink) ## Key features [¶](https://www.tinybird.co/docs/about:blank#key-features) - Fully Managed: Sinks require no additional tooling or infrastructure management. - Scheduled or On-Demand: Run exports on a defined schedule using cron expressions or trigger them manually when needed. - Query Parameters: Support for parameterized queries allows flexible data filtering and transformation. - Observability: Monitor Sink operations and data transfer through Service Data Sources. ## Common use cases [¶](https://www.tinybird.co/docs/about:blank#common-use-cases) Sinks enable various data integration scenarios: - Regular data exports to clients or partner systems. - Feeding data lakes and data warehouses. - Real-time data synchronization with external systems. - Event-driven architectures and data pipelines. --- URL: https://www.tinybird.co/docs/publish/charts Last update: 2024-12-13T10:17:28.000Z Content: --- title: "Charts · Tinybird Docs" theme-color: "#171612" description: "Create beautiful, fast charts of your Tinybird data." --- # Charts [¶](https://www.tinybird.co/docs/about:blank#charts) Charts are a great way to visualize your data. You can create and publish easy, fast Charts in Tinybird from any of your published API Endpoints. <-figure-> ![Example Tinybird dashboard showing multiple chart types](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcharts-dashboard.png&w=3840&q=75) <-figcaption-> Example Tinybird Charts dashboard Check out the [live demo](https://guide-tinybird-charts.vercel.app/) to see an example of Charts in action. You can't create or manage charts using Tinybird CLI. ## Overview [¶](https://www.tinybird.co/docs/about:blank#overview) When you publish an API Endpoint, you often want to visualize the data in a more user-friendly way. Charts are a great way to do this. Tinybird provides three options: - No-code: A fast, UI-based flow for creating Charts that live in your Tinybird Workspace UI (great for internal reference use, smaller projects, and getting started). - Low code: Using the Tinybird UI to create Charts and generate an iframe, which you can then embed in your own application. - Code-strong: Using the `@tinybirdco/charts` npm React library to build out exactly what you need, in your own application, using React components. Fully customizable and secured with JWTs. You can either generate initial Chart data in the Tinybird UI, or start using the library directly. Instead of coding your own charts and dashboards from scratch, use Tinybird's pre-built Chart components. You won't have to implement the frontend and backend architecture, or any security middleware. Use the library components and JWTs, manage the token exchange flow, and interact directly with any of your published Tinybird API Endpoints. To create a Chart, you need to have a published API Endpoint. Learn how to [publish an API Endpoint here](https://www.tinybird.co/docs/docs/publish/api-endpoints). ## The Tinybird Charts library BETA [¶](https://www.tinybird.co/docs/about:blank#the-tinybird-charts-library) All options are built on the Tinybird Charts library ( `@tinybirdco/charts` ) , a modular way to build fast visualizations of your data, which leverages [Apache ECharts](https://echarts.apache.org/en/index.html) . You can use Tinybird Charts with any level of customization, and also use your Tinybird data with any third party library. ### Components [¶](https://www.tinybird.co/docs/about:blank#components) The library provides the following components: - `AreaChart` - `BarChart` - `BarList` - `DonutChart` - `LineChart` - `PieChart` - `Table` All components share the same API, making it easy to switch between different types of charts. The Tinybird Charts library is currently in public beta. If you have any feedback or suggestions, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). ## Create a Chart in the UI (no-code) [¶](https://www.tinybird.co/docs/about:blank#create-a-chart-in-the-ui-no-code) 1. In your Workspace, navigate to the Overview page of one of your API Endpoints. 2. Select the "Create Chart" button (top right). 3. Configure your Chart by selecting the name, the type of Chart, and the fields you want to visualize. Under "Data", select the index and category. 4. Once you're happy with your Chart, select "Save". <-figure-> ![Example Tinybird pie chart, showing configuration options](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcharts-create-chart.png&w=3840&q=75) <-figcaption-> Example Tinybird pie Chart, showing configuration options Your Chart now lives in the API Endpoint Overview page. ## Create a Chart using an iframe (low code) [¶](https://www.tinybird.co/docs/about:blank#create-a-chart-using-an-iframe-low-code) Tinybird users frequently want to take the data from their Tinybird API Endpoints, create charts, and embed them in their own dashboard application. A low-overhead option is to take the generated iframe and drop it into your application: 1. Create a Chart using the process described above. 2. In the API Endpoint Overview page, scroll down to the "Charts" tab and select your Chart. 3. Select the `<>` tab to access the code snippets. 4. Copy and paste the ready-to-use iframe code into your application. <-figure-> ![GIF showing a user creating a Pie Chart in the Tinybird UI and generating the iframe code](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcharts-create-pie-chart.gif&w=3840&q=75) <-figcaption-> Creating a Pie Chart in the Tinybird UI and generating the iframe code ## Create a Chart using the React library (code-strong) [¶](https://www.tinybird.co/docs/about:blank#create-a-chart-using-the-react-library-code-strong) This option gives you the most customization flexibility. You'll need to be familiar with frontend development and styling. To create a Chart component and use the library, you can either create a Chart in the UI first, or use the library directly. Using the library directly means there will not be a Chart created in your Workspace, and no generated snippet, so skip to #2. ### 1. View your Chart code [¶](https://www.tinybird.co/docs/about:blank#1-view-your-chart-code) 1. In your Workspace, navigate to the API Endpoint Overview page. 2. Scroll down to the "Charts" tab and select one of your Charts. 3. Select the `<>` tab to access the code snippets. 4. You now have the code for a ready-to-use React component. ### 2. Install the library [¶](https://www.tinybird.co/docs/about:blank#2-install-the-library) Install the `@tinybirdco/charts` library locally in your project: npm install @tinybirdco/charts ### 3. Create a JWT [¶](https://www.tinybird.co/docs/about:blank#3-create-a-jwt) Calls need to be secured with a token. To learn more about the token exchange, see [Understanding the token exchange](https://www.tinybird.co/docs/docs/publish/charts/guides/charts-using-iframes-and-jwt-tokens#understanding-the-token-exchange). In React code snippets, the Chart components are authenticated using the `token` prop, where you paste your [Tinybird Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens). You can limit how often you or your users can fetch Tinybird APIs on a per-endpoint or per-user basis. See [Rate limits for JWTs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#rate-limits-for-jwts). #### Create a JWT [¶](https://www.tinybird.co/docs/about:blank#create-a-jwt) There is wide support for creating JWTs in many programming languages and frameworks. See the [Tinybird JWT docs for popular options](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#create-a-jwt-in-production). ### 4. Embed the Chart into your application [¶](https://www.tinybird.co/docs/about:blank#4-embed-the-chart-into-your-application) Copy and paste the Chart snippet (either generated by Tinybird, or constructed by you [using the same configuration](https://www.npmjs.com/package/@tinybirdco/charts#usage) ) into your application. For example: ##### Example Line Chart component code snippet import React from 'react' import { LineChart } from '@tinybirdco/charts' function MyLineChart() { return ( ) } ### 5. Fetch data [¶](https://www.tinybird.co/docs/about:blank#5-fetch-data) The most common approach for fetching and rendering data is to directly use a single Chart component (or group of individual components) by passing the required props. These are included by default within each generated code snippet, so for most use cases, you should be able to simply copy, paste, and have the Chart you want. The library offers [many additional props](https://www.npmjs.com/package/@tinybirdco/charts?activeTab=readme#api) for further customization, including many that focus specifically on fetching data. See [6. Customization](https://www.tinybird.co/docs/docs/publish/charts#6-customization) for more. #### Alternative approaches and integrations [¶](https://www.tinybird.co/docs/about:blank#alternative-approaches-and-integrations) Depending on your needs, you have additional options: - Wrapping components within `` to share styles, query configuration, and custom loading and error states[ among several Chart components](https://www.npmjs.com/package/@tinybirdco/charts?activeTab=readme#reusing-styles-and-query-config-using-the-chartprovider) . - Adding your own fetcher to the `ChartProvider` (or to a specific Chart component) using the `fetcher` prop . This can be useful to add custom headers or dealing with JWT tokens. - Using the `useQuery` hook to[ fetch data and pass it directly to the component](https://www.npmjs.com/package/@tinybirdco/charts?activeTab=readme#using-the-hook) . It works with any custom component, or with any third-party library like[ Tremor](https://www.tremor.so/) or[ shadcn](https://ui.shadcn.com/) . <-figure-> ![GIF showing how to use the library with Tinybird Charts & shadcn in the UI](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcharts-tb-sd.gif&w=3840&q=75) <-figcaption-> Using the library with Tinybird Charts & shadcn ### 6. Customization [¶](https://www.tinybird.co/docs/about:blank#6-customization) Tinybird supports customization: - ** Standard customization** : Use the properties provided by the[ @tinybirdco/charts library](https://www.npmjs.com/package/@tinybirdco/charts?activeTab=readme#api) . - ** Advanced customization** : Send a specific parameter to[ customize anything within a Chart](https://www.npmjs.com/package/@tinybirdco/charts?activeTab=readme#extra-personalization-using-echarts-options) , aligning with the[ ECharts specification](https://echarts.apache.org/handbook/en/get-started/) . ### 7. Filtering [¶](https://www.tinybird.co/docs/about:blank#7-filtering) Filtering is possible by using the endpoint parameters. Use the `params` data-fetching property to pass your parameters to a chart component, `` , or the `useQuery` hook. See the [example snippet](https://www.tinybird.co/docs/about:blank#4-embed-the-chart-into-your-application) for how `params` and filter are used. ### 8. Advanced configuration (optional) [¶](https://www.tinybird.co/docs/about:blank#8-advanced-configuration-optional) #### Polling [¶](https://www.tinybird.co/docs/about:blank#polling) Control the frequency that your chart polls for new data by setting the `refreshInterval` prop (interval in milliseconds). The npm library offers [a range of additional component props](https://www.npmjs.com/package/@tinybirdco/charts) specifically for data fetching, so be sure to review them and use a combination to build the perfect chart. Use of this feature may significantly increase your billing costs. The lower the refreshInterval prop (so the more frequently you're polling for fresh data), the more requests you're making to your Tinybird API Endpoints. Read [the billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing) and understand the pricing of different operations. #### Global vs local settings [¶](https://www.tinybird.co/docs/about:blank#global-vs-local-settings) Each chart can have its own settings, or settings can be shared across a group of Chart components by wrapping them within ``. #### States [¶](https://www.tinybird.co/docs/about:blank#states) Chart components can be one of a range of states: - Success - Error - Loading ## Example use cases [¶](https://www.tinybird.co/docs/about:blank#example-use-cases) Two examples showing different ways to generate JWTs, set up a local project, and implement a Chart: 1. Guide:[ Consume APIs in a Next.js frontend with JWTs](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-apis-nextjs) . 2. Guide:[ Build charts with iframes and JWTs](https://www.tinybird.co/docs/docs/publish/charts/guides/charts-using-iframes-and-jwt-tokens) . Interested in Charts but don't have any data to use? Run the demo in the [Add Tinybird Charts to a Next.js frontend](https://www.tinybird.co/docs/docs/publish/charts/guides/add-charts-to-nextjs) guide, which uses a bootstrapped Next.js app and fake data. ## Troubleshooting [¶](https://www.tinybird.co/docs/about:blank#troubleshooting) ### Handle token errors [¶](https://www.tinybird.co/docs/about:blank#handle-token-errors) See the information on [JWT error handling](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#error-handling). ### Refresh token [¶](https://www.tinybird.co/docs/about:blank#refresh-token) See the information on [JWT refreshing and limitations](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#jwt-limitations). ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Check out the[ Tinybird Charts library](https://www.npmjs.com/package/@tinybirdco/charts) . - Understand how Tinybird[ uses Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . --- URL: https://www.tinybird.co/docs/publish/api-endpoints Last update: 2024-10-16T16:43:40.000Z Content: --- title: "API Endpoints · Tinybird Docs" theme-color: "#171612" description: "API Endpoints make it easy to use the results of your queries in applications." --- # API Endpoints [¶](https://www.tinybird.co/docs/about:blank#api-endpoints) Tinybird can turn any Pipe into an API Endpoint that you can query. For example, you can ingest your data, build SQL logic inside a Pipe, and then publish the result of your query as an HTTP API Endpoint. You can then create interactive [Charts](https://www.tinybird.co/docs/docs/publish/charts) of your data. API Endpoints make it easy to use the results of your queries in applications. Any app that can run an HTTP GET can use Tinybird API Endpoints. Tinybird represents API Endpoints using the icon. ## Create an API Endpoint [¶](https://www.tinybird.co/docs/about:blank#create-an-api-endpoint) To create an API Endpoint, you first need a [Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) . You can publish any of the queries in your Pipes as an API Endpoint. ### Using the UI [¶](https://www.tinybird.co/docs/about:blank#using-the-ui) First, [create a Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#creating-pipes-in-the-ui) in the UI. In the Pipe, select **Create API Endpoint** , then select the node that you want to publish. You can export a CSV file with the extracted data by selecting **Export CSV**. ### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-cli) First, [create a Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#creating-pipes-in-the-cli) using the Tinybird CLI. Use the following command to publish an API Endpoint from the CLI. This automatically selects the final node in the Pipe. tb pipe publish PIPE_NAME_OR_ID If you want to manually select a different node to publish, supply the node name as the final command argument: tb pipe publish PIPE_NAME_OR_ID NODE_NAME ## Secure your API Endpoints [¶](https://www.tinybird.co/docs/about:blank#secure-your-api-endpoints) Access to the APIs you publish in Tinybird are also protected with Tokens. You can limit which operations a specific Token can do through scopes. For example, you can create Tokens that are only able to do admin operations on Tinybird resources, or only have `READ` permission for a specific Data Source. See [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) to understand how they work and see what types are available. ## API gateways [¶](https://www.tinybird.co/docs/about:blank#api-gateways) API gateways allow you to cloak or rebrand Tinybird API Endpoints while meeting additional security and compliance requirements. When you publish an [API Endpoint](https://www.tinybird.co/docs/docs/publish/api-endpoints) in Tinybird, it's available through `api.tinybird.co` or the API Gateway URL that corresponds to your [Workspace](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) region. See [API Endpoint URLs](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) . API Endpoints are secured using [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) that are managed inside your Tinybird Workspace. Sometimes you might want to put the Tinybird API Endpoints behind an API Gateway. For example: - To present a unified brand experience to your users. - To avoid exposing Tokens and the underlying technology. - To comply with regulations around data privacy and security. - To add Tinybird to an existing API architecture. ### Alternative approaches [¶](https://www.tinybird.co/docs/about:blank#alternative-approaches) You can meet the requirements satisfied by an API gateway through other methods. - Use[ JSON Web Tokens (JWTs)](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#json-web-tokens-jwts) to habe your application call Tinybird API Endpoints from the frontend without proxying through your backend. - Appropriately scope the Token used inside your application. Exposing a read-only Token has limited security concerns as it can't be used to modify data. You can invalidate the Token at any time. - Use row-level security to ensure that an Token only provides access to the appropriate data. ### Amazon API Gateway [¶](https://www.tinybird.co/docs/about:blank#amazon-api-gateway) The steps to create a reverse proxy using Amazon API Gateway are as follows: 1. Access the API Gateway console. 2. Select** Create API** , then** HTTP API** . 3. Select** Add Integration** and then select** HTTP** . 4. Configure the integration with the method** GET** and the full URL to your Tinybird API with its Token. For example, `https://api.tinybird.co/v0/pipes/top-10-products.json?token=p.eyJ1Ijog...` 5. Set a name for the API and select** Next** . 6. On the** Routes** page, set the method to** GET** and configure the desired** Resource path** . For example,** /top-10-products** . 7. Go through the rest of the step to create the API. You can find more information about applying a custom domain name in the [Amazon API Gateway documentation](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-custom-domain-names.html). ### Google Cloud Apigee [¶](https://www.tinybird.co/docs/about:blank#google-cloud-apigee) The steps to create a reverse proxy using Apigee are as follows: 1. Access the Apigee console. 2. Add a new** Reverse Proxy** . 3. Add your** Base path** . For example,** /top-10-products** . 4. Add the** Target** . For example, `https://api.tinybird.co/v0/pipes/top-10-products.json?token=p.eyJ1Ijog...` 5. Select** Pass through** for security. 6. Select an environment to deploy the API to. 7. Deploy, and test the API. You can find more information about applying a custom domain name in the [Apigee documentation](https://cloud.google.com/apigee/docs/api-platform/publish/portal/custom-domain). ### Grafbase Edge Gateway [¶](https://www.tinybird.co/docs/about:blank#grafbase-edge-gateway) To create a new Grafbase Edge Gateway using the Grafbase CLI, follow these steps. Inside a new directory, run: npx grafbase init --template openapi-tinybird In Tinybird, open your API Endpoint page. Select **Create Chart**, **Share this API Endpoint** , then select **OpenAPI 3.0** . Copy the link that appears, including the full Token. Create an `.env` file using the following template and enter the required details. # TINYBIRD_API_URL is the URL for your published API Endpoint TINYBIRD_API_URL= # TINYBIRD_API_TOKEN is the Token with READ access to the API Endpoint TINYBIRD_API_TOKEN= # TINYBIRD_API_SCHEMA is the OpenAPI 3.0 spec URL copied from the API Endpoint docs page TINYBIRD_API_SCHEMA= You can now run the Grafbase Edge Gateway locally: npx grafbase dev Open the local Pathfinder at `http://127.0.0.1:4000` to test your Edge Gateway. Here is an example GraphQL query: query Tinybird { tinybird { topPages { data { action payload } rows } } } Make sure to replace `topPages` with the name of your API Endpoint. ### NGINX [¶](https://www.tinybird.co/docs/about:blank#nginx) The following is an example NGINX configuration file that handles a `GET` request and make the request to Tinybird on your behalf. The Token is only accessed server-side and never exposed to the user. worker_processes 1; events { worker_connections 1024; } http { server { listen 8080; server_name localhost; location /top-10-products { proxy_pass https://api.tinybird.co/v0/pipes/top-10-products.json?token=p.eyJ1Ijog...; } } } ## Query API and API Endpoints [¶](https://www.tinybird.co/docs/about:blank#query-api-and-api-endpoints) The [Query API](https://www.tinybird.co/docs/docs/api-reference/query-api) is similar to running SQL statements against a normal database instead of using it as your backend, which is useful for ad-hoc queries. Publish API Endpoints instead of using the Query API in the following situations: - You want to build and maintain all the logic in Tinybird and call the API Endpoint to fetch the result. - You want to use incremental nodes in your Pipes to simplify the development and maintenance of your queries. - You need support for query parameters and more complex logic using the Tinybird[ templating language](https://www.tinybird.co/docs/docs/cli/advanced-templates) . - You need to incorporate changes from your query, which means little downstream impact. You can monitor performance of individual API Endpoints using [pipe_stats_rt](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-pipe-stats-rt) and [pipe_stats](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-pipe-stats) , uncovering optimization opportunities. All requests to the Query API are grouped together, making it more difficult to monitor performance of a specific query. ## Errors and retries [¶](https://www.tinybird.co/docs/about:blank#errors-and-retries) API Endpoints return standard HTTP success or error codes. For errors, the response also includes extra information about what went wrong, encoded in the response as JSON. ### Error codes [¶](https://www.tinybird.co/docs/about:blank#error-codes) API Endpoints might return the following HTTP error codes: | Code | Description | | --- | --- | | 400 | Bad request. A `HTTP400` can be returned in several scenarios and typically represents a malformed request such as errors in your SQL queries or missing query parameters. | | 403 | Forbidden. The auth Token doesn't have the correct scopes. | | 404 | Not found. This usually occurs when the name of the API Endpoint is wrong or hasn't been published. | | 405 | HTTP Method not allowed. Requests to API Endpoints must use the `GET` method. | | 408 | Request timeout. This occurs when the query takes too long to complete by default this is 10 seconds. | | 414 | Request-URI Too Large. Not all APIs have the same limit but it's usually 2KB for GET requests. Reduce the URI length or use a POST request to avoid the limit. | | 429 | Too many requests. Usually occurs when an API Endpoint is hitting into rate limits. | | 499 | Connection closed. This occurs if the client closes the connection after 1 second, if this is unexpected increase the connection timeout on your end. | | 500 | Internal Server Error. Usually an unexpected transient service error. | Errors when running a query are usually reported as 400 Bad request or 500 Internal Server Error, depending on whether the error can be fixed by the caller or not. In those cases the API response has an additional HTTP header, `X-DB-Exception-Code` where you can check the internal database error, reported as a stringified number. For a full list of internal database errors, see [List of API Endpoint database errors](https://www.tinybird.co/docs/list-of-errors). ### Retries [¶](https://www.tinybird.co/docs/about:blank#retries) When implementing an API Gateway, make sure to handle potential errors and implement retry strategies where appropriate. Implement automatic retries for the following errors: - HTTP 429: Too many requests - HTTP 500: Internal Server Error Follow an exponential backoff when retrying requests that produce the previous errors. See [Exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) in Wikipedia. ### Token limitations with API Gateways [¶](https://www.tinybird.co/docs/about:blank#token-limitations-with-api-gateways) When using an API Gateway or proxy between your application and Tinybird, your proxy uses a Token to authenticate requests to Tinybird. Treat the Token as a secret and don't expose it to the client. Use a service such as Unkey to add multi-tenant API keys, rate limiting, and token usage analytics to your app at scale. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read more about how to use Tokens in the[ Tokens docs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . - Read the guide:[ "Consume APIs in a Next.js frontend with JWTs"](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-apis-nextjs) . - Understand[ Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . --- URL: https://www.tinybird.co/docs/monitoring/service-datasources Last update: 2025-02-17T10:28:12.000Z Content: --- title: "Service Data Sources · Tinybird Docs" theme-color: "#171612" description: "In addition to the Data Sources you upload, Tinybird provides other "Service Data Sources" that allow you to inspect what's going on in your account." --- # Service Data Sources [¶](https://www.tinybird.co/docs/about:blank#service-data-sources) Tinybird provides Service Data Sources that you can use to inspect what's going on in your Tinybird account, diagnose issues, monitor usage, and so on. For example, you can get real time stats about API calls or a log of every operation over your Data Sources. This is similar to using system tables in a database, although Service Data Sources contain information about the usage of the service itself. Queries made to Service Data Sources are free of charge and don't count towards your usage. However, calls to API Endpoints that use Service Data Sources do count towards API rate limits. See [Billing](https://www.tinybird.co/docs/docs/get-started/plans/billing). ## Considerations [¶](https://www.tinybird.co/docs/about:blank#considerations) - You can't use Service Data Sources in Materialized View queries. - Pass dynamic[ query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters#leverage-dynamic-parameters) to API Endpoints to then query Service Data Sources. - You can only query Organization-level Service Data Sources if you're an administrator. See[ Consumption overview](https://www.tinybird.co/docs/docs/get-started/administration/organizations#consumption-overview) . ## Service Data Sources [¶](https://www.tinybird.co/docs/about:blank#service-data-sources) The following Service Data Sources are available. ### tinybird.pipe_stats_rt [¶](https://www.tinybird.co/docs/about:blank#tinybird-pipe-stats-rt) Contains information about all requests made to your [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints) in real time. This Data Source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at [tinybird.pipe_stats](https://www.tinybird.co/docs/about:blank#tinybird-pipe-stats). Calls made against Service Data Sources are not logged and don't count towards usage limits. | Field | Type | Description | | --- | --- | --- | | `start_datetime` | `DateTime` | API call start date and time. | | `pipe_id` | `String` | Pipe Id as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) ( `query_api` in case it's a Query API request). | | `pipe_name` | `String` | Pipe name as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) ( `query_api` in case it's a Query API request). | | `duration` | `Float` | API call duration, in seconds. | | `read_bytes` | `UInt64` | API call read data, in bytes. | | `read_rows` | `UInt64` | API call rows read. | | `result_rows` | `UInt64` | Rows returned by the API call. | | `url` | `String` | URL ( `token` param is removed for security reasons). | | `error` | `UInt8` | `1` if query returned error, else `0` . | | `request_id` | `String` | API call identifier returned in `x-request-id` header. Format is ULID string. | | `token` | `String` | API call token identifier used. | | `token_name` | `String` | API call token name used. | | `status_code` | `Int32` | API call returned status code. | | `method` | `String` | API call method POST or GET. | | `parameters` | `Map(String, String)` | API call parameters used. | | `release` | `String` | Semantic version of the release (deprecated). | | `user_agent` | `Nullable(String)` | User Agent HTTP header from the request. | | `resource_tags` | `Array(String)` | Tags associated with the Pipe when the request was made. | | `cpu_time` | `Float` | CPU time used by the query, in seconds. | ### tinybird.pipe_stats [¶](https://www.tinybird.co/docs/about:blank#tinybird-pipe-stats) Aggregates the request stats in [tinybird.pipe_stats_rt](https://www.tinybird.co/docs/about:blank#tinybird-pipe-stats-rt) by day. Calls made against Service Data Sources are not logged and don't count towards usage limits. | Field | Type | Description | | --- | --- | --- | | `date` | `Date` | Request date and time. | | `pipe_id` | `String` | Pipe Id as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) . | | `pipe_name` | `String` | Name of the Pipe. | | `view_count` | `UInt64` | Request count. | | `error_count` | `UInt64` | Number of requests with error. | | `avg_duration_state` | `AggregateFunction(avg, Float32)` | Average duration state, in seconds (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `quantile_timing_state` | `AggregateFunction(quantilesTiming(0.9, 0.95, 0.99), Float64)` | 0.9, 0.95 and 0.99 quantiles state. Time, in milliseconds (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `read_bytes_sum` | `UInt64` | Total bytes read. | | `read_rows_sum` | `UInt64` | Total rows read. | | `resource_tags` | `Array(String)` | All the tags associated with the resource when the aggregated requests were made. | ### tinybird.bi_stats_rt [¶](https://www.tinybird.co/docs/about:blank#tinybird-bi-stats-rt) Contains information about all requests to your [BI Connector interface](https://www.tinybird.co/docs/docs/work-with-data/query/bi-connector) in real time. This Data Source has a TTL of 7 days. If you need to query data older than 7 days you must use the aggregated by day data available at [tinybird.bi_stats](https://www.tinybird.co/docs/about:blank#tinybird-bi-stats). | Field | Type | Description | | --- | --- | --- | | `start_datetime` | `DateTime` | Query start timestamp. | | `query` | `String` | Executed query. | | `query_normalized` | `String` | Normalized executed query. This is the pattern of the query, without literals. Useful to analyze usage patterns. | | `error_code` | `Int32` | Error code, if any. `0` on normal execution. | | `error` | `String` | Error description, if any. Empty otherwise. | | `duration` | `UInt64` | Query duration, in milliseconds. | | `read_rows` | `UInt64` | Read rows. | | `read_bytes` | `UInt64` | Read bytes. | | `result_rows` | `UInt64` | Total rows returned. | | `result_bytes` | `UInt64` | Total bytes returned. | ### tinybird.bi_stats [¶](https://www.tinybird.co/docs/about:blank#tinybird-bi-stats) Aggregates the stats in [tinybird.bi_stats_rt](https://www.tinybird.co/docs/about:blank#tinybird-bi-stats-rt) by day. | Field | Type | Description | | --- | --- | --- | | `date` | `Date` | Stats date. | | `database` | `String` | Database identifier. | | `query_normalized` | `String` | Normalized executed query. This is the pattern of the query, without literals. Useful to analyze usage patterns. | | `view_count` | `UInt64` | Requests count. | | `error_count` | `UInt64` | Error count. | | `avg_duration_state` | `AggregateFunction(avg, Float32)` | Average duration state, in milliseconds (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `quantile_timing_state` | `AggregateFunction(quantilesTiming(0.9, 0.95, 0.99), Float64)` | 0.9, 0.95 and 0.99 quantiles state. Time, in milliseconds (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `read_bytes_sum` | `UInt64` | Total bytes read. | | `read_rows_sum` | `UInt64` | Total rows read. | | `avg_result_rows_state` | `AggregateFunction(avg, Float32)` | Total bytes returned state (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `avg_result_bytes_state` | `AggregateFunction(avg, Float32)` | Total rows returned state (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | ### tinybird.block_log [¶](https://www.tinybird.co/docs/about:blank#tinybird-block-log) The Data Source contains details about how Tinybird ingests data into your Data Sources. You can use this Service Data Source to spot problematic parts of your data. | Field | Type | Description | | --- | --- | --- | | `timestamp` | `DateTime` | Date and time of the block ingestion. | | `import_id` | `String` | Id of the import operation. | | `job_id` | `Nullable(String)` | Id of the job that ingested the block of data, if it was ingested by URL. In this case, `import_id` and `job_id` must have the same value. | | `request_id` | `String` | Id of the request that performed the operation. In this case, `import_id` and `job_id` must have the same value. Format is ULID string. | | `source` | `String` | Either the URL or `stream` or `body` keywords. | | `block_id` | `String` | Block identifier. You can cross this with the `blocks_ids` column from the[ tinybird.datasources_ops_log](https://www.tinybird.co/docs/about:blank#tinybird-datasources-ops-log) Service Data Source. | | `status` | `String` | `done` | `error` . | | `datasource_id` | `String` | Data Source consistent id. | | `datasource_name` | `String` | Data Source name when the block was ingested. | | `start_offset` | `Nullable(Int64)` | The starting byte of the block, if the ingestion was split, where this block started. | | `end_offset` | `Nullable(Int64)` | If split, the ending byte of the block. | | `rows` | `Nullable(Int32)` | How many rows it ingested. | | `parser` | `Nullable(String)` | Whether the native block parser or falling back to row by row parsing is used. | | `quarantine_lines` | `Nullable(UInt32)` | If any, how many rows went into the quarantine Data Source. | | `empty_lines` | `Nullable(UInt32)` | If any, how many empty lines were skipped. | | `bytes` | `Nullable(UInt32)` | How many bytes the block had. | | `processing_time` | `Nullable(Float32)` | How long it took, in seconds. | | `processing_error` | `Nullable(String)` | Detailed message in case of error. | When Tinybird ingests data from a URL, it splits the download in several requests, resulting in different ingestion blocks. The same happens when the data upload happens with a multipart request. ### tinybird.datasources_ops_log [¶](https://www.tinybird.co/docs/about:blank#tinybird-datasources-ops-log) Contains all operations performed to your Data Sources. Tinybird tracks the following operations: | Event | Description | | --- | --- | | `create` | A Data Source is created. | | `sync-dynamodb` | Initial synchronization from a DynamoDB table when using the[ DynamoDB Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb) | | `append` | Append operation. | | `append-hfi` | Append operation using the[ High-frequency Ingestion API](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api) . | | `append-kafka` | Append operation using the[ Kafka Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/kafka) . | | `append-dynamodb` | Append operation using the[ DynamoDB Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb) | | `replace` | A replace operation took place in the Data Source. | | `delete` | A delete operation took place in the Data Source. | | `truncate` | A truncate operation took place in the Data Source. | | `rename` | The Data Source was renamed. | | `populateview-queued` | A populate operation was queued for execution. | | `populateview` | A finished populate operation (up to 8 hours after it started). | | `copy` | A copy operation took place in the Data Source. | | `alter` | An alter operation took place in the Data Source. | Materializations are logged with same `event_type` and `operation_id` as the operation that triggers them. You can track the materialization Pipe with `pipe_id` and `pipe_name`. Tinybird logs all operations with the following information in this Data Source: | Field | Type | Description | | --- | --- | --- | | `timestamp` | `DateTime` | Date and time when the operation started. | | `event_type` | `String` | Operation being logged. | | `operation_id` | `String` | Groups rows affected by the same operation. Useful for checking materializations triggered by an append operation. | | `datasource_id` | `String` | Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. | | `datasource_name` | `String` | Name of your Data Source when the operation happened. | | `result` | `String` | `ok` | `error` | | `elapsed_time` | `Float32` | How much time the operation took, in seconds. | | `error` | `Nullable(String)` | Detailed error message if the result was error. | | `import_id` | `Nullable(String)` | Id of the import operation, if data has been ingested using one of the following operations: `create` , `append` or `replace` | | `job_id` | `Nullable(String)` | Id of the job that performed the operation, if any. If data has been ingested, `import_id` and `job_id` must have the same value. | | `request_id` | `String` | Id of the request that performed the operation. If data has been ingested, `import_id` and `request_id` must have the same value. Format is ULID string. | | `rows` | `Nullable(UInt64)` | How many rows the operations affected. This depends on `event_type` : for the `append` event, how many rows got inserted; for `delete` or `truncate` events, how many rows the Data Source had; for `replace` , how many rows the Data Source has after the operation. | | `rows_quarantine` | `Nullable(UInt64)` | How many rows went into the quarantine Data Source, if any. | | `blocks_ids` | `Array(String)` | List of blocks ids used for the operation. See the[ tinybird.block_log](https://www.tinybird.co/docs/about:blank#tinybird-block-log) Service Data Source for more details. | | `options` | `Nested(Names String, Values String)` | Tinybird stores key-value pairs with extra information for some operations. For the `replace` event, Tinybird uses the `rows_before_replace` key to track how many rows the Data Source had before the replacement happened, the `replace_condition` key shows what condition was used. For `append` and `replace` events, Tinybird stores the data `source` , for example the URL, or body/stream keywords. For `rename` event, `old_name` and `new_name` . For `populateview` you can find there the whole populate `job` metadata as a JSON string. For `alter` events, Tinybird stores `operations` , and dependent pipes as `dependencies` if they exist. | | `read_bytes` | `UInt64` | Read bytes in the operation. | | `read_rows` | `UInt64` | Read rows in the operation. | | `written_rows` | `UInt64` | Written rows in the operation. | | `written_bytes` | `UInt64` | Written bytes in the operation. | | `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. | | `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. | | `pipe_id` | `String` | If present, materialization Pipe id as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) . | | `pipe_name` | `String` | If present, materialization Pipe name as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) . | | `release` | `String` | Semantic version of the release (deprecated). | | `resource_tags` | `Array(String)` | Tags associated with the Pipe when the request was made. | | `cpu_time` | `Float32` | CPU time used by the operation, in seconds. | | `memory_usage` | `UInt64` | Memory consuptiom by the operation, in bytes. | ### tinybird.datasource_ops_stats [¶](https://www.tinybird.co/docs/about:blank#tinybird-datasource-ops-stats) Data from `datasource_ops_log` , aggregated by day. | Field | Type | Description | | --- | --- | --- | | `event_date` | `Date` | Date of the event. | | `workspace_id` | `String` | Unique identifier for the Workspace. | | `event_type` | `String` | Name of your Data Source. | | `pipe_id` | `String` | Identifier of the Pipe. | | `pipe_name` | `String` | Name of the Pipe. | | `error_count` | `UInt64` | Number of requests with an error. | | `executions` | `UInt64` | Number of executions. | | `avg_elapsed_time_state` | `Float32` | Average time spent in elapsed state. | | `quantiles_state` | `Float32` | 0.9, 0.95 and 0.99 quantiles state. Time in milliseconds (see[ Querying _state columns](https://www.tinybird.co/docs/about:blank#querying-state-columns) ). | | `read_bytes` | `UInt64` | Read bytes in the operation. | | `read_rows` | `UInt64` | Read rows in the Sink operation. | | `written_rows` | `UInt64` | Written rows in the Sink operation. | | `read_bytes` | `UInt64` | Read bytes in the operation. | | `written_bytes` | `UInt64` | Written bytes in the operation. | | `written_rows_quarantine` | `UInt64` | Quarantined rows in the operation. | | `written_bytes_quarantine` | `UInt64` | Quarantined bytes in the operation. | | `resource_tags` | `Array(String)` | Tags associated with the Pipe when the request was made. | ### tinybird.endpoint_errors [¶](https://www.tinybird.co/docs/about:blank#tinybird-endpoint-errors) It provides the last 30 days errors of your published endpoints. Tinybird logs all errors with additional information in this Data Source. | Field | Type | Description | | --- | --- | --- | | `start_datetime` | `DateTime` | Date and time when the API call started. | | `request_id` | `String` | The id of the request that performed the operation. Format is ULID string. | | `pipe_id` | `String` | If present, Pipe id as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) . | | `pipe_name` | `String` | If present, Pipe name as returned in our[ Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) . | | `params` | `Nullable(String)` | URL query params included in the request. | | `url` | `Nullable(String)` | URL pathname. | | `status_code` | `Nullable(Int32)` | HTTP error code. | | `error` | `Nullable(String)` | Error message. | | `resource_tags` | `Array(String)` | Tags associated with the Pipe when the request was made. | ### tinybird.kafka_ops_log [¶](https://www.tinybird.co/docs/about:blank#tinybird-kafka-ops-log) Contains all operations performed to your Kafka Data Sources during the last 30 days. | Field | Type | Description | | --- | --- | --- | | `timestamp` | `DateTime` | Date and time when the operation took place. | | `datasource_id` | `String` | Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. | | `topic` | `String` | Kafka topic. | | `partition` | `Int16` | Partition number, or `-1` for all partitions. | | `msg_type` | `String` | 'info' for regular messages, 'warning' for issues related to the user's Kafka cluster, deserialization or Materialized Views, and 'error' for other issues. | | `lag` | `Int64` | Number of messages behind for the partition. This is the difference between the high-water mark and the last commit offset. | | `processed_messages` | `Int32` | Messages processed for a topic and partition. | | `processed_bytes` | `Int32` | Amount of bytes processed. | | `committed_messages` | `Int32` | Messages ingested for a topic and partition. | | `msg` | `String` | Information in the case of warnings or errors. Empty otherwise. | ### tinybird.datasources_storage [¶](https://www.tinybird.co/docs/about:blank#tinybird-datasources-storage) Contains stats about your Data Sources storage. Tinybird logs maximum values per hour, the same as when it calculates storage consumption. | Field | Type | Description | | --- | --- | --- | | `datasource_id` | `String` | Id of your Data Source. The Data Source id is consistent after renaming operations. You should use the id when you want to track name changes. | | `datasource_name` | `String` | Name of your Data Source. | | `timestamp` | `DateTime` | When storage was tracked. By hour. | | `bytes` | `UInt64` | Max number of bytes the Data Source has, not including quarantine. | | `rows` | `UInt64` | Max number of rows the Data Source has, not including quarantine. | | `bytes_quarantine` | `UInt64` | Max number of bytes the Data Source has in quarantine. | | `rows_quarantine` | `UInt64` | Max number of rows the Data Source has in quarantine. | ### tinybird.releases_log (deprecated) [¶](https://www.tinybird.co/docs/about:blank#tinybird-releases-log-deprecated) Contains operations performed to your releases. Tinybird tracks the following operations: | Event | Description | | --- | --- | | `init` | First Release is created on Git sync. | | `override` | Release commit is overridden. `tb init --override-commit {{commit}}` . | | `deploy` | Resources from a commit are deployed to a Release. | | `preview` | Release status is changed to preview. | | `promote` | Release status is changed to live. | | `post` | Resources from a commit are deployed to the live Release. | | `rollback` | Rollback is done a previous Release is now live. | | `delete` | Release is deleted. | Tinybird logs all operations with additional information in this Data Source. | Field | Type | Description | | `timestamp` | `DateTime64` | Date and time when the operation took place. | | `event_type` | `String` | Name of your Data Source. | | `semver` | `String` | Semantic version identifies a release. | | `commit` | `String` | Git sha commit related to the operation. | | `token` | `String` | API call token identifier used. | | `token_name` | `String` | API call token name used. | | `result` | `String` | `ok` | `error` | | `error` | `String` | Detailed error message. | ### tinybird.sinks_ops_log [¶](https://www.tinybird.co/docs/about:blank#tinybird-sinks-ops-log) Contains all operations performed to your Sink Pipes. | Field | Type | Description | | `timestamp` | `DateTime64` | Date and time when the operation took place. | | `service` | `LowCardinality(String)` | Type of Sink (GCS, S3, and so on). | | `pipe_id` | `String` | The ID of the Sink Pipe. | | `pipe_name` | `String` | the name of the Sink Pipe. | | `token_name` | `String` | Token name used. | | `result` | `LowCardinality(String)` | `ok` | `error` | | `error` | `Nullable(String)` | Detailed error message. | | `elapsed_time` | `Float64` | The duration of the operation in seconds. | | `job_id` | `Nullable(String)` | ID of the job that performed the operation, if any. | | `read_rows` | `UInt64` | Read rows in the Sink operation. | | `written_rows` | `UInt64` | Written rows in the Sink operation. | | `read_bytes` | `UInt64` | Read bytes in the operation. | | `written_bytes` | `UInt64` | Written bytes in the operation. | | `output` | `Array(String)` | The outputs of the operation. In the case of writing to a bucket, the name of the written files. | | `parameters` | `Map(String, String)` | The parameters used. Useful to debug the parameter query values. | | `options` | `Map(String, String)` | Extra information. You can access the values with `options['key']` where key is one of: file_template, file_format, file_compression, bucket_path, execution_type. | | `cpu_time` | `Float64` | The CPU time used by the sinks, in seconds. | ### tinybird.data_transfer [¶](https://www.tinybird.co/docs/about:blank#tinybird-data-transfer) Stats of data transferred per hour by a Workspace. | Field | Type | Description | | `timestamp` | `DateTime` | Date and time data transferred is tracked. By hour. | | `event` | `LowCardinality(String)` | Type of operation generated the data (ie. `sink` ). | | `origin_provider` | `LowCardinality(String)` | Provider data was transferred from. | | `origin_region` | `LowCardinality(String)` | Region data was transferred from. | | `destination_provider` | `LowCardinality(String)` | Provider data was transferred to. | | `destination_region` | `LowCardinality(String)` | Region data was transferred to. | | `kind` | `LowCardinality(String)` | `intra` | `inter` depending if the data moves within or outside the region. | ### tinybird.jobs_log [¶](https://www.tinybird.co/docs/about:blank#tinybird-jobs-log) Contains all job executions performed in your Workspace. Tinybird logs all jobs with extra information in this Data Source: | Field | Type | Description | | --- | --- | --- | | `job_id` | `String` | Unique identifier for the job. | | `job_type` | `LowCardinality(String)` | Type of job execution. `delete_data` , `import` , `populateview` , `query` , `copy` , `copy_from_main` , `copy_from_branch` , `data_branch` , `deploy_branch` , `regression_tests` , `sink` , `sink_from_branch` . | | `workspace_id` | `String` | Unique identifier for the Workspace. | | `pipe_id` | `String` | Unique identifier for the Pipe. | | `pipe_name` | `String` | Name of the Pipe. | | `created_at` | `DateTime` | Timestamp when the job was created. | | `updated_at` | `DateTime` | Timestamp when the job was last updated. | | `started_at` | `DateTime` | Timestamp when the job execution started. | | `status` | `LowCardinality(String)` | Current status of the job. `waiting` , `working` , `done` , `error` , `cancelled` . | | `error` | `Nullable(String)` | Detailed error message if the result was error. | | `job_metadata` | `JSON String` | Additional metadata related to the job execution. | Learn more about how to track background jobs execution in the [Jobs monitoring guide](https://www.tinybird.co/docs/docs/monitoring/jobs). ## Use resource_tags to better track usage [¶](https://www.tinybird.co/docs/about:blank#use-resource-tags-to-better-track-usage) You can use tags that you've added to your resources, like Pipes or Data Sources, to analyze usage and cost attribution across your organization. For example, you can add tags for projects, environments, or versions and compare usage in later queries to Service Data Sources such as [tinybird.datasources_ops_stats](https://www.tinybird.co/docs/about:blank#tinybird-datasources-ops-stats) , which aggregates operations data by day. The following Service Data Sources support `resource_tags`: - `pipe_stats_rt` - `pipe_stats` - `endpoint_errors` - `organization.pipe_stats_rt` - `organization.pipe_stats` - `datasources_ops_log` - `datasources_ops_stats` - `organization.datasources_ops_log` - `organization.datasources_ops_stats` To add tags to resources, see [Organizing resources in Workspaces](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/organizing-resources). ## Query _state columns [¶](https://www.tinybird.co/docs/about:blank#query-state-columns) Several of the Service Data Sources include columns suffixed with `_state` . This suffix identifies columns with values that are in an intermediate aggregated state. When reading these columns, merge the intermediate states to get the final value. To merge intermediate states, wrap the column in the original aggregation function and apply the `-Merge` combinator. For example, to finalize the value of the `avg_duration_state` column, you use the `avgMerge` function: ##### finalize the value for the avg_duration_state column SELECT date, avgMerge(avg_duration_state) avg_time, quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_ms_array FROM tinybird.pipe_stats where pipe_id = 'PIPE_ID' group by date See [Combinators](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#aggregate-function-combinators) to learn more about the `-Merge` combinator. ## Organization Service Data Sources [¶](https://www.tinybird.co/docs/about:blank#organization-service-data-sources) The following is a complete list of available Organization Service Data Sources: | Field | Description | | --- | --- | | `organization.workspaces` | Lists all Organization Workspaces and related information, including name, IDs, databases, plan, when it was created, and whether it has been soft-deleted. | | `organization.processed_data` | Information related to all processed data per day per workspace. | | `organization.datasources_storage` | Equivalent to tinybird.datasources_storage but with data for all Organization Workspaces. | | `organization.pipe_stats` | Equivalent to tinybird.pipe_stats but with data for all Organization Workspaces. | | `organization.pipe_stats_rt` | Equivalent to tinybird.pipe_stats_rt but with data for all Organization Workspaces. | | `organization.datasources_ops_log` | Equivalent to tinybird.datasources_ops_log but with data for all Organization Workspaces. | | `organization.data_transfer` | Equivalent to tinybird.data_transfer but with data for all Organization Workspaces. | | `organization.jobs_log` | Equivalent to tinybird.jobs_log but with data for all Organization Workspaces. | | `organization.sinks_ops_log` | Equivalent to tinybird.sinks_ops_log but with data for all Organization Workspaces. | | `organization.bi_stats` | Equivalent to tinybird.bi_stats but with data for all Organization Workspaces. | | `organization.bi_stats_rt` | Equivalent to tinybird.bi_stats_rt but with data for all Organization Workspaces. | | `organization.endpoint_errors` | Equivalent to tinybird.endpoint_errors but with data for all Organization Workspaces. | To query Organization Service Data Sources, go to any Workspace that belongs to the Organization and use the previous as regular Service Data Source from the Playground or within Pipes. Use the admin `Token of an Organization Admin` . You can also copy your admin Token and make queries using your preferred method, like `tb sql`. ### metrics_logs Service Data Source [¶](https://www.tinybird.co/docs/about:blank#metrics-logs-service-data-source) The `metrics_logs` Service Data Source is available in all the organization's workspaces. As with the rest of Organization Service Data Sources, it's only available to Organization administrators. New records for each of the metrics monitored are added every minute with the following schema: | Field | Type | Description | | --- | --- | --- | | timestamp | DateTime | Timestamp of the metric | | cluster | LowCardinality(String) | Name of the cluster | | host | LowCardinality(String) | Name of the host | | metric | LowCardinality(String) | Name of the metric | | value | String | Value of the metric | | description | LowCardinality(String) | Description of the metric | | organization_id | String | ID of your organization | The available metrics are the following: | Metric | Description | | --- | --- | | MemoryTracking | Total amount of memory, in bytes, allocated by the server. | | OSMemoryTotal | The total amount of memory on the host system, in bytes. | | InstanceType | Instance type of the host. | | Query | Number of executing queries. | | NumberCPU | Number of CPUs. | | LoadAverage1 | The whole system load, averaged with exponential smoothing over 1 minute. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. | | LoadAverage15 | The whole system load, averaged with exponential smoothing over 15 minutes. The load represents the number of threads across all the processes (the scheduling entities of the OS kernel), that are currently running by CPU or waiting for IO, or ready to run but not being scheduled at this point of time. This number includes all the processes, not only the server. The number can be greater than the number of CPU cores, if the system is overloaded, and many processes are ready to run but waiting for CPU or IO. | | CPUUsage | The ratio of time the CPU core was running OS kernel (system) code or userspace code. This is a system-wide metric, it includes all the processes on the host machine, not just the server. This includes also the time when the CPU was under-utilized due to the reasons internal to the CPU (memory loads, pipeline stalls, branch mispredictions, running another SMT core). | --- URL: https://www.tinybird.co/docs/monitoring/monitor-data-ingestion Last update: 2025-01-21T08:14:33.000Z Content: --- title: "Monitor ingestion · Tinybird Docs" theme-color: "#171612" description: "In this guide you'll learn more about how to monitor your data source ingestion in Tinybird." --- # Monitor your ingestion [¶](https://www.tinybird.co/docs/about:blank#monitor-your-ingestion) In this guide, you can learn the basics of how to monitor your Data Source ingestion. By being aware of your ingestion pipeline and leveraging Tinybird's features, you can monitor for any issues with the [Data Flow Graph](https://www.tinybird.co/docs/docs/work-with-data/query#data-flow). Remember: Every Tinybird use case is slightly different. This guide provides guidelines and an example scenario. If you have questions or want to explore more complicated ingestion monitoring scenarios, for instance looking for outliers by using the z-score or other anomaly detection processes, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). ## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start) You don't need an active Workspace to follow this guide, only an awareness of the [core Tinybird concepts](https://www.tinybird.co/docs/docs/get-started/quick-start/core-concepts). ## Key takeaways [¶](https://www.tinybird.co/docs/about:blank#key-takeaways) 1. Understand and visualize your data pipeline. 2. Leverage the Tinybird platform and tools. 3. Be proactive: Build alerts. ### Understand your data pipeline and flow [¶](https://www.tinybird.co/docs/about:blank#understand-your-data-pipeline-and-flow) The first step to monitoring your ingestion to Tinybird is to understand what you're monitoring at a high level. When stakeholders complain about outdated data, you and your data engineers start investigating and checking the data pipelines upstream until you find the problem. Understanding how data flows through those pipelines from the origin to the end is essential, and you should always know what your data flow looks like. ### Use built-in tools [¶](https://www.tinybird.co/docs/about:blank#use-built-in-tools) Tinybird provides several tools to help you: - The[ Data Flow Graph](https://www.tinybird.co/docs/docs/work-with-data/query#data-flow) is Tinybird's data lineage diagram. It visualizes how data flows within your project. It shows all the levels of dependencies, so you can see how all your Pipes, Data Sources, and Materialized Views connect. - [ Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) are logs that allow you to keep track of almost everything happening data-wise within your system. - Use[ Time Series](https://www.tinybird.co/docs/docs/work-with-data/query#time-series) in combination with Service Data Sources to allow you to visualize data ingestion trends and issues over time. ### Build alerts [¶](https://www.tinybird.co/docs/about:blank#build-alerts) You can create a personalized alert system by integrating your Pipes and Endpoints that point to certain key Service Data Sources with third-party services. ## Identify high data volume resources [¶](https://www.tinybird.co/docs/about:blank#identify-high-data-volume-resources) You can run a query to find the resources that handle the most data by using the `datasources_ops_log` Service Data Source. For example: WITH ( SELECT sum(read_bytes + written_bytes) FROM tinybird.datasources_ops_log WHERE event_type NOT IN ('populateview', 'populateview-queued') -- AND timestamp >= now() - INTERVAL 24 HOUR -- optional time filter ) AS total, sum(read_bytes) AS processed_read, sum(written_bytes) AS processed_written SELECT datasource_name, pipe_name AS materializing_pipe, formatReadableSize(processed_read) AS read, formatReadableSize(processed_written) AS written, formatReadableSize(processed_read + processed_written) AS total_processed, (processed_read + processed_written) / total AS percentage FROM tinybird.datasources_ops_log WHERE event_type NOT IN ('populateview', 'populateview-queued') -- AND timestamp >= now() - INTERVAL 24 HOUR -- optional time filter GROUP BY datasource_name, materializing_pipe ORDER BY percentage DESC ## Example scenario [¶](https://www.tinybird.co/docs/about:blank#example-scenario) In this example, a user with a passion for ornithology has built a Workspace called `bird_spotter` . They're using it to analyze the number of birds they spot in their garden and when out on hikes. It uses Tinybird's high frequency ingestion (Events API) and an updated legacy table in BigQuery, so the Data Sources are as follows: 1. `bird_records` : A dataset containing bird viewings describing the time and bird details, which the[ Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) populates every day: 2. `birds_by_hour_and_country_from_copy` : An aggregated dataset of the bird views per hour and country, which a[ Copy Pipe](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) populates every hour: 3. `tiny_bird_records` : A dataset with a list of tiny birds (i.e. hummingbirds), which Tinybird's[ BigQuery Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/bigquery) replaces every day: The three Data Sources rely on three different methods of ingestion: appending data using the high frequency API, aggregating and copying, and syncing from BigQuery. To make sure that each of these processes is happening at the scheduled time, and without errors, the user needs to implement some monitoring. ### Monitoring ingestion and spotting errors [¶](https://www.tinybird.co/docs/about:blank#monitoring-ingestion-and-spotting-errors) The user in the example can filter the **Service Data Source** called `datasource_ops_log` by Data Source and ingestion method. By building a quick **Time Series** , they can immediately see the "shape" of their ingestion. The user can then build a robust system for monitoring. Instead of only focusing on the ingestion method, they can create 3 different Pipes that have specific logic, and expose each Pipe as a queryable Endpoint. Each Endpoint aggregates key information about each ingestion method, and count and flag errors. #### Endpoint 1: Check `append-hfi` operations in `bird_records` [¶](https://www.tinybird.co/docs/about:blank#endpoint-1-check-operations-in) SELECT toDate(timestamp) as date, sum(if(result = 'error', 1, 0)) as error_count, count() as append_count, if(append_count > 0, 1, 0) as append_flag FROM tinybird.datasources_ops_log WHERE datasource_name = 'bird_records' AND event_type = 'append-hfi' GROUP BY date ORDER BY date DESC #### Endpoint 2: Check `copy operations` in `birds_by_hour_and_country_from_copy` [¶](https://www.tinybird.co/docs/about:blank#endpoint-2-check-in) SELECT toDate(timestamp) as date, sum(if(result = 'error', 1, 0)) as error_count, count() as copy_count, if(copy_count >= 24, 1, 0) as copy_flag FROM tinybird.datasources_ops_log WHERE datasource_name = 'birds_by_hour_and_country_from_copy' AND event_type = 'copy' GROUP BY date ORDER BY date DESC #### Endpoint 3: Check `replace` operations in `tiny_bird_records` [¶](https://www.tinybird.co/docs/about:blank#endpoint-3-check-operations-in) SELECT toDate(timestamp) as date, sum(if(result = 'error', 1, 0)) as error_count, count() as replace_count, if(replace_count > 0, 1, 0) as replace_flag FROM tinybird.datasources_ops_log WHERE datasource_name = 'tiny_bird_records' AND event_type = 'replace' GROUP BY date ORDER BY date DESC ### Using the output [¶](https://www.tinybird.co/docs/about:blank#using-the-output) Because the Pipes expose API Endpoints, they can be consumed by any third-party app to build real-time alerts. The preferred way to integrate Tinybird data with your monitoring and alerting tools is by [exporting Pipe API Endpoints in Prometheus format](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format) , as it provides seamless compatibility with tools like Prometheus, Grafana or Datadog. Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. ### Example GitHub Actions implementation [¶](https://www.tinybird.co/docs/about:blank#example-github-actions-implementation) In the `bird_spotter` example repo , you can see the `scripts` and `workflows` that the user has built: - `ingest.py` and `monitor.py` are Python scripts that run daily. The first ingests data in this case from a sample csv and the second checks if the append, copy, and sync operations have happened and are error-free. Because this guide is an example scenario, there's a function that randomly chooses not to ingest, so there's always an error present. - `ingest.yml` and `monitor.yml` are yaml files that schedule those daily runs. The output of a daily check would look something like this: INFO:__main__:Alert! Ingestion operation missing. Last ingestion date isn't today: 2024-04-16 INFO:__main__:Last copy_count count is equal to 9. All fine! INFO:__main__:Last replace_count count is equal to 1. All fine! INFO:__main__:Alerts summary: INFO:__main__:Append error count: 1 INFO:__main__:Copy error count: 0 INFO:__main__:Replace error count: 0 In this instance, the ingestion script has randomly failed to append new data, and triggers an alert that the user can action. In contrast, copy operations and replace counts have run as expected: 9 copies and 1 BigQuery sync occurred since 00:00. ## Example scenario: Detect out-of-sync Data Sources [¶](https://www.tinybird.co/docs/about:blank#example-scenario-detect-out-of-sync-data-sources) Some Tinybird Connectors use async jobs to keep your Data Sources up to date. These jobs produce records with the result sent to the `datasources_ops_log` Service Data Source, both for successful and failed runs. The following example configures a new Tinybird Endpoint that reports Data Sources that are out of sync. It's then possible to leverage that data in your monitoring tool of choice, such as Grafana, Datadog, UptimeRobot, and others. ### Endpoint: Get out of sync Data Sources using `datasources_ops_log` [¶](https://www.tinybird.co/docs/about:blank#endpoint-get-out-of-sync-data-sources-using) To get the Data Sources that haven't been successfully updated in the last hour, check their sync jobs results in the `datasources_ops_log`: select datasource_id, argMax(datasource_name, timestamp) as datasource_name, max(case when result = 'ok' then timestamp end) as last_successful_sync from tinybird.datasources_ops_log where arrayExists(x -> x in ('bigquery'), Options.Values) and toDate(timestamp) >= today() - interval 30 days and result = 'ok' group by datasource_id having max(event_type = 'delete') = false and last_successful_sync < now() - interval 1 hour ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read the in-depth docs on Tinybird's[ Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . - Want to use Prometheus or Datadog?[ Consume API Endpoints in Prometheus format](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format) . - Learn how to[ Optimize your data project](https://www.tinybird.co/docs/docs/work-with-data/optimization) . - Learn about the difference between log* analytics* and log* analysis* in the blog[ "Log Analytics: how to identify trends and correlations that Log Analysis tools can't"](https://www.tinybird.co/blog-posts/log-analytics-how-to-identify-trends-and-correlations-that-log-analysis-tools-cannot) . --- URL: https://www.tinybird.co/docs/monitoring/latency Last update: 2024-12-17T18:51:47.000Z Content: --- title: "Measure API Endpoint latency · Tinybird Docs" theme-color: "#171612" description: "Latency is an essential metric to monitor in real-time applications. Learn how to measure and monitor the latency of your API Endpoints in Tinybird." --- # Measure API Endpoint latency [¶](https://www.tinybird.co/docs/about:blank#measure-api-endpoint-latency) Latency is the time it takes for a request to travel from the client to the server and back; the time it takes for a request to be sent and received. Latency is usually measured in seconds or milliseconds (ms). The lower the latency, the faster the response time. Latency is an essential metric to monitor in real-time applications. Read on to learn how latency is measured in Tinybird, and how to monitor and visualize the latency of your [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints) when data is being retrieved. ## How latency is measured [¶](https://www.tinybird.co/docs/about:blank#how-latency-is-measured) When measuring latency in an end-to-end application, you consider data ingestion, data transformation, and data retrieval. In Tinybird, latency is measured as the time it takes for a request to be sent, and the response to be sent back to the client. When calling an API Endpoint, you can check this metric defined as `elapsed` in the `statistics` object of the response: ##### Statistics object within an example Tinybird API Endpoint call { "meta": [ ... ], "data": [ ... ], "rows": 10, "statistics": { "elapsed": 0.001706275, "rows_read": 10, "bytes_read": 180 } } ## Monitor latency [¶](https://www.tinybird.co/docs/about:blank#monitor-latency) To monitor the latency of your API Endpoints, use the `pipe_stats_rt` and `pipe_stats` [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources): - `pipe_stats_rt` consists of the real-time statistics of your API Endpoints, and has a `duration` field that encapsulates the latency time in seconds. - `pipe_stats` contains the** aggregated** statistics of your API Endpoints by date, and presents a `avg_duration_state` field which is the average duration of the API Endpoint by day in seconds. Because the `avg_duration_state` field is an intermediate state, you'd need to merge it when querying the Data Source using something like `avgMerge`. Dor details on building Pipes and Endpoints that monitor the performance of your API Endpoints using the `pipe_stats_rt` and `pipe_stats` Data Sources, follow the [API Endpoint performance guide](https://www.tinybird.co/docs/docs/monitoring/analyze-endpoints-performance#example-2-analyzing-the-performance-of-api-endpoints-over-time). ## Visualize latency [¶](https://www.tinybird.co/docs/about:blank#visualize-latency) Tinybird has built-in tools to help you visualize the latency of your API Endpoints: Time Series live internally in your Workspace, while Charts give you the option to embed in an external application. ### Time series [¶](https://www.tinybird.co/docs/about:blank#time-series) In your Workspace, you can create [a Time series](https://www.tinybird.co/docs/docs/work-with-data/query#time-series) to visualize the latency of your API Endpoints over time. Point to `pipe_stats_rt` and select `duration` and `start_datetime` , or point to or `pipe_stats` and select `avgMerge(avg_duration_state)` and `date`. ### Charts [¶](https://www.tinybird.co/docs/about:blank#charts) If you want to expose your latency metrics in your own application, you can use a Tinybird-generated [Chart](https://www.tinybird.co/docs/docs/publish/charts) to expose the results of an Endpoint that queries the `pipe_stats_rt` or `pipe_stats` Data Source. Then, you can embed the Chart into your application by using the `iframe` code. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Optimize even further by[ monitoring your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) . - Read this blog on[ Monitoring global API latency](https://www.tinybird.co/blog-posts/dev-qa-global-api-latency-chronark) . --- URL: https://www.tinybird.co/docs/monitoring/jobs Last update: 2024-12-17T18:51:47.000Z Content: --- title: "Monitor jobs in your Workspace · Tinybird Docs" theme-color: "#171612" description: "Many of the operations you can run in your Workspace are executed using jobs. jobs_log provides you with an overview of all your jobs." --- # Monitor jobs in your Workspace [¶](https://www.tinybird.co/docs/about:blank#monitor-jobs-in-your-workspace) Many operations in your Tinybird Workspace, like Imports, Copy Jobs, Sinks, and Populates, are executed as background jobs within the platform. This approach ensures that the system can handle a large volume of requests efficiently without causing timeouts or delays in your workflow. Monitoring and managing jobs, for example querying job statuses, types, and execution details, is essential for maintaining a healthy Workspace. The two mechanisms for generic job monitoring are the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api) and the `jobs_log` Data Source. You can also track more specific things using dedicated Service Data Sources, such as `datasources_ops_log` for import, replaces, or copy, or `sinks_ops_log` for sink operations, or tracking jobs across [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) with `organization.jobs_log` . See [Service Data Sources docs](https://www.tinybird.co/docs/docs/monitoring/service-datasources). The Jobs API and the `jobs_log` return identical information about job execution. However, the Jobs API has some limitations: It reports only on a single Workspace, returns only 100 records, from the last 48 hours. If you want to monitor jobs outside these parameters, use the `jobs_log` Data Source. ## Track a specific job [¶](https://www.tinybird.co/docs/about:blank#track-a-specific-job) The most elemental use case is to track a specific job. You can do this using the Jobs API or SQL queries. ### Jobs API [¶](https://www.tinybird.co/docs/about:blank#jobs-api-track-specific-job) The Jobs API is a convenient way to programmatically check the status of a job. By sending a GET request, you can retrieve detailed information about a specific job. This method is particularly useful for integration into scripts or applications. curl \ -X GET "https://$TB_HOST/v0/jobs/{job_id}" \ -H "Authorization: Bearer $TOKEN" Replace `{job_id}` with the actual job ID. Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace. ### SQL queries [¶](https://www.tinybird.co/docs/about:blank#sql-queries-track-specific-job) Alternatively, you can use SQL to query the `jobs_log` Data Source from directly within a Tinybird Pipe. This method is ideal for users who are comfortable with SQL and prefer to run queries directly against the data, and then expose them with an endpoint or perform any other actions with it. SELECT * FROM tinybird.jobs_log WHERE job_id='{job_id}' Replace `{job_id}` with the desired job ID. This query retrieves all columns for the specified job, providing comprehensive details about its execution. ## Track specific job types [¶](https://www.tinybird.co/docs/about:blank#track-specific-job-types) Tracking jobs by type lets you monitor and analyze all jobs of a certain category, such as all `copy` jobs. This can help you understand the performance and status of specific job types across your entire Workspace. ### Jobs API [¶](https://www.tinybird.co/docs/about:blank#jobs-api-track-specific-job-types) You can fetch all jobs of a specific type by making a GET request against the Jobs API: curl \ -X GET "https://$TB_HOST/v0/jobs?kind=copy" \ -H "Authorization: Bearer $TOKEN" Replace `copy` with the type of job you want to track. Make sure you have set your Tinybird host ( `$TB_HOST` ) and authorization token ( `$TOKEN` ) correctly. ### SQL queries [¶](https://www.tinybird.co/docs/about:blank#sql-queries-track-specific-job-types) Alternatively, you can run an SQL query to fetch all jobs of a specific type from the `jobs_log` Data Source: SELECT * FROM tinybird.jobs_log WHERE job_type='copy' Replace `copy` with the desired job type. This query retrieves all columns for jobs of the specified type. ## Track ongoing jobs [¶](https://www.tinybird.co/docs/about:blank#track-ongoing-jobs) To keep track of jobs that are currently running, you can query the status of jobs in progress. This helps in monitoring the real-time workload and managing system performance. ### Jobs API [¶](https://www.tinybird.co/docs/about:blank#jobs-api-ongoing-jobs) By making an HTTP GET request to the Jobs API, you can fetch all jobs that are currently in the `working` status: curl \ -X GET "https://$TB_HOST/v0/jobs?status=working" \ -H "Authorization: Bearer $TOKEN" This call retrieves jobs that are actively running. Ensure you have set your Tinybird host ( `$TB_HOST` ) and authorization token ( `$TOKEN` ) correctly. ### SQL queries [¶](https://www.tinybird.co/docs/about:blank#sql-queries-ongoing-jobs) You can also use an SQL query to fetch currently running jobs from the `jobs_log` Data Source: SELECT * FROM tinybird.jobs_log WHERE status='working' This query retrieves all columns for jobs with the status `working` , allowing you to monitor ongoing operations. ## Track errored jobs [¶](https://www.tinybird.co/docs/about:blank#track-errored-jobs) Tracking errored jobs is crucial for identifying and resolving issues that may arise during job execution. Jobs API or SQL queries to `jobs_log` helps you monitor jobs that errored during the execution. ### Jobs API [¶](https://www.tinybird.co/docs/about:blank#jobs-api-errored-jobs) You can use the Jobs API to fetch details of jobs that have ended in error. Use the following `curl` command to retrieve all jobs that have a status of `error`: curl \ -X GET "https://$TB_HOST/v0/jobs?status=error" \ -H "Authorization: Bearer $TOKEN" This call fetches a list of jobs that are currently in an errored state, providing details that can be used for further analysis or debugging. Make sure you've set your Tinybird host ( `$TB_HOST` ) and authorization token ( `$TOKEN` ) correctly. ### SQL queries [¶](https://www.tinybird.co/docs/about:blank#sql-queries-errored-jobs) Alternatively, you can use SQL to query the `jobs_log` Data Source directly. Use the following SQL query to fetch job IDs, job types, and error messages for jobs that have encountered errors in the past day: SELECT job_id, job_type, error FROM tinybird.jobs_log WHERE status='error' AND created_at > now() - INTERVAL 1 DAY ### Track success rate [¶](https://www.tinybird.co/docs/about:blank#track-success-rate) Extrapolating from errored jobs, you can also use `jobs_log` to calculate the success rate of your Workspace jobs: SELECT job_type, pipe_id, countIf(status='done') AS job_success, countIf(status='error') AS job_error, job_success / (job_success + job_error) as success_rate FROM tinybird.jobs_log WHERE created_at > now() - INTERVAL 1 DAY GROUP BY job_type, pipe_id ## Get job execution metadata [¶](https://www.tinybird.co/docs/about:blank#get-job-execution-metadata) In the `jobs_log` Data Source, there is a property called `job_metadata` that contains metadata related to job executions. This includes the execution type, manual or scheduled, for Copy and Sink jobs, or the count of quarantined rows for append operations, along with many other properties. You can extract and analyze this metadata using JSON functions within SQL queries. This allows you to gain valuable information about job executions directly from the `jobs_log` Data Source. The following SQL query is an example of how to extract specific metadata fields from the `job_metadata` property, such as the import mode and counts of quarantined rows and invalid lines, and how to aggregate this data for analysis: SELECT job_type, JSONExtractString(job_metadata, 'mode') AS import_mode, sum(simpleJSONExtractUInt(job_metadata, 'quarantine_rows')) AS quarantine_rows, sum(simpleJSONExtractUInt(job_metadata, 'invalid_lines')) AS invalid_lines FROM tinybird.jobs_log WHERE job_type='import' AND created_at >= toStartOfDay(now()) GROUP BY job_type, import_mode There are many other use cases you can put together with the properties in the `job_metadata` ; see below. ## Advanced use cases [¶](https://www.tinybird.co/docs/about:blank#advanced-use-cases) Beyond basic tracking, you can leverage the `jobs_log` Data Source for more advanced use cases, such as gathering statistics and performance metrics. This can help you optimize job scheduling and resource allocation. ### Get queue status [¶](https://www.tinybird.co/docs/about:blank#get-queue-status) The following SQL query returns the number of jobs that are waiting to be executed, the number of jobs that are in progress, and how many of them are done already: SELECT job_type, countIf(status='waiting') AS jobs_in_queue, countIf(status='working') AS jobs_in_progress, countIf(status='done') AS jobs_succeeded, countIf(status='error') AS jobs_errored FROM tinybird.jobs_log WHERE created_at > now() - INTERVAL 1 DAY GROUP BY job_type ### Run time statistics grouped by type of job [¶](https://www.tinybird.co/docs/about:blank#run-time-statistics-grouped-by-type-of-job) The following SQL query calculates the maximum, minimum, median, and p95 running time (in seconds) grouped by type of job over the past day. This helps in understanding the efficiency of different job types: SELECT job_type, max(date_diff('s', started_at, updated_at)) as max_run_time_in_secs, min(date_diff('s', started_at, updated_at)) as min_run_time_in_secs, median(date_diff('s', started_at, updated_at)) as median_run_time_in_secs, quantile(0.95)(date_diff('s', started_at, updated_at)) as p95_run_time_in_secs FROM tinybird.jobs_log WHERE created_at > now() - INTERVAL 1 DAY GROUP BY job_type ### Statistics on queue time by type of job [¶](https://www.tinybird.co/docs/about:blank#statistics-on-queue-time-by-type-of-job) The following SQL query calculates the average queue time, in seconds, for a specific type of job over the past day. This can help in identifying bottlenecks in job scheduling: SELECT job_type, max(date_diff('s', created_at, started_at)) as max_run_time_in_secs, min(date_diff('s', created_at, started_at)) as min_run_time_in_secs, median(date_diff('s', created_at, started_at)) as median_run_time_in_secs, quantile(0.95)(date_diff('s', created_at, started_at)) as p95_run_time_in_secs FROM tinybird.jobs_log WHERE created_at > now() - INTERVAL 1 DAY GROUP BY job_type ### Get statistics on job completion rate [¶](https://www.tinybird.co/docs/about:blank#get-statistics-on-job-completion-rate) The following SQL query calculates the success rate by type of job (e.g., copy) and Pipe over the past day. This can help you to assess the reliability and efficiency of your workflows by measuring the completion rate of the jobs, and find potential issues and areas for improvement: SELECT job_type, pipe_id, countIf(status='done') AS job_success, countIf(status='error') AS job_error, job_success / (job_success + job_error) as success_rate FROM tinybird.jobs_log WHERE created_at > now() - INTERVAL 1 DAY GROUP BY job_type, pipe_id ### Statistics on the amount of manual vs. scheduled run jobs [¶](https://www.tinybird.co/docs/about:blank#statistics-on-the-amount-of-manual-vs-scheduled-run-jobs) The following SQL query calculates the percentage rate between manual and scheduled jobs. Understanding the distribution of manually-executed jobs versus scheduled jobs can let you know about some on-demand jobs performed for some specific reasons: SELECT job_type, countIf(JSONExtractString(job_metadata, 'execution_type')='manual') AS job_manual, countIf(JSONExtractString(job_metadata, 'execution_type')='scheduled') AS job_scheduled FROM tinybird.jobs_log WHERE job_type='copy' AND created_at > now() - INTERVAL 1 DAY GROUP BY job_type ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read up on the `jobs_log` Service Data Source specification . - Learn how to[ monitor your Workspace ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) . --- URL: https://www.tinybird.co/docs/monitoring/health-checks Last update: 2024-12-18T15:47:50.000Z Content: --- title: "Health checks · Tinybird Docs" theme-color: "#171612" description: "Use the built-in Tinybird tools to monitor your data ingestion and API Endpoint processes." --- # Check the health of your Data Sources [¶](https://www.tinybird.co/docs/about:blank#check-the-health-of-your-data-sources) After you have fixed all the possible errors in your source files, matched the Data Source schema to your needs, and done on-the-fly transformations, you can start ingesting data periodically. Knowing the status of your ingestion processes helps you to keep your data clean and consistent. ## Data Sources log [¶](https://www.tinybird.co/docs/about:blank#data-sources-log) From the **Data Sources log** in your Workspace overview, you can check whether there are new rows in quarantine, if jobs are failing, or if there is any other problem. ## Operations Log [¶](https://www.tinybird.co/docs/about:blank#operations-log) Select a Data Source to see the size of the Data Source, the number of rows, the number of rows in the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine) , and when it was last updated. The Operations log contains details of the events for the Data Source, which appears as the results of the query. ## Service Data Sources for continuous monitoring [¶](https://www.tinybird.co/docs/about:blank#service-data-sources-for-continuous-monitoring) [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) can help you with ingestion health checks. You can use them like any other Data Source in your Workspace, which means you can create API Endpoints to monitor your ingestion processes. Querying the 'tinybird.datasources_ops_log' directly, you can, for example, lists your ingest processes during the last week: ##### LISTING INGESTIONS IN THE LAST 7 DAYS SELECT * FROM tinybird.datasources_ops_log WHERE toDate(timestamp) > now() - INTERVAL 7 DAY ORDER BY timestamp DESC This query calculates the percentage of quarantined rows for a given period of time: ##### CALCULATE % OF ROWS THAT WENT TO QUARANTINE SELECT countIf(result != 'ok') / countIf(result == 'ok') * 100 percentage_failed, sum(rows_quarantine) / sum(rows) * 100 quarantined_rows FROM tinybird.datasources_ops_log The following query monitors the average duration of your periodic ingestion processes for a given Data Source: ##### CALCULATING AVERAGE INGEST DURATION SELECT avg(elapsed_time) avg_duration FROM tinybird.datasources_ops_log WHERE datasource_id = 't_8417d5126ed84802aa0addce7d1664f2' If you want to configure or build an external service that monitors these metrics, you need to create an API Endpoint and raise an alert when passing a threshold. When you receive an alert, you can check the quarantine Data Source or the Operations log to see what's going on and fix your source files or ingestion processes. ## Monitoring API Endpoints [¶](https://www.tinybird.co/docs/about:blank#monitoring-api-endpoints) You can use the 'pipe_stats' and 'pipe_stats_rt' [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) to monitor the performance of your API Endpoints. Every request to a Pipe is logged to 'tinybird.pipe_stats_rt' and kept in this Data Source for the last week. The following example API Endpoint aggregates the statistics for each hour for the selected Pipe. ##### PIPE_STATS_RT_BY_HR SELECT toStartOfHour(start_datetime) as hour, count() as view_count, round(avg(duration), 2) as avg_time, arrayElement(quantiles(0.50)(duration),1) as quantile_50, arrayElement(quantiles(0.95)(duration),1) as quantile_95, arrayElement(quantiles(0.99)(duration),1) as quantile_99 FROM tinybird.pipe_stats_rt WHERE pipe_id = 'PIPE_ID' GROUP BY hour ORDER BY hour 'pipe_stats' contains statistics about your Pipe Endpoints' API calls aggregated per day using intermediate states. ##### PIPE_STATS_BY_DATE SELECT date, sum(view_count) view_count, sum(error_count) error_count, avgMerge(avg_duration_state) avg_time, quantilesTimingMerge(0.9, 0.95, 0.99)(quantile_timing_state) quantiles_timing_in_millis_array FROM tinybird.pipe_stats WHERE pipe_id = 'PIPE_ID' GROUP BY date ORDER BY date You can use these API Endpoints to trigger alerts whenever statistics pass predefined thresholds. [Export API endpoint statistics in Prometheus format](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format) to integrate with your monitoring and alerting tools. To see how you can monitor Pipes and Data Sources health in a dashboard, see [Operational Analytics in Real Time with Tinybird and Retool](https://www.tinybird.co/blog-posts/service-data-sources-and-retool). --- URL: https://www.tinybird.co/docs/monitoring/analyze-endpoints-performance Last update: 2024-12-18T15:47:50.000Z Content: --- title: "Analyze the performance of your API Endpoints · Tinybird Docs" theme-color: "#171612" description: "Learn more about how to measure the performance of your API Endpoints." --- # Analyze the performance of your API Endpoints [¶](https://www.tinybird.co/docs/about:blank#analyze-the-performance-of-your-api-endpoints) You can use the `pipe_stats` and `pipe_stats_rt` Service Data Sources to analyze the performance of your API Endpoints. Read on to see several practical examples that show what you can do with these Data Sources. ## Knowing what to optimize [¶](https://www.tinybird.co/docs/about:blank#knowing-what-to-optimize) Before you optimize, you need to know what to optimize. The `pipe_stats` and `pipe_stats_rt` Service Data Sources let you see how your API Endpoints are performing, so you can find causes of overhead and improve performance. These Service Data Sources provide performance data and consumption data for every single request. You can also filter and sort results by Tokens to see who is accessing your API Endpoints and how often. The difference between `pipe_stats_rt` and `pipe_stats` is that `pipe_stats` provides aggregate stats, like average request duration and total read bytes, per day, whereas `pipe_stats_rt` offers the same information but without aggregation. Every single request is stored in `pipe_stats_rt` . The examples in this guide use `pipe_stats_rt` , but you can use the same logic with `pipe_stats` if you need more than 7 days of lookback. ## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start) You need a high-level understanding of Tinybird's [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources). ### Understand the core stats [¶](https://www.tinybird.co/docs/about:blank#understand-the-core-stats) This guide focuses on the following fields in the `pipe_stats_rt` Service Data Source: - `pipe_name` (String): Pipe name as returned in Pipes API. - `duration` (Float): the duration in seconds of each specific request. - `read_bytes` (UInt64): How much data was scanned for this particular request. - `read_rows` (UInt64): How many rows were scanned. - `token_name` (String): The name of the Token used in a particular request. - `status_code` (Int32): The HTTP status code returned for this particular request. You can find the full schema for `pipe_stats_rt` in the [API docs](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-pipe-stats-rt). The value of `pipe_name` is "query_api" in the event as it's a Query API request. The following section covers how to monitor query performance when using the Query API. ### Use the Query API with metadata parameters [¶](https://www.tinybird.co/docs/about:blank#use-the-query-api-with-metadata-parameters) If you are using the [Query API](https://www.tinybird.co/docs/docs/api-reference/query-api) to run queries in Tinybird you can still track query performance using the `pipe_stats_rt` Service Data Source. Add metadata related to the query as request parameters, as well as any existing parameters used in your query. For example, when running a query against the Query API you can leverage a parameter called `app_name` to track all queries from the "explorer" application. Here's an example using `curl`: ##### Using the metadata parameters with the Query API curl -X POST \ -H "Authorization: Bearer " \ --data "% SELECT * FROM events LIMIT {{Int8(my_limit, 10)}}" \ "https://api.tinybird.co/v0/sql?my_limit=10&app_name=explorer" When you run the following queries, use the `parameters` attribute to access those queries where `app_name` equals "explorer": ##### Simple Parameterized Query SELECT * FROM tinybird.pipe_stats_rt WHERE parameters['app_name'] = 'explorer' ## Detect errors in your API Endpoints [¶](https://www.tinybird.co/docs/about:blank#detect-errors-in-your-api-endpoints) If you want to monitor the number of errors per Endpoint over the last hour, you can run the following query: ##### Errors in the last hour SELECT pipe_name, status_code, count() as error_count FROM tinybird.pipe_stats_rt WHERE status_code >= 400 AND start_datetime > now() - INTERVAL 1 HOUR GROUP BY pipe_name, status_code ORDER BY status_code desc If you have errors, the query would return something like: Pipe_a | 404 | 127 Pipe_b | 403 | 32 With one query, you can see in real time if your API Endpoints are experiencing errors, and investigate further if so. ## Analyze the performance of API Endpoints over time [¶](https://www.tinybird.co/docs/about:blank#analyze-the-performance-of-api-endpoints-over-time) You can also use `pipe_stats_rt` to track how long API calls take using the `duration` field, and seeing how that changes over time. API performance is directly related to how much data you are reading per request, so if your API Endpoint is dynamic, request duration varies. For instance, it might be receiving start and end date parameters that alter how long a period is being read. ##### API Endpoint performance over time SELECT toStartOfMinute(start_datetime) t, pipe_name, avg(duration) avg_duration, quantile(.95)(duration) p95_duration, count() requests FROM tinybird.pipe_stats_rt WHERE start_datetime >= {{DateTime(start_date_time, '2022-05-01 00:00:00', description="Start date time")}} AND start_datetime < {{DateTime(end_date_time, '2022-05-25 00:00:00', description="End date time")}} GROUP BY t, pipe_name ORDER BY t desc, pipe_name ## Find the endpoints that process the most data [¶](https://www.tinybird.co/docs/about:blank#find-the-endpoints-that-process-the-most-data) You might want to find Endpoints that repeatedly scan large amounts of data. They are your best candidates for optimization to reduce time and spend. Here's an example of using `pipe_stats_rt` to find the API Endpoints that have processed the most data as a percentage of all processed data in the last 24 hours: ##### Most processed data last 24 hours WITH ( SELECT sum(read_bytes) FROM tinybird.pipe_stats_rt WHERE start_datetime >= now() - INTERVAL 24 HOUR ) as total, sum(read_bytes) as processed_byte SELECT pipe_id, quantile(0.9)(duration) as p90, formatReadableSize(processed_byte) AS processed_formatted, processed_byte*100/total as percentage FROM tinybird.pipe_stats_rt WHERE start_datetime >= now() - INTERVAL 24 HOUR GROUP BY pipe_id ORDER BY percentage DESC ### Include consumption of the Query API [¶](https://www.tinybird.co/docs/about:blank#include-consumption-of-the-query-api) If you use Tinybird's Query API to query your Data Sources directly, you probably want to include in your analysis which queries are consuming more. Whenever you use the Query API, the field `pipe_name` contain the value `query_api` . The actual query is included as part of the `q` parameter in the `url` field. You can modify the query in the previous section to extract the SQL query that's processing the data. ##### Using the Query API WITH ( SELECT sum(read_bytes) FROM tinybird.pipe_stats_rt WHERE start_datetime >= now() - INTERVAL 24 HOUR ) as total, sum(read_bytes) as processed_byte SELECT if(pipe_name = 'query_api', normalizeQuery(extractURLParameter(decodeURLComponent(url), 'q')),pipe_name) as pipe_name, quantile(0.9)(duration) as p90, formatReadableSize(processed_byte) AS processed_formatted, processed_byte*100/total as percentage FROM tinybird.pipe_stats_rt WHERE start_datetime >= now() - INTE RVAL 24 HOUR GROUP BY pipe_name ORDER BY percentage DESC ## Monitor usage of Tokens [¶](https://www.tinybird.co/docs/about:blank#monitor-usage-of-tokens) If you use your API Endpoint with different Tokens, for example if allowing different customers to check their own data, you can track and control which Tokens are being used to access these endpoints. The following example shows, for the last 24 hours, the number and size of requests per Token: ##### Token usage last 24 hours SELECT count() requests, formatReadableSize(sum(read_bytes)) as total_read_bytes, token_name FROM tinybird.pipe_stats_rt WHERE start_datetime >= now() - INTERVAL 24 HOUR GROUP BY token_name ORDER BY requests DESC To get this information, request the Token name ( `token_name` column) or id ( `token` column). Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Want to optimize further? Read[ Monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) . - Want to use Prometheus or Datadog?[ Consume API Endpoints in Prometheus format](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format) - Learn how to[ monitor jobs in your Workspace](https://www.tinybird.co/docs/docs/monitoring/jobs) . - Monitor the[ latency of your API Endpoints](https://www.tinybird.co/docs/docs/monitoring/latency) . - Learn how to[ build Charts of your data](https://www.tinybird.co/docs/docs/publish/charts) . --- URL: https://www.tinybird.co/docs/get-started/quick-start Content: --- title: "Get started with Tinybird · Tinybird Docs" theme-color: "#171612" description: "Get started with Tinybird as quickly as possible. Ingest, query, and publish data in minutes." --- # Get started with Tinybird [¶](https://www.tinybird.co/docs/about:blank#get-started-with-tinybird) With Tinybird, you can ingest data from anywhere, query and transform it using SQL, and publish it as high-concurrency, low-latency REST API endpoints. Read on to learn how to create a Workspace, ingest data, create a query, publish an API, and confirm your setup works properly using the Tinybird user interface. 1 ## Create your Tinybird account [¶](https://www.tinybird.co/docs/about:blank#create-your-tinybird-account) [Create a Tinybird account](https://www.tinybird.co/signup) . It's free and no credit card is required. See [Tinybird pricing plans](https://www.tinybird.co/docs/docs/get-started/plans/billing) for more information. [Sign up for Tinybird](https://www.tinybird.co/signup) 2 ## Select your cloud provider and region [¶](https://www.tinybird.co/docs/about:blank#select-your-cloud-provider-and-region) When logging in to Tinybird, select the cloud provider and region you want to work in. <-figure-> ![Select your region](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-region-select.png&w=3840&q=75) 3 ## Create your Workspace [¶](https://www.tinybird.co/docs/about:blank#create-your-workspace) A [Workspace](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) is an area that contains a set of Tinybird resources, including Data Sources, Pipes, nodes, API Endpoints, and Tokens. Create a Workspace named `customer_rewards` . The name must be unique. <-figure-> ![Create a Workspace](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-create-workspace.png&w=3840&q=75) 4 ## Download and ingest sample data [¶](https://www.tinybird.co/docs/about:blank#download-and-ingest-sample-data) Download the following sample data from a fictitious online coffee shop: [Download data file](https://www.tinybird.co/docs/docs/assets/sample-data-files/orders.ndjson) Select **File Upload** and follow the instructions to load the file. <-figure-> ![Upload a file with data](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-file-upload.png&w=3840&q=75) Select **Create Data Source** to automatically create the `orders` [Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources). <-figure-> ![Create a Data Source](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-create-data-source.png&w=3840&q=75) 5 ## Query data using a Pipe [¶](https://www.tinybird.co/docs/about:blank#query-data-using-a-pipe) You can create [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) to query your data using SQL. To create a Pipe, select **Pipes** and then **Create Pipe**. Name your Pipe `rewards` and add the following SQL: select count() from orders Select the node name and change it to `rewards_count`. <-figure-> ![Create a Pipe](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-create-pipe.png&w=3840&q=75) Select **Run** to preview the result of your Pipe. 6 ## Publish your query as an API [¶](https://www.tinybird.co/docs/about:blank#publish-your-query-as-an-api) You can turn any Pipe into a high-concurrency, low-latency API Endpoint. Select **Create API Endpoint** and then select the `rewards_count` node in the menu. <-figure-> ![Create an API Endpoint](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-create-api.png&w=3840&q=75) 7 ## Call your API [¶](https://www.tinybird.co/docs/about:blank#call-your-api) You can test your API endpoint using a curl command. Go to the **Output** section of the API page and select the **cURL** tab. Copy the curl command into a Terminal window and run it. <-figure-> ![Test your API Endpoint](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fquickstartui-test-api.png&w=3840&q=75) Congratulations! You have created your first API Endpoint in Tinybird. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Check the[ Tinybird CLI Quick start](https://www.tinybird.co/docs/docs/cli/quick-start) . - Try a template to get started quickly. See[ Templates](https://www.tinybird.co/templates) . - Learn more about[ User-Facing Analytics](https://www.tinybird.co/docs/docs/use-cases) in the Use Case Hub. - Learn about[ Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts) and build beautiful visualizations for your API endpoints. --- URL: https://www.tinybird.co/docs/get-started/plans Content: --- title: "Plans · Tinybird Docs" theme-color: "#171612" description: "Choose the Tinybird plan that fits your needs. Available plans are Free, Developer, and Enterprise." --- # Tinybird plans [¶](https://www.tinybird.co/docs/about:blank#tinybird-plans) Plans describe what your organization can do in Tinybird, the compute resources at your disposal, and how usage is billed. To learn about Tinybird pricing, see [Pricing](https://www.tinybird.co/pricing). ## Available plans [¶](https://www.tinybird.co/docs/about:blank#available-plans) Tinybird offers the following plan options: Free, Developer, and Enterprise. | Plan | What you get | When to upgrade | | --- | --- | --- | | Free | - All Tinybird features, with few limitations. - Use Tinybird without time limits. - You can upgrade at any time. | - You need more resources. - You're ready for production. - You want Tinybird support. | | Developer | - More resources and less limits. - You can select a machine size. - Can deal with load peaks. | - You need more performance. - You need wider limits. - You need private networking. | | Enterprise | - Get all the resources you need. - No limits in dedicated plans. - Detailed usage reports and tracking. | - You can adjust resources when needed. - You can add private networking. | You can upgrade from Free and Developer plans at any time. All plans include: - Unlimited seats - Unlimited source connections - Unlimited tables - Unlimited pipes - Unlimited endpoints - Multi-cloud, multi-region support - RBAC and Workspace admin controls. See[ Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) . - SOC2 Type II and HIPAA compliance. See[ Compliance](https://www.tinybird.co/docs/docs/get-started/compliance) . ## Free [¶](https://www.tinybird.co/docs/about:blank#free) The Free plan provides you with a production-grade instance of Tinybird, including ingest connectors, real time querying, and managed endpoints. Sinks are not available on the Free plan. See [Free plan limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#free-plan-limits) for more information on limits. ### Support [¶](https://www.tinybird.co/docs/about:blank#free-support) Support for the Free plan is available through the [Community Slack](https://www.tinybird.co/docs/docs/community) , which is monitored by the Tinybird team on a best-effort basis. ### Upgrade [¶](https://www.tinybird.co/docs/about:blank#free-upgrade) To upgrade to a Developer plan: 1. Go to** Organization settings** ,** Billing** . 2. Select** Change size** . Tinybird recommends a size based on your usage trends. 3. Select a bigger or a smaller size. Self service plans can be up to 3 vCPUs. 4. Confirm the resize. To upgrade to an Enterprise plan, [contact sales](https://www.tinybird.co/contact-us). ### Migrate from Build [¶](https://www.tinybird.co/docs/about:blank#free-migrate-from-build) Tinybird automatically detects your Build plan and offers to migrate to a new Free plan or to a Developer plan. When migrating from a Build plan, the following considerations: - You have to create an organization if you want to create a new Workspace. See[ Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) . - Limits for Workspaces not tied to an Organization remain the same. ## Developer [¶](https://www.tinybird.co/docs/about:blank#developer) The Developer plan provides you with a database within a large shared cluster. All customers on the cluster share resources. Developer plans include 25 GB of storage. When signing up for a Developer plan, you can select the size of the shared machine, up to 3 vCPUs. If you need more capacity, see [Enterprise](https://www.tinybird.co/docs/about:blank#enterprise) , which starts at 4 vCPUs. See [Billing, Developer plans](https://www.tinybird.co/docs/docs/get-started/plans/billing#developer-plans) for more information on how Developer plans are billed. See [Developer plan limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#developer-plan-limits) for more information on limits. ### Support [¶](https://www.tinybird.co/docs/about:blank#developer-support) The Developer plan includes standard support through [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co). ### Resize and upgrade [¶](https://www.tinybird.co/docs/about:blank#developer-resize-and-upgrade) You can resize Developer plans when needed. When increasing or reducing capacity, resizes are instantaneous. To resize your plan: 1. Go to** Organization settings** ,** Billing** . 2. Select** Change size** . Tinybird recommends a size based on your usage trends. 3. Select a bigger or a smaller size. Self service plans can be up to 3 vCPUs. 4. Confirm the resize. To upgrade to an Enterprise plan, [contact sales](https://www.tinybird.co/contact-us). You can resize your Developer plan only once every 24 hours. ### Downgrade [¶](https://www.tinybird.co/docs/about:blank#developer-downgrade) You can downgrade back to a Free plan by canceling your Developer plan. To cancel your Developer plan: 1. Go to** Organization settings** ,** Billing** . 2. Select** Cancel plan** . 3. Review the information and confirm. Remaining costs are charged automatically upon cancellation. After canceling your Developer plan, Free plan limits apply to your Workspaces. ### Migrate from Pro [¶](https://www.tinybird.co/docs/about:blank#developer-migrate-from-pro) Tinybird automatically detects your Pro plan and offers to migrate to a new Developer plan. When migrating from a Pro plan, you have to create an organization if you want to create a new Workspace. See [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations). To migrate to an Enterprise plan, [contact sales](https://www.tinybird.co/contact-us). After April 29, 2025, if you don't migrate your Pro plan to a new plan, it's downgraded to a free plan automatically. ## Enterprise [¶](https://www.tinybird.co/docs/about:blank#enterprise) The Enterprise plan provides you with a Tinybird cluster with at least two database servers. On a Enterprise plan, you're the only customer on your cluster. Your queries and outputs are more performant as a result. Minimum storage for Enterprise plans is 1 TB. The support package is also mandatory. See [Billing, Enterprise plans](https://www.tinybird.co/docs/docs/get-started/plans/billing#enterprise-plans) for more information on how dedicated plans are billed and how to monitor usage. ### Support [¶](https://www.tinybird.co/docs/about:blank#enterprise-support) Enterprise plans include Premium support, which includes: - Professional services hours. - A dedicated Slack channel. - Email support. ### Adjust resources [¶](https://www.tinybird.co/docs/about:blank#enterprise-adjust-resources) To adjust the limits and resources of your Enterprise plan or request a different size, [contact sales](https://www.tinybird.co/contact-us). ## Plan size estimation [¶](https://www.tinybird.co/docs/about:blank#plan-size-estimation) When you're offered to migrate or upgrade, Tinybird suggests a plan size based on the following: - vCPU usage: total active hours accrued in the last 30 days across all your Workspaces. - Queries per second: maximum queries per second reached in the last 30 days across all your Workspaces. Recommendations from Tinybird aggregate the operations of all your Workspaces. ### Estimate your plan size [¶](https://www.tinybird.co/docs/about:blank#estimate-your-plan-size) You can adapt and run the following queries in each of your Workspaces to check current usage and run your estimates: - vCPU - QPS SELECT toStartOfDay(minute) as date, sum(total_cpu_time_seconds) as sum_total_cpu_time_seconds, max(total_cpu_time_seconds) as max_total_cpu_time_seconds, (max_total_cpu_time_seconds/60) as max_cpu_time_minutes_per_minute FROM tinybird.estimated_shared_infra_cpu_time GROUP BY date ORDER BY date DESC ## Legacy plans [¶](https://www.tinybird.co/docs/about:blank#legacy-plans) Effective April 29, 2025, the Build, Pro, and Enterprise plans are no longer available for new users. If you're an existing users on paid legacy plans, you can migrate to Developer or Enterprise plans based on processed data and storage. - [ Migrate from Pro](https://www.tinybird.co/docs/about:blank#developer-migrate-from-pro) - [ Migrate from Build](https://www.tinybird.co/docs/about:blank#free-migrate-from-build) ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Familiarize yourself with[ key concepts](https://www.tinybird.co/docs/docs/get-started/plans/concepts) to better understand your bill and plan limits. - Read the[ billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing) to understand which data operations count towards your bill, and how to optimize your usage. - Learn about[ limits](https://www.tinybird.co/docs/docs/get-started/plans/limits) and how to adjust them. --- URL: https://www.tinybird.co/docs/get-started/integrations Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Integrations · Tinybird Docs" theme-color: "#171612" description: "Connect Tinybird to your database, data warehouse, streaming platform, devtools, and other applications." --- # Integrations [¶](https://www.tinybird.co/docs/about:blank#integrations) You can integrate Tinybird with various data sources, data sinks, and devtools to support your use case. - ** Native Integrations** are built and maintained by Tinybird, and integrated into the Tinybird product. - ** Guided Integrations** aren't built and maintained by Tinybird, but utilize native Tinybird APIs and/or functionality in the external tool. ## List of integrations [¶](https://www.tinybird.co/docs/about:blank#list-of-integrations) Native ## Amazon DynamoDB Use the DynamoDB Connector to ingest historical and change stream data from Amazon... [Docs](https://www.tinybird.co/docs/get-data-in/connectors/dynamodb) Guided ## Amazon Kinesis Learn how to send data from AWS Kinesis to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-aws-kinesis) Native ## Amazon S3 Use the S3 Connector to ingest files from your Amazon S3 buckets into Tinybird. [Source Docs](https://www.tinybird.co/docs/get-data-in/connectors/s3), [Sink Docs](https://www.tinybird.co/docs/publish/sinks/s3-sink) Guided ## Amazon SNS SNS is a popular pub/sub messaging system for AWS users. Here's how to use SNS to send... [Guide](https://www.tinybird.co/blog-posts/use-aws-sns-to-send-data-to-tinybird) Native ## Apache Kafka Use the Kafka Connector to ingest data streams from your Kafka cluster into Tinybird. [Source Docs](https://www.tinybird.co/docs/get-data-in/connectors/kafka), [Sink Docs](https://www.tinybird.co/docs/publish/sinks/kafka-sink) Guided ## Auth0 Log Streams Learn how to send Auth0 Logs Streams to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-auth0-logs) Guided ## Clerk With Clerk you can easily manage user auth. By integrating Clerk with Tinybird, you can... [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-clerk) Native ## Confluent Cloud Connect Tinybird to your Confluent Cloud cluster, select a topic, and Tinybird... [Docs](https://www.tinybird.co/docs/get-data-in/connectors/confluent) Guided ## Dub Learn how to connect Dub webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-dub) Guided ## Estuary Learn how to use Estuary to push data streams to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-with-estuary) Guided ## GitHub Learn how to connect GitHub Webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-github) Guided ## GitLab Learn how to connect GitLab Webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-gitlab) Native ## Google BigQuery Use the BigQuery Connector to load data from BigQuery into Tinybird. [Docs](https://www.tinybird.co/docs/get-data-in/connectors/bigquery) Guided ## Google Cloud Storage Learn how to automatically synchronize all the CSV files in a Google GCS bucket to a... [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-google-gcs) Guided ## Google Pub/Sub Learn how to send data from Google Pub/Sub to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-google-pubsub) Guided ## Grafana Learn how to create Grafana Dashboards and Alerts consuming Tinybird API Endpoints. [Guide](https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-api-endpoints-in-grafana) Guided ## Knock Learn how to connect Knock outbound webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-knock) Guided ## Mailgun Learn how to connect Mailgun webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-mailgun) Guided ## MongoDB Learn how to ingest data into Tinybird from MongoDB [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-mongodb) Guided ## MySQL A step-by-step guide to setting up Change Data Capture (CDC) with MySQL, Confluent Cloud,... [Guide](https://www.tinybird.co/blog-posts/mysql-cdc) Guided ## Orb Events Learn how to configure a Orb webhook to send events to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-orb) Guided ## PagerDuty Learn how to connect PagerDuty Webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-pagerduty) Native ## PostgreSQL The Tinybird postgresql() table function allows you to read data from your existing... [Docs](https://www.tinybird.co/docs/get-data-in/table-functions/postgresql) Native ## Prometheus Learn how to consume API Endpoints in Prometheus format. [Guide](https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format) Guided ## Python logs Learn how to send Python logs to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/python-sdk) Native ## Redpanda The Redpanda Connector allows you to ingest data from your existing Redpanda cluster and... [Docs](https://www.tinybird.co/docs/get-data-in/connectors/redpanda) Guided ## Resend With Resend you can send and receive emails programmatically. By integrating Resend with... [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-resend) Guided ## Rudderstack Learn two different methods to send events from RudderStack to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-rudderstack) Guided ## Sentry Learn how to connect Sentry Webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-sentry) Guided ## Snowflake Learn how to bring Snowflake data into Tinybird using AWS S3 or Azure Blob Storage. [S3 guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3), [Azure guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-snowflake-using-azure-blob-storage) Guided ## Stripe Learn how to connect Stripe webhooks to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-stripe) Guided ## Trigger.dev Learn how to reliably trigger Tinybird jobs with Trigger.dev. [Guide](https://www.tinybird.co/docs/publish/api-endpoints/guides/reliable-scheduling-with-trigger) Guided ## Vector.dev Learn how to ingest data from Vector.dev to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-vector) Guided ## Vercel Integration This integration will allow you to link your Tinybird Workspaces with your Vercel... [Guide](https://www.tinybird.co/docs/work-with-data/organize-your-work/integrating-vercel) Guided ## Vercel Log Drains Learn how to connect Vercel Log Drains to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-vercel-logdrains) Guided ## Vercel Webhooks Learn how to send Vercel events to Tinybird. [Guide](https://www.tinybird.co/docs/get-data-in/guides/ingest-from-vercel) --- URL: https://www.tinybird.co/docs/get-started/compliance Last update: 2024-12-12T09:38:12.000Z Content: --- title: "Compliance and certifications · Tinybird Docs" theme-color: "#171612" description: "Tinybird is committed to the highest data security and safety. See what compliance certifications are available." --- # Compliance and certifications [¶](https://www.tinybird.co/docs/about:blank#compliance-and-certifications) Data security and privacy are paramount in today's digital landscape. Tinybird's commitment to protecting your sensitive information is backed by the following compliance certifications, which ensure that we meet rigorous industry standards for data security, privacy, and operational excellence. ## SOC 2 Type II [¶](https://www.tinybird.co/docs/about:blank#soc-2-type-ii) Tinybird has obtained a SOC 2 Type II certification, in accordance with attestation standards established by the American Institute of Certified Public Accountants (AICPA), that are relevant to security, availability, processing integrity, confidentiality, and privacy for Tinybird's real-time platform for user-facing analytics. Compliance is monitored continually—with reports published annually—to confirm the robustness of Tinybird's data security. This independent assessment provides Tinybird users with assurance that their sensitive information is being handled responsibly and securely. ## HIPAA [¶](https://www.tinybird.co/docs/about:blank#hipaa) Tinybird supports its customers' Health Insurance Portability and Accountability Act (HIPAA) compliance efforts by offering Business Associate Agreements (BAAs). Additionally, Tinybird's offering allows customers to process their data constituting personal health information (PHI) in AWS, Azure, or Google Cloud—entities which themselves have entered into BAAs with Tinybird. ## Trust center [¶](https://www.tinybird.co/docs/about:blank#trust-center) To learn more about Tinybird security controls and certifications, visit the [Tinybird Trust Center](https://trust.tinybird.co/). --- URL: https://www.tinybird.co/docs/get-started/architecture Last update: 2025-01-20T11:46:12.000Z Content: --- title: "Architecture · Tinybird Docs" theme-color: "#171612" description: "Frequently asked questions about Tinybird architecture" --- # Architecture [¶](https://www.tinybird.co/docs/about:blank#architecture) Tinybird is the analytical backend for your applications: it consumes data from any source and exposes it through API Endpoints. Tinybird can sit parallel to the Data Warehouse or in front of it. The Data Warehouse allows to explore use cases like BI and data science, while Tinybird unlocks action use cases like operational applications, embedded analytics, and user-facing analytics. Read the guide ["Team integration and data governance"](https://www.tinybird.co/docs/docs/get-started/administration/team-integration-governance) to learn more about implementing Tinybird in your existing team. ## Is data stored in Tinybird? [¶](https://www.tinybird.co/docs/about:blank#is-data-stored-in-tinybird) The data you ingest into Tinybird is stored in a high performance, columnar OLAP database. Depending on the use case, you can add a TTL to control how long data is stored. ## Cloud environment vs on-premises [¶](https://www.tinybird.co/docs/about:blank#cloud-environment-vs-on-premises) Tinybird is a managed SaaS solution. Tinybird doesn't provide a "Bring Your Own Cloud" (BYOC) or on-premises deployments. If you are interested in a BYOC deployment, join the [Tinybird BYOC Waitlist](https://faster.tinybird.co/byoc-waitlist). ## Tinybird Local container [¶](https://www.tinybird.co/docs/about:blank#tinybird-local-container) You can run Tinybird locally for testing and development purposes using the [Tinybird Local container](https://www.tinybird.co/docs/docs/cli/local-container). ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Explore[ Tinybird's Customer Stories](https://www.tinybird.co/customer-stories) and see what people have built on Tinybird. - Start building using the[ quick start](https://www.tinybird.co/docs/docs/get-started/quick-start) . - Read the guide[ "Team integration and data governance"](https://www.tinybird.co/docs/docs/get-started/administration/team-integration-governance) to learn more about implementing Tinybird in your existing team. --- URL: https://www.tinybird.co/docs/get-started/administration Content: --- title: "Administration · Tinybird Docs" theme-color: "#171612" description: "Manage your organization and workspace settings, create tokens, invite users, and more." --- # Administration [¶](https://www.tinybird.co/docs/about:blank#administration) Effective administration of your Tinybird account involves managing several key components: - [ Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) : Create and manage organizations to group workspaces and team members. - [ Workspaces](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) : Set up isolated environments for your data projects. - [ Auth Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) : Generate and control access tokens for secure API authentication. - [ Governance](https://www.tinybird.co/docs/docs/get-started/administration/team-integration-governance) : Implement best practices for team collaboration and security --- URL: https://www.tinybird.co/docs/get-data-in/table-functions Content: --- title: "Table functions · Tinybird Docs" theme-color: "#171612" description: "Ingest data from remote databases into Tinybird." --- # Table functions [¶](https://www.tinybird.co/docs/about:blank#table-functions) Tinybird table functions allow you to read data from an existing database and schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables or incrementally sync your data. Tinybird supports the following table functions: - [ MySQL](https://www.tinybird.co/docs/docs/get-data-in/table-functions/mysql) - [ PostgreSQL](https://www.tinybird.co/docs/docs/get-data-in/table-functions/postgresql) - [ URL](https://www.tinybird.co/docs/docs/get-data-in/table-functions/url) ## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites) Your database needs to be open and public, exposed to the internet with publicly signed certs, so you can connect to it by passing the hostname, port, username, and password. You also need to be familiar with making cURL requests to [manage your secrets](https://www.tinybird.co/docs/about:blank#about-secrets). ### Environment Variables API [¶](https://www.tinybird.co/docs/about:blank#environment-variables-api) The Environment Variables API is currently only accessible at API level. Pasting your credentials into a Pipe node or datafile as plain text is a security risk. Instead, use the Environment Variables API to [create two new secrets](https://www.tinybird.co/docs/docs/api-reference/environment-variables-api#post-v0secrets) for your username and password. In the next step, you can interpolate your new secrets using the `tb_secret` function: {{tb_secret('db_username')}} {{tb_secret('db_password')}} ## Load an external table [¶](https://www.tinybird.co/docs/about:blank#load-an-external-table) Create a new Pipe node. Call the table function in the `FROM` and pass the connection details: ##### PostgreSQL table function example SELECT * FROM postgresql( 'aws-0-eu-central-1.TODO.com:3866', 'postgres', 'orders', {{tb_secret('db_username')}}, {{tb_secret('db_password')}}, ) Publish this node as a Copy Pipe, thereby running the query manually. You can choose to append only new data, or replace all data. ### Using datafiles [¶](https://www.tinybird.co/docs/about:blank#using-datafiles) You can also define node logic in [Pipe files](https://www.tinybird.co/docs/docs/cli/datafiles/pipe-files) . An example for a PostgreSQL eCommerce `orders_backfill` scenario, with a node called `all_orders` , would be: NODE all_orders SQL > % SELECT * FROM postgresql( 'aws-0-eu-central-1.TODO.com:3866', 'postgres', 'orders', {{tb_secret('db_username')}}, {{tb_secret('db_password')}}, ) TYPE copy TARGET_DATASOURCE orders COPY_SCHEDULE @on-demand COPY_MODE replace ## Include filters [¶](https://www.tinybird.co/docs/about:blank#include-filters) You can use a source column to filter by a value in Tinybird, for example: ##### Example Copy Pipe with PostgreSQL table function and filters SELECT * FROM postgresql( 'aws-0-eu-central-1.TODO.com:3866', 'postgres', 'orders', {{tb_secret('db_username')}}, {{tb_secret('db_password')}}, ) WHERE orderDate > (select max(orderDate) from orders) ## Schedule runs [¶](https://www.tinybird.co/docs/about:blank#schedule-runs) When publishing as a Copy Pipe, most users set it to run at a frequent interval using a cron expression. You can also trigger the Copy Pipe on demand: curl -H "Authorization: Bearer " \ -X POST "https:/tinybird.co/api/v0/pipes//run" Having on-demand Pipes in your Workspace is helpful, as you can run a full sync manually any time you need it. You might also use them for weekly full syncs. ## Synchronization strategies [¶](https://www.tinybird.co/docs/about:blank#synchronization-strategies) When copying data from your database to Tinybird, you can use one of the following strategies: - Use `COPY_MODE replace` to synchronize small dimensions tables, up to a few million rows, in a frequent schedule (1 to 5 minutes). - Use `COPY_MODE append` to do incremental appends. For example, you can append events data tagged with a timestamp. Combine it with `COPY_SCHEDULE` and filters in the Copy Pipe SQL to sync the new events. ### Timeouts [¶](https://www.tinybird.co/docs/about:blank#timeouts) When synchronizing dimensions tables with `COPY_MODE replace` and 1 minute schedule, the copy job might time out because it can't ingest the whole table in the defined schedule. Timeouts depend on several factors: - The timeout configured in your external database. - The external database load. - Network connectivity, for example when copying data from different cloud regions. Follow these tips to avoid timeouts using incremental appends: - Make sure to tag your data with an updated timestamp and use the column to filter the Copy Pipe SQL. - Configure the Copy Pipe with an incremental append strategy and 1 minute schedule. That way you make sure only new records in the last minute are ingested, thus optimizing the copy job duration. - Create an index in the external table to speed up filtering. - Create the target Data Source as a[ ReplacingMergeTree](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies#use-the-replacingmergetree-engine) using a unique or primary key as the `ENGINE_SORTING_KEY` . Rows with the same `ENGINE_SORTING_KEY` are deduplicated. Remember to use the `FINAL` keyword when querying the Data Source to force deduplication at query time. - Combine this approach with an hourly or daily replacement to get rid of deleted rows. Learn about[ how to handle deleted rows](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies#use-the-replacingmergetree-engine) when using `ReplacingMergeTree` . ## Observability [¶](https://www.tinybird.co/docs/about:blank#observability) Job executions are logged in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . You can check this log directly in the Data Source view page in the UI. Filter by `datasource_id` to monitor ingestion through the table functions from the `datasources_ops_log`: ##### Example query to the datasources_ops_log Service Data Source SELECT timestamp, event_type, result, error, job_id FROM tinybird.datasources_ops_log WHERE datasource_id = 't_1234' AND event_type = 'copy' ORDER BY timestamp DESC ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) The table functions inherit all the [limits of Copy Pipes](https://www.tinybird.co/docs/docs/get-started/plans/limits#copy-pipe-limits). Environment Variables are created at a Workspace level, so you can connect one of each external database per Tinybird Workspace. Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. ## Billing [¶](https://www.tinybird.co/docs/about:blank#billing) There are no additional or specific costs for the table function itself; only the costs associated with Copy Pipes apply. For more information on data operations and their charges, see the [billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing). --- URL: https://www.tinybird.co/docs/get-data-in/migrate Content: --- title: "Migrate data into Tinybird · Tinybird Docs" theme-color: "#171612" description: "Learn how to migrate data into Tinybird from other data platforms." --- # Migrate data into Tinybird [¶](https://www.tinybird.co/docs/about:blank#migrate-data-into-tinybird) Tinybird provides several options for migrating data from external platforms. Whether you're moving from a managed service like DoubleCloud, a real-time analytics platform like Rockset, or a traditional database like PostgreSQL, Tinybird offers migration paths to help you transition your data and workloads. Each migration guide provides detailed, step-by-step instructions for: - Moving your existing data into Tinybird. - Understanding how concepts and features map between platforms. - Setting up equivalent functionality in Tinybird. - Maintaining data consistency during the migration. The guides also cover important considerations like: - Data volume and performance requirements. - Schema management and data types. - Authentication and security. - Monitoring and observability. Select your current platform to get started with your migration to Tinybird. - [ Migrate from DoubleCloud](https://www.tinybird.co/docs/docs/get-data-in/migrate/migrate-from-doublecloud) - [ Migrate from Postgres](https://www.tinybird.co/docs/docs/get-data-in/migrate/migrate-from-postgres) - [ Migrate from Rockset](https://www.tinybird.co/docs/docs/get-data-in/migrate/migrate-from-rockset) --- URL: https://www.tinybird.co/docs/get-data-in/ingest-apis Content: --- title: "Ingest APIs · Tinybird Docs" theme-color: "#171612" description: "Ingest data into Tinybird using the Datasource and Events APIs." --- # Ingest APIs [¶](https://www.tinybird.co/docs/about:blank#ingest-apis) Tinybird provides the following APIs for ingesting data: - The** Data Sources API** lets you create and manage Data Sources, as well as import data from files (CSV, NDJSON, Parquet). You can use it to create new Data Sources from files, append data to existing Data Sources, or replace data selectively. The Data Sources API supports both local and remote files, with automatic schema inference for CSV files. - The** Events API** provides high-throughput streaming ingestion through a simple HTTP API. It's designed for sending JSON events individually or in batches using NDJSON format. The Events API is optimized for real-time data ingestion, supporting compression and write acknowledgements when needed. Both APIs require authentication using tokens with appropriate scopes. For detailed information about each API's capabilities and usage, see: - [ Data Sources API documentation](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/datasource-api) - [ Events API documentation](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) - [ Data Sources API Reference](https://www.tinybird.co/docs/docs/api-reference/datasource-api) - [ Events API Reference](https://www.tinybird.co/docs/docs/api-reference/events-api) --- URL: https://www.tinybird.co/docs/get-data-in/guides Content: --- title: "Ingest guides · Tinybird Docs" theme-color: "#171612" description: "Guides for ingesting data into Tinybird." --- # Ingest guides [¶](https://www.tinybird.co/docs/about:blank#ingest-guides) Tinybird provides multiple ways to bring data into the platform. While [native connectors](https://www.tinybird.co/docs/docs/get-data-in/connectors) offer the most streamlined experience, you can use the [Ingest APIs](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis) and other mechanisms to bring data from virtually any source. Each guide provides step-by-step instructions and best practices for setting up reliable data ingestion pipelines. Whether you're working with batch files, streaming events, or database synchronization, you can find examples of how to effectively bring that data into Tinybird. - [ Auth0](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-auth0-logs) - [ AWS Kinesis](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-aws-kinesis) - [ Clerk](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-clerk) - [ CSV files](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-csv-files) - [ Dub](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-dub) - [ DynamoDB Single-Table](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-dynamodb-single-table-design) - [ Estuary](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-with-estuary) - [ GitHub](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-github) - [ GitLab](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-gitlab) - [ Google Cloud Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-google-gcs) - [ Google Pub/Sub](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-google-pubsub) - [ HTTP Requests](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api) - [ Knock](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-knock) - [ Mailgun](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-mailgun) - [ MongoDB](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-mongodb) - [ NDJSON data](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data) - [ Orb](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-orb) - [ PagerDuty](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-pagerduty) - [ Postgres CDC with Redpanda Connect](https://www.tinybird.co/docs/docs/get-data-in/guides/postgres-cdc-with-redpanda-connect) - [ Python logs](https://www.tinybird.co/docs/docs/get-data-in/guides/python-sdk) - [ Resend](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-resend) - [ RudderStack](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-rudderstack) - [ Sentry](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-sentry) - [ Snowflake incremental updates](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-incremental-updates) - [ Snowflake using Azure Blob Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-azure-blob-storage) - [ Snowflake using S3](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3) - [ Stripe](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-stripe) - [ Vector.dev](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-vector) - [ Vercel (log drains)](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-vercel-logdrains) - [ Vercel (webhooks)](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-vercel) --- URL: https://www.tinybird.co/docs/get-data-in/data-sources Last update: 2025-01-29T13:58:37.000Z Content: --- title: "Data Sources · Tinybird Docs" theme-color: "#171612" description: "Data Sources contain all the data you bring into Tinybird, acting like tables in a database." --- # Data Sources [¶](https://www.tinybird.co/docs/about:blank#data-sources) When you get data into Tinybird, it's stored in a Data Source. You then write SQL queries to explore the data from the Data Source. Tinybird represents Data Sources using the icon. For example, if your event data lives in a Kafka topic, you can create a Data Source that connects directly to Kafka and writes the events to Tinybird. You can then [create a Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#creating-pipes-in-the-ui) to query fresh event data. A Data Source can also be the result of materializing a SQL query through a [Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#creating-pipes-in-the-ui). ## Create Data Sources [¶](https://www.tinybird.co/docs/about:blank#create-data-sources) You can use Tinybird's UI, CLI, and API to create Data Sources. ### Using the UI [¶](https://www.tinybird.co/docs/about:blank#create-data-sources-using-the-ui) Follow these steps to create a new Data Source: 1. In your Workspace, go to** Data Sources** . 2. Select** +** to add a new Data Source. ### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#create-data-sources-using-the-cli) You can create Data Source using the `tb datasource` command. See [tb datasource](https://www.tinybird.co/docs/docs/cli/command-ref#tb-datasource) in the CLI reference. ### Using the Events API [¶](https://www.tinybird.co/docs/about:blank#create-data-sources-using-the-events-api) If you send data to the Events API and the Data Source doesn't exist, the Events API creates a Data Source by guessing the types from the data you send. See [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api). You can still modify the Data Source using the alter options or adding a TTL as explained in the following section. ## Set the Data Source TTL [¶](https://www.tinybird.co/docs/about:blank#set-the-data-source-ttl) You can apply a TTL (Time To Live) to a Data Source in Tinybird. Use a TTL to define how long you want to store data. For example, you can define a TTL of 7 days, which means Tinybird automatically deletes data older than 7 days. You may set the TTL at the time of creating the Data Source, or set it later. Your data must have a column with a type that represents a date or datetime. Valid types are `Date` and `DateTime`. ### Using the UI [¶](https://www.tinybird.co/docs/about:blank#set-the-data-source-ttl-using-the-ui) Follow these steps to set a TTL using the Tinybird UI: 1. Select** Advanced Settings** . 2. Open the** TTL** menu. 3. Select a column that represents a date. 4. Define the TTL period in days. If you need to apply transformations to the date column, or want to use more complex logic, select the **Code editor** tab and enter SQL code to define your TTL. ### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#set-the-data-source-ttl-using-the-cli) Follow these steps to set a TTL using the Tinybird CLI: 1. Create a new Data Source and .datasource file using the `tb datasource` command. 2. Edit the .datasource file you've created. 3. Go to the Engine settings. 4. Add a new setting called `ENGINE_TTL` and enter your TTL string enclosed in double quotes. 5. Save the file. The following example shows a .datasource file with TTL defined: SCHEMA > `date` DateTime, `product_id` String, `user_id` Int64, `event` String, `extra_data` String ENGINE "MergeTree" ENGINE_PARTITION_KEY "toYear(date)" ENGINE_SORTING_KEY "date, user_id, event, extra_data" ENGINE_TTL "date + toIntervalDay(90)" ## Change Data Source TTL [¶](https://www.tinybird.co/docs/about:blank#change-data-source-ttl) You can modify the TTL of an existing Data Source, either by adding a new TTL or by updating an existing TTL. ### Using the UI [¶](https://www.tinybird.co/docs/about:blank#change-the-data-source-ttl-using-the-ui) Follow these steps to modify a TTL using the Tinybird UI: 1. Go to the Data Source details page by clicking on the Data Source with the TTL you wish to change. 2. Select the** Schema** tab. 3. Select the TTL text. 4. A dialog opens. Select the menu. 5. Select the field to use for the TTL. 6. Change the TTL interval. 7. Select** Save** . The updated TTL value appears in the Data Source's schema page. ### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#change-the-data-source-ttl-using-the-cli) Follow these steps to modify a TTL using the Tinybird CLI: 1. Open the .datasource file. 2. Go to the Engine settings. 3. If `ENGINE_TTL` doesn't exist, add it and enter your TTL enclosed in double quotes. 4. If a TTL is already defined, modify the existing setting. The following is an example TTL setting: ENGINE_TTL "date + toIntervalDay(90)" When ready, save the .datasource file and push the changes to Tinybird using the CLI: tb push DATA_SOURCE_FILE -f ## Share a Data Source [¶](https://www.tinybird.co/docs/about:blank#share-a-data-source) Workspace administrators can share a Data Source with another Workspace they've access to on the same region and cluster. To share a Data Source, follow these steps: 1. Find the Data Source you want to share inside** Data Project** . 2. Select the** More actions (⋯)** icon next to Data Source. 3. Select** Share** . 4. Type the Workspace name or ID. 5. Select** Share** . You can use the shared Data Source to create Pipes and Materialized Views in the target Workspace. Users that have access to a shared Data Source can access the `tinybird.datasources_ops_log` and the `tinybird.kafka_ops_log` Service Data Sources. ### Limitations [¶](https://www.tinybird.co/docs/about:blank#limitations) The following limitations apply to shared Data Sources: - Shared Data Sources are read-only. - You can't share a shared Data Source, only the original. - You can't check the quarantine of a shared Data Source. ## Supported engines and settings [¶](https://www.tinybird.co/docs/about:blank#supported-engines-and-settings) Workspace administrators can share a Data Source with another Workspace they've access to on the same region and cluster. To share a Data Source, follow these steps: 1. Find the Data Source you want to share inside** Data Project** . 2. Select the** More actions (⋯)** icon next to Data Source. 3. Select** Share** . 4. Type the Workspace name or ID. 5. Select** Share** . You can use the shared Data Source to create Pipes and Materialized Views in the target Workspace. Users that have access to a shared Data Source can access the `tinybird.datasources_ops_log` and the `tinybird.kafka_ops_log` Service Data Sources. ### Limitations [¶](https://www.tinybird.co/docs/about:blank#limitations) The following limitations apply to shared Data Sources: - Shared Data Sources are read-only. - You can't share a shared Data Source, only the original. - You can't check the quarantine of a shared Data Source. ## Supported engines [¶](https://www.tinybird.co/docs/about:blank#supported-engines) Tinybird features different strategies to store data, which define where and how the data is stored and also what kind of data access, queries, and availability your data has. A Tinybird Data Source uses a table engine that determines those factors. See [Engines](https://www.tinybird.co/docs/docs/sql-reference/engines). ## Supported data types [¶](https://www.tinybird.co/docs/about:blank#supported-data-types) Data types specify how Tinybird stores and processes values in a database. They determine what kind of data can fit in a column (like numbers, text, dates, etc.), how much storage space the data uses, and what operations you can perform on the values. Choosing the most appropriate data type is important for both data integrity and query performance. See [Data types](https://www.tinybird.co/docs/docs/sql-reference/data-types). ### Set a different codec [¶](https://www.tinybird.co/docs/about:blank#set-a-different-codec) Tinybird applies compression codecs to data types to optimize performance. You can override the default compression codecs by adding the `CODEC()` statement after the type declarations in your .datasource schema. For example: SCHEMA > `product_id` Int32 `json:$.product_id`, `timestamp` DateTime64(3) `json:$.timestamp` CODEC(DoubleDelta, ZSTD(1)), ## Supported file types and compression formats for ingest [¶](https://www.tinybird.co/docs/about:blank#supported-file-types-and-compression-formats-for-ingest) Tinybird supports these file types and compression formats at ingest time: | File type | Method | Accepted extensions | Compression formats supported | | --- | --- | --- | --- | | CSV | File upload, URL | `.csv` , `.csv.gz` | `gzip` | | NDJSON | File upload, URL, Events API | `.ndjson` , `.ndjson.gz` | `gzip` | | Parquet | File upload, URL | `.parquet` , `.parquet.gz` | `gzip` | | Avro | Kafka | | `gzip` | ## Quarantine Data Sources [¶](https://www.tinybird.co/docs/about:blank#quarantine-data-sources) Every Data Source you create in your Workspace has a quarantine Data Source associated that store data that doesn't fit the schema. If you send rows that don't fit the Data Source schema, they're automatically sent to the quarantine table so that the ingest process doesn't fail. By convention, quarantine Data Sources follow the naming pattern `{datasource_name}_quarantine` . You can review quarantined rows at any time or perform operations on them using Pipes. This is a useful source of information when fixing issues in the origin source or applying changes during ingest. The quarantine Data Source schema contains the columns of the original row and the following columns with information about the issues that caused the quarantine: - `c__error_column` Array(String) contains an array of all the columns that contain an invalid value. - `c__error` Array(String) contains an array of all the errors that caused the ingestion to fail and lead to store the values in quarantine. This column along the `c__error_column` allows you so easily identify which is the columns that has problems and which is the error. - `c__import_id` Nullable(String) contains the job's identifier in case the column was imported through a job. - `insertion_date` (DateTime) contains the timestamp in which the ingestion was done. See the [Quarantine guide](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine) for practical examples on using the quarantine Data Source. ## Partitioning [¶](https://www.tinybird.co/docs/about:blank#partitioning) Use partitions for data manipulation. Partitioning isn't intended to speed up `SELECT` queries: experiment with more efficient sorting keys, as defined by `ENGINE_SORTING_KEY` , for that. A bad partition key, or creating too many partitions, can negatively impact query performance. Configure partitioning using the `ENGINE_PARTITION_KEY` setting. When choosing a partition key: - Leave the `ENGINE_PARTITION_KEY` key empty. If the table is small or you aren't sure what the best partition key should be, leave it empty: Tinybird places all data in a single partition. - Use a date column. Depending on the filter, you can opt for more or less granularity based on your needs. `toYYYYMM(date_column)` or `toYear(date_column)` are valid default choices. Don't use too granular a partition key, like a customer ID or name. This could lead to the `TOO_MANY_PARTS` error. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) The following examples show how to define partitions. ##### Using an empty tuple to create a single partition ENGINE_PARTITION_KEY "tuple()" ##### Using a Date column to create monthly partitions ENGINE_PARTITION_KEY "toYYYYMM(date_column)" ##### Using a column to partition by event types ENGINE_PARTITION_KEY "event_type % 8" ### TOO_MANY_PARTS error [¶](https://www.tinybird.co/docs/about:blank#too-many-parts-error) Each insert operation creates a new part containing compressed data files and index files for each column. Tinybird merges smaller parts into bigger parts in the background based on specific rules. The goal is to maintain one large part, or a few large parts, per partition. The `TOO_MANY_PARTS` error happens when you insert data faster than Tinybird can merge the parts. Inserting data to many partitions at once multiplies the problem by the number of partitions. To prevent this error: - Batch your inserts into larger chunks instead of making many small inserts. - Limit the number of partitions you write to simultaneously . - Define a less granular partition key. If the error persists, contact [support](https://www.tinybird.co/docs/docs/get-started/plans/support). ## Upserts and deletes [¶](https://www.tinybird.co/docs/about:blank#upserts-and-deletes) See [this guide](https://www.tinybird.co/docs/docs/get-data-in/data-operations/replace-and-delete-data) . Depending on the frequency needed, you might want to convert upserts and deletes into an append operation that you can solve through [deduplication](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies). ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) There is a limit of 100 Data Sources per Workspace. --- URL: https://www.tinybird.co/docs/get-data-in/data-operations Content: --- title: "Data operations · Tinybird Docs" theme-color: "#171612" description: "Replace and delete data, recover data from quarantine, iterate a data source, and more." --- # Data operations [¶](https://www.tinybird.co/docs/about:blank#data-operations) After ingesting data into Tinybird, you might need to perform various operations to maintain and optimize your data. This section covers common data operations like: - [ Replacing and deleting data](https://www.tinybird.co/docs/docs/get-data-in/data-operations/replace-and-delete-data) : Update or remove data selectively or entirely from your Data Sources. - [ Iterating a Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source) : Make schema changes and evolve your Data Sources over time. - [ Scheduling data operations](https://www.tinybird.co/docs/docs/get-data-in/data-operations/scheduling-with-github-actions-and-cron) : Automate data operations using cron jobs or GitHub Actions. - [ Recovering from quarantine](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine) : Handle and fix data that didn't match your schema during ingestion. These operations help you maintain data quality, adapt to changing requirements, and ensure your data pipeline runs smoothly. Whether you need to fix data issues, modify schemas, or automate routine tasks, Tinybird provides the tools to manage your data effectively. --- URL: https://www.tinybird.co/docs/get-data-in/connectors Content: --- title: "Connectors · Tinybird Docs" theme-color: "#171612" description: "Learn how to connect to your data sources using Tinybird connectors." --- # Connectors [¶](https://www.tinybird.co/docs/about:blank#connectors) Tinybird Connectors are native integrations that let you seamlessly connect to and ingest data from popular data platforms and services. They provide a managed solution to stream or batch import data into Tinybird with minimal configuration. The following Connectors are available: - [ Amazon DynamoDB](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb) - [ Amazon MSK](https://www.tinybird.co/docs/docs/get-data-in/connectors/msk) - [ Amazon S3](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3) - [ Confluent](https://www.tinybird.co/docs/docs/get-data-in/connectors/confluent) - [ Google BigQuery](https://www.tinybird.co/docs/docs/get-data-in/connectors/bigquery) - [ Kafka](https://www.tinybird.co/docs/docs/get-data-in/connectors/kafka) - [ Redpanda](https://www.tinybird.co/docs/docs/get-data-in/connectors/redpanda) Each Connector is fully managed by Tinybird and requires minimal setup - typically just authentication credentials and basic configuration. They handle the complexities of: - Authentication and secure connections - Schema detection and mapping - Incremental updates and change data capture - Error handling and monitoring - Scheduling and orchestration You can configure Connectors through either the Tinybird UI or CLI, making it easy to incorporate them into your data workflows and CI/CD pipelines. --- URL: https://www.tinybird.co/docs/cli/workspaces Last update: 2024-12-18T11:12:31.000Z Content: --- title: "Manage Workspaces using the CLI · Tinybird Docs" theme-color: "#171612" description: "Learn how to switch between different Tinybird Workspaces and how to manage members using the CLI." --- # Manage Workspaces using the CLI [¶](https://www.tinybird.co/docs/about:blank#manage-workspaces-using-the-cli) If you are a member of different Workspaces, you might need to frequently switch between Workspaces when working on a project using Tinybird CLI. This requires to authenticate and select the right Workspace. ## Authenticate [¶](https://www.tinybird.co/docs/about:blank#authenticate) Authenticate using the admin Token. For example: ##### Authenticate tb auth --token ## List Workspaces [¶](https://www.tinybird.co/docs/about:blank#list-workspaces) List the Workspaces you have access to, and the one that you're currently authenticated to: ##### List Workspaces tb workspace ls ## Create a Workspace [¶](https://www.tinybird.co/docs/about:blank#create-a-workspace) You can create new empty Workspaces or create a Workspace from a template. To create Workspaces using Tinybird CLI, you need [your user Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#your-user-token). Run the following command to create a Workspace following instructions: ##### Authenticate tb workspace create You can create a Workspace directly by defining the user Token using the `--user_token` flag: ##### Authenticate tb workspace create workspace_name --user_token ## Switch to another Workspace [¶](https://www.tinybird.co/docs/about:blank#switch-to-another-workspace) You can switch to another Workspace using `--use` . For example: ##### Switch to a Workspace using the Workspace id or the Workspace name # Use the Workspace ID tb workspace use 841717b1-2472-44f9-9a81-42f1263cabe7 # Use the Workspace name tb workspace use Production To find out the IDs and names of available Workspaces, run `tb workspace ls`: tb workspace ls You can also check which Workspace you're currently in: ##### Show current Workspace tb workspace current ## Manage Workspace members [¶](https://www.tinybird.co/docs/about:blank#manage-workspace-members) You can manage Workspace members using the `workspace members` commands. ### List members [¶](https://www.tinybird.co/docs/about:blank#list-members) To list members, run `tb workspace members ls` . For example: ##### Listing current Workspace members tb workspace members ls ### Add members [¶](https://www.tinybird.co/docs/about:blank#add-members) To add members, run `tb workspace members add` . For example: ##### Adding users to the current Workspace tb workspace members add "user1@example.com,user2@example.com,user3@example.com" ### Remove members [¶](https://www.tinybird.co/docs/about:blank#remove-members) To remove members, run `tb workspace members rm` . For example: ##### Removing members from current Workspace tb workspace members rm user3@example.com You can also manage roles. For example, to set a user as admin: ##### Add admin role to user tb workspace members set-role admin user@example.com --- URL: https://www.tinybird.co/docs/cli/template-functions Last update: 2025-01-07T15:54:46.000Z Content: --- title: "Template functions · Tinybird Docs" theme-color: "#171612" description: "Template functions available in Tinybird datafiles." --- # Template functions [¶](https://www.tinybird.co/docs/about:blank#template-functions) The following template functions are available. You can use them in [datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) to accomplish different tasks. See [Advanced templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) for more information on templating. ## defined [¶](https://www.tinybird.co/docs/about:blank#defined) Checks whether a variable is defined. ##### defined function % SELECT date FROM my_table {% if defined(param) %} WHERE ... {% end %} ## column [¶](https://www.tinybird.co/docs/about:blank#column) Retrieves the column by its name from a variable. ##### column function % {% set var_1 = 'name' %} SELECT {{column(var_1)}} FROM my_table ## columns [¶](https://www.tinybird.co/docs/about:blank#columns) Retrieves columns by their name from a variable. ##### columns function % {% set var_1 = 'name,age,address' %} SELECT {{columns(var_1)}} FROM my_table ## date_diff_in_seconds [¶](https://www.tinybird.co/docs/about:blank#date-diff-in-seconds) Returns the absolute value of the difference in seconds between two `DateTime` . See [DateTime](https://www.tinybird.co/docs/docs/sql-reference/data-types/datetime). The function accepts the following parameters: - `date_1` : the first date or DateTime. - `date_2` : the second date or DateTime. - `date_format` : (optional) the format of the dates. Defaults to `'%Y-%m-%d %H:%M:%S'` , so you can pass `DateTime` as `YYYY-MM-DD hh:mm:ss` when calling the function. - `backup_date_format` : (optional) the format of the dates if the first format doesn't match. Use it when your default input format is a DateTime ( `2022-12-19 18:42:22` ) but you receive a date instead ( `2022-12-19` ). - `none_if_error` : (optional) whether to return `None` if the dates don't match the provided formats. Defaults to `False` . Use it to provide an alternate logic in case any of the dates are specified in a different format. An example of how to use the function: date_diff_in_seconds('2022-12-19T18:42:23.521Z', '2022-12-19T18:42:23.531Z', date_format='%Y-%m-%dT%H:%M:%S.%fz') The following example shows how to use the function in a datafile: ##### date_diff_in_seconds function % SELECT date, events {% if date_diff_in_seconds(date_end, date_start, date_format="%Y-%m-%dT%H:%M:%Sz") < 3600 %} FROM my_table_raw {% else %} FROM my_table_hourly_agg {% end %} WHERE date BETWEEN parseDateTimeBestEffort({{String(date_start,'2023-01-11T12:24:04Z')}}) AND parseDateTimeBestEffort({{String(date_end,'2023-01-11T12:24:05Z')}}) See [working with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time) for more information on how to work with time in Tinybird. ## date_diff_in_minutes [¶](https://www.tinybird.co/docs/about:blank#date-diff-in-minutes) Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/about:blank#date_diff_in_seconds) , but returns the difference in minutes. ## date_diff_in_hours [¶](https://www.tinybird.co/docs/about:blank#date-diff-in-hours) Same behavior as [date_diff_in_seconds](https://www.tinybird.co/docs/about:blank#date_diff_in_seconds) , but returns the difference in hours. ## date_diff_in_days [¶](https://www.tinybird.co/docs/about:blank#date-diff-in-days) Returns the absolute value of the difference in days between two dates or DateTime. ##### date_diff_in_days function % SELECT date FROM my_table {% if date_diff_in_days(date_end, date_start) < 7 %} WHERE ... {% end %} `date_format` is optional and defaults to `'%Y-%m-%d` , so you can pass DateTime as `YYYY-MM-DD` when calling the function. As with `date_diff_in_seconds`, `date_diff_in_minutes` , and `date_diff_in_hours` , other [date_formats](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) are supported. ## split_to_array [¶](https://www.tinybird.co/docs/about:blank#split-to-array) Splits comma separated values into an array. The function accepts the following parameters: `split_to_array(arr, default, separator=',')` - `arr` : the value to split. - `default` : the default value to use if `arr` is empty. - `separator` : the separator to use. Defaults to `,` . The following example splits `code` into an array of integers: ##### split_to_array function % SELECT arrayJoin(arrayMap(x -> toInt32(x), {{split_to_array(code, '')}})) as codes FROM my_table The following example splits `param` into an array of strings using `|` as the custom separator: ##### split_to_array with a custom separator function % SELECT {{split_to_array(String(param, 'hi, how are you|fine thanks'), separator='|')}} ## enumerate_with_last [¶](https://www.tinybird.co/docs/about:blank#enumerate-with-last) Creates an iterable array, returning a boolean value that allows to check if the current element is the last element in the array. You can use it alongside the [split_to_array function](https://www.tinybird.co/docs/about:blank#split_to_array). ## symbol [¶](https://www.tinybird.co/docs/about:blank#symbol) Retrieves the value of a variable. The function accepts the following parameters: `symbol(x, quote)` For example: ##### enumerate_with_last function % SELECT {% for _last, _x in enumerate_with_last(split_to_array(attr, 'amount')) %} sum({{symbol(_x)}}) as {{symbol(_x)}} {% if not _last %}, {% end %} {% end %} FROM my_table ## sql_and [¶](https://www.tinybird.co/docs/about:blank#sql-and) Creates a list of "WHERE" clauses, along with "AND" separated filters, that checks if a field () is or isn't () in a list/tuple (). The function accepts the following parameters: `sql_and(__= [, ...] )` - `` : any column in the table. - `` : one of: `in` , `not_in` , `gt` (>), `lt` (<), `gte` (>=), `lte` (<=) - `` : any of the transform type functions ( `Array(param, 'Int8')` , `String(param)` , etc.). If one parameter isn't specified, then the filter is ignored. For example: ##### sql_and function % SELECT * FROM my_table WHERE 1 {% if defined(param) or defined(param2_not_in) %} AND {{sql_and( param__in=Array(param, 'Int32', defined=False), param2__not_in=Array(param2_not_in, 'String', defined=False))}} {% end %} If this is queried with `param=1,2` and `param2_not_in=ab,bc,cd` , it translates to: ##### sql_and function - generated sql SELECT * FROM my_table WHERE 1 AND param IN [1,2] AND param2 NOT IN ['ab','bc','cd'] If this is queried with `param=1,2` only, but `param2_not_in` isn't specified, it translates to: ##### sql_and function - generated sql param missing SELECT * FROM my_table WHERE 1 AND param IN [1,2] ## Transform types functions [¶](https://www.tinybird.co/docs/about:blank#transform-types-functions) The following functions validate the type of a template variable and cast it to the desired data type. They also provide a default value if no value is passed. - `Boolean(x)` - `DateTime64(x)` - `DateTime(x)` - `Date(x)` - `Float32(x)` - `Float64(x)` - `Int8(x)` - `Int16(x)` - `Int32(x)` - `Int64(x)` - `Int128(x)` - `Int256(x)` - `UInt8(x)` - `UInt16(x)` - `UInt32(x)` - `UInt64(x)` - `UInt128(x)` - `UInt256(x)` - `String(x)` - `Array(x)` Each function accepts the following parameters: `type(x, default, description=, required=)` - `x` : the parameter or value. - `default` : (optional) the default value to use if `x` is empty. - `description` : (optional) the description of the value. - `required` : (optional) whether the value is required. For example, `Int32` in the following query, `lim` is the parameter to be cast to an `Int32`, `10` is the default value, and so on: ##### transform_type_functions % SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}} --- URL: https://www.tinybird.co/docs/cli/quick-start Last update: 2025-01-09T09:46:35.000Z Content: --- title: "Quick start · Tinybird Docs" theme-color: "#171612" description: "Get started with Tinybird CLI as quickly as possible. Ingest, query, and publish data in minutes." --- # Quick start for Tinybird command-line interface [¶](https://www.tinybird.co/docs/about:blank#quick-start-for-tinybird-command-line-interface) With Tinybird, you can ingest data from anywhere, query and transform it using SQL, and publish your data as high-concurrency, low-latency REST API endpoints. After you've [familiarized yourself with Tinybird](https://www.tinybird.co/docs/docs/get-started/quick-start) , you're ready to start automating and scripting the management of your Workspace using the Tinybird command-line interface (CLI). The Tinybird CLI is essential for all [CI/CD workflows](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration). Read on to learn how to download and configure the Tinybird CLI, create a Workspace, ingest data, create a query, publish an API, and confirm your setup works properly. ## Step 1: Create your Tinybird account [¶](https://www.tinybird.co/docs/about:blank#step-1-create-your-tinybird-account) [Create a Tinybird account](https://www.tinybird.co/signup) . It's free and no credit card is required. See [Tinybird pricing plans](https://www.tinybird.co/docs/docs/get-started/plans/billing) for more information. [Sign up for Tinybird](https://www.tinybird.co/signup) ## Step 2: Download and install the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#step-2-download-and-install-the-tinybird-cli) [Follow the instructions](https://www.tinybird.co/docs/docs/cli/install) to download and install the Tinybird command-line interface (CLI). Complete the setup and authenticate with your Tinybird account in the cloud and region you prefer. ## Step 3: Create your Workspace [¶](https://www.tinybird.co/docs/about:blank#step-3-create-your-workspace) A [Workspace](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) is an area that contains a set of Tinybird resources, including Data Sources, Pipes, nodes, API Endpoints, and Tokens. Create a Workspace named `customer_rewards` . Use a unique name. tb workspace create customer_rewards ## Step 4: Download and ingest sample data [¶](https://www.tinybird.co/docs/about:blank#step-4-download-and-ingest-sample-data) Download the following sample data from a fictitious online coffee shop: [Download data file](https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-05.parquet) The following Tinybird CLI commands infer the schema from the datafile, generate and push a .datasource file and ingest the data. tb datasource generate orders.ndjson # Infer the schema tb push orders.datasource # Upload the datasource file tb datasource append orders orders.ndjson # Ingest the data ## Step 5: Query data using a Pipe and Publish it as an API [¶](https://www.tinybird.co/docs/about:blank#step-5-query-data-using-a-pipe-and-publish-it-as-an-api) In Tinybird, you can create [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) to query your data using SQL. The following commands create a Pipe with an SQL instruction that returns the number of records Tinybird has ingested from the data file: tb pipe generate rewards 'select count() from orders' tb push rewards.pipe When you push a Pipe, Tinybird publishes it automatically as a high-concurrency, low-latency API Endpoint. ## Step 6: Call the API Endpoint [¶](https://www.tinybird.co/docs/about:blank#step-6-call-the-api-endpoint) You can test your API Endpoint using a curl command. First, create and obtain the read Token for the API Endpoint. tb token create static rewards_read_token --scope PIPES:READ --resource rewards tb token copy rewards_read_token Copy the read Token and insert it into a curl command. curl --compressed -H 'Authorization: Bearer your_read_token_here' https://api.us-east.aws.tinybird.co/v0/pipes/rewards.json You have now created your first API Endpoint in Tinybird using the CLI. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn about datafiles and their format. See[ Datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) . - Learn how advanced templates can help you. See[ Advanced templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) . - Browse the full CLI reference. See[ Command reference](https://www.tinybird.co/docs/docs/cli/command-ref) . --- URL: https://www.tinybird.co/docs/cli/local-container Last update: 2025-02-12T12:29:07.000Z Content: --- title: "Tinybird Local container · Tinybird Docs" theme-color: "#171612" description: "Learn how to run Tinybird locally using the local container." --- # Tinybird Local container [¶](https://www.tinybird.co/docs/about:blank#tinybird-local-container) You can run your own Tinybird instance locally using the `tinybird-local` container. This is useful for testing and development. For example, you can test Data Sources and Pipes in your data project before deploying them to production. Tinybird Local doesn't include the following features: - Tinybird UI - Connectors - Scheduled operations - Batch operations ## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites) To get started, you need a container runtime, like Docker or podman. ## Run Tinybird Local [¶](https://www.tinybird.co/docs/about:blank#run-tinybird-local) To run Tinybird locally, run the following command: docker run --platform linux/amd64 -p 7181:7181 --name tinybird-local -d tinybirdco/tinybird-local:latest By default, Tinybird Local runs on port 80, although you can expose it locally using any other port. ## Local authentication [¶](https://www.tinybird.co/docs/about:blank#local-authentication) To authenticate with Tinybird Local, retrieve the Workspace admin token and pass it through the CLI: TOKEN=$(curl -s http://localhost:7181/tokens | jq -r ".workspace_admin_token") tb --host http://localhost:7181 --token $TOKEN auth After you've authenticated, you can get the default Workspace with the `tb workspace ls` CLI command. For example: tb workspace ls ** Workspaces: -------------------------------------------------------------------------------------------- | name | id | role | plan | current | -------------------------------------------------------------------------------------------- | Tinybird_Local_Testing | 7afc6330-3aae-4df5-8712-eaad216c5d7d | admin | Custom | True | -------------------------------------------------------------------------------------------- ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn about datafiles and their format. See[ Datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) . - Learn how advanced templates can help you. See[ Advanced templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) . - Browse the full CLI reference. See[ Command reference](https://www.tinybird.co/docs/docs/cli/command-ref) . --- URL: https://www.tinybird.co/docs/cli/install Last update: 2024-12-18T11:12:31.000Z Content: --- title: "Install Tinybird CLI · Tinybird Docs" theme-color: "#171612" description: "Install the Tinybird CLI on Linux or macOS, or use the prebuilt Docker image." --- # Install the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#install-the-tinybird-cli) You can install Tinybird on your local machine or use a prebuilt Docker image. Read on to learn how to install and configure Tinybird CLI for use. ## Installation [¶](https://www.tinybird.co/docs/about:blank#installation) Install Tinybird CLI locally to use it on your machine. ### Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites) Tinybird CLI supports Linux and macOS 10.14 and higher. Supported Python versions are 3.8, 3.9, 3.10, 3.11, 3.12. ### Install tinybird-cli [¶](https://www.tinybird.co/docs/about:blank#install-tinybird-cli) Create a virtual environment before installing the `tinybird-cli` package: ##### Creating a virtual environment for Python 3 python3 -m venv .venv source .venv/bin/activate Then, install `tinybird-cli`: ##### Install tinybird-cli pip install tinybird-cli To update the `tinybird-cli` package, run the following command: ##### Update tinybird-cli pip install --upgrade tinybird-cli ## Docker image [¶](https://www.tinybird.co/docs/about:blank#docker-image) The official `tinybird-cli-docker` image provides a Tinybird CLI executable ready to use in your projects and pipelines. To run Tinybird CLI using Docker from the terminal, run the following commands: ##### Build local image setting your project path # Assuming a projects/data path docker run -v ~/projects/data:/mnt/data -it tinybirdco/tinybird-cli-docker cd mnt/data ## Authentication [¶](https://www.tinybird.co/docs/about:blank#authentication) Before you start using Tinybird CLI, check that you can authenticate by running `tb auth`: ##### Authenticate tb auth -i A list of available regions appears. Select your Tinybird region, then provide your admin Token. See [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens). You can also pass the Token directly with the `--token` flag. For example: ##### Authenticate tb auth --token See the API Reference docs for the [list of supported regions](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) . You can also get the list using `tb auth ls`. The `tb auth` command saves your credentials in a .tinyb file in your current directory. Add it to your .gitignore file to avoid leaking credentials. ## Integrated help [¶](https://www.tinybird.co/docs/about:blank#integrated-help) After you've installed the Tinybird CLI you can access the integrated help by using the `--help` flag: ##### Integrated help tb --help You can do the same for every available command. For example: ##### Integrated command help tb datasource --help ## Telemetry [¶](https://www.tinybird.co/docs/about:blank#telemetry) Starting from version 1.0.0b272, the Tinybird CLI collects telemetry on the use of the CLI commands and information about exceptions and crashes and sends it to Tinybird. Telemetry helps Tinybird improve the command-line experience. On each `tb` execution, the CLI collects information about your system, Python environment, the CLI version installed and the command you ran. All data is completely anonymous. To opt out of the telemetry feature, set the `TB_CLI_TELEMETRY_OPTOUT` environment variable to `1` or `true`. ## Configure your shell prompt [¶](https://www.tinybird.co/docs/about:blank#configure-your-shell-prompt) You can extract the current Tinybird Workspace and region from your .tinyb file and show it in your zsh or bash shell prompt. To extract the information programmatically, paste the following function to your shell config file: ##### Parse the .tinyb file to use the output in the PROMPT prompt_tb() { if [ -e ".tinyb" ]; then TB_CHAR=$'\U1F423' branch_name=`grep '"name":' .tinyb | cut -d : -f 2 | cut -d '"' -f 2` region=`grep '"host":' .tinyb | cut -d / -f 3 | cut -d . -f 2 | cut -d : -f 1` if [ "$region" = "tinybird" ]; then region=`grep '"host":' .tinyb | cut -d / -f 3 | cut -d . -f 1` fi TB_BRANCH="${TB_CHAR}tb:${region}=>${branch_name}" else TB_BRANCH='' fi echo $TB_BRANCH } When the function is available, you need to make the output visible on the prompt of your shell. The following example shows how to do this for zsh: ##### Include Tinybird information in the zsh prompt echo 'export PROMPT="' $PS1 ' $(prompt_tb)"' >> ~/.zshrc Restart your shell and go to the root of your project to see the Tinybird region and Workspace in your prompt. --- URL: https://www.tinybird.co/docs/cli/datafiles Content: --- title: "Datafiles · Tinybird Docs" theme-color: "#171612" description: "Datafiles describe your Tinybird resources: Data Sources, Pipes, and so on. They're the source code of your project." --- # Datafiles [¶](https://www.tinybird.co/docs/about:blank#datafiles) Datafiles describe your Tinybird resources, like Data Sources, Pipes, and so on. They're the source code of your project. You can use datafiles to manage your projects as source code and take advantage of version control. Tinybird CLI helps you produce and push datafiles to the Tinybird platform. ## Types of datafiles [¶](https://www.tinybird.co/docs/about:blank#types-of-datafiles) Tinybird uses the following types of datafiles: - Datasource files (.datasource) represent Data Sources. See[ Datasource files](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files) . - Pipe files (.pipe) represent Pipes of various types. See[ Pipe files](https://www.tinybird.co/docs/docs/cli/datafiles/pipe-files) . - Include files (.incl) are reusable fragments you can include in .datasource or .pipe files. See[ Include files](https://www.tinybird.co/docs/docs/cli/datafiles/include-files) . ## Syntactic conventions [¶](https://www.tinybird.co/docs/about:blank#syntactic-conventions) Datafiles follow the same syntactic conventions. ### Casing [¶](https://www.tinybird.co/docs/about:blank#casing) Instructions always appear at the beginning of a line in upper case. For example: ##### Basic syntax COMMAND value ANOTHER_INSTR "Value with multiple words" ### Multiple lines [¶](https://www.tinybird.co/docs/about:blank#multiple-lines) Instructions can span multiples lines. For example: ##### Multiline syntax SCHEMA > `d` DateTime, `total` Int32, `from_novoa` Int16 ## File structure [¶](https://www.tinybird.co/docs/about:blank#file-structure) The following example shows a typical `tinybird` project directory that includes subdirectories for supported types: ##### Example file structure tinybird ├── datasources/ │ └── connections/ │ └── my_connector_name.incl │ └── my_datasource.datasource ├── endpoints/ ├── includes/ ├── pipes/ ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Understand[ CI/CD processes on Tinybird](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) . - Read about[ implementing test strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies) . --- URL: https://www.tinybird.co/docs/cli/data-projects Last update: 2025-01-20T11:46:12.000Z Content: --- title: "Organize files in data projects · Tinybird Docs" theme-color: "#171612" description: "Learn how to best organize your Tinybird CLI files in versioned projects." --- # Organize files in data projects [¶](https://www.tinybird.co/docs/about:blank#organize-files-in-data-projects) A data project is a set of files that describes how your data must be stored, processed, and exposed through APIs. In the same way you maintain source code files in a repository, use a CI, make deployments, run tests, and so on, Tinybird provides tools to work following a similar pattern but with data pipelines. The source code of your project are the [datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) in Tinybird. With a data project you can: - Define how the data should flow, from schemas to API Endpoints. - Manage your datafiles using version control. - Branch your datafiles. - Run tests. - Deploy data projects. ## Ecommerce site example [¶](https://www.tinybird.co/docs/about:blank#ecommerce-site-example) Consider an ecommerce site where you have events from users and a list of products with their attributes. Your goal is to expose several API endpoints to return sales per day and top product per day. The data project file structure would look like the following: ecommerce_data_project/ datasources/ events.datasource products.datasource fixtures/ events.csv products.csv pipes/ top_product_per_day.pipe endpoints/ sales.pipe top_products.pipe To follow this tutorial, download and open the example using the following commands: ##### Clone demo git clone https://github.com/tinybirdco/ecommerce_data_project.git cd ecommerce_data_project ### Upload the project [¶](https://www.tinybird.co/docs/about:blank#upload-the-project) You can push the whole project to your Tinybird account to check everything is fine. The `tb push` command uploads the data to Tinybird, previously checking the project dependencies and the SQL syntax. In this case, use the `--push-deps` flag to push everything: ##### Push dependencies tb push --push-deps After the upload completes, the endpoints defined in our project, `sales` and `top_products` , are available and you can start pushing data to the different Data Sources. ### Define Data Sources [¶](https://www.tinybird.co/docs/about:blank#define-data-sources) Data Sources define how your data is ingested and stored. You can add data to Data Sources using the [Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api). Each Data Source is defined by a schema and other properties. See [Datasource files](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files). The following snippet shows the content of the `event.datasource` file from the ecommerce example: DESCRIPTION > # Events from users This contains all the events produced by Kafka, there are 4 fixed columns. plus a `json` column which contains the rest of the data for that event. See [documentation](https://www.tinybird.co/docs/url_for_docs) for the different events. SCHEMA > timestamp DateTime, product String, user_id String, action String json String ENGINE MergeTree ENGINE_SORTING_KEY timestamp The file describes the schema and how the data is sorted. In this case, the access pattern is most of the time by the `timestamp` column. If no `SORTING_KEY` is set, Tinybird picks one by default, date or datetime columns in most cases. To push the Data Source, run: ##### Push the events Data Source tb push datasources/events.datasource You can't override Data Sources. If you try to push a Data Source that already exists in your account you get an error. To override a Data Source, remove it or upload a new one with a different name. ### Define data Pipes [¶](https://www.tinybird.co/docs/about:blank#define-data-pipes) The content of the `pipes/top_product_per_day.pipe` file creates a data Pipe that transforms the data as it's inserted: NODE only_buy_events DESCRIPTION > filters all the buy events SQL > SELECT toDate(timestamp) date, product, JSONExtractFloat(json, 'price') AS price FROM events WHERE action = 'buy' NODE top_per_day SQL > SELECT date, topKState(10)(product) top_10, sumState(price) total_sales FROM only_buy_events GROUP BY date TYPE materialized DATASOURCE top_per_day_mv ENGINE AggregatingMergeTree ENGINE_SORTING_KEY date Each Pipe can have one or more nodes. The previous Pipe defines two nodes, `only_buy_events` and `top_per_day`. - The first node filters `buy` events and extracts some data from the `json` column. - The second node runs the aggregation. In general, use `NODE` to start a new node and then use `SQL >` to define the SQL for that Node. You can use other nodes inside the SQL. In this case, the second node uses the first one `only_buy_events`. To push the Pipe, run: ##### Populate tb push pipes/top_product_per_day.pipe --populate If you want to populate with the existing data in `events` table, use the `--populate` flag. When using the `--populate` flag you get a job URL so you can check the status of the job by checking the URL provided. See [Populate and copy data](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/populate-data) for more information on how populate jobs work. ### Define API Endpoints [¶](https://www.tinybird.co/docs/about:blank#define-api-endpoints) API Endpoints are the way you expose the data to be consumed. The following snippet shows the content of the `endpoints/top_products.pipe` file: NODE endpoint DESCRIPTION > returns top 10 products for the last week SQL > SELECT date, topKMerge(10)(top_10) AS top_10 FROM top_per_day WHERE date > today() - interval 7 day GROUP BY date The syntax is the same as in the data transformation Pipes, though you can access the results through the `{% user("apiHost") %}/v0/top_products.json?token=TOKEN` endpoint. When you push an endpoint a Token with `PIPE:READ` permissions is automatically created. You can see it from the [Tokens UI](https://app.tinybird.co/tokens) or directly from the CLI with the command `tb pipe token_read `. Alternatively, you can use the `TOKEN token_name READ` command to automatically create a Token with name `token_name` with `READ` permissions over the endpoint or add `READ` permissions to the existing `token_name` over the endpoint. For example: TOKEN public_read_token READ NODE endpoint DESCRIPTION > returns top 10 products for the last week SQL > SELECT date, topKMerge(10)(top_10) AS top_10 FROM top_per_day WHERE date > today() - interval 7 day GROUP BY date To push the endpoint, run: ##### Push the top products Pipe tb push endpoints/top_products.pipe The Token `public_read_token` was created automatically and it's provided in the test URL. You can add parameters to any endpoint. For example, parametrize the dates to be able to filter the data between two dates: NODE endpoint DESCRIPTION > returns top 10 products for the last week SQL > % SELECT date, topKMerge(10)(top_10) AS top_10 FROM top_per_day WHERE date between {{Date(start)}} AND {{Date(end)}} GRUP BY date Now, the endpoint can receive `start` and `end` parameters: `{% user("apiHost") %}/v0/top_products.json?start=2018-09-07&end=2018-09-17&token=TOKEN`. You can print the results from the CLI using the `pipe data` command. For instance, for the previous example: ##### Print the results of the top products endpoint tb pipe data top_products --start '2018-09-07' --end '2018-09-17' --format CSV For the parameters templating to work you need to start your NODE SQL definition using the character `%`. ### Override an endpoint or a data Pipe [¶](https://www.tinybird.co/docs/about:blank#override-an-endpoint-or-a-data-pipe) When working on a project, you might need to push several versions of the same file. You can override a Pipe that has already been pushed using the `--force` flag. For example: ##### Override the Pipe tb push endpoints/top_products_params.pipe --force If the endpoint has been called before, it runs regression tests with the most frequent requests. If the new version doesn't return the same data, then it's not pushed. You can see in the example how to run all the requests tested. You can force the push without running the checks using the `--no-check` flag if needed. For example: ##### Force override tb push endpoints/top_products_params.pipe --force --no-check ### Downloading datafiles from Tinybird [¶](https://www.tinybird.co/docs/about:blank#downloading-datafiles-from-tinybird) You can download datafiles using the `pull` command. For example: ##### Pull a specific file tb pull --match endpoint_im_working_on The previous command downloads the `endpoint_im_working_on.pipe` to the current folder. --- URL: https://www.tinybird.co/docs/cli/common-use-cases Last update: 2025-01-20T11:46:12.000Z Content: --- title: "CLI common use cases · Tinybird Docs" theme-color: "#171612" description: "This document shows some common use cases where the Command Line Interface (CLI) can help you on your day to day workflow." --- # Common use cases [¶](https://www.tinybird.co/docs/about:blank#common-use-cases) The following uses cases illustrate how Tinybird CLI solve common situations using available commands. ## Download Pipes and data sources from your account [¶](https://www.tinybird.co/docs/about:blank#download-pipes-and-data-sources-from-your-account) There are two ways you can start working with the CLI. You can either [start a new data project](https://www.tinybird.co/docs/docs/cli/quick-start) from scratch, or if you already have some data and API Endpoints in your Tinybird account, pull it to your local disk to continue working from there. For this second option, use the `--match` flag to filter Pipes or data sources containing the string passed as parameter. For instance, to pull all the files named `project`: ##### Pull all the project files tb pull --match project [D] writing project.datasource(demo) [D] writing project_geoindex.datasource(demo) [D] writing project_geoindex_pipe.pipe(demo) [D] writing project_agg.pipe(demo) [D] writing project_agg_API_endpoint_request_log_pipe_3379.pipe(demo) [D] writing project_exploration.pipe(demo) [D] writing project_moving_avg.pipe(demo) The pull command doesn't preserve the directory structure, so all your datafiles are downloaded to your current directory. Once the files are pulled, you can `diff` or `push` the changes to your source control repository and continue working from the command line. When you pull data sources or Pipes, your data isn't downloaded, just the data source schemas and Pipes definition, so they can be replicated easily. ## Push the entire data project [¶](https://www.tinybird.co/docs/about:blank#push-the-entire-data-project) ##### Push the whole project tb push --push-deps ## Push a Pipe with all its dependencies [¶](https://www.tinybird.co/docs/about:blank#push-a-pipe-with-all-its-dependencies) ##### Push dependencies tb push pipes/mypipe.pipe --push-deps ## Adding a new column to a Data Source [¶](https://www.tinybird.co/docs/about:blank#adding-a-new-column-to-a-data-source) Data Source schemas are mostly immutable, but you have the possibility to append new columns at the end of an existing Data Source with an Engine from the MergeTree Family or Null Engine. If you want to change columns, add columns in other positions, or modify the engine, you must first create a new version of the Data Source with the modified schema. Then ingest the data and finally point the Pipes to this new API Endpoint. To force a Pipe replacement use the `--force` flag when pushing it. If you create a new column with a `DEFAULT` o `MATERIALIZED` value, only the rows inserted after adding the column will write the data to disk. Data already in a Data Source when the column is added will not be modified, and the value will be computed at query time. That's not problematic when the expression is constant, for example a specific date or number, but for dynamic expression like `now()` or `now64()` , the returned value might change every time a select query is performed. ### Append new columns to an existing Data Source [¶](https://www.tinybird.co/docs/about:blank#append-new-columns-to-an-existing-data-source) As an example, imagine you have the following Data Source defined, and it has been already pushed to Tinybird: ##### Appending a new column to a Data Source SCHEMA > `test` Int16, `local_date` Date, `test3` Int64 If you want to append a new column, you must change the `*.datasource` file to add the new column `new_column` . You can append as many columns as you need at the same time: ##### Appending a new column to a Data Source SCHEMA > `test` Int16, `local_date` Date, `test3` Int64, `new_column` Int64 Remember that when **appending or deleting columns to an existing Data Source** , the engine of that Data Source must be of the **MergeTree** family. After appending the new column, execute `tb push my_datasource.datasource --force` and confirm the addition of the column(s). The `--force` parameter is required for this kind of operation. Existing imports will continue working once the new columns are added, even if those imports don't carry values for the added columns. In those cases, the new columns contain empty values like `0` for numeric values or `''` for Strings, or if defined, the default values in the schema. ### Create a new version of the Data Source to make additional add/change column operations [¶](https://www.tinybird.co/docs/about:blank#create-a-new-version-of-the-data-source-to-make-additional-addchange-column-operations) To create a new version of a Data Source, create a separate datafile with a different name. You can choose a helpful naming convention such as adding a `_version` suffix (e.g. `my_ds_1.datasource` ). ## Debug mode [¶](https://www.tinybird.co/docs/about:blank#debug-mode) When you work with Pipes that use several versions of different data sources, you might need to double check which version of which Data Source the Pipe is pointing at before you push it to your Tinybird account. To do so, use the `--dry-run --debug` flags like this: ##### Debug mode tb push my_pipe.pipe --dry-run --debug After you've validated the content of the Pipe, push your Pipe as normal. ## Automatic regression tests for your API Endpoints [¶](https://www.tinybird.co/docs/about:blank#automatic-regression-tests-for-your-api-endpoints) Any time you `--force` push a Pipe which has a public API Endpoint that has received requests, some automatic regression tests are executed. If the previous version of the API Endpoint returns the same data as the version you are pushing, the CLI checks for the top ten requests. This can help you validate whether you are introducing a regression in your API. Other times, you are consciously `--force` pushing a new version which returns different data. In that case you can avoid the regression tests with the `--no-check` flag: ##### Avoid regression tests tb push my_may_view_pipe.pipe --force --no-check When pushing a Pipe with a public API Endpoint, the API Endpoint will be maintained based on the node name. If the existing API Endpoint node is renamed, the last node of the Pipe will be recreated as an API Endpoint. The latter option isn't an atomic operation: The API Endpoint will be down for a few moments while the new API Endpoint is created. --- URL: https://www.tinybird.co/docs/cli/command-ref Last update: 2025-01-21T08:14:33.000Z Content: --- title: "Tinybird CLI command reference · Tinybird Docs" theme-color: "#171612" description: "The Tinybird CLI allows you to use all the Tinybird functionality directly from the command line. Get to know the command reference." --- # CLI command reference [¶](https://www.tinybird.co/docs/about:blank#cli-command-reference) The following list shows all available commands in the Tinybird command-line interface, their options, and their arguments. For examples on how to use them, see the [Quick start guide](https://www.tinybird.co/docs/docs/cli/quick-start), [Data projects](https://www.tinybird.co/docs/docs/cli/data-projects) , and [Common use cases](https://www.tinybird.co/docs/docs/cli/common-use-cases). ## tb auth [¶](https://www.tinybird.co/docs/about:blank#tb-auth) Configure your Tinybird authentication. **auth commands** | Command | Description | | --- | --- | | info OPTIONS | Gets information about the authentication that is currently being used. | | ls OPTIONS | Lists available regions to authenticate. | | use OPTIONS REGION_NAME_OR_HOST_OR_ID | Switches to a different region. You can pass the region name, the region host url, or the region index after listing available regions with `tb auth ls` . | The previous commands accept the following options: - `--token INTEGER` : Use auth Token, defaults to TB_TOKEN envvar, then to the .tinyb file. - `--host TEXT` : Set custom host if it's different than https://api.tinybird.co. Check[ this page](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) for the available list of regions. - `--region TEXT` : Set region. Run 'tb auth ls' to show available regions. - `--connector [bigquery]` : Set credentials for one of the supported connectors. - `--interactive,-i` : Show available regions and select where to authenticate to. ## tb branch [¶](https://www.tinybird.co/docs/about:blank#tb-branch) Manage your Workspace branches. **Branch commands** | Command | Description | Options | | | --- | --- | --- | | | create BRANCH_NAME | Creates a new Branch in the current 'main' Workspace. | `--last-partition` : Attaches the last modified partition from 'main' to the new Branch. `-i, --ignore-datasource DATA_SOURCE_NAME` : Ignore specified Data Source partitions. `--wait / --no-wait` : Wait for Branch jobs to finish, showing a progress bar. Disabled by default. | | | current | Shows the Branch you're currently authenticated to. | | | | data | Performs a data branch operation to bring data into the current Branch. | `--last-partition` : Attaches the last modified partition from 'main' to the new Branch. `-i, --ignore-datasource DATA_SOURCE_NAME` : Ignore specified Data Source partitions. `--wait / --no-wait` : Wait for Branch jobs to finish, showing a progress bar. Disabled by default. | | | datasource copy DATA_SOURCE_NAME | Copies data source from Main. | `--sql SQL` : Freeform SQL query to select what is copied from Main into the Environment Data Source. `--sql-from-main` : SQL query selecting all from the same Data Source in Main. `--wait / --no-wait` : Wait for copy job to finish. Disabled by default. | | | ls | Lists all the Branches available. | `--sort / --no-sort` : Sorts the list of Branches by name. Disabled by default. | | | regression-tests | Regression test commands. | `-f, --filename PATH` : The yaml file with the regression-tests definition. `--skip-regression-tests / --no-skip-regression-tests` : Flag to skip execution of regression tests. This is handy for CI Branches where regression might be flaky. `--main` : Runs regression tests in the main Branch. For this flag to work all the resources in the Branch Pipe Endpoints need to exist in the main Branch. `--wait / --no-wait` : Waits for regression job to finish, showing a progress bar. Disabled by default. | | | regression-tests coverage PIPE_NAME | Runs regression tests using coverage requests for Branch vs Main Workspace. It creates a regression-tests job. The argument supports regular expressions. Using '.*' if no Pipe name is provided. | `--assert-result / --no-assert-result` : Whether to perform an assertion on the results returned by the Endpoint. Enabled by default. Use `--no-assert-result` if you expect the endpoint output is different from current version. `--assert-result-no-error / --no-assert-result-no-error` : Whether to verify that the Endpoint doesn't return errors. Enabled by default. Use `--no-assert-result-no-error` if you expect errors from the endpoint. `--assert-result-rows-count / --no-assert-result-rows-count` : Whether to verify that the correct number of elements are returned in the results. Enabled by default. Use `--assert-result-rows-count` if you expect the numbers of elements in the endpoint output is different from current version. `--assert-result-ignore-order / --no-assert-result-ignore-order` : Whether to ignore the order of the elements in the results. Disabled by default. Use `--assert-result-ignore-order` if you expect the endpoint output is returning same elements but in different order. `--assert-time-increase-percentage INTEGER` : Allowed percentage increase in Endpoint response time. Default value is 25%. Use -1 to disable assert. `--assert-bytes-read-increase-percentage INTEGER` : Allowed percentage increase in the amount of bytes read by the endpoint. Default value is 25%. Use -1 to disable assert. `--assert-max-time FLOAT` : Max time allowed for the endpoint response time. If the response time is lower than this value then the `--assert-time-increase-percentage` isn't taken into account. `--ff, --failfast` : When set, the checker exits as soon one test fails. `--wait` : Waits for regression job to finish, showing a progress bar. Disabled by default. `--skip-regression-tests / --no-skip-regression-tests` : Flag to skip execution of regression tests. This is handy for CI environments where regression might be flaky. `--main` : Runs regression tests in the main Branch. For this flag to work all the resources in the Branch Pipe Endpoints need to exist in the main Branch. | | | regression-tests last PIPE_NAME | Runs regression tests using coverage requests for Branch vs Main Workspace. It creates a regression-tests job. The argument supports regular expressions. Using '.*' if no Pipe name is provided. | `--assert-result / --no-assert-result` : Whether to perform an assertion on the results returned by the Endpoint. Enabled by default. Use `--no-assert-result` if you expect the endpoint output is different from current version. `--assert-result-no-error / --no-assert-result-no-error` : Whether to verify that the Endpoint doesn't return errors. Enabled by default. Use `--no-assert-result-no-error` if you expect errors from the endpoint. `--assert-result-rows-count / --no-assert-result-rows-count` : Whether to verify that the correct number of elements are returned in the results. Enabled by default. Use `--assert-result-rows-count` if you expect the numbers of elements in the endpoint output is different from current version. `--assert-result-ignore-order / --no-assert-result-ignore-order` : Whether to ignore the order of the elements in the results. Disabled by default. Use `--assert-result-ignore-order` if you expect the endpoint output is returning same elements but in different order. `--assert-time-increase-percentage INTEGER` : Allowed percentage increase in Endpoint response time. Default value is 25%. Use -1 to disable assert. `--assert-bytes-read-increase-percentage INTEGER` : Allowed percentage increase in the amount of bytes read by the endpoint. Default value is 25%. Use -1 to disable assert. `--assert-max-time FLOAT` : Max time allowed for the endpoint response time. If the response time is lower than this value then the `--assert-time-increase-percentage` isn't taken into account. `--ff, --failfast` : When set, the checker exits as soon one test fails. `--wait` : Waits for regression job to finish, showing a progress bar. Disabled by default. `--skip-regression-tests / --no-skip-regression-tests` : Flag to skip execution of regression tests. This is handy for CI environments where regression might be flaky. | | | regression-tests manual PIPE_NAME | Runs regression tests using coverage requests for Branch vs Main Workspace. It creates a regression-tests job. The argument supports regular expressions. Using '.*' if no Pipe name is provided. | `--assert-result / --no-assert-result` : Whether to perform an assertion on the results returned by the Endpoint. Enabled by default. Use `--no-assert-result` if you expect the endpoint output is different from current version. `--assert-result-no-error / --no-assert-result-no-error` : Whether to verify that the Endpoint doesn't return errors. Enabled by default. Use `--no-assert-result-no-error` if you expect errors from the endpoint. `--assert-result-rows-count / --no-assert-result-rows-count` : Whether to verify that the correct number of elements are returned in the results. Enabled by default. Use `--assert-result-rows-count` if you expect the numbers of elements in the endpoint output is different from current version. `--assert-result-ignore-order / --no-assert-result-ignore-order` : Whether to ignore the order of the elements in the results. Disabled by default. Use `--assert-result-ignore-order` if you expect the endpoint output is returning same elements but in different order. `--assert-time-increase-percentage INTEGER` : Allowed percentage increase in Endpoint response time. Default value is 25%. Use -1 to disable assert. `--assert-bytes-read-increase-percentage INTEGER` : Allowed percentage increase in the amount of bytes read by the endpoint. Default value is 25%. Use -1 to disable assert. `--assert-max-time FLOAT` : Max time allowed for the endpoint response time. If the response time is lower than this value then the `--assert-time-increase-percentage` isn't taken into account. `--ff, --failfast` : When set, the checker exits as soon one test fails. `--wait` : Waits for regression job to finish, showing a progress bar. Disabled by default. `--skip-regression-tests / --no-skip-regression-tests` : Flag to skip execution of regression tests. This is handy for CI Branches where regression might be flaky. | | | rm [BRANCH_NAME_OR_ID] | Removes a Branch from the Workspace (not Main). It can't be recovered. | `--yes` : Don't ask for confirmation. | | | use [BRANCH_NAME_OR_ID] | Switches to another Branch. | | | ## tb check [¶](https://www.tinybird.co/docs/about:blank#tb-check) Checks file syntax. It only allows one option, `--debug` , which prints the internal representation. ## tb connection [¶](https://www.tinybird.co/docs/about:blank#tb-connection) Connection commands. | Command | Description | Options | | --- | --- | --- | | create COMMAND [ARGS] | Creates a connection. Available subcommands or types are `bigquery` , `kafka` , `s3` , `s3_iamrole` . | See the next table. | | ls [OPTIONS] | Lists connections. | `--connector TYPE` : Filters by connector. Available types are `bigquery` , `kafka` , `s3` , `s3_iamrole` . | | rm [OPTIONS] CONNECTION_ID_OR_NAME | Removes a connection. | `--force BOOLEAN` : Forces connection removal even if there are Data Sources using it. | ### tb connection create [¶](https://www.tinybird.co/docs/about:blank#tb-connection-create) The following subcommands and settings are available for each `tb connection create` subcommand: | Command | Description | Options | | --- | --- | --- | | create bigquery [OPTIONS] | Creates a BigQuery connection. | `--no-validate` : Doesn't validate GCP permissions. | | create kafka [OPTIONS] | Creates a Kafka connection. | `--bootstrap-servers TEXT` : Kafka Bootstrap Server in the form mykafka.mycloud.com:9092. `--key TEXT` : Key. `--secret TEXT` : Secret. `--connection-name TEXT` : Name of your Kafka connection. If not provided, it's set as the bootstrap server. `--auto-offset-reset TEXT` : Offset reset, can be 'latest' or 'earliest'. Defaults to 'latest'. `--schema-registry-url TEXT` : Avro Confluent Schema Registry URL. `--sasl-mechanism TEXT` : Authentication method for connection-based protocols. Defaults to 'PLAIN'. `--ssl-ca-pem TEXT` : Path or content of the CA Certificate file in PEM format. | | create s3 [OPTIONS] | Creates an S3 connection. | `--key TEXT` : Your Amazon S3 key with access to the buckets. `--secret TEXT` : The Amazon S3 secret for the key. `--region TEXT` : The Amazon S3 region where you buckets are located. `--connection-name TEXT` : The name of the connection to identify it in Tinybird. `--no-validate` : Don't validate S3 permissions during connection creation. | | create s3_iamrole [OPTIONS] | Creates an S3 connection (IAM role). | `--connection-name TEXT` : Name of the connection to identify it in Tinybird. `--role-arn TEXT` : The ARN of the IAM role to use for the connection. `--region TEXT` : The Amazon S3 region where the bucket is located. `--policy TEXT` : The Amazon S3 access policy: write or read. `--no-validate` : Don't validate S3 permissions during connection creation. | ## tb datasource [¶](https://www.tinybird.co/docs/about:blank#tb-datasource) Data Sources commands. | Command | Description | Options | | --- | --- | --- | | analyze OPTIONS URL_OR_FILE | Analyzes a URL or a file before creating a new data source. | | | append OPTIONS DATASOURCE_NAME URL | Appends data to an existing Data Source from URL, local file or a connector. | | | connect OPTIONS CONNECTION DATASOURCE_NAME | Deprecated. Use `tb connection create` instead. | `--kafka-topic TEXT` : For Kafka connections: topic. `--kafka-group TEXT` : For Kafka connections: group ID. `--kafka-auto-offset-reset [latest|earliest]` : Kafka auto.offset.reset config. Valid values are: ["latest", "earliest"]. `--kafka-sasl-mechanism [PLAIN|SCRAM-SHA-256|SCRAM-SHA-512]` : Kafka SASL mechanism. Valid values are: ["PLAIN", "SCRAM-SHA-256", "SCRAM-SHA-512"]. Default: "PLAIN". | | copy OPTIONS DATASOURCE_NAME | Copies data source from Main. | `--sql TEXT` : Freeform SQL query to select what is copied from Main into the Branch Data Source. `--sql-from-main` : SQL query selecting * from the same Data Source in Main. `--wait` : Wait for copy job to finish, disabled by default. | | delete OPTIONS DATASOURCE_NAME | Deletes rows from a Data Source. | `--yes` : Doesn't ask for confirmation. `--wait` : Wait for delete job to finish, disabled by default. `--dry-run` : Run the command without deleting anything. `--sql-condition` : Delete rows with SQL condition. | | generate OPTIONS FILENAMES | Generates a Data Source file based on a sample CSV file from local disk or URL. | `--force` : Overrides existing files. | | ls OPTIONS | Lists Data Sources. | `--match TEXT` : Retrieves any resources matching the pattern. eg `--match _test` . `--format [json]` : Force a type of the output. `--dry-run` : Run the command without deleting anything. | | replace OPTIONS DATASOURCE_NAME URL | Replaces the data in a Data Source from a URL, local file or a connector. | `--sql` : The SQL to extract from. `--connector` : Connector name. `--sql-condition` : Delete rows with SQL condition. | | rm OPTIONS DATASOURCE_NAME | Deletes a Data Source. | `--yes` : Doesn't ask for confirmation. | | share OPTIONS DATASOURCE_NAME WORKSPACE_NAME_OR_ID | Shares a Data Source. | `--user_token TEXT` : User token. `--yes` : Don't ask for confirmation. | | sync OPTIONS DATASOURCE_NAME | Syncs from connector defined in .datasource file. | `--yes` : Doesn't ask for confirmation. | | truncate OPTIONS DATASOURCE_NAME | Truncates a Data Source. | `--yes` : Doesn't ask for confirmation. `--cascade` : Truncate dependent Data Source attached in cascade to the given Data Source. | | unshare OPTIONS DATASOURCE_NAME WORKSPACE_NAME_OR_ID | Unshares a Data Source. | `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. `--yes` : Don't ask for confirmation. | | scheduling resume DATASOURCE_NAME | Resumes the scheduling of a Data Source. | | | scheduling pause DATASOURCE_NAME | Pauses the scheduling of a Data Source. | | | scheduling status DATASOURCE_NAME | Gets the scheduling status of a Data Source (paused or running). | | ## tb dependencies [¶](https://www.tinybird.co/docs/about:blank#tb-dependencies) Prints all Data Sources dependencies. Its options: - `--no-deps` : Prints only Data Sources with no Pipes using them. - `--match TEXT` : Retrieves any resource matching the pattern. - `--pipe TEXT` : Retrieves any resource used by Pipe. - `--datasource TEXT` : Retrieves resources depending on this Data Source. - `--check-for-partial-replace` : Retrieves dependant Data Sources that have their data replaced if a partial replace is executed in the Data Source selected. - `--recursive` : Calculates recursive dependencies. ## tb deploy [¶](https://www.tinybird.co/docs/about:blank#tb-deploy) Deploys in Tinybird pushing resources changed from previous release using Git. These are the options available for the `deploy` command: - `--dry-run` : Runs the command with static checks, without creating resources on the Tinybird account or any side effect. Doesn't check for runtime errors. - `-f, --force` : Overrides Pipes when they already exist. - `--override-datasource` : When pushing a Pipe with a materialized node if the target Data Source exists it tries to override it. - `--populate` : Populate materialized nodes when pushing them. - `--subset FLOAT` : Populates with a subset percent of the data (limited to a maximum of 2M rows), this is useful to quickly test a materialized node with some data. The subset must be greater than 0 and lower than 0.1. A subset of 0.1 means a 10% of the data in the source Data Source is used to populate the Materialized View. Use it together with `--populate` , it has precedence over `--sql-condition` . - `--sql-condition TEXT` : Populates with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `--sql-condition='date == toYYYYMM(now())'` it populates taking all the rows from the trigger Data Source which `date` is the current month. Use it together with `--populate` . `--sql-condition` isn't taken into account if the `--subset` param is present. Including in the `sql_condition` any column present in the Data Source `engine_sorting_key` makes the populate job process less data. - `--unlink-on-populate-error` : If the populate job fails the Materialized View is unlinked and new data isn't ingested there. First time a populate job fails, the Materialized View is always unlinked. - `--wait` : To be used along with `--populate` command. Waits for populate jobs to finish, showing a progress bar. Disabled by default. - `--yes` : Doesn't ask for confirmation. - `--workspace_map TEXT..., --workspace TEXT...` : Adds a Workspace path to the list of external Workspaces, usage: `--workspace name path/to/folder` . - `--timeout FLOAT` : Timeout you want to use for the job populate. - `--user_token TOKEN` : The user Token is required for sharing a Data Source that contains the SHARED_WITH entry. ## tb diff [¶](https://www.tinybird.co/docs/about:blank#tb-diff) Diffs local datafiles to the corresponding remote files in the Workspace. It works as a regular `diff` command, useful to know if the remote resources have been changed. Some caveats: - Resources in the Workspace might mismatch due to having slightly different SQL syntax, for instance: A parenthesis mismatch, `INTERVAL` expressions or changes in the schema definitions. - If you didn't specify an `ENGINE_PARTITION_KEY` and `ENGINE_SORTING_KEY` , resources in the Workspace might have default ones. The recommendation in these cases is use `tb pull` to keep your local files in sync. Remote files are downloaded and stored locally in a `.diff_tmp` directory, if working with git you can add it to `.gitignore`. The options for this command: - `--fmt / --no-fmt` : Format files before doing the diff, default is True so both files match the format. - `--no-color` : Don't colorize diff. - `--no-verbose` : List the resources changed not the content of the diff. ## tb fmt [¶](https://www.tinybird.co/docs/about:blank#tb-fmt) Formats a .datasource, .pipe or .incl file. These are the options available for the `fmt` command: - `--line-length INTEGER` : A number indicating the maximum characters per line in the node SQL, lines split based on the SQL syntax and the number of characters passed as a parameter. - `--dry-run` : Don't ask to override the local file. - `--yes` : Don't ask for confirmation to overwrite the local file. - `--diff` : Formats local file, prints the diff and exits 1 if different, 0 if equal. This command removes comments starting with # from the file, so use DESCRIPTION or a comment block instead: ##### Example comment block % {% comment this is a comment and fmt keeps it %} SELECT {% comment this is another comment and fmt keeps it %} count() c FROM stock_prices_1m You can add `tb fmt` to your git `pre-commit` hook to have your files properly formatted. If the SQL formatting results aren't the ones expected to you, you can disable it just for the blocks needed. Read [how to disable fmt](https://docs.sqlfmt.com/getting-started/disabling-sqlfmt). ## tb init [¶](https://www.tinybird.co/docs/about:blank#tb-init) Initializes folder layout. It comes with these options: - `--generate-datasources` : Generates Data Sources based on CSV, NDJSON and Parquet files in this folder. - `--folder DIRECTORY` : Folder where datafiles are placed. - `-f, --force` : Overrides existing files. - `-ir, --ignore-remote` : Ignores remote files not present in the local data project on `tb init --git` . - `--git` : Initializes Workspace with Git commits. - `--override-commit TEXT` : Use this option to manually override the reference commit of your Workspace. This is useful if a commit isn't recognized in your Git log, such as after a force push ( `git push -f` ). ## tb job [¶](https://www.tinybird.co/docs/about:blank#tb-job) Jobs commands. | Command | Description | Options | | --- | --- | --- | | cancel JOB_ID | Tries to cancel a job. | None | | details JOB_ID | Gets details for any job created in the last 48h. | None | | ls [OPTIONS] | Lists jobs. | `--status [waiting|working|done|error]` or `-s` : Shows results with the desired status. | ## tb materialize [¶](https://www.tinybird.co/docs/about:blank#tb-materialize) Analyzes the `node_name` SQL query to generate the .datasource and .pipe files needed to push a new materialize view. This command guides you to generate the Materialized View with name TARGET_DATASOURCE, the only requirement is having a valid Pipe datafile locally. Use `tb pull` to download resources from your Workspace when needed. It allows to use these options: - `--push-deps` : Push dependencies, disabled by default. - `--workspace TEXT...` : Add a Workspace path to the list of external Workspaces, usage: `--workspace name path/to/folder` . - `--no-versions` : When set, resource dependency versions aren't used, it pushes the dependencies as-is. - `--verbose` : Prints more log. - `--unlink-on-populate-error` : If the populate job fails the Materialized View is unlinked and new data isn't ingested in the Materialized View. First time a populate job fails, the Materialized View is always unlinked. ## tb pipe [¶](https://www.tinybird.co/docs/about:blank#tb-pipe) Use the following commands to manage Pipes. | Command | Description | Options | | --- | --- | --- | | append OPTIONS PIPE_NAME_OR_UID SQL | Appends a node to a Pipe. | | | copy pause OPTIONS PIPE_NAME_OR_UID | Pauses a running Copy Pipe. | | | copy resume OPTIONS PIPE_NAME_OR_UID | Resumes a paused Copy Pipe. | | | copy run OPTIONS PIPE_NAME_OR_UID | Runs an on-demand copy job. | `--wait` : Waits for the copy job to finish. `--yes` : Doesn't ask for confirmation. `--param TEXT` : Key and value of the params you want the Copy Pipe to be called with. For example: `tb pipe copy run --param foo=bar` . | | data OPTIONS PIPE_NAME_OR_UID PARAMETERS | Prints data returned by a Pipe. You can pass query parameters to the command, for example `--param_name value` . | `--query TEXT` : Runs SQL over Pipe results. `--format [json|csv]` : Return format (CSV, JSON). `-- value` : Query parameter. You can define multiple parameters and their value. For example, `--paramOne value --paramTwo value2` . | | generate OPTIONS NAME QUERY | Generates a Pipe file based on a sql query. Example: `tb pipe generate my_pipe 'select * from existing_datasource'` . | `--force` : Overrides existing files. | | ls OPTIONS | Lists Pipes. | `--match TEXT` : Retrieves any resourcing matching the pattern. For example `--match _test` . `--format [json|csv]` : Force a type of the output. | | populate OPTIONS PIPE_NAME | Populates the result of a Materialized node into the target Materialized View. | `--node TEXT` : Name of the materialized Node. Required. `--sql-condition TEXT` : Populate with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `--sql-condition='date == toYYYYMM(now())'` it populates taking all the rows from the trigger Data Source which `date` is the current month. Use it together with `--populate` . `--sql-condition` isn't taken into account if the `--subset` param is present. Including in the `sql_condition` any column present in the Data Source `engine_sorting_key` makes the populate job process less data. `--truncate` : Truncates the materialized Data Source before populating it. `--unlink-on-populate-error` : If the populate job fails the Materialized View is unlinked and new data isn't ingested in the Materialized View. First time a populate job fails, the Materialized View is always unlinked. `--wait` : Waits for populate jobs to finish, showing a progress bar. Disabled by default. | | publish OPTIONS PIPE_NAME_OR_ID NODE_UID | Changes the published node of a Pipe. | | | regression-test OPTIONS FILENAMES | Runs regression tests using last requests. | `--debug` : Prints internal representation, can be combined with any command to get more information. `--only-response-times` : Checks only response times. `--workspace_map TEXT..., --workspace TEXT...` : Add a Workspace path to the list of external Workspaces, usage: `--workspace name path/to/folder` . `--no-versions` : When set, resource dependency versions aren't used, it pushes the dependencies as-is. `-l, --limit INTEGER RANGE` : Number of requests to validate [0<=x<=100]. `--sample-by-params INTEGER RANGE` : When set, aggregates the pipe_stats_rt requests by `extractURLParameterNames(assumeNotNull(url))` and for each combination takes a sample of N requests [1<=x<=100]. `-m, --match TEXT` : Filters the checker requests by specific parameter. You can pass multiple parameters -m foo -m bar. `-ff, --failfast` : When set, the checker exits as soon as one test fails. `--ignore-order` : When set, the checker ignores the order of list properties. `--validate-processed-bytes` : When set, the checker validates that the new version doesn't process more than 25% than the current version. `--relative-change FLOAT` : When set, the checker validates the new version has less than this distance with the current version. | | rm OPTIONS PIPE_NAME_OR_ID | Deletes a Pipe. PIPE_NAME_OR_ID can be either a Pipe name or id in the Workspace or a local path to a .pipe file. | `--yes` : Doesn't ask for confirmation. | | set_endpoint OPTIONS PIPE_NAME_OR_ID NODE_UID | Same as 'publish', changes the published node of a Pipe. | | | sink run OPTIONS PIPE_NAME_OR_UID | Runs an on-demand sink job. | `--wait` : Waits for the sink job to finish. `--yes` : Don't ask for confirmation. `--dry-run` : Run the command without executing the sink job. `--param TEXT` : Key and value of the params you want the Sink Pipe to be called with. For example: `tb pipe sink run --param foo=bar` . | | stats OPTIONS PIPES | Prints Pipe stats for the last 7 days. | `--format [json]` : Forces a type of the output. To parse the output, keep in mind to use `tb --no-version-warning pipe stats` option. | | token_read OPTIONS PIPE_NAME | Retrieves a Token to read a Pipe. | | | unlink OPTIONS PIPE_NAME NODE_UID | Unlinks the output of a Pipe, whatever its type: Materialized Views, Copy Pipes, or Sinks. | | | unpublish OPTIONS PIPE_NAME NODE_UID | Unpublishes the endpoint of a Pipe. | | ## tb prompt [¶](https://www.tinybird.co/docs/about:blank#tb-prompt) Provides instructions to configure the shell prompt for Tinybird CLI. See [Configure your shell prompt](https://www.tinybird.co/docs/docs/cli/install#configure-your-shell-prompt). ## tb pull [¶](https://www.tinybird.co/docs/about:blank#tb-pull) Retrieves the latest version for project files from your Workspace. With these options: - `--folder DIRECTORY` : Folder where files are placed. - `--auto / --no-auto` : Saves datafiles automatically into their default directories (/datasources or /pipes). Default is True. - `--match TEXT` : Retrieve any resourcing matching the pattern. eg `--match _test` . - `-f, --force` : Override existing files. - `--fmt` : Format files, following the same format as `tb fmt` . ## tb push [¶](https://www.tinybird.co/docs/about:blank#tb-push) Push files to your Workspace. You can use this command with these options: - `--dry-run` : Runs the command with static checks, without creating resources on the Tinybird account or any side effect. Doesn't check for runtime errors. - `--check / --no-check` : Enables/disables output checking, enabled by default. - `--push-deps` : Pushes dependencies, disabled by default. - `--only-changes` : Pushes only the resources that have changed compared to the destination Workspace. - `--debug` : Prints internal representation, can be combined with any command to get more information. - `-f, --force` : Overrides Pipes when they already exist. - `--override-datasource` : When pushing a Pipe with a materialized node if the target Data Source exists it tries to override it. - `--populate` : Populates materialized nodes when pushing them. - `--subset FLOAT` : Populates with a subset percent of the data (limited to a maximum of 2M rows), this is useful to quickly test a materialized node with some data. The subset must be greater than 0 and lower than 0.1. A subset of 0.1 means a 10 percent of the data in the source Data Source is used to populate the Materialized View. Use it together with `--populate` , it has precedence over `--sql-condition` . - `--sql-condition TEXT` : Populates with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `--sql-condition='date == toYYYYMM(now())'` it populates taking all the rows from the trigger Data Source which `date` is the current month. Use it together with `--populate` . `--sql-condition` isn't taken into account if the `--subset` param is present. Including in the `sql_condition` any column present in the Data Source `engine_sorting_key` makes the populate job process less data. - `--unlink-on-populate-error` : If the populate job fails the Materialized View is unlinked and new data isn't ingested in the Materialized View. First time a populate job fails, the Materialized View is always unlinked. - `--fixtures` : Appends fixtures to Data Sources. - `--wait` : To be used along with `--populate` command. Waits for populate jobs to finish, showing a progress bar. Disabled by default. - `--yes` : Doesn't ask for confirmation. - `--only-response-times` : Checks only response times, when --force push a Pipe. - `--workspace TEXT..., --workspace_map TEXT...` : Add a Workspace path to the list of external Workspaces, usage: `--workspace name path/to/folder` . - `--no-versions` : When set, resource dependency versions aren't used, it pushes the dependencies as-is. - `--timeout FLOAT` : Timeout you want to use for the populate job. - `-l, --limit INTEGER RANGE` : Number of requests to validate [0<=x<=100]. - `--sample-by-params INTEGER RANGE` : When set, aggregates the `pipe_stats_rt` requests by `extractURLParameterNames(assumeNotNull(url))` and for each combination takes a sample of N requests [1<=x<=100]. - `-ff, --failfast` : When set, the checker exits as soon one test fails. - `--ignore-order` : When set, the checker ignores the order of list properties. - `--validate-processed-bytes` : When set, the checker validates that the new version doesn't process more than 25% than the current version. - `--user_token TEXT` : The User Token is required for sharing a Data Source that contains the SHARED_WITH entry. ## tb sql [¶](https://www.tinybird.co/docs/about:blank#tb-sql) Runs SQL queries over Data Sources and Pipes. - `--rows_limit INTEGER` : Max number of rows retrieved. - `--pipeline TEXT` : The name of the Pipe to run the SQL Query. - `--pipe TEXT` : The path to the .pipe file to run the SQL Query of a specific NODE. - `--node TEXT` : The NODE name. - `--format [json|csv|human]` : Output format. - `--stats / --no-stats` : Shows query stats. ## tb test [¶](https://www.tinybird.co/docs/about:blank#tb-test) Test commands. | Command | Description | Options | | --- | --- | --- | | init | Initializes a file list with a simple test suite. | `--force` : Overrides existing files. | | parse [OPTIONS] [FILES] | Reads the contents of a test file list. | | | run [OPTIONS] [FILES] | Runs the test suite, a file, or a test. | `--verbose` or `-v` : Shows results. `--fail` : Show only failed/error tests. `--concurrency [INTEGER RANGE]` or `-c [INTEGER RANGE]` : How many tests to run concurrently. | ## tb token [¶](https://www.tinybird.co/docs/about:blank#tb-token) Manage your Workspace Tokens. | Command | Description | Options | | --- | --- | --- | | copy OPTIONS TOKEN_ID | Copies a Token. | | | ls OPTIONS | Lists Tokens. | `--match TEXT` : Retrieves any Token matching the pattern. eg `--match _test` . | | refresh OPTIONS TOKEN_ID | Refreshes a Token. | `--yes` : Doesn't ask for confirmation. | | rm OPTIONS TOKEN_ID | Removes a Token. | `--yes` : Doesn't ask for confirmation. | | scopes OPTIONS TOKEN_ID | Lists Token scopes. | | | create static OPTIONS TOKEN_NAME | Creates a static Token that lasts forever. | `--scope` : Scope for the Token (e.g., `DATASOURCES:READ` ). Required. `--resource` : Resource you want to associate the scope with. `--filter` : SQL condition used to filter the values when calling with this token (eg. `--filter=value > 0` ). | | create jwt OPTIONS TOKEN_NAME | Creates a JWT Token with a fixed expiration time. | `--ttl` : Time to live (e.g., '1h', '30min', '1d'). Required. `--scope` : Scope for the token (only `PIPES:READ` is allowed for JWT tokens).Required. `--resource` : Resource associated with the scope. Required. `--fixed-params` : Fixed parameters in key=value format, multiple values separated by commas. | ## tb workspace [¶](https://www.tinybird.co/docs/about:blank#tb-workspace) Manage your Workspaces. | Command | Description | Options | | --- | --- | --- | | clear OPTIONS | Drop all the resources inside a project. This command is dangerous because it removes everything, use with care. | `--yes` : Don't ask for confirmation. `--dry-run` : Run the command without removing anything. | | create OPTIONS WORKSPACE_NAME | Creates a new Workspace for your Tinybird user. | `--starter_kit TEXT` : Uses a Tinybird starter kit as a template. `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. `--fork` : When enabled, Tinybird shares all Data Sources from the current Workspace to the new created one. | | current OPTIONS | Shows the Workspace you're currently authenticated to. | | | delete OPTIONS WORKSPACE_NAME_OR_ID | Deletes a Workspace where you are an admin. | `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. `--yes` : Don't ask for confirmation. | | ls OPTIONS | Lists all the Workspaces you have access to in the account you're currently authenticated to. | | | members add OPTIONS MEMBERS_EMAILS | Adds members to the current Workspace. | `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. | | members ls OPTIONS | Lists members in the current Workspace. | | | members rm OPTIONS | Removes members from the current Workspace. | `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. | | members set-role OPTIONS [guest|viewer|admin] MEMBERS_EMAILS | Sets the role for existing Workspace members. | `--user_token TEXT` : When passed, Tinybird won't prompt asking for it. | | use OPTIONS WORKSPACE_NAME_OR_ID | Switches to another workspace. Use `tb workspace ls` to list the workspaces you have access to. | | ## tb tag [¶](https://www.tinybird.co/docs/about:blank#tb-tag) Manage your Workspace tags. | Command | Description | Options | | --- | --- | --- | | create TAG_NAME | Creates a tag in the current Workspace. | | | ls | List all the tags of the current Workspace. | | | ls TAG_NAME | List all the resources tagged with the given tag. | | | rm TAG_NAME | Removes a tag from the current Workspace. All resources aren't tagged by the given tag anymore. | `--yes` : Don't ask for confirmation. | --- URL: https://www.tinybird.co/docs/cli/advanced-templates Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Advanced templates · Tinybird Docs" theme-color: "#171612" description: "Advanced usage of Tinybird datafiles, including how to work with query parameters." --- # Advanced templates [¶](https://www.tinybird.co/docs/about:blank#advanced-templates) When developing multiple use cases, you might want to reuse certain parts or steps of an analysis, such as data filters or similar table operations. Read on to learn about advanced usage of the datafile system when using the [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/quick-start) . Before reading this page, familiarize yourself with [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters). ## Templates reuse [¶](https://www.tinybird.co/docs/about:blank#templates-reuse) You can set up templates to reuse certain parts or steps of an analysis, such as data filters or similar table operations. To follow along, clone the [ecommerce_data_project_advanced](https://github.com/tinybirdco/ecommerce_data_project_advanced) repository: ##### Clone demo git clone https://github.com/tinybirdco/ecommerce_data_project_advanced.git cd ecommerce_data_project_advanced The repository contains the following file structure: ##### File structure ecommerce_data_project/ datasources/ events.datasource mv_top_per_day.datasource products.datasource fixtures/ events.csv products.csv endpoints/ sales.pipe top_products_between_dates.pipe top_products_last_week.pipe includes/ only_buy_events.incl top_products.incl pipes/ top_product_per_day.pipe Take a look at the `sales.pipe` API Endpoint and the `top_product_per_day.pipe` Pipe that materializes to a `mv_top_per_day` Data Source. Both use the same node, `only_buy_events` , through the usage of [include](https://www.tinybird.co/docs/docs/cli/datafiles/include-files) files: - only_buy_events - sales_pipes - top_products ##### includes/only_buy_events.incl NODE only_buy_events SQL > SELECT toDate(timestamp) date, product, joinGet('products_join_by_id', 'color', product) as color, JSONExtractFloat(json, 'price') as price FROM events where action = 'buy' When using include files to reuse logic in .datasource files, the extension of the file must be `.datasource.incl`. ## Include variables [¶](https://www.tinybird.co/docs/about:blank#include-variables) You can include variables in a node template. The following example shows two API Endpoints that display the 10 top products, each filtered by different date intervals: - top_products_1 - top_products_2 - top_products_3 ##### includes/top_products.incl NODE endpoint DESCRIPTION > returns top 10 products for the last week SQL > select date, topKMerge(10)(top_10) as top_10 from top_product_per_day {% if '$DATE_FILTER' = 'last_week' %} where date > today() - interval 7 day {% else %} where date between {{Date(start)}} and {{Date(end)}} {% end %} group by date In the previous examples, the `DATE_FILTER` variable is sent to the `top_products` include, where the variable content is retrieved using the `$` prefix with the `DATE_FILTER` reference. You can also assign an array of values to an include variable. To do this, parse the variable using function templates, as explained in [Template functions](https://www.tinybird.co/docs/docs/cli/template-functions). ### Variables and parameters [¶](https://www.tinybird.co/docs/about:blank#variables-and-parameters) Parameters are variables whose value you can change through the API Endpoint request parameters. Variables only live in the template and you can set them when declaring the `INCLUDE` or with the `set` template syntax. For example: ##### Using 'set' to declare a variable {% set my_var = 'default' %} By default, variables are interpreted as parameters. To prevent variables or private parameters from appearing in the auto-generated API Endpoint documentation, they need to start with `_` . For example: ##### Define private variables % SELECT date FROM my_table WHERE a > 10 {% if defined(_private_param) %} and b = {{Int32(_private_param)}} {% end %} You also need to use `_` as a prefix when using variables in template functions. See [Template functions](https://www.tinybird.co/docs/docs/cli/template-functions) for more information. --- URL: https://www.tinybird.co/docs/api-reference/token-api Last update: 2024-12-30T16:50:50.000Z Content: --- title: "Token API Reference · Tinybird Docs" theme-color: "#171612" description: "In order to read, append or import data into you Tinybird account, you'll need a Token with the right permissions." --- GET /v0/tokens/? [¶](https://www.tinybird.co/docs/about:blank#get--v0-tokens-?) Retrieves all workspace Static Tokens. Get all tokens [¶](https://www.tinybird.co/docs/about:blank#id2) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens" A list of your Static Tokens and their scopes will be sent in the response. Successful response [¶](https://www.tinybird.co/docs/about:blank#id3) { "tokens": [ { "name": "admin token", "description": "", "scopes": [ { "type": "ADMIN" } ], "token": "p.token" }, { "name": "import token", "description": "", "scopes": [ { "type": "DATASOURCES:CREATE" } ], "token": "p.token0" }, { "name": "token name 1", "description": "", "scopes": [ { "type": "DATASOURCES:READ", "resource": "table_name_1" }, { "type": "DATASOURCES:APPEND", "resource": "table_name_1" } ], "token": "p.token1" }, { "name": "token name 2", "description": "", "scopes": [ { "type": "PIPES:READ", "resource": "pipe_name_2" } ], "token": "p.token2" } ] } POST /v0/tokens/? [¶](https://www.tinybird.co/docs/about:blank#post--v0-tokens-?) Creates a new Token: Static or JWT Creating a new Static Token [¶](https://www.tinybird.co/docs/about:blank#id4) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/" \ -d "name=test&scope=DATASOURCES:APPEND:table_name&scope=DATASOURCES:READ:table_name"| Key | Type | Description | | --- | --- | --- | | name | String | Name of the token | | description | String | Optional. Markdown text with a description of the token. | | scope | String | Scope(s) to set. Format is[ SCOPE:TYPE[:arg][:filter]](about:blank#id2) . This is only used for the Static Tokens | Successful response [¶](https://www.tinybird.co/docs/about:blank#id6) { "name": "token_name", "description": "", "scopes": [ { "type": "DATASOURCES:APPEND", "resource": "table_name" } { "type": "DATASOURCES:READ", "resource": "table_name", "filter": "department = 1" }, ], "token": "p.token" } When creating a token with `filter` whenever you use the token to read the table, it will be filtered. For example, if table is `events_table` and `filter` is `date > '2018-01-01' and type == 'foo'` a query like `select count(1) from events_table` will become `select count(1) from events_table where date > '2018-01-01' and type == 'foo'` Creating a new token with filter [¶](https://www.tinybird.co/docs/about:blank#id7) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/" \ -d "name=test&scope=DATASOURCES:READ:table_name:column==1" If we provide an `expiration_time` in the URL, the token will be created as a JWT Token. Creating a new JWT Token [¶](https://www.tinybird.co/docs/about:blank#id8) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens?name=jwt_token&expiration_time=1710000000" \ -d '{"scopes": [{"type": "PIPES:READ", "resource": "requests_per_day", "fixed_params": {"user_id": 3}}]}' In multi-tenant applications, you can use this endpoint to create a JWT token for a specific tenant where each user has their own token with a fixed set of scopes and parameters POST /v0/tokens/(.+)/refresh [¶](https://www.tinybird.co/docs/about:blank#post--v0-tokens-(.+)-refresh) Refresh the Static Token without modifying name, scopes or any other attribute. Specially useful when a Token is leaked, or when you need to rotate a Token. Refreshing a Static Token [¶](https://www.tinybird.co/docs/about:blank#id9) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/:token_name/refresh" When successfully refreshing a token, new information will be sent in the response Successful response [¶](https://www.tinybird.co/docs/about:blank#id10) { "name": "token name", "description": "", "scopes": [ { "type": "DATASOURCES:READ", "resource": "table_name" } ], "token": "NEW_TOKEN" }| Key | Type | Description | | --- | --- | --- | | auth_token | String | Token. Ensure it has the `TOKENS` scope on it | | Code | Description | | --- | --- | | 200 | No error | | 403 | Forbidden. Provided token doesn’t have permissions to drop the token. A token is not allowed to remove itself, it needs `ADMIN` or `TOKENS` scope | GET /v0/tokens/(.+) [¶](https://www.tinybird.co/docs/about:blank#get--v0-tokens-(.+)) Fetches information about a particular Static Token. Getting token info [¶](https://www.tinybird.co/docs/about:blank#id13) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/:token" Returns a json with name and scopes. Successful response [¶](https://www.tinybird.co/docs/about:blank#id14) { "name": "token name", "description": "", "scopes": [ { "type": "DATASOURCES:READ", "resource": "table_name" } ], "token": "p.TOKEN" } DELETE /v0/tokens/(.+) [¶](https://www.tinybird.co/docs/about:blank#delete--v0-tokens-(.+)) Deletes a Static Token . Deleting a token [¶](https://www.tinybird.co/docs/about:blank#id15) curl -X DELETE \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/:token" PUT /v0/tokens/(.+) [¶](https://www.tinybird.co/docs/about:blank#put--v0-tokens-(.+)) Modifies a Static Token. More than one scope can be sent per request, all of them will be added as Token scopes. Every time a Token scope is modified, it overrides the existing one(s). editing a token [¶](https://www.tinybird.co/docs/about:blank#id16) curl -X PUT \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/tokens/?" \ -d "name=test_new_name&description=this is a test token&scope=PIPES:READ:test_pipe&scope=DATASOURCES:CREATE"| Key | Type | Description | | --- | --- | --- | | token | String | Token. Ensure it has the `TOKENS` scope on it | | name | String | Optional. Name of the token. | | description | String | Optional. Markdown text with a description of the token. | | scope | String | Optional. Scope(s) to set. Format is[ SCOPE:TYPE[:arg][:filter]](about:blank#id2) . New scope(s) will override existing ones. | Successful response [¶](https://www.tinybird.co/docs/about:blank#id18) { "name": "test", "description": "this is a test token", "scopes": [ { "type": "PIPES:READ", "resource": "test_pipe" }, { "type": "DATASOURCES:CREATE" } ] } --- URL: https://www.tinybird.co/docs/api-reference/sink-pipes-api Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Sink Pipes API Reference · Tinybird Docs" theme-color: "#171612" description: "The Sink Pipes API allows you to create, delete, schedule, and trigger Sink Pipes." --- # Sink Pipes API [¶](https://www.tinybird.co/docs/about:blank#sink-pipes-api) The Sink Pipes API allows you to create, delete, schedule, and trigger Sink Pipes. ## POST /v0/pipes/{pipe_id}/nodes/{node_id}/sink [¶](https://www.tinybird.co/docs/about:blank#post-v0pipespipe-idnodesnode-idsink) Set the Pipe as a Sink Pipe, optionally scheduled. Required token permission is `PIPES:CREATE`. ### Restrictions [¶](https://www.tinybird.co/docs/about:blank#restrictions) - You can set only one schedule per Sink Pipe. - You can't set a Sink Pipe if the Pipe is already materializing. You must unlink the Materialization first. - You can't set a Sink Pipe if the Pipe is already an API Endpoint. You must unpublish the API Endpoint first. - You can't set a Sink Pipe if the Pipe is already copying. You must unset the copy first. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) ##### Setting the Pipe as a Sink Pipe curl \ -X POST "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/sink" \ -H "Authorization: Bearer " \ -d "connection=my_connection_name" \ -d "path=s3://bucket-name/prefix" \ -d "file_template=exported_file_template" \ -d "format=csv" \ -d "compression=gz" \ -d "schedule_cron=0 */1 * * *" \ -d "write_strategy=new" ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) | Key | Type | Description | | --- | --- | --- | | connection | String | Name of the connection to holding the credentials to run the sink | | path | String | Object store prefix into which the sink will write data | | file_template | String | File template string. See[ file template](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink#file-template) for more details | | format | String | Optional. Format of the exported files. Default: CSV | | compression | String | Optional. Compression of the output files. Default: None | | schedule_cron | String | Optional. The sink's execution schedule, in crontab format. | | write_strategy | String | Optional. Default: `new` . The sink's write strategy for filenames already existing in the bucket. Values: `new` , `truncate` ; `new` adds a new file with a suffix, while `truncate` replaces the existent file. | ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "id": "t_529f46626c324674b3a84cd820ac2649", "name": "p_test", "description": null, "endpoint": null, "created_at": "2024-01-18 12:57:36.503834", "updated_at": "2024-01-18 13:01:21.435012", "parent": null, "type": "sink", "last_commit": { "content_sha": "", "path": "", "status": "changed" }, "sink_node": "t_6e8afdb8c691459b80e16541433f951b", "schedule": { "timezone": "Etc/UTC", "cron": "0 */1 * * *", "status": "running" }, "nodes": [ { "id": "t_6e8afdb8c691459b80e16541433f951b", "name": "p_test_0", "sql": "SELECT * FROM test", "description": null, "materialized": null, "cluster": null, "tags": {}, "created_at": "2024-01-18 12:57:36.503843", "updated_at": "2024-01-18 12:57:36.503843", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "node_type": "sink", "dependencies": [ "test" ], "params": [] } ] } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 404 | Pipe, Node, or data connector not found, bucket doesn't exist | | 403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe, can't assume role | | 401 | Invalid credentials (from connection) | | 400 | Invalid or missing parameters, bad ARN role, invalid region name | ## DELETE /v0/pipes/{pipe_id}/nodes/{node_id}/sink [¶](https://www.tinybird.co/docs/about:blank#delete-v0pipespipe-idnodesnode-idsink) Removes the Sink from the Pipe. This doesn't delete the Pipe nor the Node, only the sink configuration and any associated settings. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X DELETE "https://api.tinybird.co/v0/pipes/$1/nodes/$2/sink" \ -H "Authorization: Bearer " Successful response example { "id": "t_529f46626c324674b3a84cd820ac2649", "name": "p_test", "description": null, "endpoint": null, "created_at": "2024-01-18 12:57:36.503834", "updated_at": "2024-01-19 09:27:12.069650", "parent": null, "type": "default", "last_commit": { "content_sha": "", "path": "", "status": "changed" }, "nodes": [ { "id": "t_6e8afdb8c691459b80e16541433f951b", "name": "p_test_0", "sql": "SELECT * FROM test", "description": null, "materialized": null, "cluster": null, "tags": {}, "created_at": "2024-01-18 12:57:36.503843", "updated_at": "2024-01-19 09:27:12.069649", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "node_type": "standard", "dependencies": [ "test" ], "params": [] } ], "url": "https://api.split.tinybird.co/v0/pipes/p_test.json" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 404 | Pipe, Node, or data connector not found | | 403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe | | 400 | Invalid or missing parameters, Pipe isn't a Sink Pipe | ## POST /v0/pipes/{pipe_id}/sink [¶](https://www.tinybird.co/docs/about:blank#post-v0pipespipe-idsink) Triggers the Sink Pipe, creating a sink job. Allows overriding some of the sink settings for this particular execution. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) ##### Trigger a Sink Pipe with some overrides curl \ -X POST "https://api.tinybird.co/v0/pipes/p_test/sink" \ -H "Authorization: Bearer " \ -d "file_template=export_file" \ -d "format=csv" \ -d "compression=gz" \ -d "write_strategy=truncate" \ -d {key}={val} ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) | Key | Type | Description | | --- | --- | --- | | connection | String | Name of the connection to holding the credentials to run the sink | | path | String | Object store prefix into which the sink will write data | | file_template | String | File template string. See[ file template](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink#file-template) for more details | | format | String | Optional. Format of the exported files. Default: CSV | | compression | String | Optional. Compression of the output files. Default: None | | write_strategy | String | Optional. The sink's write strategy for filenames already existing in the bucket. Values: `new` , `truncate` ; `new` adds a new file with a suffix, while `truncate` replaces the existent file. | | {key} | String | Optional. Additional variables to be injected into the file template. See[ file template](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink#file-template) for more details | ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "id": "t_6e8afdb8c691459b80e16541433f951b", "name": "p_test_0", "sql": "SELECT * FROM test", "description": null, "materialized": null, "cluster": null, "tags": {}, "created_at": "2024-01-18 12:57:36.503843", "updated_at": "2024-01-19 09:27:12.069649", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "node_type": "sink", "dependencies": [ "test" ], "params": [], "job": { "id": "685e7395-3b08-492b-9fe8-2944859d6a06", "kind": "sink", "status": "waiting", "created_at": "2024-01-19 15:58:46.688525", "updated_at": "2024-01-19 15:58:46.688532", "is_cancellable": true, "job_url": "https://api.split.tinybird.co/v0/jobs/685e7395-3b08-492b-9fe8-2944859d6a06", "pipe": { "id": "t_529f46626c324674b3a84cd820ac2649", "name": "p_test" } } } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 404 | Pipe, Node, or data connector not found | | 403 | Limit reached, Query includes forbidden keywords, Pipe is already a Sink Pipe | | 400 | Invalid or missing parameters, Pipe isn't a Sink Pipe | ## GET /v0/integrations/s3/policies/trust-policy [¶](https://www.tinybird.co/docs/about:blank#get-v0integrationss3policiestrust-policy) Retrieves the trust policy to be attached to the IAM role that will be used for the connection. External IDs are different for each Workspace, but shared between Branches of the same Workspace to avoid having to change the trust policy for each Branch. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X GET "https://$TB_HOST/v0/integrations/s3/policies/trust-policy" \ -H "Authorization: Bearer " Successful response example { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Principal": { "AWS": "arn:aws:iam::123456789:root" }, "Condition": { "StringEquals": { "sts:ExternalId": "c6ee2795-aae3-4a55-a7a1-92d92fab0e41" } } } ] } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 404 | S3 integration not supported in your region | ## GET /v0/integrations/s3/policies/write-access-policy [¶](https://www.tinybird.co/docs/about:blank#get-v0integrationss3policieswrite-access-policy) Retrieves the trust policy to be attached to the IAM Role that will be used for the connection. External IDs are different for each Workspace, but shared between branches of the same Workspace to avoid having to change the trust policy for each branch. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X GET "https://$TB_HOST/v0/integrations/s3/policies/write-access-policy?bucket=test-bucket" \ -H "Authorization: Bearer " ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) | Key | Type | Description | | --- | --- | --- | | bucket | Optional[String] | Bucket to use for rendering the template. If not provided the '' placeholder is used | ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": "arn:aws:s3:::" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::/*" } ] } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | ## GET /v0/integrations/s3/settings [¶](https://www.tinybird.co/docs/about:blank#get-v0integrationss3settings) Retrieves the settings to be attached to the IAM role that will be used for the connection. External IDs are different for each Workspace, but shared between Branches of the same Workspace to avoid having to generate specific IAM roles for each of the Branches. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X GET "https://$TB_HOST/v0/integrations/s3/settings" \ -H "Authorization: Bearer " Successful response example { "principal": "arn:aws:iam:::root", "external_id": "" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 404 | S3 integration not supported in your region | ## GET /v0/datasources-bigquery-credentials [¶](https://www.tinybird.co/docs/about:blank#get-v0datasources-bigquery-credentials) Retrieves the Workspace's GCP service account to be authorized to write to the destination bucket. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X POST "${TINYBIRD_HOST}/v0/connectors" \ -H "Authorization: Bearer " \ -d "service=gcs_service_account" \ -d "name=" ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) None ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "account": "cdk-E-d83f6d01-b5c1-40-43439d@development-353413.iam.gserviceaccount.com" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 503 | Feature not enabled in your region | --- URL: https://www.tinybird.co/docs/api-reference/query-api Last update: 2024-12-30T16:50:50.000Z Content: --- title: "Query API Reference · Tinybird Docs" theme-color: "#171612" description: "The Query API allows you to query your Pipes inside Tinybird as if you were running SQL statements against a regular database." --- GET /v0/sql [¶](https://www.tinybird.co/docs/about:blank#get--v0-sql) Executes a SQL query using the engine. Running sql queries against your data [¶](https://www.tinybird.co/docs/about:blank#id1) curl --data "SELECT * FROM " https://api.tinybird.co/v0/sql As a response, it gives you the query metadata, the resulting data and some performance statistics. Successful response [¶](https://www.tinybird.co/docs/about:blank#id2) { "meta": [ { "name": "VendorID", "type": "Int32" }, { "name": "tpep_pickup_datetime", "type": "DateTime" } ], "data": [ { "VendorID": 2, "tpep_pickup_datetime": "2001-01-05 11:45:23", "tpep_dropoff_datetime": "2001-01-05 11:52:05", "passenger_count": 5, "trip_distance": 1.53, "RatecodeID": 1, "store_and_fwd_flag": "N", "PULocationID": 71, "DOLocationID": 89, "payment_type": 2, "fare_amount": 7.5, "extra": 0.5, "mta_tax": 0.5, "tip_amount": 0, "tolls_amount": 0, "improvement_surcharge": 0.3, "total_amount": 8.8 }, { "VendorID": 2, "tpep_pickup_datetime": "2002-12-31 23:01:55", "tpep_dropoff_datetime": "2003-01-01 14:59:11" } ], "rows": 3, "rows_before_limit_at_least": 4, "statistics": { "elapsed": 0.00091042, "rows_read": 4, "bytes_read": 296 } } Data can be fetched in different formats. Just append `FORMAT ` to your SQL query: Requesting different formats with SQL [¶](https://www.tinybird.co/docs/about:blank#id3) SELECT count () from < pipe > FORMAT JSON| Key | Type | Description | | --- | --- | --- | | q | String | The SQL query | | pipeline | String | (Optional) The name of the pipe. It allows writing a query like ‘SELECT * FROM _’ where ‘_’ is a placeholder for the ‘pipeline’ parameter | | output_format_json_quote_64bit_integers | int | (Optional) Controls quoting of 64-bit or bigger integers (like UInt64 or Int128) when they are output in a JSON format. Such integers are enclosed in quotes by default. This behavior is compatible with most JavaScript implementations. Possible values: 0 — Integers are output without quotes. 1 — Integers are enclosed in quotes. Default value is 0 | | output_format_json_quote_denormals | int | (Optional) Controls representation of inf and nan on the UI instead of null e.g when dividing by 0 - inf and when there is no representation of a number in Javascript - nan. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | | output_format_parquet_string_as_string | int | (Optional) Use Parquet String type instead of Binary for String columns. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | | format | Description | | --- | --- | | CSV | CSV without header | | CSVWithNames | CSV with header | | JSON | JSON including data, statistics and schema information | | TSV | TSV without header | | TSVWithNames | TSV with header | | PrettyCompact | Formatted table | | JSONEachRow | Newline-delimited JSON values (NDJSON) | | Parquet | Apache Parquet | | Prometheus | Prometheus text-based format | As you can see in the example above, timestamps do not include a time zone in their serialization. Let’s see how that relates to timestamps ingested from your original data: - If the original timestamp had no time zone associated, you’ll read back the same date and time verbatim. If you ingested the timestamp `2022-11-14 11:08:46` , for example, Tinybird sends `"2022-11-14 11:08:46"` back. This is so regardless of the time zone of the column in ClickHouse. - If the original timestamp had a time zone associated, you’ll read back the corresponding date and time in the time zone of the destination column in ClickHouse, which is UTC by default. If you ingested `2022-11-14 12:08:46.574295 +0100` , for instance, Tinybird sends `"2022-11-14 11:08:46"` back for a `DateTime` , and `"2022-11-14 06:08:46"` for a `DateTime('America/New_York')` . POST /v0/sql [¶](https://www.tinybird.co/docs/about:blank#post--v0-sql) Executes a SQL query using the engine, while providing a templated or non templated query string and the custom parameters that will be translated into the query. The custom parameters provided should not have the same name as the request parameters for this endpoint (outlined below), as they are reserved in order to get accurate results for your query. Running sql queries against your data [¶](https://www.tinybird.co/docs/about:blank#id6) For example: 1. Providing the value to the query via the POST body: curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/sql" -d \ '{ "q":"% SELECT * FROM where column_name = {{String(column_name)}}", "column_name": "column_name_value" }' 2. Providing a new value to the query from the one defined within the pipe in the POST body: curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/sql" -d \ '{ "q":"% SELECT * FROM where column_name = {{String(column_name, "column_name_value")}}", "column_name": "new_column_name_value" }' 3. Providing a non template query in the POST body: curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/sql" -d \ '{ "q":"SELECT * FROM " }' 4. Providing a non template query as a string in the POST body with a content type of "text/plain" : curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: text/plain" \ "https://api.tinybird.co/v0/sql" -d "SELECT * FROM "| Key | Type | Description | | --- | --- | --- | | pipeline | String | (Optional) The name of the pipe. It allows writing a query like ‘SELECT * FROM _’ where ‘_’ is a placeholder for the ‘pipeline’ parameter | | output_format_json_quote_64bit_integers | int | (Optional) Controls quoting of 64-bit or bigger integers (like UInt64 or Int128) when they are output in a JSON format. Such integers are enclosed in quotes by default. This behavior is compatible with most JavaScript implementations. Possible values: 0 — Integers are output without quotes. 1 — Integers are enclosed in quotes. Default value is 0 | | output_format_json_quote_denormals | int | (Optional) Controls representation of inf and nan on the UI instead of null e.g when dividing by 0 - inf and when there is no representation of a number in Javascript - nan. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | | output_format_parquet_string_as_string | int | (Optional) Use Parquet String type instead of Binary for String columns. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | As a response, it gives you the query metadata, the resulting data and some performance statistics. Successful response [¶](https://www.tinybird.co/docs/about:blank#id8) { "meta": [ { "name": "VendorID", "type": "Int32" }, { "name": "tpep_pickup_datetime", "type": "DateTime" } ], "data": [ { "VendorID": 2, "tpep_pickup_datetime": "2001-01-05 11:45:23", "tpep_dropoff_datetime": "2001-01-05 11:52:05", "passenger_count": 5, "trip_distance": 1.53, "RatecodeID": 1, "store_and_fwd_flag": "N", "PULocationID": 71, "DOLocationID": 89, "payment_type": 2, "fare_amount": 7.5, "extra": 0.5, "mta_tax": 0.5, "tip_amount": 0, "tolls_amount": 0, "improvement_surcharge": 0.3, "total_amount": 8.8 }, { "VendorID": 2, "tpep_pickup_datetime": "2002-12-31 23:01:55", "tpep_dropoff_datetime": "2003-01-01 14:59:11" } ], "rows": 3, "rows_before_limit_at_least": 4, "statistics": { "elapsed": 0.00091042, "rows_read": 4, "bytes_read": 296 } } Data can be fetched in different formats. Just append `FORMAT ` to your SQL query: Requesting different formats with SQL [¶](https://www.tinybird.co/docs/about:blank#id9) SELECT count () from < pipe > FORMAT JSON| format | Description | | --- | --- | | CSV | CSV without header | | CSVWithNames | CSV with header | | JSON | JSON including data, statistics and schema information | | TSV | TSV without header | | TSVWithNames | TSV with header | | PrettyCompact | Formatted table | | JSONEachRow | Newline-delimited JSON values (NDJSON) | | Parquet | Apache Parquet | As you can see in the example above, timestamps do not include a time zone in their serialization. Let’s see how that relates to timestamps ingested from your original data: - If the original timestamp had no time zone associated, you’ll read back the same date and time verbatim. If you ingested the timestamp `2022-11-14 11:08:46` , for example, Tinybird sends `"2022-11-14 11:08:46"` back. This is so regardless of the time zone of the column in ClickHouse. - If the original timestamp had a time zone associated, you’ll read back the corresponding date and time in the time zone of the destination column in ClickHouse, which is UTC by default. If you ingested `2022-11-14 12:08:46.574295 +0100` , for instance, Tinybird sends `"2022-11-14 11:08:46"` back for a `DateTime` , and `"2022-11-14 06:08:46"` for a `DateTime('America/New_York')` . --- URL: https://www.tinybird.co/docs/api-reference/pipe-api Content: --- title: "Pipes API Reference · Tinybird Docs" theme-color: "#171612" description: "The Pipe API enables you to manage your Pipes. With Pipes you can transform data via SQL queries and publish the results of those queries as API Endpoints." --- GET /v0/pipes/? [¶](https://www.tinybird.co/docs/about:blank#get--v0-pipes-?) Get a list of pipes in your account. getting a list of your pipes [¶](https://www.tinybird.co/docs/about:blank#id1) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes" Pipes in the response will be the ones that are accessible using a particular token with read permissions for them. Successful response [¶](https://www.tinybird.co/docs/about:blank#id2) { "pipes": [{ "id": "t_55c39255e6b548dd98cb6da4b7d62c1c", "name": "my_pipe", "description": "This is a description", "endpoint": "t_h65c788b42ce4095a4789c0d6b0156c3", "created_at": "2022-11-10 12:39:38.106380", "updated_at": "2022-11-29 13:33:40.850186", "parent": null, "nodes": [{ "id": "t_h65c788b42ce4095a4789c0d6b0156c3", "name": "my_node", "sql": "SELECT col_a, col_b FROM my_data_source", "description": null, "materialized": null, "cluster": null, "tags": {}, "created_at": "2022-11-10 12:39:47.852303", "updated_at": "2022-11-10 12:46:54.066133", "version": 0, "project": null, "result": null, "ignore_sql_errors": false "node_type": "default" }], "url": "https://api.tinybird.co/v0/pipes/my_pipe.json" }] }| Key | Type | Description | | --- | --- | --- | | dependencies | boolean | The response will include the nodes dependent data sources and pipes, default is `false` | | attrs | String | comma separated list of the pipe attributes to return in the response. Example: `attrs=name,description` | | node_attrs | String | comma separated list of the node attributes to return in the response. Example `node_attrs=id,name` | Pipes id’s are immutable so you can always refer to them in your 3rd party applications to make them compatible with Pipes once they are renamed. For lighter JSON responses consider using the `attrs` and `node_attrs` params to return exactly the attributes you need to consume. POST /v0/pipes/? [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-?) Creates a new Pipe. There are 3 ways to create a Pipe Creating a Pipe providing full JSON [¶](https://www.tinybird.co/docs/about:blank#id4) curl -X POST \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ "https://api.tinybird.co/v0/pipes" \ -d '{ "name":"pipe_name", "description": "my first pipe", "nodes": [ {"sql": "select * from my_datasource limit 10", "name": "node_00", "description": "sampled data" }, {"sql": "select count() from node_00", "name": "node_01" } ] }' If you prefer to create the minimum Pipe, and then append your transformation nodes you can set your name and first transformation node’s SQL in your POST request Creating a pipe with a name and a SQL query [¶](https://www.tinybird.co/docs/about:blank#id5) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes?name=pipename&sql=select%20*%20from%20events" Pipes can be also created as copies of other Pipes. Just use the `from` argument: Creating a pipe from another pipe [¶](https://www.tinybird.co/docs/about:blank#id6) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes?name=pipename&from=src_pipe" Bear in mind, if you use this method to overwrite an existing Pipe, the endpoint will only be maintained if the node name is the same. POST /v0/pipes/(.+)/nodes [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-nodes) Appends a new node to a Pipe. adding a new node to a pipe [¶](https://www.tinybird.co/docs/about:blank#id7) curl \ -H "Authorization: Bearer " \ -d 'select * from node_0' "https://api.tinybird.co/v0/pipes/:name/nodes?name=node_name&description=explanation" Appends a new node that creates a Materialized View adding a Materialized View using a materialized node [¶](https://www.tinybird.co/docs/about:blank#id8) curl \ -H "Authorization: Bearer " \ -d 'select id, sum(amount) as amount, date from my_datasource' "https://api.tinybird.co/v0/pipes/:name/nodes?name=node_name&description=explanation&type=materialized&datasource=new_datasource&engine=AggregatingMergeTree"| Key | Type | Description | | --- | --- | --- | | name | String | The referenceable name for the node. | | description | String | Use it to store a more detailed explanation of the node. | | token | String | Auth token. Ensure it has the `PIPE:CREATE` scope on it | | type | String | Optional. Available options are { `standard` (default), `materialized` , `endpoint` }. Use `materialized` to create a Materialized View from your new node. | | datasource | String | Required with `type=materialized` . Specifies the name of the destination Data Source where the Materialized View schema is defined. | | override_datasource | Boolean | Optional. Default `false` When the target Data Source of the Materialized View exists in the Workspace it’ll be overriden by the `datasource` specified in the request. | | populate | Boolean | Optional. Default `false` . When `true` , a job is triggered to populate the destination Data Source. | | populate_subset | Float | Optional. Populate with a subset percent of the data (limited to a maximum of 2M rows), this is useful to quickly test a materialized node with some data. The subset must be greater than 0 and lower than 0.1. A subset of 0.1 means a 10 percent of the data in the source Data Source will be used to populate the Materialized View. Use it together with `populate=true` , it has precedence over `populate_condition` | | populate_condition | String | Optional. Populate with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `populate_condition='date == toYYYYMM(now())'` it’ll populate taking all the rows from the trigger Data Source which `date` is the current month. Use it together with `populate=true` . `populate_condition` is not taken into account if the `populate_subset` param is present. Including in the `populate_condition` any column present in the Data Source `engine_sorting_key` will make the populate job process less data. | | unlink_on_populate_error | String | Optional. Default is `false` . If the populate job fails the Materialized View is unlinked and new data won’t be ingested in the Materialized View. | | engine | String | Optional. Engine for destination Materialized View. Requires the `type` parameter as `materialized` . | | engine_* | String | Optional. Engine parameters and options. Requires the `type` parameter as `materialized` and the `engine` parameter.[ Check Engine Parameters and Options for more details](https://www.tinybird.co/docs/docs/api-reference/datasource-api) | SQL query for the transformation node must be sent in the body encoded in utf-8 | Code | Description | | --- | --- | | 200 | No error | | 400 | empty or wrong SQL or API param value | | 403 | Forbidden. Provided token doesn’t have permissions to append a node to the pipe, it needs `ADMIN` or `PIPE:CREATE` | | 404 | Pipe not found | | 409 | There’s another resource with the same name, names must be unique | The Materialized View already exists | `override_datasource` cannot be performed | DELETE /v0/pipes/(.+)/nodes/(.+) [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+)-nodes-(.+)) Drops a particular transformation node in the Pipe. It does not remove related nodes so this could leave the Pipe in an unconsistent state. For security reasons, enabled nodes can’t be removed. removing a node from a pipe [¶](https://www.tinybird.co/docs/about:blank#id11) curl -X DELETE "https://api.tinybird.co/v0/pipes/:name/nodes/:node_id"| Code | Description | | --- | --- | | 204 | No error, removed node | | 400 | The node is published. Published nodes can’t be removed | | 403 | Forbidden. Provided token doesn’t have permissions to change the last node of the pipe, it needs ADMIN or IMPORT | | 404 | Pipe not found | PUT /v0/pipes/(.+)/nodes/(.+) [¶](https://www.tinybird.co/docs/about:blank#put--v0-pipes-(.+)-nodes-(.+)) Changes a particular transformation node in the Pipe Editing a Pipe’s transformation node [¶](https://www.tinybird.co/docs/about:blank#id13) curl -X PUT \ -H "Authorization: Bearer " \ -d 'select * from node_0' "https://api.tinybird.co/v0/pipes/:name/nodes/:node_id?name=new_name&description=updated_explanation"| Key | Type | Description | | --- | --- | --- | | name | String | new name for the node | | description | String | new description for the node | | token | String | Auth token. Ensure it has the `PIPE:CREATE` scope on it | Please, note that the desired SQL query should be sent in the body encoded in utf-8. | Code | Description | | --- | --- | | 200 | No error | | 400 | Empty or wrong SQL | | 403 | Forbidden. Provided token doesn’t have permissions to change the last node to the pipe, it needs `ADMIN` or `PIPE:CREATE` | | 404 | Pipe not found | | 409 | There’s another resource with the same name, names must be unique | GET /v0/pipes/(.+)\.(json|csv|ndjson|parquet|prometheus) [¶](https://www.tinybird.co/docs/about:blank#get--v0-pipes-(.+)%5C.(json%7Ccsv%7Cndjson%7Cparquet%7Cprometheus)) Returns the published node data in a particular format. Getting data for a pipe [¶](https://www.tinybird.co/docs/about:blank#pipe-get-data) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name.format"| Key | Type | Description | | --- | --- | --- | | q | String | Optional, query to execute, see API Query endpoint | | output_format_json_quote_64bit_integers | int | (Optional) Controls quoting of 64-bit or bigger integers (like UInt64 or Int128) when they are output in a JSON format. Such integers are enclosed in quotes by default. This behavior is compatible with most JavaScript implementations. Possible values: 0 — Integers are output without quotes. 1 — Integers are enclosed in quotes. Default value is 0 | | output_format_json_quote_denormals | int | (Optional) Controls representation of inf and nan on the UI instead of null e.g when dividing by 0 - inf and when there is no representation of a number in Javascript - nan. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | | output_format_parquet_string_as_string | int | (Optional) Use Parquet String type instead of Binary for String columns. Possible values: 0 - disabled, 1 - enabled. Default value is 0 | The `q` parameter is a SQL query (see [Query API](https://www.tinybird.co/docs/docs/api-reference/query-api) ). When using this endpoint to query your Pipes, you can use the `_` shortcut, which refers to your Pipe name | format | Description | | --- | --- | | csv | CSV with header | | json | JSON including data, statistics and schema information | | ndjson | One JSON object per each row | | parquet | A Parquet file. Some libraries might not properly process `UInt*` data types, if that’s your case cast those columns to signed integers with `toInt*` functions. `String` columns are exported as `Binary` , take that into account when reading the resulting Parquet file, most libraries convert from Binary to String (e.g. Spark has this configuration param: `spark.sql.parquet.binaryAsString` ) | | prometheus | Prometheus text-based format. The output table must include name (String) and value (number) as required columns, with optional help (String), timestamp (number), and type (String) (valid values: counter, gauge, histogram, summary, untyped, or empty). Labels should be a Map(String, String), and rows for the same metric with different labels must appear consecutively. The table must be sorted by the name column. | POST /v0/pipes/(.+)\.(json|csv|ndjson|parquet|prometheus) [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)%5C.(json%7Ccsv%7Cndjson%7Cparquet%7Cprometheus)) Returns the published node data in a particular format, passing the parameters in the request body. Use this endpoint when the query is too long to be passed as a query string parameter. When using the post endpoint, there are no traces of the parameters in the pipe_stats_rt Data Source. See the get endpoint for more information. GET /v0/pipes/(.+\.pipe) [¶](https://www.tinybird.co/docs/about:blank#get--v0-pipes-(.+%5C.pipe)) Get pipe information. Provided Auth Token must have read access to the Pipe. Getting information about a particular pipe [¶](https://www.tinybird.co/docs/about:blank#id16) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name" `pipe_id` and `pipe_name` are two ways to refer to the pipe in SQL queries and API endpoints the only difference is `pipe_id` never changes so it’ll work even if you change the `pipe_name` (which is the name used to display the pipe). In general you can use `pipe_id` or `pipe_name` indistinctly: Successful response [¶](https://www.tinybird.co/docs/about:blank#id17) { "id": "t_bd1c62b5e67142bd9bf9a7f113a2b6ea", "name": "events_pipe", "pipeline": { "nodes": [{ "name": "events_ds_0" "sql": "select * from events_ds_log__raw", "materialized": false }, { "name": "events_ds", "sql": "select * from events_ds_0 where valid = 1", "materialized": false }] } } You can make your Pipe’s id more descriptive by prepending information such as `t_my_events_table.bd1c62b5e67142bd9bf9a7f113a2b6ea` DELETE /v0/pipes/(.+\.pipe) [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+%5C.pipe)) Drops a Pipe from your account. Auth token in use must have the `DROP:NAME` scope. Dropping a pipe [¶](https://www.tinybird.co/docs/about:blank#id18) curl -X DELETE \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name" PUT /v0/pipes/(.+\.pipe) [¶](https://www.tinybird.co/docs/about:blank#put--v0-pipes-(.+%5C.pipe)) Changes Pipe’s metadata. When there is another Pipe with the same name an error is raised. editing a pipe [¶](https://www.tinybird.co/docs/about:blank#id19) curl -X PUT \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name?name=new_name"| Key | Type | Description | | --- | --- | --- | | name | String | new name for the pipe | | description | String | new Markdown description for the pipe | | token | String | Auth token. Ensure it has the `PIPE:CREATE` scope on it | GET /v0/pipes/(.+) [¶](https://www.tinybird.co/docs/about:blank#get--v0-pipes-(.+)) Get pipe information. Provided Auth Token must have read access to the Pipe. Getting information about a particular pipe [¶](https://www.tinybird.co/docs/about:blank#id21) curl -X GET \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name" `pipe_id` and `pipe_name` are two ways to refer to the pipe in SQL queries and API endpoints the only difference is `pipe_id` never changes so it’ll work even if you change the `pipe_name` (which is the name used to display the pipe). In general you can use `pipe_id` or `pipe_name` indistinctly: Successful response [¶](https://www.tinybird.co/docs/about:blank#id22) { "id": "t_bd1c62b5e67142bd9bf9a7f113a2b6ea", "name": "events_pipe", "pipeline": { "nodes": [{ "name": "events_ds_0" "sql": "select * from events_ds_log__raw", "materialized": false }, { "name": "events_ds", "sql": "select * from events_ds_0 where valid = 1", "materialized": false }] } } You can make your Pipe’s id more descriptive by prepending information such as `t_my_events_table.bd1c62b5e67142bd9bf9a7f113a2b6ea` DELETE /v0/pipes/(.+) [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+)) Drops a Pipe from your account. Auth token in use must have the `DROP:NAME` scope. Dropping a pipe [¶](https://www.tinybird.co/docs/about:blank#id23) curl -X DELETE \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name" PUT /v0/pipes/(.+) [¶](https://www.tinybird.co/docs/about:blank#put--v0-pipes-(.+)) Changes Pipe’s metadata. When there is another Pipe with the same name an error is raised. editing a pipe [¶](https://www.tinybird.co/docs/about:blank#id24) curl -X PUT \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:name?name=new_name"| Key | Type | Description | | --- | --- | --- | | name | String | new name for the pipe | | description | String | new Markdown description for the pipe | | token | String | Auth token. Ensure it has the `PIPE:CREATE` scope on it | --- URL: https://www.tinybird.co/docs/api-reference/jobs-api Last update: 2024-12-30T16:50:50.000Z Content: --- title: "Jobs API Reference · Tinybird Docs" theme-color: "#171612" description: "With the Jobs API, you can list the jobs for the last 48 hours or the last 100 jobs and also get the details for a specific job." --- GET /v0/jobs/? [¶](https://www.tinybird.co/docs/about:blank#get--v0-jobs-?) We can get a list of the last 100 jobs in the last 48 hours, with the possibility of filtering them by kind, status, pipe_id, pipe_name, created_after, and created_before. | Key | Type | Description | | --- | --- | --- | | kind | String | This will return only the jobs with that particular kind. Example: `kind=populateview` or `kind=copy` or `kind=import` | | status | String | This will return only the jobs with the status provided. Example: `status=done` or `status=waiting` or `status=working` or `status=error` | | pipe_id | String | This will return only the jobs associated with the provided pipe id. Example: `pipe_id=t_31a0ff508c9843b59c32f7f81a156968` | | pipe_name | String | This will return only the jobs associated with the provided pipe name. Example: `pipe_name=test_pipe` | | created_after | String | This will return jobs that were created after the provided date in the ISO 8601 standard date format. Example: `created_after=2023-06-15T18:13:25.855Z` | | created_before | String | This will return jobs that were created before the provided date in the ISO 8601 standard date format. Example: `created_before=2023-06-19T18:13:25.855Z` | Getting the latest jobs [¶](https://www.tinybird.co/docs/about:blank#id2) curl \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/jobs" \ You will get a list of jobs with the `kind`, `status`, `id` , and the `url` to access the specific information about that job. Jobs list [¶](https://www.tinybird.co/docs/about:blank#id3) { "jobs": [ { "id": "c8ae13ef-e739-40b6-8bd5-b1e07c8671c2", "kind": "import", "status": "done", "created_at": "2020-12-04 15:08:33.214377", "updated_at": "2020-12-04 15:08:33.396286", "job_url": "https://api.tinybird.co/v0/jobs/c8ae13ef-e739-40b6-8bd5-b1e07c8671c2", "datasource": { "id": "t_31a0ff508c9843b59c32f7f81a156968", "name": "my_datasource_1" } }, { "id": "1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb", "kind": "import", "status": "error", "created_at": "2020-12-04 15:08:09.051310", "updated_at": "2020-12-04 15:08:09.263055", "job_url": "https://api.tinybird.co/v0/jobs/1f6a5a3d-cfcb-4244-ba0b-0bfa1d1752fb", "datasource": { "id": "t_49806938714f4b72a225599cdee6d3ab", "name": "my_datasource_2" } } ] } Job details in `job_url` will be available for 48h after its creation. POST /v0/jobs/(.+)/cancel [¶](https://www.tinybird.co/docs/about:blank#post--v0-jobs-(.+)-cancel) With this endpoint you can try to cancel an existing Job. All jobs can be cancelled if they are in the “waiting” status, but you can’t cancel a Job in “done” or “error” status. In the case of the job of type “populate”, you can cancel it in the “working” state. After successfully starting the cancellation process, you will see two different status in the job: - “cancelling”: The Job can’t be immediately cancelled as it’s doing some work, but the cancellation will eventually happen. - “cancelled”: The Job has been completely cancelled. A Job cancellation doesn’t guarantee a complete rollback of the changes being made by it, sometimes you will need to delete new inserted rows or datasources created. The fastest way to know if a job is cancellable, is just reading the “is_cancellable” key inside the job JSON description. Depending on the Job and its status, when you try to cancel it you may get different responses: - HTTP Code 200: The Job has successfully started the cancellation process. Remember that if the Job has now a “cancelling” status, it may need some time to completely cancel itself. This request will return the status of the job. - HTTP Code 404: Job not found. - HTTP Code 403: The token provided doesn’t have access to this Job. - HTTP Code 400: Job is not in a cancellable status or you are trying to cancel a job which is already in the “cancelling” state. Try to cancel a Job [¶](https://www.tinybird.co/docs/about:blank#id4) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/jobs/:job_id/cancel" Populate Job in cancelling state right after the cancellation request. [¶](https://www.tinybird.co/docs/about:blank#id5) { "kind": "populateview", "id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c", "job_id": "32c3438d-582e-4a6f-9b57-7d7a3bfbeb8c", "status": "cancelling", "created_at": "2021-03-17 18:56:23.939380", "updated_at": "2021-03-17 18:56:44.343245", "is_cancellable": false, "datasource": { "id": "t_02043945875b4070ae975f3812444b76", "name": "your_datasource_name", "cluster": null, "tags": {}, "created_at": "2020-07-15 10:55:12.427269", "updated_at": "2020-07-15 10:55:12.427270", "statistics": null, "replicated": false, "version": 0, "project": null, "used_by": [] }, "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2", "pipe_id": "t_7fa8009023a245b696b4f2f7195b23c3", "pipe_name": "top_product_per_day", "queries": [ { "query_id": "01HSZ9WJES5QEZZM4TGDD3YFZ2", "status": "done" }, { "query_id": "01HSZ9WY6QS6XAMBHZMSNB1G75", "status": "done" }, { "query_id": "01HSZ9X8YVEQ0PXA6T2HZQFFPX", "status": "working" }, { "query_id": "01HSZQ5YX0517X81JBF9G9HB2P", "status": "waiting" }, { "query_id": "01HSZQ6PZJA3P81RC6Q6EF6HMK", "status": "waiting" }, { "query_id": "01HSZQ76D7YYFB16TFT32KXMCY", "status": "waiting" } ], "progress_percentage": 50.0 } GET /v0/jobs/(.+) [¶](https://www.tinybird.co/docs/about:blank#get--v0-jobs-(.+)) Get the details of a specific Job. You can get the details of a Job by using its ID. Get the details of a Job [¶](https://www.tinybird.co/docs/about:blank#id6) curl \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/jobs/:job_id" You will get a JSON response with the details of the Job, including the `kind`, `status`, `id`, `created_at`, `updated_at` , and the `datasource` associated with the Job. This is available for 48h after the Job creation. After that, you can consult the Job details in the Service Data Source jobs_log. Job details [¶](https://www.tinybird.co/docs/about:blank#id7) { "kind": "import", "id": "d5b869ed-3a74-45f9-af54-57350aae4cef", "job_id": "d5b869ed-3a74-45f9-af54-57350aae4cef", "status": "done", "created_at": "2024-07-22 11:47:58.207606", "updated_at": "2024-07-22 11:48:52.971327", "started_at": "2024-07-22 11:47:58.351734", "is_cancellable": false, "mode": "append", "datasource": { "id": "t_caf95c54174e48f488ea65d181eb5b75", "name": "events", "cluster": "default", "tags": { }, "created_at": "2024-07-22 11:47:51.807384", "updated_at": "2024-07-22 11:48:52.726243", "replicated": true, "version": 0, "project": null, "headers": { "cached_delimiter": "," }, "shared_with": [ ], "engine": { "engine": "MergeTree", "partition_key": "toYear(date)", "sorting_key": "date, user_id, event, extra_data" }, "description": "", "used_by": [ ], "last_commit": { "content_sha": "", "status": "changed", "path": "" }, "errors_discarded_at": null, "type": "csv" }, "import_id": "d5b869ed-3a74-45f9-af54-57350aae4cef", "url": "https://storage.googleapis.com/tinybird-assets/datasets/guides/events_50M_1.csv", "statistics": { "bytes": 1592720209, "row_count": 50000000 }, "quarantine_rows": 0, "invalid_lines": 0 } --- URL: https://www.tinybird.co/docs/api-reference/events-api Last update: 2025-02-07T08:04:20.000Z Content: --- title: "Events API Reference · Tinybird Docs" theme-color: "#171612" description: "With the Tinybird Events API you can ingest thousands of JSON events per second." --- # Events API [¶](https://www.tinybird.co/docs/about:blank#events-api) The Events API allows you to ingest JSON events with a simple HTTP POST request. See [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api). All endpoints require authentication using a Token with `DATASOURCE:APPEND` or `DATASOURCE:CREATE` scope. ## POST /v0/events [¶](https://www.tinybird.co/docs/about:blank#post-v0events) Use this endpoint to send NDJSON (new-line delimited JSON) events to a [Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources). ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) | Key | Type | Description | | --- | --- | --- | | name | String | name or ID of the target Data Source to append data to it | | wait | Boolean | 'false' by default. Set to 'true' to wait until the write is acknowledged by the database. Enabling this flag makes possible to retry on database errors, but it introduces additional latency. It's recommended to enable it in use cases in which data loss avoidance is critical. Disable it otherwise. | ### Return HTTP status codes [¶](https://www.tinybird.co/docs/about:blank#return-http-status-codes) | Status Code | Description | | --- | --- | | 200 | The data has been inserted into the database. The write has been acknowledged. The request 'wait' parameter was enabled. | | 202 | The data has been processed, and it will be send to the database eventually. The write hasn't been acknowledged yet. The request 'wait' parameter was disabled. | | 400 | The request is invalid. The body will contain more information. A common cause is missing the 'name' parameter. No data has been inserted, but the request shouldn't be retried. | | 403 | The token isn't valid. The request shouldn't be retried. | | 404 | The token's Workspace doesn't belong to this cluster. The Workspace is probably removed or in another cluster. The request shouldn't be retried, ensure the token's region and the Tinybird domain matches. | | 422 | The ingestion has been partially completed due to an error in a Materialized View. Retrying may result in data duplication, but not retrying may result in data loss. The general advice is to not retry, review attached Materialized Views, and contact us if the issue persists. | | 429 | The request/second limit has been reached. Default limit is 1000 requests/second, contact us for increased capacity. The request may be retried after a while. Use exponential backoff with a limited amount of retries. | | 500 | An unexpected error has occurred. The body will contain more information. Retrying is the general advice, contact with us if the issue persists. | | 503 | The service is temporarily unavailable. The body may contain more information. A common cause is to have reached a throughput limit, or to have attached a Materialized View with an issue. No data has been inserted, and it's safe to retry. Contact with us if the issue persists. | | 0x07 GOAWAY | HTTP2 only. Too many alive connections. Recreate the connection and retry. | ### Compression [¶](https://www.tinybird.co/docs/about:blank#compression) You can compress JSON events with Gzip and send the compressed payload to the Events API. You must include the header `Content-Encoding: gzip` with the request. ## Examples [¶](https://www.tinybird.co/docs/about:blank#examples) ### NDJSON messages [¶](https://www.tinybird.co/docs/about:blank#ndjson-messages) The following example shows how to push single NDJSON messages using the Events API: ##### Send individual NDJSON messages curl \ -H "Authorization: Bearer " \ -d '{"date": "2020-04-05 00:05:38", "city": "Chicago"}' \ 'https://api.tinybird.co/v0/events?name=events_test' ### JSON messages [¶](https://www.tinybird.co/docs/about:blank#json-messages) The following example shows how to push single JSON messages using the Events API: ##### Send individual JSON messages curl \ -H "Authorization: Bearer " \ -d $'{ \ "date": "2020-04-05 00:05:38", \ "city": "Chicago" \ }' \ 'https://api.tinybird.co/v0/events?name=events_test&format=json' ### Multiple NDJSON messages [¶](https://www.tinybird.co/docs/about:blank#multiple-ndjson-messages) The following example shows how to push multiple NDJSON messages using the Events API. Notice the '$' before the JSON events. It's needed in order for Bash to replace the '\n'. curl doesn't do it automatically. ##### Send many NDJSON events. curl \ -H "Authorization: Bearer " \ -d $'{"date": "2020-04-05 00:05:38", "city": "Chicago"}\n{"date": "2020-04-05 00:07:22", "city": "Madrid"}\n' \ 'https://api.tinybird.co/v0/events?name=events_test' ### Gzip compressed payload [¶](https://www.tinybird.co/docs/about:blank#gzip-compressed-payload) The following example shows how to push a Gzip compressed payload using the Events API, where 'body.gz' is a batch of NDJSON events. ##### Send a Gzip compressed payload. curl \ -H "Authorization: Bearer " \ -H "Content-Encoding: gzip" \ --data-binary @body.gz \ 'https://api.tinybird.co/v0/events?name=events_example' --- URL: https://www.tinybird.co/docs/api-reference/environment-variables-api Last update: 2025-01-20T11:46:12.000Z Content: --- title: "Environment Variables API · Tinybird Docs" theme-color: "#171612" description: "The Environment Variables API allows you to create, update, delete and list environment variables that can be used in Pipes in a Workspace." --- # Environment Variables API [¶](https://www.tinybird.co/docs/about:blank#environment-variables-api) Use the Environment Variables API to create, update, delete, and list environment variables that you can use in Pipes in a Workspace. Environment variables allow you to store sensitive information, such as access secrets and host names, in your Workspace. Environment variables are encrypted at rest. Using the Environment Variables API requires a Workspace admin token. ## Environment variables types [¶](https://www.tinybird.co/docs/about:blank#environment-variables-types) The Environment Variables API supports different types of environment variables: | Environment variable type | Comments | | --- | --- | | `secret` | Used to store passwords and other secrets, automatically prevents Endpoint from exposing its value. It's the default type. | ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. The Environment Variables API has the following limits: - 5 requests per second. - 100 environment variables per Workspace. - 8 KB max size of the `value` attribute. ## Templating [¶](https://www.tinybird.co/docs/about:blank#templating) After creating environment variables in a Workspace, you can use the `tb_secret` template function to replace the original value: % SELECT * FROM postgresql('host:post', 'database', 'table', 'user', {{tb_secret('pg_password')}}) Environment variables values are rendered as `String` data type. If you need to use a different type, use any of the [functions to cast a String value to a given type](https://www.tinybird.co/docs/docs/sql-reference/functions) . For example: % SELECT * FROM table WHERE int_value = toUInt8({{tb_secret('int_secret')}}) ## Staging and production use [¶](https://www.tinybird.co/docs/about:blank#staging-and-production-use) If you have staging and production Workspaces, create the same environment variables with the same name in both Workspaces, changing only their corresponding value. Tinybird doesn't allow you to create an API Endpoint when exposing environment variables with `type=secret` in a SELECT clause. So, while it's possible to have a node that uses the logic `SELECT {{tb_secret('username')}}` , you can't publish that node as a Copy Pipe or API Endpoint. ## Branch use [¶](https://www.tinybird.co/docs/about:blank#branch-use) You can use environment variables in Branches, but you must create them in the main Workspace initially. Environment variables have the same value in the main Workspace as in the Branches. You can't create a environment variable in a Branch to be deployed in the main Workspace. ## POST /v0/variables/? [¶](https://www.tinybird.co/docs/about:blank#post-v0variables) Creates a new environment variable. ### Restrictions [¶](https://www.tinybird.co/docs/about:blank#restrictions) Environment variables names are unique for a Workspace. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X POST "https://$TB_HOST/v0/variables" \ -H "Authorization: Bearer " \ -d "type=secret" \ -d "name=test_password" \ -d "value=test" ### Request parameters [¶](https://www.tinybird.co/docs/about:blank#request-parameters) | Key | Type | Description | | --- | --- | --- | | type | String (optional) | The type of the variable. Defaults to `secret` | | name | String | The name of the variable | | value | String | The variable value | ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "name": "test_token", "created_at": "2024-06-21T10:27:57", "updated_at": "2024-06-21T10:27:57", "edited_by": "token: 'admin token'" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 400 | Invalid or missing parameters | | 403 | Limit reached or invalid token | | 404 | Workspace not found | ## DELETE /v0/variables/(.+) [¶](https://www.tinybird.co/docs/about:blank#delete-v0variables) Deletes a environment variable. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X DELETE "https://$TB_HOST/v0/variables/test_password" \ -H "Authorization: Bearer " ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "ok": true } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 400 | Invalid or missing parameters | | 403 | Limit reached or token invalid | | 404 | Workspace or variable not found | ## PUT /v0/variables/(.+) [¶](https://www.tinybird.co/docs/about:blank#put-v0variables) Updates a environment variable. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X PUT "https://$TB_HOST/v0/variables/test_password" \ -H "Authorization: Bearer " \ -d "value=new_value" ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "name": "test_password", "type": "secret", "created_at": "2024-06-21T10:27:57", "updated_at": "2024-06-21T10:29:57", "edited_by": "token: 'admin token'" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 400 | Invalid or missing parameters | | 403 | Limit reached or token invalid | | 404 | Workspace or variable not found | ## GET /v0/variables/? [¶](https://www.tinybird.co/docs/about:blank#get-v0variables) Retrieves all Workspace environment variables. The value isn't returned. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X GET "https://$TB_HOST/v0/variables" \ -H "Authorization: Bearer " ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "variables": [ { "name": "test_token", "type": "secret", "created_at": "2024-06-21T10:27:57", "updated_at": "2024-06-21T10:27:57", "edited_by": "token: 'admin token'" }, { "name": "test_token2", "type": "secret", "created_at": "2024-06-21T10:27:57", "updated_at": "2024-06-21T10:29:57", "edited_by": "token: 'admin token'" } ] } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 400 | Invalid or missing parameters | | 403 | Limit reached or token invalid | | 404 | Workspace not found | ## GET /v0/variables/(.+) [¶](https://www.tinybird.co/docs/about:blank#get-v0variables) Fetches information about a particular environment variable. The value isn't returned. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) curl \ -X GET "https://$TB_HOST/v0/variables/test_password" \ -H "Authorization: Bearer " ### Successful response example [¶](https://www.tinybird.co/docs/about:blank#successful-response-example) { "name": "test_password", "type": "secret", "created_at": "2024-06-21T10:27:57", "updated_at": "2024-06-21T10:27:57", "edited_by": "token: 'admin token'" } ### Response codes [¶](https://www.tinybird.co/docs/about:blank#response-codes) | Code | Description | | --- | --- | | 200 | OK | | 400 | Invalid or missing parameters | | 403 | Limit reached or token invalid | | 404 | Workspace or variable not found | --- URL: https://www.tinybird.co/docs/api-reference/datasource-api Last update: 2024-12-30T16:50:50.000Z Content: --- title: "Data Sources API Reference · Tinybird Docs" theme-color: "#171612" description: "The Data Source API enables you to create, manage and import data into your Data Sources." --- POST /v0/datasources/? [¶](https://www.tinybird.co/docs/about:blank#post--v0-datasources-?) This endpoint supports 3 modes to enable 3 distinct operations, depending on the parameters provided: > - Create a new Data Source with a schema - Append data to an existing Data Source - Replace data in an existing Data Source The mode is controlled by setting the `mode` parameter, for example, `-d "mode=create"` . Each mode has different [rate limits](https://www.tinybird.co/docs/docs/api-reference/overview#limits). When importing remote files by URL, if the server hosting the remote file supports HTTP Range headers, the import process will be parallelized. | KEY | TYPE | DESCRIPTION | | --- | --- | --- | | mode | String | Default: `create` . Other modes: `append` and `replace` . The `create` mode creates a new Data Source and attempts to import the data of the CSV if a URL is provided or the body contains any data. The `append` mode inserts the new rows provided into an existing Data Source (it will also create it if it does not exist yet). The `replace` mode will remove the previous Data Source and its data and replace it with the new one; Pipes or queries pointing to this Data Source will immediately start returning data from the new one and without disruption once the replace operation is complete. The `create` mode will automatically name the Data Source if no `name` parameter is provided; for the `append` and `replace` modes to work, the `name` parameter must be provided and the schema must be compatible. | | name | String | Optional. Name of the Data Source to create, append or replace data. This parameter is mandatory when using the `append` or `replace` modes. | | url | String | Optional. The URL of the CSV with the data to be imported | | dialect_delimiter | String | Optional. The one-character string separating the fields. We try to guess the delimiter based on the CSV contents using some statistics, but sometimes we fail to identify the correct one. If you know your CSV’s field delimiter, you can use this parameter to explicitly define it. | | dialect_new_line | String | Optional. The one- or two-character string separating the records. We try to guess the delimiter based on the CSV contents using some statistics, but sometimes we fail to identify the correct one. If you know your CSV’s record delimiter, you can use this parameter to explicitly define it. | | dialect_escapechar | String | Optional. The escapechar removes any special meaning from the following character. This is useful if the CSV does not use double quotes to encapsulate a column but uses double quotes in the content of a column and it is escaped with, e.g. a backslash. | | schema | String | Optional. Data Source schema in the format ‘column_name Type, column_name_2 Type2…’. When creating a Data Source with format `ndjson` the `schema` must include the `jsonpath` for each column, see the `JSONPaths` section for more details. | | engine | String | Optional. Engine for the underlying data. Requires the `schema` parameter. | | engine_* | String | Optional. Engine parameters and options, check the[ Engines](https://www.tinybird.co/docs/concepts/data-sources.html#supported-engines) section for more details | | progress | String | Default: `false` . When using `true` and sending the data in the request body, Tinybird will return block status while loading using Line-delimited JSON. | | token | String | Auth token with create or append permissions. Required only if no Bearer Authorization header is found | | type_guessing | String | Default: `true` The `type_guessing` parameter is not taken into account when replacing or appending data to an existing Data Source. When using `false` all columns are created as `String` otherwise it tries to guess the column types based on the CSV contents. Sometimes you are not familiar with the data and the first step is to get familiar with it: by disabling the type guessing, we enable you to quickly import everything as strings that you can explore with SQL and cast to the right type or shape in whatever way you see fit via a Pipe. | | debug | String | Optional. Enables returning debug information from logs. It can include `blocks` , `block_log` and/or `hook_log` | | replace_condition | String | Optional. When used in combination with the `replace` mode it allows you to replace a portion of your Data Source that matches the `replace_condition` SQL statement with the contents of the `url` or query passed as a parameter. See this[ guide](https://www.tinybird.co/guide/replacing-and-deleting-data#replace-data-selectively) to learn more. | | replace_truncate_when_empty | Boolean | Optional. When used in combination with the `replace` mode it allows truncating the Data Source when empty data is provided. Not supported when `replace_condition` is specified | | format | String | Default: `csv` . Indicates the format of the data to be ingested in the Data Source. By default is `csv` and you should specify `format=ndjson` for NDJSON format, and `format=parquet` for Parquet files. | **Examples** Creating a CSV Data Source from a schema [¶](https://www.tinybird.co/docs/about:blank#id2) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String, date Date, close Float32" Creating a CSV Data Source from a local CSV file with schema inference [¶](https://www.tinybird.co/docs/about:blank#id3) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources?name=stocks" \ -F csv = @local_file.csv Creating a CSV Data Source from a remote CSV file with schema inference [¶](https://www.tinybird.co/docs/about:blank#id4) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d url = 'https://.../data.csv' Creating an empty Data Source with a ReplacingMergeTree engine and custom engine settings [¶](https://www.tinybird.co/docs/about:blank#id5) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "schema=pk UInt64, insert_date Date, close Float32" \ -d "engine=ReplacingMergeTree" \ -d "engine_sorting_key=pk" \ -d "engine_ver=insert_date" \ -d "name=test123" \ -d "engine_settings=index_granularity=2048, ttl_only_drop_parts=false" Appending data to a Data Source from a local CSV file [¶](https://www.tinybird.co/docs/about:blank#id6) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources?name=data_source_name&mode=append" \ -F csv = @local_file.csv Appending data to a Data Source from a remote CSV file [¶](https://www.tinybird.co/docs/about:blank#id7) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d mode = 'append' \ -d name = 'data_source_name' \ -d url = 'https://.../data.csv' Replacing data with a local file [¶](https://www.tinybird.co/docs/about:blank#id8) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources?name=data_source_name&mode=replace" \ -F csv = @local_file.csv Replacing data with a remote file from a URL [¶](https://www.tinybird.co/docs/about:blank#id9) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d mode = 'replace' \ -d name = 'data_source_name' \ --data-urlencode "url=http://example.com/file.csv" GET /v0/datasources/? [¶](https://www.tinybird.co/docs/about:blank#get--v0-datasources-?) getting a list of your Data Sources [¶](https://www.tinybird.co/docs/about:blank#id10) curl \ -H "Authorization: Bearer " \ -X GET "https://api.tinybird.co/v0/datasources" Get a list of the Data Sources in your account. The token you use to query the available Data Sources will determine what Data Sources get returned: only those accessible with the token you are using will be returned in the response. Successful response [¶](https://www.tinybird.co/docs/about:blank#id11) { "datasources": [{ "id": "t_a049eb516ef743d5ba3bbe5e5749433a", "name": "your_datasource_name", "cluster": "tinybird", "tags": {}, "created_at": "2019-11-13 13:53:05.340975", "updated_at": "2022-02-11 13:11:19.464343", "replicated": true, "version": 0, "project": null, "headers": {}, "shared_with": [ "89496c21-2bfe-4775-a6e8-97f1909c8fff" ], "engine": { "engine": "MergeTree", "engine_sorting_key": "example_column_1", "engine_partition_key": "", "engine_primary_key": "example_column_1" }, "description": "", "used_by": [], "type": "csv", "columns": [{ "name": "example_column_1", "type": "Date", "codec": null, "default_value": null, "jsonpath": null, "nullable": false, "normalized_name": "example_column_1" }, { "name": "example_column_2", "type": "String", "codec": null, "default_value": null, "jsonpath": null, "nullable": false, "normalized_name": "example_column_2" } ], "statistics": { "bytes": 77822, "row_count": 226188 }, "new_columns_detected": {}, "quarantine_rows": 0 }] }| Key | Type | Description | | --- | --- | --- | | attrs | String | comma separated list of the Data Source attributes to return in the response. Example: `attrs=name,id,engine` . Leave empty to return a full response | Note that the `statistics` ’s `bytes` and `row_count` attributes might be `null` depending on how the Data Source was created. POST /v0/datasources/(.+)/alter [¶](https://www.tinybird.co/docs/about:blank#post--v0-datasources-(.+)-alter) Modify the Data Source schema. This endpoint supports the operation to alter the following fields of a Data Source: | Key | Type | Description | | --- | --- | --- | | schema | String | Optional. Set the whole schema that adds new columns to the existing ones of a Data Source. | | description | String | Optional. Sets the description of the Data Source. | | kafka_store_raw_value | Boolean | Optional. Default: false. When set to true, the ‘value’ column of a Kafka Data Source will save the JSON as a raw string. | | kafka_store_headers | Boolean | Optional. Default: false. When set to true, the ‘headers’ of a Kafka Data Source will be saved as a binary map. | | ttl | String | Optional. Set to any value accepted in ClickHouse for a TTL or to ‘false’ to remove the TTL. | | dry | Boolean | Optional. Default: false. Set to true to show what would be modified in the Data Source, without running any modification at all. | The schema parameter can be used to add new columns at the end of the existing ones in a Data Source. Be aware that currently we don’t validate if the change will affect the existing MVs (Materialized Views) attached to the Data Source to be modified, so this change may break existing MVs. For example, avoid changing a Data Source that has a MV created with something like `SELECT * FROM Data Source ...` . If you want to have forward compatible MVs with column additions, create them especifying the columns instead of using the `*` operator. Also, take in account that, for now, the only engines supporting adding new columns are those inside the MergeTree family. To add a column to a Data Source, call this endpoint with the Data Source name and the new schema definition. For example, having a Data Source created like this: Creating a Data Source from a schema [¶](https://www.tinybird.co/docs/about:blank#id14) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String, date Date, close Float32" if you want to add a new column ‘concept String’, you need to call this endpoint with the new schema: Adding a new column to an existing Data Source [¶](https://www.tinybird.co/docs/about:blank#id15) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/stocks/alter" \ -d "schema=symbol String, date Date, close Float32, concept String" If everything went ok, you will get the operations done in the response: ADD COLUMN operation resulted from the schema change. [¶](https://www.tinybird.co/docs/about:blank#id16) { "operations": [ "ADD COLUMN `concept` String" ] } You can also view the inferred operations without executing them adding `dry=true` in the parameters. - To modify the description of a Data Source: Modifying the description a Data Source [¶](https://www.tinybird.co/docs/about:blank#id17) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/stocks/alter" \ -d "name=stocks" \ -d "description=My new description" - To save in the “value” column of a Kafka Data Source the JSON as a raw string: Saving the raw string in the value column of a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#id18) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/stocks/alter" \ -d "name=stocks" \ -d "kafka_store_raw_value=true" -d "kafka_store_headers=true" - To modify the TTL of a Data Source: Modifying the TTL of a Data Source [¶](https://www.tinybird.co/docs/about:blank#id19) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/stocks/alter" \ -d "name=stocks" \ -d "ttl=12 hours" - To remove the TTL of a Data Source: Modifying the TTL of a Data Source [¶](https://www.tinybird.co/docs/about:blank#id20) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "ttl=false" - To add default values to the columns of a Data Source: Modifying default values [¶](https://www.tinybird.co/docs/about:blank#id21) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String DEFAULT '-', date Date DEFAULT now(), close Float32 DEFAULT 1.1" - To add default values to the columns of a NDJSON Data Source, add the default definition after the jsonpath definition: Modifying default values in a NDJSON Data Source [¶](https://www.tinybird.co/docs/about:blank#id22) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String `json: $ .symbol` DEFAULT '-', date Date `json: $ .date` DEFAULT now(), close `json: $ .close` Float32 DEFAULT 1.1" - To make a column nullable, change the type of the column adding the Nullable type prefix to old one: Converting column “close” to Nullable [¶](https://www.tinybird.co/docs/about:blank#id23) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String `json: $ .symbol, date Date `json: $ .date`, close `json: $ .close` Nullable(Float32)" - To drop a column, just remove the column from the schema definition. It will not be possible removing columns that are part of the primary or partition key: Remove column “close” from the Data Source [¶](https://www.tinybird.co/docs/about:blank#id24) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources" \ -d "name=stocks" \ -d "schema=symbol String `json: $ .symbol, date Date `json: $ .date`" You can also alter the JSONPaths of existing Data Sources. In that case you have to specify the [JSONPath](https://www.tinybird.co/docs/docs/guides/ingesting-data/ingest-ndjson-data) in the schema in the same way as when you created the Data Source. POST /v0/datasources/(.+)/truncate [¶](https://www.tinybird.co/docs/about:blank#post--v0-datasources-(.+)-truncate) Truncates a Data Source in your account. If the Data Source has dependent Materialized Views, those **won’t** be truncated in cascade. In case you want to delete data from other dependent Materialized Views, you’ll have to do a subsequent call to this method. Auth token in use must have the `DATASOURCES:CREATE` scope. Truncating a Data Source [¶](https://www.tinybird.co/docs/about:blank#id25) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/name/truncate" This works as well for the `quarantine` table of a Data Source. Remember that the quarantine table for a Data Source has the same name but with the “_quarantine” suffix. Truncating the quarantine table from a Data Source [¶](https://www.tinybird.co/docs/about:blank#id26) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/datasources/:name_quarantine/truncate" POST /v0/datasources/(.+)/delete [¶](https://www.tinybird.co/docs/about:blank#post--v0-datasources-(.+)-delete) Deletes rows from a Data Source in your account given a SQL condition. Auth token in use must have the `DATASOURCES:CREATE` scope. Deleting rows from a Data Source given a SQL condition [¶](https://www.tinybird.co/docs/about:blank#id27) curl \ -H "Authorization: Bearer " \ --data "delete_condition=(country='ES')" \ "https://api.tinybird.co/v0/datasources/:name/delete" When deleting rows from a Data Source, the response will not be the final result of the deletion but a Job. You can check the job status and progress using the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api) . In the response, `id`, `job_id` , and `delete_id` should have the same value: Delete API Response [¶](https://www.tinybird.co/docs/about:blank#id28) { "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "job_url": "https://api.tinybird.co/v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b", "job": { "kind": "delete_data", "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "status": "waiting", "created_at": "2023-04-11 13:52:32.423207", "updated_at": "2023-04-11 13:52:32.423213", "started_at": null, "is_cancellable": true, "datasource": { "id": "t_c45d5ae6781b41278fcee365f5bxxxxx", "name": "shopping_data" }, "delete_condition": "event = 'search'" }, "status": "waiting", "delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b" } To check on the progress of the delete job, use the `job_id` from the Delete API response to query the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api). For example, to check on the status of the above delete job: checking the status of the delete job [¶](https://www.tinybird.co/docs/about:blank#id29) curl \ -H "Authorization: Bearer " \ https://api.tinybird.co/v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b Would respond with: Job API Response [¶](https://www.tinybird.co/docs/about:blank#id30) { "kind": "delete_data", "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b", "status": "done", "created_at": "2023-04-11 13:52:32.423207", "updated_at": "2023-04-11 13:52:37.330020", "started_at": "2023-04-11 13:52:32.842861", "is_cancellable": false, "datasource": { "id": "t_c45d5ae6781b41278fcee365f5bc2d35", "name": "shopping_data" }, "delete_condition": " event = 'search'", "rows_affected": 100 } ### Data Source engines supported Tinybird uses ClickHouse as the underlying storage technology. ClickHouse features different strategies to store data, these different strategies define not only where and how the data is stored but what kind of data access, queries, and availability your data has. In ClickHouse terms, a Tinybird Data Source uses a [Table Engine](https://clickhouse.tech/docs/en/engines/table_engines/) that determines those factors. Currently, Tinybird supports deleting data for data sources with the following Engines: - MergeTree - ReplacingMergeTree - SummingMergeTree - AggregatingMergeTree - CollapsingMergeTree - VersionedCollapsingMergeTree ### Dependent views deletion If the Data Source has dependent Materialized Views, those won’t be cascade deleted. In case you want to delete data from other dependent Materialized Views, you’ll have to do a subsequent call to this method for the affected view with a proper `delete_condition` . This applies as well to the associated `quarantine` Data Source. | KEY | TYPE | DESCRIPTION | | --- | --- | --- | | delete_condition | String | Mandatory. A string representing the WHERE SQL clause you’d add to a regular DELETE FROM WHERE statement. Most of the times you might want to write a simple `delete_condition` such as `column_name=value` but any valid SQL statement including conditional operators is valid | | dry_run | String | Default: `false` . It allows you to test the deletion. When using `true` it will execute all deletion validations and return number of matched `rows_to_be_deleted` . | GET /v0/datasources/(.+) [¶](https://www.tinybird.co/docs/about:blank#get--v0-datasources-(.+)) Getting information about a particular Data Source [¶](https://www.tinybird.co/docs/about:blank#id32) curl \ -H "Authorization: Bearer " \ -X GET "https://api.tinybird.co/v0/datasources/datasource_name" Get Data Source information and stats. The token provided must have read access to the Data Source. Successful response [¶](https://www.tinybird.co/docs/about:blank#id33) { "id": "t_bd1c62b5e67142bd9bf9a7f113a2b6ea", "name": "datasource_name", "statistics": { "bytes": 430833, "row_count": 3980 }, "used_by": [{ "id": "t_efdc62b5e67142bd9bf9a7f113a34353", "name": "pipe_using_datasource_name" }] "updated_at": "2018-09-07 23:50:32.322461", "created_at": "2018-11-28 23:50:32.322461", "type": "csv" }| Key | Type | Description | | --- | --- | --- | | attrs | String | comma separated list of the Data Source attributes to return in the response. Example: `attrs=name,id,engine` . Leave empty to return a full response | `id` and `name` are two ways to refer to the Data Source in SQL queries and API endpoints. The only difference is that the `id` never changes; it will work even if you change the `name` (which is the name used to display the Data Source in the UI). In general you can use `id` or `name` indistinctively: Using the above response as an example: `select count(1) from events_table` is equivalent to `select count(1) from t_bd1c62b5e67142bd9bf9a7f113a2b6ea` The id `t_bd1c62b5e67142bd9bf9a7f113a2b6ea` is not a descriptive name so you can add a description like `t_my_events_datasource.bd1c62b5e67142bd9bf9a7f113a2b6ea` The `statistics` property contains information about the table. Those numbers are an estimation: `bytes` is the estimated data size on disk and `row_count` the estimated number of rows. These statistics are updated whenever data is appended to the Data Source. The `used_by` property contains the list of pipes that are using this data source. Only Pipe `id` and `name` are sent. The `type` property indicates the `format` used when the Data Source was created. Available formats are `csv`, `ndjson` , and `parquet` . The Data Source `type` indicates what file format you can use to ingest data. DELETE /v0/datasources/(.+) [¶](https://www.tinybird.co/docs/about:blank#delete--v0-datasources-(.+)) Dropping a Data Source [¶](https://www.tinybird.co/docs/about:blank#id35) curl \ -H "Authorization: Bearer " \ -X DELETE "https://api.tinybird.co/v0/datasources/:name" Drops a Data Source from your account. | Key | Type | Description | | --- | --- | --- | | force | String | Default: `false` . The `force` parameter is taken into account when trying to delete Materialized Views. By default, when using `false` the deletion will not be carried out; you can enable it by setting it to `true` . If the given Data Source is being used as the trigger of a Materialized Node, it will not be deleted in any case. | | dry_run | String | Default: `false` . It allows you to test the deletion. When using `true` it will execute all deletion validations and return the possible affected materializations and other dependencies of a given Data Source. | | token | String | Auth token. Only required if no Bearer Authorization header is sent. It must have `DROP:datasource_name` scope for the given Data Source. | PUT /v0/datasources/(.+) [¶](https://www.tinybird.co/docs/about:blank#put--v0-datasources-(.+)) Update Data Source attributes Updating the name of a Data Source [¶](https://www.tinybird.co/docs/about:blank#id37) curl \ -H "Authorization: Bearer " \ -X PUT "https://api.tinybird.co/v0/datasources/:name?name=new_name" Promoting a Data Source to a Snowflake one [¶](https://www.tinybird.co/docs/about:blank#id38) curl \ -H "Authorization: Bearer " \ -X PUT "https://api.tinybird.co/v0/datasources/:name" \ -d "connector=1d8232bf-2254-4d68-beff-4dd9aa505ab0" \ -d "service=snowflake" \ -d "cron=*/30 * * * *" \ -d "query=select a, b, c from test" \ -d "mode=replace" \ -d "external_data_source=database.schema.table" \ -d "ingest_now=True" \| Key | Type | Description | | --- | --- | --- | | name | String | new name for the Data Source | | token | String | Auth token. Only required if no Bearer Authorization header is sent. It should have `DATASOURCES:CREATE` scope for the given Data Source | | connector | String | Connector ID to link it to | | service | String | Type of service to promote it to. Only ‘snowflake’ or ‘bigquery’ allowed | | cron | String | Cron-like pattern to execute the connector’s job | | query | String | Optional: custom query to collect from the external data source | | mode | String | Only replace is allowed for connectors | | external_data_source | String | External data source to use for Snowflake | | ingest_now | Boolean | To ingest the data immediately instead of waiting for the first execution determined by cron | --- URL: https://www.tinybird.co/docs/api-reference/analyze-api Last update: 2024-12-30T16:50:50.000Z Content: --- title: "Analyze API Reference · Tinybird Docs" theme-color: "#171612" description: "The Analyze API allows you analyze a given NDJSON, CSV, or Parquet file to generate a Tinybird Data Source schema." --- POST /v0/analyze/? [¶](https://www.tinybird.co/docs/about:blank#post--v0-analyze-?) The Analyze API takes a sample of a supported file ( `csv`, `ndjson`, `parquet` ) and guesses the file format, schema, columns, types, nullables and JSONPaths (in the case of NDJSON paths). This is a helper endpoint to create Data Sources without having to write the schema manually. Take into account Tinybird’s guessing algorithm is not deterministic since it takes a random portion of the file passed to the endpoint, that means it can guess different types or nullables depending on the sample analyzed. We recommend to double check the schema guessed in case you have to make some manual adjustments. Analyze a local file [¶](https://www.tinybird.co/docs/about:blank#id1) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/analyze" \ -F "file=@path_to_local_file" Analyze a remote file [¶](https://www.tinybird.co/docs/about:blank#id2) curl \ -H "Authorization: Bearer " \ -G -X POST "https://api.tinybird.co/v0/analyze" \ --data-urlencode "url=https://example.com/file" Analyze response [¶](https://www.tinybird.co/docs/about:blank#id3) { "analysis": { "columns": [ { "path": "$.a_nested_array.nested_array[:]", "recommended_type": "Array(Int16)", "present_pct": 3, "name": "a_nested_array_nested_array" }, { "path": "$.an_array[:]", "recommended_type": "Array(Int16)", "present_pct": 3, "name": "an_array" }, { "path": "$.field", "recommended_type": "String", "present_pct": 1, "name": "field" }, { "path": "$.nested.nested_field", "recommended_type": "String", "present_pct": 1, "name": "nested_nested_field" } ], "schema": "a_nested_array_nested_array Array(Int16) `json:$.a_nested_array.nested_array[:]`, an_array Array(Int16) `json:$.an_array[:]`, field String `json:$.field`, nested_nested_field String `json:$.nested.nested_field`" }, "preview": { "meta": [ { "name": "a_nested_array_nested_array", "type": "Array(Int16)" }, { "name": "an_array", "type": "Array(Int16)" }, { "name": "field", "type": "String" }, { "name": "nested_nested_field", "type": "String" } ], "data": [ { "a_nested_array_nested_array": [ 1, 2, 3 ], "an_array": [ 1, 2, 3 ], "field": "test", "nested_nested_field": "bla" } ], "rows": 1, "statistics": { "elapsed": 0.000310539, "rows_read": 2, "bytes_read": 142 } } } The `columns` attribute contains the guessed columns and for each one: - `path` : The JSONPath syntax in the case of NDJSON/Parquet files - `recommended_type` : The guessed database type - `present_pct` : If the value is lower than 1 then there was nulls in the sample used for guessing - `name` : The recommended column name The `schema` attribute is ready to be used in the [Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) The `preview` contains up to 10 rows of the content of the file. --- URL: https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Implementing test strategies · Tinybird Docs" theme-color: "#171612" description: "Learn about different strategies for testing your data project. You'll learn how to implement regression tests, data quality tests, and fixture tests." --- # Implementing test strategies [¶](https://www.tinybird.co/docs/about:blank#implementing-test-strategies) Tinybird provides you with a suite of tools to test your project. This means you can make changes and be confident that they won't break the API Endpoints you've deployed. ## Overview [¶](https://www.tinybird.co/docs/about:blank#overview) This walkthrough is based on the [Web Analytics template](https://www.tinybird.co/templates) . You can follow along using your existing Tinybird project, or by cloning the Web Analytics template. If you need to, create a new Workspace by clicking on the following button: <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcreate-workspace-with-web-analytics.png&w=3840&q=75) ### Generate mock data [¶](https://www.tinybird.co/docs/about:blank#generate-mock-data) If you want to send fake data to your project, use Tinybird's [mockingbird](https://mockingbird.tinybird.co/docs/cli) CLI tool. You'll need to run the following command to start receiving dummy events: ##### Command to populate the Web Analytics template with fake events mockingbird-cli tinybird \ --template "Web Analytics template" \ --token \ --datasource "analytics_events" \ # The region should be "eu_gcp", "us_gcp" --endpoint "" \ --eps 100 \ --limit 1000 ## Testing strategies [¶](https://www.tinybird.co/docs/about:blank#testing-strategies) You can implement three different testing strategies in a Tinybird data project: Regression tests, data quality tests, and fixture tests. All three of them are included as part of the [Continuous Integration workflow](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration). **Regression tests** prevent you from breaking working APIs. They run automatically on each commit to a Pull Request, or when trying to overwrite a Pipe with an API Endpoint in a Workspace. They compare both the output and performance of your API Endpoints using the previous and current versions of the Pipe endpoints. **Data quality tests** warn you about anomalies in the data. As opposed to regression tests, you do have to write data quality tests to cover one or more criteria over your data: the presence of null values, duplicates, out-of-range values, etc. Data quality tests are usually scheduled by users to run over production data as well. **Fixture tests** are like "manual tests" for your API Endpoints. They check that a given call to a given Pipe endpoint with a set of parameters and a known set of data (fixtures) returns a deterministic response. They're useful for coverage testing and for when you are developing or debugging new business logic that requires very specific data scenarios. When creating fixture tests, you must provide both the test and any data fixtures. ## Regression tests [¶](https://www.tinybird.co/docs/about:blank#regression-tests) When one of your API Endpoints is integrated into a production environment (such as a web/mobile application or a dashboard), you want to make sure that any change in the Pipe doesn't change the output of the API endpoint. In other words, you want the same version of an API Endpoint to return the same data for the same requests. Tinybird provides you with automatic regression tests that run any time you push a new change to an API Endpoint. Here's an example you can follow along by reading. Imagine you have a `top_browsers` Pipe: ##### Definition of the top_browsers.pipe file DESCRIPTION > Top Browsers ordered by most visits. Accepts `date_from` and `date_to` date filter. Defaults to last 7 days. Also `skip` and `limit` parameters for pagination. TOKEN "dashboard" READ NODE endpoint DESCRIPTION > Group by browser and calculate hits and visits SQL > % SELECT browser, uniqMerge(visits) as visits, countMerge(hits) as hits FROM analytics_sources_mv WHERE {% if defined(date_from) %} date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}} {% else %} date >= timestampAdd(today(), interval -7 day) {% end %} {% if defined(date_to) %} and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}} {% else %} and date <= today() {% end %} GROUP BY browser ORDER BY visits desc LIMIT {{Int32(skip, 0)}},{{Int32(limit, 50)}} These requests are sent to the API Endpoint: ##### Let's run some requests to the API Endpoint to see the output curl https://api.tinybird.co/v0/pipes/top_browsers.json?token={TOKEN} curl https://api.tinybird.co/v0/pipes/top_browsers.json?token={TOKEN}&date_from=2020-04-23&date_to=2030-04-23 Now, it's possible to filter by `device` and modify the API Endpoint: ##### Definition of the top_browsers.pipe file DESCRIPTION > Top Browsers ordered by most visits. Accepts `date_from` and `date_to` date filter. Defaults to last 7 days. Also `skip` and `limit` parameters for pagination. TOKEN "dashboard" READ NODE endpoint DESCRIPTION > Group by browser and calculate hits and visits SQL > % SELECT browser, uniqMerge(visits) as visits, countMerge(hits) as hits FROM analytics_sources_mv WHERE {% if defined(date_from) %} date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}} {% else %} date >= timestampAdd(today(), interval -7 day) {% end %} {% if defined(date_to) %} and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}} {% else %} and date <= today() {% end %} {% if defined(device) %} and device = {{String(device, description="Device to filter", required=False)}} {% end %} GROUP BY browser ORDER BY visits desc LIMIT {{Int32(skip, 0)}},{{Int32(limit, 50)}} At this point, you'd create a new Pull Request like [this example](https://github.com/tinybirdco/use-case-examples/pull/213) with the changes above, so the API Endpoint is tested for regressions. On the standard Tinybird [Continuous Integration pipeline](https://github.com/tinybirdco/use-case-examples/actions/runs/7616008955/job/20741856936?pr=213) , changes are deployed to a branch, and there's a `Run Pipe regression tests` step that runs the following command on that branch: ##### Run regression tests tb branch regression-tests coverage --wait The next step creates a Job that runs the coverage of the API Endpoints. The Job tests all combinations of parameters by running at least one request for each combination, and comparing the results of the new and old versions of the Pipe: ##### Run coverage regression tests ## In case the endpoints don't have any requests, we will show a warning so you can delete the endpoint if it's not needed or it's expected 🚨 No requests found for the endpoint analytics_hits - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint trend - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_locations - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint kpis - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_sources - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_pages - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies ## If the endpoint has been requested, we will run the regression tests and show the results ## This validation is running for each combination of parameters and comparing the results from the branch against the resources that we have copied from the production OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?&pipe_checker=true - 0.318s (9.0%) 8.59 KB (0.0%) OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_start=2020-04-23&date_end=2030-04-23&pipe_checker=true - 0.267s (-58.0%) 8.59 KB (0.0%) OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-04-23&date_to=2030-04-23&pipe_checker=true - 0.297s (-26.0%) 0 bytes (0%) 🚨 No requests found for the endpoint top_devices - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies ==== Performance metrics ==== --------------------------------------------------------------------- | top_browsers(coverage) | Origin | Branch | Delta | --------------------------------------------------------------------- | min response time | 0.293 seconds | 0.267 seconds | -8.83 % | | max response time | 0.639 seconds | 0.318 seconds | -50.16 % | | mean response time | 0.445 seconds | 0.294 seconds | -33.87 % | | median response time | 0.402 seconds | 0.297 seconds | -26.23 % | | p90 response time | 0.639 seconds | 0.318 seconds | -50.16 % | | min read bytes | 0 bytes | 0 bytes | +0.0 % | | max read bytes | 8.59 KB | 8.59 KB | +0.0 % | | mean read bytes | 5.73 KB | 5.73 KB | +0.0 % | | median read bytes | 8.59 KB | 8.59 KB | +0.0 % | | p90 read bytes | 8.59 KB | 8.59 KB | +0.0 % | --------------------------------------------------------------------- ==== Results Summary ==== -------------------------------------------------------------------------------------------- | Endpoint | Test | Run | Passed | Failed | Mean response time | Mean read bytes | -------------------------------------------------------------------------------------------- | analytics_hits | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | trend | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_locations | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | kpis | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_sources | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_pages | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_browsers | coverage | 3 | 3 | 0 | -33.87 % | +0.0 % | | top_devices | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | -------------------------------------------------------------------------------------------- As you can see, regression tests are run for each combination of parameters and the results are compared against the Workspace. In this case, there are no regression issues since adding a new filter. Let's see what happens if you introduce a breaking change in the Pipe definition. First, you'd run some requests using the device filter: ##### Let's run some requests using the device filter curl https://api.tinybird.co/v0/pipes/top_browsers.json?token={TOKEN}&device=mobile-android curl https://api.tinybird.co/v0/pipes/top_browsers.json?token={TOKEN}&date_from=2020-04-23&date_to=2030-04-23&device=desktop Then, introduce a breaking change in the Pipe definition by removing the `device` filter that was added in the previous step: ##### Definition of the top_browsers.pipe file DESCRIPTION > Top Browsers ordered by most visits. Accepts `date_from` and `date_to` date filter. Defaults to last 7 days. Also `skip` and `limit` parameters for pagination. TOKEN "dashboard" READ NODE endpoint DESCRIPTION > Group by browser and calculate hits and visits SQL > % SELECT browser, uniqMerge(visits) as visits, countMerge(hits) as hits FROM analytics_sources_mv WHERE {% if defined(date_from) %} date >= {{Date(date_from, description="Starting day for filtering a date range", required=False)}} {% else %} date >= timestampAdd(today(), interval -7 day) {% end %} {% if defined(date_to) %} and date <= {{Date(date_to, description="Finishing day for filtering a date range", required=False)}} {% else %} and date <= today() {% end %} GROUP BY browser ORDER BY visits desc LIMIT {{Int32(skip, 0)}},{{Int32(limit, 50)}} At this point, you'd create a Pull Request [like this example](https://github.com/tinybirdco/use-case-examples/pull/215) with the change above so the API Endpoint is tested for regressions. This time, the output is slightly different than before: ##### Run coverage regression tests 🚨 No requests found for the endpoint analytics_hits - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint trend - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_locations - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint kpis - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_sources - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies 🚨 No requests found for the endpoint top_pages - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies ## The requests without using the device filter are still working, but the other will return a different number of rows, value OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?&pipe_checker=true - 0.3s (-45.0%) 8.59 KB (0.0%) FAIL - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?device=mobile-android&pipe_checker=true - 0.274s (-43.0%) 8.59 KB (-2.0%) OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_start=2020-04-23&date_end=2020-04-23&pipe_checker=true - 0.341s (18.0%) 8.59 KB (0.0%) OK - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-04-23&date_to=2020-04-23&pipe_checker=true - 0.314s (-49.0%) 0 bytes (0%) FAIL - top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-01-23&date_to=2030-04-23&device=desktop&pipe_checker=true - 0.218s (-58.0% skipped < 0.3) 8.59 KB (-2.0%) 🚨 No requests found for the endpoint top_devices - coverage (Skipping validation). 💡 See this guide for more info about the regression tests => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies ==== Failures Detail ==== ❌ top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?device=mobile-android&pipe_checker=true ** 1 != 4 : Unexpected number of result rows count, this might indicate regression. 💡 Hint: Use `--no-assert-result-rows-count` if it's expected and want to skip the assert. Origin Workspace: https://api.tinybird.co/v0/pipes/top_browsers.json?device=mobile-android&pipe_checker=true&__tb__semver=0.0.0 Test Branch: https://api.tinybird.co/v0/pipes/top_browsers.json?device=mobile-android&pipe_checker=true ❌ top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-01-23&date_to=2030-04-23&device=desktop&pipe_checker=true ** 3 != 4 : Unexpected number of result rows count, this might indicate regression. 💡 Hint: Use `--no-assert-result-rows-count` if it's expected and want to skip the assert. Origin Workspace: https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-01-23&date_to=2030-04-23&device=desktop&pipe_checker=true&__tb__semver=0.0.0 Test Branch: https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-01-23&date_to=2030-04-23&device=desktop&pipe_checker=true ==== Performance metrics ==== Error: ** Check Failures Detail above for more information. If the results are expected, skip asserts or increase thresholds, see 💡 Hints above (note skip asserts flags are applied to all regression tests, so use them when it makes sense). If you are using the CI template for GitHub or GitLab you can add skip asserts flags as labels to the MR and they are automatically applied. Find available flags to skip asserts and thresholds here => https://www.tinybird.co/docs/work-with-data/strategies/implementing-test-strategies#testing-strategies --------------------------------------------------------------------- | top_browsers(coverage) | Origin | Branch | Delta | --------------------------------------------------------------------- | min response time | 0.29 seconds | 0.218 seconds | -24.61 % | | max response time | 0.612 seconds | 0.341 seconds | -44.28 % | | mean response time | 0.491 seconds | 0.289 seconds | -41.06 % | | median response time | 0.523 seconds | 0.3 seconds | -42.72 % | | p90 response time | 0.612 seconds | 0.341 seconds | -44.28 % | | min read bytes | 0 bytes | 0 bytes | +0.0 % | | max read bytes | 8.8 KB | 8.59 KB | -2.33 % | | mean read bytes | 6.95 KB | 6.87 KB | -1.18 % | | median read bytes | 8.59 KB | 8.59 KB | +0.0 % | | p90 read bytes | 8.8 KB | 8.59 KB | -2.33 % | --------------------------------------------------------------------- ==== Results Summary ==== -------------------------------------------------------------------------------------------- | Endpoint | Test | Run | Passed | Failed | Mean response time | Mean read bytes | -------------------------------------------------------------------------------------------- | analytics_hits | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | trend | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_locations | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | kpis | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_sources | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_pages | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | | top_browsers | coverage | 5 | 3 | 2 | -41.06 % | -1.18 % | | top_devices | coverage | 0 | 0 | 0 | +0.0 % | +0.0 % | -------------------------------------------------------------------------------------------- ❌ FAILED top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?device=mobile-android&pipe_checker=true ❌ FAILED top_browsers(coverage) - https://api.tinybird.co/v0/pipes/top_browsers.json?date_from=2020-01-23&date_to=2030-04-23&device=desktop&pipe_checker=true Because the API Endpoint filter changed, the request response changed, and the regression testing warns you. If the change is expected, you can skip the assertion by adding the following labels to the Pull Request: - `--no-assert-result` : If you expect the API Endpoint output to be different from the current version - `--no-assert-result-no-error` : If you expect errors from the API Endpoint - `--no-assert-result-rows-count` : If you expect the number of elements in the API Endpoint output to be different than the current version - `--assert-result-ignore-order` : If you expect the API Endpoint output is returning the same elements but in a different order - `--assert-time-increase-percentage -1` : If you expect the API Endpoint execution time to increase more than 25% from the current version - `--assert-bytes-read-increase-percentage -1` : If you expect the API Endpoint bytes read to increase more than 25% from the current version - `--assert-max-time` : If you expect the API Endpoint execution time to vary but you don't want to assert the increase in time up to a certain threshold. For instance, if you want to allow your API Endpoints to respond in up to 1s and you don't want to assert any increase in time percentage use `--assert-max-time 1` . By default is 0.3s. These flags will be applied to ALL the regression tests and are advised to be used for one-time breaking changes. Define a file in `tests/regression.yaml` to configure the behavior of the regression tests for each API Endpoint. Follow the docs in [Configure regression tests](https://www.tinybird.co/docs/about:blank#configure-regression-tests). For this example, you would add the label `--no-assert-result-rows-count` to the PR as you'd expect the number of rows to be different and you'd want to remove the filter. <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fskip-assert-result-rows-count.png&w=3840&q=75) At this point, the regression tests would pass and the PR could be merged. ### How regression tests work [¶](https://www.tinybird.co/docs/about:blank#how-regression-tests-work) The regression test functionality is powered by `tinybird.pipe_stats_rt` , one of the [service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) that is available to all users by default. There are three options to run regression tests: `coverage`, `last` and `manual`. - The `coverage` option will run at least 1 request for each combination of parameters. - The `last` option will run the last N requests for each combination of parameters. - The `manual` option will run the requests you define in the configuration file `tests/regression.yaml` . When you run `tb branch regression-tests coverage --wait` , Tinybird generates a job that queries `tinybird.pipe_stats_rt` to gather all the possible combinations of queries done in the last 7 days for each API Endpoint. ##### Simplification of the query used to gather all the possible combinations SELECT ## Using this function we extract all the parameters used in each request extractURLParameterNames(assumeNotNull(url)) as params, ## According to the option `--sample-by-params`, we run one query for each combination of parameters or more groupArraySample({sample_by_params if sample_by_params > 0 else 1})(url) as endpoint_url FROM tinybird.pipe_stats_rt WHERE pipe_name = '{pipe_name}' ## According to the option `--match`, we will filter only the requests that contain that parameter ## This is especially useful when you want to validate a new parameter you want to introduce or you have optimized the endpoint in that specific case { " AND " + " AND ".join([f"has(params, '{match}')" for match in matches]) if matches and len(matches) > 0 else ''} GROUP BY params FORMAT JSON When you run `tb branch regression-tests last --wait` , Tinybird generates a job that queries `tinybird.pipe_stats_rt` to gather the last N requests for each API Endpoint. ##### Query to get the last N requests SELECT url FROM tinybird.pipe_stats_rt WHERE pipe_name = '{pipe_name}' ## According to the option `--match`, we will filter only the requests that contain that parameter ## This is especially useful when you want to validate a new parameter you want to introduce or you have optimize the endpoint in that specific case { " AND " + " AND ".join([f"has(params, '{match}')" for match in matches]) if matches and len(matches) > 0 else ''} ## According to the option `--limit` by default 100 LIMIT {limit} FORMAT JSON ### Configure regression tests [¶](https://www.tinybird.co/docs/about:blank#configure-regression-tests) By default, the CI Workflow uses the `coverage` option when running the regression-test: ##### Default command to run regression tests in the CI Workflow tb branch regression-tests coverage --wait But if it finds a `tests/regression.yaml` file in your project, it uses the configuration defined in that file. ##### Run regression tests for all endpoints in a test branch using a configuration file tb branch regression-tests -f tests/regression.yaml --wait If the file `tests/regression.yaml` is present, only `--skip-regression-tests` and `--no-skip-regression-tests` labels in the Pull Request will take effect The configuration file is a YAML file with the following structure: - pipe: '.*' # regular expression that selects all API Endpoints in the Workspace tests: # list of tests to run for this Pipe - [coverage|last|manual]: # testing strategy to use (coverage, last, or manual) config: # configuration options for this strategy assert_result: bool = True # whether to perform an assertion on the results returned by the endpoint assert_result_no_error: bool = True # whether to verify that the endpoint doesn't return errors assert_result_rows_count: bool = True # whether to verify that the correct number of elements are returned in the results assert_result_ignore_order: bool = False # whether to ignore the order of the elements in the results assert_time_increase_percentage: int = 25 # allowed percentage increase in endpoint response time. use -1 to disable assert assert_bytes_read_increase_percentage: int = 25 # allowed percentage increase in the amount of bytes read by the endpoint. use -1 to disable assert assert_max_time: float = 1 # Max time allowed for the endpoint response time. If the response time is lower than this value then the assert_time_increase_percentage isn't taken into account. Default is 0.3 skip: bool = False # whether the test should be skipped, use it to skip individual Pipe tests failfast: bool = False # whether the test should stop at the first error encountered Note that the order of preference for the configuration options is from bottom to top, so the configuration options specified for a particular Pipe take precedence over the options specified earlier (higher up) in the configuration file. Here's an example YAML file with two entries for two regular expressions of a Pipe, where one overrides the configuration of the other: - pipe: 'top_.*' tests: - coverage: config: # Default config but reducing thresholds from default 25 and expecting different order in the response payload assert_time_increase_percentage: 15 assert_bytes_read_increase_percentage: 15 assert_result_ignore_order: true - last: config: # default config but not asserting performance and failing at first occurrence assert_time_increase_percentage: -1 assert_bytes_read_increase_percentage: -1 failfast: true limit: 5 # Default value will be 10 - pipe: 'top_pages' tests: - coverage: - manual: params: - {param1: value1, param2: value2} - {param1: value3, param2: value4} config: failfast: true # Override config for top_pages executing coverage with default config and two manual requests ## Data quality tests [¶](https://www.tinybird.co/docs/about:blank#data-quality-tests) Data quality tests are meant to cover scenarios that don't have to happen in your production data. For example, you can check that the data isn't duplicated or you don't have values out of range. Data quality tests are run with the `tb test` command. You can use them in two different ways: - Run them periodically over your production data. - Use them as part of your test suite in Continuous Integration with a Branch or fixtures. Here's an example you can follow along by reading - no need to clone anything. In the Web Analytics example, you can use data quality tests to validate that there are no duplicate entries with the same session_id in the `analytics_events`. Run `tb test init` to generate a dummy test file in `tests/default.yaml`: ##### Example of tests generated by running tb test init # This test should always pass as numbers(5) returns values from [1,5] - this_test_should_pass: max_bytes_read: null max_time: null pipe: null sql: SELECT * FROM numbers(5) WHERE 0 # This test should always fail as numbers(5) returns values from [1,5]. Therefore the SQL will return a value - this_test_should_fail: max_bytes_read: null max_time: null pipe: null sql: SELECT * FROM numbers(5) WHERE 1 # If max_time is specified, the test will show a warning if the query takes more than the threshold - this_test_should_pass_over_time: max_bytes_read: null max_time: 1.0e-07 pipe: null sql: SELECT * FROM numbers(5) WHERE 0 # if max_bytes_read is specified, the test will show a warning if the query reads more bytes than the threshold - this_test_should_pass_over_bytes: max_bytes_read: 5 max_time: null pipe: null sql: SELECT sum(number) AS total FROM numbers(5) HAVING total>1000 # The combination of both - this_test_should_pass_over_time_and_bytes: max_bytes_read: 5 max_time: 1.0e-07 pipe: null sql: SELECT sum(number) AS total FROM numbers(5) HAVING total>1000 These tests check that the SQL query returns an empty result. If the result isn't empty, the test fails. In this example, you'd write a similar test to check that there are no duplicate entries with the same session_id in the `analytics_events`: - no_duplicate_entries: max_bytes_read: null max_time: null sql: | SELECT date, sumMerge(total_sales) total_sales FROM top_products_view GROUP BY date HAVING total_sales < 0 You can follow along with the [example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/216) that would be made, and see the [Run tests](https://github.com/tinybirdco/use-case-examples/actions/runs/7626991493/job/20774698416?pr=216) workflow output: ##### Output of data quality tests -------------------------- semver: 0.0.2 file: ./tests/default.yaml test: no_duplicate_entries status: Pass elapsed: 0.001710404 ms -------------------------- Totals: Total Pass: 1 If the test fails, the CI workflow will fail, and the output returns the value of the SQL query. You can also test the output of a Pipe. For instance, in the Web Analytics example, you could add a validation: 1. First, query the API Endpoint `top_products` with the parameters `date_start` and `date_end` specified in the test. 2. Then, run the SQL query from the result of the previous step. ##### Data quality test for a Pipe - products_by_date: max_bytes_read: null max_time: null sql: | SELECT 1 FROM top_products HAVING count() = 0 pipe: name: top_products params: date_start: '2020-01-01' date_end: '2020-12-31' ## Fixture tests [¶](https://www.tinybird.co/docs/about:blank#fixture-tests) Regression tests confirm the backward compatibility of your API Endpoints when overwriting them, and data quality tests cover scenarios that might not happen with test or production data. Sometimes, you need to cover a very specific scenario, or a use case that is being developed and you don't have production data for it. This is when to use fixture tests. To configure fixture testing you need: - A script to run fixture tests, like[ this example](https://github.com/tinybirdco/ci/blob/main/scripts/exec_test.sh) . The script is automatically created when you[ connect your Workspace to Git](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) . - Data fixtures: These are datafiles placed in the `datasources/fixtures` folder of your project. Their name must match the name of a Data Source. - Fixture tests in the `tests` folder. In the Continuous Integration job, a Branch is created. If fixture data exists in the `datasources/fixtures` folder, is appended to the Data Sources in the Branch, and the fixture tests are run. To effectively use data fixtures you should: - Use data that don't collide with production data, to avoid unexpected results in regression testing. - Use only data fixtures for landing Data Sources since Materialized Views are automatically populated. - Use future dates in the data fixtures to avoid problems with the TTL of the Data Sources. Fixture tests are placed inside the `tests` folder of your project. If you have a lot of tests, create subfolders to organize the tests by module or API Endpoint. Each fixture test requires two files: - One to indicate a request to an API Endpoint with the naming convention `.test` - One to indicate the exact response to the API Endpoint with the naming convention `.test.result` For instance, to test the output of the `top_browsers` endpoint, create a `simple_top_browsers.test` fixture test with this content: ##### top_browsers.test tb pipe data top_browsers --date_from 2100-01-01 --date_to 2100-02-01 --format CSV The test makes a request to the `top_browsers` API endpoint passing the `date_from` and `date_to` parameters and the response is `CSV` Now we need a `simple_top_browsers.test.result` with the expected result given our data fixtures: ##### simple_top_browsers.test.result "browser","visits","hits" "chrome",1,1 With this approach, your tests should run like this [example Pull Request](https://github.com/tinybirdco/use-case-examples/pull/217/) . Fixture tests are run as part of the Continuous Integration pipeline and the Job fails if the tests fail. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Ffixture-tests-CI-step.png&w=3840&q=75) ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn how to use[ staging and production Workspaces](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/staging-and-production-workspaces) . - Check out an[ example test Pull Request](https://github.com/tinybirdco/use-case-examples/pull/217/) . --- URL: https://www.tinybird.co/docs/work-with-data/strategies/deployment-strategies Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Deployment strategies · Tinybird Docs" theme-color: "#171612" description: "Things to take into account when building your Continuous Deployment pipeline." --- # Deployment strategies [¶](https://www.tinybird.co/docs/about:blank#deployment-strategies) This page explains deployment strategies when you are doing one of the following: - Adding, updating or deleting resources - Maintaining streaming ingestion - Handling user API requests It covers the default method for implementing Continuous Deployment (CD), how to bypass the default deployment strategy to create custom deployments, and finally, strategies to take into account when migrating data. ## How deployment works [¶](https://www.tinybird.co/docs/about:blank#how-deployment-works) With the Git integration, your project in Git is the real source of truth and you should expect your Workspace(s) to be a working version of the resources in Git. In the default [CI/CD workflow templates](https://github.com/tinybirdco/ci) the `tb deploy` command is used to deploy changes to the Workspace. This does the following: - Checks the current commit in the Workspace and validates that is an ancestor of the commit in the Pull Request being deployed. If not, usually you have to `git rebase` your branch. - Performs a `git diff` from the current branch to the main branch so it can get a list of the datafiles that changed. - Deploys them both in CI and CD. ## Alter strategy [¶](https://www.tinybird.co/docs/about:blank#alter-strategy) Updates existing resources that have been changed. This is the default deployment strategy when using `tb deploy` . Use it to add a new column to a Data Source or change the TTL. Not all operations can be performed with this strategy. For instance, you can't change the Sorting Key of a Data Source with this strategy. An example use case can be found here: [Add column to a Data Source](https://github.com/tinybirdco/use-case-examples/tree/main/add_nullable_column_to_landing_data_source). ## Versioning strategy [¶](https://www.tinybird.co/docs/about:blank#versioning-strategy) When you want to make a breaking change to some resource, create a new version of that resource. You can perform this strategy in different steps each one corresponding to a Pull Request like this: - Create a new Branch with the new resource(s) (Pipe or Data Source) with a different name and deploy it. - Make sure any backfill operation is run over the new resources so data in the main Workspace is in sync. - Create a new Branch to connect the corresponding dependencies to the new resource(s) and deploy it. - Create a new Branch to `git rm` the old resources and deploy it. This strategy allows for a staged and controlled deployment of breaking changes. You keep old and new versions of the resource(s) in the main Workspace, until your end user application is rolled out. ## Custom deployments [¶](https://www.tinybird.co/docs/about:blank#custom-deployments) The `tb deploy` command allows you to deploy a project with a single command. Under the hood, this command performs a series of actions that are common to most deployments. However, your project might have specific requirements that mean you need to customize these actions, or run additional actions after the deployment process. Use a custom deployment when you need to perform some operation that's not directly supported by `tb deploy` or require several steps like automating a backfill operation. ### Custom deployment actions [¶](https://www.tinybird.co/docs/about:blank#custom-deployment-actions) The `deploy.sh` file allows you to overwrite the default deployment process, so instead of running the default `tb deploy` command you can run a custom shell script to deploy your changes. Use custom deployment actions if the default deployment process isn't suitable for your project. For example, you might want to deploy resources in a specific order or handle errors differently. To create custom deployment actions: - If you are using the Tinybird CI/CD templates, export an environment variable in your CI/CD templates with name `VERSION` . You can find it in the `.tinyenv` file in the data project. - Create a `deploy/$VERSION/deploy.sh` file and ensure it has execution permissions. For example, `chmod +x -R deploy/0.0.1/` . - In the `deploy.sh` file, write the Tinybird CLI commands you want to execute during the deployment process. - The CI/CD pipelines will find the `deploy.sh` file and execute it when deploying the matching version of the project. It's important to note that custom deployments run in CI, so use it to validate that the custom deployment will work when merging the Pull Request. Once you merge a Pull Request with a custom deployment make sure you update the `VERSION` environment variable, so this custom deployment doesn't run on the next Pull Request. After a custom deployment, if you did not run `tb deploy` make sure your Workspace is synchronized with the Git main branch head commit by running `tb init --override-commit HEAD_COMMIT_ID`. ## Deploying common use cases [¶](https://www.tinybird.co/docs/about:blank#deploying-common-use-cases) Check out [the Use Case repository](https://github.com/tinybirdco/use-case-examples) for common use cases and scenarios. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Practice iterating on one of Tinybird's examples in the[ Use Case repository](https://github.com/tinybirdco/use-case-examples) . - Learn about[ backfill strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/backfill-strategies) . --- URL: https://www.tinybird.co/docs/work-with-data/strategies/deduplication-strategies Last update: 2024-12-18T15:47:50.000Z Content: --- title: "Deduplicate data in your Data Source · Tinybird Docs" theme-color: "#171612" description: "Learn several strategies for deduplicating data in Tinybird." --- # Deduplicate data in your Data Source [¶](https://www.tinybird.co/docs/about:blank#deduplicate-data-in-your-data-source) Sometimes you might need to deduplicate data, for example to receive updates or data from a transactional database through CDC. You might want to retrieve only the latest data point, or keep a historic record of the evolution of the attributes of an object over time. Because Tinybird doesn't enforce uniqueness for primary keys when inserting rows, you need to follow different strategies to deduplicate data with minimal side effects. ## Deduplication strategies [¶](https://www.tinybird.co/docs/about:blank#deduplication-strategies) You can use one of the following strategies to deduplicate your data. | Method | When to use | | --- | --- | | [ Deduplicate at query time](https://www.tinybird.co/docs/about:blank#deduplicate-at-query-time) | Deduplicate data at query time if you are still prototyping or the Data Source is small. | | [ Use ReplacingMergeTree](https://www.tinybird.co/docs/about:blank#use-the-replacingmergetree-engine) | Use `ReplacingMergeTree` or `AggregatingMergeTree` for greater performance. | | [ Snapshot based deduplication](https://www.tinybird.co/docs/about:blank#snapshot-based-deduplication) | If data freshness isn't required, generate periodic snapshots of the data and take advantage of subsequent Materialized Views for rollups. | | [ Hybrid approach using Lambda architecture](https://www.tinybird.co/docs/about:blank#hybrid-approach-using-lambda-architecture) | When you need to overcome engine approach limitations while preserving freshness, combine approaches in a Lambda architecture. | For dimensional and small tables, a periodical full replace is usually the best option. ## Example case [¶](https://www.tinybird.co/docs/about:blank#example-case) Consider a dataset from a social media analytics company that wants to track some data content over time. You receive an event with the latest info for each post, identified by `post_id` . The three fields, `views`, `likes`, `tags` , vary from event to event. For example: ##### post.ndjson { "timestamp": "2024-07-02T02:22:17", "post_id": 956, "views": 856875, "likes": 2321, "tags": "Sports" } ## Deduplicate at query time [¶](https://www.tinybird.co/docs/about:blank#deduplicate-at-query-time) Imagine you're only interested in the latest value of views for each post. In that case, you can deduplicate data on `post_id` and get the latest value with these strategies: - Get the max date for each post in a subquery and then filter by its results. - Group data by `post_id` and use the `argMax` function. - Use the `LIMIT BY` clause. Select `Subquery`, `argMax` , or `LIMIT BY` to see the example queries for each. - Subquery - argMax - LIMIT BY ##### Deduplicating data on post_id using Subquery SELECT * FROM posts_info WHERE (post_id, timestamp) IN ( SELECT post_id, max(timestamp) FROM posts_info GROUP BY post_id ) Depending on your data and how you define the sorting keys in your Data Sources to store it on disk, one approach is faster than the others. In general, deduplicating at query time is fine if the size of your data is small. If you have lots of data, use a specific Engine that takes care of deduplication for you. ## Use the ReplacingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#use-the-replacingmergetree-engine) If you've lots of data and you're interested in the latest insertion for each unique key, use the ReplacingMergeTree engine with the following options: `ENGINE_SORTING_KEY`, `ENGINE_VER` , and `ENGINE_IS_DELETED`. - Rows with the same `ENGINE_SORTING_KEY` are deduplicated. You can select one or more columns. - If you specify a type for `ENGINE_VER` , the row with the highest `ENGINE_VER` for each unique `ENGINE_SORTING_KEY` is kept, for example a timestamp. - `ENGINE_IS_DELETED` is only active if you use `ENGINE_VER` . This column determines whether the row represents the state or is to be deleted; `1` is a deleted row, `0` is a state row. The type must be `UInt8` . - You can omit `ENGINE_VER` , so that the last inserted row for each unique `ENGINE_SORTING_KEY` is kept. Aggregation or rollups in Materialized Views built on top of ReplacingMergeTree queries always contain duplicated data. ### Define a Data Source [¶](https://www.tinybird.co/docs/about:blank#define-a-data-source) Define a Data Source like the following: ##### post_views_rmt.datasource DESCRIPTION > Data Source to save post info. ReplacingMergeTree Engine. SCHEMA > `post_id` Int32 `json:$.post_id`, `views` Int32 `json:$.views`, `likes` Int32 `json:$.likes`, `tag` String `json:$.tag`, `timestamp` DateTime `json:$.timestamp`, `_is_deleted` UInt8 `json:$._is_deleted` ENGINE "ReplacingMergeTree" ENGINE_PARTITION_KEY "" ENGINE_SORTING_KEY "post_id" ENGINE_VER "timestamp" ENGINE_IS_DELETED "_is_deleted" ReplacingMergeTree deduplicates during a merge, and merges can't be controlled. Consider adding the `FINAL` clause, or an alternative deduplication method, to apply the merge at query time. Note also that rows are masked, not removed, when using `FINAL`. - FINAL - Subquery - argMax - LIMIT BY ##### Deduplicating data on post_id using FINAL SELECT * FROM posts_info_rmt FINAL You can define the `posts_info_rmt` as the landing Data Source, the one you send events to, or as a Materialized View from `posts_info` . You can also create a Data Source with an `AggregatingMergeTree` Engine using `maxState(ts)` and `argMaxState(field,ts)`. ## Snapshot based deduplication [¶](https://www.tinybird.co/docs/about:blank#snapshot-based-deduplication) Use [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) to take a query result and write it to a new Data Source in the following situations: - You need other Sorting Keys that might change with updates. - You need to do rollups and want to use Materialized Views. - Response times are too long with a `ReplacingMergeTree` . The following is an example snapshot: ##### post_generate_snapshot.pipe NODE gen_snapshot SQL > SELECT post_id, argMax(views, timestamp) views, argMax(likes, timestamp) likes, argMax(tag, timestamp) tag, max(timestamp) as ts, toStartOfMinute(now()) - INTERVAL 1 MINUTE as snapshot_ts FROM posts_info WHERE timestamp <= toStartOfMinute(now()) - INTERVAL 1 MINUTE GROUP BY post_id TYPE copy TARGET_DATASOURCE post_snapshot COPY_MODE replace COPY_SCHEDULE 0 * * * * Because the `TARGET_DATASOURCE` engine is a MergeTree, you can use fields that you expect to be updated as sorting keys in the ReplacingMergeTree. ##### post_snapshot.datasource SCHEMA > `post_id` Int32, `views` Int32, `likes` Int32, `tag` String, `ts` DateTime, `snapshot_ts` DateTime ENGINE "MergeTree" ENGINE_PARTITION_KEY "" ENGINE_SORTING_KEY "tag, post_id" ## Hybrid approach using Lambda architecture [¶](https://www.tinybird.co/docs/about:blank#hybrid-approach-using-lambda-architecture) Snapshots might decrease data freshness, and running Copy Pipes too frequently might be more expensive than Materialized Views. A way to mitigate these issues is to combine batch and real-time processing, reading the latest snapshot and incorporating the changes that happened since then. This pattern is described in the [Lambda architecture](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-architecture) guide. See a practical example in the [CDC using Lambda](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-example-cdc) guide. Using the `post_snapshot` Data Source created before, the real-time Pipe would be like the following: ##### latest_values.pipe NODE get_latest_changes SQL > SELECT max(timestamp) last_ts, post_id, argMax(views, timestamp) views, argMax(likes, timestamp) likes, argMax(tag, timestamp) tag FROM posts_info_rmt WHERE timestamp > (SELECT max(snapshot_ts) FROM post_snapshot) GROUP BY post_id NODE get_snapshot SQL > SELECT last_ts, post_id, views, likes, tag FROM posts_info_rmt WHERE snapshot_ts = (SELECT max(snapshot_ts) FROM post_snapshot) AND post_id NOT IN (SELECT post_id FROM get_latest_changes) NODE combine_both SQL > SELECT * FROM get_snapshot UNION ALL SELECT * FROM get_latest_changes ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read the[ Materialized Views docs](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views#creating-a-materialized-view-in-the-tinybird-ui) . - Read the[ Lambda architecture guide](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-architecture) . - Visualize your data using[ Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts) . --- URL: https://www.tinybird.co/docs/work-with-data/strategies/backfill-strategies Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Backfill strategies for iterations · Tinybird Docs" theme-color: "#171612" description: "When iterating Data Sources or Materialized Views, you will often need to backfill data from a Tinybird Data Source to another. This guide will help you understand the different strategies to do it in a safe way." --- # Backfill strategies for iterations [¶](https://www.tinybird.co/docs/about:blank#backfill-strategies-for-iterations) Backfilling data is the process of filling in missing data that didn't exist before. Whether you're changing data types, changing the sorting key, or redefining whole views, at some point you may need to run a backfill from the previous version of your Data Source or Materialized View to the new one. This page introduces the key challenges of backfilling real-time data, and covers the different strategies to run a backfill when you are iterating a Data Source or Materialized View. Before you start iterating and making critical changes to your Data Sources, Materialized Views, and Pipes, it's crucial to read the [Deployment Strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deployment-strategies) docs. ## The challenge of backfilling real-time data [¶](https://www.tinybird.co/docs/about:blank#the-challenge-of-backfilling-real-time-data) The iteration of Data Sources or Materialized Views often needs a careful approach to backfill data. This process becomes critical, especially when you create a new version of a Data Source or Materialized View, which results in creating a new, empty Data Source or Materialized View. The main challenge lies in migrating historical data while continuously ingesting new real time data. See the detailed explanation [Best practices for Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices). ## Use case [¶](https://www.tinybird.co/docs/about:blank#use-case) Imagine you have the following Data Source deployed in our main Workspace: ##### analytics_events.datasource SCHEMA > `timestamp` DateTime `json:$.timestamp`, `session_id` String `json:$.session_id`, `action` LowCardinality(String) `json:$.action`, `version` LowCardinality(String) `json:$.version`, `payload` String `json:$.payload` ENGINE "MergeTree" ENGINE_PARTITION_KEY "toYYYYMM(timestamp)" ENGINE_SORTING_KEY "timestamp" You want to modify the sorting key from `timestamp` to `action, timestamp` . This change requires you to create a new Data Source, for example `events1.datasource`. When merging the Pull Request, you will have `analytics_events` and `analytics_events_1` in your main Workspace (also in the branch while in CI). This means that you will have two Data Sources one representing the new version you want to deploy which since is newly created is empty. So, how you can sync the data between the two Data Sources? This is when you use a backfill. ## How to move data in Tinybird [¶](https://www.tinybird.co/docs/about:blank#how-to-move-data-in-tinybird) Reminder: "Running a backfill" just means copying all the data from one Data Source to another Data Source. There are different ways to move data in Tinybird: ### Using Copy Pipes [¶](https://www.tinybird.co/docs/about:blank#using-copy-pipes) A Copy Pipe is a Pipe used to copy data from one Data Source to another Data Source. This method is useful for one-time moves of data or scheduled executions (for example, every day at 00:00), but it's not recommended if you want to keep the data in sync between two Data Sources. In the context of a backfill, you could use the following Pipe to copy the data from one Data Source to another. (Later, you will explain why you need the `timestamp BETWEEN {{DateTime(start_backfill_timestamp)}} AND {{DateTime(end_backfill_timestamp)}}` condition). ##### backfill_data.pipe file NODE node SQL > % SELECT * FROM analytics_events WHERE timestamp BETWEEN {{DateTime(start_backfill_timestamp)}} AND {{DateTime(end_backfill_timestamp)}} TYPE COPY TARGET_DATASOURCE analytics_events_1 Once deployed, you would need to run the following command to execute the copy: ##### Command to run the Copy Pipe with the backfill_timestamp parameter tb pipe copy run backfill_data --param start_backfill_timestamp='1970-01-01 00:00:00' --param end_backfill_timestamp='2024-01-31 00:00:00' --wait --yes You can read more about it in our [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) docs. ### Using Materialized Views [¶](https://www.tinybird.co/docs/about:blank#using-materialized-views) A Materialized View is a Pipe that will materialize the data from one Data Source to another Data Source. This method is useful to keep the data in sync between two Data Sources. ##### sync_data.pipe file NODE node SQL > % SELECT * FROM analytics_events WHERE timestamp > '2024-01-31 00:00:00' TYPE materialized DATASOURCE analytics_events_1 By default, a Materialized view will only materialize the new incoming data; it won't process the old data. It can be forced by using `tb pipe populate` command using the CLI, but be careful as this can lead to duplicate or loss of data as explained in the previous section. Combining both methods, you will see how you can start syncing both Data Sources and start backfilling data. ## Scenarios for backfill strategies [¶](https://www.tinybird.co/docs/about:blank#scenarios-for-backfill-strategies) Depending on your use case and ingestion pattern, there are different recommended strategies to backfilling data in Tinybird. The complexity of this migration depends on several factors, notably the presence of streaming ingestion. The most common scenarios are: Tinybird is actively improving this workflow. Reach out to Tinybird support ( [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) ) if you have any questions. - ** Scenario 1: I'm not in Production** . - ** Scenario 2: Full replacement every few hours** . - ** Scenario 3: Streaming ingestion WITH incremental timestamp column** . - ** Scenario 4: Streaming ingestion WITH NOT incremental timestamp column** . ### Scenario 1: I'm not in Production [¶](https://www.tinybird.co/docs/about:blank#scenario-1-im-not-in-production) If you aren't in production or the data from that Data Source isn't being used and **you can accept losing data** , you can opt-in by create a new Data Source and start using it right away. Alternatively you can remove and re-create the original Data Source using a custom deployment. Once you start to append data to the Data Source, you will start seeing data in the new Data Source. ### Scenario 2: Full replacement every few hours [¶](https://www.tinybird.co/docs/about:blank#scenario-2-full-replacement-every-few-hours) If you are running a full replacement every few hours, you can create a Materialized View the two Data Sources. To sync the data between the two Data Sources, you will use a Materialized Pipe (MV) that will materialize the data from the old Data Source to the new one. Something like this: ##### Materialize data from old to new Data Source NODE migration_node SQL > SELECT * FROM analytics_events TYPE materialized DATASOURCE analytics_events_1 You would deploy this new Pipe along with the modified Data Source. Once you deploy the Pull Request, you will have this Materialized Pipe along with the new Data Source and the Pipe will materialize the data from the old Data Source. At this point, you would just need to wait until the full replacement is executed, the new Data Source will have all the data, after that you can create a new Pull Request to connect the new Data Source to the rest of your Pipe Endpoints. ### Scenario 3: Streaming ingestion WITH incremental timestamp column [¶](https://www.tinybird.co/docs/about:blank#scenario-3-streaming-ingestion-with-incremental-timestamp-column) If you have streaming ingestion using the [events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) with a huge ingest rate, you can use the following strategy to not be impacted by [the backfilling challenge with real-time data](https://www.tinybird.co/docs/about:blank#the-challenge-of-backfilling-real-time-data). To use this strategy successfully, you must have an incremental timestamp column with the same time zone in your Data Source. In our example, you have the `timestamp` column. First, create a new Pipe that will materialize the data from old Data Source to the new one, **but filter by a future timestamp** . For example, if you are deploying the Pull Request on `2024-02-02 13:00:00` , you can use `timestamp > '2024-02-02 13:30:00'`. ##### sync_data.pipe file NODE node SQL > SELECT * FROM analytics_events WHERE timestamp > '2024-02-02 13:30:00' TYPE materialized DATASOURCE analytics_events_1 Tinybird is using the `timestamp > '2024-02-02 13:30:00'` condition to only materialize data that is newer than the `2024-02-02 13:30:00` timestamp. Then, create a Copy Pipe with the same SQL statement, but instead of filtering by a specific future timestamp, you will use two parameters to filter by a timestamp range. This allows us to have better control of the backfilling process. For example, if you are moving very large amounts of data, split the backfilling process into different batches to avoid overloading the system. ##### backfill_data.pipe file NODE node SQL > % SELECT * FROM analytics_events WHERE timestamp BETWEEN {{DateTime(start_backfill_timestamp)}} AND {{DateTime(end_backfill_timestamp)}} TYPE COPY TARGET_DATASOURCE analytics_events_1 Once you have these changes in code, create a Pull Request and the CI Workflow will generate a new Branch. #### CI workflow [¶](https://www.tinybird.co/docs/about:blank#ci-workflow) Once the CI Workflow has finished successfully, a new Branch will be created. For the following steps, use the CLI. If you don't have it installed, you can follow the [CLI installation docs](https://www.tinybird.co/docs/docs/cli/install). First, you should be able to authenticate in the Branch by copying the Token from the Branch or using these commands: ##### Authenticate in the Branch # You can use `tb auth -i` to authenticate in the branch tb auth -i # Or you can switch to the branch if you are already authenticated tb branch ls # By default, the CI Workflow will create a branch following the pattern `tmp_ci-`. tb branch use It's best to run the Copy Pipe outside of the CI Workflow. As you don't have continuous ingestion in the Branch, don't wait for the future filter timestamp. Instead, run directly the Copy Pipe to backfill the data by running the following command: ##### Run the Copy Pipe tb pipe copy run backfill_data --param start_backfill_timestamp='1970-01-01 00:00:00' --param end_backfill_timestamp='2024-02-02 13:30:00' --wait --yes After the Copy Pipe has finished, you will have all the data in the new Data Source. You can compare the number of rows from both Data Sources with the following commands: ##### Compare the number of rows tb sql "SELECT count() FROM analytics_events" tb sql "SELECT count() FROM analytics_events_1" #### CD workflow [¶](https://www.tinybird.co/docs/about:blank#cd-workflow) Now that you have tested the backfilling process in the Branch, you can merge the Pull Request and the CD Workflow, the operation will be exactly the same as in the Branch: first deploy the resources then run the data operations either manually (recommended) or automate it with a custom deployment. **You should verify that have deployed the new Data Source before the timestamp you have used in the Materialized Pipe. Otherwise, you will be missing data in the new Data Source**. For example, if you have used `timestamp > '2024-02-02 13:30:00'` in the Materialized Pipe, you should verify that you have deployed before `2024-02-02 13:30:00`. If you have deployed after `2024-02-02 13:30:00` , you will need to remove the Data Source and start again the process using a different timestamp. At `2024-02-02 13:30:00` , yuo will start seeing data in the new Data Source, that's when you will execute the same process you have done in the CI Workflow to backfill the data. First, you will need to authenticate in the Workspace by running the following command: ##### Authenticate in the Workspace tb auth -i Then, you will need to run the Copy Pipe to backfill the data by running the following command: ##### Run the Copy Pipe tb pipe copy run backfill_data --param start_backfill_timestamp='1970-01-01 00:00:00' --param end_backfill_timestamp='2024-02-02 13:30:00' --wait --yes If the Copy Pipe fails, you can re-run the same command without duplicating data. **The Copy Pipe will only copy the data if the process is successful.** If you get any error like `MEMORY LIMIT` , you can also run the Copy Pipe in batches. For example, you could run the Copy Pipe with a timestamp range of 1 hour, 1 day, 1 week, depending on the amount of data you are moving. Once the Copy Pipe has finished, you will have all the data in the new Data Source. You can compare the number of rows by running the following command: ##### Compare the number of rows tb sql "SELECT count() FROM analytics_events" tb sql "SELECT count() FROM analytics_events" At this point, you should have the same number of rows in both places and you can connect the new Data Source with the rest of the Dataflow. Finally, you have the Data Source with the new schema and all the data migrated from the previous one. The Data Source is receiving real-time data directly and now the next step is to remove the Materialized Pipe and Copy Pipe you have used to backfill the data. To do that, you would just need to create a new Pull Request and remove ( `git rm` ) the Materialized Pipe and Copy Pipe you have used to backfill the data. Once, the Pull Request is merged, the resources will be automatically removed, you can double check this operation while in CI. ### Scenario 4: Streaming ingestion WITH NOT incremental timestamp column [¶](https://www.tinybird.co/docs/about:blank#scenario-4-streaming-ingestion-with-not-incremental-timestamp-column) If you have streaming ingestion, but you don't have an incremental timestamp column, you can use one of the following strategies to backfill data in Tinybird. Reach out to Tinybird support ( [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) ) if you have any questions or you aren't sure how to proceed. - ** Strategy 1** : Run a populate, but be aware that you may be impacted by[ the previously-mentioned challenges of backfilling real-time data](https://www.tinybird.co/docs/about:blank#the-challenge-of-backfilling-real-time-data) . - ** Strategy 2** : Move ingestion to the new Data Source until you finish backfilling data,** but the data in your old Data Source will be outdated until the new Data Source is fully in sync** . #### Strategy 1: Run a populate [¶](https://www.tinybird.co/docs/about:blank#strategy-1-run-a-populate) Before following this strategy, you should be aware of the [backfilling challenge with real-time data](https://www.tinybird.co/docs/about:blank#the-challenge-of-backfilling-real-time-data). Consider that the use case is the same as the previous one, but you don't have an incremental timestamp column. You can't rely on the `timestamp` column to filter the data as it's not incremental. First, create a Materialized Pipe that will materialize the data from the old Data Source to the new one. ##### backfill_data.pipe file NODE migrating_node SQL > SELECT * FROM analytics_events TYPE materialized DATASOURCE analytics_events_1 To run the backfill, you will use the `tb pipe populate` command. This command will materialize the data from the old Data Source to the new one and as you don't need to wait until a future timestamp, you can run it inside the CI/CD Workflow. You can create a custom deployment using `VERSION=1.0.0` in the `.tinyenv` file and placing the custom deployment script in the `deploy/1.0.0` folder: ##### Scripts generated inside the deploy folder deploy/1.0.0 ├── deploy.sh ## This is the script that will be executed during the deployment You will need to modify the `deploy.sh` script to run the `tb pipe populate` command: ##### deploy.sh script #!/bin/bash # This script will be executed after the deployment # You can use it to run any command after the deployment # Run the populate Pipe tb pipe populate backfill_data --node migrating_node --wait Once you have these changes in the code, you will create a Pull Request and the CI Workflow will generate a new Branch with the new Data Source. Now, you should verify that everything is working as expected as you did in the previous section. # You can use `tb auth -i` to authenticate in the branch tb auth -i # Or you can switch to the branch if you are already authenticated tb branch ls # By default, the CI Workflow will create a branch following the pattern `tmp_ci-`. tb branch use # Also, you could compare the number of rowsby running the following command: tb sql "SELECT count() FROM analytics_events" tb sql "SELECT count() FROM analytics_events_1" Once you have verified that everything is working as expected, merge the Pull Request and the CD Workflow will generate a new Data Source in the Main Branch. Once the CD Workflow has finished successfully, you verify the same way as you did in the Branch. #### Strategy 2: Move ingestion to the new Data Source [¶](https://www.tinybird.co/docs/about:blank#strategy-2-move-ingestion-to-the-new-data-source) Consider that the use case is the same as the previous one. You don't have an incremental timestamp column. You can't rely on the `timestamp` column to filter the data as it's not incremental and you don't want to run a populate as you might be impacted by [the backfilling challenge with real-time data](https://www.tinybird.co/docs/about:blank#the-challenge-of-backfilling-real-time-data). In this case, you can move the ingestion to the new Data Source until you finish backfilling data. First, create a Copy Pipe that will copy the data from the old Data Source to the new one. ##### backfill_data.pipe file NODE migrate_data SQL > SELECT * FROM analytics_events TYPE COPY TARGET_DATASOURCE analytics_events_1 You could also parametrize the Copy Pipe to filter by a parameter. This allows you to have better control of the backfilling process. Once you have these changes in our code, create a Pull Request and the CI Workflow will generate a new Branch with the new Data Source and the Copy Pipe. Now, run the Copy Pipe to backfill the data. To do that, authenticate in the branch by either copying the Token from the branch. ##### Authenticate in the branch # You can use `tb auth -i` to authenticate in the branch tb auth -i # Or you can switch to the branch if you are already authenticated tb branch ls # By default, the CI Workflow will create a branch following the pattern `tmp_ci-`. tb branch use # Once you have authenticated in the branch, you can run the Copy Pipe by running the following command: tb pipe copy run backfill_data --node migrate_data --wait --yes Once the Copy Pipe has finished, you will have all the data in the new Data Source. As you are likely not ingesting data into your Branch, both numbers should match. ##### Compare the number of rows tb sql "SELECT count() FROM analytics_events" tb sql "SELECT count() FROM analytics_events_1" Now, merge the Pull Request and the CD Workflow will generate the new resources in the main Branch. At this point, you should modify the ingestion to start ingesting data into the new Data Source. **Keep in mind that while you are ingesting data into the new Data Source, you will stop ingesting data into the old Data Source**. Once all the ingestion is pointing to the new Data Source, you should verify that new data is being ingested into the new Data Source and nothing is being ingested into the old one. To do that you could query the Data Source directly or the Service Data Source [tinybird.datasources_ops_log](https://www.tinybird.co/docs/docs/monitoring/service-datasources). At this point, you can start backfilling data by running the Copy Pipe. To do that, you would need to run the following command: ##### Run the Copy Pipe tb pipe copy run backfill_data --node migrate_data --wait --yes There are sometimes when the Data Source you are modifying has downstream dependencies, in that case when creating the new version of the Data Source you need to make sure that you create new version of all downstream dependencies to avoid connecting two different Data Sources receiving data to the same part of the Dataflow hence duplicating data. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) If you're familiar with backfilling strategies, check out the [Deployment Strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deployment-strategies) docs. --- URL: https://www.tinybird.co/docs/work-with-data/strategies/backfill-from-external Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Backfill from external sources · Tinybird Docs" theme-color: "#171612" description: "Best practices for bringing your data into Tinybird from external sources." --- # Backfill data into Tinybird from external sources [¶](https://www.tinybird.co/docs/about:blank#backfill-data-into-tinybird-from-external-sources) You can efficiently backfill data into Tinybird from external systems, or migrate data between Tinybird clusters, by following these best practices. ## Use Parquet for optimal ingestion [¶](https://www.tinybird.co/docs/about:blank#use-parquet-for-optimal-ingestion) The most efficient way to ingest data into Tinybird from external sources is to use the Parquet format. See [NDJSON and Parquet files](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/datasource-api#ndjson-and-parquet-files). When preparing your data, generate Parquet files sized between 1 GB and 5 GB. This file size range ensures optimal throughput. ## Use the S3 Connector [¶](https://www.tinybird.co/docs/about:blank#use-the-s3-connector) Using the [S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3) is the preferred way to ingest the data for backfills from external sources due to its ease of use. First, prepare the Parquet files by uploading them to your S3 bucket. Then, set up the S3 Connector in Tinybird to automatically ingest files from that bucket. This approach streamlines the process and manages rate limits automatically, reducing manual workload. ## CLI or API as an alternative [¶](https://www.tinybird.co/docs/about:blank#cli-or-api-as-an-alternative) If using S3 isn't an option, you can use the Tinybird CLI or the Tinybird API: - [ Tinybird CLI](https://www.tinybird.co/docs/docs/cli) : The CLI simplifies ingestion and automatically handles rate limits, making it a reliable alternative. - [ Tinybird API](https://www.tinybird.co/docs/docs/api-reference/datasource-api#post--v0-datasources-?) : When using the API directly, manage[ rate limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#ingestion-limits-api) manually to prevent throttling. ## Migrate data between Tinybird clusters [¶](https://www.tinybird.co/docs/about:blank#migrate-data-between-tinybird-clusters) If you need to migrate data from a shared Tinybird cluster to a new Tinybird cluster: 1. Use[ Tinybird Sinks](https://www.tinybird.co/docs/docs/publish/sinks/s3-sink) to export data as Parquet files. 2. Configure the sink to export the Parquet files to an S3 bucket. 3. Use the[ S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3) in the new Tinybird cluster to ingest the data efficiently. This method leverages Tinybird tools to ensure a smooth migration while preserving the efficiency of the backfill process. ## Ingest only to the landing layer [¶](https://www.tinybird.co/docs/about:blank#ingest-only-to-the-landing-layer) When backfilling data from external sources: - Ingest raw data directly into the landing layer to keep ingestion and transformation steps separate. - Separate downstream[ materializations](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) : Avoid triggering downstream materializations during this process. If you already have materializations configured, consider unlinking them temporarily to simplify the ingestion step. ## Maximize throughput [¶](https://www.tinybird.co/docs/about:blank#maximize-throughput) Monitor the throughput during the backfill process. If you find it insufficient or notice underutilization of your dedicated cluster: - [ Monitor jobs](https://www.tinybird.co/docs/docs/monitoring/jobs) and also your[ dedicated cluster](https://www.tinybird.co/docs/docs/get-started/administration/organizations#dedicated-infrastructure-monitoring) . - Reach out to Tinybird support. Support can help adjust rate limits or scale components, if allowed by your configuration. ## Address memory errors [¶](https://www.tinybird.co/docs/about:blank#address-memory-errors) If memory errors occur during the ingestion or migration process: - Consider[ optimizing](https://www.tinybird.co/docs/docs/work-with-data/optimization) the use case, for example by adjusting transformations, splitting large files into smaller chunks, and so on. - If optimizations don't resolve the issue, evaluate the need for a cluster upgrade to meet resource demands. ## Summary [¶](https://www.tinybird.co/docs/about:blank#summary) For backfilling or migrating data into Tinybird: 1. Use Parquet files for efficient ingestion. 2. Leverage the S3 Connector for simplicity and rate limit management. 3. To migrate between Tinybird clusters, use Sinks to export data to S3, and reingest the Sink into the new cluster. 4. Target the landing layer to separate ingestion from downstream processing. 5. Monitor performance and contact support if you need adjustments to rate limits or cluster capacity. --- URL: https://www.tinybird.co/docs/work-with-data/query/sql-best-practices Last update: 2025-01-22T09:19:22.000Z Content: --- title: "SQL best practices · Tinybird Docs" theme-color: "#171612" description: "Learn the best practices when working with a huge amount of data." --- # Best practices for SQL queries [¶](https://www.tinybird.co/docs/about:blank#best-practices-for-sql-queries) When you're trying to process significant amounts of data, following best practices help you create faster and more robust queries. Follow these principles when writing queries meant for Tinybird: 1. The best data is the data you don't write. 2. The second best data is the one you don't read. The less data you read, the better. 3. Sequential reads are much faster. 4. The less data you process after read, the better. 5. Perform complex operations later in the processing pipeline. The following sections analyze how performance improves after implementing each principle. To follow the examples, download the [NYC Taxi Trip](https://storage.googleapis.com/tinybird-demo/yellow_trip_data_2018/yellow_tripdata_2018-01.csv) and import it using a Data Source. See [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources). ## The best data is the one you don't write [¶](https://www.tinybird.co/docs/about:blank#the-best-data-is-the-one-you-dont-write) Don't save data that you don't need, as it impacts memory usage, causing queries to take more time. ## The second best data is the one you don't read [¶](https://www.tinybird.co/docs/about:blank#the-second-best-data-is-the-one-you-dont-read) To avoid reading data that you don't need, apply filters as soon as possible. For example, consider a list of the trips whose distance is greater than 10 miles and that took place between `2017-01-31 14:00:00` and `2017-01-31 15:00:00` . Additionally, you want to retrieve the trips ordered by date. The following examples show the difference between applying the filters at the end or at the beginning of the Pipe. The first approach orders all the data by date: ##### node rule2_data_read_NOT_OK SELECT * FROM nyc_taxi ORDER BY tpep_pickup_datetime ASC After the data is sorted, you can filter. This approach takes around 30 to 60 ms after adding the time of both steps. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fbest-practices-faster-sql-queries-1.png&w=3840&q=75) Compare the number of scanned rows (139.26k) and the size of data (10.31MB) to the number of scanned rows (24.58k) and the size of data (1.82MB): you only need to scan 24.58k rows. Both values directly impact the query execution time and also affect other queries you might be running at the same time. Bandwidth is also a factor you need to keep in mind. The following example shows what happens if the filter is applied before the sorting: ##### node rule2_data_read_OK SELECT * FROM nyc_taxi WHERE (trip_distance > 10) AND ((tpep_pickup_datetime >= '2017-01-31 14:00:00') AND (tpep_pickup_datetime <= '2017-01-31 15:00:00')) ORDER BY tpep_pickup_datetime ASC If the filter is applied before the sorting, it takes only 1 to 10 ms. The size of the data read is 1.82 MB, while the number of rows read is 24.58k: they're much smaller figures than the ones in the first example. This significant difference happens because in the first approach you are sorting all the data available, even the data that you don't need for your query, while in the second approach you are sorting only the rows you need. As filtering is the fastest operation, always filter first. ## Sequential reads are much faster [¶](https://www.tinybird.co/docs/about:blank#sequential-reads-are-much-faster) To carry out sequential reads, define indexes correctly. Indexes should be defined based on the queries that are going to be run. The following example simulates a case by ordering the data based on the columns. For example, if you want to query the data by date, compare what happens when the data is sorted by date to when it's sorted by any other column. The first approach sorts the data by another column, `passenger_count`: ##### node rule3_sequential_read_NOT_OK SELECT * FROM nyc_taxi ORDER BY passenger_count ASC After you've sorted the data by `passenger_count` , filter it by date: <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fbest-practices-faster-sql-queries-2.png&w=3840&q=75) This approach takes around 5-10 ms, the number of scanned rows is 26.73k, and the size of data is 1.98 MB. For the second approach, sort the data by date: ##### node rule3_ordered_by_date_OK SELECT * FROM nyc_taxi ORDER BY tpep_pickup_datetime ASC After it's sorted by date, filter it: <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fbest-practices-faster-sql-queries-3.png&w=3840&q=75) When the data is sorted by date and the query uses date for filtering, it takes 1-2 ms, the number of scanned rows is 10.35k and the size of data is 765.53KB. The more data you have, the greater the difference between both approaches. When dealing with significant amounts of data, sequential reads can be much faster. Therefore, define the indexes taking into account the queries that you want to make. ## The less data you process after read, the better [¶](https://www.tinybird.co/docs/about:blank#the-less-data-you-process-after-read-the-better) If you only need two columns, only retrieve those. Consider a case where you only need the following: `vendorid`, `tpep_pickup_datetime` , and `trip_distance`. When retrieving all the columns instead of the previous three, you need around 140-180 ms and the size of data is 718.55MB: ##### NODE RULE4_LESS_DATA_NOT_OK SELECT * FROM ( SELECT * FROM nyc_taxi order by tpep_dropoff_datetime ) When retrieving only the columns you need, it takes around 35-60 ms: ##### node rule4_less_data_OK SELECT * FROM ( SELECT vendorid, tpep_pickup_datetime, trip_distance FROM nyc_taxi order by tpep_dropoff_datetime ) With analytical databases, not retrieving unnecessary columns make queries much more performant and efficient. Process only the data you need. ## Perform complex operations later in the processing pipeline [¶](https://www.tinybird.co/docs/about:blank#perform-complex-operations-later-in-the-processing-pipeline) Perform complex operations, such as joins or aggregations, as late as possible in the processing pipeline. As you filter all the data at the beginning, the number of rows at the end of the pipeline is lower and, therefore, the cost of executing complex operations is also lower. Using the example dataset, aggregate the data: ##### node rule5_complex_operation_NOT_OK SELECT vendorid, pulocationid, count(*) FROM nyc_taxi GROUP BY vendorid, pulocationid Apply the filter: <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fbest-practices-faster-sql-queries-4.png&w=3840&q=75) If you apply the filter after aggregating the data, it takes around 50-70 ms to retrieve the data, the number of scanned rows is 9.71m, and the size of data is 77.68 MB. If you filter before aggregating the data: ##### node rule5_complex_operation_OK SELECT vendorid, pulocationid, count(*) FROM nyc_taxi WHERE vendorid < 10 GROUP BY vendorid, pulocationid The query takes only 20-40 ms, although the number of scanned rows and the size of data is the same as in the previous approach. ## Additional guidance [¶](https://www.tinybird.co/docs/about:blank#additional-guidance) Follow these additional recommendations when creating SQL queries. ### Avoid full scans [¶](https://www.tinybird.co/docs/about:blank#avoid-full-scans) The less data you read in your queries, the faster they are. There are different strategies you can follow to avoid reading all the data in a Data Source, or doing a full scan, in your queries: - Always filter first. - Use indices by setting a proper `ENGINE_SORTING_KEY` in the Data Source. - The column names present in the `ENGINE_SORTING_KEY` should be the ones you use for filtering in the `WHERE` clause. You don't need to sort by all the columns you use for filtering, only the ones to filter first. - The order of the columns in the `ENGINE_SORTING_KEY` is important: from left to right ordered by relevance. The columns that matter the most for filtering and have less cardinality should go first. Consider the following Data Source, which is sorted by date by default: ##### Data Source: data_source_sorted_by_date SCHEMA > `id` Int64, `amount` Int64, `date` DateTime ENGINE "MergeTree" ENGINE_SORTING_KEY "id, date" The following query is slower because it filters data using a column other than the ones defined in the `ENGINE_SORTING_KEY` instruction: ##### Not filtering by any column present in the ENGINE_SORTING_KEY SELECT * FROM data_source_sorted_by_date WHERE amount > 30 The following query is faster because it filters data using a column defined in the `ENGINE_SORTING_KEY` instruction: ##### Filtering first by columns present in the ENGINE_SORTING_KEY SELECT * FROM data_source_sorted_by_date WHERE id = 135246 AND date > now() - INTERVAL 3 DAY AND amount > 30 ## Avoid big joins [¶](https://www.tinybird.co/docs/about:blank#avoid-big-joins) When doing a `JOIN` , the data in the Data Source on the right side loads in memory to perform the operation. `JOIN` s over tables of more than 1 million rows might lead to `MEMORY_LIMIT` errors when used in Materialized Views, affecting ingestion. Avoid joining big Data Sources by filtering the data in the Data Source on the right side. For example, the following pattern is less efficient because it's joining a Data Source with too many rows: ##### Doing a JOIN with a Data Source with too many rows SELECT left.id AS id, left.date AS day, right.response_id AS response_id FROM left_data_source AS left INNER JOIN big_right_data_source AS right ON left.id = right.id The following query is faster and more efficient because if prefilters the Data Source before the `JOIN`: ##### Prefilter the joined Data Source for better performance SELECT left.id AS id, left.date AS day, right.response_id AS response_id FROM left_data_source AS left INNER JOIN ( SELECT id, response_id FROM big_right_data_source WHERE id IN (SELECT id FROM left_data_source) ) AS right ON left.id = right.id ## Memory issues [¶](https://www.tinybird.co/docs/about:blank#memory-issues) Sometimes, you might reach the memory limit when running a query. This is usually because of the following reasons: - Lot of columns are used: try to reduce the amount of columns used in the query. As this isn't always possible, try to change data types or merge some columns. - A cross `JOIN` or some operation that generates a lot of rows: it might happen if the cross `JOIN` is done with two Data Sources with a large amount of rows. Try to rewrite the query to avoid the cross `JOIN` . - A massive `GROUP BY` : try to filter out rows before executing the `GROUP BY` . If you are getting a memory error while populating a Materialized View, the solutions are the same. Consider that the populate process runs in 1 million rows chunks, so if you hit memory limits, the cause might be one of the following: 1. There is a `JOIN` and the right table is large. 2. There is an `ARRAY JOIN` with a huge array that make the number of rows significantly increase. To check if a populate process could break, create a Pipe with the same query as the Materialized View and replace the source table with a node that gets 1 million rows from the source table. The following example shows an unoptimized Materialized View Pipe: ##### original Materialized View Pipe SQL NODE materialized SQL > select date, count() c from source_table group by date The following query shows the transformed Pipe: ##### Transformed Pipe to check how the Materialized View would process the data NODE limited SQL > select * from source_table limit 1000000 NODE materialized SQL > select date, count() c from limited group by date ## Nested aggregate functions [¶](https://www.tinybird.co/docs/about:blank#nested-aggregate-functions) You can't nest aggregate functions or use an alias of an aggregate function that's used in another aggregate function. For example, the following query causes an error due to a nested aggregate function: ##### Error on using nested aggregate function SELECT max(avg(number)) as max_avg_number FROM my_datasource The following query causes an error due to a nested aggregate with alias: ##### Error on using nested aggregate function with alias SELECT avg(number) avg_number, max(avg_number) max_avg_number FROM my_datasource Instead of using nested aggregate functions, use a subquery. The following example shows how to use aggregate functions in a subquery: ##### Using aggregate functions in a subquery SELECT avg_number as number, max_number FROM ( SELECT avg(number) as avg_number, max(number) as max_number FROM numbers(10) ) The following example shows how to nest aggregate functions using a subquery: ##### Nesting aggregate functions using a subquery SELECT max(avg_number) as number FROM ( SELECT avg(number) as avg_number, max(number) as max_number FROM numbers(10) ) ## Merge aggregate functions [¶](https://www.tinybird.co/docs/about:blank#merge-aggregate-functions) Columns with `AggregateFunction` types such as `count`, `avg` , and others precalculate their aggregated values using intermediate states. When you query those columns you have to add the `-Merge` combinator to the aggregate function to get the final aggregated results. Use `-Merge` aggregated states as late in the pipeline as possible. Intermediate states are stored in binary format, which explains why you might see special characters when selecting columns with the `AggregateFunction` type. For example, consider the following query: ##### Getting 'result' as aggregate function SELECT result FROM my_datasource The result contains special characters: | AggregateFunction(count) | | --- | | @33M@ | | �o�@ | When selecting columns with the `AggregateFunction` type use `-Merge` the intermediate states to get the aggregated result for that column. This operation might compute several rows, so use `-Merge` as late in the pipeline as possible. Consider the following query: ##### Getting 'result' as UInt64 -- Getting the 'result' column aggregated using countMerge. Values are UInt64 SELECT countMerge(result) as result FROM my_datasource The result is the following: | UInt64 | | --- | | 1646597 | --- URL: https://www.tinybird.co/docs/work-with-data/query/query-parameters Last update: 2025-02-04T14:06:01.000Z Content: --- title: "Using query parameters · Tinybird Docs" theme-color: "#171612" description: "Query parameters are great for any value of the query that you might want control dynamically from your applications." --- # Using query parameters [¶](https://www.tinybird.co/docs/about:blank#using-query-parameters) Query parameters are great for any value of the query that you might want control dynamically from your applications. For example, you can get your API Endpoint to answer different questions by passing a different value as query parameter. Using dynamic parameters means you can do things like: - Filtering as part of a `WHERE` clause. - Changing the number of results as part of a `LIMIT` clause. - Sorting order as part of an `ORDER BY` clause. - Selecting specific columns for `ORDER BY` or `GROUP BY` clauses. ## Define dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#define-dynamic-parameters) To make a query dynamic, start the query with a `%` character. That signals the engine that it needs to parse potential parameters. Tinybird automatically inserts the `%` character in the first line when you add a parameter to a Node. After you have created a dynamic query, you can define parameters by using the following pattern `{{([,, description=<"This is a description">, required=])}}` . For example: ##### Simple select clause using dynamic parameters % SELECT * FROM TR LIMIT {{Int32(lim, 10, description="Limit the number of rows in the response", required=False)}} The previous query returns 10 results by default, or however many are specified on the `lim` parameter when requesting data from that API Endpoint. ## Use Pipes API Endpoints with dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#use-pipes-api-endpoints-with-dynamic-parameters) When using a Data Pipes API Endpoint that uses parameters, pass in the desired parameters. Using the previous example where `lim` sets the amount of maximum rows you want to get, the request would look like this: ##### Using a data Pipes API Endpoint containing dynamic parameters curl -d https://api.tinybird.co/v0/pipes/tr_pipe?lim=20&token=.... You can specify parameters in more than one node in a Data Pipe. When invoking the API Endpoint through its URL, the passed parameters are included in the request. You can't use query parameters in nodes that are published as [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views). ## Leverage dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#leverage-dynamic-parameters) As well as using dynamic parameters in your API Endpoints, you can then leverage them further downstream for monitoring purposes. When you pass a parameter to your queries, you can build Pipes to reference the parameters and query the Service Data Sources with them, even if you don't use them in the API Endpoints themselves. Review the [Service Data Sources docs](https://www.tinybird.co/docs/docs/monitoring/service-datasources) to use the available options. For example, using the `user_agent` column on `pipe_stats_rt` shows which user agent made the request. Pass any additional things you need as a parameter to improve visibility and avoid, or get insights into, incidents and Workspace performance. This process helps you forward things like user agent or others from any app requests all the way through to Tinybird, and track if the request was done in the app and details like which device was used. ##### Example query to the pipe_stats_rt Service Data Source leveraging a passed 'referrer' parameter SELECT toStartOfMinute(start_datetime) as date, count(), parameters['referrer'] FROM tinybird.pipes_stats_rt WHERE ( pipe_id = '' and status_code != 429) or pipe_name = '' and status_code != 429) ) and start_datetime > now() - interval - 1 hour GROUP BY date, parameters['referrer'] ORDER BY count() DESC, date DESC ## Available data types for dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#available-data-types-for-dynamic-parameters) You can use the following data types for dynamic parameters: - `Boolean` : Accepts `True` and `False` as values, as well as strings like `'TRUE'` , `'FALSE'` , `'true'` , `'false'` , `'1'` , or `'0'` , or the integers `1` and `0` . - `String` : For any string values. - `DateTime64` , `DateTime` and `Date` : Accepts values like `YYYY-MM-DD HH:MM:SS.MMM` , `YYYY-MM-DD HH:MM:SS` and `YYYYMMDD` respectively. - `Float32` and `Float64` : Accepts floating point numbers of either 32 or 64 bit precision. - `Int` or `Integer` : Accepts integer numbers of any precision. - `Int8` , `Int16` , `Int32` , `Int64` , `Int128` , `Int256` and `UInt8` , `UInt16` , `UInt32` , `UInt64` , `UInt128` , `UInt256` : Accepts signed or unsigned integer numbers of the specified precision. ### Use column parameters [¶](https://www.tinybird.co/docs/about:blank#use-column-parameters) You can use `column` to pass along column names of a defined type as parameters, like: ##### Using column dynamic parameters % SELECT * FROM TR ORDER BY {{column(order_by, 'timestamp')}} LIMIT {{Int32(lim, 10)}} Always define the `column` function's second argument, the one for the default value. The alternative for not defining the argument is to validate that the first argument is defined, but this only has an effect on the execution of the API Endpoint. A placeholder is used in the development of the Pipes. ##### Validate the column parameter when not defining a default value % SELECT * FROM TR {% if defined(order_by) %} ORDER BY {{column(order_by)}} {% end %} ### Pass arrays [¶](https://www.tinybird.co/docs/about:blank#pass-arrays) You can pass along a list of values with the `Array` function for parameters, like so: ##### Passing arrays as dynamic parameters % SELECT * FROM TR WHERE access_type IN {{Array(access_numbers, 'Int32', default='101,102,110')}} ## Send stringified JSON as parameter [¶](https://www.tinybird.co/docs/about:blank#send-stringified-json-as-parameter) Consider the following stringified JSON: "filters": [ { "operand": "date", "operator": "equals", "value": "2018-01-02" }, { "operand": "high", "operator": "greater_than", "value": "100" }, { "operand": "symbol", "operator": "in_list", "value": "AAPL,AMZN" } ] You can use the `JSON()` function to use `filters` as a query parameter. The following example shows to use the `filters` field from the JSON snippet with the stock_prices_1m sample dataset. % SELECT symbol, date, high FROM stock_prices_1m WHERE 1 {% if defined(filters) %} {% for item in JSON(filters, '[]') %} {% if item.get('operator', '') == 'equals' %} AND {{ column(item.get('operand', '')) }} == {{ item.get('value', '') }} {% elif item.get('operator') == 'greater_than' %} AND {{ column(item.get('operand', '')) }} > {{ item.get('value', '') }} {% elif item.get('operator') == 'in_list' %} AND {{ column(item.get('operand', '')) }} IN splitByChar(',',{{ item.get('value', '') }}) {% end %} {% end %} {% end %} When accessing the fields in a JSON object, use the following syntax: item.get('Field', 'Default value to avoid SQL errors'). ### Pagination [¶](https://www.tinybird.co/docs/about:blank#pagination) You paginate results by adding `LIMIT` and `OFFSET` clauses to your query. You can parameterize the values of these clauses, allowing you to pass pagination values as query parameters to your API Endpoint. Use the `LIMIT` clause to select only the first `n` rows of a query result. Use the `OFFSET` clause to skip `n` rows from the beginning of a query result. Together, you can dynamically chunk the results of a query up into pages. For example, the following query introduces two dynamic parameters `page_size` and `page` which lets you control the pagination of a query result using query parameters on the URL of an API Endpoint. ##### Paging results using dynamic parameters % SELECT * FROM TR LIMIT {{Int32(page_size, 100)}} OFFSET {{Int32(page, 0) * Int32(page_size, 100)}} You can also use pages to perform calculations such as `count()` . The following example counts the total number of pages: ##### Operation on a paginated endpoint % SELECT count() as total_rows, ceil(total_rows/{{Int32(page_size, 100)}}) pages FROM endpoint_to_paginate The addition of a `LIMIT` clause to a query also adds the `rows_before_limit_at_least` field to the response metadata. `rows_before_limit_at_least` is the lower bound on the number of rows returned by the query after transformations but before the limit was applied, and can be useful for response handling calculations. To get consistent pagination results, add an `ORDER BY` clause to your paginated queries. ## Advanced templating using dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#advanced-templating-using-dynamic-parameters) To build more complex queries, use flow control operators like `if`, `else` and `elif` in combination with the `defined()` function, which helps you to check if a parameter whether a parameter has been received and act accordingly. Tinybird's templating system is based on the [Tornado Python framework](https://github.com/tornadoweb/tornado) , and uses Python syntax. You must enclose control statements in curly brackets with percentages `{%..%}` as in the following example: ##### Advanced templating using dynamic parameters % SELECT toDate(start_datetime) as day, countIf(status_code < 400) requests, countIf(status_code >= 400) errors, avg(duration) avg_duration FROM log_events WHERE endsWith(user_email, {{String(email, 'gmail.com')}}) AND start_datetime >= {{DateTime(start_date, '2019-09-20 00:00:00')}} AND start_datetime <= {{DateTime(end_date, '2019-10-10 00:00:00')}} {% if method != 'All' %} AND method = {{String(method,'POST')}} {% end %} GROUP BY day ORDER BY day DESC ### Validate presence of a parameter [¶](https://www.tinybird.co/docs/about:blank#validate-presence-of-a-parameter) ##### Validate if a param is in the query % select * from table {% if defined(my_filter) %} where attr > {{Int32(my_filter)}} {% end %} When you call the API Endpoint with `/v0/pipes/:PIPE.json?my_filter=20` it applies the filter. ### Default parameter values and placeholders [¶](https://www.tinybird.co/docs/about:blank#default-parameter-values-and-placeholders) Following best practices, you should set default parameter values as follows: ##### Default parameter values % SELECT * FROM table WHERE attr > {{Int32(my_filter, 10)}} When you call the API Endpoint with `/v0/pipes/:PIPE.json` without setting any value to `my_filter` , it automatically applies the default value of 10. If you don't set a default value for a parameter, you should validate that the parameter is defined before using it in the query as explained previously. If you don't validate the parameter and it's not defined, the query might fail. Tinybird populates the parameter with a placeholder value based on the data type. For instance, numerical data types are populated with 0, strings with `__no_value__` , and date and timestamps with `2019-01-01` and `2019-01-01 00:00:00` respectively. You could try yourself with a query like this: ##### Get placeholder values % SELECT {{String(param)}} as placeholder_string, {{Int32(param)}} as placeholder_num, {{Boolean(param)}} as placeholder_bool, {{Float32(param)}} as placeholder_float, {{Date(param)}} as placeholder_date, {{DateTime(param)}} as placeholder_ts, {{Array(param)}} as placeholder_array This returns the following values: { "placeholder_string": "__no_value__", "placeholder_num": 0, "placeholder_bool": 0, "placeholder_float": 0, "placeholder_date": "2019-01-01", "placeholder_ts": "2019-01-01 00:00:00", "placeholder_array": ["__no_value__0","__no_value__1"] } ### Test dynamic parameters [¶](https://www.tinybird.co/docs/about:blank#test-dynamic-parameters) Any dynamic parameters you create appears in the UI. Select **Test new values** to open a test dialog populated with the default value of your parameters. The test dialog helps you test different Pipe values than the default ones without impacting production environments. Use the View API page to see API Endpoint metrics resulting from that specific combination of parameters. Close the dialog to bring the Pipe back to its default production state. When testing parameters, you can modify both the SQL code and the parameters. ### Cascade parameters [¶](https://www.tinybird.co/docs/about:blank#cascade-parameters) Parameters with the same name in different Pipes are cascaded down the dependency chain. For example, if you publish Pipe A with the parameter `foo` , and then Pipe B which uses Pipe A as a Data Source also with the parameter `foo` , then when you call the API Endpoint of Pipe B with `foo=bar` , the value of `foo` will be `bar` in both Pipes. ### Throw errors [¶](https://www.tinybird.co/docs/about:blank#throw-errors) The following example stops the API Endpoint processing and returns a 400 error: ##### Validate if a param is defined and throw an error if it's not defined % {% if not defined(my_filter) %} {{ error('my_filter (int32) query param is required') }} {% end %} select * from table where attr > {{Int32(my_filter)}} The `custom_error` function is an advanced version of `error` where you can customize the response and other aspects. The function gets an object as the first argument, which is sent as JSON, and the status_code as a second argument, which defaults to 400. ##### Validate if a param is defined and throw an error if it's not defined % {% if not defined(my_filter) %} {{ custom_error({'error_id': 10001, 'error': 'my_filter (int32) query param is required'}) }} {% end %} select * from table where attr > {{Int32(my_filter)}} ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) You can't use query parameters in nodes that are published as [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) , only as API Endpoints or in on-demand Copies or Sinks. You can use query parameters in scheduled Sinks and Copies, but must have a default. That default is used in the scheduled execution. The preview step fails if the default doesn't exist. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) Thanks to the magic of dynamic parameters, you can create flexible API Endpoints with ease, so you don't need to manage or test dozens of Pipes. Be sure you're familiar with the [5 rules for faster SQL queries](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices). --- URL: https://www.tinybird.co/docs/work-with-data/query/pipes Last update: 2025-01-20T11:46:12.000Z Content: --- title: "Pipes · Tinybird Docs" theme-color: "#171612" description: "Pipes help you to bring your SQL queries together to achieve a purpose, like publishing an API Endpoint. Learn all about Pipes here!" --- # Pipes [¶](https://www.tinybird.co/docs/about:blank#pipes) ## What is a Pipe? [¶](https://www.tinybird.co/docs/about:blank#what-is-a-pipe) A Pipe is a collection of one or more SQL queries (each query is called a [Node](https://www.tinybird.co/docs/about:blank#nodes) ). Tinybird represents Pipes using the icon. ## What should I use Pipes for? [¶](https://www.tinybird.co/docs/about:blank#what-should-i-use-pipes-for) Use Pipes to build features over your data. Write SQL that joins, aggregates, or otherwise transforms your data and publish the result. You have three options to publish the result of a Pipe: [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints), [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) , and [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes). A Pipe can only have a single output at one time. This means that you can't create a Materialized View and an API Endpoint from the same Pipe, at the same time. Once your Pipe is published as an API Endpoint, you can create [Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts) : Interactive, customizable charts of your data. ## Creating Pipes [¶](https://www.tinybird.co/docs/about:blank#creating-pipes) You can create as many Pipes in your Workspace as you need. ### Creating Pipes in the UI [¶](https://www.tinybird.co/docs/about:blank#creating-pipes-in-the-ui) To add a Pipe, click the Plus (+) icon in the left side navigation bar next to the Pipes section (see Mark 1 below). <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fconcepts-pipes-creating-a-pipe-ui-1.png&w=3840&q=75) At the top of your new Pipe, you can change the name & description. **Click on the name or description** to start entering text (see Mark 1 below). <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fconcepts-pipes-creating-a-pipe-ui-2.png&w=3840&q=75) In time, you might end up with a lot of Pipes. Tinybird doesn't yet offer a way to organize your Pipes into folders, but a quick and easy alternative is to group your Pipe names by name - like `mktg-` so all your marketing Pipes are together. Pipes are ordered alphabetically, and must always start with a letter (but you can use numbers, dots, and underscores in the rest of the name). ### Creating Pipes in the CLI [¶](https://www.tinybird.co/docs/about:blank#creating-pipes-in-the-cli) #### tb pipe generate [¶](https://www.tinybird.co/docs/about:blank#tb-pipe-generate) You can use `tb pipe generate` to generate a .pipe file. You must provide a name for the Pipe & a single SQL statement. This command will generate the necessary syntax for a single-Node Pipe inside the file. You can open the file in any text editor to change the name, description, query and add more nodes. Defining your Pipes in files allows you to version control your Tinybird resources with git. The SQL statement must be wrapped in quotes or the command will fail. tb pipe generate my_pipe_name "SELECT 1" Generating the .pipe file does **not** create the Pipe in Tinybird. When you are finished editing the file, you must push the Pipe to Tinybird. tb push my_pipe_name.pipe If you list your Pipes, you'll see that your Pipe exists in Tinybird. tb pipe ls ** Pipes: -------------------------------------------------------------------- | version | name | published date | nodes | -------------------------------------------------------------------- | | my_pipe_name | 2022-11-30 21:44:55 | 1 | -------------------------------------------------------------------- ## Format descriptions using Markdown [¶](https://www.tinybird.co/docs/about:blank#format-descriptions-using-markdown) It is possible to use Markdown syntax in Pipe description fields so you can add richer formatting. Here's an example using headings, bold, and external links: ## This is my first Pipe! You can use **Markdown** in descriptions. Add [links](https://www.tinybird.co) to other bits of info! This will be rendered in the UI like this: <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fconcepts-pipes-markdown-in-pipes-desc-1.png&w=3840&q=75) ## Nodes [¶](https://www.tinybird.co/docs/about:blank#nodes) ### What is a Node? [¶](https://www.tinybird.co/docs/about:blank#what-is-a-node) A node is a container for a single SQL `SELECT` statement. Nodes live within Pipes, and you can have many sequential nodes inside the same Pipe. A query in a node can read data from a [Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources) , other nodes inside the same Pipe, or from [API Endpoint](https://www.tinybird.co/docs/docs/publish/api-endpoints) nodes in other Pipes. ### What should I use nodes for? [¶](https://www.tinybird.co/docs/about:blank#what-should-i-use-nodes-for) Nodes allow you to break your query logic down into multiple smaller queries. You can then chain nodes together to build the logic incrementally. Each node can be developed & tested individually. This makes it much easier to build complex query logic in Tinybird as you avoid creating large monolithic queries with many sub-queries. --- URL: https://www.tinybird.co/docs/work-with-data/query/guides Content: --- title: "Querying data guides · Tinybird Docs" theme-color: "#171612" description: "Guides for querying data from Tinybird." --- # Querying data guides [¶](https://www.tinybird.co/docs/about:blank#querying-data-guides) Learn how to query data in Tinybird. Each guide provides detailed instructions on how to best query data in Tinybird. - [ Adapt Postgres queries](https://www.tinybird.co/docs/docs/work-with-data/query/guides/adapt-postgres-queries) - [ Build a lambda architecture](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-architecture) - [ Lambda use case: CDC](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-example-cdc) - [ Rollup aggregations](https://www.tinybird.co/docs/docs/work-with-data/query/guides/dynamic-aggregation) - [ Working with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time) --- URL: https://www.tinybird.co/docs/work-with-data/query/bi-connector Last update: 2025-02-13T13:12:54.000Z Content: --- title: "BI Connector · Tinybird Docs" theme-color: "#171612" description: "Learn about the BI Connector in Tinybird, including how to connect and best practices." --- # BI Connector [¶](https://www.tinybird.co/docs/about:blank#bi-connector) The BI Connector is a PostgreSQL-compatible interface to data in Tinybird. You can connect many of your favorite tools to Tinybird, such as Tableau, Apache SuperSet, or Grafana. All Data Sources and published Pipes created in Tinybird are available as standard PostgreSQL tables. You can query using standard PostgreSQL syntax and use any tool that implements the PostgreSQL protocol. Tinybird uses TLS 1.2 and higher for BI Connector connections. The BI Connector isn't active by default. Contact Tinybird support ( [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) ) to discuss if your use case is supported. After the connector is activated, Tinybird sends you all the connection details. ## Compatibility [¶](https://www.tinybird.co/docs/about:blank#compatibility) The BI Connector has been successfully tested on the following solutions: - DBeaver - PowerBI - Tableau - Grafana - Metabase - Superset - Klipfolio - Anodot ## Connect to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-to-tinybird) You can connect and access your database as you would with regular PostgreSQL. You can use your command-line tool, any graphical interface such as DBbeaver, or any other SQL client such as, for example, your favorite BI tool. For example: psql -U --host bi.us-east.tinybird.co --port 5432 -d After connecting to your database, in the public schema, if you list all the available views you get all the available Data Sources and endpoints. If you're using the PostgreSQL command line you can type `\dv` to get the list of views. If you list all the tables, a table per Data Source appears. The name of the table is the identifier of the Data Source. Don't use those IDs. Instead, find your tables with their user-friendly names in the views space. ## Best practices [¶](https://www.tinybird.co/docs/about:blank#best-practices) ### 1. Avoid queries on exposed Pipes [¶](https://www.tinybird.co/docs/about:blank#1-avoid-queries-on-exposed-pipes) Both your Pipes and Data Sources are exposed in the BI Connector as tables. Query against Data Sources tables only, and avoid using any Pipes tables. In general, queries on Data Sources tables are much faster & easier to optimize. ### 2. Use Materialized Views [¶](https://www.tinybird.co/docs/about:blank#2-use-materialized-views) With Tinybird you can build Pipes that materialize the result as a [Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) . Materialized Views appear in the BI Connector as tables. Rather than querying the raw data via the BI Connector, build your transformations, filters, aggregations, or joins in standard Tinybird Pipes to create a Materialized View. You can then use the BI Connector to read the pre-prepared Materialized View. For example, if your dashboard shows events aggregated by day, create a Materialized View with pre-aggregated data by day, and have your BI chart read this. ### 3. Use dedicated Materialized Views for different charts [¶](https://www.tinybird.co/docs/about:blank#3-use-dedicated-materialized-views-for-different-charts) If you have multiple charts or widgets consuming data, try to use Materialized Views that are specifically built for that chart or widget. This helps to avoid additional transformation operations. ### 4. Use efficient sorting keys [¶](https://www.tinybird.co/docs/about:blank#4-use-efficient-sorting-keys) Data stored in Tinybird is sorted using a sorting key. This helps to efficiently find data that is relevant to your query. Use the appropriate sorting keys for your data according to the needs of your dashboard. For example, if your dashboard is filtering by time, you should have time in the table's sorting key. ### 5. Avoid column mappings [¶](https://www.tinybird.co/docs/about:blank#5-avoid-column-mappings) Avoid using column mappings. Filtering by a mapped column can mean that the index (sorting key) isn't being used. ### 6. Avoid JOINs [¶](https://www.tinybird.co/docs/about:blank#6-avoid-joins) Don't do JOINs over the BI Connector where possible. Instead, do any JOINs inside a Pipe and materialize the result as a Materialized View, then query the already-JOINed data. ### 7. Monitor usage with bi_stats_rt [¶](https://www.tinybird.co/docs/about:blank#7-monitor-usage-with-bi-stats-rt) The [bi_stats_rt](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-bi-stats-rt) table contains observability data about your usage of the BI Connector. ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) The following limits apply: - Query timeout is 10 seconds. - Resources are limited to 1 thread per query. - Concurrent queries are limited to 100 per cluster. - Queries over the BI Connector are only allocated 1 CPU core per query. This only applies to the BI Connector, not to standard Tinybird Pipes. - Single queries mustn't return more than 1 GB of data at a time. - The total amount of data returned by all queries executed within a 5-minute period must not exceed 6 GB. This is to prevent overloading the system with too much data in a short period of time. Some settings can be adjusted on request. Contact Tinybird support ( [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) ) to discuss if your use case can be supported. ## Limitations [¶](https://www.tinybird.co/docs/about:blank#limitations) The following limitations apply: - You can't use parameters in your endpoints. However, you can still filter your data using SQL `WHERE` clauses. - Some widgets of BI tools create SQL queries that aren't compatible with the BI connector. - Some JOIN operations run in Postgres and not Tinybird, which means they might run much, much more slowly. Avoid JOINs over the BI Connector. - Sometimes, PostgreSQL might rewrite the query so that filters might end up not using sorting keys. Sorting keys allow you to run faster queries, and therefore those resulting queries might be slower and more expensive since they would require full table scans. - Querying large datasets can result in a timeout. Large generally means many millions of rows, but this is also affected by how many columns and how large the values are. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read up on[ Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) and getting the most from your data. - Understand how to use and query Tinybird's[ Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . --- URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/populate-data Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Populate and copy data between Data Sources · Tinybird Docs" theme-color: "#171612" description: "Learn how populating data within Tinybird works, including the details of reingesting data and important considerations for maintaining data integrity." --- # Populate and copy data between Data Sources [¶](https://www.tinybird.co/docs/about:blank#populate-and-copy-data-between-data-sources) You can use Tinybird to populate Data Sources using existing data through the Populate and Copy operations. Both are often used in similar scenarios, with the main distinction being that you can schedule Copy jobs, and that they have more restrictions. Read on to learn how populating data within Tinybird works, including the details of reingesting data and important considerations for maintaining data integrity. ## Populate by partition [¶](https://www.tinybird.co/docs/about:blank#populate-by-partition) Populating data by partition requires as many steps as partitions in the origin Data Source. You can use this approach for progress tracking and dynamic resource recalculation while the job is in progress. You can also retry steps in case of a memory limit error, which is safer and more reliable. The following diagram shows a populate scenario that involves two Data Sources: Given that Data Source A has 3 partitions: - The job processes the data partition by partition from the source, `Data Source A` - After the data has been processed, it updates the data on the destination, `Data Source B` ## Understanding the Data Flow [¶](https://www.tinybird.co/docs/about:blank#understanding-the-data-flow) As a use case expands, it might develop a complex Data Flow. Here are three key points to consider: - Data is processed by partition from the origin: each step handles data from a single partition of the origin Data Source. - When more than one Materialized Pipe exists for the same Data Source, the execution order isn't deterministic. - Destination Data Sources in the Data Flow only use the data from the specific partition being processed. The following examples illustrates the behavior of Populate jobs in different scenarios. ### Case 1: Joining data from a Data Source that isn't a destination in the Data Flow [¶](https://www.tinybird.co/docs/about:blank#case-1-joining-data-from-a-data-source-that-isnt-a-destination-in-the-data-flow) When using a Data Source (Data Source C) in a Materialized Pipe query, if Data Source C isn't a destination Data Source for any other Materialized Pipe in the Data Flow, it uses all available data in Data Source C at that moment. ### Case 2: Joining data from a Data Source that is a destination in the same Materialized View [¶](https://www.tinybird.co/docs/about:blank#case-2-joining-data-from-a-data-source-that-is-a-destination-in-the-same-materialized-view) When using the destination Data Source `Data Source B` in the Materialized Pipe query, `Data Source B` doesn't join any data. This occurs because the data is processed by partition, and the required partition isn't available in the destination at that time. ### Case 3: Joining data from a Data Source that is a destination in another Materialized View [¶](https://www.tinybird.co/docs/about:blank#case-3-joining-data-from-a-data-source-that-is-a-destination-in-another-materialized-view) When using a Data Source (Data Source C) in a Materialized Pipe (Materialized Pipe 3) query that is the destination of another Materialized Pipe (Materialized Pipe 2) in the Data Flow, it retrieves the data ingested during the process. Whether Data Source C contains data before the view on Materialized Pipe 3 runs isn't deterministic. Because the order depends on the internal ID, you can't determine which Data Source is updated first. To control the order of the Data Flow, run each populate operation separately: 1. Run a populate over Materialized Pipe 1 to populate the data from Data Source A to Data Source B. To prevent automatic data propagation through the rest of the Materialized Views, either unlink the views or truncate the dependent Data Sources if they are repopulated. 2. Perform separate populate operations on Materialized Pipe 2 and Materialized Pipe 3, instead of a single operation on Materialized Pipe 1. ## Learn more [¶](https://www.tinybird.co/docs/about:blank#learn-more) Before you use Populate and Copy operations for different [backfill strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/backfill-strategies) , understand how they work within the Data Flow and its limitations. Read also the [Materialized Views guide](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices) : populating data while continuing ingestion into the origin Data Source might lead to duplicated data. --- URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/materialized-views Content: --- title: "Materialized Views · Tinybird Docs" theme-color: "#171612" description: "Materialized Views process your data at ingest time to increase the performance of your queries and endpoints." --- # Materialized Views [¶](https://www.tinybird.co/docs/about:blank#materialized-views) A Materialized View is the continuous, streaming result of a Pipe saved as a new Data Source. As new data is ingested into the origin Data Source, the transformed results from the Pipe are continually inserted in the new Materialized View, which you can query as any other Data Source. Tinybird represents Materialized Views using the icon. Preprocessing data at ingest time reduces latency and cost-per-query, and can significantly improve the performance of your API Endpoints. For example, you can transform the data through SQL queries, using calculations such as counts, sums, averages, or arrays, or transformations like string manipulations or joins. The resulting Materialized View acts as a Data Source you can query or publish. Typical use cases of Materialized Views include: - Aggregating, sorting, or filtering data at ingest time. - Improving the speed of a query that's taking too much time to run. - Simplifying query development by automating common filters and aggregations. - Reducing the amount of data processed by a single query. - Changing an existing schema for a different use case. You can create a new Materialized View and populate it with all existing data without any cost. On-going incremental writes to Materialized Views count towards your Tinybird usage. ## Create Materialized Views [¶](https://www.tinybird.co/docs/about:blank#create-materialized-views) To create a Materialized View using the Tinybird UI, follow these steps: 1. Write your Pipe using as many nodes as needed. 2. Select the downward arrow (▽) next to** Create API Endpoint** and select** Create Materialized View** . 3. Select the node you want to use as output. 4. Edit the name of the destination Data Source. 5. Adjust the Engine Type, Sorting Keys, and so on. For a detailed example, see [Example of Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/example-mv-ui). ### Error messages [¶](https://www.tinybird.co/docs/about:blank#error-messages) With any Materialized View, Tinybird runs a speed simulation to ensure that the Materialized View won't produce any lag. If you're getting an error in Tinybird that your query isn't compatible with real-time ingestion, review your Materialized View query setup. Review [the 5 rules for faster queries](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) and keep the following principles in mind: 1. Avoid doing huge joins without filtering. 2. Use `GROUP BY` before filtering. 3. Remember that array `JOIN` s are slow. 4. Filter the right side of JOINs to speed up Materialized Views. If you're batching, especially when ingesting from Kinesis, consider decreasing the amount of data you batch. ## Create Materialized Views (CLI) [¶](https://www.tinybird.co/docs/about:blank#create-materialized-views-cli) Consider an `origin` data source, for example `my_origin.datasource` , like the following: ##### Origin data source SCHEMA > `id` Int16, `local_date` Date, `name` String, `count` Int64 You might want to create an optimized version of the Data Source that preaggregates `count` for each ID. To do this, create a new Data Source that uses a `SimpleAggregateFunction` as a Materialized View. First, define the `destination` data source, for example `my_destination.datasource`: ##### Destination data source SCHEMA > `id` Int16, `local_date` Date, `name` String, `total_count` SimpleAggregateFunction(sum, UInt64) ENGINE "AggregatingMergeTree" ENGINE_PARTITION_KEY "toYYYYMM(local_date)" ENGINE_SORTING_KEY "local_date,id" Write a transformation Pipe, for example `my_transformation.pipe`: ##### Transformation Pipe NODE transformation_node SQL > SELECT id, local_date, name, sum(count) as total_count FROM my_origin GROUP BY id, local_date, name TYPE materialized DATASOURCE my_destination Once you have the origin and destination Data Sources defined and the transformation Pipe, you can push them: ##### Push the Materialized Views tb push my_origin.datasource tb push my_destination.datasource tb push my_transformation.pipe --populate Any time you ingest data into `my_origin` , the data in `my_destination` is automatically updated. For a detailed example, see [Example of Materialized View (CLI)](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/example-mv-cli). ### Guided process using tb materialize [¶](https://www.tinybird.co/docs/about:blank#guided-process-using-tb-materialize) Alternatively, you can use the `tb materialize` command to generate the target .datasource file needed to push a new Materialized View. The goal of the command is to guide you through all the needed steps to create a Materialized View. Given a Pipe, `tb materialize`: 1. Asks you which node of the Pipe you want to materialize. By default, it selects the last one in the Pipe. If there's only one, it's automatically selected, skipping asking you. From the selected query, the commands guesses the best parameters for the following steps. 2. It warns you of the errors the query has, if any, that prevents it from materializing. If everything is correct, it continues. 3. It creates the target Data Source file that receives the results of the materialization, setting default engine parameters. If you are materializing an aggregation you should make sure the `ENGINE_SORTING_KEY` columns in the .datasource file are in the right order you are going to filter the table. 4. It modifies the query to set up the materialization settings and pushes the Pipe to create the materialization. You can skip the Pipe checks if needed as well. 5. It asks you if you want to populate the Materialized View with existing data. If you select to populate, it asks you if you want to use a subset of the data or fully populate with all existing data. 6. It creates a backup file of the Pipe adding the `_bak` suffix to the file extension. It completes the aggregate functions with `-State` combinators where needed and adds the target Data Source name. The backup file is preserved in case you want to recover the original query. The command generates and modifies the files involved in the materialization. If you run into an error or you need to modify something in the materialization, you can reuse the files as a better starting point. ### Force populate Materialized Views [¶](https://www.tinybird.co/docs/about:blank#force-populate-materialized-views) Sometimes you might want to force populate a Materialized View, most likely because you changed the transformation in the Pipe and you want the data from the origin Data Source to be reingested. You can do this using `tb push` and the `--force` and `--populate` flags: ##### Populate a Materialized View tb push my_may_view_pipe.pipe --force --populate The response contains a [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api) `job_url` that you can use to check progress and status of the job. ## Query Materialized Views [¶](https://www.tinybird.co/docs/about:blank#query-materialized-views) If the type of engine in your Data Source is `MergeTree` and you're not doing aggregations, you can query a Materialized View as a standard Data Source. If the engine is `AggregatingMergeTree`, `SummingMergeTree` or other special engine, and you have functions in your Pipe with the `-State` modifier, use the `-Merge` modifier, or `max()` and group by the Sorting Key columns. See [Best practices for Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices). For Deduplication use cases with `ReplacingMergeTree` , see [Deduplication strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies#use-the-replacingmergetree-engine). ## Limitations [¶](https://www.tinybird.co/docs/about:blank#limitations) Materialized Views work as insert triggers, which means a delete or truncate operation on your original Data Source doesn't affect the related Materialized Views. As transformation and ingestion in the Materialized View is done on each block of inserted data in the original Data Source, some operations such as `GROUP BY`, `ORDER BY`, `DISTINCT` and `LIMIT` might need a specific `engine` , such as `AggregatingMergeTree` or `SummingMergeTree` , which can handle data aggregations. The Data Source resulting from a Materialized View generated using `JOIN` is automatically updated only if and when a new operation is performed over the Data Source in the `FROM`. You can't create Materialized Views that depend on the `UNION` of several Data Sources. ## Considerations on populates [¶](https://www.tinybird.co/docs/about:blank#considerations-on-populates) As described in [Best practices for Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices) , populates are the process to move the data that was already in the original Data Source through to the new Materialized View. If you have a continuous or streaming ingestion into the original Data Source, the populate might produce duplicated rows in the Materialized View. The populate runs as a separate job that goes partition by partition moving the existing data into the Materialized View. At the same time, the Materialized View is already automatically receiving new rows. There may be an overlap where the populate job moves a partition that includes rows that were already ingested into the Materialized View. To handle this scenario, see [Backfill strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/backfill-strategies). ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn how to make the most of Materialized Views. See[ Best Practices](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices) . - Review[ the 5 rules for faster queries](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) . --- URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/copy-pipes Last update: 2025-02-13T16:23:58.000Z Content: --- title: "Copy Pipes · Tinybird Docs" theme-color: "#171612" description: "Copy Pipes allow you to capture the result of a Pipe at a moment in time, and write the result into a target Data Source. They can be run on a schedule, or executed on demand." --- # Copy Pipes [¶](https://www.tinybird.co/docs/about:blank#copy-pipes) Copy Pipes are an extension of Tinybird's [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) . Copy Pipes allow you to capture the result of a Pipe at a moment in time, and write the result into a target [Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources) . They can be run on a schedule, or executed on demand. Copy Pipes are great for use cases like: - Event-sourced snapshots, such as change data capture (CDC). - Copy data from Tinybird to another location in Tinybird to experiment. - De-duplicate with snapshots. Copy Pipes should not be confused with [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) . Materialized Views continuously re-evaluate a query as new events are inserted, while Copy Pipes create a single snapshot at a given point in time. Tinybird represents Copy Pipes using the icon. ## Best practices [¶](https://www.tinybird.co/docs/about:blank#best-practices) A Copy Pipe executes the Pipe's query on each run to export the result. This means that the size of the copy operation is tied to the size of your data and the complexity of the Pipe's query. As Copy Pipes can run frequently, it's strongly recommended that you [follow the best practices for faster SQL](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) to optimize your queries, and understand the following best practices for optimizing your Copy Pipes. ### 1. Round datetime filters to your schedule [¶](https://www.tinybird.co/docs/about:blank#1-round-datetime-filters-to-your-schedule) Queries in Copy Pipes should always have a time window filter that aligns with the execution schedule. For example, a Copy Pipe that runs once a day typically has a filter that filters for yesterday's data, and an hourly schedule usually means a filter to get results from the previous hour. Copy Pipe job aren't guaranteed to run exactly at the scheduled time. If a Copy Pipe is scheduled to run at 16:00:00, the job could run at 16:00:01 or even 16:05:05. To get the time at which the Copy Pipe job was received (not executed), use `job_timestamp` in your time window filter and round it to the schedule window. This special [parameter](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) will default to the timestamp of when the job was received, making it a lot more reliable to queue lags. For example, if your Copy Pipe is scheduled hourly, instead of writing: SELECT * FROM datasource WHERE datetime >= now() - interval 1 hour AND datetime < now() Use `toStartOfHour()` and `job_timestamp` to round the time filter to the hour and get the time at which the Copy Pipe job was received: SELECT * FROM datasource WHERE datetime >= toStartOfHour({{DateTime(job_timestamp)}}) - interval 1 hour AND datetime < toStartOfHour({{DateTime(job_timestamp)}}) Doing this means that, even if the Copy Pipe's execution is delayed (perhaps being triggered at 16:00:01, 17:00:30, and 18:00:02) you still maintain consistent copies of data regardless of the delay, with no gaps or overlaps. ### 2. Account for late data in your schedule [¶](https://www.tinybird.co/docs/about:blank#2-account-for-late-data-in-your-schedule) There are many reasons why you might have late-arriving data: system downtime in your message queues, network outages, or something else. These are largely unavoidable and will occur at some point in your streaming journey. You should **account for potential delays ahead of time**. When using Copy Pipes, include some headroom in your schedule to allow for late data. How much headroom you give to your schedule is up to you, but some useful guidance is that you should consider the Service Level Agreements (SLAs) both up and downstream of Tinybird. For instance, if your streaming pipeline has 5 minute downtime SLAs, then most of your late data should be less than 5 minutes. Similarly, consider if you have any SLAs from data consumers who are expecting timely data in Tinybird. If you schedule a Copy Pipe every 5 minutes (16:00, 16:05, 16:10...), there could be events with a timestamp of 15:59:59 that don't arrive in Tinybird until 16:00:01 (2 seconds late!). If the Copy Pipe executes at exactly 16:00:00, these events could be lost. There are two ways to add headroom to your schedule: #### Option 1: Delay the execution [¶](https://www.tinybird.co/docs/about:blank#option-1-delay-the-execution) The first option is to simply delay the schedule. For example, if you want to create a Copy Pipe that creates 5 minute snapshots, you could delay the schedule by 1 minute, so that instead of running at 17:00, 17:05, 17:10, etc. it would instead run at 17:01, 17:06, 17:11. To achieve this, you could use the cron expression `1-59/5 * * * *`. If you use this method, you must combine it with the advice from the first tip in this best practices guide to [Round datetime filters to your schedule](https://www.tinybird.co/docs/about:blank#1-round-datetime-filters-to-your-schedule) . For example: SELECT * FROM datasource WHERE datetime >= toStartOfFiveMinutes({{DateTime(job_timestamp)}}) - interval 5 minute AND datetime < toStartOfFiveMinutes({{DateTime(job_timestamp)}}) #### Option 2: Filter the result with headroom [¶](https://www.tinybird.co/docs/about:blank#option-2-filter-the-result-with-headroom) Another strategy is to keep your schedule as desired (17:00, 17:05, 17:10, etc.) but apply a filter in the query to add some headroom. For example, you can move the copy window by 15 seconds: WITH (SELECT toStartOfFiveMinutes({{DateTime(job_timestamp)}}) - interval 15 second) AS snapshot SELECT snapshot, * FROM datasource WHERE timestamp >= snapshot - interval 5 minute AND timestamp < snapshot With this method, a Copy Pipe that executes at 17:00 will copy data from 16:54:45 to 16:59:45. At 17:05, it would copy data from 16:59:45 to 17:04:45, and so on. It is worth noting that this can be confusing to data consumers who might notice that the data timestamps don't perfectly align with the schedule, so consider whether you'll need extra documentation. ### 3. Write a snapshot timestamp [¶](https://www.tinybird.co/docs/about:blank#3-write-a-snapshot-timestamp) There are many reasons why you might want to capture a snapshot timestamp, as it documents when a particular row was written. This helps you identify which execution of the Copy Pipe is responsible for which row, which is useful for debugging or auditing. For example: SELECT toStartOfHour({{DateTime(job_timestamp)}}) as snapshot_id, * FROM datasource WHERE timestamp >= snapshot_id - interval 1 hour AND timestamp < snapshot_id In this example, you're adding a new column at the start of the result which contains the rounded timestamp of the execution time. By applying an alias to this column, you can re-use it in the query as your [rounded datetime filter](https://www.tinybird.co/docs/about:blank#1-round-datetime-filters-to-your-schedule) , saving you a bit of typing. ### 4. Backfilling data on-demand [¶](https://www.tinybird.co/docs/about:blank#4-backfilling-data-on-demand) Copy Pipes can be executed following a schedule or on-demand. All the previous best practices on this page are focused on scheduled executions. There might be cases when you want to use the same Copy Pipe to do a backfill. For example, you want to execute the Copy Pipe only on data from last year, to fill in a gap behind your fresh data. To do this, you can parameterize the filters. When you run an on-demand Copy Pipe with parameters, you can modify the values of the parameters before execution. Scheduled Copy Pipes with parameters use the default value for any parameters. This means you can reuse the same Copy Pipe for your fresh, scheduled runs as well as any ad-hoc backfills. If you have used `job_timestamp` in your time window filter, you can specify its value when running an on-demand Copy Pipe to the date you want to backfill.For example, consider the following Copy Pipe which copies data from the last hour: SELECT toStartOfHour({{DateTime(job_timestamp)}}), * FROM datasource WHERE datetime >= toStartOfHour({{DateTime(job_timestamp)}}) - interval 1 hour AND datetime < toStartOfHour({{DateTime(job_timestamp)}}) You might specify `job_timestamp` to be any datetime in the past, and the same Copy Pipe would copy the data from the last hour to the specified datetime. With this, the same Copy Pipe can perform both functions: being executed on a regular schedule to keep up with fresh data, and being executed on-demand when needed. ## Configure a Copy Pipe (CLI) [¶](https://www.tinybird.co/docs/about:blank#configure-a-copy-pipe-cli) To create a Copy Pipe from the CLI, you need to create a .pipe file. This file follows the same format as [any other .pipe file](https://www.tinybird.co/docs/docs/cli/datafiles) , including defining nodes that contain your SQL queries. In this file, define the queries that will filter and transform the data as needed. The final result of all queries should be the result that you want to write into a Data Source. You must define which node contains the final result. To do this, include the following parameters at the end of the Node: TYPE COPY TARGET_DATASOURCE datasource_name COPY_SCHEDULE --(optional) a cron expression or @on-demand. If not defined, it would default to @on-demand. COPY_MODE append --(Optional) The strategy to ingest data for copy jobs. One of `append` or `replace`, if empty the default strategy is `append`. There can be only one copy node per Pipe, and no other outputs, such as [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) or [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints). Copy Pipes can either be scheduled, or executed on-demand. This is configured using the `COPY_SCHEDULE` setting. To schedule a Copy Pipe, configure `COPY_SCHEDULE` with a cron expression. On-demand Copy Pipes are defined by configuring `COPY_SCHEDULE` with the value `@on-demand`. Note that all schedules are executed in the UTC time zone. If you are configuring a schedule that runs at a specific time, be careful to consider that you will need to convert the desired time from your local time zone into UTC. Here is an example of a Copy Pipe that is scheduled every hour and writes the results of a query into the `sales_hour_copy` Data Source: NODE daily_sales SQL > % SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales FROM teams WHERE day BETWEEN toStartOfDay({{DateTime(job_timestamp)}}) - interval 1 day AND toStartOfDay({{DateTime(job_timestamp)}}) and country = {{ String(country, 'US')}} GROUP BY day, country TYPE COPY TARGET_DATASOURCE sales_hour_copy COPY_SCHEDULE 0 * * * * Before pushing the Copy Pipe to your Workspace, make sure that the target Data Source already exists and has a schema that matches the output of the query result. Data Sources will not be created automatically when a Copy Pipe runs. If you push the target Data Source and the Copy Pipe at the same time, be sure to use the `--push-deps` option in the CLI. ### Change Copy Pipe Token reference [¶](https://www.tinybird.co/docs/about:blank#change-copy-pipe-token-reference) Tinybird automatically creates a [Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) each time a scheduled copy is created, to read from the Pipe and copy the results. To change the Token strategy (for instance, to share the same one across each copy, rather than individual Tokens for each), update the [Token reference in the .pipe datafile](https://www.tinybird.co/docs/docs/cli/datafiles/pipe-files). **Important** : Once a Token is created and associated with a copy, it's crucial to handle it with care. Avoid deleting the Token or modifying its scopes, as this will break the import process and disrupt scheduled copies. The Token is essential for maintaining the connection and ensuring that data is copied correctly. ## Execute Copy Pipes (CLI) [¶](https://www.tinybird.co/docs/about:blank#execute-copy-pipes-cli) Copy Pipes can either be scheduled, or executed on-demand. When a Copy Pipe is pushed with a schedule, it will automatically be executed as per the schedule you defined. If you need to pause the scheduler, you can run `tb pipe copy pause [pipe_name]` , and use `tb pipe copy resume [pipe_name]` to resume. Note that you can't customize the values of dynamic parameters on a scheduled Copy Pipe. Any parameters will use their default values. When a Copy Pipe is pushed without a schedule, using the `@on-demand` directive, you can run `tb pipe copy run [pipe_name]` to trigger the Copy Pipe as needed. You can pass parameter values to the Copy Pipe by using the `param` flag, e.g., `--param key=value`. You can run `tb job ls` to see any running jobs, as well as any jobs that have finished during the last 48 hours. If you remove a Copy Pipe from your Workspace, the schedule will automatically stop and no more copies will be executed. ## Configure a Copy Pipe (UI) [¶](https://www.tinybird.co/docs/about:blank#configure-a-copy-pipe-ui) To create a Copy Pipe from the UI, [follow the process to create a standard Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#creating-pipes-in-the-ui). After writing your queries: 1. Select the node that contains the final result. 2. Select the actions button next to the Node. 3. Select** Create Copy Job** . To configure the frequency: 1. Select whether the Copy Pipe should be scheduled using a cron expression, or run on-demand, using the** Frequency** menu. 2. If you select a cron expression, configure the expression. 3. Select** Next** to continue. All schedules run in the UTC time zone. If you are configuring a schedule that runs at a specific time, you might need to convert the desired time from your local time zone into UTC. ### On-demand [¶](https://www.tinybird.co/docs/about:blank#on-demand) If you selected **on-demand** as the frequency for the Copy Pipe, you can customize the values for any parameters of the Pipe. You can also configure whether the Copy Pipe should write results into a new, or existing, Data Source, using the radial buttons. If you use an existing Data Source, you can select which one to use from the menu of your Data Sources. Only Data Sources with a compatible schema are in the menu. If you create a new Data Source, Tinybird guides you through creating the new Data Source. Select **Next** to continue and go through the standard [Create Data Source wizard](https://www.tinybird.co/docs/docs/get-data-in/data-sources#creating-data-sources-in-the-ui). ### Scheduled [¶](https://www.tinybird.co/docs/about:blank#scheduled) If you selected **cron expression** as the frequency for the Copy Pipe, a preview of the result appears. You can't configure parameter values for a scheduled Copy Pipe. Review the results and select **Next** to continue. Finally, you can configure whether the Copy Pipe should write results into a new, or existing, Data Source, using the radial buttons. If you use an existing Data Source, you can select which one to use from the menu of your Data Sources. Only Data Sources with a compatible schema is in the menu. If you create a new Data Source, Tinybird guides you through creating the new Data Source. Select **Next** to continue and go through the standard [Create Data Source wizard](https://www.tinybird.co/docs/docs/get-data-in/data-sources#creating-data-sources-in-the-ui). ## Run Copy Pipes (UI) [¶](https://www.tinybird.co/docs/about:blank#run-copy-pipes-ui) To run a Copy Pipe in the UI, navigate to the Pipe, and select the **Copying** button. From the options, select **Run copy now**. You can't customize the values of dynamic parameters on a scheduled Copy Pipe. Any parameters use their default values. ## Iterating a Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#iterating-a-copy-pipe) Copy Pipes can be iterated using [version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work) like any other resource in your Data Project. However, you need to understand how connections work in Branches and deployments to select the appropriate strategy for your desired changes. Branches don't execute on creation by default recurrent jobs, like the scheduled ones for Copy Pipes. To iterate a Copy Pipe, create a new one, or recreate the existing one, with the desired configuration. The new Copy Pipe starts executing from the Branch, without affecting the unchanged production resource. This means you can test the changes without mixing the test resource with your production exports. [This example](https://github.com/tinybirdco/use-case-examples/tree/main/change_copy_pipe_time_granularity) , shows how to change the Copy Pipe time granularity, adding an extra step for backfill the old data. ## Monitoring [¶](https://www.tinybird.co/docs/about:blank#monitoring) Tinybird provides a high level metrics page for each Copy Pipe in the UI, as well as exposing low-level observability data through the [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources). You can view high level status and statistics about your Copy Pipes in the Tinybird UI from the Copy Pipe's details page. To access the details page, navigate to the Pipe, and select **View Copy Job**. The details page shows summaries of the Copy Pipe's current status and configuration, as well as Charts showing the performance of previous executions. You can also monitor your Copy Pipes using the [datasource_ops_log Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . This Data Source contains data about all your operations in Tinybird. Logs that relate to Copy Pipes can be identified by a value of `copy` in the `event_type` column. For example, the following query aggregates the Processed Data from Copy Pipes, for the current month, for a given Data Source name. SELECT toStartOfMonth(timestamp) month, sum(read_bytes + written_bytes) processed_data FROM tinybird.datasources_ops_log WHERE datasource_name = '{YOUR_DATASOURCE_NAME}' AND event_type = 'copy' AND timestamp >= toStartOfMonth({{DateTime(job_timestamp)}}) GROUP BY month Using this Data Source, you can also write queries to determine average job duration, amount of errors, error messages, and more. ## Execution [¶](https://www.tinybird.co/docs/about:blank#execution) Copy Jobs are processed through a First-In-First-Out (FIFO) queue system. The execution behavior is governed by the following parallelism settings. ### Default concurrency [¶](https://www.tinybird.co/docs/about:blank#default-concurrency) Each environment supports two concurrent Copy Jobs. In shared environments, this limit applies collectively across all users, meaning the two parallel slots are shared among all users in that environment: - Copy Jobs are processed in FIFO queue order. - Multiple jobs from the same Copy Pipe can execute concurrently. - For scheduled Copy Pipe jobs, immediate execution at the configured time isn't guaranteed. ### Dedicated environments [¶](https://www.tinybird.co/docs/about:blank#dedicated-environments) In dedicated environments, the concurrent job limit can be increased upon request to accommodate your specific workload requirements. For optimization strategies and guidelines on preventing duplicate data copies, refer to the [Best Practices section](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes#best-practices). ## Billing [¶](https://www.tinybird.co/docs/about:blank#billing) Processed Data and Storage are the two metrics that Tinybird uses for [billing](https://www.tinybird.co/docs/docs/get-started/plans/billing) Copy Pipes. A Copy Pipe executes the Pipe's query (Processed Data) and writes the result into a Data Source (Storage). Any processed data and storage incurred by a Copy Pipe is charged at the standard rate for your billing plan (see ["Tinybird plans"](https://www.tinybird.co/docs/docs/get-started/plans) ). See the [Monitoring section](https://www.tinybird.co/docs/about:blank#monitoring) for guidance on monitoring your usage of Copy Pipes. ## Limits [¶](https://www.tinybird.co/docs/about:blank#limits) Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more. ### Free and Developer [¶](https://www.tinybird.co/docs/about:blank#free-and-developer) The schedule applied to a Copy Pipe doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed some time after the scheduled time. ### Enterprise [¶](https://www.tinybird.co/docs/about:blank#enterprise) A maximum execution time of 50% of the scheduling period, 30 minutes max, means that if the Copy Pipe is scheduled to run every minute, the operation can take up to 30 seconds. If it's scheduled to run every 5 minutes, the job can last up to 2m30s, and so forth. This is to prevent overlapping jobs, which can impact results. The schedule applied to a Copy Pipe doesn't guarantee that the job executes immediately at the configured time. When the configured time elapses, the job is placed into a job queue. It is possible that, if the queue is busy, the job could be delayed and executed some time after the scheduled time. Distribute the jobs over a wider period of time rather than grouping them close together. For Enterprise customers, these settings can be customized. Reach out to your Customer Success team directly, or email us at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co). ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Understand how to use Tinybird's[ dynamic query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) . - Read up on configuring, executing, and iterating[ Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) . --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/working-with-version-control Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Working with version control · Tinybird Docs" theme-color: "#171612" description: "With Tinybird, you can take your existing version control knowledge from software engineering, and apply it immediately to your real-time data products." --- # Working with version control [¶](https://www.tinybird.co/docs/about:blank#working-with-version-control) With Tinybird, you can take your existing version control knowledge from software engineering, and apply it immediately to your real-time data products, using Git. ## How the Git integration works [¶](https://www.tinybird.co/docs/about:blank#how-the-git-integration-works) Tinybird's Git integration creates a bi-directional link between your Tinybird Workspace and a remote Git repository. This means you can work on Tinybird resources locally, push to Git and then sync to Tinybird. In this way, Git becomes the ultimate source of truth for your project. You can then follow standard development patterns, such as working in feature branches, using pull requests, and running CI/CD pipelines to validate changes and deploy to production. When you connect your Workspace to a Git repository, use the following [CI/CD](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) to deploy to the Workspace. The CI/CD pipelines need to be executed outside of Tinybird in a runner such as GitHub Action or GitLab Runner. You can connect your Workspace to Git using the [CLI](https://www.tinybird.co/docs/about:blank#connect-your-workspace-to-git-from-the-cli). You can connect both new and existing Tinybird Workspaces with Git at any time. If you're connecting an existing Workspace for the first time, your project syncs with the remote Git-based repository the moment you connect it. If you make changes in Tinybird after connecting with Git, you can create a Branch and merge the changes with a Pull Request. ## Project structure [¶](https://www.tinybird.co/docs/about:blank#project-structure) A Tinybird project is represented by a collection of text files (called [datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) ) that are organized in folders. You can initialize a new Tinybird project with automatic scaffolding using the `tb init` CLI command. This creates the following files and folders: - /datasources - /datasources/fixtures - /endpoints - /pipes - /tests - /scripts - /scripts/exec_test.sh - /scripts/append_fixtures.sh - /deploy If you have an existing project created in the UI, use the `tb pull --force` command to download all resources from your Workspace, creating the same structure as above. The purpose of these files and folder is as follows: - `datasources` : Where you put your .datasource files. - `datasources/fixtures` : Place as many CSV or NDJSON files that will be pushed when using the default `./scripts/append_fixtures.sh` script. They need to share name with a .datasource file. - `endpoints` : You can use this folder to create a logical separation between non-Endpoint Pipes and Endpoint Pipes, though it's not necessary. By default, all .pipe files will be placed in the `pipes/` directory when pulled from Tinybird. - `pipes` : Where you put your .pipe files. - `tests` : Where you put[ data quality and fixture tests](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies) . - `scripts` : Useful scripts for common operations like data migrations, fixtures tests, etc. - `deploy` : Custom deployment shell scripts. ## Connect your Workspace to Git from the CLI [¶](https://www.tinybird.co/docs/about:blank#connect-your-workspace-to-git-from-the-cli) To connect your Workspace to Git, you will need a Tinybird [Workspace](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) and a Git repository. You can either connect a pre-existing repository, or create a new one as part of this process. If you don't already have the CLI installed, follow the instructions [here](https://www.tinybird.co/docs/docs/cli/install). To initialize your Workspace with Git, run the `tb init --git` command. It performs the following actions: - Checks there are no differences between your local files and Tinybird Workspace. - Saves a reference to the current Git repository commit in the Workspace. This commit reference is used later on to diff Workspace resources and resources in a branch, to ease deployment. ##### Initialize Tinybird with a Git repository tb init --git ** - /datasources already exists, skipping ** - /datasources/fixtures already exists, skipping ** - /endpoints already exists, skipping ** - /pipes already exists, skipping ** - /tests already exists, skipping ** - /scripts already exists, skipping ** - /deploy already exists, skipping ** - '.tinyenv' already exists, skipping ** Checking diffs between remote Workspace and local. Hint: use 'tb diff' to check if your data project and Workspace synced Pulling datasources [####################################] 100% Pulling pipes [####################################] 100% Pulling tokens [####################################] 100% ** No diffs detected for 'workspace' Once complete, create CI/CD actions to integrate Tinybird with your Git provider. [These actions](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) should be based on your development pipeline, and the CLI commands Tinybird provides offer an excellent basis upon which to validate changes and deploy to Tinybird safely from Git. Add the `.tinyb` file to your `.gitignore` to avoid pushing the Tinybird configuration files to your Git provider. ##### Pushing the CI/CD actions to your Git provider echo ".tinyb" >> .gitignore git add . git commit -m "Add Tinybird CI/CD actions" git push You must save your Workspace admin [Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) as a secret in your repository. For example, with GitHub, go to your repository settings, and under Secrets and Variables / Actions, add the Token value in a secret called `TB_ADMIN_TOKEN`. You can make your shell PROMPT print your current Workspace by following this [CLI guide](https://www.tinybird.co/docs/docs/cli/install#configure-your-shell-prompt) ## Protecting the main Workspace [¶](https://www.tinybird.co/docs/about:blank#protecting-the-main-workspace) Once you decide to use this version-controlled workflow, the Git repository becomes your single source of truth. You'll want to keep production protected so users can't modify resources and break the Git workflow. If you want to prevent users from making changes to a the Workspace from the CLI or API, open the Tinybird UI and navigate to the Workspace settings menu, then the **Members** tab, and assign the **Viewer** role. Users with a Viewer role aren't able to create, edit, or delete resources or run data operations. They are allowed to create a new Branch and change resources there. ## Development workflow [¶](https://www.tinybird.co/docs/about:blank#development-workflow) This section explains how to safely develop new features using [Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches). ### Develop using the CLI [¶](https://www.tinybird.co/docs/about:blank#develop-using-the-cli) When prototyping a new API Endpoint or changing its logic, the UI is the easiest and fastest way to iterate and validate your changes, as you can see the results of your changes in real time. You can use a branch and then `tb pull` your changes to push them to Git. But, when making changes like data migrations or column types, you need to use the CLI and modify the datafiles. Make sure you're familiar with [the Tinybird CLI docs](https://www.tinybird.co/docs/docs/cli/quick-start). For changes like these, use the Git workflow: Create a new branch, make the changes in the datafiles, and create a Pull Request to validate the changes. For further guidance, read the [CI/CD docs](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration). This image visualizes each process step when setting up and working with Git via the Tinybird CLI: <-figure-> ![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fconnect-to-git.png&w=3840&q=75) ## Exploration workflow [¶](https://www.tinybird.co/docs/about:blank#exploration-workflow) ### The Playground [¶](https://www.tinybird.co/docs/about:blank#the-playground) Once you have protected production from user modifications, you might not be able to create Pipes directly. To explore the data, you need to either create a new Branch, or use [the Playground](https://www.tinybird.co/docs/docs/work-with-data/query#use-the-playground). Once you've prototyped your new Pipe, download the Pipe from the UI and commit the change to a new Git branch to follow the CI workflow. By default, Playground content is private to your Workspace view. However, you have the option to share your Playground with other Workspace members. ## Troubleshooting [¶](https://www.tinybird.co/docs/about:blank#troubleshooting) This section covers some of the problems you might face when working with version control. ### Connect to more than one Workspace [¶](https://www.tinybird.co/docs/about:blank#connect-to-more-than-one-workspace) You can have one Git repo connected to 2 Tinybird Workspaces - a common scenario is using this for [staging and production Workspaces](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/staging-and-production-workspaces) . Just use a different `ADMIN_TOKEN` in the GitHub Actions. ### Initialization [¶](https://www.tinybird.co/docs/about:blank#initialization) When your Git repository and Tinybird Workspace have the same resources, `tb init --git` allows you to easily start up a CI/CD pipeline. However, when these aren't in sync, you might experience one of these three scenarios: **Problem: There are resources in the Workspace that aren't present in your Git repository.** Remove them from your Workspace (they are probably unnecessary) or run `tb pull --match ""` to download the resource(s) locally and push them to the Git repository before continuing. **Problem: There are resources in your Git repository that aren't present in your Workspace.** Either remove them from the project or run `tb push` them to the Workspace and re-run the `init` command. **Problem: There are differences between the resources in the Git repository and the ones in your Workspace.** In this instance, you must decide which version is the source of truth. If it's the resources in the Workspace, run `tb pull --force` to overwrite your local files. If it's your local files, run `tb push --force` file by file so the Workspace is synchronized with your project in Git. Use `tb diff` to check if your project is synced. It diffs local [datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) to the corresponding resources in the Workspace. ### Git and Workspace no longer synced [¶](https://www.tinybird.co/docs/about:blank#git-and-workspace-no-longer-synced) When you deploy, the Git commit ID of that deployment is stored in the Workspace. This means you can use the `git diff` command to compare your Branch against the main one, and know which resources have been modified and need to be deployed. If you introduce a manual change in your Workspace or Git outside of the CI/CD workflow, they will no longer be synced. To get both components to once again be in sync with each other, use the command `tb init --override-commit ` . This command will override the Git commit ID from the Workspace. ## Common use cases [¶](https://www.tinybird.co/docs/about:blank#common-use-cases) Version control allows you to incrementally change or iterate your data project. It's ideal for managing changes like adding a column, changing data types, redefining whole views, and lots more. If you're new to using version control or want to be sure how to do it on Tinybird, there's an entire repository of use cases available: [tinybirdco/use-case-examples](https://github.com/tinybirdco/use-case-examples). ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read about[ datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) , the text files that describe your Tinybird resources (Data Sources, Pipes). - Understand[ CI/CD processes on Tinybird](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) . --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/staging-and-production-workspaces Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Staging and production Workspaces · Tinybird Docs" theme-color: "#171612" description: "Tinybird projects can be deployed to multiple Workspaces. You can use this to create production and staging environments for your project." --- # Staging and production Workspaces [¶](https://www.tinybird.co/docs/about:blank#staging-and-production-workspaces) Tinybird projects can be deployed to multiple Workspaces that contain different data or different connection settings. You can use this to create production and staging environments for your project. Keep your Workspaces clear and distinct. While you can deploy a project to **multiple** Workspaces, you should avoid deploying multiple projects to the **same** Workspace. This page covers how to set up a CI/CD workflow with staging and production Workspaces. This setup includes: - A staging Workspace with staging data and connection settings, used to validate change to your project before deploying to production. - A production Workspace with production data and connection settings, used to serve your data to real users. - A CI/CD workflow that will run CI over the staging Workspace, deploy manually to the staging Workspace, and finally automatically promote to the production Workspace upon merge. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fstaging-prod.png&w=3840&q=75) ## Example project setup [¶](https://www.tinybird.co/docs/about:blank#example-project-setup) Create a staging and production Workspaces and authenticate using your Workspace admin Token. ##### Create Workspaces tb workspace create staging_acme --user_token tb workspace create pro_acme --user_token Push the project to the production Workspace: ##### Recreating the project tb workspace use pro_acme tb push --push-deps --fixtures Finally, push the project to the staging Workspace: ##### Recreating the project tb workspace use staging_acme tb push --push-deps --fixtures Once you have the project deployed to both Workspaces make sure you [connect them to Git](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) by: 1. Running `tb auth` using the admin token of the Workspace. 2. Running `tb init --git` to connect the Workspace to your git main branch. You'll need to run both steps twice (once for each Workspace). When running `tb init --git` you need to make sure you end up with a setup like this one: - `tinybird_ci_stg.yml` : Runs CI over the staging Workspace. - `tinybird_cd_stg.yml` : Runs CD over the staging Workspace. You can run this job manually or following any other strategy. - `tinybird_cd.yml` : Runs CD over the production Workspace on merge a branch to the main git branch. ## Configuring CI/CD [¶](https://www.tinybird.co/docs/about:blank#configuring-cicd) The following CI/CD jobs are based on the examples in the [Continuous Integration](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) section. A common pattern is to run CD in the staging Workspace before moving to production. Below is an example of a GitHub Actions workflow that will perform CI/CD on the staging Workspace. ##### Staging CI pipeline name: Tinybird - Staging CI Workflow on: workflow_dispatch: pull_request: paths: - './**' branches: - main types: [opened, reopened, labeled, unlabeled, synchronize, closed] concurrency: ${{ github.workflow }}-${{ github.event.pull_request.number }} jobs: staging_ci: # run CI to staging Workspace uses: tinybirdco/ci/.github/workflows/ci.yml@main with: data_project_dir: . tb_env: stg secrets: tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_STG }} # set the Workspace admin Token from the staging Workspace in a new secret tb_host: https://app.tinybird.co ##### Staging CD pipeline name: Tinybird - Staging CD Workflow on: workflow_dispatch: jobs: staging_cd: # deploy changes to staging Workspace uses: tinybirdco/ci/.github/workflows/cd.yml@main with: data_project_dir: . tb_env: stg secrets: tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN_STG }} # set the Workspace admin Token from the staging Workspace in a new secret tb_host: https://app.tinybird.co The important part is that you are using the admin token from the staging Workspace, stored in a secret with name `TB_ADMIN_TOKEN_STG` . Additionally, you are setting the `tb_env` variable to `stg` , usage of the `tb_env` input variable is described in the next section. The CD GitHub Action for the production Workspace, would be like this: ##### Production CD pipeline name: Tinybird - Production CD Workflow on: workflow_dispatch: push: paths: - './**' branches: - main jobs: production_cd: # deploy changes to production Workspace uses: tinybirdco/ci/.github/workflows/cd.yml@main with: data_project_dir: . tb_env: prod secrets: tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN }} # set the Workspace admin Token from the production Workspace in a new secret tb_host: https://app.tinybird.co In this case, the job will run on merge to the git `main` branch, using the admin token from the production Workspace and setting the `tb_env` variable to `prod` ## Workspace connection credentials and environment variables [¶](https://www.tinybird.co/docs/about:blank#workspace-connection-credentials-and-environment-variables) You'll likely use different credentials or connection parameters for each Workspace, for instance if you have a Kafka topic/S3 bucket/etc. for staging, and a different one for production. Use environment variables to manage these credentials. If you are using your own CI/CD pipelines see the [docs on Include and environment variables](https://www.tinybird.co/docs/docs/cli/datafiles/include-files#include-with-environment-variables). If you are using the ones provided in this [GitHub repository](https://github.com/tinybirdco/ci) you can follow this strategy: - Set the `tb_env` input variable as shown in the previous section to a value depending on the environment to be deployed. - Then you have available a `$TB_ENV` environment to configure connection credentials via `include` files as described in[ this section](https://www.tinybird.co/docs/docs/cli/datafiles/include-files#include-with-environment-variables) - Now you can create `connection_details_stg.incl` and `connection_details_prod.incl` and use them inside datafiles like this: `INCLUDE connection_details_${TB_ENV}.incl` . Since `$TB_ENV` has a different value depending on the pipeline for the staging or production environments, resources will be deployed using their environment specific credentials. You can see a working example [here](https://github.com/tinybirdco/use-case-examples/pull/351/files) ## Staging Workspace vs Branch [¶](https://www.tinybird.co/docs/about:blank#staging-workspace-vs-branch) Branches are intended to be ephemeral or short-lived. They are useful for testing changes while you are developing them. Typically, when you begin to develop a new feature, you'll begin by creating a new Branch. Your development work takes place on the Branch, and you can test your changes as you go. On the other hand Workspaces are intended to be permanent or long-lived. You'll use a Workspace to deploy your project into production, as testing environments or to experiment with new projects. Staging Workspaces are optional, and different teams might use them differently, for example: - You don't want to test with your production data, so you have a separate well known subset of data in staging. - You want to perform integration testing with the development version of your project before moving it to the production Workspace. - You want to test out a complex deployment or data migration before deploying to the production Workspace. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Learn more about[ working with version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) . - Learn how to integrate Workspaces with Git in a[ Continuous Integration and Deployment (CI/CD)](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) pipeline. --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/organizing-resources Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Organizing resources in Workspaces · Tinybird Docs" theme-color: "#171612" description: "Tinybird projects can have many resources. Here's how to organize them." --- # Organizing resources in Workspaces [¶](https://www.tinybird.co/docs/about:blank#organizing-resources-in-workspaces) When working with Tinybird, projects can quickly accumulate a multitude of resources. Effective organization is essential for maintaining efficiency and clarity. While there's no one-size-fits-all approach, Tinybird recommends structuring each Workspace around a specific use case. This practice keeps related resources together and simplifies collaboration and management. In the Tinybird UI, your Data Project is automatically divided into two main folders: Pipes and Data Sources. These folders are fundamental and can't be deleted or removed. However, you have the flexibility to further organize your resources by assigning specific attributes using Tags. ## Organizing resources in the UI [¶](https://www.tinybird.co/docs/about:blank#organizing-resources-in-the-ui) ### Assigning Tags to Resources [¶](https://www.tinybird.co/docs/about:blank#assigning-tags-to-resources) Tags are a powerful way to categorize and manage your resources. You can add tags to both Pipes and Data Sources: - For Pipes: You'll find the option to add tags directly under the title of the Pipe or in the Pipe list screen. <-figure-> ![pipe tags](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Forganizing-resources-1.png&w=3840&q=75) - For Data Sources: Tags can be added in the metadata section of each Data Source or in the Data Source list screen. <-figure-> ![datasource tags](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Forganizing-resources-2.png&w=3840&q=75) When you create a tag, it becomes available to assign to multiple resources, allowing for consistent categorization across your project. <-figure-> ![creating tag](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Forganizing-resources-3.png&w=3840&q=75) If you didn't assign tags when creating your resources, you can always add or modify them later from the general lists of Pipes and Data Sources. <-figure-> ![bulk tags](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Forganizing-resources-4.png&w=3840&q=75) ### Filtering Resources [¶](https://www.tinybird.co/docs/about:blank#filtering-resources) One of the main advantages of using tags is the ability to filter your resources in the sidebar. <-figure-> ![tag filters](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Forganizing-resources-5.png&w=3840&q=75) For instance, if you have a tag called “latest” to denote the most recent versions of your resources, you can easily filter the sidebar to display only those tagged resources. This is especially helpful when your Workspace contains numerous resources, and you need to focus on specific versions or categories to streamline your workflow. By effectively using tags, you can enhance your productivity and maintain a well-organized Workspace, making it easier to navigate and manage your Tinybird projects. ### Renaming Tags [¶](https://www.tinybird.co/docs/about:blank#renaming-tags) If you need to change the name of a tag, go to any resource that it's tagged with the one you want to replace. Click on it and choose **"Rename tag"** <-figure-> ![renaming tags](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Frenaming-tags.gif&w=3840&q=75) ### Deleting Tags permanently [¶](https://www.tinybird.co/docs/about:blank#deleting-tags-permanently) If you need to delete a tag from the whole Workspace, go to any resource that it's tagged with the one you want to delete. Click on it and choose **"Delete permanently"** You'll see a list with all resources attached to the tag. Once you confirm, the tag will be deleted and the resources dettached from the tag. <-figure-> ![deleting tags](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fdeleting-tags.gif&w=3840&q=75) ## Organizing resources using the CLI [¶](https://www.tinybird.co/docs/about:blank#organizing-resources-using-the-cli) Tags are supported in CLI versions 5.8.0 and above. ### Tags in datafiles [¶](https://www.tinybird.co/docs/about:blank#tags-in-datafiles) Tags aren't restricted to working in the UI. If you use the CLI to manage your projects, you can also add tags to Data Sources and Pipes using their datafiles. The tags syntax is `TAGS` **followed by comma-separated values** , at the top section of the files. TAGS "stock_case, stats" To tag a Data Source, write the TAGS directive right before `SCHEMA` ##### tinybird/datasources/example.datasource TOKEN tracker APPEND DESCRIPTION > Analytics events **landing data source** TAGS "stock_case, stats" SCHEMA > `timestamp` DateTime `json:$.timestamp`, `session_id` String `json:$.session_id`, `action` LowCardinality(String) `json:$.action`, ... To tag a Pipe, write the `TAGS` directive right before the `NODE` s settings ##### tinybird/pipes/sales_by_hour_mv.pipe DESCRIPTION materialized Pipe to aggregate sales per hour in the sales_by_hour Data Source TAGS "stock_case, stats" NODE daily_sales SQL > SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales FROM teams GROUP BY day, country TYPE MATERIALIZED DATASOURCE sales_by_hour When you deploy your changes, the resource will get tagged. You can use the tags to filter the data project later in the UI. Tags get created automatically. You don't have to worry about creating a new one before tagging a resource. If the tag you've written doesn't exist, pushing that file will create it. To know more about publishing changes to your data project, read the [Deployment Strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deployment-strategies) docs. To know more about the datafiles syntax, read the [datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) doc in the CLI section. ### tb tag [¶](https://www.tinybird.co/docs/about:blank#tb-tag) `tb tag` is a CLI command to manage your Workspace tags. You can create, list and delete tags in your workspace. Please refer to the [tb tag command reference](https://www.tinybird.co/docs/docs/cli/command-ref#tb-tag) for more details. Do you miss any functionality in the tags system? Would you like any other command in the CLI regarding tags? Do you have in mind any other feature that would benefit from the tagging system? Share your feedback by contacting Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/integrating-vercel Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Vercel Integration · Tinybird Docs" theme-color: "#171612" description: "In this guide you'll learn how to integrate your Vercel Project with a Tinybird Workspace and sync your Tokens." --- # Integrating Vercel with Tinybird [¶](https://www.tinybird.co/docs/about:blank#integrating-vercel-with-tinybird) This integration will allow you to link your Tinybird Workspaces with your Vercel projects to sync Tinybird [Static Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#what-should-i-use-tokens-for) into Vercel Environment Variables. This integration makes it easy to use Tinybird as a purpose-built analytics backend from within the Vercel Marketplace. Build any kind of analytics, be it web analytics, structured logging, telemetry, anomaly detection, personalization or anything else you can think of - and never have to worry about infrastructure, scale or security. ## Add the Tinybird integration [¶](https://www.tinybird.co/docs/about:blank#add-the-tinybird-integration) There's basically two stages to this process - adding Tinybird to your Vercel account to set the overall scope of the integration, and then connecting specific Vercel Projects to Tinybird Workspaces. To get started, you can either search for Tinybird in the Vercel integration marketplace, which looks like this: <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-1.png&w=3840&q=75) Or you can just use this [link](https://vercel.com/integrations/tinybird/new) to go straight to creating a new Tinybird integration. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-2.png&w=3840&q=75) Either way, click **Add integration** to get started and then select your target Vercel account. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-3.png&w=3840&q=75) Next, select the scope of which Projects the Tinybird integration will be added to - this controls which Projects will be presented as options to connect to a Tinybird Workspace. You can limit the selection to a specific project if you want. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-4.png&w=3840&q=75) Finally in this section, review and approve the required permissions to allow this integration to function. You need to be able to read Project and Account information, and manage the environment variables in the projects in order to provide Tokens and URLS for connectivity. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-5.png&w=3840&q=75) The integration workflow will launch straight into connecting a Project to a Workspace, but you can come back to it later if you want via the Integrations dashboard in your Vercel Account ## Integrate a Vercel project [¶](https://www.tinybird.co/docs/about:blank#integrate-a-vercel-project) You can set up a new Workspace integration from the Vercel Account integrations control panel by selecting the Tinybird integration, and then the **Configure** button. Note that you'll already be in this workflow the first time you add the integration to the account. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-6.png&w=3840&q=75) Once in this control panel, you'll be able to review any integrations present, for now let's add our first one. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-7.png&w=3840&q=75) First in this workflow, select which Tinybird region you want to connect to <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-8.png&w=3840&q=75) Next, select at least one Vercel Project to include - they'll all get the Environment variables added <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-9.png&w=3840&q=75) Now you need a Tinybird Workspace to integrate with - you can either pick one you already have, or you can create a shiny new one by popping out into the Tinybird UI. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-10.png&w=3840&q=75) Now to select which Static Tokens you want to have sync'd into the Vercel Projects as Environment Variables. If you don't see the Token you want in the list, you can pop out into the Tinybird UI and create one with the scope you want. If this is your first time, you can just use the default admin Token to get started with and reduce the scope later. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-11.png&w=3840&q=75) And that's it! Now your Project(s) will have these Tokens available ready to use. Note that the Environment variables follow a convenient naming format `TB__` In your Tinybird UI, at the bottom of the main dropdown menu, you'll see a new option for Integrations - You can review your various integrations here. If you want to update and Integration, rerun the workflow by clicking the **Add project** button. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-14.png&w=3840&q=75) And if you head to your Tokens page within the Tinybird UI, you'll see a handy notification if that Token is being actively synchronized. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-12.png&w=3840&q=75) ## Removing the integration [¶](https://www.tinybird.co/docs/about:blank#removing-the-integration) To remove Tinybird from your Vercel Account, go to the Integrations management page and select **Remove integration**. <-figure-> ![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-integrating-vercel-13.png&w=3840&q=75) This removes the Integration and environment variables. Note that your Tinybird Workspaces and Tokens isn't removed. --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/continuous-integration Last update: 2025-01-07T15:48:10.000Z Content: --- title: "Continuous Integration and Deployment (CI/CD) · Tinybird Docs" theme-color: "#171612" description: "How to implement Continuous Integration and Deployment workflows for your Tinybird data project." --- # Continuous integration and continuous deployment (CI/CD) [¶](https://www.tinybird.co/docs/about:blank#continuous-integration-and-continuous-deployment-cicd) Once you connect your data project and Workspace [through Git](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) you can implement a Continuous Integration (CI) and Continuous Deployment (CD) workflow to automate interaction with Tinybird. This page covers how CI and CD work using a walkthrough example. CI/CD pipelines require the use of: - [ Datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) - [ CLI commands](https://www.tinybird.co/docs/docs/cli/command-ref) - [ Tinybird Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches) ## How continuous integration works [¶](https://www.tinybird.co/docs/about:blank#how-continuous-integration-works) As you expand and iterate on your data projects, you can continuously validate your API Endpoints. In the same way that you write integration and acceptance tests for source code in a software project, you can write automated tests for your API Endpoints to run on each Pull or Merge request. Continuous Integration can help with: - Linting: Syntax and formatting on[ datafiles](https://www.tinybird.co/docs/docs/cli/datafiles) . - Correctness: Making sure you can push your changes to a Tinybird Workspace. - Quality: Running fixture tests or data quality tests or both to validate the changes in the Pull Request. - Regression: Running automatic regression tests to validate endpoint performance and data quality. The following section uses the CI template, GitHub Actions, and the Tinybird CLI to demonstrate how to test your API Endpoints on any new commit to a Pull Request. Set these optional environment variables to adapt your CI/CD workflow: - `TB_VERSION_WARNING=0` : Don't print CLI version warning message if there's a new version available. - `TB_SKIP_REGRESSION=0` : Skip regression tests. ### Building the CI/CD pipeline [¶](https://www.tinybird.co/docs/about:blank#building-the-cicd-pipeline) This section demonstrates automating CI/CD pipelines using GitHub as the provider with a GitHub Action, but you can use any suitable platform. The examples on this page use the Tinybird's CI and CD templates in [this repository](https://github.com/tinybirdco/ci) . You can find examples for Gitlab in that repository as well. You can those example templates or build your own pipelines inspired by them. That way you can adapt them to suit your data project needs and integrate them better with the CI/CD workflow you use for other parts of your toolset. These steps use the [Tinybird CLI commands](https://www.tinybird.co/docs/docs/cli/command-ref) so you can fully reproduce the pipeline locally. Remember to add a new secret with the Workspace administrator [Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) to the repository's settings to be able to run the needed commands from the CLI. #### 1. Trigger the CI workflow [¶](https://www.tinybird.co/docs/about:blank#1-trigger-the-ci-workflow) ##### Run the CI workflow on each commit to a Pull Request, when labelling, or with other kinds of triggers name: Tinybird - CI Workflow on: workflow_dispatch: pull_request: branches: - main types: [opened, reopened, labeled, unlabeled, synchronize, closed] Key points: The CI workflow triggers when a new Pull Request opens, reopens, synchronizs or updates labels and the base branch has to be `main` . On closed, it deletes the Tinybird branch created for CI. #### 2. Configure the CI job [¶](https://www.tinybird.co/docs/about:blank#2-configure-the-ci-job) ##### Use the workflow configuration defined in the uses reference jobs: ci: # ci using Branches from Workspace 'web_analytics_starter_kit' uses: tinybirdco/ci/.github/workflows/ci.yml@main with: data_project_dir: . secrets: tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN }} # set Workspace admin Token in GitHub secrets tb_host: https://api.tinybird.co You can combine the CI trigger and the CI job configuration into a single YAML workflow file and store it in `/my_repo/.github/workflows` . If you aren't already familiar with GitHub actions you can checkout the [quickstart guide](https://docs.github.com/en/actions/writing-workflows/quickstart). If your data project directory isn't in the root of the Git repository, change the `data_project_dir` variable. About secrets: - `tb_host` : The URL of the region you want to use. - `tb_admin_token` : The Workspace admin Token. This grants all the permissions for a specific Workspace. You can find more information in the[ Tokens docs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . ### The CI workflow [¶](https://www.tinybird.co/docs/about:blank#the-ci-workflow) A potential CI workflow could run the following steps: 1. Configuration: set up dependencies and installs the Tinybird CLI to run the required commands. 2. Check the data project syntax and the authentication. 3. Create a new ephemeral CI Tinybird Branch. 4. Push the changes to the Branch. 5. Run tests in the Branch. 6. Delete the Branch. #### 0. Workflow configuration [¶](https://www.tinybird.co/docs/about:blank#0-workflow-configuration) defaults: run: working-directory: ${{ inputs.data_project_dir }} if: ${{ github.event.action != 'closed' }} steps: - uses: actions/checkout@master with: fetch-depth: 300 ref: ${{ github.event.pull_request.head.sha }} - uses: actions/setup-python@v5 with: python-version: "3.11" architecture: "x64" cache: 'pip' - name: Validate input run: | [[ "${{ secrets.tb_admin_token }}" ]] || { echo "Go to the tokens section in your Workspace, copy the 'admin token' and set TB_ADMIN_TOKEN as a Secret in your Git repository"; exit 1; } - name: Set environment variables run: | _ENV_FLAGS="${ENV_FLAGS:=--last-partition --wait}" _NORMALIZED_BRANCH_NAME=$(echo $DATA_PROJECT_DIR | rev | cut -d "/" -f 1 | rev | tr '.-' '_') GIT_BRANCH=${GITHUB_HEAD_REF} echo "GIT_BRANCH=$GIT_BRANCH" >> $GITHUB_ENV echo "_ENV_FLAGS=$_ENV_FLAGS" >> $GITHUB_ENV echo "_NORMALIZED_BRANCH_NAME=$_NORMALIZED_BRANCH_NAME" >> $GITHUB_ENV Key points: This sets the default `working-directory` to the `data_project_dir` variable, check outs the `main` branch to get the head commit, checks the `TB_ADMIN_TOKEN` , and installs Python 3.11. #### 1. Install the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#1-install-the-tinybird-cli) - name: Install Tinybird CLI run: | if [ -f "requirements.txt" ]; then pip install -r requirements.txt else pip install tinybird-cli fi - name: Tinybird version run: tb --version Workflow actions use the Tinybird CLI to interact with your Workspace, create a test Branch, and run the tests. You can use a `requirements.txt` file to pin a tinybird-cli version to avoid automatically install the latest version. You can run this workflow locally by having a local data project and the CLI authenticated to your Tinybird Workspace. #### 2. Check the data project syntax and the authentication [¶](https://www.tinybird.co/docs/about:blank#2-check-the-data-project-syntax-and-the-authentication) - name: Check all the datafiles syntax run: tb check - name: Check auth run: tb --host ${{ secrets.tb_host }} --token ${{ secrets.tb_admin_token }} auth info #### 3. Create a new Tinybird Branch to deploy changes and run the tests [¶](https://www.tinybird.co/docs/about:blank#3-create-a-new-tinybird-branch-to-deploy-changes-and-run-the-tests) A Branch is an isolated copy of the resources in your Workspace at a specific point in time. It's designed to be temporary and disposable so that you can develop and test changes before deploying them to your Workspace. Each CI job creates a Branch. In this example, the Tinybird Brand name uses `github.event.pull_request.number` as a unique identifier so multiple tests can run in parallel. If a Branch with the same name exist, it's removed and recreated again. The `tb branch create` command creates new Branches. Once you merge your changes with the Pull Request, the workflow deletes your Tinybird Branch. - name: Try to delete previous Branch run: | output=$(tb --host ${{ secrets.tb_host }} --token ${{ secrets.tb_admin_token }} branch ls) BRANCH_NAME="tmp_ci_${_NORMALIZED_BRANCH_NAME}_${{ github.event.pull_request.number }}" # Check if the branch name exists in the output if echo "$output" | grep -q "\b$BRANCH_NAME\b"; then tb \ --host ${{ secrets.tb_host }} \ --token ${{ secrets.tb_admin_token }} \ branch rm $BRANCH_NAME \ --yes else echo "Skipping clean up: The Branch '$BRANCH_NAME' doesn't exist." fi - name: Create new test Branch with data run: | tb \ --host ${{ secrets.tb_host }} \ --token ${{ secrets.tb_admin_token }} \ branch create tmp_ci_${_NORMALIZED_BRANCH_NAME}_${{ github.event.pull_request.number }} \ ${_ENV_FLAGS} Set the `_ENV_FLAGS` variable to `--last-partition --wait` to attach the most recently ingested data in the Workspace. This way, you can run the tests using the same data as in production. Alternatively, leave it empty and use fixtures. #### 4. Deploy changes to the Tinybird Branch [¶](https://www.tinybird.co/docs/about:blank#4-deploy-changes-to-the-tinybird-branch) You can push the changes in your current Pull Request to the test Branch previously created in two ways: ##### Standard deployment Use `tb deploy` if you connected your data project and Workspace [through Git](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) . This command pushes the file changes based on the result of `git diff` between the latest commit deployed to the Workspace and the current git branch HEAD commit. If your data project and Workspace aren't connected through Git, you can use `tb push --only-changes --force --yes` . This command pushes the file changes based on the result of `tb diff` between the local changes in the git branch and the remote changes in the Tinybird branch. Common `tb push` options: - `--only-changes` : Deploys the changed datafiles and its dependencies. - `--force` : Overrides any existing Pipe. - `--yes` : Confirms any alter to a Data Source. - `--no-check` : Avoid running regression tests when overwriting a Pipe Endpoint. ##### Custom deploy command Alternatively, for more complex changes, you can decide how to deploy the changes to the test Branch. This is convenient, for instance, if additionally to deploy the datafiles you want to automate some other data operation, such as a running a copy Pipe, truncate a Data Source, etc. For this to work, you have to place an executable shell script file in `deploy/$VERSION/deploy.sh` with the CLI commands to push the changes. `$VERSION` should be a global variable and unique to the current active Pull Request. You can find it in the `.tinyenv` file in the data project. - name: Deploy changes to the test Branch run: | DEPLOY_FILE=./deploy/${VERSION}/deploy.sh if [ ! -f "$DEPLOY_FILE" ]; then echo "$DEPLOY_FILE not found, running default tb deploy command" tb deploy fi - name: Custom deployment to the test Branch run: | DEPLOY_FILE=./deploy/${VERSION}/deploy.sh if [ -f "$DEPLOY_FILE" ]; then echo "$DEPLOY_FILE found" if ! [ -x "$DEPLOY_FILE" ]; then echo "Error: You don't have permission to execute '$DEPLOY_FILE'. Run:" echo "> chmod +x $DEPLOY_FILE" echo "and commit your changes" exit 1 else $DEPLOY_FILE fi fi #### 5. Run the tests [¶](https://www.tinybird.co/docs/about:blank#5-run-the-tests) You can now run your test suite. This is an optional step but recommended if you want to make sure everything works as expected. Tinybird provides three type of tests by default, but you can include any test needed for your deployment pipeline: - Data fixture tests: These test specific business logic based on fixture data, see `datasources/fixtures` . - Data quality tests: These test precise data scenarios. - Regression tests: These test that requests to your API Endpoints are still working as expected. For these tests to work, you must attach production data using the `--last-partition` flag when creating the test Branch. To learn more about testing Tinybird data projects, refer to the [Implementing test strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies) docs. - name: Get regression labels id: regression_labels uses: SamirMarin/get-labels-action@v0 with: github_token: ${{ secrets.GITHUB_TOKEN }} label_key: regression - name: Run Pipe regression tests run: | source .tinyenv echo ${{ steps.regression_labels.outputs.labels }} REGRESSION_LABELS=$(echo "${{ steps.regression_labels.outputs.labels }}" | awk -F, '{for (i=1; i<=NF; i++) if ($i ~ /^--/) print $i}' ORS=',' | sed 's/,$//') echo ${REGRESSION_LABELS} CONFIG_FILE=./tests/regression.yaml BASE_CMD="tb branch regression-tests" LABELS_CMD="$(echo ${REGRESSION_LABELS} | tr , ' ')" if [ -f ${CONFIG_FILE} ]; then echo "Config file found: ${CONFIG_FILE}" ${BASE_CMD} -f ${CONFIG_FILE} --wait ${LABELS_CMD} else echo "Config file not found at '${CONFIG_FILE}', running with default values" ${BASE_CMD} coverage --wait ${LABELS_CMD} fi - name: Append fixtures run: | if [ -f ./scripts/append_fixtures.sh ]; then echo "append_fixtures script found" ./scripts/append_fixtures.sh fi - name: Run fixture tests run: | if [ -f ./scripts/exec_test.sh ]; then ./scripts/exec_test.sh fi - name: Run data quality tests run: | tb test run -v -c 4 You can find the reference `append_fixtures` and `exec_test` scripts in [this repository](https://github.com/tinybirdco/ci/tree/main/scripts). #### 6. Delete the Branch [¶](https://www.tinybird.co/docs/about:blank#6-delete-the-branch) By default, the workflow doesn't delete Branches until it's merged into the main Workspace. The following step runs after the tests: - name: Try to delete previous Branch run: | output=$(tb --host ${{ secrets.tb_host }} --token ${{ secrets.tb_admin_token }} branch ls) BRANCH_NAME="tmp_ci_${_NORMALIZED_BRANCH_NAME}_${{ github.event.pull_request.number }}" # Check if the branch name exists in the output if echo "$output" | grep -q "\b$BRANCH_NAME\b"; then tb \ --host ${{ secrets.tb_host }} \ --token ${{ secrets.tb_admin_token }} \ branch rm $BRANCH_NAME \ --yes else echo "Skipping clean up: The Branch '$BRANCH_NAME' doesn't exist." fi You can have up to simultaneous 3 Branches per Workspace at any time. Contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) if you need to increase this limit. ## How continuous deployment works [¶](https://www.tinybird.co/docs/about:blank#how-continuous-deployment-works) Once a Pull Request passes CI and a peer reviews and approves it, it's time to merge it to your main Git branch. Continuous Deployment or sometimes Continuous Delivery automatically deploys changes to the Workspace. While efficient, this workflow comes with several challenges, most of them related to handling the current state of your Tinybird Workspace. For instance: - As opposed to when you deploy a stateless app, deployments to a Workspace are incremental, based on the previous resources in the Workspace. - Resources or operations that run programatically and deal with handling state: populating operations or permission handling. - Performing deployments in the same Workspace; you need to be aware of this and implement a policy to avoid collisions from different Pull Requests deploying at the same time, or regressions. As deployments rely on Git commits to push resources, your Branches **must not** be out-of-date when merging. Use your Git provider to control branch freshness. The CD workflow explained here is a guide relevant to many of the most common use cases. However, some complex deployments may require additional knowledge and expertise from the team deploying the change. Continuous Deployment helps with: - Correctness: Ensuring you can push your changes to a Tinybird Workspace. - Deployment: Deploying the changes to the Workspace automatically. - Data Operations: Centralizing data operations required after pushing resources to the Workspace. The following section uses the generated CD template, GitHub Actions, and the Tinybird CLI to explain how to deploy Pull Request changes after merging. ### Configure the CD job [¶](https://www.tinybird.co/docs/about:blank#configure-the-cd-job) ##### CD workflow name: Tinybird - CD Workflow on: workflow_dispatch: push: branches: - main jobs: cd: # deploy changes to Workspace 'Web Analytics template' uses: tinybirdco/ci/.github/workflows/cd.yml@main with: data_project_dir: . secrets: tb_admin_token: ${{ secrets.TB_ADMIN_TOKEN }} # set Workspace admin Token in GitHub secrets tb_host: https://api.tinybird.co You can use this YAML workflow file and store it in `/my_repo/.github/workflows` . This workflow deploys on merge to `main` to the Workspace defined by the `TB_ADMIN_TOKEN` set as secret in the GitHub repository's Settings. If your data project directory isn't in the root of the Git repository, change the `data_project_dir` variable. About secrets: - `tb_host` : The URL of the region you want to use. - `tb_admin_token` : The Workspace admin Token. This grants all the permissions for a specific Workspace. You can find more information in the[ Tokens docs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . ### The CD workflow [¶](https://www.tinybird.co/docs/about:blank#the-cd-workflow) The CD pipeline deploys the changes to the main Workspace the same way the CI pipeline deploys them to the Tinybird Branch. Run CD workflow on merging a PR to keep your Workspace in sync with the git repository main branch HEAD commit. The CD workflow performs the following steps: 1. Configuration 2. Install the Tinybird CLI 3. Checks authentication 4. Pushes changes 5. Post-deployment #### 0. Workflow configuration [¶](https://www.tinybird.co/docs/about:blank#0-workflow-configuration) Same as the CI workflow. #### 1. Install the Tinybird CLI and check authentication [¶](https://www.tinybird.co/docs/about:blank#1-install-the-tinybird-cli-and-check-authentication) Worflow actions use the Tinybird CLI to interact with your Workspace. You can run this workflow locally by having a local data project and the CLI authenticated to your Tinybird Workspace. This step is equivalent to, but not identical to, the CI workflow step 1. - name: Install Tinybird CLI run: | if [ -f "requirements.txt" ]; then pip install -r requirements.txt else pip install tinybird-cli fi - name: Tinybird version run: tb --version - name: Check auth run: tb --host ${{ secrets.tb_host }} --token ${{ secrets.tb_admin_token }} auth info #### 2. Deploy changes [¶](https://www.tinybird.co/docs/about:blank#2-deploy-changes) Use the same exact strategy that you used in CI. If you did automatic deployment through git, then use `tb deploy` , otherwise `tb push --only-changes --force`. If you did a custom deployment for this specific PR make sure the same exact script runs in CD. - name: Deploy changes to the main Workspace run: | DEPLOY_FILE=./deploy/${VERSION}/deploy.sh if [ ! -f "$DEPLOY_FILE" ]; then echo "$DEPLOY_FILE not found, running default tb push command" tb deploy fi - name: Custom deployment to the main Workspace run: | DEPLOY_FILE=./deploy/${VERSION}/deploy.sh if [ -f "$DEPLOY_FILE" ]; then echo "$DEPLOY_FILE found" if ! [ -x "$DEPLOY_FILE" ]; then echo "Error: You don't have permission to execute '$DEPLOY_FILE'. Run:" echo "> chmod +x $DEPLOY_FILE" echo "and commit your changes" exit 1 else $DEPLOY_FILE fi fi ## Other git providers [¶](https://www.tinybird.co/docs/about:blank#other-git-providers) Most git vendors provide a way to run CI/CD pipelines. This example here is a guide on how you can use the Tinybird CLI + git to build CI/CD pipelines with GitHub actions, but you can create similar pipelines with other providers or adapt your own pipelines to support CI/CD to a Tinybird Workspace. You can find the a similar workflow for GitLab [here](https://github.com/tinybirdco/ci/blob/main/.gitlab/README.md). --- URL: https://www.tinybird.co/docs/work-with-data/organize-your-work/branches Last update: 2025-01-28T07:55:04.000Z Content: --- title: "Branches · Tinybird Docs" theme-color: "#171612" description: "Branches allow you to create an isolated copy of your Tinybird project, make changes and develop new features, then merge those changes back into the Workspace." --- # Branches [¶](https://www.tinybird.co/docs/about:blank#branches) ## What's a Branch? [¶](https://www.tinybird.co/docs/about:blank#whats-a-branch) Branches are isolated, temporary copies of your Tinybird project and the resources it contains (including [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources), [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes), [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints) ). They are inspired by Git branches, allowing you to make changes and develop new features, then merge those changes back into the Workspace. Tinybird represents branches using the icon. ## What should you use Branches for? [¶](https://www.tinybird.co/docs/about:blank#what-should-you-use-branches-for) Use Branches to develop new features without affecting production or other developers. Tinybird Branches mirror the intent behind Git branches, and you can use them for the same reasons. Branches are temporary, rather than long lived, so they're ideal for building new features and prototyping. Remove Branches when their work is complete, either merged or deleted. Common uses for Branches include: - Designing new Data Sources or Pipes. - Iterating the schema of a Data Source. - Testing the backfill strategy of a new Data Source. - Changing the output of a Pipe. - Modifying API Endpoint query parameters. Branches are temporary. If you need a long-lived non-production environment, take a look at the docs on [staging and production Workspaces](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/staging-and-production-workspaces). ## Data in Branches [¶](https://www.tinybird.co/docs/about:blank#data-in-branches) Creating a Branch copies all the **resources** from the Workspace to the new Branch, but it doesn't copy all the **production data**. When creating a Branch, you can select to copy a segment of data from production into the new Branch. This duplicates the latest active partition up to a maximum of 50 GB of data from all production Data Sources, and attaches the copies to the new Branch Data Sources. This data is a copy and is physically separate from the production data. This means you can ingest or drop data from the Branch and production data isn't affected. If you select not to copy data, the new Branch contains no data, and you need to ingest data into the Branch Data Sources. When you delete a Branch, the Branch data is also deleted. There is a soft limit on the number of Branches you can create per [Workspace](https://www.tinybird.co/docs/docs/get-started/plans/limits#workspace-limits) . Contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) if you need to increase this limit. ## Branch users [¶](https://www.tinybird.co/docs/about:blank#branch-users) When you create a Branch, Tinybird adds all Workspace users to the new Branch with their current roles. If you don't see a user in the Branch, it's probably because the user joined the Workspace after creating the Branch. Adding or removing members is only available in main Branch. Although users with the Viewer role can't modify a Workspace, they can modify Branches of which they're members. ## Branch Tokens [¶](https://www.tinybird.co/docs/about:blank#branch-tokens) When creating a new Branch, all [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) from the Workspace mirror in the new Branch. These are **new** Tokens that are completely independent of the production Tokens, but with the same names and scopes. This means you can delete or update the scopes of the Tokens in the new Branch without affecting production. When you are on a branch in the CLI using the command `tb branch use BRANCH_NAME_OR_ID` the administrator token of the branch is on the [.tinyb file](https://www.tinybird.co/docs/docs/cli/install#authentication) . If you need this token for CI/CD processes, you can parse it from this auth file. ## Create a Branch [¶](https://www.tinybird.co/docs/about:blank#create-a-branch) You can create a Branch via the Tinybird UI, Tinybird CLI, or automatically as part of your CI/CD process. ### Using the Tinybird UI [¶](https://www.tinybird.co/docs/about:blank#using-the-tinybird-ui) To create a new Branch with the Tinybird UI, select the `Branch` dropdown in the top bar near `main` , and select `New Branch` . Type the Branch name and select the `Create Branch` button. ### Using the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-tinybird-cli) To create a new Branch with the Tinybird CLI, use the `tb branch create` command. See the [tb branch Command Reference](https://www.tinybird.co/docs/docs/cli/command-ref#tb-branch) for more information. ### Using a pull request [¶](https://www.tinybird.co/docs/about:blank#using-a-pull-request) To create a new Branch with a Pull Request (PR), push your local feature branch to your Git provider and open a new PR. The CI/CD actions should sync the branch from your PR to Tinybird. ## Delete a Branch [¶](https://www.tinybird.co/docs/about:blank#delete-a-branch) You can delete a Branch via the Tinybird UI, Tinybird CLI, or automatically as part of your CI/CD process. ### Using the Tinybird UI [¶](https://www.tinybird.co/docs/about:blank#using-the-tinybird-ui) To delete a Branch with the Tinybird UI, select the Branch you want to delete in the top bar. Then select `Settings` and from that menu select `Advanced Settings` . Verify you have selected the correct Branch and select the red `Delete Branch` button. You must verify this operation and select `Delete` again. ### Using the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-tinybird-cli) To delete a Branch with the Tinybird CLI, use the `tb branch rm` command. See the [tb branch Command Reference](https://www.tinybird.co/docs/docs/cli/command-ref#tb-branch) for more information. ### Using a pull request [¶](https://www.tinybird.co/docs/about:blank#using-a-pull-request) To delete a Branch from a Pull Request (PR), close or merge the associated PR in your Git provider. The automated CI/CD actions delete the Branch from Tinybird. ## Limitations [¶](https://www.tinybird.co/docs/about:blank#limitations) **JOIN Engine Data Sources** Data Sources using the JOIN Engine aren't fully supported in Branches. You can create a Branch with JOIN Engine Data Sources, but the data isn't copied into the new Branch. Tinybird deprecated the JOIN Engine and plans to remove it in a future release. You shouldn't create new JOIN Engine Data Sources. If you need support, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). --- URL: https://www.tinybird.co/docs/work-with-data/optimization/opt201-fix-mistakes Last update: 2025-01-17T07:58:01.000Z Content: --- title: "How to fix common mistakes · Tinybird Docs" theme-color: "#171612" description: "In this guide, you'll learn more tips on how to optimize your Tinybird project." --- # Optimizations 201: Fix common mistakes [¶](https://www.tinybird.co/docs/about:blank#optimizations-201-fix-common-mistakes) In this guide, you'll learn the top 5 questions to ask yourself and fix common pitfalls in your data project. ## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites) You'll need to read [Optimizations 101: Detect inefficiencies](https://www.tinybird.co/docs/opt101-detect-inefficiencies) first. It'll give you an idea of which Pipe is the worst performing, and the particular characteristics of that performance, so you can start to look into common causes. It's also a really good idea to read [Best practices for faster SQL](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) and the [Thinking in Tinybird](https://www.tinybird.co/blog-posts/thinking-in-tinybird) blog post. This guide walks through 5 common questions (the "usual suspects"). If you find a badly-performing Pipe, ask yourself these 5 questions first before exploring other, more nuanced problem areas. ## 1. Are you aggregating or transforming data at query time? [¶](https://www.tinybird.co/docs/about:blank#1-are-you-aggregating-or-transforming-data-at-query-time) Calculating the `count()`, `sum()` , or `avg()` , or casting to another data type is common in a published API Endpoint. **As your data scales, you may be processing more data than necessary** , as you run the same query each time there is a request to that API Endpoint. If this is the case, you should create a [Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views). In a traditional database, you have to schedule Materialized Views to run on a regular cadence. Although this helps pre-process large amounts of data, the batch nature renders your data stale. In Tinybird, **Materialized Views let you incrementally pre-aggregate, transform, and filter large Data Sources upon ingestion** . By shifting the computational load from query time to ingestion time, you'll scan less data and keep your API Endpoints fast. Read the docs to [create a Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views#creating-a-materialized-view-in-the-tinybird-ui). ## 2. Are you filtering by the fields in the sorting key? [¶](https://www.tinybird.co/docs/about:blank#2-are-you-filtering-by-the-fields-in-the-sorting-key) The sorting key is important. **It determines the way data is indexed and stored in your Data Source** , and is crucial for the performance of your API Endpoints. Setting the right sorting key allows you to have all the data you need, for any given query, as close as possible. In all databases (including Tinybird), indexing allows you to not read the data that you don't need, which speeds up some operations (like filtering) hugely. The goal of sorting keys (SKs) is to reduce scan size and discard as much data as possible when examining the `WHERE` clauses in the queries. In short, a good Sort Key is what will help you [avoid expensive, time-consuming full scans](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices#avoid-full-scans). Some good rules of thumb for setting Sorting Keys: - Order matters: Data will be stored based on the Sort Key order. - Between 3 and 5 columns is good enough. More will probably penalize. - `timestamp` is often a** bad candidate for being the first element** of the SK. - If you have a multi-tenant app, `customer_id` is a** great candidate for being the first element** of the SK. One common mistake is to use the [partition key](https://www.tinybird.co/docs/docs/get-data-in/data-sources#partitioning) for filtering. Use the sorting key, not the partition key. ## 3. Are you using the best data types? [¶](https://www.tinybird.co/docs/about:blank#3-are-you-using-the-best-data-types) If you do need to read data, you should try to use the smallest types that can get the job done. A common examples are timestamps. Do you really need millisecond precision? Often when users start doing data analytics, they aren't sure what the data will look like, and how to query it in their application. But now that you have your API Endpoint or Pipe, you can (and should!) go back and see if your schema best supports your resulting use-case. It's common to use *simple* types to begin, like `String`, `Int` , and `DateTime` , but as you go further in the application implementation, you should review the data types you selected at the beginning. When reviewing your data types, focus on the following points: - ** Downsizing types** , to select a different data type with a lower size. For instance, UUID fields can be typed as UUID fields instead of string types, you can use unsigned integers (UInt) instead of integers (Int) where there aren't negative values or you could use a Date instead of DateTime. - Examine string** cardinality** to perhaps use `LowCardinality()` if there are less than 100k uniques. - ** Nullable** columns are** bigger and slower** and can't be sorting keys, so use `coalesce()` . **Sorting key** and **data type** changes are done by changing your schema, which means [iterating the Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source) . See an example of these types of changes in the `thinking-in-tinybird` demo repo. ## 4. Are you doing complex operations early in the processing pipeline? [¶](https://www.tinybird.co/docs/about:blank#4-are-you-doing-complex-operations-early-in-the-processing-pipeline) Operations such as joins or aggregations get increasingly expensive as your data grows. Filter your data first to reduce the number of rows, then **perform the more complex operations later in the pipeline**. Follow this example: [Rule 5 for faster SQL](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices#perform-complex-operations-later-in-the-processing-pipeline). ## 5. Are you joining two or more data sources? [¶](https://www.tinybird.co/docs/about:blank#5-are-you-joining-two-or-more-data-sources) It's a common scenario to want to **enrich your events with some dimension tables** , so that you are materializing a JOIN. This kind of approach could process more data than you need, so here are some tips to follow in order to reduce it: - Try to switch out JOINs and replace them with a** subquery** : `WHERE column IN (SELECT column FROM dimensions)` . - If the join is needed, try to** filter the right table first** (better if you can use a field in the sorting key). - Remember that the** Materialization is only triggered when you ingest data in the left Data Source** (the one you use to do a `SELECT … FROM datasource` ). So, if you need to recalculate data from the past, creating a Materialized View isn't the right approach. Instead, check this guide about[ Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) . ### Understanding the Materialized JOIN issue [¶](https://www.tinybird.co/docs/about:blank#understanding-the-materialized-join-issue) #### The issue [¶](https://www.tinybird.co/docs/about:blank#the-issue) There's a [common pitfall](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views#limitations) when working with Materialized Views: *Materialized Views generated using JOIN clauses are tricky. The resulting Data Source will be only automatically updated if and when a new operation is performed over the Data Source in the FROM.* Since Materialized Views work with the result of a SQL query, you can use JOINs and any other SQL feature. But JOINs should be used with caution. SELECT a.id, a.value, b.value FROM a LEFT JOIN b USING id If you insert data in `a` (LEFT SIDE), data will be processed as expected... but what happens if you add data to `b` (RIGHT SIDE)? It will not be processed. This is because a Materialized View is only triggered when its source table receives inserts. It's just a trigger on the source table and knows nothing about the joined table. Note that this doesn't only apply to JOIN queries, and is relevant when introducing **any** table external to the Materialized View's SELECT statement, e.g. using a IN SELECT. It can become more complex if you need to deal with [stream joins](https://github.com/tinybirdco/streaming_join_demo) . However, this guide focuses on the basic setup as doing JOINs implies something most people don't realize. These JOINs can be very expensive because you're reading the small amount of rows being ingested (LEFT SIDE) plus a full scan of the joined table (RIGHT SIDE) as you don't know anything about it. #### The optimization [¶](https://www.tinybird.co/docs/about:blank#the-optimization) Sometimes, to easily detect these cases, it's useful to review the `read_bytes` / `write_bytes` ratio. If you're reading way more than writing, most likely you're doing some JOINs within the MV. You can easily change this by **adding a filter in the right side** , rewriting the previous query as follows: SELECT a.id, a.value, b.value FROM a LEFT JOIN ( SELECT id, value FROM b WHERE b.id IN (SELECT id FROM a) ) b USING id This might sound counter-intuitive when writing a query the first time because you're basically reading `a` twice. However, you need to think `a` is usually smaller than `b` because you're only reading the block of rows you're ingesting. To see a real improvement, you will need the fields you're using to filter to be in the sorting key of `b` . Most of the time you'll use the "joining key", but you can use any other potential field that allows you to hit the index in `b` and filter the right side of the JOIN. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Check out the Monitoring docs and guides for more tips, like[ using Time Series to analyze patterns](https://www.tinybird.co/docs/docs/monitoring/latency#time-series) . - Explore[ this example repo](https://github.com/tinybirdco/tb-usage-cost-metrics/tree/main) to analyze Processed Data. It may not be 100% accurate to[ billing](https://www.tinybird.co/docs/docs/get-started/plans/billing) , as Tinybird tracks certain operations differently in Service Data Sources, but it's a great proxy. --- URL: https://www.tinybird.co/docs/work-with-data/optimization/opt101-detect-inefficiencies Last update: 2025-01-20T11:46:12.000Z Content: --- title: "How to detect inefficient resources · Tinybird Docs" theme-color: "#171612" description: "In this guide, you'll learn where to look when looking to optimize your Tinybird project." --- # Optimizations 101: Detect inefficient resources [¶](https://www.tinybird.co/docs/about:blank#optimizations-101-detect-inefficient-resources) This guide shows you how to find your most inefficient resources - typically the slowest API Endpoints, and the ones processing the most data. ## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites) You don't need an active Workspace to understand this guide, but it can be useful to have one open to compare and apply the ideas to your own situation. ## 1. Orient yourself with the Overview [¶](https://www.tinybird.co/docs/about:blank#1-orient-yourself-with-the-overview) First, navigate to the Overview page of your Workspace: <-figure-> ![Overview of a Workspace showing stats about Pipes performance](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-overview.png&w=3840&q=75) <-figcaption-> Workspace Overview page To find the slowest API Endpoints: **Set the period to show performance information using the top right dropdown:** - Try to cover a large period for a good average. - If you've made changes recently, try to only include the period for the current version, to make sure you're analyzing the performance of the new version. **Then use the different Consumption summaries to examine your usage:** - BI Connector (if you're using it). - Query API for direct API queries. - A separate section for Pipes. **Check the dedicated Pipes section:** - In Pipes, you get a view of the total number of requests over the period, the average latency, the total data processed for that Pipe, and the average data processed. - Sort by `Processed` to bring the Pipes processing the most data over the period to the top. This effectively shows which Pipe is the "most expensive" and gives you a list of where to start your investigations. Additionally, you can sort by `avg. latency` if you're interested in finding the slowest Endpoints. Consider that the column `requests` value can help you decide which Pipes to optimize, because you could ignore Pipes with few requests. - Finally, select the name of a Pipe to open it in the UI for further investigation. ## 2. Analyze a Pipe Endpoint [¶](https://www.tinybird.co/docs/about:blank#2-analyze-a-pipe-endpoint) When you open up a Pipe in the UI, you have a few different options to gain insights into its performance: <-figure-> ![Pipe page with a spotlight on View API and performance metrics](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-pipe-analysis-spotlight.png&w=3840&q=75) <-figcaption-> Pipe page: View API and performance metrics The **View API** button (top right corner) takes you to the Pipe's performance and testing page. Under each node in the Pipe, there's **performance information** about the query that node is running (processed bytes, rows, columns, time). Note that these node performance metrics are from executing the query with a `LIMIT 20` , so may give you smaller results unless the query reads all the rows for processing. You can modify the query in the node to force it to process all the data, such as by hashing a column or forcing a count of all rows. By doing this, you'll have a clear indication of data processed and time execution. If you select **Explain** under any Node, you'll see how the nodes are integrated into the query that is run through the engine: <-figure-> ![Pipe page](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-pipe-analysis-explain.png&w=3840&q=75) <-figcaption-> Pipe page This can be a very good way to spot if you're making a common mistake, such as not filtering your data before you aggregate it. These common mistakes will be explained later. The node with the tick icon is the one published as the Endpoint: The performance metrics under this node are still shown with `LIMIT 20` . Open the **View API** button (or just select the tick icon) to bring up the Endpoint performance page for further analysis. ## 3. Use the View API Performance page [¶](https://www.tinybird.co/docs/about:blank#3-use-the-view-api-performance-page) Coming from the **View API** button from the Pipe page, you'll go to the Endpoint page. <-figure-> ![Endpoint page](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-endpoint-page.png&w=3840&q=75) <-figcaption-> Endpoint page Here you can see the specific performance metrics of this Pipe as it's called by your application. This is a good place to see if there's a lot of variance in requests (in other words, to check if the performance of the Endpoint changes with time for any reason). Towards the bottom of that page, you see the sample snippets: <-figure-> ![Endpoint snippet](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-endpoint-snippet.png&w=3840&q=75) <-figcaption-> Endpoint snippet You can copy this sample code and run it in a new browser tab. This executes the query exactly as your application would, and the performance metrics will be recorded and reported to you on the Endpoint page. This allows you to see the exact impact of any changes you make to your Pipe. ## 4. Use params to find common Pipe usage patterns [¶](https://www.tinybird.co/docs/about:blank#4-use-params-to-find-common-pipe-usage-patterns) You'll need to be familiar with [dynamic parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) and [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) for the next 2 sections. Once you have a dynamic API Endpoint with parameters in use, it can be really useful to observe common parameter patterns from the application (for instance, which parameters and values force the Endpoints to process more data). You can then use these observations to drive new optimizations on the Pipe, such as ensuring that a column that is commonly used as parameter is in the sorting key. Use the `pipe_stats` and `pipe_stats_rt` [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . You can also explore some common Pipe usage queries in the [Monitor API performance guide](https://www.tinybird.co/docs/docs/monitoring/analyze-endpoints-performance). Specifically, you can execute the following query to get the number of requests, average processed data, and average duration of your API Endpoints and parameters in the last 24 hours: ##### check pipe_stats_rt SELECT pipe_name, parameters, count() AS total_requests, formatReadableSize(avg(read_bytes)) AS avg_read, avg(duration) AS avg_duration FROM tinybird.pipe_stats_rt WHERE start_datetime > now() - INTERVAL 1 DAY GROUP BY pipe_name, parameters ORDER BY total_requests DESC ## 5. Measure Materialized Views and Copy Pipes performance [¶](https://www.tinybird.co/docs/about:blank#5-measure-materialized-views-and-copy-pipes-performance) You should also check the Pipes that move data from one Data Source to another, like Materialized Views and Copy Pipes. You can track them in `tinybird.datasources_ops_log` and in the Materialized View page or Copy page. <-figure-> ![Materialized View page](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Foptimiz101-mv-log.png&w=3840&q=75) <-figcaption-> Materialized View page ##### check processed data for data source operations SELECT datasource_name, formatReadableSize(sum(read_bytes) + sum(written_bytes)) AS f_processed_data, formatReadableSize(sum(read_bytes)) AS f_read_bytes, formatReadableSize(sum(written_bytes)) AS f_written_bytes, round(sum(read_bytes) / sum(written_bytes), 2) AS f_ratio FROM tinybird.datasources_ops_log WHERE timestamp > now() - INTERVAL 1 DAY AND pipe_name != '' GROUP BY datasource_name ORDER BY sum(read_bytes) + sum(written_bytes) DESC Pay special attention to **Materialized Views with JOINs** , since they are prone to scan more data than needed if your SQL isn't optimized. Basically, JOINs should be with subqueries of pre-filtered data. ## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps) - Read[ Optimizations 201: Fix common mistakes](https://www.tinybird.co/docs/docs/work-with-data/optimization/opt201-fix-mistakes) . - Check out the Monitoring docs and guides for more tips, like[ using Time Series to analyze patterns](https://www.tinybird.co/docs/docs/monitoring/latency#time-series) . - Explore[ this example repo](https://github.com/tinybirdco/tb-usage-cost-metrics/tree/main) to analyze Processed Data. It may not be 100% accurate to[ billing](https://www.tinybird.co/docs/docs/get-started/plans/billing) , as Tinybird tracks certain operations differently in Service Data Sources, but it's a great proxy. --- URL: https://www.tinybird.co/docs/sql-reference/engines/versionedcollapsingmergetree Last update: 2025-01-23T08:42:52.000Z Content: --- title: "VersionedCollapsingMergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the VersionedCollapsingMergeTree engine for collapsing rows." --- # VersionedCollapsingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#versionedcollapsingmergetree-engine) This engine allows to quickly write object states that are continually changing and deletes old object states in the background. This significantly reduces the volume of storage. The engine inherits from MergeTree and adds the logic for collapsing rows to the algorithm for merging data parts. `VersionedCollapsingMergeTree` serves the same purpose as [CollapsingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/collapsingmergetree) but uses a different collapsing algorithm that allows inserting the data in any order with multiple threads. ## Engine parameters [¶](https://www.tinybird.co/docs/about:blank#engine-parameters) You can use the following parameters to configure the engine. ### sign [¶](https://www.tinybird.co/docs/about:blank#sign) `ENGINE_SIGN` is the name of the column with the type of row: `1` is a “state” row, `-1` is a “cancel” row. The column data type should be `Int8`. ### version [¶](https://www.tinybird.co/docs/about:blank#version) `ENGINE_VERSION` is the name of the column with the version of the object state. The column data type should be `UInt*`. ## Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) ##### vcmt.datasource SCHEMA > UserID UInt64 `json:$.UserID`, PageViews UInt8 `json:$.PageViews`, Duration UInt8 `json:$.Duration`, Sign Int8 `json:$.Sign`, Version UInt16 `json:$.Version` ENGINE "CollapsingMergeTree" ENGINE_SORTING_KEY "UserID" ENGINE_SIGN "Sign" ENGINE_VERSION "Version" tb push datasources/vcmt.datasource TB_HOST=$(cat .tinyb | jq ".host" -r) TB_TOKEN=$(cat .tinyb | jq ".token" -r) curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d '{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": 1, "Version": 1}' \ "$TB_HOST/v0/events?name=vcmt" tb sql "select * from vcmt" --------------------------------------------------------------- | UserID | PageViews | Duration | Sign | Version | --------------------------------------------------------------- | 4324182021466249494 | 5 | 146 | 1 | 1 | --------------------------------------------------------------- Let's add another row with the same `UserID` and negate the previous row (same row with -1 as `Sign` value). `{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": -1, "Version": 2}` curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d $'{"UserID": 4324182021466249494, "PageViews": 6, "Duration": 185, "Sign": 1, "Version": 2}\n{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": -1, "Version": 1}\n' \ "$TB_HOST/v0/events?name=vcmt" tb sql "select * from vcmt" --------------------------------------------------------------- | UserID | PageViews | Duration | Sign | Version | --------------------------------------------------------------- | 4324182021466249494 | 5 | 146 | 1 | 1 | | 4324182021466249494 | 6 | 185 | 1 | 2 | | 4324182021466249494 | 5 | 146 | -1 | 1 | --------------------------------------------------------------- What happened here? Collapsing happens during merges, which are performed asynchronously and can't be controlled, so you should force it using `FINAL` or query using aggregation. tb sql "select * from vcmt final" --------------------------------------------------------------- | UserID | PageViews | Duration | Sign | Version | --------------------------------------------------------------- | 4324182021466249494 | 6 | 185 | 1 | 2 | --------------------------------------------------------------- tb sql "SELECT UserID, sum(PageViews * Sign) AS PageViews, sum(Duration * Sign) AS Duration, Version FROM vcmt GROUP BY UserID, Version HAVING sum(Sign) > 0 ORDER BY Version DESC LIMIT 1 BY UserID" -------------------------------------------------------- | UserID | PageViews | Duration | Version | -------------------------------------------------------- | 4324182021466249494 | 6 | 185 | 2 | -------------------------------------------------------- ## Query clauses [¶](https://www.tinybird.co/docs/about:blank#query-clauses) When creating a `VersionedCollapsingMergeTree` table, the same clauses as when creating a `MergeTree` table are required. ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/engines/summingmergetree Last update: 2025-01-07T15:48:10.000Z Content: --- title: "SummingMergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the SummingMergeTree engine for summarizing rows." --- # SummingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#summingmergetree-engine) The engine inherits from MergeTree. The difference is that when merging data parts for `SummingMergeTree` Data Sources, the database replaces all the rows with the same primary key with one row which contains summarized values for the columns with the numeric data type. If the sorting key is composed in a way that a single key value corresponds to large number of rows, this significantly reduces storage volume and speeds up data selection. Remember that, if not specified, the `ENGINE_PRIMARY_KEY` matches the `ENGINE_SORTING_KEY`. Use the SummingMergeTree engine together with `MergeTree` . Store complete data in `MergeTree` table, and use `SummingMergeTree` for aggregated data storing. Such an approach prevents you from losing valuable data due to an incorrectly composed primary key. [AggregatingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/aggregatingmergetree) is generally a better choice because it allows more operations and behaves similarly to SummingMergeTree. ## Query clauses [¶](https://www.tinybird.co/docs/about:blank#query-clauses) When creating a `SummingMergeTree` table, the same clauses as when creating a `MergeTree` table are required. ## Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) ##### summing_merge_tree.datasource SCHEMA > `key` UInt32, `value` UInt32 ENGINE "SummingMergeTree" ENGINE_SORTING_KEY "key" $ cat fixtures/summt.csv key,value 1,1 1,2 2,1 $ tb datasource append summing_merge_tree fixtures/summt.csv $ tb sql "SELECT key, sum(value) FROM summing_merge_tree GROUP BY key" ┌─key─┬─sum(value)─┐ │ 1 │ 3 │ │ 2 │ 1 │ └─────┴────────────┘ ## Data Processing [¶](https://www.tinybird.co/docs/about:blank#data-processing) When rows are inserted into a Data Source, they are saved as-is. SummingMergeTree engine merges the inserted parts of data periodically and this is when rows with the same primary key are summed and replaced with one for each resulting part of data. Merge process is async and trigger can't be controlled, i.e. the summation will be incomplete. Therefore when querying, the aggregate function `sum()` and `GROUP BY` clause should be used as described in the example above. ### Common Rules for Summation [¶](https://www.tinybird.co/docs/about:blank#common-rules-for-summation) The values in the columns with the numeric data type are summarized. If the values were 0 in all of the columns for summation, the row is deleted. If the column isn't in the primary key and isn't summarized, an arbitrary value is selected from the existing ones. The values aren't summarized for columns in the primary key. ### Summation in the Aggregatefunction Columns [¶](https://www.tinybird.co/docs/about:blank#summation-in-the-aggregatefunction-columns) For columns of the [AggregateFunction type](https://www.tinybird.co/docs/docs/sql-reference/data-types/aggregatefunction) , SummingMergeTree engine behaves as the [AggregatingMergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/aggregatingmergetree) engine, aggregating according to the function. ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/engines/replacingmergetree Last update: 2025-01-07T15:48:10.000Z Content: --- title: "ReplacingMergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the ReplacingMergeTree engine for deduplicating rows." --- # ReplacingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#replacingmergetree-engine) The engine differs from [MergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/mergetree) in that it removes duplicate entries with the same sorting key value. Data deduplication occurs only during a merge. Merging occurs in the background at an unknown time, so you can’t plan for it. Some data may remain unprocessed. `ReplacingMergeTree` is suitable for clearing out duplicate data in the background to save space, but it doesn't guarantee the absence of duplicates. Check [Deduplication strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies) for more information about how to handle duplicates. ## Creating a ReplacingMergeTree Data Source [¶](https://www.tinybird.co/docs/about:blank#creating-a-replacingmergetree-data-source) ##### post_views_rmt.datasource DESCRIPTION > Data Source to save latest version of post info. ReplacingMergeTree Engine. SCHEMA > `post_id` Int32 `json:$.post_id`, `views` Int32 `json:$.views`, `likes` Int32 `json:$.likes`, `tag` String `json:$.tag`, `timestamp` DateTime `json:$.timestamp`, `_is_deleted` UInt8 `json:$._is_deleted` DEFAULT 0 ENGINE "ReplacingMergeTree" ENGINE_PARTITION_KEY "" ENGINE_SORTING_KEY "post_id" ENGINE_VER "timestamp" ENGINE_IS_DELETED "_is_deleted" ## ReplacingMergeTree parameters [¶](https://www.tinybird.co/docs/about:blank#replacingmergetree-parameters) ### ENGINE_VER [¶](https://www.tinybird.co/docs/about:blank#engine-ver) `ENGINE_VER` is the column with the version number. Type `UInt*`, `Date`, `DateTime` or `DateTime64` . Optional parameter. When merging, `ReplacingMergeTree` from all the rows with the same `ENGINE_SORTING_KEY` leaves only one: - The last in the selection, if `ENGINE_VER` not set. A selection is a set of rows in a set of parts participating in the merge. The most recently created part, the last insert, is the last one in the selection. After deduplication, the last row from the most recent insert remains for each unique sorting key. - With the maximum version, if `ENGINE_VER` specified. If `ENGINE_VER` is the same for several rows, then it uses "if `ENGINE_VER` isn't specified" rule for them, i.e. the most recent inserted row remains. ### ENGINE_IS_DELETED [¶](https://www.tinybird.co/docs/about:blank#engine-is-deleted) `ENGINE_IS_DELETED` is the name of the column used during a merge to determine whether the data in this row represents the state or is to be deleted. `1` is a "deleted" row, `0` is a "state" row. `ENGINE_IS_DELETED` can only be enabled when `ENGINE_VER` is used. Note it's a mask, not an actual delete, so queries to the Data Source with `FINAL` are equivalent to queries with `WHERE = 0`. No matter the operation on the data, the version must be increased. If two inserted rows have the same version number, the last inserted row is the one kept. ## Query time de-duplication and FINAL [¶](https://www.tinybird.co/docs/about:blank#query-time-de-duplication-and-final) At merge time, the ReplacingMergeTree identifies duplicate rows, using the values of the `ENGINE_SORTING_KEY` columns as a unique identifier, and retains only the highest version. This, however, offers eventual correctness only - it doesn't guarantee rows are deduplicated. Queries can, therefore, produce incorrect answers due to update and delete rows being considered in queries. To obtain correct answers, complement background merges with query time deduplication and deletion removal. You can do this by using the `FINAL` operator. ## Example: [¶](https://www.tinybird.co/docs/about:blank#example) echo '{ "timestamp": "2024-07-02T02:22:17", "post_id": 956, "views": 856875, "likes": 2321, "tag": "Sports" }' > post_views.ndjson echo '{ "timestamp": "2024-08-02T03:22:17", "post_id": 956, "views": 956875, "likes": 3321, "tag": "Sports" }' >> post_views.ndjson echo '{ "timestamp": "2024-08-01T00:00:00", "post_id": 1, "views": 56875, "likes": 321, "tag": "Music" }' >> post_views.ndjson echo '{ "timestamp": "2024-09-01T00:00:00", "post_id": 1, "views": 56875, "likes": 321, "tag": "Music", "_is_deleted": 1 }' >> post_views.ndjson tb datasource append post_views_rmt post_views.ndjson tb sql "select * from post_views_rmt final" ------------------------------------------------------------------------- | post_id | views | likes | tag | timestamp | _is_deleted | ------------------------------------------------------------------------- | 956 | 956875 | 3321 | Sports | 2024-08-02 03:22:17 | 0 | ------------------------------------------------------------------------- ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/engines/null Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Null engine · Tinybird Docs" theme-color: "#171612" description: "Use the Null engine to ignore data." --- # Null engine [¶](https://www.tinybird.co/docs/about:blank#null-engine) When writing to a `Null` table, data is ignored. When reading from a `Null` Data Source, the response is empty. You can use the Null engine to create a Materialized View from a `Null` table that transforms the data as it's sent to the Null Data Source so the Target Data Source will receive the transformed data and the original will be discarded. Example: for masking some PII data. ##### event.datasource SCHEMA > `action` LowCardinality(String) `json:$.action`, `timestamp` DateTime64(3) `json:$.timestamp`, `version` LowCardinality(String) `json:$.version`, `userEmail` String `json:$.userEmail` DEFAULT '', `workspaceId` String `json:$.workspaceId` DEFAULT '', `payload` String `json:$.payload` ENGINE Null ##### mask_events_mat.pipe DESCRIPTION > Mask events for mat view NODE maskEvents SQL > SELECT action, timestamp, version, cityHash64(userEmail) as userEmailMasked, workspaceId, payload FROM event TYPE Materialized DATASOURCE event_masked ##### event_masked.datasource SCHEMA > `action` LowCardinality(String), `timestamp` DateTime64(3), `version` LowCardinality(String), `userEmailMasked` UInt64, `workspaceId` String , `payload` String ENGINE MergeTree ENGINE_PARTITION_KEY toYear(timestamp) ENGINE_SORTING_KEY action, workspaceId, timestamp --- URL: https://www.tinybird.co/docs/sql-reference/engines/mergetree Last update: 2025-01-07T15:48:10.000Z Content: --- title: "MergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the MergeTree engine for high data ingest rates and huge data volumes." --- # MergeTree engine [¶](https://www.tinybird.co/docs/about:blank#mergetree-engine) The `MergeTree` engine is designed for high data ingest rates and huge data volumes. Is the default engine for Data Sources. Key features: - The table's sorting key (defined by `ENGINE_SORTING_KEY` ) determines the sort order within each table part. - The sorting key doesn't reference individual rows but blocks of 8192 rows called granules. - You can partition tables using an arbitrary partition expression. Partition pruning ensures partitions are omitted from reading when the query allows it. When partitioning you should be aware that partitioning isn't the fastest way to speed up queries, in contrast to the `ENGINE_SORTING_KEY` . A partition is the minimum unit of data for async operations, but for streaming ingest you should not write to more than one or two partitions. ## Creating a MergeTree Data Source [¶](https://www.tinybird.co/docs/about:blank#creating-a-mergetree-data-source) Here's how to define a MergeTree data source: SCHEMA > `event_date` Date `json:$.date`, `event_type` LowCardinality(String) `json:$.event_type`, `user_id` UInt64 `json:$.user_id`, `payload` JSON `json:$.payload` ENGINE "MergeTree" ENGINE_SORTING_KEY "event_type, event_date, user_id" [ENGINE_PARTITION_KEY "toYYYYMM(event_date)"] [ENGINE_PRIMARY_KEY "event_type, event_date"] [ENGINE_TTL "event_date + INTERVAL 30 DAY"] [ENGINE_SETTINGS index_granularity=8192] [INDEXES > index_name index_expression TYPE index_type GRANULARITY index_granularity] ### Engine Settings [¶](https://www.tinybird.co/docs/about:blank#engine-settings) - `ENGINE_SORTING_KEY` - Defines the sorting key. This is a tuple of column names or arbitrary expressions that determines how data is sorted within each part. For example: `"event_type, event_date"` . - `ENGINE_PARTITION_KEY` - Optional. Defines the partitioning key. In most cases, you don't need a partition key, and if you do need to partition, generally you don't need a partition key more granular than by month. To partition by month, use `"toYYYYMM(event_date)"` . - `ENGINE_PRIMARY_KEY` - Optional. Defines the primary key. If not specified, the `ENGINE_PRIMARY_KEY` matches the `ENGINE_SORTING_KEY` . It can be a subset of the `ENGINE_SORTING_KEY` . Useful for Engines where you need a long sorting key for aggregation or deudplication purposes, but a shorter primary key will be better for keeping the index size smaller. - `ENGINE_TTL` - Optional. Defines rules for data expiration. The expression must result in a Date or DateTime, for example `"event_date + INTERVAL 1 DAY"` . - `ENGINE_SETTINGS` - Optional. Defines settings for the engine. For example, `"index_granularity=8192"` . - `INDEXES >` - Optional. Defines indexes for the engine. ## Data Storage Internals [¶](https://www.tinybird.co/docs/about:blank#data-storage-internals) A table consists of data parts sorted by sorting key. When data is inserted in a table, separate data parts are created and each of them is lexicographically sorted by sorting key. For example, if the sorting key is `"event_type, event_date"` , the data in the part is sorted by `event_type` , and within each `event_type` , it's ordered by `event_date`. When data is inserted in a DataSource, separate data parts are created and each of them is lexicographically sorted by sorting key. Data belonging to the same sorting key is stored in the same data part. Data belonging to different partitions are separated into different parts. In the background, the DataSource merges data parts for more efficient storage. Parts belonging to different partitions aren't merged. The merge mechanism doesn't guarantee that all rows with the same primary key will be in the same data part. Each data part is logically divided into granules. A granule is the smallest indivisible data set that Tinybird reads when selecting data. Tinybird doesn't split rows or values, so each granule always contains an integer number of rows. The first row of a granule is marked with the value of the primary key for the row. For each data part, Tinybird creates an index file that stores the marks. For each column, whether it’s in the primary key or not, Tinybird also stores the same marks. These marks let you find data directly in column files. The granule size is restricted by the `index_granularity` setting of the table engine. The number of rows in a granule lays in the `[1, index_granularity]` range, depending on the size of the rows. ### Sorting Keys and Indexes in Queries [¶](https://www.tinybird.co/docs/about:blank#sorting-keys-and-indexes-in-queries) Take the `"event_type, event_date"` sorting key as an example. In this case, the sorting and index can be illustrated as follows: Whole data: [-------------------------------------------------------------------------] event_type: [aaaaaaaaaaaaaaaaaabbbbcdeeeeeeeeeeeeefgggggggghhhhhhhhhiiiiiiiiikllllllll] event_date: [1111111222222233331233211111222222333211111112122222223111112223311122333] Marks: | | | | | | | | | | | a,1 a,2 a,3 b,3 e,2 e,3 g,1 h,2 i,1 i,3 l,3 Marks numbers: 0 1 2 3 4 5 6 7 8 9 10 If the data query specifies: - `event_type in ('a', 'h')` , the server reads the data in the ranges of marks `[0, 3)` and `[6, 8)` . - `event_type = 'a' AND event_date = 3` , the server reads the data in the ranges of marks `[1, 3)` and `[7, 8)` . - `event_date = 3` , the server reads the data in the range of marks `[1, 10]` . The examples above show that it's always more effective to use an index than a full scan. ### Data Skipping Indexes [¶](https://www.tinybird.co/docs/about:blank#data-skipping-indexes) SCHEMA > `event_date` Date `json:$.date`, `event_type` LowCardinality(String) `json:$.event_type`, `user_id` UInt64 `json:$.user_id`, `payload` JSON `json:$.payload` ENGINE "MergeTree" ENGINE_SORTING_KEY "event_type, event_date, user_id" INDEXES > index_name index_expression TYPE index_type GRANULARITY index_granularity For tables from the `*MergeTree` family, data skipping indices can be specified. These indices aggregate some information about the specified expression on blocks, which consist of `granularity_value` granules (the size of the granule is specified using the `index_granularity` setting in the table engine). Then these aggregates are used in `SELECT` queries for reducing the amount of data to read from the disk by skipping big blocks of data where the `where` query can't be satisfied. The `GRANULARITY` clause can be omitted, the default value of `granularity_value` is 1. **Example** SCHEMA > u64 UInt64, i32 Int32, s String INDEXES > INDEX idx1 u64 TYPE bloom_filter GRANULARITY 3, INDEX idx2 u64 * i32 TYPE minmax GRANULARITY 3, INDEX idx3 u64 * length(s) TYPE set(1000) GRANULARITY 4 Indices from the example can be used by Tinybird to reduce the amount of data to read from disk in the following queries: SELECT count() FROM ds WHERE u64 == 10 SELECT count() FROM ds WHERE u64 * i32 >= 1234 SELECT count() FROM ds WHERE u64 * length(s) == 1234 Data skipping indexes can also be created on composite columns: -- on columns of type Map: INDEX map_key_index mapKeys(map_column) TYPE bloom_filter INDEX map_value_index mapValues(map_column) TYPE bloom_filter -- on columns of type Tuple: INDEX tuple_1_index tuple_column.1 TYPE bloom_filter INDEX tuple_2_index tuple_column.2 TYPE bloom_filter -- on columns of type Nested: INDEX nested_1_index col.nested_col1 TYPE bloom_filter INDEX nested_2_index col.nested_col2 TYPE bloom_filter ### Available Types of Indices [¶](https://www.tinybird.co/docs/about:blank#available-types-of-indices) #### MinMax [¶](https://www.tinybird.co/docs/about:blank#minmax) Stores extremes of the specified expression (if the expression is `tuple` , then it stores extremes for each element of `tuple` ), uses stored info for skipping blocks of data like the primary key. Syntax: `minmax` #### Set [¶](https://www.tinybird.co/docs/about:blank#set) Stores unique values of the specified expression (no more than `max_rows` rows, `max_rows=0` means “no limits”). Uses the values to check if the `WHERE` expression isn't satisfiable on a block of data. Syntax: `set(max_rows)` #### Bloom Filter [¶](https://www.tinybird.co/docs/about:blank#bloom-filter) Stores a [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) for the specified columns. An optional `false_positive` parameter with possible values between 0 and 1 specifies the probability of receiving a false positive response from the filter. Default value: 0.025. Supported data types: `Int*`, `UInt*`, `Float*`, `Enum`, `Date`, `DateTime`, `String`, `FixedString`, `Array`, `LowCardinality`, `Nullable`, `UUID` and `Map` . For the `Map` data type, the client can specify if the index should be created for keys or values using mapKeys or mapValues function. Syntax: `bloom_filter([false_positive])` #### N-gram Bloom Filter [¶](https://www.tinybird.co/docs/about:blank#n-gram-bloom-filter) Stores a [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) that contains all n-grams from a block of data. Only works with datatypes: String, FixedString and Map. Can be used for optimization of `EQUALS`, `LIKE` and `IN` expressions. Syntax: `ngrambf_v1(n, size_of_bloom_filter_in_bytes, number_of_hash_functions, random_seed)` - `n` : ngram size, - `size_of_bloom_filter_in_bytes` : Bloom filter size in bytes (you can use large values here, for example, 256 or 512, because it can be compressed well). - `number_of_hash_functions` : The number of hash functions used in the Bloom filter. - `random_seed` : The seed for Bloom filter hash functions. #### Token Bloom Filter [¶](https://www.tinybird.co/docs/about:blank#token-bloom-filter) The same as `ngrambf_v1` , but stores tokens instead of ngrams. Tokens are sequences separated by non-alphanumeric characters. Syntax: `tokenbf_v1(size_of_bloom_filter_in_bytes, number_of_hash_functions, random_seed)` ### Functions Support [¶](https://www.tinybird.co/docs/about:blank#functions-support) Conditions in the `WHERE` clause contains calls of the functions that operate with columns. If the column is a part of an index, Tinybird tries to use this index when performing the functions. Tinybird supports different subsets of functions for using indexes. Indexes of type `set` can be utilized by all functions. The other index types are supported as follows: | Function (operator) / Index | primary key | minmax | ngrambf_v1 | tokenbf_v1 | bloom_filter | full_text | | --- | --- | --- | --- | --- | --- | --- | | equals (=, ==) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | notEquals(!=, <>) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | like | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | | notLike | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | | match | ✗ | ✗ | ✔ | ✔ | ✗ | ✔ | | startsWith | ✔ | ✔ | ✔ | ✔ | ✗ | ✔ | | endsWith | ✗ | ✗ | ✔ | ✔ | ✗ | ✔ | | multiSearchAny | ✗ | ✗ | ✔ | ✗ | ✗ | ✔ | | in | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | notIn | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | | less (<) | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | greater (>) | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | lessOrEquals (<=) | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | greaterOrEquals (>=) | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | empty | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | notEmpty | ✔ | ✔ | ✗ | ✗ | ✗ | ✗ | | has | ✗ | ✗ | ✔ | ✔ | ✔ | ✔ | | hasAny | ✗ | ✗ | ✔ | ✔ | ✔ | ✗ | | hasAll | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ | | hasToken | ✗ | ✗ | ✗ | ✔ | ✗ | ✔ | | hasTokenOrNull | ✗ | ✗ | ✗ | ✔ | ✗ | ✔ | | hasTokenCaseInsensitive (*) | ✗ | ✗ | ✗ | ✔ | ✗ | ✗ | | hasTokenCaseInsensitiveOrNull (*) | ✗ | ✗ | ✗ | ✔ | ✗ | ✗ | Functions with a constant argument that is less than ngram size can’t be used by `ngrambf_v1` for query optimization. (*) For `hasTokenCaseInsensitive` and `hasTokenCaseInsensitiveOrNull` to be effective, the `tokenbf_v1` index must be created on lowercased data, for example `INDEX idx (lower(str_col)) TYPE tokenbf_v1(512, 3, 0)`. Bloom filters can have false positive matches, so the `ngrambf_v1`, `tokenbf_v1` , and `bloom_filter` indexes can't be used for optimizing queries where the result of a function is expected to be false. For example: - Can be optimized: - `s LIKE '%test%'` - `NOT s NOT LIKE '%test%'` - `s = 1` - `NOT s != 1` - `startsWith(s, 'test')` - Can not be optimized: - `NOT s LIKE '%test%'` - `s NOT LIKE '%test%'` - `NOT s = 1` - `s != 1` - `NOT startsWith(s, 'test')` ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/engines/engine-settings Last update: 2024-12-05T17:20:25.000Z Content: --- title: "Engine settings · Tinybird Docs" theme-color: "#171612" description: "Settings for Tinybird table engines" --- # Engine settings [¶](https://www.tinybird.co/docs/about:blank#engine-settings) You can customize table engine settings to better match your needs. The engine options are: `engine_partition_key`, `engine_sorting_key`, `engine_primary_key`, `engine_sampling_key`, `engine_ttl` and `engine_settings`. If `engine_partition_key` is empty or not passed as a parameter, the underlying Data Source doesn't have any partition unless there's a `Date` column. In that case the Data Source is partitioned by year. If you want to create a Data Source with no partitions, send `engine_partition_key=tuple()`. Don't change the following settings unless you are familiar with them and understand their impact. If you're unsure, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community). The supported engine settings are: - `index_granularity` - `merge_with_ttl_timeout` - `ttl_only_drop_parts` - `min_bytes_for_wide_part` - `min_rows_for_wide_part` ## index_granularity [¶](https://www.tinybird.co/docs/about:blank#index-granularity) Maximum number of data rows between the marks of an index. Default value: 8192. ## merge_with_ttl_timeout [¶](https://www.tinybird.co/docs/about:blank#merge-with-ttl-timeout) Minimum delay in seconds before repeating a merge with delete TTL. Default value: `14400` seconds (4 hours). ## ttl_only_drop_parts [¶](https://www.tinybird.co/docs/about:blank#ttl-only-drop-parts) Controls whether data parts are fully dropped in MergeTree tables when all rows in that part have expired according to their `TTL` settings. When `ttl_only_drop_parts` is disabled (by default), only the rows that have expired based on their TTL settings are removed. When `ttl_only_drop_parts` is enabled, the entire part is dropped if all rows in that part have expired according to their `TTL` settings. Default value: 0. ## min_bytes_for_wide_part [¶](https://www.tinybird.co/docs/about:blank#min-bytes-for-wide-part) Minimum number of bytes/rows in a data part that can be stored in `Wide` format. You can set one, both or none of these settings. --- URL: https://www.tinybird.co/docs/sql-reference/engines/collapsingmergetree Last update: 2025-01-07T15:48:10.000Z Content: --- title: "CollapsingMergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the CollapsingMergeTree engine for collapsing rows." --- # CollapsingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#collapsingmergetree-engine) The `CollapsingMergeTree` engine inherits from MergeTree and adds the logic of rows collapsing to data parts merge algorithm. `CollapsingMergeTree` asynchronously deletes, or collapses, pairs of rows if all of the fields in a sorting key ( `ENGINE_SORTING_KEY` ) are equivalent except the particular field `Sign` , which can have `1` and `-1` values. Rows without a pair are kept. The engine may significantly reduce the volume of storage and increase the efficiency of `SELECT` queries. ## CollapsingMergeTree parameters [¶](https://www.tinybird.co/docs/about:blank#collapsingmergetree-parameters) ### sign [¶](https://www.tinybird.co/docs/about:blank#sign) `ENGINE_SIGN` - name of the column with the type of row: `1` is a “state” row, `-1` is a “cancel” row. The column data type should be `Int8`. ## Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) ##### cmt.datasource SCHEMA > UserID UInt64 `json:$.UserID`, PageViews UInt8 `json:$.PageViews`, Duration UInt8 `json:$.Duration`, Sign Int8 `json:$.Sign` ENGINE "CollapsingMergeTree" ENGINE_SORTING_KEY "UserID" ENGINE_SIGN "Sign" tb push datasources/cmt.datasource TB_HOST=$(cat .tinyb | jq ".host" -r) TB_TOKEN=$(cat .tinyb | jq ".token" -r) curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d '{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": 1}' \ "$TB_HOST/v0/events?name=cmt" tb sql "select * from cmt" ----------------------------------------------------- | UserID | PageViews | Duration | Sign | ----------------------------------------------------- | 4324182021466249494 | 5 | 146 | 1 | ----------------------------------------------------- Let's add another row with the same `UserID` and negate the previous row (same row with -1 as `Sign` value). `{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": -1}` curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d $'{"UserID": 4324182021466249494, "PageViews": 6, "Duration": 185, "Sign": 1}\n{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": -1}\n' \ "$TB_HOST/v0/events?name=cmt" tb sql "select * from cmt" ----------------------------------------------------- | UserID | PageViews | Duration | Sign | ----------------------------------------------------- | 4324182021466249494 | 5 | 146 | 1 | | 4324182021466249494 | 6 | 185 | 1 | | 4324182021466249494 | 5 | 146 | -1 | ----------------------------------------------------- What happened here? Collapsing happens during merges, which are performed asynchronously and can't be controlled, so you should force it using `FINAL` or query using aggregation. tb sql "select * from cmt final" ----------------------------------------------------- | UserID | PageViews | Duration | Sign | ----------------------------------------------------- | 4324182021466249494 | 6 | 185 | 1 | ----------------------------------------------------- tb sql "SELECT UserID, sum(PageViews * Sign) AS PageViews, sum(Duration * Sign) AS Duration FROM cmt GROUP BY UserID HAVING sum(Sign) > 0" ---------------------------------------------- | UserID | PageViews | Duration | ---------------------------------------------- | 4324182021466249494 | 6 | 185 | ---------------------------------------------- A different approach would be to negate all the numeric columns and then sum them up. `{"UserID": 4324182021466249494, "PageViews": -5, "Duration": -146, "Sign": -1}` curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d '{"UserID": 4324182021466249494, "PageViews": 5, "Duration": 146, "Sign": 1}' \ "$TB_HOST/v0/events?name=cmt" tb sql "select * from cmt" ----------------------------------------------------- | UserID | PageViews | Duration | Sign | ----------------------------------------------------- | 4324182021466249494 | 5 | 146 | 1 | ----------------------------------------------------- curl \ -H "Authorization: Bearer $TB_TOKEN" \ -d $'{"UserID": 4324182021466249494, "PageViews": 6, "Duration": 185, "Sign": 1}\n{"UserID": 4324182021466249494, "PageViews": -5, "Duration": -146, "Sign": -1}\n' \ "$TB_HOST/v0/events?name=cmt" tb sql "SELECT UserID, sum(PageViews) AS PageViews, sum(Duration) AS Duration FROM UAct GROUP BY UserID" ---------------------------------------------- | UserID | PageViews | Duration | ---------------------------------------------- | 4324182021466249494 | 6 | 185 | ---------------------------------------------- ## Query clauses [¶](https://www.tinybird.co/docs/about:blank#query-clauses) When creating a `CollapsingMergeTree` table, the same query clauses are required as when creating a `MergeTree` table. ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/engines/aggregatingmergetree Last update: 2025-01-23T08:42:52.000Z Content: --- title: "AggregatingMergeTree engine · Tinybird Docs" theme-color: "#171612" description: "Use the AggregatingMergeTree engine for incremental data aggregation." --- # AggregatingMergeTree engine [¶](https://www.tinybird.co/docs/about:blank#aggregatingmergetree-engine) Use the `AggregatingMergeTree` engine for incremental data aggregation, including for aggregated materialized views. The engine processes all columns with the following types: - [ AggregateFunction](https://www.tinybird.co/docs/docs/sql-reference/data-types/aggregatefunction) - [ SimpleAggregateFunction](https://www.tinybird.co/docs/docs/sql-reference/data-types/simpleaggregatefunction) Use `AggregatingMergeTree` if it reduces the number of rows by orders of magnitude. ## Creating an AggregatingMergeTree Data Source [¶](https://www.tinybird.co/docs/about:blank#creating-an-aggregatingmergetree-data-source) The `AggregatingMergeTree` Data Source is usually created using a Materialized View from a `MergeTree` Data Source. Example: ##### user_activity.datasource SCHEMA > `timestamp` DateTime `json:$.timestamp`, `user_id` UInt32 `json:$.user_id`, `activity_type` String `json:$.activity_type`, `payload` JSON `json:$.payload` ENGINE "MergeTree" ENGINE_PARTITION_KEY "toYYYYMM(timestamp)" ENGINE_SORTING_KEY "activity_type, user_id, timestamp" ##### total_daily_activities_mat.pipe NODE total_daily_activities_1 SQL > SELECT activity_type, toDate(timestamp) AS date, countState() AS total_activities FROM user_activity GROUP BY activity_type, date TYPE materialized DATASOURCE total_daily_activities_mv ##### total_daily_activities_mv.datasource SCHEMA > `activity_type` String, `date` Date, `total_activities` AggregateFunction(count) ENGINE "AggregatingMergeTree" ENGINE_PARTITION_KEY "toYYYYMM(date)" ENGINE_SORTING_KEY "activity_type, date" ## Querying an AggregatingMergeTree Data Source [¶](https://www.tinybird.co/docs/about:blank#querying-an-aggregatingmergetree-data-source) As explained in the [MergeTree](https://www.tinybird.co/docs/docs/sql-reference/engines/mergetree) engine, parts are merged in the background, so you are never sure if the data is merged or not. That's why you should always use the `-Merge()` modifier after a `-State()` when needed. ##### Querying an AggregatingMergeTree Data Source SELECT activity_type, date, countMerge(total_activities) AS daily_total_activities FROM total_daily_activities_mv GROUP BY activity_type, date ## Settings [¶](https://www.tinybird.co/docs/about:blank#settings) For a list of supported settings, see [Engine settings](https://www.tinybird.co/docs/docs/sql-reference/engines/engine-settings). --- URL: https://www.tinybird.co/docs/sql-reference/functions/window-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Window functions · Tinybird Docs" theme-color: "#171612" description: "Functions for window calculations." --- # Window functions [¶](https://www.tinybird.co/docs/about:blank#window-functions) Window functions perform calculations across a set of table rows that are somehow related to the current row. They are often used for ranking, aggregating, and calculating running totals. ## dense_rank Ranks the current row within its partition without gaps. In other words, if the value of any new row encountered is equal to the value of one of the previous rows then it will receive the next successive rank without any gaps in ranking. The rank function provides the same behaviour, but with gaps in ranking. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) Alias: `denseRank` (case-sensitive) dense_rank (column_name) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A number for the current row within its partition, without gaps in ranking. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) The following example is based on the example provided in the video instructional Ranking window functions in Tinybird. Query: SELECT player, salary, dense_rank() OVER (ORDER BY salary DESC) AS dense_rank FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 150000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 150000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─dense_rank─┐ │ Gary Chen │ 195000 │ 1 │ │ Robert George │ 195000 │ 1 │ │ Charles Juarez │ 190000 │ 2 │ │ Michael Stanley │ 150000 │ 3 │ │ Douglas Benson │ 150000 │ 3 │ │ Scott Harrison │ 150000 │ 3 │ │ James Henderson │ 140000 │ 4 │ └─────────────────┴────────┴────────────┘ ## first_value Returns the first value evaluated within its ordered frame. By default, NULL arguments are skipped, however the `RESPECT NULLS` modifier can be used to override this behaviour. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) first_value (column_name) [[RESPECT NULLS] | [IGNORE NULLS]] OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([PARTITION BY grouping_column] [ORDER BY sorting_column]) Alias: `any`. Using the optional modifier `RESPECT NULLS` after `first_value(column_name)` will ensure that `NULL` arguments aren't skipped. Alias: `first_value_respect_nulls` For more detail on window function syntax see: Window Functions - Syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The first value evaluated within its ordered frame. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) In this example the `first_value` function is used to find the highest paid footballer from a fictional dataset of salaries of Premier League football players. Query: SELECT player, salary, first_value(player) OVER (ORDER BY salary DESC) AS highest_paid_player FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 150000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 150000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─highest_paid_player─┐ │ Gary Chen │ 196000 │ Gary Chen │ │ Robert George │ 195000 │ Gary Chen │ │ Charles Juarez │ 190000 │ Gary Chen │ │ Scott Harrison │ 180000 │ Gary Chen │ │ Douglas Benson │ 150000 │ Gary Chen │ │ James Henderson │ 140000 │ Gary Chen │ │ Michael Stanley │ 100000 │ Gary Chen │ └─────────────────┴────────┴─────────────────────┘ ## lagInFrame Returns a value evaluated at the row that is at a specified physical offset row before the current row within the ordered frame. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) lagInFrame(x[, offset[, default]]) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : Column name. - `offset` : Offset to apply. (U)Int*. (Optional - `1` by default). - `default` : Value to return if calculated row exceeds the boundaries of the window frame. (Optional - default value of column type when omitted). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value evaluated at the row that is at a specified physical offset before the current row within the ordered frame. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) This example looks at historical data for a specific stock and uses the `lagInFrame` function to calculate a day-to-day delta and percentage change in the closing price of the stock. Query: SELECT date, close, lagInFrame(close, 1, close) OVER (ORDER BY date ASC) AS previous_day_close, COALESCE(ROUND(close - previous_day_close, 2)) AS delta, COALESCE(ROUND((delta / previous_day_close) * 100, 2)) AS percent_change FROM ( select c1::Date as date, c2::Float32 as open, c3::Float32 as high, c4::Float32 as low, c5::Float32 as close, c6::UInt32 as volume from values ( ('2024-06-03', 113.62, 115.00, 112.00, 115.00, 438392000), ('2024-06-04', 115.72, 116.60, 114.04, 116.44, 403324000), ('2024-06-05', 118.37, 122.45, 117.47, 122.44, 528402000), ('2024-06-06', 124.05, 125.59, 118.32, 121.00, 664696000), ('2024-06-07', 119.77, 121.69, 118.02, 120.89, 412386000) ) ) ORDER BY date DESC Result: ┌───────date─┬──close─┬─previous_day_close─┬─delta─┬─percent_change─┐ │ 2024-06-07 │ 120.89 │ 121 │ -0.11 │ -0.09 │ │ 2024-06-06 │ 121 │ 122.44 │ -1.44 │ -1.18 │ │ 2024-06-05 │ 122.44 │ 116.44 │ 6 │ 5.15 │ │ 2024-06-04 │ 116.44 │ 115 │ 1.44 │ 1.25 │ │ 2024-06-03 │ 115 │ 115 │ 0 │ 0 │ └────────────┴────────┴────────────────────┴───────┴────────────────┘ ## last_value Returns the last value evaluated within its ordered frame. By default, NULL arguments are skipped, however the `RESPECT NULLS` modifier can be used to override this behaviour. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) last_value (column_name) [[RESPECT NULLS] | [IGNORE NULLS]] OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) Alias: `anyLast`. Using the optional modifier `RESPECT NULLS` after `first_value(column_name)` will ensure that `NULL` arguments aren't skipped. Alias: `lastValueRespectNulls` For more detail on window function syntax see: Window Functions - Syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The last value evaluated within its ordered frame. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) In this example the `last_value` function is used to find the lowest paid footballer from a fictional dataset of salaries of Premier League football players. Query: SELECT player, salary, last_value(player) OVER (ORDER BY salary DESC RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS lowest_paid_player FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 196000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 100000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 180000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─lowest_paid_player─┐ │ Gary Chen │ 196000 │ Michael Stanley │ │ Robert George │ 195000 │ Michael Stanley │ │ Charles Juarez │ 190000 │ Michael Stanley │ │ Scott Harrison │ 180000 │ Michael Stanley │ │ Douglas Benson │ 150000 │ Michael Stanley │ │ James Henderson │ 140000 │ Michael Stanley │ │ Michael Stanley │ 100000 │ Michael Stanley │ └─────────────────┴────────┴────────────────────┘ ## leadInFrame Returns a value evaluated at the row that is offset rows after the current row within the ordered frame. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) leadInFrame(x[, offset[, default]]) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : Column name. - `offset` : Offset to apply. (U)Int*. (Optional - `1` by default). - `default` : Value to return if calculated row exceeds the boundaries of the window frame. (Optional - default value of column type when omitted). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - value evaluated at the row that is offset rows after the current row within the ordered frame. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT n, m, leadInFrame(m) OVER ( PARTITION BY n ORDER BY m ASC ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING ) AS next FROM ( select (number % 12) + 1 n, toStartOfMonth(today()) - INTERVAL number MONTH m from numbers(36) order by m ) Result: -------------------------------- | n | m | next | -------------------------------- | 1 | 2022-12-01 | 2023-12-01 | | 1 | 2023-12-01 | 2024-12-01 | | 1 | 2024-12-01 | 1970-01-01 | | 2 | 2022-11-01 | 2023-11-01 | | 2 | 2023-11-01 | 2024-11-01 | | 2 | 2024-11-01 | 1970-01-01 | | 3 | 2022-10-01 | 2023-10-01 | | 3 | 2023-10-01 | 2024-10-01 | | 3 | 2024-10-01 | 1970-01-01 | | 4 | 2022-09-01 | 2023-09-01 | | 4 | 2023-09-01 | 2024-09-01 | | 4 | 2024-09-01 | 1970-01-01 | | 5 | 2022-08-01 | 2023-08-01 | | 5 | 2023-08-01 | 2024-08-01 | | 5 | 2024-08-01 | 1970-01-01 | | 6 | 2022-07-01 | 2023-07-01 | | 6 | 2023-07-01 | 2024-07-01 | | 6 | 2024-07-01 | 1970-01-01 | | 7 | 2022-06-01 | 2023-06-01 | | 7 | 2023-06-01 | 2024-06-01 | | 7 | 2024-06-01 | 1970-01-01 | | 8 | 2022-05-01 | 2023-05-01 | | 8 | 2023-05-01 | 2024-05-01 | | 8 | 2024-05-01 | 1970-01-01 | | 9 | 2022-04-01 | 2023-04-01 | | 9 | 2023-04-01 | 2024-04-01 | | 9 | 2024-04-01 | 1970-01-01 | | 10 | 2022-03-01 | 2023-03-01 | | 10 | 2023-03-01 | 2024-03-01 | | 10 | 2024-03-01 | 1970-01-01 | | 11 | 2022-02-01 | 2023-02-01 | | 11 | 2023-02-01 | 2024-02-01 | | 11 | 2024-02-01 | 1970-01-01 | | 12 | 2022-01-01 | 2023-01-01 | | 12 | 2023-01-01 | 2024-01-01 | | 12 | 2024-01-01 | 1970-01-01 | -------------------------------- ## nth_value Returns the first non-NULL value evaluated against the nth row (offset) in its ordered frame. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) nth_value (x, offset) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : Column name. - `offset` : nth row to evaluate current row against. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The first non-NULL value evaluated against the nth row (offset) in its ordered frame. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) In this example the `nth-value` function is used to find the third-highest salary from a fictional dataset of salaries of Premier League football players. Query: SELECT player, salary, nth_value(player,3) OVER(ORDER BY salary DESC) AS third_highest_salary FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 100000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 180000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─third_highest_salary─┐ │ Gary Chen │ 195000 │ │ │ Robert George │ 195000 │ │ │ Charles Juarez │ 190000 │ Charles Juarez │ │ Scott Harrison │ 180000 │ Charles Juarez │ │ Douglas Benson │ 150000 │ Charles Juarez │ │ James Henderson │ 140000 │ Charles Juarez │ │ Michael Stanley │ 100000 │ Charles Juarez │ └─────────────────┴────────┴──────────────────────┘ ## percent_rank returns the relative rank (i.e. percentile) of rows within a window partition. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) Alias: `percentRank` (case-sensitive) percent_rank (column_name) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING]] | [window_name]) FROM table_name WINDOW window_name as ([PARTITION BY grouping_column] [ORDER BY sorting_column] RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) The default and required window frame definition is `RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`. For more detail on window function syntax see: Window Functions - Syntax. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT player, salary, percent_rank() OVER (ORDER BY salary DESC) AS percent_rank FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 150000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 150000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬───────percent_rank─┐ │ Gary Chen │ 195000 │ 0 │ │ Robert George │ 195000 │ 0 │ │ Charles Juarez │ 190000 │ 0.3333333333333333 │ │ Michael Stanley │ 150000 │ 0.5 │ │ Scott Harrison │ 150000 │ 0.5 │ │ Douglas Benson │ 150000 │ 0.5 │ │ James Henderson │ 140000 │ 1 │ └─────────────────┴────────┴────────────────────┘ ## rank Ranks the current row within its partition with gaps. In other words, if the value of any row it encounters is equal to the value of a previous row then it will receive the same rank as that previous row. The rank of the next row is then equal to the rank of the previous row plus a gap equal to the number of times the previous rank was given. The dense_rank function provides the same behaviour but without gaps in ranking. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) rank (column_name) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A number for the current row within its partition, including gaps. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) The following example is based on the example provided in the video instructional Ranking window functions in Tinybird. Query: SELECT player, salary, rank() OVER (ORDER BY salary DESC) AS rank FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 150000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 150000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M'), ('South Hampton Seagulls', 'Douglas Benson', 150000, 'M'), ('South Hampton Seagulls', 'James Henderson', 140000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─rank─┐ │ Gary Chen │ 195000 │ 1 │ │ Robert George │ 195000 │ 1 │ │ Charles Juarez │ 190000 │ 3 │ │ Douglas Benson │ 150000 │ 4 │ │ Michael Stanley │ 150000 │ 4 │ │ Scott Harrison │ 150000 │ 4 │ │ James Henderson │ 140000 │ 7 │ └─────────────────┴────────┴──────┘ ## row_number Numbers the current row within its partition starting from 1. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) row_number (column_name) OVER ([[PARTITION BY grouping_column] [ORDER BY sorting_column] [ROWS or RANGE expression_to_bound_rows_withing_the_group]] | [window_name]) FROM table_name WINDOW window_name as ([[PARTITION BY grouping_column] [ORDER BY sorting_column]) For more detail on window function syntax see: Window Functions - Syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A number for the current row within its partition. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) The following example is based on the example provided in the video instructional Ranking window functions in Tinybird. Query: SELECT player, salary, row_number() OVER (ORDER BY salary DESC) AS row_number FROM ( select c1::String as team, c2::String as player, c3::UInt32 as salary, c4::String as position from values ( ('Port Elizabeth Barbarians', 'Gary Chen', 195000, 'F'), ('New Coreystad Archdukes', 'Charles Juarez', 190000, 'F'), ('Port Elizabeth Barbarians', 'Michael Stanley', 150000, 'D'), ('New Coreystad Archdukes', 'Scott Harrison', 150000, 'D'), ('Port Elizabeth Barbarians', 'Robert George', 195000, 'M') ) ) Result: ┌─player──────────┬─salary─┬─row_number─┐ │ Gary Chen │ 195000 │ 1 │ │ Robert George │ 195000 │ 2 │ │ Charles Juarez │ 190000 │ 3 │ │ Michael Stanley │ 150000 │ 4 │ │ Scott Harrison │ 150000 │ 5 │ └─────────────────┴────────┴────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/uuid-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "UUID functions · Tinybird Docs" theme-color: "#171612" description: "Functions for working with UUIDs." --- # Functions for working with UUIDs [¶](https://www.tinybird.co/docs/about:blank#functions-for-working-with-uuids) The following functions are available for working with UUIDs. ## generateUUIDv4 [¶](https://www.tinybird.co/docs/about:blank#generateuuidv4) Generates a version 4 UUID. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) generateUUIDv4([expr]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : An arbitrary expression used to bypass common subexpression elimination if the function is called multiple times in a query. The value of the expression has no effect on the returned UUID. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A value of type UUIDv4. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT generateUUIDv4() uuid Result: ┌─────────────────────────────────uuid─┐ │ f4bf890f-f9dc-4332-ad5c-0c18e73f28e9 │ └──────────────────────────────────────┘ ### Example with multiple UUIDs generated per row [¶](https://www.tinybird.co/docs/about:blank#example-with-multiple-uuids-generated-per-row) SELECT generateUUIDv4(1), generateUUIDv4(2) ┌─generateUUIDv4(1)────────────────────┬─generateUUIDv4(2)────────────────────┐ │ 2d49dc6e-ddce-4cd0-afb8-790956df54c1 │ 8abf8c13-7dea-4fdf-af3e-0e18767770e6 │ └──────────────────────────────────────┴──────────────────────────────────────┘ ## generateUUIDv7 [¶](https://www.tinybird.co/docs/about:blank#generateuuidv7) Generates a version 7 UUID. The generated UUID contains the current Unix timestamp in milliseconds (48 bits), followed by version "7" (4 bits), a counter (42 bit) to distinguish UUIDs within a millisecond (including a variant field "2", 2 bit), and a random field (32 bits). For any given timestamp (unix_ts_ms), the counter starts at a random value and is incremented by 1 for each new UUID until the timestamp changes. In case the counter overflows, the timestamp field is incremented by 1 and the counter is reset to a random new start value. Function `generateUUIDv7` guarantees that the counter field within a timestamp increments monotonically across all function invocations in concurrently running threads and queries. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) generateUUIDv7([expr]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : An arbitrary expression used to bypass common subexpression elimination if the function is called multiple times in a query. The value of the expression has no effect on the returned UUID. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A value of type UUIDv7. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) First, create a table with a column of type UUID, then insert a generated UUIDv7 into the table. SELECT generateUUIDv7() uuid Result: ┌─────────────────────────────────uuid─┐ │ 018f05af-f4a8-778f-beee-1bedbc95c93b │ └──────────────────────────────────────┘ ### Example with multiple UUIDs generated per row [¶](https://www.tinybird.co/docs/about:blank#example-with-multiple-uuids-generated-per-row) SELECT generateUUIDv7(1), generateUUIDv7(2) ┌─generateUUIDv7(1)────────────────────┬─generateUUIDv7(2)────────────────────┐ │ 018f05c9-4ab8-7b86-b64e-c9f03fbd45d1 │ 018f05c9-4ab8-7b86-b64e-c9f12efb7e16 │ └──────────────────────────────────────┴──────────────────────────────────────┘ ## empty [¶](https://www.tinybird.co/docs/about:blank#empty) Checks whether the input UUID is empty. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) empty(UUID) The UUID is considered empty if it contains all zeros (zero UUID). The function also works for Arrays and Strings. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A UUID. UUID. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `1` for an empty UUID or `0` for a non-empty UUID. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) To generate the UUID value, Tinybird provides the generateUUIDv4 function. Query: SELECT empty(generateUUIDv4()) Result: ┌─empty(generateUUIDv4())─┐ │ 0 │ └─────────────────────────┘ ## notEmpty [¶](https://www.tinybird.co/docs/about:blank#notempty) Checks whether the input UUID is non-empty. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) notEmpty(UUID) The UUID is considered empty if it contains all zeros (zero UUID). The function also works for Arrays or Strings. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A UUID. UUID. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `1` for a non-empty UUID or `0` for an empty UUID. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) To generate the UUID value, Tinybird provides the generateUUIDv4 function. Query: SELECT notEmpty(generateUUIDv4()) Result: ┌─notEmpty(generateUUIDv4())─┐ │ 1 │ └────────────────────────────┘ ## toUUID [¶](https://www.tinybird.co/docs/about:blank#touuid) Converts a value of type String to a UUID. toUUID(string) ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) The UUID type value. ### Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) SELECT toUUID('61f0c404-5cb3-11e7-907b-a6006ad3dba0') AS uuid Result: ┌─────────────────────────────────uuid─┐ │ 61f0c404-5cb3-11e7-907b-a6006ad3dba0 │ └──────────────────────────────────────┘ ## toUUIDOrDefault [¶](https://www.tinybird.co/docs/about:blank#touuidordefault) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : String of 36 characters or FixedString(36). String. - `default` : UUID to be used as the default if the first argument can't be converted to a UUID type. UUID. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) UUID toUUIDOrDefault(string, default) ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) The UUID type value. ### Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) This first example returns the first argument converted to a UUID type as it can be converted: SELECT toUUIDOrDefault('61f0c404-5cb3-11e7-907b-a6006ad3dba0', cast('59f0c404-5cb3-11e7-907b-a6006ad3dba0' as UUID)) Result: ┌─toUUIDOrDefault('61f0c404-5cb3-11e7-907b-a6006ad3dba0', CAST('59f0c404-5cb3-11e7-907b-a6006ad3dba0', 'UUID'))─┐ │ 61f0c404-5cb3-11e7-907b-a6006ad3dba0 │ └───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ This second example returns the second argument (the provided default UUID) as the first argument can't be converted to a UUID type: SELECT toUUIDOrDefault('-----61f0c404-5cb3-11e7-907b-a6006ad3dba0', cast('59f0c404-5cb3-11e7-907b-a6006ad3dba0' as UUID)) Result: ┌─toUUIDOrDefault('-----61f0c404-5cb3-11e7-907b-a6006ad3dba0', CAST('59f0c404-5cb3-11e7-907b-a6006ad3dba0', 'UUID'))─┐ │ 59f0c404-5cb3-11e7-907b-a6006ad3dba0 │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ## toUUIDOrNull [¶](https://www.tinybird.co/docs/about:blank#touuidornull) Takes an argument of type String and tries to parse it into UUID. If failed, returns NULL. toUUIDOrNull(string) ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) The Nullable(UUID) type value. ### Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) SELECT toUUIDOrNull('61f0c404-5cb3-11e7-907b-a6006ad3dba0T') AS uuid Result: ┌─uuid─┐ │ ᴺᵁᴸᴸ │ └──────┘ ## toUUIDOrZero [¶](https://www.tinybird.co/docs/about:blank#touuidorzero) It takes an argument of type String and tries to parse it into UUID. If failed, returns zero UUID. toUUIDOrZero(string) ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) The UUID type value. ### Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) SELECT toUUIDOrZero('61f0c404-5cb3-11e7-907b-a6006ad3dba0T') AS uuid Result: ┌─────────────────────────────────uuid─┐ │ 00000000-0000-0000-0000-000000000000 │ └──────────────────────────────────────┘ ## UUIDStringToNum [¶](https://www.tinybird.co/docs/about:blank#uuidstringtonum) Accepts `string` containing 36 characters in the format `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` , and returns a FixedString(16) as its binary representation, with its format optionally specified by `variant` ( `Big-endian` by default). ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) UUIDStringToNum(string[, variant = 1]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : A String of 36 characters or FixedString - `variant` : Integer, representing a variant as specified by RFC4122. 1 = `Big-endian` (default), 2 = `Microsoft` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) FixedString(16) ### Usage example [¶](https://www.tinybird.co/docs/about:blank#usage-example) SELECT '612f3c40-5d3b-217e-707b-6a546a3d7b29' AS uuid, UUIDStringToNum(uuid) AS bytes Result: ┌─uuid─────────────────────────────────┬─bytes────────────┐ │ 612f3c40-5d3b-217e-707b-6a546a3d7b29 │ a/<@];!~p{jTj={) │ └──────────────────────────────────────┴──────────────────┘ SELECT '612f3c40-5d3b-217e-707b-6a546a3d7b29' AS uuid, UUIDStringToNum(uuid, 2) AS bytes Result: ┌─uuid─────────────────────────────────┬─bytes────────────┐ │ 612f3c40-5d3b-217e-707b-6a546a3d7b29 │ @` functions and cast behave differently in some cases, for example in case of LowCardinality: cast removes LowCardinality trait `to` functions don't. The same with Nullable, this behaviour isn't compatible with SQL standard, and it can be changed using cast_keep_nullable setting. Be aware of potential data loss if values of a datatype are converted to a smaller datatype (for example from `Int64` to `Int32` ) or between incompatible datatypes (for example from `String` to `Int` ). Make sure to check carefully if the result is as expected. Example: SELECT toTypeName(toLowCardinality('') AS val) AS source_type, toTypeName(toString(val)) AS to_type_result_type, toTypeName(CAST(val, 'String')) AS cast_result_type ┌─source_type────────────┬─to_type_result_type────┬─cast_result_type─┐ │ LowCardinality(String) │ LowCardinality(String) │ String │ └────────────────────────┴────────────────────────┴──────────────────┘ SELECT toTypeName(toNullable('') AS val) AS source_type, toTypeName(toString(val)) AS to_type_result_type, toTypeName(CAST(val, 'String')) AS cast_result_type ┌─source_type──────┬─to_type_result_type─┬─cast_result_type─┐ │ Nullable(String) │ Nullable(String) │ String │ └──────────────────┴─────────────────────┴──────────────────┘ SELECT toTypeName(toNullable('') AS val) AS source_type, toTypeName(toString(val)) AS to_type_result_type, toTypeName(CAST(val, 'String')) AS cast_result_type SETTINGS cast_keep_nullable = 1 ┌─source_type──────┬─to_type_result_type─┬─cast_result_type─┐ │ Nullable(String) │ Nullable(String) │ Nullable(String) │ └──────────────────┴─────────────────────┴──────────────────┘ ## toBool [¶](https://www.tinybird.co/docs/about:blank#tobool) Converts an input value to a value of type `Bool` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toBool(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string. Expression. Supported arguments: - Values of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. - Strings `true` or `false` (case-insensitive). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `true` or `false` based on evaluation of the argument. Bool. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toBool(toUInt8(1)), toBool(toInt8(-1)), toBool(toFloat32(1.01)), toBool('true'), toBool('false'), toBool('FALSE') Result: toBool(toUInt8(1)): true toBool(toInt8(-1)): true toBool(toFloat32(1.01)): true toBool('true'): true toBool('false'): false toBool('FALSE'): false ## toInt8 [¶](https://www.tinybird.co/docs/about:blank#toint8) Converts an input value to a value of type `Int8` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt8(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt8('0xc0fe');` . If the input value can't be represented within the bounds of Int8, overflow or underflow of the result occurs. This isn't considered an error. For example: `SELECT toInt8(128) == -128;`. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit integer value. Int8. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8(-8), toInt8(-8.8), toInt8('-8') Result: Row 1: ────── toInt8(-8): -8 toInt8(-8.8): -8 toInt8('-8'): -8 ## toInt8OrZero [¶](https://www.tinybird.co/docs/about:blank#toint8orzero) Like `toInt8` , this function converts an input value to a value of type Int8 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt8OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of ordinary Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt8OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit integer value if successful, otherwise `0` . Int8. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8OrZero('-8'), toInt8OrZero('abc') Result: Row 1: ────── toInt8OrZero('-8'): -8 toInt8OrZero('abc'): 0 ## toInt8OrNull [¶](https://www.tinybird.co/docs/about:blank#toint8ornull) Like `toInt8` , this function converts an input value to a value of type Int8 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt8OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt8OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit integer value if successful, otherwise `NULL` . Int8 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8OrNull('-8'), toInt8OrNull('abc') Result: Row 1: ────── toInt8OrNull('-8'): -8 toInt8OrNull('abc'): ᴺᵁᴸᴸ ## toInt8OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint8ordefault) Like `toInt8` , this function converts an input value to a value of type Int8 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt8OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int8` is unsuccessful. Int8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt8OrDefault('0xc0fe', CAST('-1', 'Int8'));` . If the input value can't be represented within the bounds of Int8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int8. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8OrDefault('-8', CAST('-1', 'Int8')), toInt8OrDefault('abc', CAST('-1', 'Int8')) Result: Row 1: ────── toInt8OrDefault('-8', CAST('-1', 'Int8')): -8 toInt8OrDefault('abc', CAST('-1', 'Int8')): -1 ## toInt16 [¶](https://www.tinybird.co/docs/about:blank#toint16) Converts an input value to a value of type `Int16` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt16(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt16('0xc0fe');` . If the input value can't be represented within the bounds of Int16, overflow or underflow of the result occurs. This isn't considered an error. For example: `SELECT toInt16(32768) == -32768;`. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit integer value. Int16. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt16(-16), toInt16(-16.16), toInt16('-16') Result: Row 1: ────── toInt16(-16): -16 toInt16(-16.16): -16 toInt16('-16'): -16 ## toInt16OrZero [¶](https://www.tinybird.co/docs/about:blank#toint16orzero) Like `toInt16` , this function converts an input value to a value of type Int16 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt16OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt16OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int16, overflow or underflow of the result occurs. This isn't considered as an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit integer value if successful, otherwise `0` . Int16. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt16OrZero('-16'), toInt16OrZero('abc') Result: Row 1: ────── toInt16OrZero('-16'): -16 toInt16OrZero('abc'): 0 ## toInt16OrNull [¶](https://www.tinybird.co/docs/about:blank#toint16ornull) Like `toInt16` , this function converts an input value to a value of type Int16 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt16OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt16OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int16, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit integer value if successful, otherwise `NULL` . Int16 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt16OrNull('-16'), toInt16OrNull('abc') Result: Row 1: ────── toInt16OrNull('-16'): -16 toInt16OrNull('abc'): ᴺᵁᴸᴸ ## toInt16OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint16ordefault) Like `toInt16` , this function converts an input value to a value of type Int16 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt16OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int16` is unsuccessful. Int16. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt16OrDefault('0xc0fe', CAST('-1', 'Int16'));` . If the input value can't be represented within the bounds of Int16, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int16. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt16OrDefault('-16', CAST('-1', 'Int16')), toInt16OrDefault('abc', CAST('-1', 'Int16')) Result: Row 1: ────── toInt16OrDefault('-16', CAST('-1', 'Int16')): -16 toInt16OrDefault('abc', CAST('-1', 'Int16')): -1 ## toInt32 [¶](https://www.tinybird.co/docs/about:blank#toint32) Converts an input value to a value of type `Int32` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt32(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt32('0xc0fe');` . If the input value can't be represented within the bounds of Int32, the result over or under flows. This isn't considered an error. For example: `SELECT toInt32(2147483648) == -2147483648;` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit integer value. Int32. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt32(-32), toInt32(-32.32), toInt32('-32') Result: Row 1: ────── toInt32(-32): -32 toInt32(-32.32): -32 toInt32('-32'): -32 ## toInt32OrZero [¶](https://www.tinybird.co/docs/about:blank#toint32orzero) Like `toInt32` , this function converts an input value to a value of type Int32 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt32OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt32OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit integer value if successful, otherwise `0` . Int32 The function uses rounding towards zero, meaning it truncate fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt32OrZero('-32'), toInt32OrZero('abc') Result: Row 1: ────── toInt32OrZero('-32'): -32 toInt32OrZero('abc'): 0 ## toInt32OrNull [¶](https://www.tinybird.co/docs/about:blank#toint32ornull) Like `toInt32` , this function converts an input value to a value of type Int32 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt32OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt32OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit integer value if successful, otherwise `NULL` . Int32 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt32OrNull('-32'), toInt32OrNull('abc') Result: Row 1: ────── toInt32OrNull('-32'): -32 toInt32OrNull('abc'): ᴺᵁᴸᴸ ## toInt32OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint32ordefault) Like `toInt32` , this function converts an input value to a value of type Int32 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt32OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int32` is unsuccessful. Int32. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt32OrDefault('0xc0fe', CAST('-1', 'Int32'));` . If the input value can't be represented within the bounds of Int32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int32. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt32OrDefault('-32', CAST('-1', 'Int32')), toInt32OrDefault('abc', CAST('-1', 'Int32')) Result: Row 1: ────── toInt32OrDefault('-32', CAST('-1', 'Int32')): -32 toInt32OrDefault('abc', CAST('-1', 'Int32')): -1 ## toInt64 [¶](https://www.tinybird.co/docs/about:blank#toint64) Converts an input value to a value of type `Int64` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt64(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported types: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt64('0xc0fe');` . If the input value can't be represented within the bounds of Int64, the result over or under flows. This isn't considered an error. For example: `SELECT toInt64(9223372036854775808) == -9223372036854775808;` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit integer value. Int64. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt64(-64), toInt64(-64.64), toInt64('-64') Result: Row 1: ────── toInt64(-64): -64 toInt64(-64.64): -64 toInt64('-64'): -64 ## toInt64OrZero [¶](https://www.tinybird.co/docs/about:blank#toint64orzero) Like `toInt64` , this function converts an input value to a value of type Int64 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt64OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt64OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit integer value if successful, otherwise `0` . Int64. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt64OrZero('-64'), toInt64OrZero('abc') Result: Row 1: ────── toInt64OrZero('-64'): -64 toInt64OrZero('abc'): 0 ## toInt64OrNull [¶](https://www.tinybird.co/docs/about:blank#toint64ornull) Like `toInt64` , this function converts an input value to a value of type Int64 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt64OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt64OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit integer value if successful, otherwise `NULL` . Int64 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt64OrNull('-64'), toInt64OrNull('abc') Result: Row 1: ────── toInt64OrNull('-64'): -64 toInt64OrNull('abc'): ᴺᵁᴸᴸ ## toInt64OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint64ordefault) Like `toInt64` , this function converts an input value to a value of type Int64 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt64OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int64` is unsuccessful. Int64. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt64OrDefault('0xc0fe', CAST('-1', 'Int64'));` . If the input value can't be represented within the bounds of Int64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int64. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt64OrDefault('-64', CAST('-1', 'Int64')), toInt64OrDefault('abc', CAST('-1', 'Int64')) Result: Row 1: ────── toInt64OrDefault('-64', CAST('-1', 'Int64')): -64 toInt64OrDefault('abc', CAST('-1', 'Int64')): -1 ## toInt128 [¶](https://www.tinybird.co/docs/about:blank#toint128) Converts an input value to a value of type `Int128` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt128(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt128('0xc0fe');` . If the input value can't be represented within the bounds of Int128, the result over or under flows. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit integer value. Int128. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt128(-128), toInt128(-128.8), toInt128('-128') Result: Row 1: ────── toInt128(-128): -128 toInt128(-128.8): -128 toInt128('-128'): -128 ## toInt128OrZero [¶](https://www.tinybird.co/docs/about:blank#toint128orzero) Like `toInt128` , this function converts an input value to a value of type Int128 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt128OrZero(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt128OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit integer value if successful, otherwise `0` . Int128. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt128OrZero('-128'), toInt128OrZero('abc') Result: Row 1: ────── toInt128OrZero('-128'): -128 toInt128OrZero('abc'): 0 ## toInt128OrNull [¶](https://www.tinybird.co/docs/about:blank#toint128ornull) Like `toInt128` , this function converts an input value to a value of type Int128 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt128OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt128OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit integer value if successful, otherwise `NULL` . Int128 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt128OrNull('-128'), toInt128OrNull('abc') Result: Row 1: ────── toInt128OrNull('-128'): -128 toInt128OrNull('abc'): ᴺᵁᴸᴸ ## toInt128OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint128ordefault) Like `toInt128` , this function converts an input value to a value of type Int128 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt128OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int128` is unsuccessful. Int128. Supported arguments: - (U)Int8/16/32/64/128/256. - Float32/64. - String representations of (U)Int8/16/32/128/256. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt128OrDefault('0xc0fe', CAST('-1', 'Int128'));` . If the input value can't be represented within the bounds of Int128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int128. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt128OrDefault('-128', CAST('-1', 'Int128')), toInt128OrDefault('abc', CAST('-1', 'Int128')) Result: Row 1: ────── toInt128OrDefault('-128', CAST('-1', 'Int128')): -128 toInt128OrDefault('abc', CAST('-1', 'Int128')): -1 ## toInt256 [¶](https://www.tinybird.co/docs/about:blank#toint256) Converts an input value to a value of type `Int256` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt256(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt256('0xc0fe');` . If the input value can't be represented within the bounds of Int256, the result over or under flows. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit integer value. Int256. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt256(-256), toInt256(-256.256), toInt256('-256') Result: Row 1: ────── toInt256(-256): -256 toInt256(-256.256): -256 toInt256('-256'): -256 ## toInt256OrZero [¶](https://www.tinybird.co/docs/about:blank#toint256orzero) Like `toInt256` , this function converts an input value to a value of type Int256 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt256OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt256OrZero('0xc0fe');` . If the input value can't be represented within the bounds of Int256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit integer value if successful, otherwise `0` . Int256. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt256OrZero('-256'), toInt256OrZero('abc') Result: Row 1: ────── toInt256OrZero('-256'): -256 toInt256OrZero('abc'): 0 ## toInt256OrNull [¶](https://www.tinybird.co/docs/about:blank#toint256ornull) Like `toInt256` , this function converts an input value to a value of type Int256 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt256OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toInt256OrNull('0xc0fe');` . If the input value can't be represented within the bounds of Int256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit integer value if successful, otherwise `NULL` . Int256 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt256OrNull('-256'), toInt256OrNull('abc') Result: Row 1: ────── toInt256OrNull('-256'): -256 toInt256OrNull('abc'): ᴺᵁᴸᴸ ## toInt256OrDefault [¶](https://www.tinybird.co/docs/about:blank#toint256ordefault) Like `toInt256` , this function converts an input value to a value of type Int256 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toInt256OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Int256` is unsuccessful. Int256. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` - String representations of binary and hexadecimal values, e.g. `SELECT toInt256OrDefault('0xc0fe', CAST('-1', 'Int256'));` If the input value can't be represented within the bounds of Int256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit integer value if successful, otherwise returns the default value if passed or `0` if not. Int256. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt256OrDefault('-256', CAST('-1', 'Int256')), toInt256OrDefault('abc', CAST('-1', 'Int256')) Result: Row 1: ────── toInt256OrDefault('-256', CAST('-1', 'Int256')): -256 toInt256OrDefault('abc', CAST('-1', 'Int256')): -1 ## toUInt8 [¶](https://www.tinybird.co/docs/about:blank#touint8) Converts an input value to a value of type `UInt8` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt8(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt8('0xc0fe');` . If the input value can't be represented within the bounds of UInt8, overflow or underflow of the result occurs. This isn't considered an error. For example: `SELECT toUInt8(256) == 0;`. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit unsigned integer value. UInt8. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8(8), toUInt8(8.8), toUInt8('8') Result: Row 1: ────── toUInt8(8): 8 toUInt8(8.8): 8 toUInt8('8'): 8 ## toUInt8OrZero [¶](https://www.tinybird.co/docs/about:blank#touint8orzero) Like `toUInt8` , this function converts an input value to a value of type UInt8 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt8OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of ordinary Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt8OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit unsigned integer value if successful, otherwise `0` . UInt8. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8OrZero('-8'), toUInt8OrZero('abc') Result: Row 1: ────── toUInt8OrZero('-8'): 0 toUInt8OrZero('abc'): 0 ## toUInt8OrNull [¶](https://www.tinybird.co/docs/about:blank#touint8ornull) Like `toUInt8` , this function converts an input value to a value of type UInt8 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt8OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt8OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit unsigned integer value if successful, otherwise `NULL` . UInt8 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8OrNull('8'), toUInt8OrNull('abc') Result: Row 1: ────── toUInt8OrNull('8'): 8 toUInt8OrNull('abc'): ᴺᵁᴸᴸ ## toUInt8OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint8ordefault) Like `toUInt8` , this function converts an input value to a value of type UInt8 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt8OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `UInt8` is unsuccessful. UInt8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt8OrDefault('0xc0fe', CAST('0', 'UInt8'));` . If the input value can't be represented within the bounds of UInt8, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 8-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt8. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8OrDefault('8', CAST('0', 'UInt8')), toUInt8OrDefault('abc', CAST('0', 'UInt8')) Result: Row 1: ────── toUInt8OrDefault('8', CAST('0', 'UInt8')): 8 toUInt8OrDefault('abc', CAST('0', 'UInt8')): 0 ## toUInt16 [¶](https://www.tinybird.co/docs/about:blank#touint16) Converts an input value to a value of type `UInt16` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt16(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt16('0xc0fe');` . If the input value can't be represented within the bounds of UInt16, overflow or underflow of the result occurs. This isn't considered an error. For example: `SELECT toUInt16(65536) == 0;`. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit unsigned integer value. UInt16. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt16(16), toUInt16(16.16), toUInt16('16') Result: Row 1: ────── toUInt16(16): 16 toUInt16(16.16): 16 toUInt16('16'): 16 ## toUInt16OrZero [¶](https://www.tinybird.co/docs/about:blank#touint16orzero) Like `toUInt16` , this function converts an input value to a value of type UInt16 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt16OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt16OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt16, overflow or underflow of the result occurs. This isn't considered as an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit unsigned integer value if successful, otherwise `0` . UInt16. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt16OrZero('16'), toUInt16OrZero('abc') Result: Row 1: ────── toUInt16OrZero('16'): 16 toUInt16OrZero('abc'): 0 ## toUInt16OrNull [¶](https://www.tinybird.co/docs/about:blank#touint16ornull) Like `toUInt16` , this function converts an input value to a value of type UInt16 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt16OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt16OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt16, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit unsigned integer value if successful, otherwise `NULL` . UInt16 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt16OrNull('16'), toUInt16OrNull('abc') Result: Row 1: ────── toUInt16OrNull('16'): 16 toUInt16OrNull('abc'): ᴺᵁᴸᴸ ## toUInt16OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint16ordefault) Like `toUInt16` , this function converts an input value to a value of type UInt16 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt16OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `UInt16` is unsuccessful. UInt16. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt16OrDefault('0xc0fe', CAST('0', 'UInt16'));` . If the input value can't be represented within the bounds of UInt16, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 16-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt16. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt16OrDefault('16', CAST('0', 'UInt16')), toUInt16OrDefault('abc', CAST('0', 'UInt16')) Result: Row 1: ────── toUInt16OrDefault('16', CAST('0', 'UInt16')): 16 toUInt16OrDefault('abc', CAST('0', 'UInt16')): 0 ## toUInt32 [¶](https://www.tinybird.co/docs/about:blank#touint32) Converts an input value to a value of type `UInt32` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt32(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt32('0xc0fe');` . If the input value can't be represented within the bounds of UInt32, the result over or under flows. This isn't considered an error. For example: `SELECT toUInt32(4294967296) == 0;` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit unsigned integer value. UInt32. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt32(32), toUInt32(32.32), toUInt32('32') Result: Row 1: ────── toUInt32(32): 32 toUInt32(32.32): 32 toUInt32('32'): 32 ## toUInt32OrZero [¶](https://www.tinybird.co/docs/about:blank#touint32orzero) Like `toUInt32` , this function converts an input value to a value of type UInt32 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt32OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt32OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit unsigned integer value if successful, otherwise `0` . UInt32 The function uses rounding towards zero , meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt32OrZero('32'), toUInt32OrZero('abc') Result: Row 1: ────── toUInt32OrZero('32'): 32 toUInt32OrZero('abc'): 0 ## toUInt32OrNull [¶](https://www.tinybird.co/docs/about:blank#touint32ornull) Like `toUInt32` , this function converts an input value to a value of type UInt32 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt32OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt32OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit unsigned integer value if successful, otherwise `NULL` . UInt32 / NULL. The function uses rounding towards zero , meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt32OrNull('32'), toUInt32OrNull('abc') Result: Row 1: ────── toUInt32OrNull('32'): 32 toUInt32OrNull('abc'): ᴺᵁᴸᴸ ## toUInt32OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint32ordefault) Like `toUInt32` , this function converts an input value to a value of type UInt32 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt32OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `UInt32` is unsuccessful. UInt32. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt32OrDefault('0xc0fe', CAST('0', 'UInt32'));` . If the input value can't be represented within the bounds of UInt32, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt32. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt32OrDefault('32', CAST('0', 'UInt32')), toUInt32OrDefault('abc', CAST('0', 'UInt32')) Result: Row 1: ────── toUInt32OrDefault('32', CAST('0', 'UInt32')): 32 toUInt32OrDefault('abc', CAST('0', 'UInt32')): 0 ## toUInt64 [¶](https://www.tinybird.co/docs/about:blank#touint64) Converts an input value to a value of type `UInt64` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt64(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported types: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt64('0xc0fe');` . If the input value can't be represented within the bounds of UInt64, the result over or under flows. This isn't considered an error. For example: `SELECT toUInt64(18446744073709551616) == 0;` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit unsigned integer value. UInt64. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt64(64), toUInt64(64.64), toUInt64('64') Result: Row 1: ────── toUInt64(64): 64 toUInt64(64.64): 64 toUInt64('64'): 64 ## toUInt64OrZero [¶](https://www.tinybird.co/docs/about:blank#touint64orzero) Like `toUInt64` , this function converts an input value to a value of type UInt64 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt64OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt64OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit unsigned integer value if successful, otherwise `0` . UInt64. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt64OrZero('64'), toUInt64OrZero('abc') Result: Row 1: ────── toUInt64OrZero('64'): 64 toUInt64OrZero('abc'): 0 ## toUInt64OrNull [¶](https://www.tinybird.co/docs/about:blank#touint64ornull) Like `toUInt64` , this function converts an input value to a value of type UInt64 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt64OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt64OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit unsigned integer value if successful, otherwise `NULL` . UInt64 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt64OrNull('64'), toUInt64OrNull('abc') Result: Row 1: ────── toUInt64OrNull('64'): 64 toUInt64OrNull('abc'): ᴺᵁᴸᴸ ## toUInt64OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint64ordefault) Like `toUInt64` , this function converts an input value to a value of type UInt64 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt64OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `defauult` (optional): The default value to return if parsing to type `UInt64` is unsuccessful. UInt64. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt64OrDefault('0xc0fe', CAST('0', 'UInt64'));` . If the input value can't be represented within the bounds of UInt64, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt64. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt64OrDefault('64', CAST('0', 'UInt64')), toUInt64OrDefault('abc', CAST('0', 'UInt64')) Result: Row 1: ────── toUInt64OrDefault('64', CAST('0', 'UInt64')): 64 toUInt64OrDefault('abc', CAST('0', 'UInt64')): 0 ## toUInt128 [¶](https://www.tinybird.co/docs/about:blank#touint128) Converts an input value to a value of type `UInt128` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt128(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt128('0xc0fe');` . If the input value can't be represented within the bounds of UInt128, the result over or under flows. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit unsigned integer value. UInt128. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt128(128), toUInt128(128.8), toUInt128('128') Result: Row 1: ────── toUInt128(128): 128 toUInt128(128.8): 128 toUInt128('128'): 128 ## toUInt128OrZero [¶](https://www.tinybird.co/docs/about:blank#touint128orzero) Like `toUInt128` , this function converts an input value to a value of type UInt128 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt128OrZero(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt128OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit unsigned integer value if successful, otherwise `0` . UInt128. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt128OrZero('128'), toUInt128OrZero('abc') Result: Row 1: ────── toUInt128OrZero('128'): 128 toUInt128OrZero('abc'): 0 ## toUInt128OrNull [¶](https://www.tinybird.co/docs/about:blank#touint128ornull) Like `toUInt128` , this function converts an input value to a value of type UInt128 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt128OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. Expression / String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt128OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit unsigned integer value if successful, otherwise `NULL` . UInt128 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt128OrNull('128'), toUInt128OrNull('abc') Result: Row 1: ────── toUInt128OrNull('128'): 128 toUInt128OrNull('abc'): ᴺᵁᴸᴸ ## toUInt128OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint128ordefault) Like `toUInt128` , this function converts an input value to a value of type UInt128 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt128OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `UInt128` is unsuccessful. UInt128. Supported arguments: - (U)Int8/16/32/64/128/256. - Float32/64. - String representations of (U)Int8/16/32/128/256. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt128OrDefault('0xc0fe', CAST('0', 'UInt128'));` . If the input value can't be represented within the bounds of UInt128, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 128-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt128. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt128OrDefault('128', CAST('0', 'UInt128')), toUInt128OrDefault('abc', CAST('0', 'UInt128')) Result: Row 1: ────── toUInt128OrDefault('128', CAST('0', 'UInt128')): 128 toUInt128OrDefault('abc', CAST('0', 'UInt128')): 0 ## toUInt256 [¶](https://www.tinybird.co/docs/about:blank#touint256) Converts an input value to a value of type `UInt256` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt256(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Unsupported arguments: - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt256('0xc0fe');` . If the input value can't be represented within the bounds of UInt256, the result over or under flows. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit unsigned integer value. Int256. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt256(256), toUInt256(256.256), toUInt256('256') Result: Row 1: ────── toUInt256(256): 256 toUInt256(256.256): 256 toUInt256('256'): 256 ## toUInt256OrZero [¶](https://www.tinybird.co/docs/about:blank#touint256orzero) Like `toUInt256` , this function converts an input value to a value of type UInt256 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt256OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `0` ): - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt256OrZero('0xc0fe');` . If the input value can't be represented within the bounds of UInt256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit unsigned integer value if successful, otherwise `0` . UInt256. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt256OrZero('256'), toUInt256OrZero('abc') Result: Row 1: ────── toUInt256OrZero('256'): 256 toUInt256OrZero('abc'): 0 ## toUInt256OrNull [¶](https://www.tinybird.co/docs/about:blank#touint256ornull) Like `toUInt256` , this function converts an input value to a value of type UInt256 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt256OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256. Unsupported arguments (return `\N` ) - String representations of Float32/64 values, including `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toUInt256OrNull('0xc0fe');` . If the input value can't be represented within the bounds of UInt256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit unsigned integer value if successful, otherwise `NULL` . UInt256 / NULL. The function uses rounding towards zero, meaning it truncates fractional digits of numbers. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt256OrNull('256'), toUInt256OrNull('abc') Result: Row 1: ────── toUInt256OrNull('256'): 256 toUInt256OrNull('abc'): ᴺᵁᴸᴸ ## toUInt256OrDefault [¶](https://www.tinybird.co/docs/about:blank#touint256ordefault) Like `toUInt256` , this function converts an input value to a value of type UInt256 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUInt256OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `UInt256` is unsuccessful. UInt256. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values of type Float32/64. Arguments for which the default value is returned: - String representations of Float32/64 values, including `NaN` and `Inf` - String representations of binary and hexadecimal values, e.g. `SELECT toUInt256OrDefault('0xc0fe', CAST('0', 'UInt256'));` If the input value can't be represented within the bounds of UInt256, overflow or underflow of the result occurs. This isn't considered an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 256-bit unsigned integer value if successful, otherwise returns the default value if passed or `0` if not. UInt256. - The function uses rounding towards zero, meaning it truncates fractional digits of numbers. - The default value type should be the same as the cast type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt256OrDefault('-256', CAST('0', 'UInt256')), toUInt256OrDefault('abc', CAST('0', 'UInt256')) Result: Row 1: ────── toUInt256OrDefault('-256', CAST('0', 'UInt256')): 0 toUInt256OrDefault('abc', CAST('0', 'UInt256')): 0 ## toFloat32 [¶](https://www.tinybird.co/docs/about:blank#tofloat32) Converts an input value to a value of type `Float32` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat32(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values of type (U)Int8/16/32/64/128/256. - String representations of (U)Int8/16/32/128/256. - Values of type Float32/64, including `NaN` and `Inf` . - String representations of Float32/64, including `NaN` and `Inf` (case-insensitive). Unsupported arguments: - String representations of binary and hexadecimal values, e.g. `SELECT toFloat32('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit floating point value. Float32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat32(42.7), toFloat32('42.7'), toFloat32('NaN') Result: Row 1: ────── toFloat32(42.7): 42.7 toFloat32('42.7'): 42.7 toFloat32('NaN'): nan ## toFloat32OrZero [¶](https://www.tinybird.co/docs/about:blank#tofloat32orzero) Like `toFloat32` , this function converts an input value to a value of type Float32 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat32OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256, Float32/64. Unsupported arguments (return `0` ): - String representations of binary and hexadecimal values, e.g. `SELECT toFloat32OrZero('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit Float value if successful, otherwise `0` . Float32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat32OrZero('42.7'), toFloat32OrZero('abc') Result: Row 1: ────── toFloat32OrZero('42.7'): 42.7 toFloat32OrZero('abc'): 0 ## toFloat32OrNull [¶](https://www.tinybird.co/docs/about:blank#tofloat32ornull) Like `toFloat32` , this function converts an input value to a value of type Float32 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat32OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256, Float32/64. Unsupported arguments (return `\N` ): - String representations of binary and hexadecimal values, e.g. `SELECT toFloat32OrNull('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit Float value if successful, otherwise `\N` . Float32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat32OrNull('42.7'), toFloat32OrNull('abc') Result: Row 1: ────── toFloat32OrNull('42.7'): 42.7 toFloat32OrNull('abc'): ᴺᵁᴸᴸ ## toFloat32OrDefault [¶](https://www.tinybird.co/docs/about:blank#tofloat32ordefault) Like `toFloat32` , this function converts an input value to a value of type Float32 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat32OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Float32` is unsuccessful. Float32. Supported arguments: - Values of type (U)Int8/16/32/64/128/256. - String representations of (U)Int8/16/32/128/256. - Values of type Float32/64, including `NaN` and `Inf` . - String representations of Float32/64, including `NaN` and `Inf` (case-insensitive). Arguments for which the default value is returned: - String representations of binary and hexadecimal values, e.g. `SELECT toFloat32OrDefault('0xc0fe', CAST('0', 'Float32'));` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 32-bit Float value if successful, otherwise returns the default value if passed or `0` if not. Float32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat32OrDefault('8', CAST('0', 'Float32')), toFloat32OrDefault('abc', CAST('0', 'Float32')) Result: Row 1: ────── toFloat32OrDefault('8', CAST('0', 'Float32')): 8 toFloat32OrDefault('abc', CAST('0', 'Float32')): 0 ## toFloat64 [¶](https://www.tinybird.co/docs/about:blank#tofloat64) Converts an input value to a value of type `Float64` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat64(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. Supported arguments: - Values of type (U)Int8/16/32/64/128/256. - String representations of (U)Int8/16/32/128/256. - Values of type Float32/64, including `NaN` and `Inf` . - String representations of type Float32/64, including `NaN` and `Inf` (case-insensitive). Unsupported arguments: - String representations of binary and hexadecimal values, e.g. `SELECT toFloat64('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit floating point value. Float64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat64(42.7), toFloat64('42.7'), toFloat64('NaN') Result: Row 1: ────── toFloat64(42.7): 42.7 toFloat64('42.7'): 42.7 toFloat64('NaN'): nan ## toFloat64OrZero [¶](https://www.tinybird.co/docs/about:blank#tofloat64orzero) Like `toFloat64` , this function converts an input value to a value of type Float64 but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat64OrZero(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256, Float32/64. Unsupported arguments (return `0` ): - String representations of binary and hexadecimal values, e.g. `SELECT toFloat64OrZero('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit Float value if successful, otherwise `0` . Float64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat64OrZero('42.7'), toFloat64OrZero('abc') Result: Row 1: ────── toFloat64OrZero('42.7'): 42.7 toFloat64OrZero('abc'): 0 ## toFloat64OrNull [¶](https://www.tinybird.co/docs/about:blank#tofloat64ornull) Like `toFloat64` , this function converts an input value to a value of type Float64 but returns `NULL` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat64OrNull(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A String representation of a number. String. Supported arguments: - String representations of (U)Int8/16/32/128/256, Float32/64. Unsupported arguments (return `\N` ): - String representations of binary and hexadecimal values, e.g. `SELECT toFloat64OrNull('0xc0fe');` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit Float value if successful, otherwise `\N` . Float64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat64OrNull('42.7'), toFloat64OrNull('abc') Result: Row 1: ────── toFloat64OrNull('42.7'): 42.7 toFloat64OrNull('abc'): ᴺᵁᴸᴸ ## toFloat64OrDefault [¶](https://www.tinybird.co/docs/about:blank#tofloat64ordefault) Like `toFloat64` , this function converts an input value to a value of type Float64 but returns the default value in case of an error. If no `default` value is passed then `0` is returned in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFloat64OrDefault(expr[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression / String. - `default` (optional): The default value to return if parsing to type `Float64` is unsuccessful. Float64. Supported arguments: - Values of type (U)Int8/16/32/64/128/256. - String representations of (U)Int8/16/32/128/256. - Values of type Float32/64, including `NaN` and `Inf` . - String representations of Float32/64, including `NaN` and `Inf` (case-insensitive). Arguments for which the default value is returned: - String representations of binary and hexadecimal values, e.g. `SELECT toFloat64OrDefault('0xc0fe', CAST('0', 'Float64'));` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 64-bit Float value if successful, otherwise returns the default value if passed or `0` if not. Float64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFloat64OrDefault('8', CAST('0', 'Float64')), toFloat64OrDefault('abc', CAST('0', 'Float64')) Result: Row 1: ────── toFloat64OrDefault('8', CAST('0', 'Float64')): 8 toFloat64OrDefault('abc', CAST('0', 'Float64')): 0 ## toDate [¶](https://www.tinybird.co/docs/about:blank#todate) Converts the argument to Date data type. If the argument is DateTime or DateTime64, it truncates it and leaves the date component of the DateTime: SELECT now() AS x, toDate(x) ┌───────────────────x─┬─toDate(now())─┐ │ 2022-12-30 13:44:17 │ 2022-12-30 │ └─────────────────────┴───────────────┘ If the argument is a String, it's parsed as Date or DateTime. If it was parsed as DateTime, the date component is being used: SELECT toDate('2022-12-30') AS x, toTypeName(x) ┌──────────x─┬─toTypeName(toDate('2022-12-30'))─┐ │ 2022-12-30 │ Date │ └────────────┴──────────────────────────────────┘ 1 row in set. Elapsed: 0.001 sec. SELECT toDate('2022-12-30 01:02:03') AS x, toTypeName(x) ┌──────────x─┬─toTypeName(toDate('2022-12-30 01:02:03'))─┐ │ 2022-12-30 │ Date │ └────────────┴───────────────────────────────────────────┘ If the argument is a number and looks like a UNIX timestamp (is greater than 65535), it's interpreted as a DateTime, then truncated to Date in the current timezone. The timezone argument can be specified as a second argument of the function. The truncation to Date depends on the timezone: SELECT now() AS current_time, toUnixTimestamp(current_time) AS ts, toDateTime(ts) AS time_Amsterdam, toDateTime(ts, 'Pacific/Apia') AS time_Samoa, toDate(time_Amsterdam) AS date_Amsterdam, toDate(time_Samoa) AS date_Samoa, toDate(ts) AS date_Amsterdam_2, toDate(ts, 'Pacific/Apia') AS date_Samoa_2 Row 1: ────── current_time: 2022-12-30 13:51:54 ts: 1672404714 time_Amsterdam: 2022-12-30 13:51:54 time_Samoa: 2022-12-31 01:51:54 date_Amsterdam: 2022-12-30 date_Samoa: 2022-12-31 date_Amsterdam_2: 2022-12-30 date_Samoa_2: 2022-12-31 The example above demonstrates how the same UNIX timestamp can be interpreted as different dates in different time zones. If the argument is a number and it's smaller than 65536, it's interpreted as the number of days since 1970-01-01 (the first UNIX day) and converted to Date. It corresponds to the internal numeric representation of the `Date` data type. Example: SELECT toDate(12345) ┌─toDate(12345)─┐ │ 2003-10-20 │ └───────────────┘ This conversion doesn't depend on timezones. If the argument doesn't fit in the range of the Date type, it results in an implementation-defined behavior, that can saturate to the maximum supported date or overflow: SELECT toDate(10000000000.) ┌─toDate(10000000000.)─┐ │ 2106-02-07 │ └──────────────────────┘ The function `toDate` can be also written in alternative forms: SELECT now() AS time, toDate(time), DATE(time), CAST(time, 'Date') ┌────────────────time─┬─toDate(now())─┬─DATE(now())─┬─CAST(now(), 'Date')─┐ │ 2022-12-30 13:54:58 │ 2022-12-30 │ 2022-12-30 │ 2022-12-30 │ └─────────────────────┴───────────────┴─────────────┴─────────────────────┘ ## toDateOrZero [¶](https://www.tinybird.co/docs/about:blank#todateorzero) The same as toDate but returns lower boundary of Date if an invalid argument is received. Only String argument is supported. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateOrZero('2022-12-30'), toDateOrZero('') Result: ┌─toDateOrZero('2022-12-30')─┬─toDateOrZero('')─┐ │ 2022-12-30 │ 1970-01-01 │ └────────────────────────────┴──────────────────┘ ## toDateOrNull [¶](https://www.tinybird.co/docs/about:blank#todateornull) The same as toDate but returns `NULL` if an invalid argument is received. Only String argument is supported. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateOrNull('2022-12-30'), toDateOrNull('') Result: ┌─toDateOrNull('2022-12-30')─┬─toDateOrNull('')─┐ │ 2022-12-30 │ ᴺᵁᴸᴸ │ └────────────────────────────┴──────────────────┘ ## toDateOrDefault [¶](https://www.tinybird.co/docs/about:blank#todateordefault) Like toDate but if unsuccessful, returns a default value which is either the second argument (if specified), or otherwise the lower boundary of Date. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateOrDefault(expr [, default_value]) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateOrDefault('2022-12-30'), toDateOrDefault('', '2023-01-01'::Date) Result: ┌─toDateOrDefault('2022-12-30')─┬─toDateOrDefault('', CAST('2023-01-01', 'Date'))─┐ │ 2022-12-30 │ 2023-01-01 │ └───────────────────────────────┴─────────────────────────────────────────────────┘ ## toDateTime [¶](https://www.tinybird.co/docs/about:blank#todatetime) Converts an input value to DateTime. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTime(expr[, time_zone ]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, Int, Date or DateTime. - `time_zone` : Time zone. String. If `expr` is a number, it's interpreted as the number of seconds since the beginning of the Unix Epoch (as Unix timestamp). If `expr` is a String, it may be interpreted as a Unix timestamp or as a string representation of date / date with time. Thus, parsing of short numbers' string representations (up to 4 digits) is explicitly disabled due to ambiguity, e.g. a string `'1999'` may be both a year (an incomplete string representation of Date / DateTime) or a unix timestamp. Longer numeric strings are allowed. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A date time. DateTime ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTime('2022-12-30 13:44:17'), toDateTime(1685457500, 'UTC') Result: ┌─toDateTime('2022-12-30 13:44:17')─┬─toDateTime(1685457500, 'UTC')─┐ │ 2022-12-30 13:44:17 │ 2023-05-30 14:38:20 │ └───────────────────────────────────┴───────────────────────────────┘ ## toDateTimeOrZero [¶](https://www.tinybird.co/docs/about:blank#todatetimeorzero) The same as toDateTime but returns lower boundary of DateTime if an invalid argument is received. Only String argument is supported. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTimeOrZero('2022-12-30 13:44:17'), toDateTimeOrZero('') Result: ┌─toDateTimeOrZero('2022-12-30 13:44:17')─┬─toDateTimeOrZero('')─┐ │ 2022-12-30 13:44:17 │ 1970-01-01 00:00:00 │ └─────────────────────────────────────────┴──────────────────────┘ ## toDateTimeOrNull [¶](https://www.tinybird.co/docs/about:blank#todatetimeornull) The same as toDateTime but returns `NULL` if an invalid argument is received. Only String argument is supported. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTimeOrNull('2022-12-30 13:44:17'), toDateTimeOrNull('') Result: ┌─toDateTimeOrNull('2022-12-30 13:44:17')─┬─toDateTimeOrNull('')─┐ │ 2022-12-30 13:44:17 │ ᴺᵁᴸᴸ │ └─────────────────────────────────────────┴──────────────────────┘ ## toDateTimeOrDefault [¶](https://www.tinybird.co/docs/about:blank#todatetimeordefault) Like toDateTime but if unsuccessful, returns a default value which is either the third argument (if specified), or otherwise the lower boundary of DateTime. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTimeOrDefault(expr [, time_zone [, default_value]]) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTimeOrDefault('2022-12-30 13:44:17'), toDateTimeOrDefault('', 'UTC', '2023-01-01'::DateTime('UTC')) Result: ┌─toDateTimeOrDefault('2022-12-30 13:44:17')─┬─toDateTimeOrDefault('', 'UTC', CAST('2023-01-01', 'DateTime(\'UTC\')'))─┐ │ 2022-12-30 13:44:17 │ 2023-01-01 00:00:00 │ └────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────┘ ## toDate32 [¶](https://www.tinybird.co/docs/about:blank#todate32) Converts the argument to the Date32 data type. If the value is outside the range, `toDate32` returns the border values supported by Date32. If the argument has Date type, it's borders are taken into account. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDate32(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, UInt32 or Date. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A calendar date. Type Date32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) 1. The value is within the range: SELECT toDate32('1955-01-01') AS value, toTypeName(value) ┌──────value─┬─toTypeName(toDate32('1925-01-01'))─┐ │ 1955-01-01 │ Date32 │ └────────────┴────────────────────────────────────┘ 1. The value is outside the range: SELECT toDate32('1899-01-01') AS value, toTypeName(value) ┌──────value─┬─toTypeName(toDate32('1899-01-01'))─┐ │ 1900-01-01 │ Date32 │ └────────────┴────────────────────────────────────┘ 1. With Date argument: SELECT toDate32(toDate('1899-01-01')) AS value, toTypeName(value) ┌──────value─┬─toTypeName(toDate32(toDate('1899-01-01')))─┐ │ 1970-01-01 │ Date32 │ └────────────┴────────────────────────────────────────────┘ ## toDate32OrZero [¶](https://www.tinybird.co/docs/about:blank#todate32orzero) The same as toDate32 but returns the min value of Date32 if an invalid argument is received. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDate32OrZero('1899-01-01'), toDate32OrZero('') Result: ┌─toDate32OrZero('1899-01-01')─┬─toDate32OrZero('')─┐ │ 1900-01-01 │ 1900-01-01 │ └──────────────────────────────┴────────────────────┘ ## toDate32OrNull [¶](https://www.tinybird.co/docs/about:blank#todate32ornull) The same as toDate32 but returns `NULL` if an invalid argument is received. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDate32OrNull('1955-01-01'), toDate32OrNull('') Result: ┌─toDate32OrNull('1955-01-01')─┬─toDate32OrNull('')─┐ │ 1955-01-01 │ ᴺᵁᴸᴸ │ └──────────────────────────────┴────────────────────┘ ## toDate32OrDefault [¶](https://www.tinybird.co/docs/about:blank#todate32ordefault) Converts the argument to the Date32 data type. If the value is outside the range, `toDate32OrDefault` returns the lower border value supported by Date32. If the argument has Date type, it's borders are taken into account. Returns default value if an invalid argument is received. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDate32OrDefault('1930-01-01', toDate32('2020-01-01')), toDate32OrDefault('xx1930-01-01', toDate32('2020-01-01')) Result: ┌─toDate32OrDefault('1930-01-01', toDate32('2020-01-01'))─┬─toDate32OrDefault('xx1930-01-01', toDate32('2020-01-01'))─┐ │ 1930-01-01 │ 2020-01-01 │ └─────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┘ ## toDateTime64 [¶](https://www.tinybird.co/docs/about:blank#todatetime64) Converts an input value to a value of type DateTime64. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTime64(expr, scale, [timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, UInt32, Float or DateTime. - `scale` - Tick size (precision): 10-precision seconds. Valid range: [ 0 : 9 ]. - `timezone` (optional) - Time zone of the specified datetime64 object. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A calendar date and time of day, with sub-second precision. DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) 1. The value is within the range: SELECT toDateTime64('1955-01-01 00:00:00.000', 3) AS value, toTypeName(value) ┌───────────────────value─┬─toTypeName(toDateTime64('1955-01-01 00:00:00.000', 3))─┐ │ 1955-01-01 00:00:00.000 │ DateTime64(3) │ └─────────────────────────┴────────────────────────────────────────────────────────┘ 1. As decimal with precision: SELECT toDateTime64(1546300800.000, 3) AS value, toTypeName(value) ┌───────────────────value─┬─toTypeName(toDateTime64(1546300800., 3))─┐ │ 2019-01-01 00:00:00.000 │ DateTime64(3) │ └─────────────────────────┴──────────────────────────────────────────┘ Without the decimal point the value is still treated as Unix Timestamp in seconds: SELECT toDateTime64(1546300800000, 3) AS value, toTypeName(value) ┌───────────────────value─┬─toTypeName(toDateTime64(1546300800000, 3))─┐ │ 2282-12-31 00:00:00.000 │ DateTime64(3) │ └─────────────────────────┴────────────────────────────────────────────┘ 1. With `timezone` : SELECT toDateTime64('2019-01-01 00:00:00', 3, 'Asia/Istanbul') AS value, toTypeName(value) ┌───────────────────value─┬─toTypeName(toDateTime64('2019-01-01 00:00:00', 3, 'Asia/Istanbul'))─┐ │ 2019-01-01 00:00:00.000 │ DateTime64(3, 'Asia/Istanbul') │ └─────────────────────────┴─────────────────────────────────────────────────────────────────────┘ ## toDateTime64OrZero [¶](https://www.tinybird.co/docs/about:blank#todatetime64orzero) Like toDateTime64, this function converts an input value to a value of type DateTime64 but returns the min value of DateTime64 if an invalid argument is received. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTime64OrZero(expr, scale, [timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, UInt32, Float or DateTime. - `scale` - Tick size (precision): 10-precision seconds. Valid range: [ 0 : 9 ]. - `timezone` (optional) - Time zone of the specified DateTime64 object. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A calendar date and time of day, with sub-second precision, otherwise the minimum value of `DateTime64` : `1970-01-01 01:00:00.000` . DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTime64OrZero('2008-10-12 00:00:00 00:30:30', 3) AS invalid_arg Result: ┌─────────────invalid_arg─┐ │ 1970-01-01 01:00:00.000 │ └─────────────────────────┘ ## toDateTime64OrNull [¶](https://www.tinybird.co/docs/about:blank#todatetime64ornull) Like toDateTime64, this function converts an input value to a value of type DateTime64 but returns `NULL` if an invalid argument is received. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTime64OrNull(expr, scale, [timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, UInt32, Float or DateTime. - `scale` - Tick size (precision): 10-precision seconds. Valid range: [ 0 : 9 ]. - `timezone` (optional) - Time zone of the specified DateTime64 object. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A calendar date and time of day, with sub-second precision, otherwise `NULL` . DateTime64/NULL. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTime64OrNull('1976-10-18 00:00:00.30', 3) AS valid_arg, toDateTime64OrNull('1976-10-18 00:00:00 30', 3) AS invalid_arg Result: ┌───────────────valid_arg─┬─invalid_arg─┐ │ 1976-10-18 00:00:00.300 │ ᴺᵁᴸᴸ │ └─────────────────────────┴─────────────┘ ## toDateTime64OrDefault [¶](https://www.tinybird.co/docs/about:blank#todatetime64ordefault) Like toDateTime64, this function converts an input value to a value of type DateTime64, but returns either the default value of DateTime64 or the provided default if an invalid argument is received. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDateTime64OrNull(expr, scale, [timezone, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : The value. String, UInt32, Float or DateTime. - `scale` - Tick size (precision): 10-precision seconds. Valid range: [ 0 : 9 ]. - `timezone` (optional) - Time zone of the specified DateTime64 object. - `default` (optional) - Default value to return if an invalid argument is received. DateTime64. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A calendar date and time of day, with sub-second precision, otherwise the minimum value of `DateTime64` or the `default` value if provided. DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDateTime64OrDefault('1976-10-18 00:00:00 30', 3) AS invalid_arg, toDateTime64OrDefault('1976-10-18 00:00:00 30', 3, 'UTC', toDateTime64('2001-01-01 00:00:00.00',3)) AS invalid_arg_with_default Result: ┌─────────────invalid_arg─┬─invalid_arg_with_default─┐ │ 1970-01-01 01:00:00.000 │ 2000-12-31 23:00:00.000 │ └─────────────────────────┴──────────────────────────┘ ## toDecimal32 [¶](https://www.tinybird.co/docs/about:blank#todecimal32) Converts an input value to a value of type `Decimal(9, S)` with scale of `S` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal32(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. - `S` : Scale parameter between 0 and 9, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values or string representations of type Float32/64. Unsupported arguments: - Values or string representations of Float32/64 values `NaN` and `Inf` (case-insensitive). - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal32('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal32`: `( -1 * 10^(9 - S), 1 * 10^(9 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an exception. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal32(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal32('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(9, S)` . Decimal32(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal32(2, 1) AS a, toTypeName(a) AS type_a, toDecimal32(4.2, 2) AS b, toTypeName(b) AS type_b, toDecimal32('4.2', 3) AS c, toTypeName(c) AS type_c Result: Row 1: ────── a: 2 type_a: Decimal(9, 1) b: 4.2 type_b: Decimal(9, 2) c: 4.2 type_c: Decimal(9, 3) ## toDecimal32OrZero [¶](https://www.tinybird.co/docs/about:blank#todecimal32orzero) Like `toDecimal32` , this function converts an input value to a value of type Decimal(9, S) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal32OrZero(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 9, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal32OrZero('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal32`: `( -1 * 10^(9 - S), 1 * 10^(9 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(9, S)` if successful, otherwise `0` with `S` decimal places. Decimal32(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal32OrZero(toString(-1.111), 5) AS a, toTypeName(a), toDecimal32OrZero(toString('Inf'), 5) as b, toTypeName(b) Result: Row 1: ────── a: -1.111 toTypeName(a): Decimal(9, 5) b: 0 toTypeName(b): Decimal(9, 5) ## toDecimal32OrNull [¶](https://www.tinybird.co/docs/about:blank#todecimal32ornull) Like `toDecimal32` , this function converts an input value to a value of type Nullable(Decimal(9, S)) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal32OrNull(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 9, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal32OrNull('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal32`: `( -1 * 10^(9 - S), 1 * 10^(9 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Nullable(Decimal(9, S))` if successful, otherwise value `NULL` of the same type. Decimal32(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal32OrNull(toString(-1.111), 5) AS a, toTypeName(a), toDecimal32OrNull(toString('Inf'), 5) as b, toTypeName(b) Result: Row 1: ────── a: -1.111 toTypeName(a): Nullable(Decimal(9, 5)) b: ᴺᵁᴸᴸ toTypeName(b): Nullable(Decimal(9, 5)) ## toDecimal32OrDefault [¶](https://www.tinybird.co/docs/about:blank#todecimal32ordefault) Like `toDecimal32` , this function converts an input value to a value of type Decimal(9, S) but returns the default value in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal32OrDefault(expr, S[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 9, specifying how many digits the fractional part of a number can have. UInt8. - `default` (optional): The default value to return if parsing to type `Decimal32(S)` is unsuccessful. Decimal32(S). Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal32OrDefault('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal32`: `( -1 * 10^(9 - S), 1 * 10^(9 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal32OrDefault(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal32OrDefault('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(9, S)` if successful, otherwise returns the default value if passed or `0` if not. Decimal32(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal32OrDefault(toString(0.0001), 5) AS a, toTypeName(a), toDecimal32OrDefault('Inf', 0, CAST('-1', 'Decimal32(0)')) AS b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Decimal(9, 5) b: -1 toTypeName(b): Decimal(9, 0) ## toDecimal64 [¶](https://www.tinybird.co/docs/about:blank#todecimal64) Converts an input value to a value of type `Decimal(18, S)` with scale of `S` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal64(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. - `S` : Scale parameter between 0 and 18, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values or string representations of type Float32/64. Unsupported arguments: - Values or string representations of Float32/64 values `NaN` and `Inf` (case-insensitive). - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal64('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal64`: `( -1 * 10^(18 - S), 1 * 10^(18 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an exception. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal64(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal64('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(18, S)` . Decimal64(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal64(2, 1) AS a, toTypeName(a) AS type_a, toDecimal64(4.2, 2) AS b, toTypeName(b) AS type_b, toDecimal64('4.2', 3) AS c, toTypeName(c) AS type_c Result: Row 1: ────── a: 2 type_a: Decimal(18, 1) b: 4.2 type_b: Decimal(18, 2) c: 4.2 type_c: Decimal(18, 3) ## toDecimal64OrZero [¶](https://www.tinybird.co/docs/about:blank#todecimal64orzero) Like `toDecimal64` , this function converts an input value to a value of type Decimal(18, S) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal64OrZero(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 18, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal64OrZero('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal64`: `( -1 * 10^(18 - S), 1 * 10^(18 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(18, S)` if successful, otherwise `0` with `S` decimal places. Decimal64(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal64OrZero(toString(0.0001), 18) AS a, toTypeName(a), toDecimal64OrZero(toString('Inf'), 18) as b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Decimal(18, 18) b: 0 toTypeName(b): Decimal(18, 18) ## toDecimal64OrNull [¶](https://www.tinybird.co/docs/about:blank#todecimal64ornull) Like `toDecimal64` , this function converts an input value to a value of type Nullable(Decimal(18, S)) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal64OrNull(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 18, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal64OrNull('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal64`: `( -1 * 10^(18 - S), 1 * 10^(18 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Nullable(Decimal(18, S))` if successful, otherwise value `NULL` of the same type. Decimal64(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal64OrNull(toString(0.0001), 18) AS a, toTypeName(a), toDecimal64OrNull(toString('Inf'), 18) as b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Nullable(Decimal(18, 18)) b: ᴺᵁᴸᴸ toTypeName(b): Nullable(Decimal(18, 18)) ## toDecimal64OrDefault [¶](https://www.tinybird.co/docs/about:blank#todecimal64ordefault) Like `toDecimal64` , this function converts an input value to a value of type Decimal(18, S) but returns the default value in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal64OrDefault(expr, S[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 18, specifying how many digits the fractional part of a number can have. UInt8. - `default` (optional): The default value to return if parsing to type `Decimal64(S)` is unsuccessful. Decimal64(S). Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal64OrDefault('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal64`: `( -1 * 10^(18 - S), 1 * 10^(18 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal64OrDefault(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal64OrDefault('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(18, S)` if successful, otherwise returns the default value if passed or `0` if not. Decimal64(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal64OrDefault(toString(0.0001), 18) AS a, toTypeName(a), toDecimal64OrDefault('Inf', 0, CAST('-1', 'Decimal64(0)')) AS b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Decimal(18, 18) b: -1 toTypeName(b): Decimal(18, 0) ## toDecimal128 [¶](https://www.tinybird.co/docs/about:blank#todecimal128) Converts an input value to a value of type `Decimal(38, S)` with scale of `S` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal128(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. - `S` : Scale parameter between 0 and 38, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values or string representations of type Float32/64. Unsupported arguments: - Values or string representations of Float32/64 values `NaN` and `Inf` (case-insensitive). - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal128('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal128`: `( -1 * 10^(38 - S), 1 * 10^(38 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an exception. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal128(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal128('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(38, S)` . Decimal128(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal128(99, 1) AS a, toTypeName(a) AS type_a, toDecimal128(99.67, 2) AS b, toTypeName(b) AS type_b, toDecimal128('99.67', 3) AS c, toTypeName(c) AS type_c Result: Row 1: ────── a: 99 type_a: Decimal(38, 1) b: 99.67 type_b: Decimal(38, 2) c: 99.67 type_c: Decimal(38, 3) ## toDecimal128OrZero [¶](https://www.tinybird.co/docs/about:blank#todecimal128orzero) Like `toDecimal128` , this function converts an input value to a value of type Decimal(38, S) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal128OrZero(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 38, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal128OrZero('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal128`: `( -1 * 10^(38 - S), 1 * 10^(38 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(38, S)` if successful, otherwise `0` with `S` decimal places. Decimal128(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal128OrZero(toString(0.0001), 38) AS a, toTypeName(a), toDecimal128OrZero(toString('Inf'), 38) as b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Decimal(38, 38) b: 0 toTypeName(b): Decimal(38, 38) ## toDecimal128OrNull [¶](https://www.tinybird.co/docs/about:blank#todecimal128ornull) Like `toDecimal128` , this function converts an input value to a value of type Nullable(Decimal(38, S)) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal128OrNull(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 38, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal128OrNull('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal128`: `( -1 * 10^(38 - S), 1 * 10^(38 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Nullable(Decimal(38, S))` if successful, otherwise value `NULL` of the same type. Decimal128(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal128OrNull(toString(1/42), 38) AS a, toTypeName(a), toDecimal128OrNull(toString('Inf'), 38) as b, toTypeName(b) Result: Row 1: ────── a: 0.023809523809523808 toTypeName(a): Nullable(Decimal(38, 38)) b: ᴺᵁᴸᴸ toTypeName(b): Nullable(Decimal(38, 38)) ## toDecimal128OrDefault [¶](https://www.tinybird.co/docs/about:blank#todecimal128ordefault) Like `toDecimal128` , this function converts an input value to a value of type Decimal(38, S) but returns the default value in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal128OrDefault(expr, S[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 38, specifying how many digits the fractional part of a number can have. UInt8. - `default` (optional): The default value to return if parsing to type `Decimal128(S)` is unsuccessful. Decimal128(S). Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal128OrDefault('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal128`: `( -1 * 10^(38 - S), 1 * 10^(38 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal128OrDefault(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal128OrDefault('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(38, S)` if successful, otherwise returns the default value if passed or `0` if not. Decimal128(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal128OrDefault(toString(1/42), 18) AS a, toTypeName(a), toDecimal128OrDefault('Inf', 0, CAST('-1', 'Decimal128(0)')) AS b, toTypeName(b) Result: Row 1: ────── a: 0.023809523809523808 toTypeName(a): Decimal(38, 18) b: -1 toTypeName(b): Decimal(38, 0) ## toDecimal256 [¶](https://www.tinybird.co/docs/about:blank#todecimal256) Converts an input value to a value of type `Decimal(76, S)` with scale of `S` . Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal256(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression returning a number or a string representation of a number. Expression. - `S` : Scale parameter between 0 and 76, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - Values or string representations of type (U)Int8/16/32/64/128/256. - Values or string representations of type Float32/64. Unsupported arguments: - Values or string representations of Float32/64 values `NaN` and `Inf` (case-insensitive). - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal256('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal256`: `( -1 * 10^(76 - S), 1 * 10^(76 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an exception. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal256(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal256('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(76, S)` . Decimal256(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal256(99, 1) AS a, toTypeName(a) AS type_a, toDecimal256(99.67, 2) AS b, toTypeName(b) AS type_b, toDecimal256('99.67', 3) AS c, toTypeName(c) AS type_c Result: Row 1: ────── a: 99 type_a: Decimal(76, 1) b: 99.67 type_b: Decimal(76, 2) c: 99.67 type_c: Decimal(76, 3) ## toDecimal256OrZero [¶](https://www.tinybird.co/docs/about:blank#todecimal256orzero) Like `toDecimal256` , this function converts an input value to a value of type Decimal(76, S) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal256OrZero(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 76, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal256OrZero('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal256`: `( -1 * 10^(76 - S), 1 * 10^(76 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(76, S)` if successful, otherwise `0` with `S` decimal places. Decimal256(S). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimal256OrZero(toString(0.0001), 76) AS a, toTypeName(a), toDecimal256OrZero(toString('Inf'), 76) as b, toTypeName(b) Result: Row 1: ────── a: 0.0001 toTypeName(a): Decimal(76, 76) b: 0 toTypeName(b): Decimal(76, 76) ## toDecimal256OrNull [¶](https://www.tinybird.co/docs/about:blank#todecimal256ornull) Like `toDecimal256` , this function converts an input value to a value of type Nullable(Decimal(76, S)) but returns `0` in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal256OrNull(expr, S) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 76, specifying how many digits the fractional part of a number can have. UInt8. Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal256OrNull('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal256`: `( -1 * 10^(76 - S), 1 * 10^(76 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Nullable(Decimal(76, S))` if successful, otherwise value `NULL` of the same type. Decimal256(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal256OrNull(toString(1/42), 76) AS a, toTypeName(a), toDecimal256OrNull(toString('Inf'), 76) as b, toTypeName(b) Result: Row 1: ────── a: 0.023809523809523808 toTypeName(a): Nullable(Decimal(76, 76)) b: ᴺᵁᴸᴸ toTypeName(b): Nullable(Decimal(76, 76)) ## toDecimal256OrDefault [¶](https://www.tinybird.co/docs/about:blank#todecimal256ordefault) Like `toDecimal256` , this function converts an input value to a value of type Decimal(76, S) but returns the default value in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimal256OrDefault(expr, S[, default]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : A String representation of a number. String. - `S` : Scale parameter between 0 and 76, specifying how many digits the fractional part of a number can have. UInt8. - `default` (optional): The default value to return if parsing to type `Decimal256(S)` is unsuccessful. Decimal256(S). Supported arguments: - String representations of type (U)Int8/16/32/64/128/256. - String representations of type Float32/64. Unsupported arguments: - String representations of Float32/64 values `NaN` and `Inf` . - String representations of binary and hexadecimal values, e.g. `SELECT toDecimal256OrDefault('0xc0fe', 1);` . An overflow can occur if the value of `expr` exceeds the bounds of `Decimal256`: `( -1 * 10^(76 - S), 1 * 10^(76 - S) )`. Excessive digits in a fraction are discarded (not rounded). Excessive digits in the integer part will lead to an error. Conversions drop extra digits and could operate in an unexpected way when working with Float32/Float64 inputs as the operations are performed using floating point instructions. For example: `toDecimal256OrDefault(1.15, 2)` is equal to `1.14` because 1.15 * 100 in floating point is 114.99. You can use a String input so the operations use the underlying integer type: `toDecimal256OrDefault('1.15', 2) = 1.15` ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value of type `Decimal(76, S)` if successful, otherwise returns the default value if passed or `0` if not. Decimal256(S). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT toDecimal256OrDefault(toString(1/42), 76) AS a, toTypeName(a), toDecimal256OrDefault('Inf', 0, CAST('-1', 'Decimal256(0)')) AS b, toTypeName(b) Result: Row 1: ────── a: 0.023809523809523808 toTypeName(a): Decimal(76, 76) b: -1 toTypeName(b): Decimal(76, 0) ## toString [¶](https://www.tinybird.co/docs/about:blank#tostring) Functions for converting between numbers, strings (but not fixed strings), dates, and dates with times. All these functions accept one argument. When converting to or from a string, the value is formatted or parsed using the same rules as for the TabSeparated format (and almost all other text formats). If the string can’t be parsed, an exception is thrown and the request is canceled. When converting dates to numbers or vice versa, the date corresponds to the number of days since the beginning of the Unix epoch. When converting dates with times to numbers or vice versa, the date with time corresponds to the number of seconds since the beginning of the Unix epoch. The date and date-with-time formats for the toDate/toDateTime functions are defined as follows: YYYY-MM-DD YYYY-MM-DD hh:mm:ss As an exception, if converting from UInt32, Int32, UInt64, or Int64 numeric types to Date, and if the number is greater than or equal to 65536, the number is interpreted as a Unix timestamp (and not as the number of days) and is rounded to the date. This allows support for the common occurrence of writing `toDate(unix_timestamp)` , which otherwise would be an error and would require writing the more cumbersome `toDate(toDateTime(unix_timestamp))`. Conversion between a date and a date with time is performed the natural way: by adding a null time or dropping the time. Conversion between numeric types uses the same rules as assignments between different numeric types in C++. Additionally, the toString function of the DateTime argument can take a second String argument containing the name of the time zone. Example: `Asia/Yekaterinburg` In this case, the time is formatted according to the specified time zone. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT now() AS ts, time_zone, toString(ts, time_zone) AS str_tz_datetime FROM system.time_zones WHERE time_zone LIKE 'Europe%' LIMIT 10 Result: ┌──────────────────ts─┬─time_zone─────────┬─str_tz_datetime─────┐ │ 2023-09-08 19:14:59 │ Europe/Amsterdam │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Andorra │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Astrakhan │ 2023-09-08 23:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Athens │ 2023-09-08 22:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Belfast │ 2023-09-08 20:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Belgrade │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Berlin │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Bratislava │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Brussels │ 2023-09-08 21:14:59 │ │ 2023-09-08 19:14:59 │ Europe/Bucharest │ 2023-09-08 22:14:59 │ └─────────────────────┴───────────────────┴─────────────────────┘ Also see the `toUnixTimestamp` function. ## toFixedString [¶](https://www.tinybird.co/docs/about:blank#tofixedstring) Converts a String type argument to a FixedString(N) type (a string of fixed length N). If the string has fewer bytes than N, it's padded with null bytes to the right. If the string has more bytes than N, an exception is thrown. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toFixedString(s, N) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `s` : A String to convert to a fixed string. String. - `N` : Length N. UInt8 ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - An N length fixed string of `s` . FixedString. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFixedString('foo', 8) AS s Result: ┌─s─────────────┐ │ foo\0\0\0\0\0 │ └───────────────┘ ## toStringCutToZero [¶](https://www.tinybird.co/docs/about:blank#tostringcuttozero) Accepts a String or FixedString argument. Returns the String with the content truncated at the first zero byte found. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toStringCutToZero(s) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toFixedString('foo', 8) AS s, toStringCutToZero(s) AS s_cut Result: ┌─s─────────────┬─s_cut─┐ │ foo\0\0\0\0\0 │ foo │ └───────────────┴───────┘ Query: SELECT toFixedString('foo\0bar', 8) AS s, toStringCutToZero(s) AS s_cut Result: ┌─s──────────┬─s_cut─┐ │ foo\0bar\0 │ foo │ └────────────┴───────┘ ## toDecimalString [¶](https://www.tinybird.co/docs/about:blank#todecimalstring) Converts a numeric value to String with the number of fractional digits in the output specified by the user. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toDecimalString(number, scale) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `number` : Value to be represented as String, Int, UInt, Float, Decimal, - `scale` : Number of fractional digits, UInt8. - Maximum scale for Decimal and Int, UInt types is 77 (it's the maximum possible number of significant digits for Decimal), - Maximum scale for Float is 60. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Input value represented as String with given number of fractional digits (scale). The number is rounded up or down according to common arithmetic in case requested scale is smaller than original number's scale. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toDecimalString(CAST('64.32', 'Float64'), 5) Result: ┌toDecimalString(CAST('64.32', 'Float64'), 5)─┐ │ 64.32000 │ └─────────────────────────────────────────────┘ ## reinterpretAsUInt8 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint8) Performs byte reinterpretation by treating the input value as a value of type UInt8. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt8(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt8. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt8. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8(257) AS x, toTypeName(x), reinterpretAsUInt8(x) AS res, toTypeName(res) Result: ┌─x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 1 │ Int8 │ 1 │ UInt8 │ └───┴───────────────┴─────┴─────────────────┘ ## reinterpretAsUInt16 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint16) Performs byte reinterpretation by treating the input value as a value of type UInt16. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt16(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt16. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt16. UInt16. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8(257) AS x, toTypeName(x), reinterpretAsUInt16(x) AS res, toTypeName(res) Result: ┌─x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 1 │ UInt8 │ 1 │ UInt16 │ └───┴───────────────┴─────┴─────────────────┘ ## reinterpretAsUInt32 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint32) Performs byte reinterpretation by treating the input value as a value of type UInt32. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt32(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt32. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt32. UInt32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt16(257) AS x, toTypeName(x), reinterpretAsUInt32(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ UInt16 │ 257 │ UInt32 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsUInt64 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint64) Performs byte reinterpretation by treating the input value as a value of type UInt64. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt64(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt64. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt64. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt32(257) AS x, toTypeName(x), reinterpretAsUInt64(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ UInt32 │ 257 │ UInt64 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsUInt128 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint128) Performs byte reinterpretation by treating the input value as a value of type UInt128. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt128(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt128. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt128. UInt128. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt64(257) AS x, toTypeName(x), reinterpretAsUInt128(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ UInt64 │ 257 │ UInt128 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsUInt256 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuint256) Performs byte reinterpretation by treating the input value as a value of type UInt256. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUInt256(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as UInt256. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as UInt256. UInt256. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt128(257) AS x, toTypeName(x), reinterpretAsUInt256(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ UInt128 │ 257 │ UInt256 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt8 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint8) Performs byte reinterpretation by treating the input value as a value of type Int8. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt8(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int8. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int8. Int8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toUInt8(257) AS x, toTypeName(x), reinterpretAsInt8(x) AS res, toTypeName(res) Result: ┌─x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 1 │ UInt8 │ 1 │ Int8 │ └───┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt16 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint16) Performs byte reinterpretation by treating the input value as a value of type Int16. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt16(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int16. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int16. Int16. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt8(257) AS x, toTypeName(x), reinterpretAsInt16(x) AS res, toTypeName(res) Result: ┌─x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 1 │ Int8 │ 1 │ Int16 │ └───┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt32 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint32) Performs byte reinterpretation by treating the input value as a value of type Int32. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt32(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int32. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int32. Int32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt16(257) AS x, toTypeName(x), reinterpretAsInt32(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ Int16 │ 257 │ Int32 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt64 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint64) Performs byte reinterpretation by treating the input value as a value of type Int64. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt64(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int64. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int64. Int64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt32(257) AS x, toTypeName(x), reinterpretAsInt64(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ Int32 │ 257 │ Int64 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt128 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint128) Performs byte reinterpretation by treating the input value as a value of type Int128. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt128(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int128. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int128. Int128. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt64(257) AS x, toTypeName(x), reinterpretAsInt128(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ Int64 │ 257 │ Int128 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsInt256 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasint256) Performs byte reinterpretation by treating the input value as a value of type Int256. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsInt256(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to byte reinterpret as Int256. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Int256. Int256. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toInt128(257) AS x, toTypeName(x), reinterpretAsInt256(x) AS res, toTypeName(res) Result: ┌───x─┬─toTypeName(x)─┬─res─┬─toTypeName(res)─┐ │ 257 │ Int128 │ 257 │ Int256 │ └─────┴───────────────┴─────┴─────────────────┘ ## reinterpretAsFloat32 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasfloat32) Performs byte reinterpretation by treating the input value as a value of type Float32. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsFloat32(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to reinterpret as Float32. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Float32. Float32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsUInt32(toFloat32(0.2)) as x, reinterpretAsFloat32(x) Result: ┌──────────x─┬─reinterpretAsFloat32(x)─┐ │ 1045220557 │ 0.2 │ └────────────┴─────────────────────────┘ ## reinterpretAsFloat64 [¶](https://www.tinybird.co/docs/about:blank#reinterpretasfloat64) Performs byte reinterpretation by treating the input value as a value of type Float64. Unlike `CAST` , the function doesn't attempt to preserve the original value - if the target type isn't able to represent the input type, the output is meaningless. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsFloat64(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to reinterpret as Float64. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Reinterpreted value `x` as Float64. Float64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsUInt64(toFloat64(0.2)) as x, reinterpretAsFloat64(x) Result: ┌───────────────────x─┬─reinterpretAsFloat64(x)─┐ │ 4596373779694328218 │ 0.2 │ └─────────────────────┴─────────────────────────┘ ## reinterpretAsDate [¶](https://www.tinybird.co/docs/about:blank#reinterpretasdate) Accepts a string, fixed string or numeric value and interprets the bytes as a number in host order (little endian). It returns a date from the interpreted number as the number of days since the beginning of the Unix Epoch. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsDate(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : number of days since the beginning of the Unix Epoch. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Date. Date. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) If the provided string isn’t long enough, the function works as if the string is padded with the necessary number of null bytes. If the string is longer than needed, the extra bytes are ignored. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsDate(65), reinterpretAsDate('A') Result: ┌─reinterpretAsDate(65)─┬─reinterpretAsDate('A')─┐ │ 1970-03-07 │ 1970-03-07 │ └───────────────────────┴────────────────────────┘ ## reinterpretAsDateTime [¶](https://www.tinybird.co/docs/about:blank#reinterpretasdatetime) These functions accept a string and interpret the bytes placed at the beginning of the string as a number in host order (little endian). Returns a date with time interpreted as the number of seconds since the beginning of the Unix Epoch. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsDateTime(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : number of seconds since the beginning of the Unix Epoch. (U)Int*, Float, Date, DateTime, UUID, String or FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Date and Time. DateTime. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) If the provided string isn’t long enough, the function works as if the string is padded with the necessary number of null bytes. If the string is longer than needed, the extra bytes are ignored. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsDateTime(65), reinterpretAsDateTime('A') Result: ┌─reinterpretAsDateTime(65)─┬─reinterpretAsDateTime('A')─┐ │ 1970-01-01 01:01:05 │ 1970-01-01 01:01:05 │ └───────────────────────────┴────────────────────────────┘ ## reinterpretAsString [¶](https://www.tinybird.co/docs/about:blank#reinterpretasstring) This function accepts a number, date or date with time and returns a string containing bytes representing the corresponding value in host order (little endian). Null bytes are dropped from the end. For example, a UInt32 type value of 255 is a string that is one byte long. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsString(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to reinterpret to string. (U)Int*, Float, Date, DateTime. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - String containing bytes representing `x` . String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsString(toDateTime('1970-01-01 01:01:05')), reinterpretAsString(toDate('1970-03-07')) Result: ┌─reinterpretAsString(toDateTime('1970-01-01 01:01:05'))─┬─reinterpretAsString(toDate('1970-03-07'))─┐ │ A │ A │ └────────────────────────────────────────────────────────┴───────────────────────────────────────────┘ ## reinterpretAsFixedString [¶](https://www.tinybird.co/docs/about:blank#reinterpretasfixedstring) This function accepts a number, date or date with time and returns a FixedString containing bytes representing the corresponding value in host order (little endian). Null bytes are dropped from the end. For example, a UInt32 type value of 255 is a FixedString that is one byte long. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsFixedString(x) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `x` : value to reinterpret to string. (U)Int*, Float, Date, DateTime. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Fixed string containing bytes representing `x` . FixedString. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT reinterpretAsFixedString(toDateTime('1970-01-01 01:01:05')), reinterpretAsFixedString(toDate('1970-03-07')) Result: ┌─reinterpretAsFixedString(toDateTime('1970-01-01 01:01:05'))─┬─reinterpretAsFixedString(toDate('1970-03-07'))─┐ │ A │ A │ └─────────────────────────────────────────────────────────────┴────────────────────────────────────────────────┘ ## reinterpretAsUUID [¶](https://www.tinybird.co/docs/about:blank#reinterpretasuuid) In addition to the UUID functions listed here, there is dedicated UUID function documentation. Accepts a 16 byte string and returns a UUID containing bytes representing the corresponding value in network byte order (big-endian). If the string isn't long enough, the function works as if the string is padded with the necessary number of null bytes to the end. If the string is longer than 16 bytes, the extra bytes at the end are ignored. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpretAsUUID(fixed_string) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `fixed_string` : Big-endian byte string. FixedString. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The UUID type value. UUID. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) String to UUID. Query: SELECT reinterpretAsUUID(reverse(unhex('000102030405060708090a0b0c0d0e0f'))) Result: ┌─reinterpretAsUUID(reverse(unhex('000102030405060708090a0b0c0d0e0f')))─┐ │ 08090a0b-0c0d-0e0f-0001-020304050607 │ └───────────────────────────────────────────────────────────────────────┘ Going back and forth from String to UUID. Query: WITH generateUUIDv4() AS uuid, identity(lower(hex(reverse(reinterpretAsString(uuid))))) AS str, reinterpretAsUUID(reverse(unhex(str))) AS uuid2 SELECT uuid = uuid2 Result: ┌─equals(uuid, uuid2)─┐ │ 1 │ └─────────────────────┘ ## reinterpret [¶](https://www.tinybird.co/docs/about:blank#reinterpret) Uses the same source in-memory bytes sequence for `x` value and reinterprets it to destination type. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) reinterpret(x, type) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Any type. - `type` : Destination type. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Destination type value. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT reinterpret(toInt8(-1), 'UInt8') as int_to_uint, reinterpret(toInt8(1), 'Float32') as int_to_float, reinterpret('1', 'UInt32') as string_to_int Result: ┌─int_to_uint─┬─int_to_float─┬─string_to_int─┐ │ 255 │ 1e-45 │ 49 │ └─────────────┴──────────────┴───────────────┘ ## CAST [¶](https://www.tinybird.co/docs/about:blank#cast) Converts an input value to the specified data type. Unlike the reinterpret function, `CAST` tries to present the same value using the new data type. If the conversion can't be done then an exception is raised. Several syntax variants are supported. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) CAST(x, T) CAST(x AS t) x::t ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A value to convert. May be of any type. - `T` : The name of the target data type. String. - `t` : The target data type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Converted value. If the input value doesn't fit the bounds of the target type, the result overflows. For example, `CAST(-1, 'UInt8')` returns `255`. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT CAST(toInt8(-1), 'UInt8') AS cast_int_to_uint, CAST(1.5 AS Decimal(3,2)) AS cast_float_to_decimal, '1'::Int32 AS cast_string_to_int Result: ┌─cast_int_to_uint─┬─cast_float_to_decimal─┬─cast_string_to_int─┐ │ 255 │ 1.50 │ 1 │ └──────────────────┴───────────────────────┴────────────────────┘ Query: SELECT '2016-06-15 23:00:00' AS timestamp, CAST(timestamp AS DateTime) AS datetime, CAST(timestamp AS Date) AS date, CAST(timestamp, 'String') AS string, CAST(timestamp, 'FixedString(22)') AS fixed_string Result: ┌─timestamp───────────┬────────────datetime─┬───────date─┬─string──────────────┬─fixed_string──────────────┐ │ 2016-06-15 23:00:00 │ 2016-06-15 23:00:00 │ 2016-06-15 │ 2016-06-15 23:00:00 │ 2016-06-15 23:00:00\0\0\0 │ └─────────────────────┴─────────────────────┴────────────┴─────────────────────┴───────────────────────────┘ Conversion to FixedString (N) only works for arguments of type String or FixedString. Type conversion to Nullable and back is supported. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toTypeName(x) FROM t_null Result: ┌─toTypeName(x)─┐ │ Int8 │ │ Int8 │ └───────────────┘ Query: SELECT toTypeName(CAST(x, 'Nullable(UInt16)')) FROM t_null Result: ┌─toTypeName(CAST(x, 'Nullable(UInt16)'))─┐ │ Nullable(UInt16) │ │ Nullable(UInt16) │ └─────────────────────────────────────────┘ ## accurateCast(x, T) [¶](https://www.tinybird.co/docs/about:blank#accuratecastx-t) Converts `x` to the `T` data type. The difference from cast is that `accurateCast` doesn't allow overflow of numeric types during cast if type value `x` doesn't fit the bounds of type `T` . For example, `accurateCast(-1, 'UInt8')` throws an exception. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT cast(-1, 'UInt8') as uint8 Result: ┌─uint8─┐ │ 255 │ └───────┘ Query: SELECT accurateCast(-1, 'UInt8') as uint8 Result: Code: 70. DB::Exception: Received from localhost:9000. DB::Exception: Value in column Int8 can't be safely converted into type UInt8: While processing accurateCast(-1, 'UInt8') AS uint8. ## accurateCastOrNull(x, T) [¶](https://www.tinybird.co/docs/about:blank#accuratecastornullx-t) Converts input value `x` to the specified data type `T` . Always returns Nullable type and returns NULL if the cast value isn't representable in the target type. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) accurateCastOrNull(x, T) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Input value. - `T` : The name of the returned data type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The value, converted to the specified data type `T` . ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toTypeName(accurateCastOrNull(5, 'UInt8')) Result: ┌─toTypeName(accurateCastOrNull(5, 'UInt8'))─┐ │ Nullable(UInt8) │ └────────────────────────────────────────────┘ Query: SELECT accurateCastOrNull(-1, 'UInt8') as uint8, accurateCastOrNull(128, 'Int8') as int8, accurateCastOrNull('Test', 'FixedString(2)') as fixed_string Result: ┌─uint8─┬─int8─┬─fixed_string─┐ │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │ └───────┴──────┴──────────────┘ ## accurateCastOrDefault(x, T[, default_value]) [¶](https://www.tinybird.co/docs/about:blank#accuratecastordefaultx-t-default-value) Converts input value `x` to the specified data type `T` . Returns default type value or `default_value` if specified if the cast value isn't representable in the target type. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) accurateCastOrDefault(x, T) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Input value. - `T` : The name of the returned data type. - `default_value` : Default value of returned data type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The value converted to the specified data type `T` . ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toTypeName(accurateCastOrDefault(5, 'UInt8')) Result: ┌─toTypeName(accurateCastOrDefault(5, 'UInt8'))─┐ │ UInt8 │ └───────────────────────────────────────────────┘ Query: SELECT accurateCastOrDefault(-1, 'UInt8') as uint8, accurateCastOrDefault(-1, 'UInt8', 5) as uint8_default, accurateCastOrDefault(128, 'Int8') as int8, accurateCastOrDefault(128, 'Int8', 5) as int8_default, accurateCastOrDefault('Test', 'FixedString(2)') as fixed_string, accurateCastOrDefault('Test', 'FixedString(2)', 'Te') as fixed_string_default Result: ┌─uint8─┬─uint8_default─┬─int8─┬─int8_default─┬─fixed_string─┬─fixed_string_default─┐ │ 0 │ 5 │ 0 │ 5 │ │ Te │ └───────┴───────────────┴──────┴──────────────┴──────────────┴──────────────────────┘ ## toIntervalYear [¶](https://www.tinybird.co/docs/about:blank#tointervalyear) Returns an interval of `n` years of data type IntervalYear. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalYear(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of years. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` years. IntervalYear. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalYear(1) AS interval_to_year SELECT date + interval_to_year AS result Result: ┌─────result─┐ │ 2025-06-15 │ └────────────┘ ## toIntervalQuarter [¶](https://www.tinybird.co/docs/about:blank#tointervalquarter) Returns an interval of `n` quarters of data type IntervalQuarter. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalQuarter(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of quarters. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` quarters. IntervalQuarter. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalQuarter(1) AS interval_to_quarter SELECT date + interval_to_quarter AS result Result: ┌─────result─┐ │ 2024-09-15 │ └────────────┘ ## toIntervalMonth [¶](https://www.tinybird.co/docs/about:blank#tointervalmonth) Returns an interval of `n` months of data type IntervalMonth. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalMonth(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of months. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` months. IntervalMonth. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalMonth(1) AS interval_to_month SELECT date + interval_to_month AS result Result: ┌─────result─┐ │ 2024-07-15 │ └────────────┘ ## toIntervalWeek [¶](https://www.tinybird.co/docs/about:blank#tointervalweek) Returns an interval of `n` weeks of data type IntervalWeek. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalWeek(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of weeks. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` weeks. IntervalWeek. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalWeek(1) AS interval_to_week SELECT date + interval_to_week AS result Result: ┌─────result─┐ │ 2024-06-22 │ └────────────┘ ## toIntervalDay [¶](https://www.tinybird.co/docs/about:blank#tointervalday) Returns an interval of `n` days of data type IntervalDay. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalDay(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of days. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` days. IntervalDay. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalDay(5) AS interval_to_days SELECT date + interval_to_days AS result Result: ┌─────result─┐ │ 2024-06-20 │ └────────────┘ ## toIntervalHour [¶](https://www.tinybird.co/docs/about:blank#tointervalhour) Returns an interval of `n` hours of data type IntervalHour. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalHour(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of hours. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` hours. IntervalHour. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalHour(12) AS interval_to_hours SELECT date + interval_to_hours AS result Result: ┌──────────────result─┐ │ 2024-06-15 12:00:00 │ └─────────────────────┘ ## toIntervalMinute [¶](https://www.tinybird.co/docs/about:blank#tointervalminute) Returns an interval of `n` minutes of data type IntervalMinute. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalMinute(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of minutes. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` minutes. IntervalMinute. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalMinute(12) AS interval_to_minutes SELECT date + interval_to_minutes AS result Result: ┌──────────────result─┐ │ 2024-06-15 00:12:00 │ └─────────────────────┘ ## toIntervalSecond [¶](https://www.tinybird.co/docs/about:blank#tointervalsecond) Returns an interval of `n` seconds of data type IntervalSecond. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalSecond(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of seconds. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` seconds. IntervalSecond. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDate('2024-06-15') AS date, toIntervalSecond(30) AS interval_to_seconds SELECT date + interval_to_seconds AS result Result: ┌──────────────result─┐ │ 2024-06-15 00:00:30 │ └─────────────────────┘ ## toIntervalMillisecond [¶](https://www.tinybird.co/docs/about:blank#tointervalmillisecond) Returns an interval of `n` milliseconds of data type IntervalMillisecond. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalMillisecond(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of milliseconds. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` milliseconds. IntervalMilliseconds. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime('2024-06-15') AS date, toIntervalMillisecond(30) AS interval_to_milliseconds SELECT date + interval_to_milliseconds AS result Result: ┌──────────────────result─┐ │ 2024-06-15 00:00:00.030 │ └─────────────────────────┘ ## toIntervalMicrosecond [¶](https://www.tinybird.co/docs/about:blank#tointervalmicrosecond) Returns an interval of `n` microseconds of data type IntervalMicrosecond. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalMicrosecond(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of microseconds. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` microseconds. IntervalMicrosecond. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime('2024-06-15') AS date, toIntervalMicrosecond(30) AS interval_to_microseconds SELECT date + interval_to_microseconds AS result Result: ┌─────────────────────result─┐ │ 2024-06-15 00:00:00.000030 │ └────────────────────────────┘ ## toIntervalNanosecond [¶](https://www.tinybird.co/docs/about:blank#tointervalnanosecond) Returns an interval of `n` nanoseconds of data type IntervalNanosecond. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toIntervalNanosecond(n) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : Number of nanoseconds. Integer numbers or string representations thereof, and float numbers. (U)Int*/Float*/String. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Interval of `n` nanoseconds. IntervalNanosecond. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime('2024-06-15') AS date, toIntervalNanosecond(30) AS interval_to_nanoseconds SELECT date + interval_to_nanoseconds AS result Result: ┌────────────────────────result─┐ │ 2024-06-15 00:00:00.000000030 │ └───────────────────────────────┘ ## parseDateTime [¶](https://www.tinybird.co/docs/about:blank#parsedatetime) Converts a String to DateTime according to a MySQL format string. This function is the opposite operation of function formatDateTime. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTime(str[, format[, timezone]]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `str` : The String to be parsed - `format` : The format string. Optional. `%Y-%m-%d %H:%i:%s` if not specified. - `timezone` : Timezone. Optional. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) Return a DateTime value parsed from the input string according to a MySQL-style format string. ### Format specifiers [¶](https://www.tinybird.co/docs/about:blank#format-specifiers) All format specifiers listed in formatDateTime except: - %Q: Quarter (1-4) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT parseDateTime('2021-01-04+23:00:00', '%Y-%m-%d+%H:%i:%s') ┌─parseDateTime('2021-01-04+23:00:00', '%Y-%m-%d+%H:%i:%s')─┐ │ 2021-01-04 23:00:00 │ └───────────────────────────────────────────────────────────┘ Alias: `TO_TIMESTAMP`. ## parseDateTimeOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetimeorzero) Same as for parseDateTime except that it returns zero date when it encounters a date format that can't be processed. ## parseDateTimeOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetimeornull) Same as for parseDateTime except that it returns `NULL` when it encounters a date format that can't be processed. Alias: `str_to_date`. ## parseDateTimeInJodaSyntax [¶](https://www.tinybird.co/docs/about:blank#parsedatetimeinjodasyntax) Similar to parseDateTime, except that the format string is in Joda instead of MySQL syntax. This function is the opposite operation of function formatDateTimeInJodaSyntax. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTimeInJodaSyntax(str[, format[, timezone]]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `str` : The String to be parsed - `format` : The format string. Optional. `yyyy-MM-dd HH:mm:ss` if not specified. - `timezone` : Timezone. Optional. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) Return a DateTime value parsed from the input string according to a Joda-style format string. ### Format specifiers [¶](https://www.tinybird.co/docs/about:blank#format-specifiers) All format specifiers listed in formatDateTimeInJoda are supported, except: - S: fraction of second - z: time zone - Z: time zone offset/id ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT parseDateTimeInJodaSyntax('2023-02-24 14:53:31', 'yyyy-MM-dd HH:mm:ss', 'Europe/Minsk') ┌─parseDateTimeInJodaSyntax('2023-02-24 14:53:31', 'yyyy-MM-dd HH:mm:ss', 'Europe/Minsk')─┐ │ 2023-02-24 14:53:31 │ └─────────────────────────────────────────────────────────────────────────────────────────┘ ## parseDateTimeInJodaSyntaxOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetimeinjodasyntaxorzero) Same as for parseDateTimeInJodaSyntax except that it returns zero date when it encounters a date format that can't be processed. ## parseDateTimeInJodaSyntaxOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetimeinjodasyntaxornull) Same as for parseDateTimeInJodaSyntax except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTime64 [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64) Converts a String to DateTime64 according to a MySQL format string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTime64(str[, format[, timezone]]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `str` : The String to be parsed. - `format` : The format string. Optional. `%Y-%m-%d %H:%i:%s.%f` if not specified. - `timezone` : Timezone. Optional. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) Return a DateTime64 value parsed from the input string according to a MySQL-style format string. The precision of the returned value is 6. ## parseDateTime64OrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64orzero) Same as for parseDateTime64 except that it returns zero date when it encounters a date format that can't be processed. ## parseDateTime64OrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64ornull) Same as for parseDateTime64 except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTime64InJodaSyntax [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64injodasyntax) Converts a String to DateTime64 according to a Joda format string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTime64InJodaSyntax(str[, format[, timezone]]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `str` : The String to be parsed. - `format` : The format string. Optional. `yyyy-MM-dd HH:mm:ss` if not specified. - `timezone` : Timezone. Optional. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) Return a DateTime64 value parsed from the input string according to a Joda-style format string. The precision of the returned value equal to the number of `S` placeholders in the format string (but at most 6). ## parseDateTime64InJodaSyntaxOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64injodasyntaxorzero) Same as for parseDateTime64InJodaSyntax except that it returns zero date when it encounters a date format that can't be processed. ## parseDateTime64InJodaSyntaxOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64injodasyntaxornull) Same as for parseDateTime64InJodaSyntax except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTimeBestEffort [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffort) ## parseDateTime32BestEffort [¶](https://www.tinybird.co/docs/about:blank#parsedatetime32besteffort) Converts a date and time in the String representation to DateTime data type. The function parses ISO 8601, RFC 1123 - 5.2.14 RFC-822 Date and Time Specification, Tinybird’s and some other date and time formats. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTimeBestEffort(time_string [, time_zone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_string` : String containing a date and time to convert. String. - `time_zone` : Time zone. The function parses `time_string` according to the time zone. String. ### Supported non-standard formats [¶](https://www.tinybird.co/docs/about:blank#supported-non-standard-formats) - A string containing 9..10 digit unix timestamp. - A string with a date and a time component: `YYYYMMDDhhmmss` , `DD/MM/YYYY hh:mm:ss` , `DD-MM-YY hh:mm` , `YYYY-MM-DD hh:mm:ss` , etc. - A string with a date, but no time component: `YYYY` , `YYYYMM` , `YYYY*MM` , `DD/MM/YYYY` , `DD-MM-YY` etc. - A string with a day and time: `DD` , `DD hh` , `DD hh:mm` . In this case `MM` is substituted by `01` . - A string that includes the date and time along with time zone offset information: `YYYY-MM-DD hh:mm:ss ±h:mm` , etc. For example, `2020-12-12 17:36:00 -5:00` . - A syslog timestamp: `Mmm dd hh:mm:ss` . For example, `Jun 9 14:20:32` . For all of the formats with separator the function parses months names expressed by their full name or by the first three letters of a month name. Examples: `24/DEC/18`, `24-Dec-18`, `01-September-2018`. If the year isn't specified, it's considered to be equal to the current year. If the resulting DateTime happen to be in the future (even by a second after the current moment), then the current year is substituted by the previous year. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `time_string` converted to the DateTime data type. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT parseDateTimeBestEffort('23/10/2020 12:12:57') AS parseDateTimeBestEffort Result: ┌─parseDateTimeBestEffort─┐ │ 2020-10-23 12:12:57 │ └─────────────────────────┘ Query: SELECT parseDateTimeBestEffort('Sat, 18 Aug 2018 07:22:16 GMT', 'Asia/Istanbul') AS parseDateTimeBestEffort Result: ┌─parseDateTimeBestEffort─┐ │ 2018-08-18 10:22:16 │ └─────────────────────────┘ Query: SELECT parseDateTimeBestEffort('1284101485') AS parseDateTimeBestEffort Result: ┌─parseDateTimeBestEffort─┐ │ 2015-07-07 12:04:41 │ └─────────────────────────┘ Query: SELECT parseDateTimeBestEffort('2018-10-23 10:12:12') AS parseDateTimeBestEffort Result: ┌─parseDateTimeBestEffort─┐ │ 2018-10-23 10:12:12 │ └─────────────────────────┘ Query: SELECT toYear(now()) as year, parseDateTimeBestEffort('10 20:19') Result: ┌─year─┬─parseDateTimeBestEffort('10 20:19')─┐ │ 2023 │ 2023-01-10 20:19:00 │ └──────┴─────────────────────────────────────┘ Query: WITH now() AS ts_now, formatDateTime(ts_around, '%b %e %T') AS syslog_arg SELECT ts_now, syslog_arg, parseDateTimeBestEffort(syslog_arg) FROM (SELECT arrayJoin([ts_now - 30, ts_now + 30]) AS ts_around) Result: ┌──────────────ts_now─┬─syslog_arg──────┬─parseDateTimeBestEffort(syslog_arg)─┐ │ 2023-06-30 23:59:30 │ Jun 30 23:59:00 │ 2023-06-30 23:59:00 │ │ 2023-06-30 23:59:30 │ Jul 1 00:00:00 │ 2022-07-01 00:00:00 │ └─────────────────────┴─────────────────┴─────────────────────────────────────┘ ## parseDateTimeBestEffortUS [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffortus) This function behaves like parseDateTimeBestEffort for ISO date formats, e.g. `YYYY-MM-DD hh:mm:ss` , and other date formats where the month and date components can be unambiguously extracted, e.g. `YYYYMMDDhhmmss`, `YYYY-MM`, `DD hh` , or `YYYY-MM-DD hh:mm:ss ±h:mm` . If the month and the date components can't be unambiguously extracted, e.g. `MM/DD/YYYY`, `MM-DD-YYYY` , or `MM-DD-YY` , it prefers the US date format instead of `DD/MM/YYYY`, `DD-MM-YYYY` , or `DD-MM-YY` . As an exception from the latter, if the month is bigger than 12 and smaller or equal than 31, this function falls back to the behavior of parseDateTimeBestEffort, e.g. `15/08/2020` is parsed as `2020-08-15`. ## parseDateTimeBestEffortOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffortornull) ## parseDateTime32BestEffortOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetime32besteffortornull) Same as for parseDateTimeBestEffort except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTimeBestEffortOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffortorzero) ## parseDateTime32BestEffortOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetime32besteffortorzero) Same as for parseDateTimeBestEffort except that it returns zero date or zero date time when it encounters a date format that can't be processed. ## parseDateTimeBestEffortUSOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffortusornull) Same as parseDateTimeBestEffortUS function except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTimeBestEffortUSOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetimebesteffortusorzero) Same as parseDateTimeBestEffortUS function except that it returns zero date ( `1970-01-01` ) or zero date with time ( `1970-01-01 00:00:00` ) when it encounters a date format that can't be processed. ## parseDateTime64BestEffort [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffort) Same as parseDateTimeBestEffort function but also parse milliseconds and microseconds and returns DateTime data type. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) parseDateTime64BestEffort(time_string [, precision [, time_zone]]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_string` : String containing a date or date with time to convert. String. - `precision` : Required precision. `3` : for milliseconds, `6` : for microseconds. Default: `3` . Optional. UInt8. - `time_zone` : Timezone. The function parses `time_string` according to the timezone. Optional. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `time_string` converted to the DateTime data type. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT parseDateTime64BestEffort('2021-01-01') AS a, toTypeName(a) AS t UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 01:01:00.12346') AS a, toTypeName(a) AS t UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 01:01:00.12346',6) AS a, toTypeName(a) AS t UNION ALL SELECT parseDateTime64BestEffort('2021-01-01 01:01:00.12346',3,'Asia/Istanbul') AS a, toTypeName(a) AS t FORMAT PrettyCompactMonoBlock Result: ┌──────────────────────────a─┬─t──────────────────────────────┐ │ 2021-01-01 01:01:00.123000 │ DateTime64(3) │ │ 2021-01-01 00:00:00.000000 │ DateTime64(3) │ │ 2021-01-01 01:01:00.123460 │ DateTime64(6) │ │ 2020-12-31 22:01:00.123000 │ DateTime64(3, 'Asia/Istanbul') │ └────────────────────────────┴────────────────────────────────┘ ## parseDateTime64BestEffortUS [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffortus) Same as for parseDateTime64BestEffort, except that this function prefers US date format ( `MM/DD/YYYY` etc.) in case of ambiguity. ## parseDateTime64BestEffortOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffortornull) Same as for parseDateTime64BestEffort except that it returns `NULL` when it encounters a date format that can't be processed. ## parseDateTime64BestEffortOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffortorzero) Same as for parseDateTime64BestEffort except that it returns zero date or zero date time when it encounters a date format that can't be processed. ## parseDateTime64BestEffortUSOrNull [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffortusornull) Same as for parseDateTime64BestEffort, except that this function prefers US date format ( `MM/DD/YYYY` etc.) in case of ambiguity and returns `NULL` when it encounters a date format that can't be processed. ## parseDateTime64BestEffortUSOrZero [¶](https://www.tinybird.co/docs/about:blank#parsedatetime64besteffortusorzero) Same as for parseDateTime64BestEffort, except that this function prefers US date format ( `MM/DD/YYYY` etc.) in case of ambiguity and returns zero date or zero date time when it encounters a date format that can't be processed. ## toLowCardinality [¶](https://www.tinybird.co/docs/about:blank#tolowcardinality) Converts input parameter to the LowCardinality version of same data type. To convert data from the `LowCardinality` data type use the CAST function. For example, `CAST(x as String)`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toLowCardinality(expr) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `expr` : Expression resulting in one of the supported data types. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - Result of `expr` . LowCardinality of the type of `expr` . ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT toLowCardinality('1') Result: ┌─toLowCardinality('1')─┐ │ 1 │ └───────────────────────┘ ## toUnixTimestamp64Milli [¶](https://www.tinybird.co/docs/about:blank#tounixtimestamp64milli) Converts a `DateTime64` to a `Int64` value with fixed millisecond precision. The input value is scaled up or down appropriately depending on its precision. The output value is a timestamp in UTC, not in the timezone of `DateTime64`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUnixTimestamp64Milli(value) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : DateTime64 value with any precision. DateTime64. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to the `Int64` data type. Int64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime64('2009-02-13 23:31:31.011', 3, 'UTC') AS dt64 SELECT toUnixTimestamp64Milli(dt64) Result: ┌─toUnixTimestamp64Milli(dt64)─┐ │ 1234567891011 │ └──────────────────────────────┘ ## toUnixTimestamp64Micro [¶](https://www.tinybird.co/docs/about:blank#tounixtimestamp64micro) Converts a `DateTime64` to a `Int64` value with fixed microsecond precision. The input value is scaled up or down appropriately depending on its precision. The output value is a timestamp in UTC, not in the timezone of `DateTime64`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUnixTimestamp64Micro(value) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : DateTime64 value with any precision. DateTime64. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to the `Int64` data type. Int64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime64('1970-01-15 06:56:07.891011', 6, 'UTC') AS dt64 SELECT toUnixTimestamp64Micro(dt64) Result: ┌─toUnixTimestamp64Micro(dt64)─┐ │ 1234567891011 │ └──────────────────────────────┘ ## toUnixTimestamp64Nano [¶](https://www.tinybird.co/docs/about:blank#tounixtimestamp64nano) Converts a `DateTime64` to a `Int64` value with fixed nanosecond precision. The input value is scaled up or down appropriately depending on its precision. The output value is a timestamp in UTC, not in the timezone of `DateTime64`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toUnixTimestamp64Nano(value) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : DateTime64 value with any precision. DateTime64. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to the `Int64` data type. Int64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH toDateTime64('1970-01-01 00:20:34.567891011', 9, 'UTC') AS dt64 SELECT toUnixTimestamp64Nano(dt64) Result: ┌─toUnixTimestamp64Nano(dt64)─┐ │ 1234567891011 │ └─────────────────────────────┘ ## fromUnixTimestamp64Milli [¶](https://www.tinybird.co/docs/about:blank#fromunixtimestamp64milli) Converts an `Int64` to a `DateTime64` value with fixed millisecond precision and optional timezone. The input value is scaled up or down appropriately depending on its precision. Please note that input value is treated as a UTC timestamp, not timestamp at the given (or implicit) timezone. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) fromUnixTimestamp64Milli(value[, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : value with any precision. Int64. - `timezone` : (optional) timezone name of the result. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to DateTime64 with precision `3` . DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH CAST(1234567891011, 'Int64') AS i64 SELECT fromUnixTimestamp64Milli(i64, 'UTC') AS x, toTypeName(x) Result: ┌───────────────────────x─┬─toTypeName(x)────────┐ │ 2009-02-13 23:31:31.011 │ DateTime64(3, 'UTC') │ └─────────────────────────┴──────────────────────┘ ## fromUnixTimestamp64Micro [¶](https://www.tinybird.co/docs/about:blank#fromunixtimestamp64micro) Converts an `Int64` to a `DateTime64` value with fixed microsecond precision and optional timezone. The input value is scaled up or down appropriately depending on its precision. Please note that input value is treated as a UTC timestamp, not timestamp at the given (or implicit) timezone. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) fromUnixTimestamp64Micro(value[, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : value with any precision. Int64. - `timezone` : (optional) timezone name of the result. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to DateTime64 with precision `6` . DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH CAST(1234567891011, 'Int64') AS i64 SELECT fromUnixTimestamp64Micro(i64, 'UTC') AS x, toTypeName(x) Result: ┌──────────────────────────x─┬─toTypeName(x)────────┐ │ 1970-01-15 06:56:07.891011 │ DateTime64(6, 'UTC') │ └────────────────────────────┴──────────────────────┘ ## fromUnixTimestamp64Nano [¶](https://www.tinybird.co/docs/about:blank#fromunixtimestamp64nano) Converts an `Int64` to a `DateTime64` value with fixed nanosecond precision and optional timezone. The input value is scaled up or down appropriately depending on its precision. Please note that input value is treated as a UTC timestamp, not timestamp at the given (or implicit) timezone. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) fromUnixTimestamp64Nano(value[, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `value` : value with any precision. Int64. - `timezone` : (optional) timezone name of the result. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `value` converted to DateTime64 with precision `9` . DateTime64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: WITH CAST(1234567891011, 'Int64') AS i64 SELECT fromUnixTimestamp64Nano(i64, 'UTC') AS x, toTypeName(x) Result: ┌─────────────────────────────x─┬─toTypeName(x)────────┐ │ 1970-01-01 00:20:34.567891011 │ DateTime64(9, 'UTC') │ └───────────────────────────────┴──────────────────────┘ ## formatRow [¶](https://www.tinybird.co/docs/about:blank#formatrow) Converts arbitrary expressions into a string via given format. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) formatRow(format, x, y, ...) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `format` : Text format. For example, CSV, TSV. - `x` , `y` , ...: Expressions. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A formatted string. (for text formats it's usually terminated with the new line character). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT formatRow('CSV', number, 'good') FROM numbers(3) Result: ┌─formatRow('CSV', number, 'good')─┐ │ 0,"good" │ │ 1,"good" │ │ 2,"good" │ └──────────────────────────────────┘ ### Note: If format contains suffix/prefix, it will be written in each row. [¶](https://www.tinybird.co/docs/about:blank#note-if-format-contains-suffixprefix-it-will-be-written-in-each-row) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT formatRow('CustomSeparated', number, 'good') FROM numbers(3) SETTINGS format_custom_result_before_delimiter='\n', format_custom_result_after_delimiter='' Result: ┌─formatRow('CustomSeparated', number, 'good')─┐ │ 0 good │ │ 1 good │ │ 2 good │ └──────────────────────────────────────────────┘ Note: Only row-based formats are supported in this function. ## formatRowNoNewline [¶](https://www.tinybird.co/docs/about:blank#formatrownonewline) Converts arbitrary expressions into a string via given format. Differs from formatRow in that this function trims the last `\n` if any. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) formatRowNoNewline(format, x, y, ...) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `format` : Text format. For example, CSV, TSV. - `x` , `y` , ...: Expressions. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A formatted string. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT formatRowNoNewline('CSV', number, 'good') FROM numbers(3) Result: ┌─formatRowNoNewline('CSV', number, 'good')─┐ │ 0,"good" │ │ 1,"good" │ │ 2,"good" │ └───────────────────────────────────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/tuple-map-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Tuple map functions · Tinybird Docs" theme-color: "#171612" description: "Functions for tuple maps." --- # Tuple map functions [¶](https://www.tinybird.co/docs/about:blank#tuple-map-functions) The following functions are available for tuple maps. ## map [¶](https://www.tinybird.co/docs/about:blank#map) Creates a value of type Map(key, value) from key-value pairs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) map(key1, value1[, key2, value2, ...]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `key_n` : The keys of the map entries. Any type supported as key type of Map. - `value_n` : The values of the map entries. Any type supported as value type of Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A map containing `key:value` pairs. Map(key, value). ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT map('key1', number, 'key2', number * 2) FROM numbers(3) Result: ┌─map('key1', number, 'key2', multiply(number, 2))─┐ │ {'key1':0,'key2':0} │ │ {'key1':1,'key2':2} │ │ {'key1':2,'key2':4} │ └──────────────────────────────────────────────────┘ ## mapFromArrays [¶](https://www.tinybird.co/docs/about:blank#mapfromarrays) Creates a map from an array or map of keys and an array or map of values. The function is a convenient alternative to syntax `CAST([...], 'Map(key_type, value_type)')`. For example, instead of writing - `CAST((['aa', 'bb'], [4, 5]), 'Map(String, UInt32)')` , or - `CAST([('aa',4), ('bb',5)], 'Map(String, UInt32)')` you can write `mapFromArrays(['aa', 'bb'], [4, 5])`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapFromArrays(keys, values) Alias: `MAP_FROM_ARRAYS(keys, values)` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `keys` : Array or map of keys to create the map from Array or Map. If `keys` is an array, Tinybird accepts `Array(Nullable(T))` or `Array(LowCardinality(Nullable(T)))` as its type as long as it doesn't contain NULL value. - `values` - Array or map of values to create the map from Array or Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A map with keys and values constructed from the key array and value array/map. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: select mapFromArrays(['a', 'b', 'c'], [1, 2, 3]) Result: ┌─mapFromArrays(['a', 'b', 'c'], [1, 2, 3])─┐ │ {'a':1,'b':2,'c':3} │ └───────────────────────────────────────────┘ `mapFromArrays` also accepts arguments of type Map. These are cast to array of tuples during execution. SELECT mapFromArrays([1, 2, 3], map('a', 1, 'b', 2, 'c', 3)) Result: ┌─mapFromArrays([1, 2, 3], map('a', 1, 'b', 2, 'c', 3))─┐ │ {1:('a',1),2:('b',2),3:('c',3)} │ └───────────────────────────────────────────────────────┘ SELECT mapFromArrays(map('a', 1, 'b', 2, 'c', 3), [1, 2, 3]) Result: ┌─mapFromArrays(map('a', 1, 'b', 2, 'c', 3), [1, 2, 3])─┐ │ {('a',1):1,('b',2):2,('c',3):3} │ └───────────────────────────────────────────────────────┘ ## extractKeyValuePairs [¶](https://www.tinybird.co/docs/about:blank#extractkeyvaluepairs) Converts a string of key-value pairs to a Map(String, String). Parsing is tolerant towards noise (e.g. log files). Key-value pairs in the input string consist of a key, followed by a key-value delimiter, and a value. Key value pairs are separated by a pair delimiter. Keys and values can be quoted. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extractKeyValuePairs(data[, key_value_delimiter[, pair_delimiter[, quoting_character]]]) Alias: - `str_to_map` - `mapFromString` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `data` - String to extract key-value pairs from. String or FixedString. - `key_value_delimiter` - Single character delimiting keys and values. Defaults to `:` . String or FixedString. - `pair_delimiters` - Set of character delimiting pairs. Defaults to `` , `,` and `;` . String or FixedString. - `quoting_character` - Single character used as quoting character. Defaults to `"` . String or FixedString. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - A of key-value pairs. Type: Map(String, String) ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query SELECT extractKeyValuePairs('name:neymar, age:31 team:psg,nationality:brazil') as kv Result: ┌─kv──────────────────────────────────────────────────────────────────────┐ │ {'name':'neymar','age':'31','team':'psg','nationality':'brazil'} │ └─────────────────────────────────────────────────────────────────────────┘ With a single quote `'` as quoting character: SELECT extractKeyValuePairs('name:\'neymar\';\'age\':31;team:psg;nationality:brazil,last_key:last_value', ':', ';,', '\'') as kv Result: ┌─kv───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ {'name':'neymar','age':'31','team':'psg','nationality':'brazil','last_key':'last_value'} │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Escape sequences without escape sequences support: SELECT extractKeyValuePairs('age:a\\x0A\\n\\0') AS kv Result: ┌─kv─────────────────────┐ │ {'age':'a\\x0A\\n\\0'} │ └────────────────────────┘ To restore a map string key-value pairs serialized with `toString`: SELECT map('John', '33', 'Paula', '31') AS m, toString(m) as map_serialized, extractKeyValuePairs(map_serialized, ':', ',', '\'') AS map_restored Result: Row 1: ────── m: {'John':'33','Paula':'31'} map_serialized: {'John':'33','Paula':'31'} map_restored: {'John':'33','Paula':'31'} ## extractKeyValuePairsWithEscaping [¶](https://www.tinybird.co/docs/about:blank#extractkeyvaluepairswithescaping) Same as `extractKeyValuePairs` but supports escaping. Supported escape sequences: `\x`, `\N`, `\a`, `\b`, `\e`, `\f`, `\n`, `\r`, `\t`, `\v` and `\0`. Non standard escape sequences are returned as it's (including the backslash) unless they are one of the following: `\\`, `'`, `"`, `backtick`, `/`, `=` or ASCII control characters (c <= 31). This function will satisfy the use case where pre-escaping and post-escaping aren't suitable. For instance, consider the following input string: `a: "aaaa\"bbb"` . The expected output is: `a: aaaa\"bbbb`. - Pre-escaping: Pre-escaping it will output: `a: "aaaa"bbb"` and `extractKeyValuePairs` will then output: `a: aaaa` - Post-escaping: `extractKeyValuePairs` will output `a: aaaa\` and post-escaping will keep it as it's. Leading escape sequences will be skipped in keys and will be considered invalid for values. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Escape sequences with escape sequence support turned on: SELECT extractKeyValuePairsWithEscaping('age:a\\x0A\\n\\0') AS kv Result: ┌─kv────────────────┐ │ {'age':'a\n\n\0'} │ └───────────────────┘ ## mapAdd [¶](https://www.tinybird.co/docs/about:blank#mapadd) Collect all the keys and sum corresponding values. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapAdd(arg1, arg2 [, ...]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) Arguments are maps or tuples of two arrays, where items in the first array represent keys, and the second array contains values for the each key. All key arrays should have same type, and all value arrays should contain items which are promoted to the one type (Int64, UInt64 or Float64). The common promoted type is used as a type for the result array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Depending on the arguments returns one map or tuple, where the first array contains the sorted keys and the second array contains values. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query with `Map` type: SELECT mapAdd(map(1,1), map(1,1)) Result: ┌─mapAdd(map(1, 1), map(1, 1))─┐ │ {1:2} │ └──────────────────────────────┘ Query with a tuple: SELECT mapAdd(([toUInt8(1), 2], [1, 1]), ([toUInt8(1), 2], [1, 1])) as res, toTypeName(res) as type Result: ┌─res───────────┬─type───────────────────────────────┐ │ ([1,2],[2,2]) │ Tuple(Array(UInt8), Array(UInt64)) │ └───────────────┴────────────────────────────────────┘ ## mapSubtract [¶](https://www.tinybird.co/docs/about:blank#mapsubtract) Collect all the keys and subtract corresponding values. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapSubtract(Tuple(Array, Array), Tuple(Array, Array) [, ...]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) Arguments are maps or tuples of two arrays, where items in the first array represent keys, and the second array contains values for the each key. All key arrays should have same type, and all value arrays should contain items which are promote to the one type (Int64, UInt64 or Float64). The common promoted type is used as a type for the result array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Depending on the arguments returns one map or tuple, where the first array contains the sorted keys and the second array contains values. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query with `Map` type: SELECT mapSubtract(map(1,1), map(1,1)) Result: ┌─mapSubtract(map(1, 1), map(1, 1))─┐ │ {1:0} │ └───────────────────────────────────┘ Query with a tuple map: SELECT mapSubtract(([toUInt8(1), 2], [toInt32(1), 1]), ([toUInt8(1), 2], [toInt32(2), 1])) as res, toTypeName(res) as type Result: ┌─res────────────┬─type──────────────────────────────┐ │ ([1,2],[-1,0]) │ Tuple(Array(UInt8), Array(Int64)) │ └────────────────┴───────────────────────────────────┘ ## mapPopulateSeries [¶](https://www.tinybird.co/docs/about:blank#mappopulateseries) Fills missing key-value pairs in a map with integer keys. To support extending the keys beyond the largest value, a maximum key can be specified. More specifically, the function returns a map in which the the keys form a series from the smallest to the largest key (or `max` argument if it specified) with step size of 1, and corresponding values. If no value is specified for a key, a default value is used as value. In case keys repeat, only the first value (in order of appearance) is associated with the key. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapPopulateSeries(map[, max]) mapPopulateSeries(keys, values[, max]) For array arguments the number of elements in `keys` and `values` must be the same for each row. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) Arguments are Maps or two Arrays, where the first and second array contains keys and values for the each key. Mapped arrays: - `map` : Map with integer keys. Map. or - `keys` : Array of keys. Array(Int). - `values` : Array of values. Array(Int). - `max` : Maximum key value. Optional. Int8, Int16, Int32, Int64, Int128, Int256. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Depending on the arguments a Map or a Tuple of two Arrays: keys in sorted order, and values the corresponding keys. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query with `Map` type: SELECT mapPopulateSeries(map(1, 10, 5, 20), 6) Result: ┌─mapPopulateSeries(map(1, 10, 5, 20), 6)─┐ │ {1:10,2:0,3:0,4:0,5:20,6:0} │ └─────────────────────────────────────────┘ Query with mapped arrays: SELECT mapPopulateSeries([1,2,4], [11,22,44], 5) AS res, toTypeName(res) AS type Result: ┌─res──────────────────────────┬─type──────────────────────────────┐ │ ([1,2,3,4,5],[11,22,0,44,0]) │ Tuple(Array(UInt8), Array(UInt8)) │ └──────────────────────────────┴───────────────────────────────────┘ ## mapContains [¶](https://www.tinybird.co/docs/about:blank#mapcontains) Returns if a given key is contained in a given map. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapContains(map, key) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map` : Map. Map. - `key` : Key. Type must match the key type of `map` . ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `1` if `map` contains `key` , `0` if not. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapContains(a, 'name') FROM ( select (c1, c2)::Map(String, String) as a from values((['name', 'age'], ['eleven', '11']), (['number', 'position'], ['twelve', '6.0'])) ) Result: ┌─mapContains(a, 'name')─┐ │ 1 │ │ 0 │ └────────────────────────┘ ## mapKeys [¶](https://www.tinybird.co/docs/about:blank#mapkeys) Returns the keys of a given map. This function can be optimized by enabling setting optimize_functions_to_subcolumns. With enabled setting, the function only reads the keys subcolumn instead the whole map. The query `SELECT mapKeys(m) FROM table` is transformed to `SELECT m.keys FROM table`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapKeys(map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map` : Map. Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array containing all keys from the `map` . Array. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapKeys(a) FROM ( select (c1, c2)::Map(String, String) as a from values((['name', 'age'], ['eleven', '11']), (['number', 'position'], ['twelve', '6.0'])) ) Result: ┌─mapKeys(a)────────────┐ │ ['name','age'] │ │ ['number','position'] │ └───────────────────────┘ ## mapValues [¶](https://www.tinybird.co/docs/about:blank#mapvalues) Returns the values of a given map. This function can be optimized by enabling setting optimize_functions_to_subcolumns. With enabled setting, the function only reads the values subcolumn instead the whole map. The query `SELECT mapValues(m) FROM table` is transformed to `SELECT m.values FROM table`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapValues(map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map` : Map. Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array containing all the values from `map` . Array. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapValues(a) FROM ( select (c1, c2)::Map(String, String) as a from values((['name', 'age'], ['eleven', '11']), (['number', 'position'], ['twelve', '6.0'])) ) Result: ┌─mapValues(a)─────┐ │ ['eleven','11'] │ │ ['twelve','6.0'] │ └──────────────────┘ ## mapContainsKeyLike [¶](https://www.tinybird.co/docs/about:blank#mapcontainskeylike) ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapContainsKeyLike(map, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map` : Map. Map. - `pattern` - String pattern to match. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `1` if `map` contains `key` like specified pattern, `0` if not. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapContainsKeyLike(a, 'a%') FROM ( select (c1, c2)::Map(String, String) as a from values((['abc', 'def'], ['abc', 'def']), (['hij', 'klm'], ['hij', 'klm'])) ) Result: ┌─mapContainsKeyLike(a, 'a%')─┐ │ 1 │ │ 0 │ └─────────────────────────────┘ ## mapExtractKeyLike [¶](https://www.tinybird.co/docs/about:blank#mapextractkeylike) Give a map with string keys and a LIKE pattern, this function returns a map with elements where the key matches the pattern. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapExtractKeyLike(map, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map` : Map. Map. - `pattern` - String pattern to match. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A map containing elements the key matching the specified pattern. If no elements match the pattern, an empty map is returned. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapExtractKeyLike(a, 'a%') FROM ( select (c1, c2)::Map(String, String) as a from values((['abc', 'def'], ['abc', 'def']), (['hij', 'klm'], ['hij', 'klm'])) ) Result: ┌─mapExtractKeyLike(a, 'a%')─┐ │ {'abc':'abc'} │ │ {} │ └────────────────────────────┘ ## mapApply [¶](https://www.tinybird.co/docs/about:blank#mapapply) Applies a function to each element of a map. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapApply(func, map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `func` : Lambda function. - `map` : Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns a map obtained from the original map by application of `func(map1[i], ..., mapN[i])` for each element. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapApply((k, v) -> (k, v * 10), _map) AS r FROM ( SELECT map('key1', number, 'key2', number * 2) AS _map FROM numbers(3) ) Result: ┌─r─────────────────────┐ │ {'key1':0,'key2':0} │ │ {'key1':10,'key2':20} │ │ {'key1':20,'key2':40} │ └───────────────────────┘ ## mapFilter [¶](https://www.tinybird.co/docs/about:blank#mapfilter) Filters a map by applying a function to each map element. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapFilter(func, map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `func` - Lambda function. - `map` : Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns a map containing only the elements in `map` for which `func(map1[i], ..., mapN[i])` returns something other than 0. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapFilter((k, v) -> ((v % 2) = 0), _map) AS r FROM ( SELECT map('key1', number, 'key2', number * 2) AS _map FROM numbers(3) ) Result: ┌─r───────────────────┐ │ {'key1':0,'key2':0} │ │ {'key2':2} │ │ {'key1':2,'key2':4} │ └─────────────────────┘ ## mapUpdate [¶](https://www.tinybird.co/docs/about:blank#mapupdate) ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapUpdate(map1, map2) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `map1` Map. - `map2` Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns a map1 with values updated of values for the corresponding keys in map2. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapUpdate(map('key1', 0, 'key3', 0), map('key1', 10, 'key2', 10)) AS map Result: ┌─map────────────────────────────┐ │ {'key3':0,'key1':10,'key2':10} │ └────────────────────────────────┘ ## mapConcat [¶](https://www.tinybird.co/docs/about:blank#mapconcat) Concatenates multiple maps based on the equality of their keys. If elements with the same key exist in more than one input map, all elements are added to the result map, but only the first one is accessible via operator `[]` ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapConcat(maps) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `maps` – Arbitrarily many Maps. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns a map with concatenated maps passed as arguments. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT mapConcat(map('key1', 1, 'key3', 3), map('key2', 2)) AS map Result: ┌─map──────────────────────────┐ │ {'key1':1,'key3':3,'key2':2} │ └──────────────────────────────┘ Query: SELECT mapConcat(map('key1', 1, 'key2', 2), map('key1', 3)) AS map, map['key1'] Result: ┌─map──────────────────────────┬─elem─┐ │ {'key1':1,'key2':2,'key1':3} │ 1 │ └──────────────────────────────┴──────┘ ## mapExists([func,], map) [¶](https://www.tinybird.co/docs/about:blank#mapexistsfunc-map) Returns 1 if at least one key-value pair in `map` exists for which `func(key, value)` returns something other than 0. Otherwise, it returns 0. `mapExists` is a higher-order function. You can pass a lambda function to it as the first argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapExists((k, v) -> (v = 1), map('k1', 1, 'k2', 2)) AS res Result: ┌─res─┐ │ 1 │ └─────┘ ## mapAll([func,] map) [¶](https://www.tinybird.co/docs/about:blank#mapallfunc-map) Returns 1 if `func(key, value)` returns something other than 0 for all key-value pairs in `map` . Otherwise, it returns 0. Note that the `mapAll` is a higher-order function. You can pass a lambda function to it as the first argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT mapAll((k, v) -> (v = 1), map('k1', 1, 'k2', 2)) AS res Result: ┌─res─┐ │ 0 │ └─────┘ ## mapSort([func,], map) [¶](https://www.tinybird.co/docs/about:blank#mapsortfunc-map) Sorts the elements of a map in ascending order. If the `func` function is specified, the sorting order is determined by the result of the `func` function applied to the keys and values of the map. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) SELECT mapSort(map('key2', 2, 'key3', 1, 'key1', 3)) AS map ┌─map──────────────────────────┐ │ {'key1':3,'key2':2,'key3':1} │ └──────────────────────────────┘ SELECT mapSort((k, v) -> v, map('key2', 2, 'key3', 1, 'key1', 3)) AS map ┌─map──────────────────────────┐ │ {'key3':1,'key2':2,'key1':3} │ └──────────────────────────────┘ For more details see the reference for `arraySort` function. ## mapPartialSort [¶](https://www.tinybird.co/docs/about:blank#mappartialsort) Sorts the elements of a map in ascending order with additional `limit` argument allowing partial sorting. If the `func` function is specified, the sorting order is determined by the result of the `func` function applied to the keys and values of the map. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapPartialSort([func,] limit, map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `func` – Optional function to apply to the keys and values of the map. Lambda function. - `limit` – Elements in range [1..limit] are sorted. (U)Int. - `map` – Map to sort. Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Partially sorted map. Map. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT mapPartialSort((k, v) -> v, 2, map('k1', 3, 'k2', 1, 'k3', 2)) ┌─mapPartialSort(lambda(tuple(k, v), v), 2, map('k1', 3, 'k2', 1, 'k3', 2))─┐ │ {'k2':1,'k3':2,'k1':3} │ └───────────────────────────────────────────────────────────────────────────┘ ## mapReverseSort([func,], map) [¶](https://www.tinybird.co/docs/about:blank#mapreversesortfunc-map) Sorts the elements of a map in descending order. If the `func` function is specified, the sorting order is determined by the result of the `func` function applied to the keys and values of the map. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) SELECT mapReverseSort(map('key2', 2, 'key3', 1, 'key1', 3)) AS map ┌─map──────────────────────────┐ │ {'key3':1,'key2':2,'key1':3} │ └──────────────────────────────┘ SELECT mapReverseSort((k, v) -> v, map('key2', 2, 'key3', 1, 'key1', 3)) AS map ┌─map──────────────────────────┐ │ {'key1':3,'key2':2,'key3':1} │ └──────────────────────────────┘ For more details see function arrayReverseSort. ## mapPartialReverseSort [¶](https://www.tinybird.co/docs/about:blank#mappartialreversesort) Sorts the elements of a map in descending order with additional `limit` argument allowing partial sorting. If the `func` function is specified, the sorting order is determined by the result of the `func` function applied to the keys and values of the map. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) mapPartialReverseSort([func,] limit, map) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `func` – Optional function to apply to the keys and values of the map. Lambda function. - `limit` – Elements in range [1..limit] are sorted. (U)Int. - `map` – Map to sort. Map. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Partially sorted map. Map. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT mapPartialReverseSort((k, v) -> v, 2, map('k1', 3, 'k2', 1, 'k3', 2)) ┌─mapPartialReverseSort(lambda(tuple(k, v), v), 2, map('k1', 3, 'k2', 1, 'k3', 2))─┐ │ {'k1':3,'k3':2,'k2':1} │ └──────────────────────────────────────────────────────────────────────────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/tuple-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Tuple functions · Tinybird Docs" theme-color: "#171612" description: "Functions for working with tuples." --- # Tuple functions [¶](https://www.tinybird.co/docs/about:blank#tuple-functions) The following functions are available for working with tuples. ## tuple [¶](https://www.tinybird.co/docs/about:blank#tuple) A function that allows grouping multiple columns. For columns C1, C2, ... with the types T1, T2, ..., it returns a named Tuple(C1 T1, C2 T2, ...) type tuple containing these columns if their names are unique and can be treated as unquoted identifiers, otherwise a Tuple(T1, T2, ...) is returned. There is no cost to execute the function. Tuples are normally used as intermediate values for an argument of IN operators, or for creating a list of formal parameters of lambda functions. Tuples can’t be written to a table. The function implements the operator `(x, y, ...)`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tuple(x, y, ...) ## tupleElement [¶](https://www.tinybird.co/docs/about:blank#tupleelement) A function that allows getting a column from a tuple. If the second argument is a number `index` , it's the column index, starting from 1. If the second argument is a string `name` , it represents the name of the element. Besides, Tinybird can provide the third optional argument, such that when index out of bounds or no element exist for the name, the default value returned instead of throwing an exception. The second and third arguments, if provided, must be constants. There is no cost to execute the function. The function implements operators `x.index` and `x.name`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleElement(tuple, index, [, default_value]) tupleElement(tuple, name, [, default_value]) ## untuple [¶](https://www.tinybird.co/docs/about:blank#untuple) Performs syntactic substitution of tuple elements in the call location. The names of the result columns are implementation-specific and subject to change. Do not assume specific column names after `untuple`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) untuple(x) You can use the `EXCEPT` expression to skip columns as a result of the query. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : A `tuple` function, column, or tuple of elements. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - None. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Input table: ┌─key─┬─v1─┬─v2─┬─v3─┬─v4─┬─v5─┬─v6────────┐ │ 1 │ 10 │ 20 │ 40 │ 30 │ 15 │ (33,'ab') │ │ 2 │ 25 │ 65 │ 70 │ 40 │ 6 │ (44,'cd') │ │ 3 │ 57 │ 30 │ 20 │ 10 │ 5 │ (55,'ef') │ │ 4 │ 55 │ 12 │ 7 │ 80 │ 90 │ (66,'gh') │ │ 5 │ 30 │ 50 │ 70 │ 25 │ 55 │ (77,'kl') │ └─────┴────┴────┴────┴────┴────┴───────────┘ Example of using a `Tuple` -type column as the `untuple` function parameter: Query: SELECT untuple(v6) FROM kv Result: ┌─_ut_1─┬─_ut_2─┐ │ 33 │ ab │ │ 44 │ cd │ │ 55 │ ef │ │ 66 │ gh │ │ 77 │ kl │ └───────┴───────┘ Example of using an `EXCEPT` expression: Query: SELECT untuple((* EXCEPT (v2, v3),)) FROM kv Result: ┌─key─┬─v1─┬─v4─┬─v5─┬─v6────────┐ │ 1 │ 10 │ 30 │ 15 │ (33,'ab') │ │ 2 │ 25 │ 40 │ 6 │ (44,'cd') │ │ 3 │ 57 │ 10 │ 5 │ (55,'ef') │ │ 4 │ 55 │ 80 │ 90 │ (66,'gh') │ │ 5 │ 30 │ 25 │ 55 │ (77,'kl') │ └─────┴────┴────┴────┴───────────┘ ## tupleHammingDistance [¶](https://www.tinybird.co/docs/about:blank#tuplehammingdistance) Returns the Hamming Distance between two tuples of the same size. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleHammingDistance(tuple1, tuple2) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple1` : First tuple. Tuple. - `tuple2` : Second tuple. Tuple. Tuples should have the same type of the elements. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The Hamming distance. The result type is calculated the same way it's for Arithmetic functions, based on the number of elements in the input tuples. SELECT toTypeName(tupleHammingDistance(tuple(0), tuple(0))) AS t1, toTypeName(tupleHammingDistance((0, 0), (0, 0))) AS t2, toTypeName(tupleHammingDistance((0, 0, 0), (0, 0, 0))) AS t3, toTypeName(tupleHammingDistance((0, 0, 0, 0), (0, 0, 0, 0))) AS t4, toTypeName(tupleHammingDistance((0, 0, 0, 0, 0), (0, 0, 0, 0, 0))) AS t5 ┌─t1────┬─t2─────┬─t3─────┬─t4─────┬─t5─────┐ │ UInt8 │ UInt16 │ UInt32 │ UInt64 │ UInt64 │ └───────┴────────┴────────┴────────┴────────┘ ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleHammingDistance((1, 2, 3), (3, 2, 1)) AS HammingDistance Result: ┌─HammingDistance─┐ │ 2 │ └─────────────────┘ Can be used with MinHash functions for detection of semi-duplicate strings: SELECT tupleHammingDistance(wordShingleMinHash(string), wordShingleMinHashCaseInsensitive(string)) AS HammingDistance FROM (SELECT 'Bazinga is a column-oriented database management system for online analytical processing of queries.' AS string) Result: ┌─HammingDistance─┐ │ 2 │ └─────────────────┘ ## tupleToNameValuePairs [¶](https://www.tinybird.co/docs/about:blank#tupletonamevaluepairs) Turns a named tuple into an array of (name, value) pairs. For a `Tuple(a T, b T, ..., c T)` returns `Array(Tuple(String, T), ...)` in which the `Strings` represents the named fields of the tuple and `T` are the values associated with those names. All values in the tuple should be of the same type. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleToNameValuePairs(tuple) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple` : Named tuple. Tuple with any types of values. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - An array with (name, value) pairs. Array(Tuple(String, ...)). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleToNameValuePairs(col) FROM (select (c1,c2)::Tuple(user_ID UInt64, session_ID UInt64) as col from values((100, 2502), (1, 100))) Result: ┌─tupleToNameValuePairs(col)────────────┐ │ [('user_ID',100),('session_ID',2502)] │ │ [('user_ID',1),('session_ID',100)] │ └───────────────────────────────────────┘ It is possible to transform columns to rows using this function: SELECT arrayJoin(tupleToNameValuePairs(col)) FROM (select (c1,c2,c3)::Tuple(CPU Float64, Memory Float64, Disk Float64) as col from values((tuple(3.3, 5.5, 6.6)))) Result: ┌─arrayJoin(tupleToNameValuePairs(col))─┐ │ ('CPU',3.3) │ │ ('Memory',5.5) │ │ ('Disk',6.6) │ └───────────────────────────────────────┘ If you pass a simple tuple to the function, Tinybird uses the indexes of the values as their names: SELECT tupleToNameValuePairs(tuple(3, 2, 1)) Result: ┌─tupleToNameValuePairs(tuple(3, 2, 1))─┐ │ [('1',3),('2',2),('3',1)] │ └───────────────────────────────────────┘ ## tuplePlus [¶](https://www.tinybird.co/docs/about:blank#tupleplus) Calculates the sum of corresponding values of two tuples of the same size. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tuplePlus(tuple1, tuple2) Alias: `vectorSum`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple1` : First tuple. Tuple. - `tuple2` : Second tuple. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with the sum. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tuplePlus((1, 2), (2, 3)) Result: ┌─tuplePlus((1, 2), (2, 3))─┐ │ (3,5) │ └───────────────────────────┘ ## tupleMinus [¶](https://www.tinybird.co/docs/about:blank#tupleminus) Calculates the subtraction of corresponding values of two tuples of the same size. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleMinus(tuple1, tuple2) Alias: `vectorDifference`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple1` : First tuple. Tuple. - `tuple2` : Second tuple. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with the result of subtraction. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleMinus((1, 2), (2, 3)) Result: ┌─tupleMinus((1, 2), (2, 3))─┐ │ (-1,-1) │ └────────────────────────────┘ ## tupleMultiply [¶](https://www.tinybird.co/docs/about:blank#tuplemultiply) Calculates the multiplication of corresponding values of two tuples of the same size. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleMultiply(tuple1, tuple2) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple1` : First tuple. Tuple. - `tuple2` : Second tuple. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with the multiplication. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleMultiply((1, 2), (2, 3)) Result: ┌─tupleMultiply((1, 2), (2, 3))─┐ │ (2,6) │ └───────────────────────────────┘ ## tupleDivide [¶](https://www.tinybird.co/docs/about:blank#tupledivide) Calculates the division of corresponding values of two tuples of the same size. Note that division by zero will return `inf`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleDivide(tuple1, tuple2) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple1` : First tuple. Tuple. - `tuple2` : Second tuple. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with the result of division. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleDivide((1, 2), (2, 3)) Result: ┌─tupleDivide((1, 2), (2, 3))─┐ │ (0.5,0.6666666666666666) │ └─────────────────────────────┘ ## tupleNegate [¶](https://www.tinybird.co/docs/about:blank#tuplenegate) Calculates the negation of the tuple values. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleNegate(tuple) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple` : Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with the result of negation. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleNegate((1, 2)) Result: ┌─tupleNegate((1, 2))─┐ │ (-1,-2) │ └─────────────────────┘ ## tupleMultiplyByNumber [¶](https://www.tinybird.co/docs/about:blank#tuplemultiplybynumber) Returns a tuple with all values multiplied by a number. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleMultiplyByNumber(tuple, number) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple` : Tuple. - `number` : Multiplier. Int/UInt, Float or Decimal. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with multiplied values. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleMultiplyByNumber((1, 2), -2.1) Result: ┌─tupleMultiplyByNumber((1, 2), -2.1)─┐ │ (-2.1,-4.2) │ └─────────────────────────────────────┘ ## tupleDivideByNumber [¶](https://www.tinybird.co/docs/about:blank#tupledividebynumber) Returns a tuple with all values divided by a number. Note that division by zero will return `inf`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleDivideByNumber(tuple, number) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuple` : Tuple. - `number` : Divider. Int/UInt, Float or Decimal. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple with divided values. Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tupleDivideByNumber((1, 2), 0.5) Result: ┌─tupleDivideByNumber((1, 2), 0.5)─┐ │ (2,4) │ └──────────────────────────────────┘ ## tupleConcat [¶](https://www.tinybird.co/docs/about:blank#tupleconcat) Combines tuples passed as arguments. tupleConcat(tuples) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `tuples` – Arbitrary number of arguments of Tuple type. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT tupleConcat((1, 2), (3, 4), (true, false)) AS res ┌─res──────────────────┐ │ (1,2,3,4,true,false) │ └──────────────────────┘ ## tupleIntDiv [¶](https://www.tinybird.co/docs/about:blank#tupleintdiv) Does integer division of a tuple of numerators and a tuple of denominators, and returns a tuple of the quotients. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleIntDiv(tuple_num, tuple_div) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `tuple_div` : Tuple of divisor values. Tuple of numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the quotients of `tuple_num` and `tuple_div` . Tuple of integer values. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) - If either `tuple_num` or `tuple_div` contain non-integer values then the result is calculated by rounding to the nearest integer for each non-integer numerator or divisor. - An error will be thrown for division by 0. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleIntDiv((15, 10, 5), (5, 5, 5)) Result: ┌─tupleIntDiv((15, 10, 5), (5, 5, 5))─┐ │ (3,2,1) │ └─────────────────────────────────────┘ Query: SELECT tupleIntDiv((15, 10, 5), (5.5, 5.5, 5.5)) Result: ┌─tupleIntDiv((15, 10, 5), (5.5, 5.5, 5.5))─┐ │ (2,1,0) │ └───────────────────────────────────────────┘ ## tupleIntDivOrZero [¶](https://www.tinybird.co/docs/about:blank#tupleintdivorzero) Like tupleIntDiv it does integer division of a tuple of numerators and a tuple of denominators, and returns a tuple of the quotients. It doesn't throw an error for 0 divisors, but rather returns the quotient as 0. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleIntDivOrZero(tuple_num, tuple_div) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `tuple_div` : Tuple of divisor values. Tuple of numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the quotients of `tuple_num` and `tuple_div` . Tuple of integer values. - Returns 0 for quotients where the divisor is 0. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) - If either `tuple_num` or `tuple_div` contain non-integer values then the result is calculated by rounding to the nearest integer for each non-integer numerator or divisor as in tupleIntDiv. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleIntDivOrZero((5, 10, 15), (0, 0, 0)) Result: ┌─tupleIntDivOrZero((5, 10, 15), (0, 0, 0))─┐ │ (0,0,0) │ └───────────────────────────────────────────┘ ## tupleIntDivByNumber [¶](https://www.tinybird.co/docs/about:blank#tupleintdivbynumber) Does integer division of a tuple of numerators by a given denominator, and returns a tuple of the quotients. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleIntDivByNumber(tuple_num, div) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `div` : The divisor value. Numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the quotients of `tuple_num` and `div` . Tuple of integer values. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) - If either `tuple_num` or `div` contain non-integer values then the result is calculated by rounding to the nearest integer for each non-integer numerator or divisor. - An error will be thrown for division by 0. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleIntDivByNumber((15, 10, 5), 5) Result: ┌─tupleIntDivByNumber((15, 10, 5), 5)─┐ │ (3,2,1) │ └─────────────────────────────────────┘ Query: SELECT tupleIntDivByNumber((15.2, 10.7, 5.5), 5.8) Result: ┌─tupleIntDivByNumber((15.2, 10.7, 5.5), 5.8)─┐ │ (2,1,0) │ └─────────────────────────────────────────────┘ ## tupleIntDivOrZeroByNumber [¶](https://www.tinybird.co/docs/about:blank#tupleintdivorzerobynumber) Like tupleIntDivByNumber it does integer division of a tuple of numerators by a given denominator, and returns a tuple of the quotients. It doesn't throw an error for 0 divisors, but rather returns the quotient as 0. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleIntDivOrZeroByNumber(tuple_num, div) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `div` : The divisor value. Numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the quotients of `tuple_num` and `div` . Tuple of integer values. - Returns 0 for quotients where the divisor is 0. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) - If either `tuple_num` or `div` contain non-integer values then the result is calculated by rounding to the nearest integer for each non-integer numerator or divisor as in tupleIntDivByNumber. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleIntDivOrZeroByNumber((15, 10, 5), 5) Result: ┌─tupleIntDivOrZeroByNumber((15, 10, 5), 5)─┐ │ (3,2,1) │ └───────────────────────────────────────────┘ Query: SELECT tupleIntDivOrZeroByNumber((15, 10, 5), 0) Result: ┌─tupleIntDivOrZeroByNumber((15, 10, 5), 0)─┐ │ (0,0,0) │ └───────────────────────────────────────────┘ ## tupleModulo [¶](https://www.tinybird.co/docs/about:blank#tuplemodulo) Returns a tuple of the moduli (remainders) of division operations of two tuples. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleModulo(tuple_num, tuple_mod) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `tuple_div` : Tuple of modulus values. Tuple of numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the remainders of division of `tuple_num` and `tuple_div` . Tuple of non-zero integer values. - An error is thrown for division by zero. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleModulo((15, 10, 5), (5, 3, 2)) Result: ┌─tupleModulo((15, 10, 5), (5, 3, 2))─┐ │ (0,1,1) │ └─────────────────────────────────────┘ ## tupleModuloByNumber [¶](https://www.tinybird.co/docs/about:blank#tuplemodulobynumber) Returns a tuple of the moduli (remainders) of division operations of a tuple and a given divisor. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tupleModuloByNumber(tuple_num, div) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `tuple_num` : Tuple of numerator values. Tuple of numeric type. - `div` : The divisor value. Numeric type. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Tuple of the remainders of division of `tuple_num` and `div` . Tuple of non-zero integer values. - An error is thrown for division by zero. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tupleModuloByNumber((15, 10, 5), 2) Result: ┌─tupleModuloByNumber((15, 10, 5), 2)─┐ │ (1,0,1) │ └─────────────────────────────────────┘ ## flattenTuple [¶](https://www.tinybird.co/docs/about:blank#flattentuple) Returns a flattened `output` tuple from a nested named `input` tuple. Elements of the `output` tuple are the paths from the original `input` tuple. For instance: `Tuple(a Int, Tuple(b Int, c Int)) -> Tuple(a Int, b Int, c Int)`. `flattenTuple` can be used to select all paths from type `Object` as separate columns. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) flattenTuple(input) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : Nested named tuple to flatten. Tuple. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - `output` tuple whose elements are paths from the original `input` . Tuple. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT flattenTuple(t) FROM (select (c1,c2,c3)::Tuple(t1 Nested(a UInt32, s String), b UInt32, t2 Tuple(k String, v UInt32)) as t from values((([(1, 'a'), (2, 'b')], 3, ('c', 4))))) Result: ┌─flattenTuple(t)───────────┐ │ ([1,2],['a','b'],3,'c',4) │ └───────────────────────────┘ ## Distance functions [¶](https://www.tinybird.co/docs/about:blank#distance-functions) All supported functions are described in [distance functions documentation](https://www.tinybird.co/docs/docs/sql-reference/functions/distance-functions). --- URL: https://www.tinybird.co/docs/sql-reference/functions/time-window-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Time window functions · Tinybird Docs" theme-color: "#171612" description: "Functions for time windows." --- # Time Window functions [¶](https://www.tinybird.co/docs/about:blank#time-window-functions) Time window functions return the inclusive lower and exclusive upper bound of the corresponding window. ## tumble [¶](https://www.tinybird.co/docs/about:blank#tumble) A tumbling time window assigns records to non-overlapping, continuous windows with a fixed duration ( `interval` ). ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tumble(time_attr, interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `interval` : Window interval in Interval. - `timezone` : Timezone name (optional). ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The inclusive lower and exclusive upper bound of the corresponding tumbling window. Tuple(DateTime, DateTime). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tumble(now(), toIntervalDay('1')) Result: ┌─tumble(now(), toIntervalDay('1'))─────────────┐ │ ('2024-07-04 00:00:00','2024-07-05 00:00:00') │ └───────────────────────────────────────────────┘ ## tumbleStart [¶](https://www.tinybird.co/docs/about:blank#tumblestart) Returns the inclusive lower bound of the corresponding tumbling window. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tumbleStart(time_attr, interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `interval` : Window interval in Interval. - `timezone` : Timezone name (optional). The parameters above can also be passed to the function as a tuple. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The inclusive lower bound of the corresponding tumbling window. DateTime, Tuple or UInt32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tumbleStart(now(), toIntervalDay('1')) Result: ┌─tumbleStart(now(), toIntervalDay('1'))─┐ │ 2024-07-04 00:00:00 │ └────────────────────────────────────────┘ ## tumbleEnd [¶](https://www.tinybird.co/docs/about:blank#tumbleend) Returns the exclusive upper bound of the corresponding tumbling window. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tumbleEnd(time_attr, interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `interval` : Window interval in Interval. - `timezone` : Timezone name (optional). The parameters above can also be passed to the function as a tuple. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The inclusive lower bound of the corresponding tumbling window. DateTime, Tuple or UInt32. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT tumbleEnd(now(), toIntervalDay('1')) Result: ┌─tumbleEnd(now(), toIntervalDay('1'))─┐ │ 2024-07-05 00:00:00 │ └──────────────────────────────────────┘ ## hop [¶](https://www.tinybird.co/docs/about:blank#hop) A hopping time window has a fixed duration ( `window_interval` ) and hops by a specified hop interval ( `hop_interval` ). If the `hop_interval` is smaller than the `window_interval` , hopping windows are overlapping. Thus, records can be assigned to multiple windows. hop(time_attr, hop_interval, window_interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `hop_interval` : Positive Hop interval. Interval. - `window_interval` : Positive Window interval. Interval. - `timezone` : Timezone name (optional). ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The inclusive lower and exclusive upper bound of the corresponding hopping window. Tuple(DateTime, DateTime)`. Since one record can be assigned to multiple hop windows, the function only returns the bound of the first window when hop function is used without `WINDOW VIEW`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT hop(now(), INTERVAL '1' DAY, INTERVAL '2' DAY) Result: ┌─hop(now(), toIntervalDay('1'), toIntervalDay('2'))─┐ │ ('2024-07-03 00:00:00','2024-07-05 00:00:00') │ └────────────────────────────────────────────────────┘ ## hopStart [¶](https://www.tinybird.co/docs/about:blank#hopstart) Returns the inclusive lower bound of the corresponding hopping window. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hopStart(time_attr, hop_interval, window_interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `hop_interval` : Positive Hop interval. Interval. - `window_interval` : Positive Window interval. Interval. - `timezone` : Timezone name (optional). The parameters above can also be passed to the function as a tuple. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The inclusive lower bound of the corresponding hopping window. DateTime, Tuple or UInt32. Since one record can be assigned to multiple hop windows, the function only returns the bound of the first window when hop function is used without `WINDOW VIEW`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT hopStart(now(), INTERVAL '1' DAY, INTERVAL '2' DAY) Result: ┌─hopStart(now(), toIntervalDay('1'), toIntervalDay('2'))─┐ │ 2024-07-03 00:00:00 │ └─────────────────────────────────────────────────────────┘ ## hopEnd [¶](https://www.tinybird.co/docs/about:blank#hopend) Returns the exclusive upper bound of the corresponding hopping window. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hopEnd(time_attr, hop_interval, window_interval [, timezone]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `time_attr` : Date and time. DateTime. - `hop_interval` : Positive Hop interval. Interval. - `window_interval` : Positive Window interval. Interval. - `timezone` : Timezone name (optional). The parameters above can also be passed to the function as a tuple. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) - The exclusive upper bound of the corresponding hopping window. DateTime, Tuple or UInt32. Since one record can be assigned to multiple hop windows, the function only returns the bound of the first window when hop function is used without `WINDOW VIEW`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT hopEnd(now(), INTERVAL '1' DAY, INTERVAL '2' DAY) Result: ┌─hopEnd(now(), toIntervalDay('1'), toIntervalDay('2'))─┐ │ 2024-07-05 00:00:00 │ └───────────────────────────────────────────────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/time-series-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Time series functions · Tinybird Docs" theme-color: "#171612" description: "Functions for time series." --- # Time series functions [¶](https://www.tinybird.co/docs/about:blank#time-series-functions) Below functions are used for series data analysis. ## seriesOutliersDetectTukey [¶](https://www.tinybird.co/docs/about:blank#seriesoutliersdetecttukey) Detects outliers in series data using Tukey Fences. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) seriesOutliersDetectTukey(series) seriesOutliersDetectTukey(series, min_percentile, max_percentile, K) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `series` - An array of numeric values. - `min_percentile` - The minimum percentile to be used to calculate inter-quantile range (IQR). The value must be in range [0.02,0.98]. The default is 0.25. - `max_percentile` - The maximum percentile to be used to calculate inter-quantile range (IQR). The value must be in range [0.02,0.98]. The default is 0.75. - `K` - Non-negative constant value to detect mild or stronger outliers. The default value is 1.5. At least four data points are required in `series` to detect outliers. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns an array of the same length as the input array where each value represents score of possible anomaly of corresponding element in the series. A non-zero score indicates a possible anomaly. Array. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT seriesOutliersDetectTukey([-3, 2, 15, 3, 5, 6, 4, 5, 12, 45, 12, 3, 3, 4, 5, 6]) AS print_0 Result: ┌───────────print_0─────────────────┐ │[0,0,0,0,0,0,0,0,0,27,0,0,0,0,0,0] │ └───────────────────────────────────┘ Query: SELECT seriesOutliersDetectTukey([-3, 2, 15, 3, 5, 6, 4.50, 5, 12, 45, 12, 3.40, 3, 4, 5, 6], 0.2, 0.8, 1.5) AS print_0 Result: ┌─print_0──────────────────────────────┐ │ [0,0,0,0,0,0,0,0,0,19.5,0,0,0,0,0,0] │ └──────────────────────────────────────┘ ## seriesPeriodDetectFFT [¶](https://www.tinybird.co/docs/about:blank#seriesperioddetectfft) Finds the period of the given series data data using FFT FFT - Fast Fourier transform ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) seriesPeriodDetectFFT(series) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `series` - An array of numeric values ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A real value equal to the period of series data. NaN when number of data points are less than four. Float64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT seriesPeriodDetectFFT([1, 4, 6, 1, 4, 6, 1, 4, 6, 1, 4, 6, 1, 4, 6, 1, 4, 6, 1, 4, 6]) AS print_0 Result: ┌───────────print_0──────┐ │ 3 │ └────────────────────────┘ SELECT seriesPeriodDetectFFT(arrayMap(x -> abs((x % 6) - 3), range(1000))) AS print_0 Result: ┌─print_0─┐ │ 6 │ └─────────┘ ## seriesDecomposeSTL [¶](https://www.tinybird.co/docs/about:blank#seriesdecomposestl) Decomposes a series data using STL (Seasonal-Trend Decomposition Procedure Based on Loess) into a season, a trend and a residual component. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) seriesDecomposeSTL(series, period) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `series` - An array of numeric values - `period` - A positive integer The number of data points in `series` should be at least twice the value of `period`. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - An array of four arrays where the first array include seasonal components, the second array - trend, the third array - residue component, and the fourth array - baseline(seasonal + trend) component. Array. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT seriesDecomposeSTL([10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34], 3) AS print_0 Result: ┌───────────print_0──────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ [[ -13.529999, -3.1799996, 16.71, -13.53, -3.1799996, 16.71, -13.53, -3.1799996, 16.71, -13.530001, -3.18, 16.710001, -13.530001, -3.1800003, 16.710001, -13.530001, -3.1800003, 16.710001, -13.530001, -3.1799994, 16.71, -13.529999, -3.1799994, 16.709997 ], [ 23.63, 23.63, 23.630003, 23.630001, 23.630001, 23.630001, 23.630001, 23.630001, 23.630001, 23.630001, 23.630001, 23.63, 23.630001, 23.630001, 23.63, 23.630001, 23.630001, 23.63, 23.630001, 23.630001, 23.630001, 23.630001, 23.630001, 23.630003 ], [ 0, 0.0000019073486, -0.0000019073486, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -0.0000019073486, 0, 0 ], [ 10.1, 20.449999, 40.340004, 10.100001, 20.45, 40.34, 10.100001, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.1, 20.45, 40.34, 10.100002, 20.45, 40.34 ]] │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/string-search-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Search functions · Tinybird Docs" theme-color: "#171612" description: "Functions for searching in strings." --- # Functions for searching in strings [¶](https://www.tinybird.co/docs/about:blank#functions-for-searching-in-strings) All functions in this section search case-sensitively by default. Case-insensitive search is usually provided by separate function variants. Functions in this section also assume that the searched string, referred to in this section as `haystack` , and the search string, referred to in this section as `needle` , are single-byte encoded text. If this assumption is violated, no exception is thrown and results are undefined. Search with UTF-8 encoded strings is usually provided by separate function variants. Likewise, if a UTF-8 function variant is used and the input strings aren't UTF-8 encoded text, no exception is thrown and the results are undefined. Note that no automatic Unicode normalization is performed, however you can use the normalizeUTF8*() functions for that. General strings functions and functions for replacing in strings are described separately. ## position [¶](https://www.tinybird.co/docs/about:blank#position) Returns the position (in bytes, starting at 1) of a substring `needle` in a string `haystack`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) position(haystack, needle[, start_pos]) Alias: - `position(needle IN haystack)` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Substring to be searched. String. - `start_pos` – Position (1-based) in `haystack` at which the search starts. UInt. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Starting position in bytes and counting from 1, if the substring was found. UInt64. - 0, if the substring was not found. UInt64. If substring `needle` is empty, these rules apply: - if no `start_pos` was specified: return `1` - if `start_pos = 0` : return `1` - if `start_pos >= 1` and `start_pos <= length(haystack) + 1` : return `start_pos` - otherwise: return `0` The same rules also apply to functions `locate`, `positionCaseInsensitive`, `positionUTF8` and `positionCaseInsensitiveUTF8`. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT position('Hello, world!', '!') Result: ┌─position('Hello, world!', '!')─┐ │ 13 │ └────────────────────────────────┘ Example with `start_pos` argument: Query: SELECT position('Hello, world!', 'o', 1), position('Hello, world!', 'o', 7) Result: ┌─position('Hello, world!', 'o', 1)─┬─position('Hello, world!', 'o', 7)─┐ │ 5 │ 9 │ └───────────────────────────────────┴───────────────────────────────────┘ Example for `needle IN haystack` syntax: Query: SELECT 6 = position('/' IN s) FROM (SELECT 'Hello/World' AS s) Result: ┌─equals(6, position(s, '/'))─┐ │ 1 │ └─────────────────────────────┘ Examples with empty `needle` substring: Query: SELECT position('abc', ''), position('abc', '', 0), position('abc', '', 1), position('abc', '', 2), position('abc', '', 3), position('abc', '', 4), position('abc', '', 5) Result: ┌─position('abc', '')─┬─position('abc', '', 0)─┬─position('abc', '', 1)─┬─position('abc', '', 2)─┬─position('abc', '', 3)─┬─position('abc', '', 4)─┬─position('abc', '', 5)─┐ │ 1 │ 1 │ 1 │ 2 │ 3 │ 4 │ 0 │ └─────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┘ ## locate [¶](https://www.tinybird.co/docs/about:blank#locate) Like position but with arguments `haystack` and `locate` switched. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) locate(needle, haystack[, start_pos]) ## positionCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#positioncaseinsensitive) A case insensitive invariant of position. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT positionCaseInsensitive('Hello, world!', 'hello') Result: ┌─positionCaseInsensitive('Hello, world!', 'hello')─┐ │ 1 │ └───────────────────────────────────────────────────┘ ## positionUTF8 [¶](https://www.tinybird.co/docs/about:blank#positionutf8) Like position but assumes `haystack` and `needle` are UTF-8 encoded strings. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Function `positionUTF8` correctly counts character `ö` (represented by two points) as a single Unicode codepoint: Query: SELECT positionUTF8('Motörhead', 'r') Result: ┌─position('Motörhead', 'r')─┐ │ 5 │ └────────────────────────────┘ ## positionCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#positioncaseinsensitiveutf8) Like positionUTF8 but searches case-insensitively. ## multiSearchAllPositions [¶](https://www.tinybird.co/docs/about:blank#multisearchallpositions) Like position but returns an array of positions (in bytes, starting at 1) for multiple `needle` substrings in a `haystack` string. All `multiSearch*()` functions only support up to 28 needles. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAllPositions(haystack, [needle1, needle2, ..., needleN]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of the starting position in bytes and counting from 1, if the substring was found. - 0, if the substring was not found. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchAllPositions('Hello, World!', ['hello', '!', 'world']) Result: ┌─multiSearchAllPositions('Hello, World!', ['hello', '!', 'world'])─┐ │ [0,13,0] │ └───────────────────────────────────────────────────────────────────┘ ## multiSearchAllPositionsCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#multisearchallpositionscaseinsensitive) Like multiSearchAllPositions but ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAllPositionsCaseInsensitive(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of the starting position in bytes and counting from 1 (if the substring was found). - 0 if the substring was not found. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchAllPositionsCaseInsensitive('Tinybird',['c','h']) Result: ["1","6"] ## multiSearchAllPositionsUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchallpositionsutf8) Like multiSearchAllPositions but assumes `haystack` and the `needle` substrings are UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAllPositionsUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 encoded string in which the search is performed. String. - `needle` : UTF-8 encoded substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of the starting position in bytes and counting from 1 (if the substring was found). - 0 if the substring was not found. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `Tinybird` as a UTF-8 string, find the positions of `C` ( `\x43` ) and `H` ( `\x48` ). Query: SELECT multiSearchAllPositionsUTF8('\x43\x6c\x69\x63\x6b\x48\x6f\x75\x73\x65',['\x43','\x48']) Result: ["1","6"] ## multiSearchAllPositionsCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchallpositionscaseinsensitiveutf8) Like multiSearchAllPositionsUTF8 but ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAllPositionsCaseInsensitiveUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 encoded string in which the search is performed. String. - `needle` : UTF-8 encoded substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of the starting position in bytes and counting from 1 (if the substring was found). - 0 if the substring was not found. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `Tinybird` as a UTF-8 string, find the positions of `c` ( `\x63` ) and `h` ( `\x68` ). Query: SELECT multiSearchAllPositionsCaseInsensitiveUTF8('\x43\x6c\x69\x63\x6b\x48\x6f\x75\x73\x65',['\x63','\x68']) Result: ["1","6"] ## multiSearchFirstPosition [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstposition) Like `position` but returns the leftmost offset in a `haystack` string which matches any of multiple `needle` strings. Functions `multiSearchFirstPositionCaseInsensitive`, `multiSearchFirstPositionUTF8` and `multiSearchFirstPositionCaseInsensitiveUTF8` provide case-insensitive and/or UTF-8 variants of this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstPosition(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Leftmost offset in a `haystack` string which matches any of multiple `needle` strings. - 0, if there was no match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchFirstPosition('Hello World',['llo', 'Wor', 'ld']) Result: 3 ## multiSearchFirstPositionCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstpositioncaseinsensitive) Like `multiSearchFirstPosition` but ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstPositionCaseInsensitive(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Array of substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Leftmost offset in a `haystack` string which matches any of multiple `needle` strings. - 0, if there was no match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchFirstPositionCaseInsensitive('HELLO WORLD',['wor', 'ld', 'ello']) Result: 2 ## multiSearchFirstPositionUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstpositionutf8) Like `multiSearchFirstPosition` but assumes `haystack` and `needle` to be UTF-8 strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstPositionUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : Array of UTF-8 substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Leftmost offset in a `haystack` string which matches any of multiple `needle` strings. - 0, if there was no match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Find the leftmost offset in UTF-8 string `hello world` which matches any of the given needles. Query: SELECT multiSearchFirstPositionUTF8('\x68\x65\x6c\x6c\x6f\x20\x77\x6f\x72\x6c\x64',['wor', 'ld', 'ello']) Result: 2 ## multiSearchFirstPositionCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstpositioncaseinsensitiveutf8) Like `multiSearchFirstPosition` but assumes `haystack` and `needle` to be UTF-8 strings and ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstPositionCaseInsensitiveUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : Array of UTF-8 substrings to be searched. Array ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Leftmost offset in a `haystack` string which matches any of multiple `needle` strings, ignoring case. - 0, if there was no match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Find the leftmost offset in UTF-8 string `HELLO WORLD` which matches any of the given needles. Query: SELECT multiSearchFirstPositionCaseInsensitiveUTF8('\x48\x45\x4c\x4c\x4f\x20\x57\x4f\x52\x4c\x44',['wor', 'ld', 'ello']) Result: 2 ## multiSearchFirstIndex [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstindex) Returns the index `i` (starting from 1) of the leftmost found needlei in the string `haystack` and 0 otherwise. Functions `multiSearchFirstIndexCaseInsensitive`, `multiSearchFirstIndexUTF8` and `multiSearchFirstIndexCaseInsensitiveUTF8` provide case-insensitive and/or UTF-8 variants of this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstIndex(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - index (starting from 1) of the leftmost found needle. Otherwise 0, if there was no match. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchFirstIndex('Hello World',['World','Hello']) Result: 1 ## multiSearchFirstIndexCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstindexcaseinsensitive) Returns the index `i` (starting from 1) of the leftmost found needlei in the string `haystack` and 0 otherwise. Ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstIndexCaseInsensitive(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - index (starting from 1) of the leftmost found needle. Otherwise 0, if there was no match. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchFirstIndexCaseInsensitive('hElLo WoRlD',['World','Hello']) Result: 1 ## multiSearchFirstIndexUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstindexutf8) Returns the index `i` (starting from 1) of the leftmost found needlei in the string `haystack` and 0 otherwise. Assumes `haystack` and `needle` are UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstIndexUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : Array of UTF-8 substrings to be searched. Array ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - index (starting from 1) of the leftmost found needle, Otherwise 0, if there was no match. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `Hello World` as a UTF-8 string, find the first index of UTF-8 strings `Hello` and `World`. Query: SELECT multiSearchFirstIndexUTF8('\x48\x65\x6c\x6c\x6f\x20\x57\x6f\x72\x6c\x64',['\x57\x6f\x72\x6c\x64','\x48\x65\x6c\x6c\x6f']) Result: 1 ## multiSearchFirstIndexCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchfirstindexcaseinsensitiveutf8) Returns the index `i` (starting from 1) of the leftmost found needlei in the string `haystack` and 0 otherwise. Assumes `haystack` and `needle` are UTF-8 encoded strings. Ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchFirstIndexCaseInsensitiveUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : Array of UTF-8 substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - index (starting from 1) of the leftmost found needle. Otherwise 0, if there was no match. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `HELLO WORLD` as a UTF-8 string, find the first index of UTF-8 strings `hello` and `world`. Query: SELECT multiSearchFirstIndexCaseInsensitiveUTF8('\x48\x45\x4c\x4c\x4f\x20\x57\x4f\x52\x4c\x44',['\x68\x65\x6c\x6c\x6f','\x77\x6f\x72\x6c\x64']) Result: 1 ## multiSearchAny [¶](https://www.tinybird.co/docs/about:blank#multisearchany) Returns 1, if at least one string needlei matches the string `haystack` and 0 otherwise. Functions `multiSearchAnyCaseInsensitive`, `multiSearchAnyUTF8` and `multiSearchAnyCaseInsensitiveUTF8` provide case-insensitive and/or UTF-8 variants of this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAny(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if there was at least one match. - 0, if there was not at least one match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchAny('Tinybird',['C','H']) Result: 1 ## multiSearchAnyCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#multisearchanycaseinsensitive) Like multiSearchAny but ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiSearchAnyCaseInsensitive(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `needle` : Substrings to be searched. Array ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if there was at least one case-insensitive match. - 0, if there was not at least one case-insensitive match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT multiSearchAnyCaseInsensitive('Tinybird',['c','h']) Result: 1 ## multiSearchAnyUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchanyutf8) Like multiSearchAny but assumes `haystack` and the `needle` substrings are UTF-8 encoded strings. *Syntax* * multiSearchAnyUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : UTF-8 substrings to be searched. Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if there was at least one match. - 0, if there was not at least one match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `Tinybird` as a UTF-8 string, check if there are any `C` ('\x43') or `H` ('\x48') letters in the word. Query: SELECT multiSearchAnyUTF8('\x43\x6c\x69\x63\x6b\x48\x6f\x75\x73\x65',['\x43','\x48']) Result: 1 ## multiSearchAnyCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#multisearchanycaseinsensitiveutf8) Like multiSearchAnyUTF8 but ignores case. *Syntax* * multiSearchAnyCaseInsensitiveUTF8(haystack, [needle1, needle2, ..., needleN]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : UTF-8 substrings to be searched. Array ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if there was at least one case-insensitive match. - 0, if there was not at least one case-insensitive match. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Given `Tinybird` as a UTF-8 string, check if there is any letter `h` ( `\x68` ) in the word, ignoring case. Query: SELECT multiSearchAnyCaseInsensitiveUTF8('\x43\x6c\x69\x63\x6b\x48\x6f\x75\x73\x65',['\x68']) Result: 1 ## match [¶](https://www.tinybird.co/docs/about:blank#match) Returns whether string `haystack` matches the regular expression `pattern` in re2 regular expression syntax. Matching is based on UTF-8, e.g. `.` matches the Unicode code point `¥` which is represented in UTF-8 using two bytes. The regular expression must not contain null bytes. If the haystack or the pattern aren't valid UTF-8, then the behavior is undefined. Unlike re2's default behavior, `.` matches line breaks. To disable this, prepend the pattern with `(?-s)`. If you only want to search substrings in a string, you can use functions like or position instead - they work much faster than this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) match(haystack, pattern) Alias: `haystack REGEXP pattern operator` ## multiMatchAny [¶](https://www.tinybird.co/docs/about:blank#multimatchany) Like `match` but returns 1 if at least one of the patterns match and 0 otherwise. If you only want to search multiple substrings in a string, you can use function multiSearchAny instead - it works much faster than this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiMatchAny(haystack, \[pattern1, pattern2, ..., patternn\]) ## multiMatchAnyIndex [¶](https://www.tinybird.co/docs/about:blank#multimatchanyindex) Like `multiMatchAny` but returns any index that matches the haystack. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiMatchAnyIndex(haystack, \[pattern1, pattern2, ..., patternn\]) ## multiMatchAllIndices [¶](https://www.tinybird.co/docs/about:blank#multimatchallindices) Like `multiMatchAny` but returns the array of all indices that match the haystack in any order. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiMatchAllIndices(haystack, \[pattern1, pattern2, ..., patternn\]) ## multiFuzzyMatchAny [¶](https://www.tinybird.co/docs/about:blank#multifuzzymatchany) Like `multiMatchAny` but returns 1 if any pattern matches the haystack within a constant edit distance. This function relies on the experimental feature of hyperscan library, and can be slow for some corner cases. The performance depends on the edit distance value and patterns used, but it's always more expensive compared to a non-fuzzy variants. `multiFuzzyMatch*()` function family don't support UTF-8 regular expressions (it threats them as a sequence of bytes) due to restrictions of hyperscan. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiFuzzyMatchAny(haystack, distance, \[pattern1, pattern2, ..., patternn\]) ## multiFuzzyMatchAnyIndex [¶](https://www.tinybird.co/docs/about:blank#multifuzzymatchanyindex) Like `multiFuzzyMatchAny` but returns any index that matches the haystack within a constant edit distance. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiFuzzyMatchAnyIndex(haystack, distance, \[pattern1, pattern2, ..., patternn\]) ## multiFuzzyMatchAllIndices [¶](https://www.tinybird.co/docs/about:blank#multifuzzymatchallindices) Like `multiFuzzyMatchAny` but returns the array of all indices in any order that match the haystack within a constant edit distance. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) multiFuzzyMatchAllIndices(haystack, distance, \[pattern1, pattern2, ..., patternn\]) ## extract [¶](https://www.tinybird.co/docs/about:blank#extract) Returns the first match of a regular expression in a string. If `haystack` doesn't match the `pattern` regex, an empty string is returned. If the regular expression has capturing groups, the function matches the input string against the first capturing group. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extract(haystack, pattern) *Arguments* * - `haystack` : Input string. String. - `pattern` : Regular expression with re2 regular expression syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The first match of the regular expression in the haystack string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT extract('number: 1, number: 2, number: 3', '\\d+') AS result Result: ┌─result─┐ │ 1 │ └────────┘ ## extractAll [¶](https://www.tinybird.co/docs/about:blank#extractall) Returns an array of all matches of a regular expression in a string. If `haystack` doesn't match the `pattern` regex, an empty string is returned. The behavior with respect to sub-patterns is the same as in function `extract`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extractAll(haystack, pattern) *Arguments* * - `haystack` : Input string. String. - `pattern` : Regular expression with re2 regular expression syntax. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of matches of the regular expression in the haystack string. Array(String). ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT extractAll('number: 1, number: 2, number: 3', '\\d+') AS result Result: ┌─result────────┐ │ ['1','2','3'] │ └───────────────┘ ## extractAllGroupsHorizontal [¶](https://www.tinybird.co/docs/about:blank#extractallgroupshorizontal) Matches all groups of the `haystack` string using the `pattern` regular expression. Returns an array of arrays, where the first array includes all fragments matching the first group, the second array - matching the second group, etc. This function is slower than extractAllGroupsVertical. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extractAllGroupsHorizontal(haystack, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : Input string. String. - `pattern` : Regular expression with re2 regular expression syntax. Must contain groups, each group enclosed in parentheses. If `pattern` contains no groups, an exception is thrown. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of arrays of matches. Array. If `haystack` doesn't match the `pattern` regex, an array of empty arrays is returned. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT extractAllGroupsHorizontal('abc=111, def=222, ghi=333', '("[^"]+"|\\w+)=("[^"]+"|\\w+)') Result: ┌─extractAllGroupsHorizontal('abc=111, def=222, ghi=333', '("[^"]+"|\\w+)=("[^"]+"|\\w+)')─┐ │ [['abc','def','ghi'],['111','222','333']] │ └──────────────────────────────────────────────────────────────────────────────────────────┘ ## extractGroups [¶](https://www.tinybird.co/docs/about:blank#extractgroups) Match all groups of given input string with a given regular expression, returns an array of arrays of matches. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extractGroups(haystack, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : Input string. String. - `pattern` : Regular expression with re2 regular expression syntax. Must contain groups, each group enclosed in parentheses. If `pattern` contains no groups, an exception is thrown. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of arrays of matches. Array. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT extractGroups('hello abc=111 world', '("[^"]+"|\\w+)=("[^"]+"|\\w+)') AS result Result: ┌─result────────┐ │ ['abc','111'] │ └───────────────┘ ## extractAllGroupsVertical [¶](https://www.tinybird.co/docs/about:blank#extractallgroupsvertical) Matches all groups of the `haystack` string using the `pattern` regular expression. Returns an array of arrays, where each array includes matching fragments from every group. Fragments are grouped in order of appearance in the `haystack`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) extractAllGroupsVertical(haystack, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : Input string. String. - `pattern` : Regular expression with re2 regular expression syntax. Must contain groups, each group enclosed in parentheses. If `pattern` contains no groups, an exception is thrown. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Array of arrays of matches. Array. If `haystack` doesn't match the `pattern` regex, an empty array is returned. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT extractAllGroupsVertical('abc=111, def=222, ghi=333', '("[^"]+"|\\w+)=("[^"]+"|\\w+)') Result: ┌─extractAllGroupsVertical('abc=111, def=222, ghi=333', '("[^"]+"|\\w+)=("[^"]+"|\\w+)')─┐ │ [['abc','111'],['def','222'],['ghi','333']] │ └────────────────────────────────────────────────────────────────────────────────────────┘ ## like [¶](https://www.tinybird.co/docs/about:blank#like) Returns whether string `haystack` matches the LIKE expression `pattern`. A LIKE expression can contain normal characters and the following metasymbols: - `%` indicates an arbitrary number of arbitrary characters (including zero characters). - `_` indicates a single arbitrary character. - `\` is for escaping literals `%` , `_` and `\` . Matching is based on UTF-8, e.g. `_` matches the Unicode code point `¥` which is represented in UTF-8 using two bytes. If the haystack or the LIKE expression aren't valid UTF-8, the behavior is undefined. No automatic Unicode normalization is performed, you can use the normalizeUTF8*() functions for that. To match against literal `%`, `_` and `\` (which are LIKE metacharacters), prepend them with a backslash: `\%`, `\_` and `\\`. The backslash loses its special meaning (i.e. is interpreted literally) if it prepends a character different than `%`, `_` or `\`. For LIKE expressions of the form `%needle%` , the function is as fast as the `position` function. All other LIKE expressions are internally converted to a regular expression and executed with a performance similar to function `match`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) like(haystack, pattern) Alias: `haystack LIKE pattern` (operator) ## notLike [¶](https://www.tinybird.co/docs/about:blank#notlike) Like `like` but negates the result. Alias: `haystack NOT LIKE pattern` (operator) ## ilike [¶](https://www.tinybird.co/docs/about:blank#ilike) Like `like` but searches case-insensitively. Alias: `haystack ILIKE pattern` (operator) ## notILike [¶](https://www.tinybird.co/docs/about:blank#notilike) Like `ilike` but negates the result. Alias: `haystack NOT ILIKE pattern` (operator) ## ngramDistance [¶](https://www.tinybird.co/docs/about:blank#ngramdistance) Calculates the 4-gram distance between a `haystack` string and a `needle` string. For this, it counts the symmetric difference between two multisets of 4-grams and normalizes it by the sum of their cardinalities. Returns a Float32 between 0 and 1. The smaller the result is, the more similar the strings are to each other. Functions `ngramDistanceCaseInsensitive`, `ngramDistanceUTF8`, `ngramDistanceCaseInsensitiveUTF8` provide case-insensitive and/or UTF-8 variants of this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramDistance(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First comparison string. String literal - `needle` : Second comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the similarity between the two strings. Float32 ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) This function will throw an exception if constant `needle` or `haystack` arguments are more than 32Kb in size. If any non-constant `haystack` or `needle` arguments are more than 32Kb in size, then the distance is always 1. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) The more similar two strings are to each other, the closer the result will be to 0 (identical). Query: SELECT ngramDistance('Tinybird','Tinybird!') Result: 0.06666667 The less similar two strings are to each, the larger the result will be. Query: SELECT ngramDistance('Tinybird','House') Result: 0.5555556 ## ngramDistanceCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#ngramdistancecaseinsensitive) Provides a case-insensitive variant of ngramDistance. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramDistanceCaseInsensitive(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First comparison string. String literal - `needle` : Second comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the similarity between the two strings. Float32 ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) With ngramDistance differences in case will affect the similarity value: Query: SELECT ngramDistance('Tinybird','tinybird') Result: 0.71428573 With ngramDistanceCaseInsensitive case is ignored so two identical strings differing only in case will now return a low similarity value: Query: SELECT ngramDistanceCaseInsensitive('Tinybird','tinybird') Result: 0 ## ngramDistanceUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramdistanceutf8) Provides a UTF-8 variant of ngramDistance. Assumes that `needle` and `haystack` strings are UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramDistanceUTF8(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First UTF-8 encoded comparison string. String literal - `needle` : Second UTF-8 encoded comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the similarity between the two strings. Float32 ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramDistanceUTF8('abcde','cde') Result: 0.5 ## ngramDistanceCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramdistancecaseinsensitiveutf8) Provides a case-insensitive variant of ngramDistanceUTF8. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramDistanceCaseInsensitiveUTF8(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First UTF-8 encoded comparison string. String literal - `needle` : Second UTF-8 encoded comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the similarity between the two strings. Float32 ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramDistanceCaseInsensitiveUTF8('abcde','CDE') Result: 0.5 ## ngramSearch [¶](https://www.tinybird.co/docs/about:blank#ngramsearch) Like `ngramDistance` but calculates the non-symmetric difference between a `needle` string and a `haystack` string, i.e. the number of n-grams from the needle minus the common number of n-grams normalized by the number of `needle` n-grams. Returns a Float32 between 0 and 1. The bigger the result is, the more likely `needle` is in the `haystack` . This function is useful for fuzzy string search. Also see function `soundex`. Functions `ngramSearchCaseInsensitive`, `ngramSearchUTF8`, `ngramSearchCaseInsensitiveUTF8` provide case-insensitive and/or UTF-8 variants of this function. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramSearch(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First comparison string. String literal - `needle` : Second comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the likelihood of the `needle` being in the `haystack` . Float32 ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramSearch('Hello World','World Hello') Result: 0.5 ## ngramSearchCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#ngramsearchcaseinsensitive) Provides a case-insensitive variant of ngramSearch. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramSearchCaseInsensitive(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First comparison string. String literal - `needle` : Second comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the likelihood of the `needle` being in the `haystack` . Float32 The bigger the result is, the more likely `needle` is in the `haystack`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramSearchCaseInsensitive('Hello World','hello') Result: 1 ## ngramSearchUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramsearchutf8) Provides a UTF-8 variant of ngramSearch in which `needle` and `haystack` are assumed to be UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramSearchUTF8(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First UTF-8 encoded comparison string. String literal - `needle` : Second UTF-8 encoded comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the likelihood of the `needle` being in the `haystack` . Float32 The bigger the result is, the more likely `needle` is in the `haystack`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramSearchUTF8('абвгдеёжз', 'гдеёзд') Result: 0.5 ## ngramSearchCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramsearchcaseinsensitiveutf8) Provides a case-insensitive variant of ngramSearchUTF8 in which `needle` and `haystack`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) ngramSearchCaseInsensitiveUTF8(haystack, needle) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : First UTF-8 encoded comparison string. String literal - `needle` : Second UTF-8 encoded comparison string. String literal ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Value between 0 and 1 representing the likelihood of the `needle` being in the `haystack` . Float32 The bigger the result is, the more likely `needle` is in the `haystack`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT ngramSearchCaseInsensitiveUTF8('абвГДЕёжз', 'АбвгдЕЁжз') Result: 0.57142854 ## countSubstrings [¶](https://www.tinybird.co/docs/about:blank#countsubstrings) Returns how often a substring `needle` occurs in a string `haystack`. Functions `countSubstringsCaseInsensitive` and `countSubstringsCaseInsensitiveUTF8` provide case-insensitive and case-insensitive + UTF-8 variants of this function respectively. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) countSubstrings(haystack, needle[, start_pos]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Substring to be searched. String. - `start_pos` – Position (1-based) in `haystack` at which the search starts. UInt. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The number of occurrences. UInt64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) SELECT countSubstrings('aaaa', 'aa') Result: ┌─countSubstrings('aaaa', 'aa')─┐ │ 2 │ └───────────────────────────────┘ Example with `start_pos` argument: SELECT countSubstrings('abc___abc', 'abc', 4) Result: ┌─countSubstrings('abc___abc', 'abc', 4)─┐ │ 1 │ └────────────────────────────────────────┘ ## countSubstringsCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#countsubstringscaseinsensitive) Returns how often a substring `needle` occurs in a string `haystack` . Ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) countSubstringsCaseInsensitive(haystack, needle[, start_pos]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Substring to be searched. String. - `start_pos` – Position (1-based) in `haystack` at which the search starts. UInt. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The number of occurrences. UInt64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT countSubstringsCaseInsensitive('AAAA', 'aa') Result: ┌─countSubstringsCaseInsensitive('AAAA', 'aa')─┐ │ 2 │ └──────────────────────────────────────────────┘ Example with `start_pos` argument: Query: SELECT countSubstringsCaseInsensitive('abc___ABC___abc', 'abc', 4) Result: ┌─countSubstringsCaseInsensitive('abc___ABC___abc', 'abc', 4)─┐ │ 2 │ └─────────────────────────────────────────────────────────────┘ ## countSubstringsCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#countsubstringscaseinsensitiveutf8) Returns how often a substring `needle` occurs in a string `haystack` . Ignores case and assumes that `haystack` is a UTF8 string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) countSubstringsCaseInsensitiveUTF8(haystack, needle[, start_pos]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : UTF-8 string in which the search is performed. String. - `needle` : Substring to be searched. String. - `start_pos` – Position (1-based) in `haystack` at which the search starts. UInt. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The number of occurrences. UInt64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT countSubstringsCaseInsensitiveUTF8('ложка, кошка, картошка', 'КА') Result: ┌─countSubstringsCaseInsensitiveUTF8('ложка, кошка, картошка', 'КА')─┐ │ 4 │ └────────────────────────────────────────────────────────────────────┘ Example with `start_pos` argument: Query: SELECT countSubstringsCaseInsensitiveUTF8('ложка, кошка, картошка', 'КА', 13) Result: ┌─countSubstringsCaseInsensitiveUTF8('ложка, кошка, картошка', 'КА', 13)─┐ │ 2 │ └────────────────────────────────────────────────────────────────────────┘ ## countMatches [¶](https://www.tinybird.co/docs/about:blank#countmatches) Returns the number of regular expression matches for a `pattern` in a `haystack`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) countMatches(haystack, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : The string to search in. String. - `pattern` : The regular expression with re2 regular expression syntax. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The number of matches. UInt64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) SELECT countMatches('foobar.com', 'o+') Result: ┌─countMatches('foobar.com', 'o+')─┐ │ 2 │ └──────────────────────────────────┘ SELECT countMatches('aaaa', 'aa') Result: ┌─countMatches('aaaa', 'aa')────┐ │ 2 │ └───────────────────────────────┘ ## countMatchesCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#countmatchescaseinsensitive) Returns the number of regular expression matches for a pattern in a haystack like `countMatches` but matching ignores the case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) countMatchesCaseInsensitive(haystack, pattern) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : The string to search in. String. - `pattern` : The regular expression with re2 regular expression syntax. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The number of matches. UInt64. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT countMatchesCaseInsensitive('AAAA', 'aa') Result: ┌─countMatchesCaseInsensitive('AAAA', 'aa')────┐ │ 2 │ └──────────────────────────────────────────────┘ ## regexpExtract [¶](https://www.tinybird.co/docs/about:blank#regexpextract) Extracts the first string in `haystack` that matches the regexp pattern and corresponds to the regex group index. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) regexpExtract(haystack, pattern[, index]) Alias: `REGEXP_EXTRACT(haystack, pattern[, index])`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String, in which regexp pattern will to be matched. String. - `pattern` : String, regexp expression, must be constant. String. - `index` – An integer number greater or equal 0 with default 1. It represents which regex group to extract. UInt or Int. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) `pattern` may contain multiple regexp groups, `index` indicates which regex group to extract. An index of 0 means matching the entire regular expression. String. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) SELECT regexpExtract('100-200', '(\\d+)-(\\d+)', 1), regexpExtract('100-200', '(\\d+)-(\\d+)', 2), regexpExtract('100-200', '(\\d+)-(\\d+)', 0), regexpExtract('100-200', '(\\d+)-(\\d+)') Result: ┌─regexpExtract('100-200', '(\\d+)-(\\d+)', 1)─┬─regexpExtract('100-200', '(\\d+)-(\\d+)', 2)─┬─regexpExtract('100-200', '(\\d+)-(\\d+)', 0)─┬─regexpExtract('100-200', '(\\d+)-(\\d+)')─┐ │ 100 │ 200 │ 100-200 │ 100 │ └──────────────────────────────────────────────┴──────────────────────────────────────────────┴──────────────────────────────────────────────┴───────────────────────────────────────────┘ ## hasSubsequence [¶](https://www.tinybird.co/docs/about:blank#hassubsequence) Returns 1 if `needle` is a subsequence of `haystack` , or 0 otherwise. A subsequence of a string is a sequence that can be derived from the given string by deleting zero or more elements without changing the order of the remaining elements. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasSubsequence(haystack, needle) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Subsequence to be searched. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if needle is a subsequence of haystack, 0 otherwise. UInt8. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT hasSubsequence('garbage', 'arg') Result: ┌─hasSubsequence('garbage', 'arg')─┐ │ 1 │ └──────────────────────────────────┘ ## hasSubsequenceCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#hassubsequencecaseinsensitive) Like hasSubsequence but searches case-insensitively. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasSubsequenceCaseInsensitive(haystack, needle) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. String. - `needle` : Subsequence to be searched. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if needle is a subsequence of haystack, 0 otherwise UInt8. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT hasSubsequenceCaseInsensitive('garbage', 'ARG') Result: ┌─hasSubsequenceCaseInsensitive('garbage', 'ARG')─┐ │ 1 │ └─────────────────────────────────────────────────┘ ## hasSubsequenceUTF8 [¶](https://www.tinybird.co/docs/about:blank#hassubsequenceutf8) Like hasSubsequence but assumes `haystack` and `needle` are UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasSubsequenceUTF8(haystack, needle) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. UTF-8 encoded String. - `needle` : Subsequence to be searched. UTF-8 encoded String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if needle is a subsequence of haystack, 0, otherwise. UInt8. ## hasSubsequenceCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#hassubsequencecaseinsensitiveutf8) Like hasSubsequenceUTF8 but searches case-insensitively. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasSubsequenceCaseInsensitiveUTF8(haystack, needle) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `haystack` : String in which the search is performed. UTF-8 encoded String. - `needle` : Subsequence to be searched. UTF-8 encoded String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if needle is a subsequence of haystack, 0 otherwise. UInt8. ## hasToken [¶](https://www.tinybird.co/docs/about:blank#hastoken) Returns 1 if a given token is present in a haystack, or 0 otherwise. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasToken(haystack, token) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `token` : Maximal length substring between two non alphanumeric ASCII characters (or boundaries of haystack). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if the token is present in the haystack, 0 otherwise. UInt8. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Token must be a constant string. Supported by tokenbf_v1 index specialization. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT hasToken('Hello World','Hello') 1 ## hasTokenOrNull [¶](https://www.tinybird.co/docs/about:blank#hastokenornull) Returns 1 if a given token is present, 0 if not present, and null if the token is ill-formed. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasTokenOrNull(haystack, token) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `token` : Maximal length substring between two non alphanumeric ASCII characters (or boundaries of haystack). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if the token is present in the haystack, 0 if it's not present, and null if the token is ill formed. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Token must be a constant string. Supported by tokenbf_v1 index specialization. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Where `hasToken` would throw an error for an ill-formed token, `hasTokenOrNull` returns `null` for an ill-formed token. Query: SELECT hasTokenOrNull('Hello World','Hello,World') null ## hasTokenCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#hastokencaseinsensitive) Returns 1 if a given token is present in a haystack, 0 otherwise. Ignores case. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasTokenCaseInsensitive(haystack, token) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `token` : Maximal length substring between two non alphanumeric ASCII characters (or boundaries of haystack). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if the token is present in the haystack, 0 otherwise. UInt8. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Token must be a constant string. Supported by tokenbf_v1 index specialization. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT hasTokenCaseInsensitive('Hello World','hello') 1 ## hasTokenCaseInsensitiveOrNull [¶](https://www.tinybird.co/docs/about:blank#hastokencaseinsensitiveornull) Returns 1 if a given token is present in a haystack, 0 otherwise. Ignores case and returns null if the token is ill-formed. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) hasTokenCaseInsensitiveOrNull(haystack, token) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `haystack` : String in which the search is performed. String. - `token` : Maximal length substring between two non alphanumeric ASCII characters (or boundaries of haystack). ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - 1, if the token is present in the haystack, 0 if the token isn't present, otherwise `null` if the token is ill-formed. UInt8. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Token must be a constant string. Supported by tokenbf_v1 index specialization. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Where `hasTokenCaseInsensitive` would throw an error for an ill-formed token, `hasTokenCaseInsensitiveOrNull` returns `null` for an ill-formed token. Query: SELECT hasTokenCaseInsensitiveOrNull('Hello World','hello,world') null --- URL: https://www.tinybird.co/docs/sql-reference/functions/string-replace-functions Last update: 2025-01-24T09:36:45.000Z Content: --- title: "Replace functions · Tinybird Docs" theme-color: "#171612" description: "Functions for replacing in strings." --- # Functions for replacing in strings [¶](https://www.tinybird.co/docs/about:blank#functions-for-replacing-in-strings) General strings functions and functions for searching in strings are described separately. See [String search functions](https://www.tinybird.co/docs/string-search-functions). ## overlay [¶](https://www.tinybird.co/docs/about:blank#overlay) Replace part of the string `input` with another string `replace` , starting at the 1-based index `offset`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) overlay(s, replace, offset[, length]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : A string type String. - `replace` : A string type String. - `offset` : An integer type Int (1-based). If `offset` is negative, it's counted from the end of the string `s` . - `length` : Optional. An integer type Int. `length` specifies the length of the snippet within the input string `s` to be replaced. If `length` isn't specified, the number of bytes removed from `s` equals the length of `replace` ; otherwise `length` bytes are removed. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT overlay('My father is from Mexico.', 'mother', 4) AS res Result: ┌─res──────────────────────┐ │ My mother is from Mexico.│ └──────────────────────────┘ SELECT overlay('My father is from Mexico.', 'dad', 4, 6) AS res Result: ┌─res───────────────────┐ │ My dad is from Mexico.│ └───────────────────────┘ ## overlayUTF8 [¶](https://www.tinybird.co/docs/about:blank#overlayutf8) Replace part of the string `input` with another string `replace` , starting at the 1-based index `offset`. Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) overlayUTF8(s, replace, offset[, length]) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : A string type String. - `replace` : A string type String. - `offset` : An integer type Int (1-based). If `offset` is negative, it's counted from the end of the input string `s` . - `length` : Optional. An integer type Int. `length` specifies the length of the snippet within the input string `s` to be replaced. If `length` isn't specified, the number of characters removed from `s` equals the length of `replace` ; otherwise `length` characters are removed. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT overlay('Mein Vater ist aus Österreich.', 'der Türkei', 20) AS res Result: ┌─res───────────────────────────┐ │ Mein Vater ist aus der Türkei.│ └───────────────────────────────┘ ## replaceOne [¶](https://www.tinybird.co/docs/about:blank#replaceone) Replaces the first occurrence of the substring `pattern` in `haystack` by the `replacement` string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) replaceOne(haystack, pattern, replacement) ## replaceAll [¶](https://www.tinybird.co/docs/about:blank#replaceall) Replaces all occurrences of the substring `pattern` in `haystack` by the `replacement` string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) replaceAll(haystack, pattern, replacement) Alias: `replace`. ## replaceRegexpOne [¶](https://www.tinybird.co/docs/about:blank#replaceregexpone) Replaces the first occurrence of the substring matching the regular expression `pattern` (in re2 syntax) in `haystack` by the `replacement` string. `replacement` can contain substitutions `\0-\9`. Substitutions `\1-\9` correspond to the 1st to 9th capturing group (submatch), substitution `\0` corresponds to the entire match. To use a verbatim `\` character in the `pattern` or `replacement` strings, escape it using `\`. Also keep in mind that string literals require extra escaping. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) replaceRegexpOne(haystack, pattern, replacement) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Converting ISO dates to American format: SELECT DISTINCT EventDate, replaceRegexpOne(toString(EventDate), '(\\d{4})-(\\d{2})-(\\d{2})', '\\2/\\3/\\1') AS res FROM test.hits LIMIT 7 FORMAT TabSeparated Result: 2014-03-17 03/17/2014 2014-03-18 03/18/2014 2014-03-19 03/19/2014 2014-03-20 03/20/2014 2014-03-21 03/21/2014 2014-03-22 03/22/2014 2014-03-23 03/23/2014 Copying a string ten times: SELECT replaceRegexpOne('Hello, World!', '.*', '\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0') AS res Result: ┌─res────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Hello, World!Hello, World!Hello, World!Hello, World!Hello, World!Hello, World!Hello, World!Hello, World!Hello, World!Hello, World! │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ## replaceRegexpAll [¶](https://www.tinybird.co/docs/about:blank#replaceregexpall) Like `replaceRegexpOne` but replaces all occurrences of the pattern. Alias: `REGEXP_REPLACE`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT replaceRegexpAll('Hello, World!', '.', '\\0\\0') AS res Result: ┌─res────────────────────────┐ │ HHeelllloo,, WWoorrlldd!! │ └────────────────────────────┘ As an exception, if a regular expression worked on an empty substring, the replacement isn't made more than once, e.g.: SELECT replaceRegexpAll('Hello, World!', '^', 'here: ') AS res Result: ┌─res─────────────────┐ │ here: Hello, World! │ └─────────────────────┘ ## regexpQuoteMeta [¶](https://www.tinybird.co/docs/about:blank#regexpquotemeta) Adds a backslash before these characters with special meaning in regular expressions: `\0`, `\\`, `|`, `(`, `)`, `^`, `$`, `.`, `[`, `]`, `?`, `*`, `+`, `{`, `:`, `-`. This implementation slightly differs from re2::RE2::QuoteMeta. It escapes zero byte as `\0` instead of `\x00` and it escapes only required characters. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) regexpQuoteMeta(s) ## format [¶](https://www.tinybird.co/docs/about:blank#format) Format the `pattern` string with the values (strings, integers, etc.) listed in the arguments, similar to formatting in Python. The pattern string can contain replacement fields surrounded by curly braces `{}` . Anything not contained in braces is considered literal text and copied verbatim into the output. Literal brace character can be escaped by two braces: `{{ '{{' }}` and `{{ '}}' }}` . Field names can be numbers (starting from zero) or empty (then they are implicitly given monotonically increasing numbers). ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) format(pattern, s0, s1, ...) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT format('{1} {0} {1}', 'World', 'Hello') ┌─format('{1} {0} {1}', 'World', 'Hello')─┐ │ Hello World Hello │ └─────────────────────────────────────────┘ With implicit numbers: SELECT format('{} {}', 'Hello', 'World') ┌─format('{} {}', 'Hello', 'World')─┐ │ Hello World │ └───────────────────────────────────┘ ## translate [¶](https://www.tinybird.co/docs/about:blank#translate) Replaces characters in the string `s` using a one-to-one character mapping defined by `from` and `to` strings. `from` and `to` must be constant ASCII strings of the same size. Non-ASCII characters in the original string aren't modified. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) translate(s, from, to) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT translate('Hello, World!', 'delor', 'DELOR') AS res Result: ┌─res───────────┐ │ HELLO, WORLD! │ └───────────────┘ ## translateUTF8 [¶](https://www.tinybird.co/docs/about:blank#translateutf8) Like translate but assumes `s`, `from` and `to` are UTF-8 encoded strings. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) translateUTF8(s, from, to) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : A string type String. - `from` : A string type String. - `to` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT translateUTF8('Münchener Straße', 'üß', 'us') AS res ┌─res──────────────┐ │ Munchener Strase │ └──────────────────┘ ## printf [¶](https://www.tinybird.co/docs/about:blank#printf) The `printf` function formats the given string with the values (strings, integers, floating-points etc.) listed in the arguments, similar to printf function in C++. The format string can contain format specifiers starting with `%` character. Anything not contained in `%` and the following format specifier is considered literal text and copied verbatim into the output. Literal `%` character can be escaped by `%%`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) printf(format, arg1, arg2, ...) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: select printf('%%%s %s %d', 'Hello', 'World', 2024) ┌─printf('%%%s %s %d', 'Hello', 'World', 2024)─┐ │ %Hello World 2024 │ └──────────────────────────────────────────────┘ --- URL: https://www.tinybird.co/docs/sql-reference/functions/string-functions Last update: 2025-01-23T08:42:52.000Z Content: --- title: "Strings functions · Tinybird Docs" theme-color: "#171612" description: "Functions for working with strings." --- # Functions for Working with Strings [¶](https://www.tinybird.co/docs/about:blank#functions-for-working-with-strings) Functions for searching in strings and for replacing in strings are described separately. See [String search functions](https://www.tinybird.co/docs/string-search-functions) and [String replace functions](https://www.tinybird.co/docs/string-replace-functions) and [String replace functions](https://www.tinybird.co/docs/string-replace-functions). ## empty [¶](https://www.tinybird.co/docs/about:blank#empty) Checks whether the input string is empty. A string is considered non-empty if it contains at least one byte, even if this byte is a space or the null byte. The function is also available for arrays and UUIDs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) empty(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Input value. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `1` for an empty string or `0` for a non-empty string. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT empty('') Result: ┌─empty('')─┐ │ 1 │ └───────────┘ ## notEmpty [¶](https://www.tinybird.co/docs/about:blank#notempty) Checks whether the input string is non-empty. A string is considered non-empty if it contains at least one byte, even if this byte is a space or the null byte. The function is also available for arrays and UUIDs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) notEmpty(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Input value. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `1` for a non-empty string or `0` for an empty string string. UInt8. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT notEmpty('text') Result: ┌─notEmpty('text')─┐ │ 1 │ └──────────────────┘ ## length [¶](https://www.tinybird.co/docs/about:blank#length) Returns the length of a string in bytes rather than in characters or Unicode code points. The function also works for arrays. Alias: `OCTET_LENGTH` ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) length(s) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : An input string or array. String/Array. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Length of the string or array `s` in bytes. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT length('Hello, world!') Result: ┌─length('Hello, world!')─┐ │ 13 │ └─────────────────────────┘ Query: SELECT length([1, 2, 3, 4]) Result: ┌─length([1, 2, 3, 4])─┐ │ 4 │ └──────────────────────┘ ## lengthUTF8 [¶](https://www.tinybird.co/docs/about:blank#lengthutf8) Returns the length of a string in Unicode code points rather than in bytes or characters. It assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. Aliases: - `CHAR_LENGTH` - `CHARACTER_LENGTH` ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) lengthUTF8(s) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : String containing valid UTF-8 encoded text. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Length of the string `s` in Unicode code points. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT lengthUTF8('Здравствуй, мир!') Result: ┌─lengthUTF8('Здравствуй, мир!')─┐ │ 16 │ └────────────────────────────────┘ ## left [¶](https://www.tinybird.co/docs/about:blank#left) Returns a substring of string `s` with a specified `offset` starting from the left. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) left(s, offset) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : The string to calculate a substring from. String or FixedString. - `offset` : The number of bytes of the offset. UInt*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - For positive `offset` : A substring of `s` with `offset` many bytes, starting from the left of the string. - For negative `offset` : A substring of `s` with `length(s) - |offset|` bytes, starting from the left of the string. - An empty string if `length` is 0. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT left('Hello', 3) Result: Hel Query: SELECT left('Hello', -3) Result: He ## leftUTF8 [¶](https://www.tinybird.co/docs/about:blank#leftutf8) Returns a substring of a UTF-8 encoded string `s` with a specified `offset` starting from the left. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) leftUTF8(s, offset) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : The UTF-8 encoded string to calculate a substring from. String or FixedString. - `offset` : The number of bytes of the offset. UInt*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - For positive `offset` : A substring of `s` with `offset` many bytes, starting from the left of the string. - For negative `offset` : A substring of `s` with `length(s) - |offset|` bytes, starting from the left of the string. - An empty string if `length` is 0. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT leftUTF8('Привет', 4) Result: Прив Query: SELECT leftUTF8('Привет', -4) Result: Пр ## leftPad [¶](https://www.tinybird.co/docs/about:blank#leftpad) Pads a string from the left with spaces or with a specified string (multiple times, if needed) until the resulting string reaches the specified `length`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) leftPad(string, length[, pad_string]) Alias: `LPAD` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : Input string that should be padded. String. - `length` : The length of the resulting string. UInt or Int. If the value is smaller than the input string length, then the input string is shortened to `length` characters. - `pad_string` : The string to pad the input string with. String. Optional. If not specified, then the input string is padded with spaces. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A left-padded string of the given length. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT leftPad('abc', 7, '*'), leftPad('def', 7) Result: ┌─leftPad('abc', 7, '*')─┬─leftPad('def', 7)─┐ │ ****abc │ def │ └────────────────────────┴───────────────────┘ ## leftPadUTF8 [¶](https://www.tinybird.co/docs/about:blank#leftpadutf8) Pads the string from the left with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Unlike leftPad which measures the string length in bytes, the string length is measured in code points. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) leftPadUTF8(string, length[, pad_string]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : Input string that should be padded. String. - `length` : The length of the resulting string. UInt or Int. If the value is smaller than the input string length, then the input string is shortened to `length` characters. - `pad_string` : The string to pad the input string with. String. Optional. If not specified, then the input string is padded with spaces. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A left-padded string of the given length. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT leftPadUTF8('абвг', 7, '*'), leftPadUTF8('дежз', 7) Result: ┌─leftPadUTF8('абвг', 7, '*')─┬─leftPadUTF8('дежз', 7)─┐ │ ***абвг │ дежз │ └─────────────────────────────┴────────────────────────┘ ## right [¶](https://www.tinybird.co/docs/about:blank#right) Returns a substring of string `s` with a specified `offset` starting from the right. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) right(s, offset) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : The string to calculate a substring from. String or FixedString. - `offset` : The number of bytes of the offset. UInt*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - For positive `offset` : A substring of `s` with `offset` many bytes, starting from the right of the string. - For negative `offset` : A substring of `s` with `length(s) - |offset|` bytes, starting from the right of the string. - An empty string if `length` is 0. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT right('Hello', 3) Result: llo Query: SELECT right('Hello', -3) Result: lo ## rightUTF8 [¶](https://www.tinybird.co/docs/about:blank#rightutf8) Returns a substring of UTF-8 encoded string `s` with a specified `offset` starting from the right. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) rightUTF8(s, offset) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `s` : The UTF-8 encoded string to calculate a substring from. String or FixedString. - `offset` : The number of bytes of the offset. UInt*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - For positive `offset` : A substring of `s` with `offset` many bytes, starting from the right of the string. - For negative `offset` : A substring of `s` with `length(s) - |offset|` bytes, starting from the right of the string. - An empty string if `length` is 0. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT rightUTF8('Привет', 4) Result: ивет Query: SELECT rightUTF8('Привет', -4) Result: ет ## rightPad [¶](https://www.tinybird.co/docs/about:blank#rightpad) Pads a string from the right with spaces or with a specified string (multiple times, if needed) until the resulting string reaches the specified `length`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) rightPad(string, length[, pad_string]) Alias: `RPAD` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : Input string that should be padded. String. - `length` : The length of the resulting string. UInt or Int. If the value is smaller than the input string length, then the input string is shortened to `length` characters. - `pad_string` : The string to pad the input string with. String. Optional. If not specified, then the input string is padded with spaces. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A left-padded string of the given length. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT rightPad('abc', 7, '*'), rightPad('abc', 7) Result: ┌─rightPad('abc', 7, '*')─┬─rightPad('abc', 7)─┐ │ abc**** │ abc │ └─────────────────────────┴────────────────────┘ ## rightPadUTF8 [¶](https://www.tinybird.co/docs/about:blank#rightpadutf8) Pads the string from the right with spaces or a specified string (multiple times, if needed) until the resulting string reaches the given length. Unlike rightPad which measures the string length in bytes, the string length is measured in code points. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) rightPadUTF8(string, length[, pad_string]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `string` : Input string that should be padded. String. - `length` : The length of the resulting string. UInt or Int. If the value is smaller than the input string length, then the input string is shortened to `length` characters. - `pad_string` : The string to pad the input string with. String. Optional. If not specified, then the input string is padded with spaces. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A right-padded string of the given length. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT rightPadUTF8('абвг', 7, '*'), rightPadUTF8('абвг', 7) Result: ┌─rightPadUTF8('абвг', 7, '*')─┬─rightPadUTF8('абвг', 7)─┐ │ абвг*** │ абвг │ └──────────────────────────────┴─────────────────────────┘ ## lower [¶](https://www.tinybird.co/docs/about:blank#lower) Converts the ASCII Latin symbols in a string to lowercase. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) lower(input) Alias: `lcase` ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT lower('TINYBIRD') ┌─lower('TINYBIRD')─┐ │ tinybird │ └───────────────────┘ ## upper [¶](https://www.tinybird.co/docs/about:blank#upper) Converts the ASCII Latin symbols in a string to uppercase. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) upper(input) Alias: `ucase` ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT upper('tinybird') ┌─upper('tinybird')─┐ │ TINYBIRD │ └────────────────────┘ ## lowerUTF8 [¶](https://www.tinybird.co/docs/about:blank#lowerutf8) Converts a string to lowercase, assuming that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. Does not detect the language, e.g. for Turkish the result might not be exactly correct (i/İ vs. i/I). If the length of the UTF-8 byte sequence is different for upper and lower case of a code point (such as `ẞ` and `ß` ), the result may be incorrect for this code point. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) lowerUTF8(input) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT lowerUTF8('MÜNCHEN') as Lowerutf8 Result: ┌─Lowerutf8─┐ │ münchen │ └───────────┘ ## upperUTF8 [¶](https://www.tinybird.co/docs/about:blank#upperutf8) Converts a string to uppercase, assuming that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. Does not detect the language, e.g. for Turkish the result might not be exactly correct (i/İ vs. i/I). If the length of the UTF-8 byte sequence is different for upper and lower case of a code point (such as `ẞ` and `ß` ), the result may be incorrect for this code point. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) upperUTF8(input) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A String data type value. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT upperUTF8('München') as Upperutf8 Result: ┌─Upperutf8─┐ │ MÜNCHEN │ └───────────┘ ## isValidUTF8 [¶](https://www.tinybird.co/docs/about:blank#isvalidutf8) Returns 1, if the set of bytes constitutes valid UTF-8-encoded text, otherwise 0. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) isValidUTF8(input) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `input` : A string type String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Returns `1` , if the set of bytes constitutes valid UTF-8-encoded text, otherwise `0` . Query: SELECT isValidUTF8('\xc3\xb1') AS valid, isValidUTF8('\xc3\x28') AS invalid Result: ┌─valid─┬─invalid─┐ │ 1 │ 0 │ └───────┴─────────┘ ## toValidUTF8 [¶](https://www.tinybird.co/docs/about:blank#tovalidutf8) Replaces invalid UTF-8 characters by the `�` (U+FFFD) character. All running in a row invalid characters are collapsed into the one replacement character. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) toValidUTF8(input_string) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `input_string` : Any set of bytes represented as the String data type object. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A valid UTF-8 string. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT toValidUTF8('\x61\xF0\x80\x80\x80b') ┌─toValidUTF8('a����b')─┐ │ a�b │ └───────────────────────┘ ## repeat [¶](https://www.tinybird.co/docs/about:blank#repeat) Concatenates a string as many times with itself as specified. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) repeat(s, n) Alias: `REPEAT` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `s` : The string to repeat. String. - `n` : The number of times to repeat the string. UInt* or Int*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A string containing string `s` repeated `n` times. If `n` <= 0, the function returns the empty string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT repeat('abc', 10) Result: ┌─repeat('abc', 10)──────────────┐ │ abcabcabcabcabcabcabcabcabcabc │ └────────────────────────────────┘ ## space [¶](https://www.tinybird.co/docs/about:blank#space) Concatenates a space ( `` ) as many times with itself as specified. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) space(n) Alias: `SPACE`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `n` : The number of times to repeat the space. UInt* or Int*. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) The string containing string `` repeated `n` times. If `n` <= 0, the function returns the empty string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT space(3) Result: ┌─space(3) ────┐ │ │ └──────────────┘ ## reverse [¶](https://www.tinybird.co/docs/about:blank#reverse) Reverses the sequence of bytes in a string. ## reverseUTF8 [¶](https://www.tinybird.co/docs/about:blank#reverseutf8) Reverses a sequence of Unicode code points in a string. Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ## concat [¶](https://www.tinybird.co/docs/about:blank#concat) Concatenates the given arguments. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) concat(s1, s2, ...) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) Values of arbitrary type. Arguments which aren't of types String or FixedString are converted to strings using their default serialization. As this decreases performance, it's not recommended to use non-String/FixedString arguments. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) The String created by concatenating the arguments. If any of arguments is `NULL` , the function returns `NULL`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT concat('Hello, ', 'World!') Result: ┌─concat('Hello, ', 'World!')─┐ │ Hello, World! │ └─────────────────────────────┘ Query: SELECT concat(42, 144) Result: ┌─concat(42, 144)─┐ │ 42144 │ └─────────────────┘ ## concatAssumeInjective [¶](https://www.tinybird.co/docs/about:blank#concatassumeinjective) Like concat but assumes that `concat(s1, s2, ...) → sn` is injective. Can be used for optimization of GROUP BY. A function is called injective if it returns for different arguments different results. In other words: different arguments never produce identical result. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) concatAssumeInjective(s1, s2, ...) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) Values of type String or FixedString. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) The String created by concatenating the arguments. If any of argument values is `NULL` , the function returns `NULL`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Input table: ##### key_val.datasource SCHEMA > key1 String, key2 String, value UInt32 ENGINE "MergeTree" tb push datasources/key_val.datasource echo "\ 'Hello, ','World',1\n\ 'Hello, ','World',2\n\ 'Hello, ','World!',3\n\ 'Hello',', World!',2\ " > key_val.csv tb datasource append key_val key_val.csv tb sql "select * from key_val" ------------------------------ | key1 | key2 | value | ------------------------------ | Hello | , World! | 2 | | Hello, | World | 1 | | Hello, | World | 2 | | Hello, | World! | 3 | ------------------------------ Query: SELECT concatAssumeInjective(key1, key2), sum(value) FROM key_val GROUP BY concatAssumeInjective(key1, key2) Result: -------------------------------------------------- | concatAssumeInjective(key1, key2) | sum(value) | -------------------------------------------------- | Hello, World! | 3 | | Hello, World! | 2 | | Hello, World | 3 | -------------------------------------------------- ## concatWithSeparator [¶](https://www.tinybird.co/docs/about:blank#concatwithseparator) Concatenates the given strings with a given separator. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) concatWithSeparator(sep, expr1, expr2, expr3...) Alias: `concat_ws` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - sep: separator. Const String or FixedString. - exprN: expression to be concatenated. Arguments which aren't of types String or FixedString are converted to strings using their default serialization. As this decreases performance, it's not recommended to use non-String/FixedString arguments. ### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values) The String created by concatenating the arguments. If any of the argument values is `NULL` , the function returns `NULL`. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT concatWithSeparator('a', '1', '2', '3', '4') Result: ┌─concatWithSeparator('a', '1', '2', '3', '4')─┐ │ 1a2a3a4 │ └──────────────────────────────────────────────┘ ## concatWithSeparatorAssumeInjective [¶](https://www.tinybird.co/docs/about:blank#concatwithseparatorassumeinjective) Like `concatWithSeparator` but assumes that `concatWithSeparator(sep, expr1, expr2, expr3...) → result` is injective. Can be used for optimization of GROUP BY. A function is called injective if it returns for different arguments different results. In other words: different arguments never produce identical result. ## substring [¶](https://www.tinybird.co/docs/about:blank#substring) Returns the substring of a string `s` which starts at the specified byte index `offset` . Byte counting starts from 1. If `offset` is 0, an empty string is returned. If `offset` is negative, the substring starts `pos` characters from the end of the string, rather than from the beginning. An optional argument `length` specifies the maximum number of bytes the returned substring may have. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) substring(s, offset[, length]) Aliases: - `substr` - `mid` - `byteSlice` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `s` : The string to calculate a substring from. String, FixedString or Enum - `offset` : The starting position of the substring in `s` . (U)Int*. - `length` : The maximum length of the substring. (U)Int*. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A substring of `s` with `length` many bytes, starting at index `offset` . String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT 'database' AS db, substr(db, 5), substr(db, 5, 1) Result: ┌─db───────┬─substring('database', 5)─┬─substring('database', 5, 1)─┐ │ database │ base │ b │ └──────────┴──────────────────────────┴─────────────────────────────┘ ## substringUTF8 [¶](https://www.tinybird.co/docs/about:blank#substringutf8) Returns the substring of a string `s` which starts at the specified byte index `offset` for Unicode code points. Byte counting starts from `1` . If `offset` is `0` , an empty string is returned. If `offset` is negative, the substring starts `pos` characters from the end of the string, rather than from the beginning. An optional argument `length` specifies the maximum number of bytes the returned substring may have. Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) substringUTF8(s, offset[, length]) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `s` : The string to calculate a substring from. String, FixedString or Enum - `offset` : The starting position of the substring in `s` . (U)Int*. - `length` : The maximum length of the substring. (U)Int*. Optional. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A substring of `s` with `length` many bytes, starting at index `offset`. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT 'Täglich grüßt das Murmeltier.' AS str, substringUTF8(str, 9), substringUTF8(str, 9, 5) Täglich grüßt das Murmeltier. grüßt das Murmeltier. grüßt ## substringIndex [¶](https://www.tinybird.co/docs/about:blank#substringindex) Returns the substring of `s` before `count` occurrences of the delimiter `delim` , as in Spark or MySQL. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) substringIndex(s, delim, count) Alias: `SUBSTRING_INDEX` ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - s: The string to extract substring from. String. - delim: The character to split. String. - count: The number of occurrences of the delimiter to count before extracting the substring. If count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. UInt or Int ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT substringIndex('www.tinybird.co', '.', 2) Result: ┌─substringIndex('www.tinybird.co', '.', 2)─┐ │ www.tinybird │ └───────────────────────────────────────────┘ ## substringIndexUTF8 [¶](https://www.tinybird.co/docs/about:blank#substringindexutf8) Returns the substring of `s` before `count` occurrences of the delimiter `delim` , specifically for Unicode code points. Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) substringIndexUTF8(s, delim, count) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `s` : The string to extract substring from. String. - `delim` : The character to split. String. - `count` : The number of occurrences of the delimiter to count before extracting the substring. If count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. UInt or Int ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A substring String of `s` before `count` occurrences of `delim`. ### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details) Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT substringIndexUTF8('www.straßen-in-europa.de', '.', 2) www.straßen-in-europa ## appendTrailingCharIfAbsent [¶](https://www.tinybird.co/docs/about:blank#appendtrailingcharifabsent) Appends character `c` to string `s` if `s` is non-empty and doesn't end with character `c`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) appendTrailingCharIfAbsent(s, c) ## convertCharset [¶](https://www.tinybird.co/docs/about:blank#convertcharset) Returns string `s` converted from the encoding `from` to encoding `to`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) convertCharset(s, from, to) ## base58Encode [¶](https://www.tinybird.co/docs/about:blank#base58encode) Encodes a string using Base58 in the "Bitcoin" alphabet. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base58Encode(plaintext) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `plaintext` : String column or constant. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the encoded value of the argument. String or FixedString. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base58Encode('Encoded') Result: ┌─base58Encode('Encoded')─┐ │ 3dc8KtHrwM │ └─────────────────────────┘ ## base58Decode [¶](https://www.tinybird.co/docs/about:blank#base58decode) Accepts a string and decodes it using Base58 encoding scheme using "Bitcoin" alphabet. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base58Decode(encoded) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `encoded` : String or FixedString. If the string isn't a valid Base58-encoded value, an exception is thrown. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base58Decode('3dc8KtHrwM') Result: ┌─base58Decode('3dc8KtHrwM')─┐ │ Encoded │ └────────────────────────────┘ ## tryBase58Decode [¶](https://www.tinybird.co/docs/about:blank#trybase58decode) Like `base58Decode` but returns an empty string in case of error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tryBase58Decode(encoded) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `encoded` : String or FixedString. If the string isn't a valid Base58-encoded value, returns an empty string in case of error. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tryBase58Decode('3dc8KtHrwM') as res, tryBase58Decode('invalid') as res_invalid ┌─res─────┬─res_invalid─┐ │ Encoded │ │ └─────────┴─────────────┘ ## base64Encode [¶](https://www.tinybird.co/docs/about:blank#base64encode) Encodes a String or FixedString as base64, according to RFC 4648. Alias: `TO_BASE64`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base64Encode(plaintext) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `plaintext` : String column or constant. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the encoded value of the argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base64Encode('tinybird') Result: ┌─base64Encode('tinybird')--─┐ │ Y2xpY2tob3VzZQ== │ └────────────────────────────┘ ## base64URLEncode [¶](https://www.tinybird.co/docs/about:blank#base64urlencode) Encodes an URL (String or FixedString) as base64 with URL-specific modifications, according to RFC 4648. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base64URLEncode(url) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `url` : String column or constant. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the encoded value of the argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base64URLEncode('https://tinybird.com') Result: ┌─base64URLEncode('https://tinybird.com')--─┐ │ aHR0cHM6Ly90aW55YmlyZC5jb20 │ └───────────────────────────────────────────┘ ## base64Decode [¶](https://www.tinybird.co/docs/about:blank#base64decode) Accepts a String and decodes it from base64, according to RFC 4648. Throws an exception in case of an error. Alias: `FROM_BASE64`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base64Decode(encoded) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `encoded` : String column or constant. If the string isn't a valid Base64-encoded value, an exception is thrown. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base64Decode('Y2xpY2tob3VzZQ==') Result: ┌─base64Decode('Y2xpY2tob3VzZQ==')─┐ │ tinybird │ └──────────────────────────────────┘ ## base64URLDecode [¶](https://www.tinybird.co/docs/about:blank#base64urldecode) Accepts a base64-encoded URL and decodes it from base64 with URL-specific modifications, according to RFC 4648. Throws an exception in case of an error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) base64URLDecode(encodedUrl) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `encodedURL` : String column or constant. If the string isn't a valid Base64-encoded value with URL-specific modifications, an exception is thrown. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT base64URLDecode('aHR0cDovL2NsaWNraG91c2UuY29t') Result: ┌─base64URLDecode('aHR0cHM6Ly90aW55YmlyZC5jb20')─┐ │ https://tinybird.co │ └────────────────────────────────────────────────┘ ## tryBase64Decode [¶](https://www.tinybird.co/docs/about:blank#trybase64decode) Like `base64Decode` but returns an empty string in case of error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tryBase64Decode(encoded) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `encoded` : String column or constant. If the string isn't a valid Base64-encoded value, returns an empty string. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tryBase64Decode('RW5jb2RlZA==') as res, tryBase64Decode('invalid') as res_invalid ┌─res────────┬─res_invalid─┐ │ tinybird │ │ └────────────┴─────────────┘ ## tryBase64URLDecode [¶](https://www.tinybird.co/docs/about:blank#trybase64urldecode) Like `base64URLDecode` but returns an empty string in case of error. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) tryBase64URLDecode(encodedUrl) ### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters) - `encodedURL` : String column or constant. If the string isn't a valid Base64-encoded value with URL-specific modifications, returns an empty string. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - A string containing the decoded value of the argument. ### Examples [¶](https://www.tinybird.co/docs/about:blank#examples) Query: SELECT tryBase64URLDecode('aHR0cHM6Ly90aW55YmlyZC5jb20') as res, tryBase64Decode('aHR0cHM6Ly9jbGlja') as res_invalid ┌─res────────────────────┬─res_invalid─┐ │ https://tinybird.com │ │ └────────────────────────┴─────────────┘ ## endsWith [¶](https://www.tinybird.co/docs/about:blank#endswith) Returns whether string `str` ends with `suffix`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) endsWith(str, suffix) ## endsWithUTF8 [¶](https://www.tinybird.co/docs/about:blank#endswithutf8) Returns whether string `str` ends with `suffix` , the difference between `endsWithUTF8` and `endsWith` is that `endsWithUTF8` match `str` and `suffix` by UTF-8 characters. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) endsWithUTF8(str, suffix) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT endsWithUTF8('中国', '\xbd'), endsWith('中国', '\xbd') Result: ┌─endsWithUTF8('中国', '½')─┬─endsWith('中国', '½')─┐ │ 0 │ 1 │ └──────────────────────────┴──────────────────────┘ ## startsWith [¶](https://www.tinybird.co/docs/about:blank#startswith) Returns whether string `str` starts with `prefix`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) startsWith(str, prefix) ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT startsWith('Spider-Man', 'Spi') ## startsWithUTF8 [¶](https://www.tinybird.co/docs/about:blank#startswithutf8) Returns whether string `str` starts with `prefix` , the difference between `startsWithUTF8` and `startsWith` is that `startsWithUTF8` match `str` and `suffix` by UTF-8 characters. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT startsWithUTF8('中国', '\xe4'), startsWith('中国', '\xe4') Result: ┌─startsWithUTF8('中国', '⥩─┬─startsWith('中国', '⥩─┐ │ 0 │ 1 │ └────────────────────────────┴────────────────────────┘ ## trim [¶](https://www.tinybird.co/docs/about:blank#trim) Removes the specified characters from the start or end of a string. If not specified otherwise, the function removes whitespace (ASCII-character 32). ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) trim([[LEADING|TRAILING|BOTH] trim_character FROM] input_string) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `trim_character` : Specified characters for trim. String. - `input_string` : String for trim. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A string without leading and/or trailing specified characters. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT trim(BOTH ' ()' FROM '( Hello, world! )') Result: ┌─trim(BOTH ' ()' FROM '( Hello, world! )')─┐ │ Hello, world! │ └───────────────────────────────────────────────┘ ## trimLeft [¶](https://www.tinybird.co/docs/about:blank#trimleft) Removes the consecutive occurrences of whitespace (ASCII-character 32) from the start of a string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) trimLeft(input_string) Alias: `ltrim(input_string)`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `input_string` : string to trim. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A string without leading common whitespaces. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT trimLeft(' Hello, world! ') Result: ┌─trimLeft(' Hello, world! ')─┐ │ Hello, world! │ └─────────────────────────────────────┘ ## trimRight [¶](https://www.tinybird.co/docs/about:blank#trimright) Removes the consecutive occurrences of whitespace (ASCII-character 32) from the end of a string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) trimRight(input_string) Alias: `rtrim(input_string)`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `input_string` : string to trim. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A string without trailing common whitespaces. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT trimRight(' Hello, world! ') Result: ┌─trimRight(' Hello, world! ')─┐ │ Hello, world! │ └──────────────────────────────────────┘ ## trimBoth [¶](https://www.tinybird.co/docs/about:blank#trimboth) Removes the consecutive occurrences of whitespace (ASCII-character 32) from both ends of a string. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) trimBoth(input_string) Alias: `trim(input_string)`. ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `input_string` : string to trim. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) A string without leading and trailing common whitespaces. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT trimBoth(' Hello, world! ') Result: ┌─trimBoth(' Hello, world! ')─┐ │ Hello, world! │ └─────────────────────────────────────┘ ## CRC32 [¶](https://www.tinybird.co/docs/about:blank#crc32) Returns the CRC32 checksum of a string using CRC-32-IEEE 802.3 polynomial and initial value `0xffffffff` (zlib implementation). The result type is UInt32. ## CRC32IEEE [¶](https://www.tinybird.co/docs/about:blank#crc32ieee) Returns the CRC32 checksum of a string, using CRC-32-IEEE 802.3 polynomial. The result type is UInt32. ## CRC64 [¶](https://www.tinybird.co/docs/about:blank#crc64) Returns the CRC64 checksum of a string, using CRC-64-ECMA polynomial. The result type is UInt64. ## normalizeQuery [¶](https://www.tinybird.co/docs/about:blank#normalizequery) Replaces literals, sequences of literals and complex aliases (containing whitespace, more than two digits or at least 36 bytes long such as UUIDs) with placeholder `?`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeQuery(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Sequence of characters. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Sequence of characters with placeholders. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT normalizeQuery('[1, 2, 3, x]') AS query Result: ┌─query────┐ │ [?.., x] │ └──────────┘ ## normalizeQueryKeepNames [¶](https://www.tinybird.co/docs/about:blank#normalizequerykeepnames) Replaces literals, sequences of literals with placeholder `?` but doesn't replace complex aliases (containing whitespace, more than two digits or at least 36 bytes long such as UUIDs). This helps better analyze complex query logs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeQueryKeepNames(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Sequence of characters. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Sequence of characters with placeholders. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT normalizeQuery('SELECT 1 AS aComplexName123'), normalizeQueryKeepNames('SELECT 1 AS aComplexName123') Result: ┌─normalizeQuery('SELECT 1 AS aComplexName123')─┬─normalizeQueryKeepNames('SELECT 1 AS aComplexName123')─┐ │ SELECT ? AS `?` │ SELECT ? AS aComplexName123 │ └───────────────────────────────────────────────┴────────────────────────────────────────────────────────┘ ## normalizedQueryHash [¶](https://www.tinybird.co/docs/about:blank#normalizedqueryhash) Returns identical 64bit hash values without the values of literals for similar queries. Can be helpful to analyze query logs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizedQueryHash(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Sequence of characters. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Hash value. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) Query: SELECT normalizedQueryHash('SELECT 1 AS `xyz`') != normalizedQueryHash('SELECT 1 AS `abc`') AS res Result: ┌─res─┐ │ 1 │ └─────┘ ## normalizedQueryHashKeepNames [¶](https://www.tinybird.co/docs/about:blank#normalizedqueryhashkeepnames) Like normalizedQueryHash it returns identical 64bit hash values without the values of literals for similar queries but it doesn't replace complex aliases (containing whitespace, more than two digits or at least 36 bytes long such as UUIDs) with a placeholder before hashing. Can be helpful to analyze query logs. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizedQueryHashKeepNames(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : Sequence of characters. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - Hash value. UInt64. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT normalizedQueryHash('SELECT 1 AS `xyz123`') != normalizedQueryHash('SELECT 1 AS `abc123`') AS normalizedQueryHash SELECT normalizedQueryHashKeepNames('SELECT 1 AS `xyz123`') != normalizedQueryHashKeepNames('SELECT 1 AS `abc123`') AS normalizedQueryHashKeepNames Result: ┌─normalizedQueryHash─┐ │ 0 │ └─────────────────────┘ ┌─normalizedQueryHashKeepNames─┐ │ 1 │ └──────────────────────────────┘ ## normalizeUTF8NFC [¶](https://www.tinybird.co/docs/about:blank#normalizeutf8nfc) Converts a string to NFC normalized form, assuming the string is valid UTF8-encoded text. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeUTF8NFC(words) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `words` : UTF8-encoded input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - String transformed to NFC normalization form. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT length('â'), normalizeUTF8NFC('â') AS nfc, length(nfc) AS nfc_len Result: ┌─length('â')─┬─nfc─┬─nfc_len─┐ │ 2 │ â │ 2 │ └─────────────┴─────┴─────────┘ ## normalizeUTF8NFD [¶](https://www.tinybird.co/docs/about:blank#normalizeutf8nfd) Converts a string to NFD normalized form, assuming the string is valid UTF8-encoded text. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeUTF8NFD(words) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `words` : UTF8-encoded input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - String transformed to NFD normalization form. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT length('â'), normalizeUTF8NFD('â') AS nfd, length(nfd) AS nfd_len Result: ┌─length('â')─┬─nfd─┬─nfd_len─┐ │ 2 │ â │ 3 │ └─────────────┴─────┴─────────┘ ## normalizeUTF8NFKC [¶](https://www.tinybird.co/docs/about:blank#normalizeutf8nfkc) Converts a string to NFKC normalized form, assuming the string is valid UTF8-encoded text. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeUTF8NFKC(words) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `words` : UTF8-encoded input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - String transformed to NFKC normalization form. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT length('â'), normalizeUTF8NFKC('â') AS nfkc, length(nfkc) AS nfkc_len Result: ┌─length('â')─┬─nfkc─┬─nfkc_len─┐ │ 2 │ â │ 2 │ └─────────────┴──────┴──────────┘ ## normalizeUTF8NFKD [¶](https://www.tinybird.co/docs/about:blank#normalizeutf8nfkd) Converts a string to NFKD normalized form, assuming the string is valid UTF8-encoded text. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) normalizeUTF8NFKD(words) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `words` : UTF8-encoded input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - String transformed to NFKD normalization form. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT length('â'), normalizeUTF8NFKD('â') AS nfkd, length(nfkd) AS nfkd_len Result: ┌─length('â')─┬─nfkd─┬─nfkd_len─┐ │ 2 │ â │ 3 │ └─────────────┴──────┴──────────┘ ## encodeXMLComponent [¶](https://www.tinybird.co/docs/about:blank#encodexmlcomponent) Escapes characters with special meaning in XML such that they can afterwards be place into a XML text node or attribute. The following characters are replaced: `<`, `&`, `>`, `"`, `'`. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) encodeXMLComponent(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : An input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The escaped string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT encodeXMLComponent('Hello, "world"!') SELECT encodeXMLComponent('<123>') SELECT encodeXMLComponent('&tinybird') SELECT encodeXMLComponent('\'foo\'') Result: Hello, "world"! <123> &tinybird 'foo&apos ## decodeXMLComponent [¶](https://www.tinybird.co/docs/about:blank#decodexmlcomponent) Un-escapes substrings with special meaning in XML. These substrings are: `"` `&` `'` `>` `<` This function also replaces numeric character references with Unicode characters. Both decimal (like `✓` ) and hexadecimal ( `✓` ) forms are supported. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) decodeXMLComponent(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : An input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The un-escaped string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT decodeXMLComponent(''foo'') SELECT decodeXMLComponent('< Σ >') Result: 'foo' < Σ > ## decodeHTMLComponent [¶](https://www.tinybird.co/docs/about:blank#decodehtmlcomponent) Un-escapes substrings with special meaning in HTML. For example: `ℏ` `>` `♦` `♥` `<` etc. This function also replaces numeric character references with Unicode characters. Both decimal (like `✓` ) and hexadecimal ( `✓` ) forms are supported. ### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax) decodeHTMLComponent(x) ### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments) - `x` : An input string. String. ### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value) - The un-escaped string. String. ### Example [¶](https://www.tinybird.co/docs/about:blank#example) SELECT decodeHTMLComponent(''CH') SELECT decodeHTMLComponent('I♥Tinybird') Result: 'CH' I♥Tinybird' ## extractTextFromHTML [¶](https://www.tinybird.co/docs/about:blank#extracttextfromhtml) This function extracts plain text from HTML or XHTML. It doesn't conform 100% to the HTML, XML or XHTML specification but the implementation is reasonably accurate and fast. The rules are the following: 1. Comments are skipped. Example: `` . Comment must end with `-->` . Nested comments are disallowed. Note: constructions like `` and `` aren't valid comments in HTML but they are skipped by other rules. 2. CDATA is pasted verbatim. Note: CDATA is XML/XHTML-specific and processed on a "best-effort" basis. 3. `script` and `style` elements are removed with all their content. Note: it's assumed that closing tag can't appear inside content. For example, in JS string literal has to be escaped like `"<\/script>"` . Note: comments and CDATA are possible inside `script` or `style` - then closing tags aren't searched inside CDATA. Example: `]]>` . But they are still searched inside comments. Sometimes it becomes complicated: ` var y = "-->"; alert(x + y);` Note: `script` and `style` can be the names of XML namespaces - then they aren't treated like usual `script` or `style` elements. Example: `Hello` . Note: whitespaces are possible after closing tag name: `` but not before: `< / script>` . 4. Other tags or tag-like elements are skipped without inner content. Example: `.` Note: it's expected that this HTML is illegal: `` Note: it also skips something like tags: `<>` , `` , etc. Note: tag without end is skipped to the end of input: `world` , `Helloworld` - there is no whitespace in HTML, but the function inserts it. Also consider: `Hello

world

` , `Hello
world` . This behavior is reasonable for data analysis, e.g. to convert HTML to a bag of words. 7. Also note that correct handling of whitespaces requires the support of `
`   and CSS `display`   and `white-space`   properties.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

extractTextFromHTML(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : input text. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Extracted text. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The first example contains several tags and a comment and also shows whitespace processing. The second example shows `CDATA` and `script` tag processing. In the third example text is extracted from the full HTML response received by the url function.

SELECT extractTextFromHTML(' 
 A text withtags.  
 ')
SELECT extractTextFromHTML('CDATA]]> ')
SELECT extractTextFromHTML(html) FROM url('http://www.donothingfor2minutes.com/', RawBLOB, 'html String') Result:

A text with tags .
The content within CDATA
Do Nothing for 2 Minutes 2:00  
## ascii [¶](https://www.tinybird.co/docs/about:blank#ascii)

Returns the ASCII code point (as Int32) of the first character of string `s`.

If `s` is empty, the result is 0. If the first character isn't an ASCII character or not part of the Latin-1 supplement range of UTF-16, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ascii(s)
## soundex [¶](https://www.tinybird.co/docs/about:blank#soundex)

Returns the Soundex code of a string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

soundex(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The Soundex code of the input value. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select soundex('aksel') Result:

┌─soundex('aksel')─┐
│ A240             │
└──────────────────┘
## punycodeEncode [¶](https://www.tinybird.co/docs/about:blank#punycodeencode)

Returns the Punycode representation of a string. The string must be UTF8-encoded, otherwise the behavior is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

punycodeEncode(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A Punycode representation of the input value. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select punycodeEncode('München') Result:

┌─punycodeEncode('München')─┐
│ Mnchen-3ya                │
└───────────────────────────┘
## punycodeDecode [¶](https://www.tinybird.co/docs/about:blank#punycodedecode)

Returns the UTF8-encoded plaintext of a Punycode-encoded string. If no valid Punycode-encoded string is given, an exception is thrown.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

punycodeEncode(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Punycode-encoded string. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The plaintext of the input value. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select punycodeDecode('Mnchen-3ya') Result:

┌─punycodeDecode('Mnchen-3ya')─┐
│ München                      │
└──────────────────────────────┘
## tryPunycodeDecode [¶](https://www.tinybird.co/docs/about:blank#trypunycodedecode)

Like `punycodeDecode` but returns an empty string if no valid Punycode-encoded string is given.

## idnaEncode [¶](https://www.tinybird.co/docs/about:blank#idnaencode)

Returns the ASCII representation (ToASCII algorithm) of a domain name according to the Internationalized Domain Names in Applications (IDNA) mechanism. The input string must be UTF-encoded and translatable to an ASCII string, otherwise an exception is thrown. Note: No percent decoding or trimming of tabs, spaces or control characters is performed.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

idnaEncode(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A ASCII representation according to the IDNA mechanism of the input value. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select idnaEncode('straße.münchen.de') Result:

┌─idnaEncode('straße.münchen.de')─────┐
│ xn--strae-oqa.xn--mnchen-3ya.de     │
└─────────────────────────────────────┘
## tryIdnaEncode [¶](https://www.tinybird.co/docs/about:blank#tryidnaencode)

Like `idnaEncode` but returns an empty string in case of an error instead of throwing an exception.

## idnaDecode [¶](https://www.tinybird.co/docs/about:blank#idnadecode)

Returns the Unicode (UTF-8) representation (ToUnicode algorithm) of a domain name according to the Internationalized Domain Names in Applications (IDNA) mechanism. In case of an error (e.g. because the input is invalid), the input string is returned. Note that repeated application of `idnaEncode()` and `idnaDecode()` doesn't necessarily return the original string due to case normalization.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

idnaDecode(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A Unicode (UTF-8) representation according to the IDNA mechanism of the input value. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select idnaDecode('xn--strae-oqa.xn--mnchen-3ya.de') Result:

┌─idnaDecode('xn--strae-oqa.xn--mnchen-3ya.de')─┐
│ straße.münchen.de                             │
└───────────────────────────────────────────────┘
## byteHammingDistance [¶](https://www.tinybird.co/docs/about:blank#bytehammingdistance)

Calculates the hamming distance between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

byteHammingDistance(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT byteHammingDistance('karolin', 'kathrin') Result:

┌─byteHammingDistance('karolin', 'kathrin')─┐
│                                         3 │
└───────────────────────────────────────────┘ Alias: `mismatches`

## stringJaccardIndex [¶](https://www.tinybird.co/docs/about:blank#stringjaccardindex)

Calculates the Jaccard similarity index between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stringJaccardIndex(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT stringJaccardIndex('Tinybird', 'mouse') Result:

┌─stringJaccardIndex('Tinybird', 'mouse')─┐
│                                       0.4 │
└───────────────────────────────────────────┘
## stringJaccardIndexUTF8 [¶](https://www.tinybird.co/docs/about:blank#stringjaccardindexutf8)

Like stringJaccardIndex but for UTF8-encoded strings.

## editDistance [¶](https://www.tinybird.co/docs/about:blank#editdistance)

Calculates the edit distance between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

editDistance(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT editDistance('Tinybird', 'mouse') Result:

┌─editDistance('Tinybird', 'mouse')─┐
│                                   6 │
└─────────────────────────────────────┘ Alias: `levenshteinDistance`

## editDistanceUTF8 [¶](https://www.tinybird.co/docs/about:blank#editdistanceutf8)

Calculates the edit distance between two UTF8 strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

editDistanceUTF8(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT editDistanceUTF8('我是谁', '我是我') Result:

┌─editDistanceUTF8('我是谁', '我是我')──┐
│                                   1 │
└─────────────────────────────────────┘ Alias: `levenshteinDistanceUTF8`

## damerauLevenshteinDistance [¶](https://www.tinybird.co/docs/about:blank#dameraulevenshteindistance)

Calculates the Damerau-Levenshtein distance between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

damerauLevenshteinDistance(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT damerauLevenshteinDistance('Tinybird', 'mouse') Result:

┌─damerauLevenshteinDistance('Tinybird', 'mouse')─┐
│                                                 6 │
└───────────────────────────────────────────────────┘
## jaroSimilarity [¶](https://www.tinybird.co/docs/about:blank#jarosimilarity)

Calculates the Jaro similarity between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

jaroSimilarity(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT jaroSimilarity('Tinybird', 'click') Result:

┌─jaroSimilarity('Tinybird', 'click')─┐
│                    0.8333333333333333 │
└───────────────────────────────────────┘
## jaroWinklerSimilarity [¶](https://www.tinybird.co/docs/about:blank#jarowinklersimilarity)

Calculates the Jaro-Winkler similarity between two byte strings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

jaroWinklerSimilarity(string1, string2)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT jaroWinklerSimilarity('Tinybird', 'click') Result:

┌─jaroWinklerSimilarity('Tinybird', 'click')─┐
│                           0.8999999999999999 │
└──────────────────────────────────────────────┘
## initcap [¶](https://www.tinybird.co/docs/about:blank#initcap)

Convert the first letter of each word to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters.

Because `initCap` converts only the first letter of each word to upper case you may observe unexpected behaviour for words containing apostrophes or capital letters. For example:

SELECT initCap('mother''s daughter'), initCap('joe McAdam') will return

┌─initCap('mother\'s daughter')─┬─initCap('joe McAdam')─┐
│ Mother'S Daughter             │ Joe Mcadam            │
└───────────────────────────────┴───────────────────────┘ This is a known behaviour, with no plans currently to fix it.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

initcap(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `val`   with the first letter of each word converted to upper case. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT initcap('building for fast') Result:

┌─initcap('building for fast')─┐
│ Building For Fast            │
└──────────────────────────────┘
## initcapUTF8 [¶](https://www.tinybird.co/docs/about:blank#initcaputf8)

Like initcap, `initcapUTF8` converts the first letter of each word to upper case and the rest to lower case. Assumes that the string contains valid UTF-8 encoded text. If this assumption is violated, no exception is thrown and the result is undefined.

This function doesn't detect the language, e.g. for Turkish the result might not be exactly correct (i/İ vs. i/I). If the length of the UTF-8 byte sequence is different for upper and lower case of a code point, the result may be incorrect for this code point.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

initcapUTF8(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `val`   with the first letter of each word converted to upper case. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT initcapUTF8('не тормозит') Result:

┌─initcapUTF8('не тормозит')─┐
│ Не Тормозит                │
└────────────────────────────┘
## firstLine [¶](https://www.tinybird.co/docs/about:blank#firstline)

Returns the first line from a multi-line string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

firstLine(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Input value. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first line of the input value or the whole value if there is no line   separators. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select firstLine('foo\nbar\nbaz') Result:

┌─firstLine('foo\nbar\nbaz')─┐
│ foo                        │
└────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/splitting-merging-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Split and merge functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for splitting and merging data."
---




# Functions for splitting strings [¶](https://www.tinybird.co/docs/about:blank#functions-for-splitting-strings)

The following functions are used to split strings into substrings.

## splitByChar [¶](https://www.tinybird.co/docs/about:blank#splitbychar)

Splits a string into substrings separated by a specified character. Uses a constant string `separator` which consists of exactly one character. Returns an array of selected substrings. Empty substrings may be selected if the separator occurs at the beginning or end of the string, or if there are multiple consecutive separators.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

splitByChar(separator, s[, max_substrings]))
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `separator`   : The separator which should contain exactly one character. String.
- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. If `max_substrings`   > 0, the returned array will contain at most `max_substrings`   substrings, otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Empty substrings may be selected when:

- A separator occurs at the beginning or end of the string
- There are multiple consecutive separators
- The original string `s`   is empty.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT splitByChar(',', '1,2,3,abcde') Result:

┌─splitByChar(',', '1,2,3,abcde')─┐
│ ['1','2','3','abcde']           │
└─────────────────────────────────┘
## splitByString [¶](https://www.tinybird.co/docs/about:blank#splitbystring)

Splits a string into substrings separated by a string. It uses a constant string `separator` of multiple characters as the separator. If the string `separator` is empty, it will split the string `s` into an array of single characters.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

splitByString(separator, s[, max_substrings]))
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `separator`   : The separator. String.
- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. When `max_substrings`   > 0, the returned substrings will be no more than `max_substrings`   , otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Empty substrings may be selected when:

- A non-empty separator occurs at the beginning or end of the string
- There are multiple consecutive non-empty separators
- The original string `s`   is empty while the separator isn't empty.

Setting splitby_max_substrings_includes_remaining_string (default: 0) controls if the remaining string is included in the last element of the result array when argument `max_substrings` > 0.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT splitByString(', ', '1, 2 3, 4,5, abcde') Result:

┌─splitByString(', ', '1, 2 3, 4,5, abcde')─┐
│ ['1','2 3','4,5','abcde']                 │
└───────────────────────────────────────────┘ SELECT splitByString('', 'abcde') Result:

┌─splitByString('', 'abcde')─┐
│ ['a','b','c','d','e']      │
└────────────────────────────┘
## splitByRegexp [¶](https://www.tinybird.co/docs/about:blank#splitbyregexp)

Splits a string into substrings separated by a regular expression. It uses a regular expression string `regexp` as the separator. If the `regexp` is empty, it will split the string `s` into an array of single characters. If no match is found for this regular expression, the string `s` won't be split.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

splitByRegexp(regexp, s[, max_substrings]))
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `regexp`   : Regular expression. Constant. String or FixedString.
- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. When `max_substrings`   > 0, the returned substrings will be no more than `max_substrings`   , otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Empty substrings may be selected when:

- A non-empty regular expression match occurs at the beginning or end of the string
- There are multiple consecutive non-empty regular expression matches
- The original string `s`   is empty while the regular expression isn't empty.

Setting splitby_max_substrings_includes_remaining_string (default: 0) controls if the remaining string is included in the last element of the result array when argument `max_substrings` > 0.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT splitByRegexp('\\d+', 'a12bc23de345f') Result:

┌─splitByRegexp('\\d+', 'a12bc23de345f')─┐
│ ['a','bc','de','f']                    │
└────────────────────────────────────────┘ SELECT splitByRegexp('', 'abcde') Result:

┌─splitByRegexp('', 'abcde')─┐
│ ['a','b','c','d','e']      │
└────────────────────────────┘
## splitByWhitespace [¶](https://www.tinybird.co/docs/about:blank#splitbywhitespace)

Splits a string into substrings separated by whitespace characters. Returns an array of selected substrings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

splitByWhitespace(s[, max_substrings]))
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. When `max_substrings`   > 0, the returned substrings will be no more than `max_substrings`   , otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Setting splitby_max_substrings_includes_remaining_string (default: 0) controls if the remaining string is included in the last element of the result array when argument `max_substrings` > 0.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT splitByWhitespace('  1!  a,  b.  ') Result:

┌─splitByWhitespace('  1!  a,  b.  ')─┐
│ ['1!','a,','b.']                    │
└─────────────────────────────────────┘
## splitByNonAlpha [¶](https://www.tinybird.co/docs/about:blank#splitbynonalpha)

Splits a string into substrings separated by whitespace and punctuation characters. Returns an array of selected substrings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

splitByNonAlpha(s[, max_substrings]))
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. When `max_substrings`   > 0, the returned substrings will be no more than `max_substrings`   , otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Setting splitby_max_substrings_includes_remaining_string (default: 0) controls if the remaining string is included in the last element of the result array when argument `max_substrings` > 0.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT splitByNonAlpha('  1!  a,  b.  ') ┌─splitByNonAlpha('  1!  a,  b.  ')─┐
│ ['1','a','b']                     │
└───────────────────────────────────┘
## arrayStringConcat [¶](https://www.tinybird.co/docs/about:blank#arraystringconcat)

Concatenates string representations of values listed in the array with the separator. `separator` is an optional parameter: a constant string, set to an empty string by default. Returns the string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayStringConcat(arr\[, separator\])
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayStringConcat(['12/05/2021', '12:50:00'], ' ') AS DateString Result:

┌─DateString──────────┐
│ 12/05/2021 12:50:00 │
└─────────────────────┘
## alphaTokens [¶](https://www.tinybird.co/docs/about:blank#alphatokens)

Selects substrings of consecutive bytes from the ranges a-z and A-Z.Returns an array of substrings.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

alphaTokens(s[, max_substrings])) Alias: `splitByAlpha`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   : The string to split. String.
- `max_substrings`   : An optional `Int64`   defaulting to 0. When `max_substrings`   > 0, the returned substrings will be no more than `max_substrings`   , otherwise the function will return as many substrings as possible.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- An array of selected substrings. Array(String).

Setting splitby_max_substrings_includes_remaining_string (default: 0) controls if the remaining string is included in the last element of the result array when argument `max_substrings` > 0.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT alphaTokens('abca1abc') ┌─alphaTokens('abca1abc')─┐
│ ['abca','abc']          │
└─────────────────────────┘
## extractAllGroups [¶](https://www.tinybird.co/docs/about:blank#extractallgroups)

Extracts all groups from non-overlapping substrings matched by a regular expression.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

extractAllGroups(text, regexp)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `text`   : String or FixedString.
- `regexp`   : Regular expression. Constant. String or FixedString.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- If the function finds at least one matching group, it returns `Array(Array(String))`   column, clustered by group_id (1 to N, where N is number of capturing groups in `regexp`   ). If there is no matching group, it returns an empty array. Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT extractAllGroups('abc=123, 8="hkl"', '("[^"]+"|\\w+)=("[^"]+"|\\w+)') Result:

┌─extractAllGroups('abc=123, 8="hkl"', '("[^"]+"|\\w+)=("[^"]+"|\\w+)')─┐
│ [['abc','123'],['8','"hkl"']]                                         │
└───────────────────────────────────────────────────────────────────────┘
## ngrams [¶](https://www.tinybird.co/docs/about:blank#ngrams)

Splits a UTF-8 string into n-grams of `ngramsize` symbols.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngrams(string, ngramsize)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String or FixedString.
- `ngramsize`   : The size of an n-gram. UInt.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array with n-grams. Array(String).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT ngrams('Tinybird', 3) Result:

┌─ngrams('Tinybird', 3)─────────────────┐
│ ['Tin','iny','nyb','ybi','bir','ird'] │
└───────────────────────────────────────┘
## tokens [¶](https://www.tinybird.co/docs/about:blank#tokens)

Splits a string into tokens using non-alphanumeric ASCII characters as separators.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `input_string`   : Any set of bytes represented as the String data type object.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The resulting array of tokens from input string. Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT tokens('test1,;\\ test2,;\\ test3,;\\   test4') AS tokens Result:

┌─tokens────────────────────────────┐
│ ['test1','test2','test3','test4'] │
└───────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/rounding-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Rounding functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for rounding."
---




# Rounding functions [¶](https://www.tinybird.co/docs/about:blank#rounding-functions)

The following functions are used to round numbers.

## floor [¶](https://www.tinybird.co/docs/about:blank#floor)

Returns the largest rounded number less than or equal `x`. A rounded number is a multiple of 1 / 10 * N, or the nearest number of the appropriate data type if 1 / 10 * N isn’t exact.

Integer arguments may be rounded with negative `N` argument, with non-negative `N` the function returns `x` , i.e. does nothing.

If rounding causes an overflow (for example, `floor(-128, -1)` ), the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

floor(x[, N])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   - The value to round. Float*, Decimal*, or (U)Int*.
- `N`   . (U)Int*. The default is zero, which means rounding to an integer. Can be negative.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A rounded number of the same type as `x`.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT floor(123.45, 1) AS rounded Result:

┌─rounded─┐
│   123.4 │
└─────────┘ Query:

SELECT floor(123.45, -1) Result:

┌─rounded─┐
│     120 │
└─────────┘
## ceiling [¶](https://www.tinybird.co/docs/about:blank#ceiling)

Like `floor` but returns the smallest rounded number greater than or equal `x`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ceiling(x[, N]) Alias: `ceil`

## truncate [¶](https://www.tinybird.co/docs/about:blank#truncate)

Like `floor` but returns the rounded number with largest absolute value that has an absolute value less than or equal to `x` ‘s.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

truncate(x[, N]) Alias: `trunc`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT truncate(123.499, 1) as res ┌───res─┐
│ 123.4 │
└───────┘
## round [¶](https://www.tinybird.co/docs/about:blank#round)

Rounds a value to a specified number of decimal places.

The function returns the nearest number of the specified order. If the input value has equal distance to two neighboring numbers, the function uses banker’s rounding for Float* inputs and rounds away from zero for the other number types (Decimal*.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

round(x[, N])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : A number to round. Float*, Decimal*, or (U)Int*.
- `N`   : The number of decimal places to round to. Integer. Defaults to `0`  .
  - If `N > 0`     , the function rounds to the right of the decimal point.
  - If `N < 0`     , the function rounds to the left of the decimal point.
  - If `N = 0`     , the function rounds to the next integer.

### Returned value: [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A rounded number of the same type as `x`.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Example with `Float` inputs:

SELECT number / 2 AS x, round(x) FROM system.numbers LIMIT 3 ┌───x─┬─round(divide(number, 2))─┐
│   0 │                        0 │
│ 0.5 │                        0 │
│   1 │                        1 │
└─────┴──────────────────────────┘ Example with `Decimal` inputs:

SELECT cast(number / 2 AS  Decimal(10,4)) AS x, round(x) FROM system.numbers LIMIT 3 ┌───x─┬─round(CAST(divide(number, 2), 'Decimal(10, 4)'))─┐
│   0 │                                                0 │
│ 0.5 │                                                1 │
│   1 │                                                1 │
└─────┴──────────────────────────────────────────────────┘ To retain trailing zeros, enable setting `output_format_decimal_trailing_zeros`:

SELECT cast(number / 2 AS  Decimal(10,4)) AS x, round(x) FROM system.numbers LIMIT 3 settings output_format_decimal_trailing_zeros=1 ┌──────x─┬─round(CAST(divide(number, 2), 'Decimal(10, 4)'))─┐
│ 0.0000 │                                           0.0000 │
│ 0.5000 │                                           1.0000 │
│ 1.0000 │                                           1.0000 │
└────────┴──────────────────────────────────────────────────┘ Examples of rounding to the nearest number:

round(3.2, 0) = 3
round(4.1267, 2) = 4.13
round(22,-1) = 20
round(467,-2) = 500
round(-467,-2) = -500 Banker’s rounding.

round(3.5) = 4
round(4.5) = 4
round(3.55, 1) = 3.6
round(3.65, 1) = 3.6
## roundBankers [¶](https://www.tinybird.co/docs/about:blank#roundbankers)

Rounds a number to a specified decimal position.

If the rounding number is halfway between two numbers, the function uses banker’s rounding. Banker's rounding is a method of rounding fractional numbers When the rounding number is halfway between two numbers, it's rounded to the nearest even digit at the specified decimal position. For example: 3.5 rounds up to 4, 2.5 rounds down to 2. It's the default rounding method for floating point numbers defined in IEEE 754. The round function performs the same rounding for floating point numbers. The `roundBankers` function also rounds integers the same way, for example, `roundBankers(45, -1) = 40`.

In other cases, the function rounds numbers to the nearest integer.

Using banker’s rounding, you can reduce the effect that rounding numbers has on the results of summing or subtracting these numbers.

For example, sum numbers 1.5, 2.5, 3.5, 4.5 with different rounding:

- No rounding: 1.5 + 2.5 + 3.5 + 4.5 = 12.
- Banker’s rounding: 2 + 2 + 4 + 4 = 12.
- Rounding to the nearest integer: 2 + 3 + 4 + 5 = 14.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

roundBankers(x [, N])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `N > 0` : The function rounds the number to the given position right of the decimal point. Example: `roundBankers(3.55, 1) = 3.6`. - `N < 0` : The function rounds the number to the given position left of the decimal point. Example: `roundBankers(24.55, -1) = 20`. - `N = 0` : The function rounds the number to an integer. In this case the argument can be omitted. Example: `roundBankers(2.5) = 2`.

- `x`   : A number to round. Float*, Decimal*, or (U)Int*.
- `N`   : The number of decimal places to round to. Integer. Defaults to `0`  .
  - If `N > 0`     , the function rounds to the right of the decimal point.
  - If `N < 0`     , the function rounds to the left of the decimal point.
  - If `N = 0`     , the function rounds to the next integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A value rounded by the banker’s rounding method.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT number / 2 AS x, roundBankers(x, 0) AS b fROM system.numbers limit 10 Result:

┌───x─┬─b─┐
│   0 │ 0 │
│ 0.5 │ 0 │
│   1 │ 1 │
│ 1.5 │ 2 │
│   2 │ 2 │
│ 2.5 │ 2 │
│   3 │ 3 │
│ 3.5 │ 4 │
│   4 │ 4 │
│ 4.5 │ 4 │
└─────┴───┘ Examples of Banker’s rounding:

roundBankers(0.4) = 0
roundBankers(-3.5) = -4
roundBankers(4.5) = 4
roundBankers(3.55, 1) = 3.6
roundBankers(3.65, 1) = 3.6
roundBankers(10.35, 1) = 10.4
roundBankers(10.755, 2) = 10.76
## roundToExp2 [¶](https://www.tinybird.co/docs/about:blank#roundtoexp2)

Accepts a number. If the number is less than one, it returns `0` . Otherwise, it rounds the number down to the nearest (whole non-negative) degree of two.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

roundToExp2(num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `num`   : A number to round. UInt/Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `0`   , for `num`   $\lt 1$. UInt8.
- `num`   rounded down to the nearest (whole non-negative) degree of two. UInt/Float equivalent to the input type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT *, roundToExp2(*) FROM system.numbers WHERE number IN (0, 2, 5, 10, 19, 50) Result:

┌─number─┬─roundToExp2(number)─┐
│      0 │                   0 │
│      2 │                   2 │
│      5 │                   4 │
│     10 │                   8 │
│     19 │                  16 │
│     50 │                  32 │
└────────┴─────────────────────┘
## roundDuration [¶](https://www.tinybird.co/docs/about:blank#roundduration)

Accepts a number. If the number is less than one, it returns `0` . Otherwise, it rounds the number down to numbers from the set of commonly used durations: `1, 10, 30, 60, 120, 180, 240, 300, 600, 1200, 1800, 3600, 7200, 18000, 36000`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

roundDuration(num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `num`   : A number to round to one of the numbers in the set of common durations. UInt/Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `0`   , for `num`   $\lt 1$.
- Otherwise, one of: `1, 10, 30, 60, 120, 180, 240, 300, 600, 1200, 1800, 3600, 7200, 18000, 36000`   . UInt16.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT *, roundDuration(*) FROM system.numbers WHERE number IN (0, 9, 19, 47, 101, 149, 205, 271, 421, 789, 1423, 2345, 4567, 9876, 24680, 42573) Result:

┌─number─┬─roundDuration(number)─┐
│      0 │                     0 │
│      9 │                     1 │
│     19 │                    10 │
│     47 │                    30 │
│    101 │                    60 │
│    149 │                   120 │
│    205 │                   180 │
│    271 │                   240 │
│    421 │                   300 │
│    789 │                   600 │
│   1423 │                  1200 │
│   2345 │                  1800 │
│   4567 │                  3600 │
│   9876 │                  7200 │
│  24680 │                 18000 │
│  42573 │                 36000 │
└────────┴───────────────────────┘
## roundAge [¶](https://www.tinybird.co/docs/about:blank#roundage)

Accepts a number within various commonly used ranges of human age and returns either a maximum or a minimum within that range.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

roundAge(num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `age`   : A number representing an age in years. UInt/Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `0`   , for $age \lt 1$.
- Returns `17`   , for $1 \leq age \leq 17$.
- Returns `18`   , for $18 \leq age \leq 24$.
- Returns `25`   , for $25 \leq age \leq 34$.
- Returns `35`   , for $35 \leq age \leq 44$.
- Returns `45`   , for $45 \leq age \leq 54$.
- Returns `55`   , for $age \geq 55$.

Type: UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT *, roundAge(*) FROM system.numbers WHERE number IN (0, 5, 20, 31, 37, 54, 72) Result:

┌─number─┬─roundAge(number)─┐
│      0 │                0 │
│      5 │               17 │
│     20 │               18 │
│     31 │               25 │
│     37 │               35 │
│     54 │               45 │
│     72 │               55 │
└────────┴──────────────────┘
## roundDown [¶](https://www.tinybird.co/docs/about:blank#rounddown)

Accepts a number and rounds it down to an element in the specified array. If the value is less than the lowest bound, the lowest bound is returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

roundDown(num, arr)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `num`   : A number to round down. Numeric.
- `arr`   : Array of elements to round `age`   down to. Array of UInt/Float type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number rounded down to an element in `arr`   . If the value is less than the lowest bound, the lowest bound is returned. UInt/Float type deduced from the type of `arr`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT *, roundDown(*, [3, 4, 5]) FROM system.numbers WHERE number IN (0, 1, 2, 3, 4, 5) Result:

┌─number─┬─roundDown(number, [3, 4, 5])─┐
│      0 │                            3 │
│      1 │                            3 │
│      2 │                            3 │
│      3 │                            3 │
│      4 │                            4 │
│      5 │                            5 │
└────────┴──────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/random-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Random number functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for generating random numbers."
---




# Functions for generating random numbers [¶](https://www.tinybird.co/docs/about:blank#functions-for-generating-random-numbers)

The following functions are used to generate random numbers.

All functions in this section accept zero or one arguments. The only use of the argument (if provided) is to prevent common subexpression elimination such that two different executions within a row of the same random function return different random values.

## rand [¶](https://www.tinybird.co/docs/about:blank#rand)

Returns a random UInt32 number with uniform distribution.

Uses a linear congruential generator with an initial state obtained from the system, which means that while it appears random, it's not truly random and can be predictable if the initial state is known. For scenarios where true randomness is crucial, consider using alternative methods like system-level calls or integrating with external libraries.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

rand() Alias: `rand32`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Returns a number of type UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT rand() 1569354847 -- Note: The actual output will be a random number, not the specific number shown in the example
## rand64 [¶](https://www.tinybird.co/docs/about:blank#rand64)

Returns a random UInt64 integer (UInt64) number

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

rand64()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Returns a number UInt64 number with uniform distribution.

Uses a linear congruential generator with an initial state obtained from the system, which means that while it appears random, it's not truly random and can be predictable if the initial state is known. For scenarios where true randomness is crucial, consider using alternative methods like system-level calls or integrating with external libraries.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT rand64() 15030268859237645412 -- Note: The actual output will be a random number, not the specific number shown in the example.
## randCanonical [¶](https://www.tinybird.co/docs/about:blank#randcanonical)

Returns a random Float64 number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randCanonical()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Returns a Float64 value between 0 (inclusive) and 1 (exclusive).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randCanonical() 0.3452178901234567 - Note: The actual output will be a random Float64 number between 0 and 1, not the specific number shown in the example.
## randConstant [¶](https://www.tinybird.co/docs/about:blank#randconstant)

Generates a single constant column filled with a random value. Unlike `rand` , this function ensures the same random value appears in every row of the generated column, making it useful for scenarios requiring a consistent random seed across rows in a single query.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randConstant([x])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- **  [x] (Optional):**   An optional expression that influences the generated random value. Even if provided, the resulting value will still be constant within the same query execution. Different queries using the same expression will likely generate different constant values.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Returns a column of type UInt32 containing the same random value in each row.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

The actual output will be different for each query execution, even with the same optional expression. The optional parameter may not significantly change the generated value compared to using `randConstant` alone.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randConstant() AS random_value | random_value |
|--------------|
| 1234567890   | SELECT randConstant(10) AS random_value | random_value |
|--------------|
| 9876543210   |
## randUniform [¶](https://www.tinybird.co/docs/about:blank#randuniform)

Returns a random Float64 drawn uniformly from interval [ `min`, `max` ].

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randUniform(min, max)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `min`   - `Float64`   - left boundary of the range,
- `max`   - `Float64`   - right boundary of the range.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

A random number of type Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randUniform(5.5, 10) FROM numbers(5) ┌─randUniform(5.5, 10)─┐
│    8.094978491443102 │
│   7.3181248914450885 │
│    7.177741903868262 │
│    6.483347380953762 │
│    6.122286382885112 │
└──────────────────────┘
## randNormal [¶](https://www.tinybird.co/docs/about:blank#randnormal)

Returns a random Float64 drawn from a normal distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randNormal(mean, variance)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mean`   - `Float64`   - mean value of distribution,
- `variance`   - `Float64`   - variance of the distribution.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randNormal(10, 2) FROM numbers(5) Result:

┌──randNormal(10, 2)─┐
│ 13.389228911709653 │
│  8.622949707401295 │
│ 10.801887062682981 │
│ 4.5220192605895315 │
│ 10.901239123982567 │
└────────────────────┘
## randLogNormal [¶](https://www.tinybird.co/docs/about:blank#randlognormal)

Returns a random Float64 drawn from a log-normal distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randLogNormal(mean, variance)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mean`   - `Float64`   - mean value of distribution,
- `variance`   - `Float64`   - variance of the distribution.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randLogNormal(100, 5) FROM numbers(5) Result:

┌─randLogNormal(100, 5)─┐
│  1.295699673937363e48 │
│  9.719869109186684e39 │
│  6.110868203189557e42 │
│  9.912675872925529e39 │
│ 2.3564708490552458e42 │
└───────────────────────┘
## randBinomial [¶](https://www.tinybird.co/docs/about:blank#randbinomial)

Returns a random UInt64 drawn from a binomial distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randBinomial(experiments, probability)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `experiments`   - `UInt64`   - number of experiments,
- `probability`   - `Float64`   - probability of success in each experiment, a value between 0 and 1.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randBinomial(100, .75) FROM numbers(5) Result:

┌─randBinomial(100, 0.75)─┐
│                      74 │
│                      78 │
│                      76 │
│                      77 │
│                      80 │
└─────────────────────────┘
## randNegativeBinomial [¶](https://www.tinybird.co/docs/about:blank#randnegativebinomial)

Returns a random UInt64 drawn from a negative binomial distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randNegativeBinomial(experiments, probability)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `experiments`   - `UInt64`   - number of experiments,
- `probability`   - `Float64`   - probability of failure in each experiment, a value between 0 and 1.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randNegativeBinomial(100, .75) FROM numbers(5) Result:

┌─randNegativeBinomial(100, 0.75)─┐
│                              33 │
│                              32 │
│                              39 │
│                              40 │
│                              50 │
└─────────────────────────────────┘
## randPoisson [¶](https://www.tinybird.co/docs/about:blank#randpoisson)

Returns a random UInt64 drawn from a Poisson distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randPoisson(n)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `n`   - `UInt64`   - mean number of occurrences.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randPoisson(10) FROM numbers(5) Result:

┌─randPoisson(10)─┐
│               8 │
│               8 │
│               7 │
│              10 │
│               6 │
└─────────────────┘
## randBernoulli [¶](https://www.tinybird.co/docs/about:blank#randbernoulli)

Returns a random UInt64 drawn from a Bernoulli distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randBernoulli(probability)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `probability`   - `Float64`   - probability of success, a value between 0 and 1.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randBernoulli(.75) FROM numbers(5) Result:

┌─randBernoulli(0.75)─┐
│                   1 │
│                   1 │
│                   0 │
│                   1 │
│                   1 │
└─────────────────────┘
## randExponential [¶](https://www.tinybird.co/docs/about:blank#randexponential)

Returns a random Float64 drawn from a exponential distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randExponential(lambda)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lambda`   - `Float64`   - lambda value.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randExponential(1/10) FROM numbers(5) Result:

┌─randExponential(divide(1, 10))─┐
│              44.71628934340778 │
│              4.211013337903262 │
│             10.809402553207766 │
│              15.63959406553284 │
│             1.8148392319860158 │
└────────────────────────────────┘
## randChiSquared [¶](https://www.tinybird.co/docs/about:blank#randchisquared)

Returns a random Float64 drawn from a Chi-square distribution - a distribution of a sum of the squares of k independent standard normal random variables.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randChiSquared(degree_of_freedom)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `degree_of_freedom`   - `Float64`   - degree of freedom.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randChiSquared(10) FROM numbers(5) Result:

┌─randChiSquared(10)─┐
│ 10.015463656521543 │
│  9.621799919882768 │
│   2.71785015634699 │
│ 11.128188665931908 │
│  4.902063104425469 │
└────────────────────┘
## randStudentT [¶](https://www.tinybird.co/docs/about:blank#randstudentt)

Returns a random Float64 drawn from a Student's t-distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randStudentT(degree_of_freedom)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `degree_of_freedom`   - `Float64`   - degree of freedom.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randStudentT(10) FROM numbers(5) Result:

┌─────randStudentT(10)─┐
│   1.2217309938538725 │
│   1.7941971681200541 │
│ -0.28192176076784664 │
│   0.2508897721303792 │
│  -2.7858432909761186 │
└──────────────────────┘
## randFisherF [¶](https://www.tinybird.co/docs/about:blank#randfisherf)

Returns a random Float64 drawn from a F-distribution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randFisherF(d1, d2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `d1`   - `Float64`   - d1 degree of freedom in `X = (S1 / d1) / (S2 / d2)`  ,
- `d2`   - `Float64`   - d2 degree of freedom in `X = (S1 / d1) / (S2 / d2)`  ,

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Random number. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT randFisherF(10, 3) FROM numbers(5) Result:

┌──randFisherF(10, 3)─┐
│   7.286287504216609 │
│ 0.26590779413050386 │
│ 0.22207610901168987 │
│  0.7953362728449572 │
│ 0.19278885985221572 │
└─────────────────────┘
## randomString [¶](https://www.tinybird.co/docs/about:blank#randomstring)

Generates a string of the specified length filled with random bytes (including zero bytes). Not all characters may be printable.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randomString(length)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `length`   : String length in bytes. Positive integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- String filled with random bytes. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT randomString(30) AS str, length(str) AS len FROM numbers(2) FORMAT Vertical Result:

Row 1:
──────
str: 3 G  :   pT ?w тi  k aV f6
len: 30

Row 2:
──────
str: 9 ,]    ^   )  ]??  8
len: 30
## randomFixedString [¶](https://www.tinybird.co/docs/about:blank#randomfixedstring)

Generates a binary string of the specified length filled with random bytes (including zero bytes). Not all characters may be printable.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randomFixedString(length)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `length`   : String length in bytes. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- String filled with random bytes. FixedString.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT randomFixedString(13) as rnd, toTypeName(rnd) Result:

┌─rnd──────┬─toTypeName(randomFixedString(13))─┐
│ j▒h㋖HɨZ'▒ │ FixedString(13)                 │
└──────────┴───────────────────────────────────┘
## randomPrintableASCII [¶](https://www.tinybird.co/docs/about:blank#randomprintableascii)

Generates a string with a random set of ASCII characters. All characters are printable. If you pass `length < 0` , the behavior of the function is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randomPrintableASCII(length)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `length`   : String length in bytes. Positive integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- String with a random set of ASCII printable characters. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT number, randomPrintableASCII(30) as str, length(str) FROM system.numbers LIMIT 3 ┌─number─┬─str────────────────────────────┬─length(randomPrintableASCII(30))─┐
│      0 │ SuiCOSTvC0csfABSw=UcSzp2.`rv8x │                               30 │
│      1 │ 1Ag NlJ &RCN:*>HVPG;PE-nO"SUFD │                               30 │
│      2 │ /"+<"wUTh:=LjJ Vm!c&hI*m#XTfzz │                               30 │
└────────┴────────────────────────────────┴──────────────────────────────────┘
## randomStringUTF8 [¶](https://www.tinybird.co/docs/about:blank#randomstringutf8)

Generates a random string of a specified length. Result string contains valid UTF-8 code points. The value of code points may be outside of the range of assigned Unicode.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

randomStringUTF8(length)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `length`   : Length of the string in code points. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- UTF-8 random string. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT randomStringUTF8(13) Result:

┌─randomStringUTF8(13)─┐
│ 𘤗𙉝д兠庇󡅴󱱎󦐪􂕌𔊹𓰛   │
└──────────────────────┘
## fuzzBits [¶](https://www.tinybird.co/docs/about:blank#fuzzbits)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

Flips the bits of String or FixedString `s` , each with probability `prob`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

fuzzBits(s, prob)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   - `String`   or `FixedString`  ,
- `prob`   - constant `Float32/64`   between 0.0 and 1.0.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Fuzzed string with same type as `s`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fuzzBits(materialize('abacaba'), 0.1)
FROM numbers(3) Result:

┌─fuzzBits(materialize('abacaba'), 0.1)─┐
│ abaaaja                               │
│ a*cjab+                               │
│ aeca2A                                │
└───────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/parametric-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Parametric Aggregate Functions · Tinybird Docs"
theme-color: "#171612"
description: "Parametric Aggregate Functions"
---




# Parametric aggregate functions [¶](https://www.tinybird.co/docs/about:blank#parametric-aggregate-functions)

Some aggregate functions can accept not only argument columns (used for compression), but a set of parameters – constants for initialization. The syntax is two pairs of brackets instead of one. The first is for parameters, and the second is for arguments.

## histogram [¶](https://www.tinybird.co/docs/about:blank#histogram)

Calculates an adaptive histogram. It doesn't guarantee precise results.

histogram(number_of_bins)(values) The functions uses [A Streaming Parallel Decision Tree Algorithm](http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf) . The borders of histogram bins are adjusted as new data enters a function. In common case, the widths of bins aren't equal.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`values` : Expression resulting in input values.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

`number_of_bins` : Upper limit for the number of bins in the histogram. The function automatically calculates the number of bins. It tries to reach the specified number of bins, but if it fails, it uses fewer bins.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of Tuples of the following format:  

`[(lower_1, upper_1, height_1), ... (lower_N, upper_N, height_N)]`  

  - `lower`   : Lower bound of the bin.   - `upper`   : Upper bound of the bin.   - `height`   : Calculated height of the bin.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT histogram(5)(number + 1)
FROM (
    SELECT *
    FROM system.numbers
    LIMIT 20
) ┌─histogram(5)(plus(number, 1))───────────────────────────────────────────┐
│ [(1,4.5,4),(4.5,8.5,4),(8.5,12.75,4.125),(12.75,17,4.625),(17,20,3.25)] │
└─────────────────────────────────────────────────────────────────────────┘ You can visualize a histogram with the bar function, for example:

WITH histogram(5)(rand() % 100) AS hist
SELECT
    arrayJoin(hist).3 AS height,
    bar(height, 0, 6, 5) AS bar
FROM
(
    SELECT *
    FROM system.numbers
    LIMIT 20
) ┌─height─┬─bar───┐
│  2.125 │ █▋    │
│   3.25 │ ██▌   │
│  5.625 │ ████▏ │
│  5.625 │ ████▏ │
│  3.375 │ ██▌   │
└────────┴───────┘ In this case, you should remember that you don't know the histogram bin borders.

## sequenceMatch [¶](https://www.tinybird.co/docs/about:blank#sequencematch)

Checks whether the sequence contains an event chain that matches the pattern.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sequenceMatch(pattern)(timestamp, cond1, cond2, ...) Events that occur at the same second may lay in the sequence in an undefined order affecting the result.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timestamp`   : Column considered to contain time data. Typical data types are `Date`   and `DateTime`   . You can also use any of the supported UInt data types.
- `cond1`  , `cond2`   : Conditions that describe the chain of events. Data type: `UInt8`   . You can pass up to 32 condition arguments. The function takes only the events described in these conditions into account. If the sequence contains data that isn’t described in a condition, the function skips them.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `pattern`   : Pattern string. See[  Pattern syntax](https://www.tinybird.co/docs/about:blank#pattern-syntax)  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- 1, if the pattern is matched.
- 0, if the pattern isn’t matched.

Type: `UInt8`.

### Pattern syntax [¶](https://www.tinybird.co/docs/about:blank#pattern-syntax)

- `(?N)`   : Matches the condition argument at position `N`   . Conditions are numbered in the `[1, 32]`   range. For example, `(?1)`   matches the argument passed to the `cond1`   parameter.
- `.*`   : Matches any number of events. You don't need conditional arguments to match this element of the pattern.
- `(?t operator value)`   : Sets the time in seconds that should separate two events. For example, pattern `(?1)(?t>1800)(?2)`   matches events that occur more than 1800 seconds from each other. An arbitrary number of any events can lay between these events. You can use the `>=`  , `>`  , `<`  , `<=`  , `==`   operators.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Consider data in the `t` table:

┌─time─┬─number─┐
│    1 │      1 │
│    2 │      3 │
│    3 │      2 │
└──────┴────────┘ Perform the query:

SELECT sequenceMatch('(?1)(?2)')(time, number = 1, number = 2) FROM t ┌─sequenceMatch('(?1)(?2)')(time, equals(number, 1), equals(number, 2))─┐
│                                                                     1 │
└───────────────────────────────────────────────────────────────────────┘ The function found the event chain where number 2 follows number 1. It skipped number 3 between them, because the number isn't described as an event. To take this number into account when searching for the event chain given in the example, make a condition for it.

SELECT sequenceMatch('(?1)(?2)')(time, number = 1, number = 2, number = 3) FROM t ┌─sequenceMatch('(?1)(?2)')(time, equals(number, 1), equals(number, 2), equals(number, 3))─┐
│                                                                                        0 │
└──────────────────────────────────────────────────────────────────────────────────────────┘ In this case, the function couldn’t find the event chain matching the pattern, because the event for number 3 occurred between 1 and 2. If in the same case you checked the condition for number 4, the sequence would match the pattern.

SELECT sequenceMatch('(?1)(?2)')(time, number = 1, number = 2, number = 4) FROM t ┌─sequenceMatch('(?1)(?2)')(time, equals(number, 1), equals(number, 2), equals(number, 4))─┐
│                                                                                        1 │
└──────────────────────────────────────────────────────────────────────────────────────────┘
## sequenceCount [¶](https://www.tinybird.co/docs/about:blank#sequencecount)

Counts the number of event chains that matched the pattern. The function searches event chains that don't overlap. It starts to search for the next chain after the current chain is matched.

Events that occur at the same second may lay in the sequence in an undefined order affecting the result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sequenceCount(pattern)(timestamp, cond1, cond2, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timestamp`   : Column considered to contain time data. Typical data types are `Date`   and `DateTime`   . You can also use any of the supported UInt data types.
- `cond1`  , `cond2`   : Conditions that describe the chain of events. Data type: `UInt8`   . You can pass up to 32 condition arguments. The function takes only the events described in these conditions into account. If the sequence contains data that isn’t described in a condition, the function skips them.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `pattern`   : Pattern string. See[  Pattern syntax](https://www.tinybird.co/docs/about:blank#pattern-syntax)  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Number of non-overlapping event chains that are matched.

Type: `UInt64`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Consider data in the `t` table:

┌─time─┬─number─┐
│    1 │      1 │
│    2 │      3 │
│    3 │      2 │
│    4 │      1 │
│    5 │      3 │
│    6 │      2 │
└──────┴────────┘ Count how many times the number 2 occurs after the number 1 with any amount of other numbers between them:

SELECT sequenceCount('(?1).*(?2)')(time, number = 1, number = 2) FROM t ┌─sequenceCount('(?1).*(?2)')(time, equals(number, 1), equals(number, 2))─┐
│                                                                       2 │
└─────────────────────────────────────────────────────────────────────────┘
### See Also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  sequenceMatch](https://www.tinybird.co/docs/about:blank#sequencematch)

## windowFunnel [¶](https://www.tinybird.co/docs/about:blank#windowfunnel)

Searches for event chains in a sliding time window and calculates the maximum number of events that occurred from the chain.

The function works according to the algorithm:

- The function searches for data that triggers the first condition in the chain and sets the event counter to 1. This is the moment when the sliding window starts.
- If events from the chain occur sequentially within the window, the counter is incremented. If the sequence of events is disrupted, the counter isn’t incremented.
- If the data has multiple event chains at varying points of completion, the function will only output the size of the longest chain.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

windowFunnel(window, [mode, [mode, ... ]])(timestamp, cond1, cond2, ..., condN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timestamp`   : Name of the column containing the timestamp. Data types supported: Date, DateTime and other unsigned integer types (note that even though timestamp supports the `UInt64`   type, it’s value can’t exceed the Int64 maximum, which is 2^63 - 1).
- `cond`   : Conditions or data describing the chain of events. UInt8.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `window`   : Length of the sliding window, it's the time interval between the first and the last condition. The unit of `window`   depends on the `timestamp`   itself and varies. Determined using the expression `timestamp of cond1 <= timestamp of cond2 <= ... <= timestamp of condN <= timestamp of cond1 + window`  .
- `mode`   : It is an optional argument. One or more modes can be set.
  - `'strict_deduplication'`     : If the same condition holds for the sequence of events, then such repeating event interrupts further processing. Note: it may work unexpectedly if several conditions hold for the same event.
  - `'strict_order'`     : Don't allow interventions of other events. E.g. in the case of `A->B->D->C`     , it stops finding `A->B->C`     at the `D`     and the max event level is 2.
  - `'strict_increase'`     : Apply conditions only to events with strictly increasing timestamps.
  - `'strict_once'`     : Count each event only once in the chain even if it meets the condition several times

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The maximum number of consecutive triggered conditions from the chain within the sliding time window. All the chains in the selection are analyzed.

Type: `Integer`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Determine if a set period of time is enough for the user to select a phone and purchase it twice in the online store.

Set the following chain of events:

1. The user logged in to their account on the store ( `eventID = 1003`   ).
2. The user searches for a phone ( `eventID = 1007, product = 'phone'`   ).
3. The user placed an order ( `eventID = 1009`   ).
4. The user made the order again ( `eventID = 1010`   ).

Input table:

SCHEMA >
    event_date Date `json:$.event_date`,
    user_id UInt32 `json:$.user_id`,
    timestamp DateTime `json:$.timestamp`,
    eventID UInt32 `json:$.eventID`,
    product String `json:$.product`

ENGINE "MergeTree" tb push datasources/trend.datasource

echo '
{"event_date": "2019-01-28", "user_id": 1, "timestamp": "2019-01-29 10:00:00", "eventID": 1003, "product": "phone"}
{"event_date": "2019-01-31", "user_id": 1, "timestamp": "2019-01-31 09:00:00", "eventID": 1007, "product": "phone"}
{"event_date": "2019-01-30", "user_id": 1, "timestamp": "2019-01-30 08:00:00", "eventID": 1009, "product": "phone"}
{"event_date": "2019-02-01", "user_id": 1, "timestamp": "2019-02-01 08:00:00", "eventID": 1010, "product": "phone"}'  > trend.ndjson

tb datasource append trend trend.ndjson Find out how far the user `user_id` could get through the chain in a period in January-February of 2019.

Query:

SELECT
    level,
    count() AS c
FROM
(
    SELECT
        user_id,
        windowFunnel(6048000000000000)(timestamp, eventID = 1003, eventID = 1009, eventID = 1007, eventID = 1010) AS level
    FROM trend
    WHERE (event_date >= '2019-01-01') AND (event_date <= '2019-02-02')
    GROUP BY user_id
)
GROUP BY level
ORDER BY level ASC Result:

┌─level─┬─c─┐
│     4 │ 1 │
└───────┴───┘
## retention [¶](https://www.tinybird.co/docs/about:blank#retention)

The function takes as arguments a set of conditions from 1 to 32 arguments of type `UInt8` that indicate whether a certain condition was met for the event. Any condition can be specified as an argument (as in WHERE).

The conditions, except the first, apply in pairs: the result of the second will be true if the first and second are true, of the third if the first and third are true, etc.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

retention(cond1, cond2, ..., cond32)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `cond`   : An expression that returns a `UInt8`   result (1 or 0).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The array of 1 or 0.

- 1: Condition was met for the event.
- 0: Condition wasn’t met for the event.

Type: `UInt8`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Let’s consider an example of calculating the `retention` function to determine site traffic.

**1.** Create a table to illustrate an example.

Input table:

Query:

SELECT '2020-01-01' as date, number as uid FROM numbers(5)  UNION ALL
SELECT '2020-01-02' as date, number as uid FROM numbers(10) UNION ALL
SELECT '2020-01-03' as date, number as uid FROM numbers(15) Result:

┌───────date─┬─uid─┐
│ 2020-01-01 │   0 │
│ 2020-01-01 │   1 │
│ 2020-01-01 │   2 │
│ 2020-01-01 │   3 │
│ 2020-01-01 │   4 │
└────────────┴─────┘
┌───────date─┬─uid─┐
│ 2020-01-02 │   0 │
│ 2020-01-02 │   1 │
│ 2020-01-02 │   2 │
│ 2020-01-02 │   3 │
│ 2020-01-02 │   4 │
│ 2020-01-02 │   5 │
│ 2020-01-02 │   6 │
│ 2020-01-02 │   7 │
│ 2020-01-02 │   8 │
│ 2020-01-02 │   9 │
└────────────┴─────┘
┌───────date─┬─uid─┐
│ 2020-01-03 │   0 │
│ 2020-01-03 │   1 │
│ 2020-01-03 │   2 │
│ 2020-01-03 │   3 │
│ 2020-01-03 │   4 │
│ 2020-01-03 │   5 │
│ 2020-01-03 │   6 │
│ 2020-01-03 │   7 │
│ 2020-01-03 │   8 │
│ 2020-01-03 │   9 │
│ 2020-01-03 │  10 │
│ 2020-01-03 │  11 │
│ 2020-01-03 │  12 │
│ 2020-01-03 │  13 │
│ 2020-01-03 │  14 │
└────────────┴─────┘ **2.** Group users by unique ID `uid` using the `retention` function.

Query:

SELECT
    uid,
    retention(date = '2020-01-01', date = '2020-01-02', date = '2020-01-03') AS r
FROM (
    SELECT '2020-01-01' as date, number as uid FROM numbers(5)  UNION ALL
    SELECT '2020-01-02' as date, number as uid FROM numbers(10) UNION ALL
    SELECT '2020-01-03' as date, number as uid FROM numbers(15))
WHERE date IN ('2020-01-01', '2020-01-02', '2020-01-03')
GROUP BY uid
ORDER BY uid ASC Result:

┌─uid─┬─r───────┐
│   0 │ [1,1,1] │
│   1 │ [1,1,1] │
│   2 │ [1,1,1] │
│   3 │ [1,1,1] │
│   4 │ [1,1,1] │
│   5 │ [0,0,0] │
│   6 │ [0,0,0] │
│   7 │ [0,0,0] │
│   8 │ [0,0,0] │
│   9 │ [0,0,0] │
│  10 │ [0,0,0] │
│  11 │ [0,0,0] │
│  12 │ [0,0,0] │
│  13 │ [0,0,0] │
│  14 │ [0,0,0] │
└─────┴─────────┘ **3.** Calculate the total number of site visits per day.

Query:

SELECT
    sum(r[1]) AS r1,
    sum(r[2]) AS r2,
    sum(r[3]) AS r3
FROM
(
    SELECT
        uid,
        retention(date = '2020-01-01', date = '2020-01-02', date = '2020-01-03') AS r
    FROM (
        SELECT '2020-01-01' as date, number as uid FROM numbers(5)  UNION ALL
        SELECT '2020-01-02' as date, number as uid FROM numbers(10) UNION ALL
        SELECT '2020-01-03' as date, number as uid FROM numbers(15))
    WHERE date IN ('2020-01-01', '2020-01-02', '2020-01-03')
    GROUP BY uid
) Result:

┌─r1─┬─r2─┬─r3─┐
│  5 │  5 │  5 │
└────┴────┴────┘ Where:

- `r1`   - the number of unique visitors who visited the site during 2020-01-01 (the `cond1`   condition).
- `r2`   - the number of unique visitors who visited the site during a specific time period between 2020-01-01 and 2020-01-02 ( `cond1`   and `cond2`   conditions).
- `r3`   - the number of unique visitors who visited the site during a specific time period on 2020-01-01 and 2020-01-03 ( `cond1`   and `cond3`   conditions).

## uniqUpTo(N)(x) [¶](https://www.tinybird.co/docs/about:blank#uniquptonx)

Calculates the number of different values of the argument up to a specified limit, `N` . If the number of different argument values is greater than `N` , this function returns `N` + 1, otherwise it calculates the exact value.

Recommended for use with small `N` s, up to 10. The maximum value of `N` is 100.

For the state of an aggregate function, this function uses the amount of memory equal to 1 + `N` * the size of one value of bytes. When dealing with strings, this function stores a non-cryptographic hash of 8 bytes; the calculation is approximated for strings.

For example, if you had a table that logs every search query made by users on your website. Each row in the table represents a single search query, with columns for the user ID, the search query, and the timestamp of the query. You can use `uniqUpTo` to generate a report that shows only the keywords that produced at least 5 unique users.

SELECT SearchPhrase
FROM SearchLog
GROUP BY SearchPhrase
HAVING uniqUpTo(4)(UserID) >= 5 `uniqUpTo(4)(UserID)` calculates the number of unique `UserID` values for each `SearchPhrase` , but it only counts up to 4 unique values. If there are more than 4 unique `UserID` values for a `SearchPhrase` , the function returns 5 (4 + 1). The `HAVING` clause then filters out the `SearchPhrase` values for which the number of unique `UserID` values is less than 5. This will give you a list of search keywords that were used by at least 5 unique users.

## sumMapFiltered [¶](https://www.tinybird.co/docs/about:blank#summapfiltered)

This function behaves the same as sumMap except that it also accepts an array of keys to filter with as a parameter. This can be especially useful when working with a high cardinality of keys.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

`sumMapFiltered(keys_to_keep)(keys, values)`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `keys_to_keep`   : Array of keys to filter with.
- `keys`   : Array of keys.
- `values`   : Array of values.

### Returned Value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT sumMapFiltered([1, 4, 8])(mapKeys(statusMap),mapValues(statusMap)) as summapfiltered
FROM (
  select c1::Date as date, c2::DateTime as timeslot, (c3::Array(UInt16), c4::Array(UInt64))::Map(UInt16, UInt64) as statusMap
  from values(
    ('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10])
  )
) Result:

┌─────summapfiltered───┐
│ ([1,4,8],[10,20,10]) │
└──────────────────────┘
## sumMapFilteredWithOverflow [¶](https://www.tinybird.co/docs/about:blank#summapfilteredwithoverflow)

This function behaves the same as sumMap except that it also accepts an array of keys to filter with as a parameter. This can be especially useful when working with a high cardinality of keys. It differs from the sumMapFiltered function in that it does summation with overflow. For example, it returns the same data type for the summation as the argument data type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

`sumMapFilteredWithOverflow(keys_to_keep)(keys, values)`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `keys_to_keep`   : Array of keys to filter with.
- `keys`   : Array of keys.
- `values`   : Array of values.

### Returned Value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

In this example you create a table `sum_map` , insert some data into it and then use both `sumMapFilteredWithOverflow` and `sumMapFiltered` and the `toTypeName` function for comparison of the result. Where `requests` was of type `UInt8` in the created table, `sumMapFiltered` has promoted the type of the summed values to `UInt64` to avoid overflow whereas `sumMapFilteredWithOverflow` has kept the type as `UInt8` which isn't large enough to store the result.

Query:

SELECT sumMapFilteredWithOverflow([1, 4, 8])(mapKeys(statusMap),mapValues(statusMap)) as summap_overflow, toTypeName(summap_overflow)
FROM (
  select c1::Date as date, c2::DateTime as timeslot, (c3::Array(UInt8), c4::Array(UInt8))::Map(UInt8, UInt8) as statusMap
  from values(
    ('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10])
  )
) Result:

┌─sum──────────────────┬─toTypeName(sum)───────────────────┐
│ ([1,4,8],[10,20,10]) │ Tuple(Array(UInt8), Array(UInt8)) │
└──────────────────────┴───────────────────────────────────┘ Query:

SELECT sumMapFiltered([1, 4, 8])(mapKeys(statusMap),mapValues(statusMap)) as summap, toTypeName(summap) 
FROM (
  select c1::Date as date, c2::DateTime as timeslot, (c3::Array(UInt8), c4::Array(UInt8))::Map(UInt8, UInt8) as statusMap
  from values(
    ('2000-01-01', '2000-01-01 00:00:00', [1, 2, 3], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:00:00', [3, 4, 5], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [4, 5, 6], [10, 10, 10]),
    ('2000-01-01', '2000-01-01 00:01:00', [6, 7, 8], [10, 10, 10])
  )
) Result:

┌─summap───────────────┬─toTypeName(summap)─────────────────┐
1. │ ([1,4,8],[10,20,10]) │ Tuple(Array(UInt8), Array(UInt64)) │
   └──────────────────────┴────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/other-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Other functions · Tinybird Docs"
theme-color: "#171612"
description: "Other available functions in Tinybird."
---




# Other functions [¶](https://www.tinybird.co/docs/about:blank#other-functions)

The following functions perform various operations in addition to the ones already described.

## basename [¶](https://www.tinybird.co/docs/about:blank#basename)

Extracts the tail of a string following its last slash or backslash. This function if often used to extract the filename from a path.

basename(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : A value of type String. Backslashes must be escaped.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A string that contains:

- The tail of the input string after its last slash or backslash. If the input string ends with a slash or backslash (e.g. `/`   or `c:\`   ), the function returns an empty string.
- The original string if there are no slashes or backslashes.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT 'some/long/path/to/file' AS a, basename(a) Result:

┌─a──────────────────────┬─basename('some\\long\\path\\to\\file')─┐
│ some\long\path\to\file │ file                                   │
└────────────────────────┴────────────────────────────────────────┘ Query:

SELECT 'some\\long\\path\\to\\file' AS a, basename(a) Result:

┌─a──────────────────────┬─basename('some\\long\\path\\to\\file')─┐
│ some\long\path\to\file │ file                                   │
└────────────────────────┴────────────────────────────────────────┘ Query:

SELECT 'some-file-name' AS a, basename(a) Result:

┌─a──────────────┬─basename('some-file-name')─┐
│ some-file-name │ some-file-name             │
└────────────────┴────────────────────────────┘
## visibleWidth [¶](https://www.tinybird.co/docs/about:blank#visiblewidth)

Calculates the approximate width when outputting values to the console in text format (tab-separated). This function is used by the system to implement Pretty formats.

`NULL` is represented as a string corresponding to `NULL` in `Pretty` formats.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

visibleWidth(x)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT visibleWidth(NULL) Result:

┌─visibleWidth(NULL)─┐
│                  4 │
└────────────────────┘
## toTypeName [¶](https://www.tinybird.co/docs/about:blank#totypename)

Returns the type name of the passed argument.

If `NULL` is passed, the function returns type `Nullable(Nothing)` , which corresponds to Tinybird's internal `NULL` representation.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toTypeName(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : A value of arbitrary type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The data type name of the input value. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT toTypeName(123) Result:

┌─toTypeName(123)─┐
│ UInt8           │
└─────────────────┘
## blockSize [¶](https://www.tinybird.co/docs/about:blank#blocksize)

In Tinybird, queries are processed in blocks (chunks). This function returns the size (row count) of the block the function is called on.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

blockSize()
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT blockSize() FROM (
    SELECT * FROM numbers(5)) Result:

┌─blockSize()─┐
│           5 │
│           5 │
│           5 │
│           5 │
│           5 │
└─────────────┘
## byteSize [¶](https://www.tinybird.co/docs/about:blank#bytesize)

Returns an estimation of uncompressed byte size of its arguments in memory.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

byteSize(argument [, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `argument`   : Value.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Estimation of byte size of the arguments in memory. UInt64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

For String arguments, the function returns the string length + 9 (terminating zero + length).

Query:

SELECT byteSize('string') Result:

┌─byteSize('string')─┐
│                 15 │
└────────────────────┘ Query:

SELECT key, byteSize(u8) AS `byteSize(UInt8)`, byteSize(u16) AS `byteSize(UInt16)`, byteSize(u32) AS `byteSize(UInt32)`, byteSize(u64) AS `byteSize(UInt64)`, byteSize(i8) AS `byteSize(Int8)`, byteSize(i16) AS `byteSize(Int16)`, byteSize(i32) AS `byteSize(Int32)`, byteSize(i64) AS `byteSize(Int64)`, byteSize(f32) AS `byteSize(Float32)`, byteSize(f64) AS `byteSize(Float64)` 
FROM (select c1::Int32 as key, c2::UInt8 as u8, c3::UInt16 as u16, c4::UInt32 as u32, c5::UInt64 as u64, c6::Int8 as i8, c7::Int16 as i16, c8::Int32 as i32, c9::Int64 as i64, c10::Float32 as f32, c11::Float64 as f64 
from values((1, 8, 16, 32, 64,  -8, -16, -32, -64, 32.32, 64.64))) Result:

key:               1
byteSize(UInt8):   1
byteSize(UInt16):  2
byteSize(UInt32):  4
byteSize(UInt64):  8
byteSize(Int8):    1
byteSize(Int16):   2
byteSize(Int32):   4
byteSize(Int64):   8
byteSize(Float32): 4
byteSize(Float64): 8 If the function has multiple arguments, the function accumulates their byte sizes.

Query:

SELECT byteSize(NULL, 1, 0.3, '') Result:

┌─byteSize(NULL, 1, 0.3, '')─┐
│                         19 │
└────────────────────────────┘
## materialize [¶](https://www.tinybird.co/docs/about:blank#materialize)

Turns a constant into a full column containing a single value. Full columns and constants are represented differently in memory. Functions usually execute different code for normal and constant arguments, although the result should typically be the same. This function can be used to debug this behavior.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

materialize(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : A constant. Constant.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A column containing a single value `x`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

In the example below the `countMatches` function expects a constant second argument. This behaviour can be debugged by using the `materialize` function to turn a constant into a full column, verifying that the function throws an error for a non-constant argument.

Query:

SELECT countMatches('foobarfoo', 'foo')
SELECT countMatches('foobarfoo', materialize('foo')) Result:

2
Code: 44. DB::Exception: Received from localhost:9000. DB::Exception: Illegal type of argument #2 'pattern' of function countMatches, expected constant String, got String
## ignore [¶](https://www.tinybird.co/docs/about:blank#ignore)

Accepts arbitrary arguments and unconditionally returns `0`. The argument is still evaluated internally, making it useful for eg. benchmarking.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ignore([arg1[, arg2[, ...]])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- Accepts arbitrarily many arguments of arbitrary type, including `NULL`  .

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `0`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ignore(0, 'Tinybird', NULL) Result:

┌─ignore(0, 'Tinybird', NULL)─┐
│                             0 │
└───────────────────────────────┘
## isConstant [¶](https://www.tinybird.co/docs/about:blank#isconstant)

Returns whether the argument is a constant expression.

A constant expression is an expression whose result is known during query analysis, i.e. before execution. For example, expressions over literals are constant expressions.

This function is mostly intended for development, debugging and demonstration.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isConstant(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Expression to check.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `1`   if `x`   is constant. UInt8.
- `0`   if `x`   is non-constant. UInt8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT isConstant(x + 1) FROM (SELECT 43 AS x) Result:

┌─isConstant(plus(x, 1))─┐
│                      1 │
└────────────────────────┘ Query:

WITH 3.14 AS pi SELECT isConstant(cos(pi)) Result:

┌─isConstant(cos(pi))─┐
│                   1 │
└─────────────────────┘ Query:

SELECT isConstant(number) FROM numbers(1) Result:

┌─isConstant(number)─┐
│                  0 │
└────────────────────┘
## bar [¶](https://www.tinybird.co/docs/about:blank#bar)

Builds a bar chart.

`bar(x, min, max, width)` draws a band with width proportional to `(x - min)` and equal to `width` characters when `x = max`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Size to display.
- `min, max`   : Integer constants. The value must fit in `Int64`  .
- `width`   : Constant, positive integer, can be fractional.

The band is drawn with accuracy to one eighth of a symbol.

Example:

SELECT
    toHour(EventTime) AS h,
    count() AS c,
    bar(c, 0, 600000, 20) AS bar
FROM test.hits
GROUP BY h
ORDER BY h ASC ┌──h─┬──────c─┬─bar────────────────┐
│  0 │ 292907 │ █████████▋         │
│  1 │ 180563 │ ██████             │
│  2 │ 114861 │ ███▋               │
│  3 │  85069 │ ██▋                │
│  4 │  68543 │ ██▎                │
│  5 │  78116 │ ██▌                │
│  6 │ 113474 │ ███▋               │
│  7 │ 170678 │ █████▋             │
│  8 │ 278380 │ █████████▎         │
│  9 │ 391053 │ █████████████      │
│ 10 │ 457681 │ ███████████████▎   │
│ 11 │ 493667 │ ████████████████▍  │
│ 12 │ 509641 │ ████████████████▊  │
│ 13 │ 522947 │ █████████████████▍ │
│ 14 │ 539954 │ █████████████████▊ │
│ 15 │ 528460 │ █████████████████▌ │
│ 16 │ 539201 │ █████████████████▊ │
│ 17 │ 523539 │ █████████████████▍ │
│ 18 │ 506467 │ ████████████████▊  │
│ 19 │ 520915 │ █████████████████▎ │
│ 20 │ 521665 │ █████████████████▍ │
│ 21 │ 542078 │ ██████████████████ │
│ 22 │ 493642 │ ████████████████▍  │
│ 23 │ 400397 │ █████████████▎     │
└────┴────────┴────────────────────┘
## transform [¶](https://www.tinybird.co/docs/about:blank#transform)

Transforms a value according to the explicitly defined mapping of some elements to other ones. There are two variations of this function:

### transform(x, array_from, array_to, default) [¶](https://www.tinybird.co/docs/about:blank#transformx-array-from-array-to-default)

`x` – What to transform.

`array_from` – Constant array of values to convert.

`array_to` – Constant array of values to convert the values in ‘from’ to.

`default` – Which value to use if ‘x’ isn't equal to any of the values in ‘from’.

`array_from` and `array_to` must have equally many elements.

Signature:

For `x` equal to one of the elements in `array_from` , the function returns the corresponding element in `array_to` , i.e. the one at the same array index. Otherwise, it returns `default` . If multiple matching elements exist `array_from` , it returns the element corresponding to the first of them.

`transform(T, Array(T), Array(U), U) -> U`

`T` and `U` can be numeric, string, or Date or DateTime types. The same letter (T or U) means that types must be mutually compatible and not necessarily equal. For example, the first argument could have type `Int64` , while the second argument could have type `Array(UInt16)`.

Example:

SELECT
    transform(SearchEngineID, [2, 3], ['Yandex', 'Google'], 'Other') AS title,
    count() AS c
FROM test.hits
WHERE SearchEngineID != 0
GROUP BY title
ORDER BY c DESC ┌─title─────┬──────c─┐
│ Yandex    │ 498635 │
│ Google    │ 229872 │
│ Other     │ 104472 │
└───────────┴────────┘
### transform(x, array_from, array_to) [¶](https://www.tinybird.co/docs/about:blank#transformx-array-from-array-to)

Similar to the other variation but has no ‘default’ argument. In case no match can be found, `x` is returned.

Example:

SELECT
    transform(domain(Referer), ['yandex.ru', 'google.ru', 'vkontakte.ru'], ['www.yandex', 'example.com', 'vk.com']) AS s,
    count() AS c
FROM test.hits
GROUP BY domain(Referer)
ORDER BY count() DESC
LIMIT 10 ┌─s──────────────┬───────c─┐
│                │ 2906259 │
│ www.yandex     │  867767 │
│ ███████.ru     │  313599 │
│ mail.yandex.ru │  107147 │
│ ██████.ru      │  100355 │
│ █████████.ru   │   65040 │
│ news.yandex.ru │   64515 │
│ ██████.net     │   59141 │
│ example.com    │   57316 │
└────────────────┴─────────┘
## formatReadableDecimalSize [¶](https://www.tinybird.co/docs/about:blank#formatreadabledecimalsize)

Given a size (number of bytes), this function returns a readable, rounded size with suffix (KB, MB, etc.) as string.

The opposite operations of this function are parseReadableSize, parseReadableSizeOrZero, and parseReadableSizeOrNull.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatReadableDecimalSize(x)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    arrayJoin([1, 1024, 1024*1024, 192851925]) AS filesize_bytes,
    formatReadableDecimalSize(filesize_bytes) AS filesize Result:

┌─filesize_bytes─┬─filesize───┐
│              1 │ 1.00 B     │
│           1024 │ 1.02 KB   │
│        1048576 │ 1.05 MB   │
│      192851925 │ 192.85 MB │
└────────────────┴────────────┘
## formatReadableSize [¶](https://www.tinybird.co/docs/about:blank#formatreadablesize)

Given a size (number of bytes), this function returns a readable, rounded size with suffix (KiB, MiB, etc.) as string.

The opposite operations of this function are parseReadableSize, parseReadableSizeOrZero, and parseReadableSizeOrNull.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatReadableSize(x) Alias: `FORMAT_BYTES`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    arrayJoin([1, 1024, 1024*1024, 192851925]) AS filesize_bytes,
    formatReadableSize(filesize_bytes) AS filesize Result:

┌─filesize_bytes─┬─filesize───┐
│              1 │ 1.00 B     │
│           1024 │ 1.00 KiB   │
│        1048576 │ 1.00 MiB   │
│      192851925 │ 183.92 MiB │
└────────────────┴────────────┘
## formatReadableQuantity [¶](https://www.tinybird.co/docs/about:blank#formatreadablequantity)

Given a number, this function returns a rounded number with suffix (thousand, million, billion, etc.) as string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatReadableQuantity(x)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    arrayJoin([1024, 1234 * 1000, (4567 * 1000) * 1000, 98765432101234]) AS number,
    formatReadableQuantity(number) AS number_for_humans Result:

┌─────────number─┬─number_for_humans─┐
│           1024 │ 1.02 thousand     │
│        1234000 │ 1.23 million      │
│     4567000000 │ 4.57 billion      │
│ 98765432101234 │ 98.77 trillion    │
└────────────────┴───────────────────┘
## formatReadableTimeDelta [¶](https://www.tinybird.co/docs/about:blank#formatreadabletimedelta)

Given a time interval (delta) in seconds, this function returns a time delta with year/month/day/hour/minute/second/millisecond/microsecond/nanosecond as string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatReadableTimeDelta(column[, maximum_unit, minimum_unit])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : A column with a numeric time delta.
- `maximum_unit`   : Optional. Maximum unit to show.
  - Acceptable values: `nanoseconds`    , `microseconds`    , `milliseconds`    , `seconds`    , `minutes`    , `hours`    , `days`    , `months`    , `years`    .
  - Default value: `years`    .
- `minimum_unit`   : Optional. Minimum unit to show. All smaller units are truncated.
  - Acceptable values: `nanoseconds`    , `microseconds`    , `milliseconds`    , `seconds`    , `minutes`    , `hours`    , `days`    , `months`    , `years`    .
  - If explicitly specified value is bigger than `maximum_unit`     , an exception will be thrown.
  - Default value: `seconds`     if `maximum_unit`     is `seconds`     or bigger, `nanoseconds`     otherwise.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    arrayJoin([100, 12345, 432546534]) AS elapsed,
    formatReadableTimeDelta(elapsed) AS time_delta ┌────elapsed─┬─time_delta ─────────────────────────────────────────────────────┐
│        100 │ 1 minute and 40 seconds                                         │
│      12345 │ 3 hours, 25 minutes and 45 seconds                              │
│  432546534 │ 13 years, 8 months, 17 days, 7 hours, 48 minutes and 54 seconds │
└────────────┴─────────────────────────────────────────────────────────────────┘ SELECT
    arrayJoin([100, 12345, 432546534]) AS elapsed,
    formatReadableTimeDelta(elapsed, 'minutes') AS time_delta ┌────elapsed─┬─time_delta ─────────────────────────────────────────────────────┐
│        100 │ 1 minute and 40 seconds                                         │
│      12345 │ 205 minutes and 45 seconds                                      │
│  432546534 │ 7209108 minutes and 54 seconds                                  │
└────────────┴─────────────────────────────────────────────────────────────────┘ SELECT
    arrayJoin([100, 12345, 432546534.00000006]) AS elapsed,
    formatReadableTimeDelta(elapsed, 'minutes', 'nanoseconds') AS time_delta ┌────────────elapsed─┬─time_delta─────────────────────────────────────┐
│                100 │ 1 minute and 40 seconds                        │
│              12345 │ 205 minutes and 45 seconds                     │
│ 432546534.00000006 │ 7209108 minutes, 54 seconds and 60 nanoseconds │
└────────────────────┴────────────────────────────────────────────────┘
## parseReadableSize [¶](https://www.tinybird.co/docs/about:blank#parsereadablesize)

Given a string containing a byte size and `B`, `KiB`, `KB`, `MiB`, `MB` , etc. as a unit (i.e. ISO/IEC 80000-13 or decimal byte unit), this function returns the corresponding number of bytes.
If the function is unable to parse the input value, it throws an exception.

The inverse operations of this function are formatReadableSize and formatReadableDecimalSize.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatReadableSize(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Readable size with ISO/IEC 80000-13 or decimal byte unit (String).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of bytes, rounded up to the nearest integer (UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    arrayJoin(['1 B', '1 KiB', '3 MB', '5.314 KiB']) AS readable_sizes,  
    parseReadableSize(readable_sizes) AS sizes ┌─readable_sizes─┬───sizes─┐
│ 1 B            │       1 │
│ 1 KiB          │    1024 │
│ 3 MB           │ 3000000 │
│ 5.314 KiB      │    5442 │
└────────────────┴─────────┘
## parseReadableSizeOrNull [¶](https://www.tinybird.co/docs/about:blank#parsereadablesizeornull)

Given a string containing a byte size and `B`, `KiB`, `KB`, `MiB`, `MB` , etc. as a unit (i.e. ISO/IEC 80000-13 or decimal byte unit), this function returns the corresponding number of bytes.
If the function is unable to parse the input value, it returns `NULL`.

The inverse operations of this function are formatReadableSize and formatReadableDecimalSize.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

parseReadableSizeOrNull(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Readable size with ISO/IEC 80000-13  or decimal byte unit (String).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of bytes, rounded up to the nearest integer, or NULL if unable to parse the input (Nullable(UInt64)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    arrayJoin(['1 B', '1 KiB', '3 MB', '5.314 KiB', 'invalid']) AS readable_sizes,  
    parseReadableSizeOrNull(readable_sizes) AS sizes ┌─readable_sizes─┬───sizes─┐
│ 1 B            │       1 │
│ 1 KiB          │    1024 │
│ 3 MB           │ 3000000 │
│ 5.314 KiB      │    5442 │
│ invalid        │    ᴺᵁᴸᴸ │
└────────────────┴─────────┘
## parseReadableSizeOrZero [¶](https://www.tinybird.co/docs/about:blank#parsereadablesizeorzero)

Given a string containing a byte size and `B`, `KiB`, `KB`, `MiB`, `MB` , etc. as a unit (i.e. ISO/IEC 80000-13 or decimal byte unit), this function returns the corresponding number of bytes. If the function is unable to parse the input value, it returns `0`.

The inverse operations of this function are formatReadableSize and formatReadableDecimalSize.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

parseReadableSizeOrZero(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Readable size with ISO/IEC 80000-13  or decimal byte unit  (String).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of bytes, rounded up to the nearest integer, or 0 if unable to parse the input (UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    arrayJoin(['1 B', '1 KiB', '3 MB', '5.314 KiB', 'invalid']) AS readable_sizes,  
    parseReadableSizeOrZero(readable_sizes) AS sizes ┌─readable_sizes─┬───sizes─┐
│ 1 B            │       1 │
│ 1 KiB          │    1024 │
│ 3 MB           │ 3000000 │
│ 5.314 KiB      │    5442 │
│ invalid        │       0 │
└────────────────┴─────────┘
## parseTimeDelta [¶](https://www.tinybird.co/docs/about:blank#parsetimedelta)

Parse a sequence of numbers followed by something resembling a time unit.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

parseTimeDelta(timestr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timestr`   : A sequence of numbers followed by something resembling a time unit.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A floating-point number with the number of seconds.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT parseTimeDelta('11s+22min') ┌─parseTimeDelta('11s+22min')─┐
│                        1331 │
└─────────────────────────────┘ SELECT parseTimeDelta('1yr2mo') ┌─parseTimeDelta('1yr2mo')─┐
│                 36806400 │
└──────────────────────────┘
## least [¶](https://www.tinybird.co/docs/about:blank#least)

Returns the smaller value of a and b.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

least(a, b)
## greatest [¶](https://www.tinybird.co/docs/about:blank#greatest)

Returns the larger value of a and b.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

greatest(a, b)
## blockNumber [¶](https://www.tinybird.co/docs/about:blank#blocknumber)

Returns a monotonically increasing sequence number of the block containing the row. The returned block number is updated on a best-effort basis, i.e. it may not be fully accurate.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

blockNumber()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Sequence number of the data block where the row is located. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT blockNumber()
FROM
(
    SELECT *
    FROM system.numbers
    LIMIT 10
) SETTINGS max_block_size = 2 Result:

┌─blockNumber()─┐
│             7 │
│             7 │
└───────────────┘
┌─blockNumber()─┐
│             8 │
│             8 │
└───────────────┘
┌─blockNumber()─┐
│             9 │
│             9 │
└───────────────┘
┌─blockNumber()─┐
│            10 │
│            10 │
└───────────────┘
┌─blockNumber()─┐
│            11 │
│            11 │
└───────────────┘
## rowNumberInBlock [¶](https://www.tinybird.co/docs/about:blank#rownumberinblock)

Returns for each block processed by `rowNumberInBlock` the number of the current row. The returned number starts for each block at 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

rowNumberInBlock()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Ordinal number of the row in the data block starting from 0. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT rowNumberInBlock()
FROM
(
    SELECT *
    FROM system.numbers_mt
    LIMIT 10
) SETTINGS max_block_size = 2 Result:

┌─rowNumberInBlock()─┐
│                  0 │
│                  1 │
└────────────────────┘
┌─rowNumberInBlock()─┐
│                  0 │
│                  1 │
└────────────────────┘
┌─rowNumberInBlock()─┐
│                  0 │
│                  1 │
└────────────────────┘
┌─rowNumberInBlock()─┐
│                  0 │
│                  1 │
└────────────────────┘
┌─rowNumberInBlock()─┐
│                  0 │
│                  1 │
└────────────────────┘
## rowNumberInAllBlocks [¶](https://www.tinybird.co/docs/about:blank#rownumberinallblocks)

Returns a unique row number for each row processed by `rowNumberInAllBlocks` . The returned numbers start at 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

rowNumberInAllBlocks()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Ordinal number of the row in the data block starting from 0. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT rowNumberInAllBlocks()
FROM
(
    SELECT *
    FROM system.numbers_mt
    LIMIT 10
)
SETTINGS max_block_size = 2 Result:

┌─rowNumberInAllBlocks()─┐
│                      0 │
│                      1 │
└────────────────────────┘
┌─rowNumberInAllBlocks()─┐
│                      4 │
│                      5 │
└────────────────────────┘
┌─rowNumberInAllBlocks()─┐
│                      2 │
│                      3 │
└────────────────────────┘
┌─rowNumberInAllBlocks()─┐
│                      6 │
│                      7 │
└────────────────────────┘
┌─rowNumberInAllBlocks()─┐
│                      8 │
│                      9 │
└────────────────────────┘
## neighbor [¶](https://www.tinybird.co/docs/about:blank#neighbor)

The window function that provides access to a row at a specified offset before or after the current row of a given column.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

neighbor(column, offset[, default_value]) The result of the function depends on the affected data blocks and the order of data in the block.

Only returns neighbor inside the currently processed data block. Because of this error-prone behavior the function is DEPRECATED, please use proper window functions instead.

The order of rows during calculation of `neighbor()` can differ from the order of rows returned to the user. To prevent that you can create a subquery with ORDER BY and call the function from outside the subquery.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : A column name or scalar expression.
- `offset`   : The number of rows to look before or ahead of the current row in `column`   . Int64.
- `default_value`   : Optional. The returned value if offset is beyond the block boundaries. Type of data blocks affected.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Value of `column`   with `offset`   distance from current row, if `offset`   isn't outside the block boundaries.
- The default value of `column`   or `default_value`   (if given), if `offset`   is outside the block boundaries.

The return type will be that of the data blocks affected or the default value type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT number, neighbor(number, 2) FROM system.numbers LIMIT 10 Result:

┌─number─┬─neighbor(number, 2)─┐
│      0 │                   2 │
│      1 │                   3 │
│      2 │                   4 │
│      3 │                   5 │
│      4 │                   6 │
│      5 │                   7 │
│      6 │                   8 │
│      7 │                   9 │
│      8 │                   0 │
│      9 │                   0 │
└────────┴─────────────────────┘ Query:

SELECT number, neighbor(number, 2, 999) FROM system.numbers LIMIT 10 Result:

┌─number─┬─neighbor(number, 2, 999)─┐
│      0 │                        2 │
│      1 │                        3 │
│      2 │                        4 │
│      3 │                        5 │
│      4 │                        6 │
│      5 │                        7 │
│      6 │                        8 │
│      7 │                        9 │
│      8 │                      999 │
│      9 │                      999 │
└────────┴──────────────────────────┘ This function can be used to compute year-over-year metric value:

Query:

WITH toDate('2018-01-01') AS start_date
SELECT
    toStartOfMonth(start_date + (number * 32)) AS month,
    toInt32(month) % 100 AS money,
    neighbor(money, -12) AS prev_year,
    round(prev_year / money, 2) AS year_over_year
FROM numbers(16) Result:

┌──────month─┬─money─┬─prev_year─┬─year_over_year─┐
│ 2018-01-01 │    32 │         0 │              0 │
│ 2018-02-01 │    63 │         0 │              0 │
│ 2018-03-01 │    91 │         0 │              0 │
│ 2018-04-01 │    22 │         0 │              0 │
│ 2018-05-01 │    52 │         0 │              0 │
│ 2018-06-01 │    83 │         0 │              0 │
│ 2018-07-01 │    13 │         0 │              0 │
│ 2018-08-01 │    44 │         0 │              0 │
│ 2018-09-01 │    75 │         0 │              0 │
│ 2018-10-01 │     5 │         0 │              0 │
│ 2018-11-01 │    36 │         0 │              0 │
│ 2018-12-01 │    66 │         0 │              0 │
│ 2019-01-01 │    97 │        32 │           0.33 │
│ 2019-02-01 │    28 │        63 │           2.25 │
│ 2019-03-01 │    56 │        91 │           1.62 │
│ 2019-04-01 │    87 │        22 │           0.25 │
└────────────┴───────┴───────────┴────────────────┘
## runningDifference [¶](https://www.tinybird.co/docs/about:blank#runningdifference)

Calculates the difference between two consecutive row values in the data block. Returns 0 for the first row, and for subsequent rows the difference to the previous row.

Only returns differences inside the currently processed data block. Because of this error-prone behavior the function is DEPRECATED, please use proper window functions instead.

The result of the function depends on the affected data blocks and the order of data in the block.

The order of rows during calculation of `runningDifference()` can differ from the order of rows returned to the user. To prevent that you can create a subquery with ORDER BY and call the function from outside the subquery.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

runningDifference(x)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    EventID,
    EventTime,
    runningDifference(EventTime) AS delta
FROM
(
    SELECT
        EventID,
        EventTime
    FROM events
    WHERE EventDate = '2016-11-24'
    ORDER BY EventTime ASC
    LIMIT 5
) Result:

┌─EventID─┬───────────EventTime─┬─delta─┐
│    1106 │ 2016-11-24 00:00:04 │     0 │
│    1107 │ 2016-11-24 00:00:05 │     1 │
│    1108 │ 2016-11-24 00:00:05 │     0 │
│    1109 │ 2016-11-24 00:00:09 │     4 │
│    1110 │ 2016-11-24 00:00:10 │     1 │
└─────────┴─────────────────────┴───────┘ Please note that the block size affects the result. The internal state of `runningDifference` state is reset for each new block.

Query:

SELECT
    number,
    runningDifference(number + 1) AS diff
FROM numbers(100000)
WHERE diff != 1 Result:

┌─number─┬─diff─┐
│      0 │    0 │
└────────┴──────┘
┌─number─┬─diff─┐
│  65536 │    0 │
└────────┴──────┘ Query:

set max_block_size=100000 -- default value is 65536!

SELECT
    number,
    runningDifference(number + 1) AS diff
FROM numbers(100000)
WHERE diff != 1 Result:

┌─number─┬─diff─┐
│      0 │    0 │
└────────┴──────┘
## runningDifferenceStartingWithFirstValue [¶](https://www.tinybird.co/docs/about:blank#runningdifferencestartingwithfirstvalue)

This function is DEPRECATED (see the note for `runningDifference` ).

Same as runningDifference, but returns the value of the first row as the value on the first row.

## runningConcurrency [¶](https://www.tinybird.co/docs/about:blank#runningconcurrency)

Calculates the number of concurrent events. Each event has a start time and an end time. The start time is included in the event, while the end time is excluded. Columns with a start time and an end time must be of the same data type. The function calculates the total number of active (concurrent) events for each event start time.



tip Events must be ordered by the start time in ascending order. If this requirement is violated the function raises an exception. Every data block is processed separately. If events from different data blocks overlap then they can't be processed correctly.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

runningConcurrency(start, end)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start`   : A column with the start time of events. Date, DateTime, or DateTime64.
- `end`   : A column with the end time of events. Date, DateTime, or DateTime64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The number of concurrent events at each event start time. UInt32

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Consider the table:

┌──────start─┬────────end─┐
│ 2021-03-03 │ 2021-03-11 │
│ 2021-03-06 │ 2021-03-12 │
│ 2021-03-07 │ 2021-03-08 │
│ 2021-03-11 │ 2021-03-12 │
└────────────┴────────────┘ Query:

SELECT start, runningConcurrency(start, end) FROM example_table Result:

┌──────start─┬─runningConcurrency(start, end)─┐
│ 2021-03-03 │                              1 │
│ 2021-03-06 │                              2 │
│ 2021-03-07 │                              3 │
│ 2021-03-11 │                              2 │
└────────────┴────────────────────────────────┘
## MACNumToString [¶](https://www.tinybird.co/docs/about:blank#macnumtostring)

Interprets a UInt64 number as a MAC address in big endian format. Returns the corresponding MAC address in format AA:BB:CC:DD:EE:FF (colon-separated numbers in hexadecimal form) as string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

MACNumToString(num)
## MACStringToNum [¶](https://www.tinybird.co/docs/about:blank#macstringtonum)

The inverse function of MACNumToString. If the MAC address has an invalid format, it returns 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

MACStringToNum(s)
## MACStringToOUI [¶](https://www.tinybird.co/docs/about:blank#macstringtooui)

Given a MAC address in format AA:BB:CC:DD:EE:FF (colon-separated numbers in hexadecimal form), returns the first three octets as a UInt64 number. If the MAC address has an invalid format, it returns 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

MACStringToOUI(s)
## getSizeOfEnumType [¶](https://www.tinybird.co/docs/about:blank#getsizeofenumtype)

Returns the number of fields in Enum. An exception is thrown if the type isn't `Enum`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

getSizeOfEnumType(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Value of type `Enum`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The number of fields with `Enum`   input values.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT getSizeOfEnumType( CAST('a' AS Enum8('a' = 1, 'b' = 2) ) ) AS x ┌─x─┐
│ 2 │
└───┘
## blockSerializedSize [¶](https://www.tinybird.co/docs/about:blank#blockserializedsize)

Returns the size on disk without considering compression.

blockSerializedSize(value[, value[, ...]])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Any value.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The number of bytes that will be written to disk for block of values without compression.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT blockSerializedSize(maxState(1)) as x Result:

┌─x─┐
│ 2 │
└───┘
## toColumnTypeName [¶](https://www.tinybird.co/docs/about:blank#tocolumntypename)

Returns the internal name of the data type that represents the value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toColumnTypeName(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Any type of value.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The internal data type name used to represent `value`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Difference between `toTypeName` and `toColumnTypeName`:

SELECT toTypeName(CAST('2018-01-01 01:02:03' AS DateTime)) Result:

┌─toTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ DateTime                                            │
└─────────────────────────────────────────────────────┘ Query:

SELECT toColumnTypeName(CAST('2018-01-01 01:02:03' AS DateTime)) Result:

┌─toColumnTypeName(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ Const(UInt32)                                             │
└───────────────────────────────────────────────────────────┘ The example shows that the `DateTime` data type is internally stored as `Const(UInt32)`.

## dumpColumnStructure [¶](https://www.tinybird.co/docs/about:blank#dumpcolumnstructure)

Outputs a detailed description of data structures in RAM

dumpColumnStructure(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Any type of value.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- A description of the column structure used for representing `value`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT dumpColumnStructure(CAST('2018-01-01 01:02:03', 'DateTime')) ┌─dumpColumnStructure(CAST('2018-01-01 01:02:03', 'DateTime'))─┐
│ DateTime, Const(size = 1, UInt32(size = 1))                  │
└──────────────────────────────────────────────────────────────┘
## defaultValueOfArgumentType [¶](https://www.tinybird.co/docs/about:blank#defaultvalueofargumenttype)

Returns the default value for the given data type.

Does not include default values for custom columns set by the user.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

defaultValueOfArgumentType(expression)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expression`   : Arbitrary type of value or an expression that results in a value of an arbitrary type.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `0`   for numbers.
- Empty string for strings.
- `ᴺᵁᴸᴸ`   for Nullable.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT defaultValueOfArgumentType( CAST(1 AS Int8) ) Result:

┌─defaultValueOfArgumentType(CAST(1, 'Int8'))─┐
│                                           0 │
└─────────────────────────────────────────────┘ Query:

SELECT defaultValueOfArgumentType( CAST(1 AS Nullable(Int8) ) ) Result:

┌─defaultValueOfArgumentType(CAST(1, 'Nullable(Int8)'))─┐
│                                                  ᴺᵁᴸᴸ │
└───────────────────────────────────────────────────────┘
## defaultValueOfTypeName [¶](https://www.tinybird.co/docs/about:blank#defaultvalueoftypename)

Returns the default value for the given type name.

Does not include default values for custom columns set by the user.

defaultValueOfTypeName(type)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `type`   : A string representing a type name.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `0`   for numbers.
- Empty string for strings.
- `ᴺᵁᴸᴸ`   for Nullable.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT defaultValueOfTypeName('Int8') Result:

┌─defaultValueOfTypeName('Int8')─┐
│                              0 │
└────────────────────────────────┘ Query:

SELECT defaultValueOfTypeName('Nullable(Int8)') Result:

┌─defaultValueOfTypeName('Nullable(Int8)')─┐
│                                     ᴺᵁᴸᴸ │
└──────────────────────────────────────────┘
## replicate [¶](https://www.tinybird.co/docs/about:blank#replicate)

Creates an array with a single value.

This function is used for the internal implementation of arrayJoin.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

replicate(x, arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The value to fill the result array with.
- `arr`   : An array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An array of the lame length as `arr` filled with value `x` . Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT replicate(1, ['a', 'b', 'c']) Result:

┌─replicate(1, ['a', 'b', 'c'])─┐
│ [1,1,1]                       │
└───────────────────────────────┘
## initializeAggregation [¶](https://www.tinybird.co/docs/about:blank#initializeaggregation)

Calculates the result of an aggregate function based on a single value. This function can be used to initialize aggregate functions with combinator -State. You can create states of aggregate functions and insert them to columns of type AggregateFunction or use initialized aggregates as default values.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

initializeAggregation (aggregate_function, arg1, arg2, ..., argN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `aggregate_function`   : Name of the aggregation function to initialize. String.
- `arg`   : Arguments of aggregate function.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Result of aggregation for every row passed to the function.

The return type is the same as the return type of function, that `initializeAggregation` takes as first argument.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT uniqMerge(state) FROM (SELECT initializeAggregation('uniqState', number % 3) AS state FROM numbers(10000)) Result:

┌─uniqMerge(state)─┐
│                3 │
└──────────────────┘ Query:

SELECT finalizeAggregation(state), toTypeName(state) FROM (SELECT initializeAggregation('sumState', number % 3) AS state FROM numbers(5)) Result:

┌─finalizeAggregation(state)─┬─toTypeName(state)─────────────┐
│                          0 │ AggregateFunction(sum, UInt8) │
│                          1 │ AggregateFunction(sum, UInt8) │
│                          2 │ AggregateFunction(sum, UInt8) │
│                          0 │ AggregateFunction(sum, UInt8) │
│                          1 │ AggregateFunction(sum, UInt8) │
└────────────────────────────┴───────────────────────────────┘ Example with `AggregatingMergeTree` table engine and `AggregateFunction` column:

##### metrics.datasource

SCHEMA >
    key UInt64,
    value AggregateFunction(sum, UInt64) DEFAULT initializeAggregation('sumState', toUInt64(0))

ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "key"
## finalizeAggregation [¶](https://www.tinybird.co/docs/about:blank#finalizeaggregation)

Given a state of aggregate function, this function returns the result of aggregation (or finalized state when using a -State combinator).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

finalizeAggregation(state)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `state`   : State of aggregation. AggregateFunction.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Value/values that was aggregated.

The return type is equal to that of any types which were aggregated.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT finalizeAggregation(( SELECT countState(number) FROM numbers(10))) Result:

┌─finalizeAggregation(_subquery16)─┐
│                               10 │
└──────────────────────────────────┘ Query:

SELECT finalizeAggregation(( SELECT sumState(number) FROM numbers(10))) Result:

┌─finalizeAggregation(_subquery20)─┐
│                               45 │
└──────────────────────────────────┘ Note that `NULL` values are ignored.

Query:

SELECT finalizeAggregation(arrayReduce('anyState', [NULL, 2, 3])) Result:

┌─finalizeAggregation(arrayReduce('anyState', [NULL, 2, 3]))─┐
│                                                          2 │
└────────────────────────────────────────────────────────────┘ Combined example:

Query:

WITH initializeAggregation('sumState', number) AS one_row_sum_state
SELECT
    number,
    finalizeAggregation(one_row_sum_state) AS one_row_sum,
    runningAccumulate(one_row_sum_state) AS cumulative_sum
FROM numbers(10) Result:

┌─number─┬─one_row_sum─┬─cumulative_sum─┐
│      0 │           0 │              0 │
│      1 │           1 │              1 │
│      2 │           2 │              3 │
│      3 │           3 │              6 │
│      4 │           4 │             10 │
│      5 │           5 │             15 │
│      6 │           6 │             21 │
│      7 │           7 │             28 │
│      8 │           8 │             36 │
│      9 │           9 │             45 │
└────────┴─────────────┴────────────────┘
## runningAccumulate [¶](https://www.tinybird.co/docs/about:blank#runningaccumulate)

Accumulates the states of an aggregate function for each row of a data block.

The state is reset for each new block of data. Because of this error-prone behavior the function is DEPRECATED, please use proper window functions instead.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

runningAccumulate(agg_state[, grouping])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `agg_state`   : State of the aggregate function. AggregateFunction.
- `grouping`   : Grouping key. Optional. The state of the function is reset if the `grouping`   value is changed. It can be any of the supported data types for which the equality operator is defined.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Each resulting row contains a result of the aggregate function, accumulated for all the input rows from 0 to the current position. `runningAccumulate`   resets states for each new data block or when the `grouping`   value changes.

Type depends on the aggregate function used.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Consider how you can use `runningAccumulate` to find the cumulative sum of numbers without and with grouping.

Query:

SELECT k, runningAccumulate(sum_k) AS res FROM (SELECT number as k, sumState(k) AS sum_k FROM numbers(10) GROUP BY k ORDER BY k) Result:

┌─k─┬─res─┐
│ 0 │   0 │
│ 1 │   1 │
│ 2 │   3 │
│ 3 │   6 │
│ 4 │  10 │
│ 5 │  15 │
│ 6 │  21 │
│ 7 │  28 │
│ 8 │  36 │
│ 9 │  45 │
└───┴─────┘ The subquery generates `sumState` for every number from `0` to `9`. `sumState` returns the state of the sum function that contains the sum of a single number.

The whole query does the following:

1. For the first row, `runningAccumulate`   takes `sumState(0)`   and returns `0`  .
2. For the second row, the function merges `sumState(0)`   and `sumState(1)`   resulting in `sumState(0 + 1)`   , and returns `1`   as a result.
3. For the third row, the function merges `sumState(0 + 1)`   and `sumState(2)`   resulting in `sumState(0 + 1 + 2)`   , and returns `3`   as a result.
4. The actions are repeated until the block ends.

The following example shows the `groupping` parameter usage:

Query:

SELECT
    grouping,
    item,
    runningAccumulate(state, grouping) AS res
FROM
(
    SELECT
        toInt8(number / 4) AS grouping,
        number AS item,
        sumState(number) AS state
    FROM numbers(15)
    GROUP BY item
    ORDER BY item ASC
) Result:

┌─grouping─┬─item─┬─res─┐
│        0 │    0 │   0 │
│        0 │    1 │   1 │
│        0 │    2 │   3 │
│        0 │    3 │   6 │
│        1 │    4 │   4 │
│        1 │    5 │   9 │
│        1 │    6 │  15 │
│        1 │    7 │  22 │
│        2 │    8 │   8 │
│        2 │    9 │  17 │
│        2 │   10 │  27 │
│        2 │   11 │  38 │
│        3 │   12 │  12 │
│        3 │   13 │  25 │
│        3 │   14 │  39 │
└──────────┴──────┴─────┘ As you can see, `runningAccumulate` merges states for each group of rows separately.

## throwIf [¶](https://www.tinybird.co/docs/about:blank#throwif)

Throw an exception if argument `x` is true.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

throwIf(x[, message[, error_code]])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - the condition to check.
- `message`   - a constant string providing a custom error message. Optional.
- `error_code`   - A constant integer providing a custom error code. Optional.

To use the `error_code` argument, configuration parameter `allow_custom_error_code_in_throwif` must be enabled.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT throwIf(number = 3, 'Too many') FROM numbers(10) Result:

↙ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.) Received exception from server (version 19.14.1):
Code: 395. DB::Exception: Received from localhost:9000. DB::Exception: Too many.
## identity [¶](https://www.tinybird.co/docs/about:blank#identity)

Returns its argument. Intended for debugging and testing. Allows to cancel using index, and get the query performance of a full scan. When the query is analyzed for possible use of an index, the analyzer ignores everything in `identity` functions. Also disables constant folding.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

identity(x)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT identity(42) Result:

┌─identity(42)─┐
│           42 │
└──────────────┘
## isDecimalOverflow [¶](https://www.tinybird.co/docs/about:blank#isdecimaloverflow)

Checks whether the Decimal value is outside its precision or outside the specified precision.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isDecimalOverflow(d, [p])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `d`   : value. Decimal.
- `p`   : precision. Optional. If omitted, the initial precision of the first argument is used. This parameter can be helpful to migrate data from/to another database or file. UInt8.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `1`   : Decimal value has more digits then allowed by its precision,
- `0`   : Decimal value satisfies the specified precision.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT isDecimalOverflow(toDecimal32(1000000000, 0), 9),
       isDecimalOverflow(toDecimal32(1000000000, 0)),
       isDecimalOverflow(toDecimal32(-1000000000, 0), 9),
       isDecimalOverflow(toDecimal32(-1000000000, 0)) Result:

1	1	1	1
## countDigits [¶](https://www.tinybird.co/docs/about:blank#countdigits)

Returns number of decimal digits need to represent a value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

countDigits(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Int or Decimal value.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of digits. UInt8.

For `Decimal` values takes into account their scales: calculates result over underlying integer type which is `(value * scale)` . For example: `countDigits(42) = 2`, `countDigits(42.000) = 5`, `countDigits(0.04200) = 4` . I.e. you may check decimal overflow for `Decimal64` with `countDecimal(x) > 18` . It's a slow variant of isDecimalOverflow.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT countDigits(toDecimal32(1, 9)), countDigits(toDecimal32(-1, 9)),
       countDigits(toDecimal64(1, 18)), countDigits(toDecimal64(-1, 18)),
       countDigits(toDecimal128(1, 38)), countDigits(toDecimal128(-1, 38)) Result:

10	10	19	19	39	39
## errorCodeToName [¶](https://www.tinybird.co/docs/about:blank#errorcodetoname)

- The textual name of an error code. LowCardinality(String).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

errorCodeToName(1) Result:

UNSUPPORTED_METHOD
## enabledProfiles [¶](https://www.tinybird.co/docs/about:blank#enabledprofiles)

Returns settings profiles, assigned to the current user both explicitly and implicitly. Explicitly assigned profiles are the same as returned by the currentProfiles function. Implicitly assigned profiles include parent profiles of other assigned profiles, profiles assigned via granted roles, profiles assigned via their own settings, and the main default profile (see the `default_profile` section in the main server configuration file).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

enabledProfiles()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- List of the enabled settings profiles. Array(String).

## minSampleSizeConversion [¶](https://www.tinybird.co/docs/about:blank#minsamplesizeconversion)

Calculates minimum required sample size for an A/B test comparing conversions (proportions) in two samples.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

minSampleSizeConversion(baseline, mde, power, alpha) Uses the formula described in this article. Assumes equal sizes of treatment and control groups. Returns the sample size required for one group (i.e. the sample size required for the whole experiment is twice the returned value).

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `baseline`   : Baseline conversion. Float.
- `mde`   : Minimum detectable effect (MDE) as percentage points (e.g. for a baseline conversion 0.25 the MDE 0.03 means an expected change to 0.25 ± 0.03). Float.
- `power`   : Required statistical power of a test (1 - probability of Type II error). Float.
- `alpha`   : Required significance level of a test (probability of Type I error). Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A named Tuple with 3 elements:

- `"minimum_sample_size"`   : Required sample size. Float64.
- `"detect_range_lower"`   : Lower bound of the range of values not detectable with the returned required sample size (i.e. all values less than or equal to `"detect_range_lower"`   are detectable with the provided `alpha`   and `power`   ). Calculated as `baseline - mde`   . Float64.
- `"detect_range_upper"`   : Upper bound of the range of values not detectable with the returned required sample size (i.e. all values greater than or equal to `"detect_range_upper"`   are detectable with the provided `alpha`   and `power`   ). Calculated as `baseline + mde`   . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following query calculates the required sample size for an A/B test with baseline conversion of 25%, MDE of 3%, significance level of 5%, and the desired statistical power of 80%:

SELECT minSampleSizeConversion(0.25, 0.03, 0.80, 0.05) AS sample_size Result:

┌─sample_size───────────────────┐
│ (3396.077603219163,0.22,0.28) │
└───────────────────────────────┘
## minSampleSizeContinuous [¶](https://www.tinybird.co/docs/about:blank#minsamplesizecontinuous)

Calculates minimum required sample size for an A/B test comparing means of a continuous metric in two samples.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

minSampleSizeContinous(baseline, sigma, mde, power, alpha) Alias: `minSampleSizeContinous`

Uses the formula described in this article. Assumes equal sizes of treatment and control groups. Returns the required sample size for one group (i.e. the sample size required for the whole experiment is twice the returned value). Also assumes equal variance of the test metric in treatment and control groups.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `baseline`   : Baseline value of a metric. Integer or Float.
- `sigma`   : Baseline standard deviation of a metric. Integer or Float.
- `mde`   : Minimum detectable effect (MDE) as percentage of the baseline value (e.g. for a baseline value 112.25 the MDE 0.03 means an expected change to 112.25 ± 112.25*0.03). Integer or Float.
- `power`   : Required statistical power of a test (1 - probability of Type II error). Integer or Float.
- `alpha`   : Required significance level of a test (probability of Type I error). Integer or Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A named Tuple with 3 elements:

- `"minimum_sample_size"`   : Required sample size. Float64.
- `"detect_range_lower"`   : Lower bound of the range of values not detectable with the returned required sample size (i.e. all values less than or equal to `"detect_range_lower"`   are detectable with the provided `alpha`   and `power`   ). Calculated as `baseline * (1 - mde)`   . Float64.
- `"detect_range_upper"`   : Upper bound of the range of values not detectable with the returned required sample size (i.e. all values greater than or equal to `"detect_range_upper"`   are detectable with the provided `alpha`   and `power`   ). Calculated as `baseline * (1 + mde)`   . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following query calculates the required sample size for an A/B test on a metric with baseline value of 112.25, standard deviation of 21.1, MDE of 3%, significance level of 5%, and the desired statistical power of 80%:

SELECT minSampleSizeContinous(112.25, 21.1, 0.03, 0.80, 0.05) AS sample_size Result:

┌─sample_size───────────────────────────┐
│ (616.2931945826209,108.8825,115.6175) │
└───────────────────────────────────────┘
## connectionId [¶](https://www.tinybird.co/docs/about:blank#connectionid)

Retrieves the connection ID of the client that submitted the current query and returns it as a UInt64 integer.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

connectionId() Alias: `connection_id`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The current connection ID. UInt64.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

This function is most useful in debugging scenarios or for internal purposes within the MySQL handler. It was created for compatibility with MySQL's `CONNECTION_ID` function It isn't typically used in production queries.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT connectionId() 0
## lowCardinalityIndices [¶](https://www.tinybird.co/docs/about:blank#lowcardinalityindices)

Returns the position of a value in the dictionary of a LowCardinality column. Positions start at 1. Since LowCardinality have per-part dictionaries, this function may return different positions for the same value in different parts.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

lowCardinalityIndices(col)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `col`   : a low cardinality column. LowCardinality.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The position of the value in the dictionary of the current part. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

##### lc_s.datasource

SCHEMA >
    s LowCardinality(String)

ENGINE "MergeTree" tb push datasources/lc_s.datasource

## create two parts:
echo 'ab\ncd\nab\nab\ndf' > lc.csv
tb datasource append lc_s lc.csv

echo 'ef\ncd\nab\ncd\nef' > lc.csv
tb datasource append lc_s lc.csv

tb sql "SELECT s, lowCardinalityIndices(s) FROM lc_s"
---------------------------------
| s  | lowCardinalityIndices(s) |
---------------------------------
| ab |                        1 |
| cd |                        2 |
| ab |                        1 |
| ab |                        1 |
| df |                        3 |
| ef |                        1 |
| cd |                        2 |
| ab |                        3 |
| cd |                        2 |
| ef |                        1 |
---------------------------------
## lowCardinalityKeys [¶](https://www.tinybird.co/docs/about:blank#lowcardinalitykeys)

Returns the dictionary values of a LowCardinality column. If the block is smaller or larger than the dictionary size, the result will be truncated or extended with default values. Since LowCardinality have per-part dictionaries, this function may return different dictionary values in different parts.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

lowCardinalityIndices(col)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `col`   : a low cardinality column. LowCardinality.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The dictionary keys. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

##### lc_s.datasource

SCHEMA >
    s LowCardinality(String)

ENGINE "MergeTree" tb push datasources/lc_s.datasource

## create two parts:
echo 'ab\ncd\nab\nab\ndf' > lc.csv
tb datasource append lc_s lc.csv

echo 'ef\ncd\nab\ncd\nef' > lc.csv
tb datasource append lc_s lc.csv

tb sql "select s, lowCardinalityKeys(s) from lc_s"
------------------------------
| s  | lowCardinalityKeys(s) |
------------------------------
| ab |                       |
| cd | ab                    |
| ab | cd                    |
| ab | df                    |
| df |                       |
| ef |                       |
| cd | ef                    |
| ab | cd                    |
| cd | ab                    |
| ef |                       |
------------------------------
## getSubcolumn [¶](https://www.tinybird.co/docs/about:blank#getsubcolumn)

Takes a table expression or identifier and constant string with the name of the sub-column, and returns the requested sub-column extracted from the expression.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

getSubcolumn(col_name, subcol_name)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `col_name`   : Table expression or identifier. Expression, Identifier.
- `subcol_name`   : The name of the sub-column. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the extracted sub-column.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT getSubcolumn(arr, 'subcolumn1'), getSubcolumn(arr, 'subcolumn2') FROM (
  select c1::Array(Tuple(subcolumn1 UInt32, subcolumn2 String)) as arr from values(([(1, 'Hello'), (2, 'World')]), ([(3, 'This'), (4, 'is'), (5, 'subcolumn')]))
) Result:

┌─getSubcolumn(arr, 'subcolumn1')─┬─getSubcolumn(arr, 'subcolumn2')─┐
│ [1,2]                           │ ['Hello','World']               │
│ [3,4,5]                         │ ['This','is','subcolumn']       │
└─────────────────────────────────┴─────────────────────────────────┘
## getTypeSerializationStreams [¶](https://www.tinybird.co/docs/about:blank#gettypeserializationstreams)

Enumerates stream paths of a data type.

This function is intended for use by developers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

getTypeSerializationStreams(col)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `col`   : Column or string representation of a data-type from which the data type will be detected.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array with all the serialization sub-stream paths.Array(String).

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT getTypeSerializationStreams(tuple('a', 1, 'b', 2)) Result:

┌─getTypeSerializationStreams(('a', 1, 'b', 2))─────────────────────────────────────────────────────────────────────────┐
1. │ ['{TupleElement(1), Regular}','{TupleElement(2), Regular}','{TupleElement(3), Regular}','{TupleElement(4), Regular}'] │
   └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Query:

SELECT getTypeSerializationStreams('Map(String, Int64)') Result:

┌─getTypeSerializationStreams('Map(String, Int64)')────────────────────────────────────────────────────────────────┐
1. │ ['{ArraySizes}','{ArrayElements, TupleElement(keys), Regular}','{ArrayElements, TupleElement(values), Regular}'] │
   └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/math-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Math functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for mathematical operations."
---




# Mathematical functions [¶](https://www.tinybird.co/docs/about:blank#mathematical-functions)

The following functions perform mathematical operations.

## e [¶](https://www.tinybird.co/docs/about:blank#e)

Returns `e` (Euler's constant).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

e()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float64.

## pi [¶](https://www.tinybird.co/docs/about:blank#pi)

Returns `π` (Pi).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

pi()
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float64.

## exp [¶](https://www.tinybird.co/docs/about:blank#exp)

Returns `e^x` , where x is the given argument to the function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exp(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT round(exp(-1), 4) Result:

┌─round(exp(-1), 4)─┐
│            0.3679 │
└───────────────────┘
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## log [¶](https://www.tinybird.co/docs/about:blank#log)

Returns the natural logarithm of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

log(x) Alias: `ln(x)`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## exp2 [¶](https://www.tinybird.co/docs/about:blank#exp2)

Returns 2 to the power of the given argument

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exp2(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## intExp2 [¶](https://www.tinybird.co/docs/about:blank#intexp2)

Like `exp` but returns a UInt64.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intExp2(x)
## log2 [¶](https://www.tinybird.co/docs/about:blank#log2)

Returns the binary logarithm of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

log2(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## exp10 [¶](https://www.tinybird.co/docs/about:blank#exp10)

Returns 10 to the power of the given argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exp10(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## intExp10 [¶](https://www.tinybird.co/docs/about:blank#intexp10)

Like `exp10` but returns a UInt64.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intExp10(x)
## log10 [¶](https://www.tinybird.co/docs/about:blank#log10)

Returns the decimal logarithm of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

log10(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## sqrt [¶](https://www.tinybird.co/docs/about:blank#sqrt)

Returns the square root of the argument.

sqrt(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## cbrt [¶](https://www.tinybird.co/docs/about:blank#cbrt)

Returns the cubic root of the argument.

cbrt(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## erf [¶](https://www.tinybird.co/docs/about:blank#erf)

If `x` is non-negative, then `erf(x / σ√2)` is the probability that a random variable having a normal distribution with standard deviation `σ` takes the value that is separated from the expected value by more than `x`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

erf(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

(three sigma rule)

SELECT erf(3 / sqrt(2)) ┌─erf(divide(3, sqrt(2)))─┐
│      0.9973002039367398 │
└─────────────────────────┘
## erfc [¶](https://www.tinybird.co/docs/about:blank#erfc)

Returns a number close to `1-erf(x)` without loss of precision for large `x` values.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

erfc(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## lgamma [¶](https://www.tinybird.co/docs/about:blank#lgamma)

Returns the logarithm of the gamma function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

lgamma(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## tgamma [¶](https://www.tinybird.co/docs/about:blank#tgamma)

Returns the gamma function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

gamma(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## sin [¶](https://www.tinybird.co/docs/about:blank#sin)

Returns the sine of the argument

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sin(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT sin(1.23) 0.9424888019316975
## cos [¶](https://www.tinybird.co/docs/about:blank#cos)

Returns the cosine of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

cos(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## tan [¶](https://www.tinybird.co/docs/about:blank#tan)

Returns the tangent of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

tan(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## asin [¶](https://www.tinybird.co/docs/about:blank#asin)

Returns the arc sine of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

asin(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## acos [¶](https://www.tinybird.co/docs/about:blank#acos)

Returns the arc cosine of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

acos(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## atan [¶](https://www.tinybird.co/docs/about:blank#atan)

Returns the arc tangent of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

atan(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float*.

## pow [¶](https://www.tinybird.co/docs/about:blank#pow)

Returns `x^y`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

pow(x, y) Alias: `power(x, y)`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   - (U)Int8/16/32/64, Float* or Decimal*
- `y`   - (U)Int8/16/32/64, Float* or Decimal*

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Type: Float64.

## cosh [¶](https://www.tinybird.co/docs/about:blank#cosh)

Returns the hyperbolic cosine of the argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

cosh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The angle, in radians. Values from the interval: `-∞ < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Values from the interval: `1 <= cosh(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT cosh(0) Result:

┌─cosh(0)──┐
│        1 │
└──────────┘
## acosh [¶](https://www.tinybird.co/docs/about:blank#acosh)

Returns the inverse hyperbolic cosine.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

acosh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Hyperbolic cosine of angle. Values from the interval: `1 <= x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The angle, in radians. Values from the interval: `0 <= acosh(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT acosh(1) Result:

┌─acosh(1)─┐
│        0 │
└──────────┘
## sinh [¶](https://www.tinybird.co/docs/about:blank#sinh)

Returns the hyperbolic sine.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sinh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The angle, in radians. Values from the interval: `-∞ < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Values from the interval: `-∞ < sinh(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT sinh(0) Result:

┌─sinh(0)──┐
│        0 │
└──────────┘
## asinh [¶](https://www.tinybird.co/docs/about:blank#asinh)

Returns the inverse hyperbolic sine.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

asinh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Hyperbolic sine of angle. Values from the interval: `-∞ < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The angle, in radians. Values from the interval: `-∞ < asinh(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT asinh(0) Result:

┌─asinh(0)─┐
│        0 │
└──────────┘
## tanh [¶](https://www.tinybird.co/docs/about:blank#tanh)

Returns the hyperbolic tangent.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

tanh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The angle, in radians. Values from the interval: `-∞ < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Values from the interval: `-1 < tanh(x) < 1`  .

Type: Float*.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT tanh(0) Result:

0
## atanh [¶](https://www.tinybird.co/docs/about:blank#atanh)

Returns the inverse hyperbolic tangent.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

atanh(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Hyperbolic tangent of angle. Values from the interval: `-1 < x < 1`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The angle, in radians. Values from the interval: `-∞ < atanh(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT atanh(0) Result:

┌─atanh(0)─┐
│        0 │
└──────────┘
## atan2 [¶](https://www.tinybird.co/docs/about:blank#atan2)

Returns the atan2 as the angle in the Euclidean plane, given in radians, between the positive x axis and the ray to the point `(x, y) ≠ (0, 0)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

atan2(y, x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `y`   : y-coordinate of the point through which the ray passes. (U)Int*, Float* or Decimal*.
- `x`   : x-coordinate of the point through which the ray passes. (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The angle `θ`   such that `-π < θ <= π`   , in radians.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT atan2(1, 1) Result:

┌────────atan2(1, 1)─┐
│ 0.7853981633974483 │
└────────────────────┘
## hypot [¶](https://www.tinybird.co/docs/about:blank#hypot)

Returns the length of the hypotenuse of a right-angle triangle. Hypot avoids problems that occur when squaring very large or very small numbers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

hypot(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The first cathetus of a right-angle triangle. (U)Int*, Float* or Decimal*.
- `y`   : The second cathetus of a right-angle triangle. (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The length of the hypotenuse of a right-angle triangle.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT hypot(1, 1) Result:

┌────────hypot(1, 1)─┐
│ 1.4142135623730951 │
└────────────────────┘
## log1p [¶](https://www.tinybird.co/docs/about:blank#log1p)

Calculates `log(1+x)` . The calculation `log1p(x)` is more accurate than `log(1+x)` for small values of x.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

log1p(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Values from the interval: `-1 < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Values from the interval: `-∞ < log1p(x) < +∞`  .

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT log1p(0) Result:

┌─log1p(0)─┐
│        0 │
└──────────┘
## sign [¶](https://www.tinybird.co/docs/about:blank#sign)

Returns the sign of a real number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sign(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Values from `-∞`   to `+∞`   . Supports all numeric types in Tinybird.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- -1 for `x < 0`
- 0 for `x = 0`
- 1 for `x > 0`

Type: Int8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Sign for the zero value:

SELECT sign(0) Result:

┌─sign(0)─┐
│       0 │
└─────────┘ Sign for the positive value:

SELECT sign(1) Result:

┌─sign(1)─┐
│       1 │
└─────────┘ Sign for the negative value:

SELECT sign(-1) Result:

┌─sign(-1)─┐
│       -1 │
└──────────┘
## sigmoid [¶](https://www.tinybird.co/docs/about:blank#sigmoid)

Returns the sigmoid function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sigmoid(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : input value. Values from the interval: `-∞ < x < +∞`   . (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Corresponding value along the sigmoid curve between 0 and 1. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT round(sigmoid(x), 5) FROM (SELECT arrayJoin([-1, 0, 1]) AS x) Result:

0.26894
0.5
0.73106
## degrees [¶](https://www.tinybird.co/docs/about:blank#degrees)

Converts radians to degrees.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

degrees(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Input in radians. (U)Int*, Float* or Decimal*.
- `x`   : Input in radians. (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Value in degrees. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT degrees(3.141592653589793) Result:

┌─degrees(3.141592653589793)─┐
│                        180 │
└────────────────────────────┘
## radians [¶](https://www.tinybird.co/docs/about:blank#radians)

Converts degrees to radians.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

radians(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Input in degrees. (U)Int*, Float* or Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Value in radians.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT radians(180) Result:

┌──────radians(180)─┐
│ 3.141592653589793 │
└───────────────────┘
## factorial [¶](https://www.tinybird.co/docs/about:blank#factorial)

Computes the factorial of an integer value. Works with any native integer type including UInt(8|16|32|64) and Int(8|16|32|64). The return type is UInt64.

The factorial of 0 is 1. Likewise, the factorial() function returns 1 for any negative value. The maximum positive value for the input argument is 20, a value of 21 or greater will cause exception throw.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

factorial(n)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT factorial(10) Result:

┌─factorial(10)─┐
│       3628800 │
└───────────────┘
## width_bucket [¶](https://www.tinybird.co/docs/about:blank#width-bucket)

Returns the number of the bucket in which `operand` falls in a histogram having `count` equal-width buckets spanning the range `low` to `high` . Returns `0` if `operand < low` , and returns `count+1` if `operand >= high`.

`operand`, `low`, `high` can be any native number type. `count` can only be unsigned native integer and its value can't be zero.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

widthBucket(operand, low, high, count) Alias: `WIDTH_BUCKET`

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT widthBucket(10.15, -8.6, 23, 18) Result:

┌─widthBucket(10.15, -8.6, 23, 18)─┐
│                               11 │
└──────────────────────────────────┘
## proportionsZTest [¶](https://www.tinybird.co/docs/about:blank#proportionsztest)

Returns test statistics for the two proportion Z-test - a statistical test for comparing the proportions from two populations `x` and `y`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

proportionsZTest(successes_x, successes_y, trials_x, trials_y, conf_level, pool_type)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `successes_x`   : Number of successes in population `x`   . UInt64.
- `successes_y`   : Number of successes in population `y`   . UInt64.
- `trials_x`   : Number of trials in population `x`   . UInt64.
- `trials_y`   : Number of trials in population `y`   . UInt64.
- `conf_level`   : Confidence level for the test. Float64.
- `pool_type`   : Selection of pooling (way in which the standard error is estimated). Can be either `unpooled`   or `pooled`   . String.

For argument `pool_type` : In the pooled version, the two proportions are averaged, and only one proportion is used to estimate the standard error. In the unpooled version, the two proportions are used separately.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `z_stat`   : Z statistic. Float64.
- `p_val`   : P value. Float64.
- `ci_low`   : The lower confidence interval. Float64.
- `ci_high`   : The upper confidence interval. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT proportionsZTest(10, 11, 100, 101, 0.95, 'unpooled') Result:

┌─proportionsZTest(10, 11, 100, 101, 0.95, 'unpooled')───────────────────────────────┐
│ (-0.20656724435948853,0.8363478437079654,-0.09345975390115283,0.07563797172293502) │
└────────────────────────────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/machine-learning-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Machine learning functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for machine learning."
---




# Machine Learning functions [¶](https://www.tinybird.co/docs/about:blank#machine-learning-functions)

The following functions are used for machine learning.

## evalMLMethod [¶](https://www.tinybird.co/docs/about:blank#evalmlmethod)

Prediction using fitted regression models uses `evalMLMethod` function.

## stochasticLinearRegression [¶](https://www.tinybird.co/docs/about:blank#stochasticlinearregression)

The `stochasticLinearRegression` aggregate function implements stochastic gradient descent method using linear model and MSE loss function. Uses `evalMLMethod` to predict on new data.

## stochasticLogisticRegression [¶](https://www.tinybird.co/docs/about:blank#stochasticlogisticregression)

The `stochasticLogisticRegression` aggregate function implements stochastic gradient descent method for binary classification problem. Uses `evalMLMethod` to predict on new data.



---

URL: https://www.tinybird.co/docs/sql-reference/functions/logical-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Logical functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for logical operations."
---




# Logical functions [¶](https://www.tinybird.co/docs/about:blank#logical-functions)

The following functions perform logical operations on arguments of arbitrary numeric types. They return either 0 or 1 as UInt8 or in some cases `NULL`.

Zero as an argument is considered `false` , non-zero values are considered `true`.

## and [¶](https://www.tinybird.co/docs/about:blank#and)

Calculates the logical conjunction of two or more values.

Setting short_circuit_function_evaluation controls whether short-circuit evaluation is used. If enabled, `val_i` is evaluated only if `(val_1 AND val_2 AND ... AND val_{i-1})` is `true` . For example, with short-circuit evaluation, no division-by-zero exception is thrown when executing the query `SELECT and(number = 2, intDiv(1, number)) FROM numbers(5)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

and(val1, val2...) Alias: The AND operator.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val1, val2, ...`   : List of at least two values. Int, UInt, Float or Nullable.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `0`   , if at least one argument evaluates to `false`  ,
- `NULL`   , if no argument evaluates to `false`   and at least one argument is `NULL`  ,
- `1`   , otherwise.

Type: UInt8 or Nullable(UInt8).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT and(0, 1, -2) Result:

┌─and(0, 1, -2)─┐
│             0 │
└───────────────┘ With `NULL`:

SELECT and(NULL, 1, 10, -2) Result:

┌─and(NULL, 1, 10, -2)─┐
│                 ᴺᵁᴸᴸ │
└──────────────────────┘
## or [¶](https://www.tinybird.co/docs/about:blank#or)

Calculates the logical disjunction of two or more values.

Setting short_circuit_function_evaluation controls whether short-circuit evaluation is used. If enabled, `val_i` is evaluated only if `((NOT val_1) AND (NOT val_2) AND ... AND (NOT val_{i-1}))` is `true` . For example, with short-circuit evaluation, no division-by-zero exception is thrown when executing the query `SELECT or(number = 0, intDiv(1, number) != 0) FROM numbers(5)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

or(val1, val2...) Alias: The OR operator.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val1, val2, ...`   : List of at least two values. Int, UInt, Float or Nullable.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   , if at least one argument evaluates to `true`  ,
- `0`   , if all arguments evaluate to `false`  ,
- `NULL`   , if all arguments evaluate to `false`   and at least one argument is `NULL`  .

Type: UInt8 or Nullable(UInt8).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT or(1, 0, 0, 2, NULL) Result:

┌─or(1, 0, 0, 2, NULL)─┐
│                    1 │
└──────────────────────┘ With `NULL`:

SELECT or(0, NULL) Result:

┌─or(0, NULL)─┐
│        ᴺᵁᴸᴸ │
└─────────────┘
## not [¶](https://www.tinybird.co/docs/about:blank#not)

Calculates the logical negation of a value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

not(val) Alias: The Negation operator.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : The value. Int, UInt, Float or Nullable.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   , if `val`   evaluates to `false`  ,
- `0`   , if `val`   evaluates to `true`  ,
- `NULL`   , if `val`   is `NULL`  .

Type: UInt8 or Nullable(UInt8).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT NOT(1) Result:

┌─not(1)─┐
│      0 │
└────────┘
## xor [¶](https://www.tinybird.co/docs/about:blank#xor)

Calculates the logical exclusive disjunction of two or more values. For more than two input values, the function first xor-s the first two values, then xor-s the result with the third value etc.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

xor(val1, val2...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val1, val2, ...`   : List of at least two values. Int, UInt, Float or Nullable.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   , for two values: if one of the values evaluates to `false`   and other doesn't,
- `0`   , for two values: if both values evaluate to `false`   or to both `true`  ,
- `NULL`   , if at least one of the inputs is `NULL`

Type: UInt8 or Nullable(UInt8).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT xor(0, 1, 1) Result:

┌─xor(0, 1, 1)─┐
│            0 │
└──────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/json-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "JSON functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for working with JSON."
---




# JSON functions [¶](https://www.tinybird.co/docs/about:blank#json-functions)

The following functions are used for working with JSON.

There are two sets of functions to parse JSON:

- `simpleJSON*`   ( `visitParam*`   ) which is made for parsing a limited subset of JSON extremely fast.
- `JSONExtract*`   which is made for parsing ordinary JSON.

## simpleJSON (visitParam) functions [¶](https://www.tinybird.co/docs/about:blank#simplejson-visitparam-functions)

Tinybird has special functions for working with simplified JSON. All these JSON functions are based on strong assumptions about what the JSON can be. They try to do as little as possible to get the job done as quickly as possible.

The following assumptions are made:

1. The field name (function argument) must be a constant.
2. The field name is somehow canonically encoded in JSON. For example: `simpleJSONHas('{"abc":"def"}', 'abc') = 1`   , but `simpleJSONHas('{"\\u0061\\u0062\\u0063":"def"}', 'abc') = 0`
3. Fields are searched for on any nesting level, indiscriminately. If there are multiple matching fields, the first occurrence is used.
4. The JSON doesn't have space characters outside of string literals.

### simpleJSONHas [¶](https://www.tinybird.co/docs/about:blank#simplejsonhas)

Checks whether there is a field named `field_name` .  The result is `UInt8`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONHas(json, field_name) Alias: `visitParamHas`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `1`   if the field exists, `0`   otherwise. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH (SELECT '{"foo":"true","qux":1}') AS json
SELECT simpleJSONHas(json, 'foo'), simpleJSONHas(json, 'bar') Result:

-----------------------------------------------------------
| simpleJSONHas(json, 'foo') | simpleJSONHas(json, 'bar') |
-----------------------------------------------------------
|                          1 |                          0 |
-----------------------------------------------------------
### simpleJSONExtractUInt [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractuint)

Parses `UInt64` from the value of the field named `field_name` . If this is a string field, it tries to parse a number from the beginning of the string. If the field doesn't exist, or it exists but doesn't contain a number, it returns `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractUInt(json, field_name) Alias: `visitParamExtractUInt`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the number parsed from the field if the field exists and contains a number, `0`   otherwise. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractUInt(json, 'foo') 
FROM (
    select c1::String as json from values(
        ('{"foo":"4e3"}'),
        ('{"foo":3.4}'),
        ('{"foo":5}'),
        ('{"foo":"not1number"}'),
        ('{"baz":2}'))
)
ORDER BY json Result:

0
4
0
3
5
### simpleJSONExtractInt [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractint)

Parses `Int64` from the value of the field named `field_name` . If this is a string field, it tries to parse a number from the beginning of the string. If the field doesn't exist, or it exists but doesn't contain a number, it returns `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractInt(json, field_name) Alias: `visitParamExtractInt`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the number parsed from the field if the field exists and contains a number, `0`   otherwise. Int64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractInt(json, 'foo') 
FROM (
    select c1::String as json from values(
        ('{"foo":"-4e3"}'),
        ('{"foo":-3.4}'),
        ('{"foo":5}'),
        ('{"foo":"not1number"}'),
        ('{"baz":2}'))
)
ORDER BY json Result:

0
-4
0
-3
5
### simpleJSONExtractFloat [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractfloat)

Parses `Float64` from the value of the field named `field_name` . If this is a string field, it tries to parse a number from the beginning of the string. If the field doesn't exist, or it exists but doesn't contain a number, it returns `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractFloat(json, field_name) Alias: `visitParamExtractFloat`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the number parsed from the field if the field exists and contains a number, `0`   otherwise. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractFloat(json, 'foo') 
FROM (
    select c1::String as json from values(
        ('{"foo":"-4e3"}'),
        ('{"foo":-3.4}'),
        ('{"foo":5}'),
        ('{"foo":"not1number"}'),
        ('{"baz":2}'))
)
ORDER BY json Result:

0
-4000
0
-3.4
5
### simpleJSONExtractBool [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractbool)

Parses a true/false value from the value of the field named `field_name` . The result is `UInt8`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractBool(json, field_name) Alias: `visitParamExtractBool`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

It returns `1` if the value of the field is `true`, `0` otherwise. This means this function will return `0` including (and not only) in the following cases:

- If the field doesn't exists.
- If the field contains `true`   as a string, e.g.: `{"field":"true"}`  .
- If the field contains `1`   as a numerical value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractBool(json, 'bar') AS barbool, simpleJSONExtractBool(json, 'foo') AS foobool 
FROM
(
    select c1::String as json from values(
        ('{"foo":false,"bar":true}'),
        ('{"foo":"true","qux":1}')
    )
)
ORDER BY json Result:

barbool   foobool  
    UInt8     UInt8  
─────────────────────
        0         0  
─────────────────────
        1         0
### simpleJSONExtractRaw [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractraw)

Returns the value of the field named `field_name` as a `String` , including separators.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractRaw(json, field_name) Alias: `visitParamExtractRaw`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the value of the field as a string, including separators if the field exists, or an empty string otherwise. `String`

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractRaw(json, 'foo')
FROM (
    select c1::String as json from values(
        ('{"foo":"-4e3"}'),
        ('{"foo":-3.4}'),
        ('{"foo":5}'),
        ('{"foo":{"def":[1,2,3]}}'),
        ('{"baz":2}')
    )
)
ORDER BY json Result:

"-4e3"
-3.4
5
{"def":[1,2,3]}
### simpleJSONExtractString [¶](https://www.tinybird.co/docs/about:blank#simplejsonextractstring)

Parses `String` in double quotes from the value of the field named `field_name`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

simpleJSONExtractString(json, field_name) Alias: `visitParamExtractString`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : The JSON in which the field is searched for. String
- `field_name`   : The name of the field to search for. String literal

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the unescaped value of a field as a string, including separators. An empty string is returned if the field doesn't contain a double quoted string, if unescaping fails or if the field doesn't exist. String.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

There is currently no support for code points in the format `\uXXXX\uYYYY` that aren't from the basic multilingual plane (they are converted to CESU-8 instead of UTF-8).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT simpleJSONExtractString(json, 'foo')
FROM (
    select c1::String as json from values(
      ('{"foo":"\\n\\u0000"}'),
      ('{"foo":"\\u263"}'),
      ('{"foo":"\\u263a"}'),
      ('{"foo":"hello}')
    )
)
ORDER BY json Result:

\n\0

☺
## JSONExtract functions [¶](https://www.tinybird.co/docs/about:blank#jsonextract-functions)

The following functions are based on simdjson, and designed for more complex JSON parsing requirements.

### isValidJSON [¶](https://www.tinybird.co/docs/about:blank#isvalidjson)

Checks that passed string is valid JSON.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isValidJSON(json)
### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT isValidJSON('{"a": "hello", "b": [-100, 200.0, 300]}') = 1
SELECT isValidJSON('not a json') = 0
### JSONHas [¶](https://www.tinybird.co/docs/about:blank#jsonhas)

If the value exists in the JSON document, `1` will be returned. If the value doesn't exist, `0` will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONHas(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `1`   if the value exists in `json`   , otherwise `0`   . UInt8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT JSONHas('{"a": "hello", "b": [-100, 200.0, 300]}', 'b') = 1
SELECT JSONHas('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', 4) = 0 The minimum index of the element is 1. Thus the element 0 doesn't exist. You may use integers to access both JSON arrays and JSON objects. For example:

SELECT JSONExtractKey('{"a": "hello", "b": [-100, 200.0, 300]}', 1) = 'a'
SELECT JSONExtractKey('{"a": "hello", "b": [-100, 200.0, 300]}', 2) = 'b'
SELECT JSONExtractKey('{"a": "hello", "b": [-100, 200.0, 300]}', -1) = 'b'
SELECT JSONExtractKey('{"a": "hello", "b": [-100, 200.0, 300]}', -2) = 'a'
SELECT JSONExtractString('{"a": "hello", "b": [-100, 200.0, 300]}', 1) = 'hello'
### JSONLength [¶](https://www.tinybird.co/docs/about:blank#jsonlength)

Return the length of a JSON array or a JSON object. If the value doesn't exist or has the wrong type, `0` will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONLength(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the length of the JSON array or JSON object. Returns `0`   if the value doesn't exist or has the wrong type. UInt64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT JSONLength('{"a": "hello", "b": [-100, 200.0, 300]}', 'b') = 3
SELECT JSONLength('{"a": "hello", "b": [-100, 200.0, 300]}') = 2
### JSONType [¶](https://www.tinybird.co/docs/about:blank#jsontype)

Return the type of a JSON value. If the value doesn't exist, `Null=0` will be returned (not usual Null, but `Null=0` of `Enum8('Null' = 0, 'String' = 34,...` ). .

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONType(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the type of a JSON value as a string, otherwise if the value doesn't exists it returns `Null=0`   . Enum.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT JSONType('{"a": "hello", "b": [-100, 200.0, 300]}') = 'Object'
SELECT JSONType('{"a": "hello", "b": [-100, 200.0, 300]}', 'a') = 'String'
SELECT JSONType('{"a": "hello", "b": [-100, 200.0, 300]}', 'b') = 'Array'
### JSONExtractUInt [¶](https://www.tinybird.co/docs/about:blank#jsonextractuint)

Parses JSON and extracts a value of UInt type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractUInt(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a UInt value if it exists, otherwise it returns `0`   . UInt64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT JSONExtractUInt('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', -1) as x, toTypeName(x) Result:

┌───x─┬─toTypeName(x)─┐
│ 300 │ UInt64        │
└─────┴───────────────┘
### JSONExtractInt [¶](https://www.tinybird.co/docs/about:blank#jsonextractint)

Parses JSON and extracts a value of Int type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractInt(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an Int value if it exists, otherwise it returns `0`   . Int64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT JSONExtractInt('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', -1) as x, toTypeName(x) Result:

┌───x─┬─toTypeName(x)─┐
│ 300 │ Int64         │
└─────┴───────────────┘
### JSONExtractFloat [¶](https://www.tinybird.co/docs/about:blank#jsonextractfloat)

Parses JSON and extracts a value of Int type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractFloat(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an Float value if it exists, otherwise it returns `0`   . Float64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT JSONExtractFloat('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', 2) as x, toTypeName(x) Result:

┌───x─┬─toTypeName(x)─┐
│ 200 │ Float64       │
└─────┴───────────────┘
### JSONExtractBool [¶](https://www.tinybird.co/docs/about:blank#jsonextractbool)

Parses JSON and extracts a boolean value. If the value doesn't exist or has a wrong type, `0` will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractBool(json\[, indices_or_keys\]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a Boolean value if it exists, otherwise it returns `0`   . Bool.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT JSONExtractBool('{"passed": true}', 'passed') Result:

┌─JSONExtractBool('{"passed": true}', 'passed')─┐
│                                             1 │
└───────────────────────────────────────────────┘
### JSONExtractString [¶](https://www.tinybird.co/docs/about:blank#jsonextractstring)

Parses JSON and extracts a string. This function is similar to `visitParamExtractString` functions. If the value doesn't exist or has a wrong type, an empty string will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractString(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an unescaped string from `json`   . If unescaping failed, if the value doesn't exist or if it has a wrong type then it returns an empty string. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT JSONExtractString('{"a": "hello", "b": [-100, 200.0, 300]}', 'a') = 'hello'
SELECT JSONExtractString('{"abc":"\\n\\u0000"}', 'abc') = '\n\0'
SELECT JSONExtractString('{"abc":"\\u263a"}', 'abc') = '☺'
SELECT JSONExtractString('{"abc":"\\u263"}', 'abc') = ''
SELECT JSONExtractString('{"abc":"hello}', 'abc') = ''
### JSONExtract [¶](https://www.tinybird.co/docs/about:blank#jsonextract)

Parses JSON and extracts a value of the given data type. This function is a generalized version of the previous `JSONExtract` functions. Meaning:

`JSONExtract(..., 'String')` returns exactly the same as `JSONExtractString()`, `JSONExtract(..., 'Float64')` returns exactly the same as `JSONExtractFloat()`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtract(json [, indices_or_keys...], return_type)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.
- `return_type`   : A string specifying the type of the value to extract. String.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value if it exists of the specified return type, otherwise it returns `0`  , `Null`   , or an empty-string depending on the specified return type. UInt64, Int64, Float64, Bool or String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT JSONExtract('{"a": "hello", "b": [-100, 200.0, 300]}', 'Tuple(String, Array(Float64))') = ('hello',[-100,200,300])
SELECT JSONExtract('{"a": "hello", "b": [-100, 200.0, 300]}', 'Tuple(b Array(Float64), a String)') = ([-100,200,300],'hello')
SELECT JSONExtract('{"a": "hello", "b": "world"}', 'Map(String, String)') = map('a',  'hello', 'b', 'world')
SELECT JSONExtract('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', 'Array(Nullable(Int8))') = [-100, NULL, NULL]
SELECT JSONExtract('{"a": "hello", "b": [-100, 200.0, 300]}', 'b', 4, 'Nullable(Int64)') = NULL
SELECT JSONExtract('{"passed": true}', 'passed', 'UInt8') = 1
SELECT JSONExtract('{"day": "Thursday"}', 'day', 'Enum8(\'Sunday\' = 0, \'Monday\' = 1, \'Tuesday\' = 2, \'Wednesday\' = 3, \'Thursday\' = 4, \'Friday\' = 5, \'Saturday\' = 6)') = 'Thursday'
SELECT JSONExtract('{"day": 5}', 'day', 'Enum8(\'Sunday\' = 0, \'Monday\' = 1, \'Tuesday\' = 2, \'Wednesday\' = 3, \'Thursday\' = 4, \'Friday\' = 5, \'Saturday\' = 6)') = 'Friday'
### JSONExtractKeysAndValues [¶](https://www.tinybird.co/docs/about:blank#jsonextractkeysandvalues)

Parses key-value pairs from JSON where the values are of the given data type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractKeysAndValues(json [, indices_or_keys...], value_type)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.
- `value_type`   : A string specifying the type of the value to extract. String.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array of parsed key-value pairs. Array(Tuple( `value_type`   )).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT JSONExtractKeysAndValues('{"x": {"a": 5, "b": 7, "c": 11}}', 'x', 'Int8') = [('a',5),('b',7),('c',11)]
### JSONExtractKeys [¶](https://www.tinybird.co/docs/about:blank#jsonextractkeys)

Parses a JSON string and extracts the keys.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractKeys(json[, a, b, c...])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : String with valid JSON.
- `a, b, c...`   : Comma-separated indices or keys that specify the path to the inner field in a nested JSON object. Each argument can be either a String to get the field by the key or an Integer to get the N-th field (indexed from 1, negative integers count from the end). If not set, the whole JSON is parsed as the top-level object. Optional parameter.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array with the keys of the JSON. Array(String).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT JSONExtractKeys('{"a": "hello", "b": [-100, 200.0, 300]}') Result:

text
┌─JSONExtractKeys('{"a": "hello", "b": [-100, 200.0, 300]}')─┐
│ ['a','b']                                                  │
└────────────────────────────────────────────────────────────┘
### JSONExtractRaw [¶](https://www.tinybird.co/docs/about:blank#jsonextractraw)

Returns part of the JSON as an unparsed string. If the part doesn't exist or has the wrong type, an empty string will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractRaw(json [, indices_or_keys]...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns part of the JSON as an unparsed string. If the part doesn't exist or has the wrong type, an empty string is returned. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT JSONExtractRaw('{"a": "hello", "b": [-100, 200.0, 300]}', 'b') = '[-100, 200.0, 300]'
### JSONExtractArrayRaw [¶](https://www.tinybird.co/docs/about:blank#jsonextractarrayraw)

Returns an array with elements of JSON array, each represented as unparsed string. If the part doesn't exist or isn’t an array, then an empty array will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractArrayRaw(json [, indices_or_keys...])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : JSON string to parse. String.
- `indices_or_keys`   : A list of zero or more arguments, each of which can be either string or integer. String, Int*.

`indices_or_keys` type:

- String = access object member by key.
- Positive integer = access the n-th member/key from the beginning.
- Negative integer = access the n-th member/key from the end.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array with elements of JSON array, each represented as unparsed string. Otherwise, an empty array is returned if the part doesn't exist or isn't an array. Array(String).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT JSONExtractArrayRaw('{"a": "hello", "b": [-100, 200.0, "hello"]}', 'b') = ['-100', '200.0', '"hello"']
### JSONExtractKeysAndValuesRaw [¶](https://www.tinybird.co/docs/about:blank#jsonextractkeysandvaluesraw)

Extracts raw data from a JSON object.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONExtractKeysAndValuesRaw(json[, p, a, t, h])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `json`   : String with valid JSON.
- `p, a, t, h`   : Comma-separated indices or keys that specify the path to the inner field in a nested JSON object. Each argument can be either a string to get the field by the key or an integer to get the N-th field (indexed from 1, negative integers count from the end). If not set, the whole JSON is parsed as the top-level object. Optional parameter.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array with `('key', 'value')`   tuples. Both tuple members are strings. Array(Tuple(String, String).
- Empty array if the requested object doesn't exist, or input JSON is invalid. Array(Tuple(String, String).

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}') Result:

┌─JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}')─┐
│ [('a','[-100,200]'),('b','{"c":{"d":"hello","f":"world"}}')]                                 │
└──────────────────────────────────────────────────────────────────────────────────────────────┘ Query:

SELECT JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}', 'b') Result:

┌─JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}', 'b')─┐
│ [('c','{"d":"hello","f":"world"}')]                                                               │
└───────────────────────────────────────────────────────────────────────────────────────────────────┘ Query:

SELECT JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}', -1, 'c') Result:

┌─JSONExtractKeysAndValuesRaw('{"a": [-100, 200.0], "b":{"c": {"d": "hello", "f": "world"}}}', -1, 'c')─┐
│ [('d','"hello"'),('f','"world"')]                                                                     │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘
### JSON_EXISTS [¶](https://www.tinybird.co/docs/about:blank#json-exists)

If the value exists in the JSON document, `1` will be returned. If the value doesn't exist, `0` will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSON_EXISTS(json, path)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : A string with valid JSON. String.
- `path`   : A string representing the path. String.

Before version 21.11 the order of arguments was wrong, i.e. JSON_EXISTS(path, json)

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `1`   if the value exists in the JSON document, otherwise `0`  .

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT JSON_EXISTS('{"hello":1}', '$.hello')
SELECT JSON_EXISTS('{"hello":{"world":1}}', '$.hello.world')
SELECT JSON_EXISTS('{"hello":["world"]}', '$.hello[*]')
SELECT JSON_EXISTS('{"hello":["world"]}', '$.hello[0]')
### JSON_QUERY [¶](https://www.tinybird.co/docs/about:blank#json-query)

Parses a JSON and extract a value as a JSON array or JSON object. If the value doesn't exist, an empty string will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSON_QUERY(json, path)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : A string with valid JSON. String.
- `path`   : A string representing the path. String.

Before version 21.11 the order of arguments was wrong, i.e. JSON_EXISTS(path, json)

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the extracted value as a JSON array or JSON object. Otherwise it returns an empty string if the value doesn't exist. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT JSON_QUERY('{"hello":"world"}', '$.hello')
SELECT JSON_QUERY('{"array":[[0, 1, 2, 3, 4, 5], [0, -1, -2, -3, -4, -5]]}', '$.array[*][0 to 2, 4]')
SELECT JSON_QUERY('{"hello":2}', '$.hello')
SELECT toTypeName(JSON_QUERY('{"hello":2}', '$.hello')) Result:

["world"]
[0, 1, 4, 0, -1, -4]
[2]
String
### JSON_VALUE [¶](https://www.tinybird.co/docs/about:blank#json-value)

Parses a JSON and extract a value as a JSON scalar. If the value doesn't exist, an empty string will be returned by default.

This function is controlled by the following settings:

- by SET `function_json_value_return_type_allow_nullable`   = `true`  , `NULL`   will be returned. If the value is complex type (such as: struct, array, map), an empty string will be returned by default.
- by SET `function_json_value_return_type_allow_complex`   = `true`   , the complex value will be returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSON_VALUE(json, path)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `json`   : A string with valid JSON. String.
- `path`   : A string representing the path. String.

Before version 21.11 the order of arguments was wrong, i.e. JSON_EXISTS(path, json)

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the extracted value as a JSON scalar if it exists, otherwise an empty string is returned. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT JSON_VALUE('{"hello":"world"}', '$.hello')
SELECT JSON_VALUE('{"array":[[0, 1, 2, 3, 4, 5], [0, -1, -2, -3, -4, -5]]}', '$.array[*][0 to 2, 4]')
SELECT JSON_VALUE('{"hello":2}', '$.hello')
SELECT toTypeName(JSON_VALUE('{"hello":2}', '$.hello'))
select JSON_VALUE('{"hello":"world"}', '$.b') settings function_json_value_return_type_allow_nullable=true
select JSON_VALUE('{"hello":{"world":"!"}}', '$.hello') settings function_json_value_return_type_allow_complex=true Result:

world
0
2
String
### toJSONString [¶](https://www.tinybird.co/docs/about:blank#tojsonstring)

Serializes a value to its JSON representation. Various data types and nested structures are supported. 64-bit integers or bigger (like `UInt64` or `Int128` ) are enclosed in quotes by default. output_format_json_quote_64bit_integers controls this behavior. Special values `NaN` and `inf` are replaced with `null` . Enable output_format_json_quote_denormals setting to show them. When serializing an Enum value, the function outputs its name.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toJSONString(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Value to serialize. Value may be of any data type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- JSON representation of the value. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The first example shows serialization of a Map. The second example shows some special values wrapped into a Tuple.

Query:

SELECT toJSONString(map('key1', 1, 'key2', 2))
SELECT toJSONString(tuple(1.25, NULL, NaN, +inf, -inf, [])) SETTINGS output_format_json_quote_denormals = 1 Result:

{"key1":1,"key2":2}
[1.25,null,"nan","inf","-inf",[]]
### JSONArrayLength [¶](https://www.tinybird.co/docs/about:blank#jsonarraylength)

Returns the number of elements in the outermost JSON array. The function returns NULL if input JSON string is invalid.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

JSONArrayLength(json) Alias: `JSON_ARRAY_LENGTH(json)`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `json`   : String with valid JSON.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- If `json`   is a valid JSON array string, returns the number of array elements, otherwise returns NULL. Nullable(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    JSONArrayLength(''),
    JSONArrayLength('[1,2,3]')

┌─JSONArrayLength('')─┬─JSONArrayLength('[1,2,3]')─┐
│                ᴺᵁᴸᴸ │                          3 │
└─────────────────────┴────────────────────────────┘
### jsonMergePatch [¶](https://www.tinybird.co/docs/about:blank#jsonmergepatch)

Returns the merged JSON object string which is formed by merging multiple JSON objects.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

jsonMergePatch(json1, json2, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `json`   : String with valid JSON.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- If JSON object strings are valid, return the merged JSON object string. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT jsonMergePatch('{"a":1}', '{"name": "joey"}', '{"name": "tom"}', '{"name": "zoey"}') AS res ┌─res───────────────────┐
│ {"a":1,"name":"zoey"} │
└───────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/ip-address-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "IP address functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for working with IP addresses."
---




# Functions for working with IP addresses [¶](https://www.tinybird.co/docs/about:blank#functions-for-working-with-ip-addresses)

## IPv4NumToString(num) [¶](https://www.tinybird.co/docs/about:blank#ipv4numtostringnum)

Takes a UInt32 number. Interprets it as an IPv4 address in big endian. Returns a string containing the corresponding IPv4 address in the format A.B.C.d (dot-separated numbers in decimal form).

Alias: `INET_NTOA`.

## IPv4StringToNum(s) [¶](https://www.tinybird.co/docs/about:blank#ipv4stringtonums)

The reverse function of IPv4NumToString. If the IPv4 address has an invalid format, it throws exception.

Alias: `INET_ATON`.

## IPv4StringToNumOrDefault(s) [¶](https://www.tinybird.co/docs/about:blank#ipv4stringtonumordefaults)

Same as `IPv4StringToNum` , but if the IPv4 address has an invalid format, it returns 0.

## IPv4StringToNumOrNull(s) [¶](https://www.tinybird.co/docs/about:blank#ipv4stringtonumornulls)

Same as `IPv4StringToNum` , but if the IPv4 address has an invalid format, it returns null.

## IPv4NumToStringClassC(num) [¶](https://www.tinybird.co/docs/about:blank#ipv4numtostringclasscnum)

Similar to IPv4NumToString, but using xxx instead of the last octet.

Example:

SELECT
    IPv4NumToStringClassC(ClientIP) AS k,
    count() AS c
FROM test.hits
GROUP BY k
ORDER BY c DESC
LIMIT 10 ┌─k──────────────┬─────c─┐
│ 83.149.9.xxx   │ 26238 │
│ 217.118.81.xxx │ 26074 │
│ 213.87.129.xxx │ 25481 │
│ 83.149.8.xxx   │ 24984 │
│ 217.118.83.xxx │ 22797 │
│ 78.25.120.xxx  │ 22354 │
│ 213.87.131.xxx │ 21285 │
│ 78.25.121.xxx  │ 20887 │
│ 188.162.65.xxx │ 19694 │
│ 83.149.48.xxx  │ 17406 │
└────────────────┴───────┘
## Pv6NumToString(x) [¶](https://www.tinybird.co/docs/about:blank#pv6numtostringx)

Accepts a FixedString(16) value containing the IPv6 address in binary format. Returns a string containing this address in text format. IPv6-mapped IPv4 addresses are output in the format ::ffff:111.222.33.44.

Alias: `INET6_NTOA`.

Examples:

SELECT IPv6NumToString(toFixedString(unhex('2A0206B8000000000000000000000011'), 16)) AS addr ┌─addr─────────┐
│ 2a02:6b8::11 │
└──────────────┘ SELECT
    IPv6NumToString(ClientIP6 AS k),
    count() AS c
FROM hits_all
WHERE EventDate = today() AND substring(ClientIP6, 1, 12) != unhex('00000000000000000000FFFF')
GROUP BY k
ORDER BY c DESC
LIMIT 10 ┌─IPv6NumToString(ClientIP6)──────────────┬─────c─┐
│ 2a02:2168:aaa:bbbb::2                   │ 24695 │
│ 2a02:2698:abcd:abcd:abcd:abcd:8888:5555 │ 22408 │
│ 2a02:6b8:0:fff::ff                      │ 16389 │
│ 2a01:4f8:111:6666::2                    │ 16016 │
│ 2a02:2168:888:222::1                    │ 15896 │
│ 2a01:7e00::ffff:ffff:ffff:222           │ 14774 │
│ 2a02:8109:eee:ee:eeee:eeee:eeee:eeee    │ 14443 │
│ 2a02:810b:8888:888:8888:8888:8888:8888  │ 14345 │
│ 2a02:6b8:0:444:4444:4444:4444:4444      │ 14279 │
│ 2a01:7e00::ffff:ffff:ffff:ffff          │ 13880 │
└─────────────────────────────────────────┴───────┘ SELECT
    IPv6NumToString(ClientIP6 AS k),
    count() AS c
FROM hits_all
WHERE EventDate = today()
GROUP BY k
ORDER BY c DESC
LIMIT 10 ┌─IPv6NumToString(ClientIP6)─┬──────c─┐
│ ::ffff:94.26.111.111       │ 747440 │
│ ::ffff:37.143.222.4        │ 529483 │
│ ::ffff:5.166.111.99        │ 317707 │
│ ::ffff:46.38.11.77         │ 263086 │
│ ::ffff:79.105.111.111      │ 186611 │
│ ::ffff:93.92.111.88        │ 176773 │
│ ::ffff:84.53.111.33        │ 158709 │
│ ::ffff:217.118.11.22       │ 154004 │
│ ::ffff:217.118.11.33       │ 148449 │
│ ::ffff:217.118.11.44       │ 148243 │
└────────────────────────────┴────────┘
## IPv6StringToNum [¶](https://www.tinybird.co/docs/about:blank#ipv6stringtonum)

The reverse function of IPv6NumToString. If the IPv6 address has an invalid format, it throws exception.

If the input string contains a valid IPv4 address, returns its IPv6 equivalent. HEX can be uppercase or lowercase.

Alias: `INET6_ATON`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

IPv6StringToNum(string)
### Argument [¶](https://www.tinybird.co/docs/about:blank#argument)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- IPv6 address in binary format. FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT addr, cutIPv6(IPv6StringToNum(addr), 0, 0) FROM (SELECT ['notaddress', '127.0.0.1', '1111::ffff'] AS addr) ARRAY JOIN addr Result:

┌─addr───────┬─cutIPv6(IPv6StringToNum(addr), 0, 0)─┐
│ notaddress │ ::                                   │
│ 127.0.0.1  │ ::ffff:127.0.0.1                     │
│ 1111::ffff │ 1111::ffff                           │
└────────────┴──────────────────────────────────────┘
## IPv6StringToNumOrDefault(s) [¶](https://www.tinybird.co/docs/about:blank#ipv6stringtonumordefaults)

Same as `IPv6StringToNum` , but if the IPv6 address has an invalid format, it returns 0.

## IPv6StringToNumOrNull(s) [¶](https://www.tinybird.co/docs/about:blank#ipv6stringtonumornulls)

Same as `IPv6StringToNum` , but if the IPv6 address has an invalid format, it returns null.

## IPv4ToIPv6(x) [¶](https://www.tinybird.co/docs/about:blank#ipv4toipv6x)

Takes a `UInt32` number. Interprets it as an IPv4 address in big endian. Returns a `FixedString(16)` value containing the IPv6 address in binary format. Examples:

SELECT IPv6NumToString(IPv4ToIPv6(IPv4StringToNum('192.168.0.1'))) AS addr ┌─addr───────────────┐
│ ::ffff:192.168.0.1 │
└────────────────────┘
## cutIPv6(x, bytesToCutForIPv6, bytesToCutForIPv4) [¶](https://www.tinybird.co/docs/about:blank#cutipv6x-bytestocutforipv6-bytestocutforipv4)

Accepts a FixedString(16) value containing the IPv6 address in binary format. Returns a string containing the address of the specified number of bytes removed in text format. For example:

WITH
    IPv6StringToNum('2001:0DB8:AC10:FE01:FEED:BABE:CAFE:F00D') AS ipv6,
    IPv4ToIPv6(IPv4StringToNum('192.168.0.1')) AS ipv4
SELECT
    cutIPv6(ipv6, 2, 0),
    cutIPv6(ipv4, 0, 2) ┌─cutIPv6(ipv6, 2, 0)─────────────────┬─cutIPv6(ipv4, 0, 2)─┐
│ 2001:db8:ac10:fe01:feed:babe:cafe:0 │ ::ffff:192.168.0.0  │
└─────────────────────────────────────┴─────────────────────┘
## IPv4CIDRToRange(ipv4, Cidr), [¶](https://www.tinybird.co/docs/about:blank#ipv4cidrtorangeipv4-cidr)

Accepts an IPv4 and an UInt8 value containing the CIDR. Return a tuple with two IPv4 containing the lower range and the higher range of the subnet.

SELECT IPv4CIDRToRange(toIPv4('192.168.5.2'), 16) ┌─IPv4CIDRToRange(toIPv4('192.168.5.2'), 16)─┐
│ ('192.168.0.0','192.168.255.255')          │
└────────────────────────────────────────────┘
## IPv6CIDRToRange(ipv6, Cidr), [¶](https://www.tinybird.co/docs/about:blank#ipv6cidrtorangeipv6-cidr)

Accepts an IPv6 and an UInt8 value containing the CIDR. Return a tuple with two IPv6 containing the lower range and the higher range of the subnet.

SELECT IPv6CIDRToRange(toIPv6('2001:0db8:0000:85a3:0000:0000:ac1f:8001'), 32) ┌─IPv6CIDRToRange(toIPv6('2001:0db8:0000:85a3:0000:0000:ac1f:8001'), 32)─┐
│ ('2001:db8::','2001:db8:ffff:ffff:ffff:ffff:ffff:ffff')                │
└────────────────────────────────────────────────────────────────────────┘
## toIPv4 [¶](https://www.tinybird.co/docs/about:blank#toipv4)

Like `IPv4StringToNum` ) but takes a string form of IPv4 address and returns value of IPv4 type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv4(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : IPv4 address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `string`   converted to the IPv4 address. IPv4.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT toIPv4('171.225.130.45') Result:

┌─toIPv4('171.225.130.45')─┐
│ 171.225.130.45           │
└──────────────────────────┘ Query:

WITH
    '171.225.130.45' as IPv4_string
SELECT
    hex(IPv4StringToNum(IPv4_string)),
    hex(toIPv4(IPv4_string)) Result:

┌─hex(IPv4StringToNum(IPv4_string))─┬─hex(toIPv4(IPv4_string))─┐
│ ABE1822D                          │ ABE1822D                 │
└───────────────────────────────────┴──────────────────────────┘
## toIPv4OrDefault [¶](https://www.tinybird.co/docs/about:blank#toipv4ordefault)

Same as `toIPv4` , but if the IPv4 address has an invalid format, it returns `0.0.0.0` (0 IPv4), or the provided IPv4 default.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv4OrDefault(string[, default])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : IP address. String.
- `default`   (optional): The value to return if `string`   has an invalid format. IPv4.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `string`   converted to the current IPv4 address. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH
    '::ffff:127.0.0.1' AS valid_IPv6_string,
    'fe80:2030:31:24' AS invalid_IPv6_string
SELECT
    toIPv4OrDefault(valid_IPv6_string) AS valid,
    toIPv4OrDefault(invalid_IPv6_string) AS default,
    toIPv4OrDefault(invalid_IPv6_string, toIPv4('1.1.1.1')) AS provided_default Result:

┌─valid───┬─default─┬─provided_default─┐
│ 0.0.0.0 │ 0.0.0.0 │ 1.1.1.1          │
└─────────┴─────────┴──────────────────┘
## toIPv4OrNull [¶](https://www.tinybird.co/docs/about:blank#toipv4ornull)

Same as `toIPv4` , but if the IPv4 address has an invalid format, it returns null.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv4OrNull(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `string`   converted to the current IPv4 address, or null if `string`   is an invalid address. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH 'fe80:2030:31:24' AS invalid_IPv6_string
SELECT toIPv4OrNull(invalid_IPv6_string) Result:

┌─toIPv4OrNull(invalid_IPv6_string)─┐
│ ᴺᵁᴸᴸ                              │
└───────────────────────────────────┘
## toIPv4OrZero [¶](https://www.tinybird.co/docs/about:blank#toipv4orzero)

Same as `toIPv4` , but if the IPv4 address has an invalid format, it returns `0.0.0.0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv4OrZero(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `string`   converted to the current IPv4 address, or `0.0.0.0`   if `string`   is an invalid address. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH 'Not an IP address' AS invalid_IPv6_string
SELECT toIPv4OrZero(invalid_IPv6_string) Result:

┌─toIPv4OrZero(invalid_IPv6_string)─┐
│ 0.0.0.0                           │
└───────────────────────────────────┘
## toIPv6 [¶](https://www.tinybird.co/docs/about:blank#toipv6)

Converts a string form of IPv6 address to IPv6 type. If the IPv6 address has an invalid format, returns an empty value. Similar to IPv6StringToNum function, which converts IPv6 address to binary format.

If the input string contains a valid IPv4 address, then the IPv6 equivalent of the IPv4 address is returned.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv6(string)
### Argument [¶](https://www.tinybird.co/docs/about:blank#argument)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- IP address. IPv6.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

WITH '2001:438:ffff::407d:1bc1' AS IPv6_string
SELECT
    hex(IPv6StringToNum(IPv6_string)),
    hex(toIPv6(IPv6_string)) Result:

┌─hex(IPv6StringToNum(IPv6_string))─┬─hex(toIPv6(IPv6_string))─────────┐
│ 20010438FFFF000000000000407D1BC1  │ 20010438FFFF000000000000407D1BC1 │
└───────────────────────────────────┴──────────────────────────────────┘ Query:

SELECT toIPv6('127.0.0.1') Result:

┌─toIPv6('127.0.0.1')─┐
│ ::ffff:127.0.0.1    │
└─────────────────────┘
## toIPv6OrDefault [¶](https://www.tinybird.co/docs/about:blank#toipv6ordefault)

Same as `toIPv6` , but if the IPv6 address has an invalid format, it returns `::` (0 IPv6) or the provided IPv6 default.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv6OrDefault(string[, default])
### Argument [¶](https://www.tinybird.co/docs/about:blank#argument)

- `string`   : IP address. String.
- `default`   (optional): The value to return if `string`   has an invalid format. IPv6.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- IPv6 address IPv6, otherwise `::`   or the provided optional default if `string`   has an invalid format.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH
    '127.0.0.1' AS valid_IPv4_string,
    '127.0.0.1.6' AS invalid_IPv4_string
SELECT
    toIPv6OrDefault(valid_IPv4_string) AS valid,
    toIPv6OrDefault(invalid_IPv4_string) AS default,
    toIPv6OrDefault(invalid_IPv4_string, toIPv6('1.1.1.1')) AS provided_default Result:

┌─valid────────────┬─default─┬─provided_default─┐
│ ::ffff:127.0.0.1 │ ::      │ ::ffff:1.1.1.1   │
└──────────────────┴─────────┴──────────────────┘
## toIPv6OrNull [¶](https://www.tinybird.co/docs/about:blank#toipv6ornull)

Same as `toIPv6` , but if the IPv6 address has an invalid format, it returns null.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv6OrNull(string)
### Argument [¶](https://www.tinybird.co/docs/about:blank#argument)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- IP address. IPv6, or null if `string`   isn't a valid format.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH '127.0.0.1.6' AS invalid_IPv4_string
SELECT toIPv6OrNull(invalid_IPv4_string) Result:

┌─toIPv6OrNull(invalid_IPv4_string)─┐
│ ᴺᵁᴸᴸ                              │
└───────────────────────────────────┘
## toIPv6OrZero [¶](https://www.tinybird.co/docs/about:blank#toipv6orzero)

Same as `toIPv6` , but if the IPv6 address has an invalid format, it returns `::`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toIPv6OrZero(string)
### Argument [¶](https://www.tinybird.co/docs/about:blank#argument)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- IP address. IPv6, or `::`   if `string`   isn't a valid format.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH '127.0.0.1.6' AS invalid_IPv4_string
SELECT toIPv6OrZero(invalid_IPv4_string) Result:

┌─toIPv6OrZero(invalid_IPv4_string)─┐
│ ::                                │
└───────────────────────────────────┘
## IPv6StringToNumOrDefault(s) [¶](https://www.tinybird.co/docs/about:blank#ipv6stringtonumordefaults)

Same as `toIPv6` , but if the IPv6 address has an invalid format, it returns 0.

## IPv6StringToNumOrNull(s) [¶](https://www.tinybird.co/docs/about:blank#ipv6stringtonumornulls)

Same as `toIPv6` , but if the IPv6 address has an invalid format, it returns null.

## isIPv4String [¶](https://www.tinybird.co/docs/about:blank#isipv4string)

Determines whether the input string is an IPv4 address or not. If `string` is IPv6 address returns `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isIPv4String(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `string`   is IPv4 address, `0`   otherwise. UInt8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT addr, isIPv4String(addr) FROM ( SELECT ['0.0.0.0', '127.0.0.1', '::ffff:127.0.0.1'] AS addr ) ARRAY JOIN addr Result:

┌─addr─────────────┬─isIPv4String(addr)─┐
│ 0.0.0.0          │                  1 │
│ 127.0.0.1        │                  1 │
│ ::ffff:127.0.0.1 │                  0 │
└──────────────────┴────────────────────┘
## isIPv6String [¶](https://www.tinybird.co/docs/about:blank#isipv6string)

Determines whether the input string is an IPv6 address or not. If `string` is IPv4 address returns `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isIPv6String(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : IP address. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `string`   is IPv6 address, `0`   otherwise. UInt8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT addr, isIPv6String(addr) FROM ( SELECT ['::', '1111::ffff', '::ffff:127.0.0.1', '127.0.0.1'] AS addr ) ARRAY JOIN addr Result:

┌─addr─────────────┬─isIPv6String(addr)─┐
│ ::               │                  1 │
│ 1111::ffff       │                  1 │
│ ::ffff:127.0.0.1 │                  1 │
│ 127.0.0.1        │                  0 │
└──────────────────┴────────────────────┘
## isIPAddressInRange [¶](https://www.tinybird.co/docs/about:blank#isipaddressinrange)

Determines if an IP address is contained in a network represented in the CIDR notation. Returns `1` if true, or `0` otherwise.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isIPAddressInRange(address, prefix) This function accepts both IPv4 and IPv6 addresses (and networks) represented as strings. It returns `0` if the IP version of the address and the CIDR don't match.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `address`   : An IPv4 or IPv6 address. String.
- `prefix`   : An IPv4 or IPv6 network prefix in CIDR. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   or `0`   . UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT isIPAddressInRange('127.0.0.1', '127.0.0.0/8') Result:

┌─isIPAddressInRange('127.0.0.1', '127.0.0.0/8')─┐
│                                              1 │
└────────────────────────────────────────────────┘ Query:

SELECT isIPAddressInRange('127.0.0.1', 'ffff::/16') Result:

┌─isIPAddressInRange('127.0.0.1', 'ffff::/16')─┐
│                                            0 │
└──────────────────────────────────────────────┘ Query:

SELECT isIPAddressInRange('::ffff:192.168.0.1', '::ffff:192.168.0.4/128') Result:

┌─isIPAddressInRange('::ffff:192.168.0.1', '::ffff:192.168.0.4/128')─┐
│                                                                  0 │
└────────────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/hash-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Hash functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for hashing data."
---




# Hash functions [¶](https://www.tinybird.co/docs/about:blank#hash-functions)

You can use hash functions for the deterministic pseudo-random shuffling of elements.

`Simhash` is a hash function, which returns close hash values for similar arguments.

## halfMD5 [¶](https://www.tinybird.co/docs/about:blank#halfmd5)

Interprets all the input parameters as strings and calculates the MD5 hash value for each of them. Then combines hashes, takes the first 8 bytes of the hash of the resulting string, and interprets them as `UInt64` in big-endian byte order.

halfMD5(par1, ...) The function is relatively slow (5 million short strings per second per processor core). Consider using the sipHash64 function instead.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT halfMD5(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS halfMD5hash, toTypeName(halfMD5hash) AS type ┌────────halfMD5hash─┬─type───┐
│ 186182704141653334 │ UInt64 │
└────────────────────┴────────┘
## MD4 [¶](https://www.tinybird.co/docs/about:blank#md4)

Calculates the MD4 from a string and returns the resulting set of bytes as FixedString(16).

## MD5 [¶](https://www.tinybird.co/docs/about:blank#md5)

Calculates the MD5 from a string and returns the resulting set of bytes as FixedString(16). If you don't need MD5 in particular, but you need a decent cryptographic 128-bit hash, use the ‘sipHash128’ function instead. If you want to get the same result as output by the md5sum utility, use lower(hex(MD5(s))).

## RIPEMD160 [¶](https://www.tinybird.co/docs/about:blank#ripemd160)

Produces RIPEMD-160 hash value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

RIPEMD160(input)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `input`   : Input string. String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A 160-bit `RIPEMD-160`   hash value of type FixedString(20).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Use the hex function to represent the result as a hex-encoded string.

Query:

SELECT HEX(RIPEMD160('The quick brown fox jumps over the lazy dog')) ┌─HEX(RIPEMD160('The quick brown fox jumps over the lazy dog'))─┐
│ 37F332F68DB77BD9D7EDD4969571AD671CF9DD3B                      │
└───────────────────────────────────────────────────────────────┘
## sipHash64 [¶](https://www.tinybird.co/docs/about:blank#siphash64)

Produces a 64-bit SipHash hash value.

sipHash64(par1,...) This is a cryptographic hash function. It works at least three times faster than the MD5 hash function.

The function interprets all the input parameters as strings and calculates the hash value for each of them. It then combines the hashes by the following algorithm:

1. The first and the second hash value are concatenated to an array which is hashed.
2. The previously calculated hash value and the hash of the third input parameter are hashed in a similar way.
3. This calculation is repeated for all remaining hash values of the original input.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of input parameters of any of the supported data types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

Note that the calculated hash values may be equal for the same input values of different argument types. This affects for example integer types of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT sipHash64(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS SipHash, toTypeName(SipHash) AS type ┌──────────────SipHash─┬─type───┐
│ 11400366955626497465 │ UInt64 │
└──────────────────────┴────────┘
## sipHash64Keyed [¶](https://www.tinybird.co/docs/about:blank#siphash64keyed)

Same as sipHash64 but additionally takes an explicit key argument instead of using a fixed key.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sipHash64Keyed((k0, k1), par1,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Same as sipHash64, but the first argument is a tuple of two UInt64 values representing the key.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT sipHash64Keyed((506097522914230528, 1084818905618843912), array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS SipHash, toTypeName(SipHash) AS type ┌─────────────SipHash─┬─type───┐
│ 8017656310194184311 │ UInt64 │
└─────────────────────┴────────┘
## sipHash128 [¶](https://www.tinybird.co/docs/about:blank#siphash128)

Like sipHash64 but produces a 128-bit hash value, i.e. the final xor-folding state is done up to 128 bits.

This 128-bit variant differs from the reference implementation and it's weaker. This version exists because, when it was written, there was no official 128-bit extension for SipHash. New projects should probably use sipHash128Reference.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sipHash128(par1,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Same as for sipHash64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 128-bit `SipHash` hash value of type FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hex(sipHash128('foo', '\x01', 3)) Result:

┌─hex(sipHash128('foo', '', 3))────┐
│ 9DE516A64A414D4B1B609415E4523F24 │
└──────────────────────────────────┘
## sipHash128Keyed [¶](https://www.tinybird.co/docs/about:blank#siphash128keyed)

Same as sipHash128 but additionally takes an explicit key argument instead of using a fixed key.

This 128-bit variant differs from the reference implementation and it's weaker. This version exists because, when it was written, there was no official 128-bit extension for SipHash. New projects should probably use sipHash128ReferenceKeyed.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sipHash128Keyed((k0, k1), par1,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Same as sipHash128, but the first argument is a tuple of two UInt64 values representing the key.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 128-bit `SipHash` hash value of type FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hex(sipHash128Keyed((506097522914230528, 1084818905618843912),'foo', '\x01', 3)) Result:

┌─hex(sipHash128Keyed((506097522914230528, 1084818905618843912), 'foo', '', 3))─┐
│ B8467F65C8B4CFD9A5F8BD733917D9BF                                              │
└───────────────────────────────────────────────────────────────────────────────┘
## sipHash128Reference [¶](https://www.tinybird.co/docs/about:blank#siphash128reference)

Like sipHash128 but implements the 128-bit algorithm from the original authors of SipHash.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sipHash128Reference(par1,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Same as for sipHash128.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 128-bit `SipHash` hash value of type FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hex(sipHash128Reference('foo', '\x01', 3)) Result:

┌─hex(sipHash128Reference('foo', '', 3))─┐
│ 4D1BE1A22D7F5933C0873E1698426260       │
└────────────────────────────────────────┘
## sipHash128ReferenceKeyed [¶](https://www.tinybird.co/docs/about:blank#siphash128referencekeyed)

Same as sipHash128Reference but additionally takes an explicit key argument instead of using a fixed key.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sipHash128ReferenceKeyed((k0, k1), par1,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Same as sipHash128Reference, but the first argument is a tuple of two UInt64 values representing the key.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 128-bit `SipHash` hash value of type FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hex(sipHash128ReferenceKeyed((506097522914230528, 1084818905618843912),'foo', '\x01', 3)) Result:

┌─hex(sipHash128ReferenceKeyed((506097522914230528, 1084818905618843912), 'foo', '', 3))─┐
│ 630133C9722DC08646156B8130C4CDC8                                                       │
└────────────────────────────────────────────────────────────────────────────────────────┘
## cityHash64 [¶](https://www.tinybird.co/docs/about:blank#cityhash64)

Produces a 64-bit CityHash hash value.

cityHash64(par1,...) This is a fast non-cryptographic hash function. It uses the CityHash algorithm for string parameters and implementation-specific fast non-cryptographic hash function for parameters with other data types. The function uses the CityHash combinator to get the final results.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Call example:

SELECT cityHash64(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS CityHash, toTypeName(CityHash) AS type ┌─────────────CityHash─┬─type───┐
│ 12072650598913549138 │ UInt64 │
└──────────────────────┴────────┘ The following example shows how to compute the checksum of the entire table with accuracy up to the row order:

SELECT groupBitXor(cityHash64(*)) FROM table
## intHash32 [¶](https://www.tinybird.co/docs/about:blank#inthash32)

Calculates a 32-bit hash code from any type of integer. This is a relatively fast non-cryptographic hash function of average quality for numbers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intHash32(int)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `int`   : Integer to hash. (U)Int*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- 32-bit hash code. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT intHash32(42) Result:

┌─intHash32(42)─┐
│    1228623923 │
└───────────────┘
## intHash64 [¶](https://www.tinybird.co/docs/about:blank#inthash64)

Calculates a 64-bit hash code from any type of integer. This is a relatively fast non-cryptographic hash function of average quality for numbers. It works faster than intHash32.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intHash64(int)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `int`   : Integer to hash. (U)Int*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- 64-bit hash code. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT intHash64(42) Result:

┌────────intHash64(42)─┐
│ 11490350930367293593 │
└──────────────────────┘
## SHA1, SHA224, SHA256, SHA512, SHA512_256 [¶](https://www.tinybird.co/docs/about:blank#sha1-sha224-sha256-sha512-sha512-256)

Calculates SHA-1, SHA-224, SHA-256, SHA-512, SHA-512-256 hash from a string and returns the resulting set of bytes as FixedString.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SHA1('s')
...
SHA512('s')
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   : Input string for SHA hash calculation. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- SHA hash as a hex-unencoded FixedString. SHA-1 returns as FixedString(20), SHA-224 as FixedString(28), SHA-256: FixedString(32), SHA-512: FixedString(64). FixedString.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Use the hex function to represent the result as a hex-encoded string.

Query:

SELECT hex(SHA1('abc')) Result:

┌─hex(SHA1('abc'))─────────────────────────┐
│ A9993E364706816ABA3E25717850C26C9CD0D89D │
└──────────────────────────────────────────┘
## BLAKE3 [¶](https://www.tinybird.co/docs/about:blank#blake3)

Calculates BLAKE3 hash string and returns the resulting set of bytes as FixedString.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

BLAKE3('s') This cryptographic hash-function is integrated with the BLAKE3 Rust library. The function is rather fast and shows approximately two times faster performance compared to SHA-2, while generating hashes of the same length as SHA-256.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- s - input string for BLAKE3 hash calculation. String.

### Return value [¶](https://www.tinybird.co/docs/about:blank#return-value)

- BLAKE3 hash as a byte array with type FixedString(32). FixedString.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Use function hex to represent the result as a hex-encoded string.

Query:

SELECT hex(BLAKE3('ABC')) Result:

┌─hex(BLAKE3('ABC'))───────────────────────────────────────────────┐
│ D1717274597CF0289694F75D96D444B992A096F1AFD8E7BBFA6EBB1D360FEDFC │
└──────────────────────────────────────────────────────────────────┘
## URLHash(url[, N]) [¶](https://www.tinybird.co/docs/about:blank#urlhashurl-n)

A fast, decent-quality non-cryptographic hash function for a string obtained from a URL using some type of normalization. `URLHash(s)` – Calculates a hash from a string without one of the trailing symbols `/`, `?` or `#` at the end, if present. `URLHash(s, N)` – Calculates a hash from a string up to the N level in the URL hierarchy, without one of the trailing symbols `/`, `?` or `#` at the end, if present. Levels are the same as in URLHierarchy.

## farmFingerprint64 [¶](https://www.tinybird.co/docs/about:blank#farmfingerprint64)

## farmHash64 [¶](https://www.tinybird.co/docs/about:blank#farmhash64)

Produces a 64-bit FarmHash or Fingerprint value. `farmFingerprint64` is preferred for a stable and portable value.

farmFingerprint64(par1, ...)
farmHash64(par1, ...) These functions use the `Fingerprint64` and `Hash64` methods respectively from all available methods.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT farmHash64(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS FarmHash, toTypeName(FarmHash) AS type ┌─────────────FarmHash─┬─type───┐
│ 17790458267262532859 │ UInt64 │
└──────────────────────┴────────┘
## javaHash [¶](https://www.tinybird.co/docs/about:blank#javahash)

Calculates JavaHash from a string, Byte, Short, Integer, Long. This hash function is neither fast nor having a good quality. The only reason to use it's when this algorithm is already used in another system and you have to calculate exactly the same result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT javaHash('')
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A `Int32` data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT javaHash(toInt32(123)) Result:

┌─javaHash(toInt32(123))─┐
│               123      │
└────────────────────────┘ Query:

SELECT javaHash('Hello, world!') Result:

┌─javaHash('Hello, world!')─┐
│               -1880044555 │
└───────────────────────────┘
## javaHashUTF16LE [¶](https://www.tinybird.co/docs/about:blank#javahashutf16le)

Calculates JavaHash from a string, assuming it contains bytes representing a string in UTF-16LE encoding.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

javaHashUTF16LE(stringUtf16le)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `stringUtf16le`   : a string in UTF-16LE encoding.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A `Int32` data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Correct query with UTF-16LE encoded string.

Query:

SELECT javaHashUTF16LE(convertCharset('test', 'utf-8', 'utf-16le')) Result:

┌─javaHashUTF16LE(convertCharset('test', 'utf-8', 'utf-16le'))─┐
│                                                      3556498 │
└──────────────────────────────────────────────────────────────┘
## hiveHash [¶](https://www.tinybird.co/docs/about:blank#hivehash)

Calculates `HiveHash` from a string.

SELECT hiveHash('') This is just JavaHash with zeroed out sign bit. This function is used in Apache Hive for versions before 3.0. This hash function is neither fast nor having a good quality. The only reason to use it's when this algorithm is already used in another system and you have to calculate exactly the same result.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `hiveHash`   hash value. Int32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hiveHash('Hello, world!') Result:

┌─hiveHash('Hello, world!')─┐
│                 267439093 │
└───────────────────────────┘
## metroHash64 [¶](https://www.tinybird.co/docs/about:blank#metrohash64)

Produces a 64-bit MetroHash hash value.

metroHash64(par1, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A UInt64 data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT metroHash64(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS MetroHash, toTypeName(MetroHash) AS type ┌────────────MetroHash─┬─type───┐
│ 14235658766382344533 │ UInt64 │
└──────────────────────┴────────┘
## jumpConsistentHash [¶](https://www.tinybird.co/docs/about:blank#jumpconsistenthash)

Calculates JumpConsistentHash form a UInt64. Accepts two arguments: a UInt64-type key and the number of buckets. Returns Int32. For more information, see the link: JumpConsistentHash

## kostikConsistentHash [¶](https://www.tinybird.co/docs/about:blank#kostikconsistenthash)

An O(1) time and space consistent hash algorithm by Konstantin 'kostik' Oblakov. Previously `yandexConsistentHash`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

kostikConsistentHash(input, n) Alias: `yandexConsistentHash` (left for backwards compatibility sake).

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `input`   : A UInt64-type key UInt64.
- `n`   : Number of buckets. UInt16.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A UInt16 data type hash value.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

It is efficient only if n <= 32768.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT kostikConsistentHash(16045690984833335023, 2) ┌─kostikConsistentHash(16045690984833335023, 2)─┐
│                                             1 │
└───────────────────────────────────────────────┘
## murmurHash2_32, murmurHash2_64 [¶](https://www.tinybird.co/docs/about:blank#murmurhash2-32-murmurhash2-64)

Produces a MurmurHash2 hash value.

murmurHash2_32(par1, ...)
murmurHash2_64(par1, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Both functions take a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The `murmurHash2_32`   function returns hash value having the UInt32 data type.
- The `murmurHash2_64`   function returns hash value having the UInt64 data type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT murmurHash2_64(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS MurmurHash2, toTypeName(MurmurHash2) AS type ┌──────────MurmurHash2─┬─type───┐
│ 11832096901709403633 │ UInt64 │
└──────────────────────┴────────┘
## gccMurmurHash [¶](https://www.tinybird.co/docs/about:blank#gccmurmurhash)

Calculates a 64-bit MurmurHash2 hash value using the same hash seed as gcc. It is portable between Clang and GCC builds.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

gccMurmurHash(par1, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `par1, ...`   : A variable number of parameters that can be any of the supported data types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Calculated hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    gccMurmurHash(1, 2, 3) AS res1,
    gccMurmurHash(('a', [1, 2, 3], 4, (4, ['foo', 'bar'], 1, (1, 2)))) AS res2 Result:

┌─────────────────res1─┬────────────────res2─┐
│ 12384823029245979431 │ 1188926775431157506 │
└──────────────────────┴─────────────────────┘
## kafkaMurmurHash [¶](https://www.tinybird.co/docs/about:blank#kafkamurmurhash)

Calculates a 32-bit MurmurHash2 hash value using the same hash seed as Kafka and without the highest bit to be compatible with Default Partitioner.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

MurmurHash(par1, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `par1, ...`   : A variable number of parameters that can be any of the supported data types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Calculated hash value. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    kafkaMurmurHash('foobar') AS res1,
    kafkaMurmurHash(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS res2 Result:

┌───────res1─┬─────res2─┐
│ 1357151166 │ 85479775 │
└────────────┴──────────┘
## murmurHash3_32, murmurHash3_64 [¶](https://www.tinybird.co/docs/about:blank#murmurhash3-32-murmurhash3-64)

Produces a MurmurHash3 hash value.

murmurHash3_32(par1, ...)
murmurHash3_64(par1, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

Both functions take a variable number of input parameters. Arguments can be any of the supported data types. For some data types calculated value of hash function may be the same for the same values even if types of arguments differ (integers of different size, named and unnamed `Tuple` with the same data, `Map` and the corresponding `Array(Tuple(key, value))` type with the same data).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The `murmurHash3_32`   function returns a UInt32 data type hash value.
- The `murmurHash3_64`   function returns a UInt64 data type hash value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT murmurHash3_32(array('e','x','a'), 'mple', 10, toDateTime('2019-06-15 23:00:00')) AS MurmurHash3, toTypeName(MurmurHash3) AS type ┌─MurmurHash3─┬─type───┐
│     2152717 │ UInt32 │
└─────────────┴────────┘
## murmurHash3_128 [¶](https://www.tinybird.co/docs/about:blank#murmurhash3-128)

Produces a 128-bit MurmurHash3 hash value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

murmurHash3_128(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : A list of expressions. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 128-bit `MurmurHash3` hash value. FixedString(16).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT hex(murmurHash3_128('foo', 'foo', 'foo')) Result:

┌─hex(murmurHash3_128('foo', 'foo', 'foo'))─┐
│ F8F7AD9B6CD4CF117A71E277E2EC2931          │
└───────────────────────────────────────────┘
## xxh3 [¶](https://www.tinybird.co/docs/about:blank#xxh3)

Produces a 64-bit xxh3 hash value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

xxh3(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : A list of expressions of any data type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A 64-bit `xxh3` hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT xxh3('Hello', 'world') Result:

┌─xxh3('Hello', 'world')─┐
│    5607458076371731292 │
└────────────────────────┘
## xxHash32, xxHash64 [¶](https://www.tinybird.co/docs/about:blank#xxhash32-xxhash64)

Calculates `xxHash` from a string. It is proposed in two flavors, 32 and 64 bits.

SELECT xxHash32('')

OR

SELECT xxHash64('')
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt32/64.

The return type will be `UInt32` for `xxHash32` and `UInt64` for `xxHash64`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT xxHash32('Hello, world!') Result:

┌─xxHash32('Hello, world!')─┐
│                 834093149 │
└───────────────────────────┘
## ngramSimHash [¶](https://www.tinybird.co/docs/about:blank#ngramsimhash)

Splits a ASCII string into n-grams of `ngramsize` symbols and returns the n-gram `simhash` . Is case sensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramSimHash(string[, ngramsize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramSimHash('Bazinga') AS Hash Result:

┌───────Hash─┐
│ 1627567969 │
└────────────┘
## ngramSimHashCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#ngramsimhashcaseinsensitive)

Splits a ASCII string into n-grams of `ngramsize` symbols and returns the n-gram `simhash` . Is case insensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramSimHashCaseInsensitive(string[, ngramsize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramSimHashCaseInsensitive('Bazinga') AS Hash Result:

┌──────Hash─┐
│ 562180645 │
└───────────┘
## ngramSimHashUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramsimhashutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and returns the n-gram `simhash` . Is case sensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramSimHashUTF8(string[, ngramsize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramSimHashUTF8('Bazinga') AS Hash Result:

┌───────Hash─┐
│ 1628157797 │
└────────────┘
## ngramSimHashCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramsimhashcaseinsensitiveutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and returns the n-gram `simhash` . Is case insensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramSimHashCaseInsensitiveUTF8(string[, ngramsize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramSimHashCaseInsensitiveUTF8('Bazinga') AS Hash Result:

┌───────Hash─┐
│ 1636742693 │
└────────────┘
## wordShingleSimHash [¶](https://www.tinybird.co/docs/about:blank#wordshinglesimhash)

Splits a ASCII string into parts (shingles) of `shinglesize` words and returns the word shingle `simhash` . Is case sensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleSimHash(string[, shinglesize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleSimHash('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Hash Result:

┌───────Hash─┐
│ 2328277067 │
└────────────┘
## wordShingleSimHashCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#wordshinglesimhashcaseinsensitive)

Splits a ASCII string into parts (shingles) of `shinglesize` words and returns the word shingle `simhash` . Is case insensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleSimHashCaseInsensitive(string[, shinglesize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleSimHashCaseInsensitive('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Hash Result:

┌───────Hash─┐
│ 2194812424 │
└────────────┘
## wordShingleSimHashUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshinglesimhashutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words and returns the word shingle `simhash` . Is case sensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleSimHashUTF8(string[, shinglesize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleSimHashUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Hash Result:

┌───────Hash─┐
│ 2328277067 │
└────────────┘
## wordShingleSimHashCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshinglesimhashcaseinsensitiveutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words and returns the word shingle `simhash` . Is case insensitive.

Can be used for detection of semi-duplicate strings with bitHammingDistance. The smaller is the Hamming Distance of the calculated `simhashes` of two strings, the more likely these strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleSimHashCaseInsensitiveUTF8(string[, shinglesize])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleSimHashCaseInsensitiveUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Hash Result:

┌───────Hash─┐
│ 2194812424 │
└────────────┘
## wyHash64 [¶](https://www.tinybird.co/docs/about:blank#wyhash64)

Produces a 64-bit wyHash64 hash value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wyHash64(string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hash value. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wyHash64('Bazinga') AS Hash Result:

┌─────────────────Hash─┐
│ 12336419557878201794 │
└──────────────────────┘
## ngramMinHash [¶](https://www.tinybird.co/docs/about:blank#ngramminhash)

Splits a ASCII string into n-grams of `ngramsize` symbols and calculates hash values for each n-gram. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case sensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHash(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHash('Bazinga') AS Tuple Result:

┌─Tuple──────────────────────────────────────┐
│ (18333312859352735453,9054248444481805918) │
└────────────────────────────────────────────┘
## ngramMinHashCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#ngramminhashcaseinsensitive)

Splits a ASCII string into n-grams of `ngramsize` symbols and calculates hash values for each n-gram. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case insensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashCaseInsensitive(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashCaseInsensitive('Bazinga') AS Tuple Result:

┌─Tuple──────────────────────────────────────┐
│ (2106263556442004574,13203602793651726206) │
└────────────────────────────────────────────┘
## ngramMinHashUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramminhashutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and calculates hash values for each n-gram. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case sensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashUTF8(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashUTF8('Bazinga') AS Tuple Result:

┌─Tuple──────────────────────────────────────┐
│ (18333312859352735453,6742163577938632877) │
└────────────────────────────────────────────┘
## ngramMinHashCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramminhashcaseinsensitiveutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and calculates hash values for each n-gram. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case insensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashCaseInsensitiveUTF8(string [, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashCaseInsensitiveUTF8('Bazinga') AS Tuple Result:

┌─Tuple───────────────────────────────────────┐
│ (12493625717655877135,13203602793651726206) │
└─────────────────────────────────────────────┘
## ngramMinHashArg [¶](https://www.tinybird.co/docs/about:blank#ngramminhasharg)

Splits a ASCII string into n-grams of `ngramsize` symbols and returns the n-grams with minimum and maximum hashes, calculated by the ngramMinHash function with the same input. Is case sensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashArg(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   n-grams each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashArg('Bazinga') AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────────────┐
│ (('ous','ick','lic','Hou','kHo','use'),('Hou','lic','ick','ous','ckH','Cli')) │
└───────────────────────────────────────────────────────────────────────────────┘
## ngramMinHashArgCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#ngramminhashargcaseinsensitive)

Splits a ASCII string into n-grams of `ngramsize` symbols and returns the n-grams with minimum and maximum hashes, calculated by the ngramMinHashCaseInsensitive function with the same input. Is case insensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashArgCaseInsensitive(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   n-grams each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashArgCaseInsensitive('Bazinga') AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────────────┐
│ (('ous','ick','lic','kHo','use','Cli'),('kHo','lic','ick','ous','ckH','Hou')) │
└───────────────────────────────────────────────────────────────────────────────┘
## ngramMinHashArgUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramminhashargutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and returns the n-grams with minimum and maximum hashes, calculated by the ngramMinHashUTF8 function with the same input. Is case sensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashArgUTF8(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   n-grams each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashArgUTF8('Bazinga') AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────────────┐
│ (('ous','ick','lic','Hou','kHo','use'),('kHo','Hou','lic','ick','ous','ckH')) │
└───────────────────────────────────────────────────────────────────────────────┘
## ngramMinHashArgCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#ngramminhashargcaseinsensitiveutf8)

Splits a UTF-8 string into n-grams of `ngramsize` symbols and returns the n-grams with minimum and maximum hashes, calculated by the ngramMinHashCaseInsensitiveUTF8 function with the same input. Is case insensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ngramMinHashArgCaseInsensitiveUTF8(string[, ngramsize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `ngramsize`   : The size of an n-gram. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   n-grams each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ngramMinHashArgCaseInsensitiveUTF8('Bazinga') AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────────────┐
│ (('ckH','ous','ick','lic','kHo','use'),('kHo','lic','ick','ous','ckH','Hou')) │
└───────────────────────────────────────────────────────────────────────────────┘
## wordShingleMinHash [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhash)

Splits a ASCII string into parts (shingles) of `shinglesize` words and calculates hash values for each word shingle. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case sensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHash(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHash('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Tuple Result:

┌─Tuple──────────────────────────────────────┐
│ (16452112859864147620,5844417301642981317) │
└────────────────────────────────────────────┘
## wordShingleMinHashCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashcaseinsensitive)

Splits a ASCII string into parts (shingles) of `shinglesize` words and calculates hash values for each word shingle. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case insensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashCaseInsensitive(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashCaseInsensitive('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Tuple Result:

┌─Tuple─────────────────────────────────────┐
│ (3065874883688416519,1634050779997673240) │
└───────────────────────────────────────────┘
## wordShingleMinHashUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words and calculates hash values for each word shingle. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case sensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashUTF8(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Tuple Result:

┌─Tuple──────────────────────────────────────┐
│ (16452112859864147620,5844417301642981317) │
└────────────────────────────────────────────┘
## wordShingleMinHashCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashcaseinsensitiveutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words and calculates hash values for each word shingle. Uses `hashnum` minimum hashes to calculate the minimum hash and `hashnum` maximum hashes to calculate the maximum hash. Returns a tuple with these hashes. Is case insensitive.

Can be used for detection of semi-duplicate strings with tupleHammingDistance. For two strings: if one of the returned hashes is the same for both strings, Tinybird thinks that those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashCaseInsensitiveUTF8(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two hashes: the minimum and the maximum. Tuple(UInt64, UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashCaseInsensitiveUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).') AS Tuple Result:

┌─Tuple─────────────────────────────────────┐
│ (3065874883688416519,1634050779997673240) │
└───────────────────────────────────────────┘
## wordShingleMinHashArg [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhasharg)

Splits a ASCII string into parts (shingles) of `shinglesize` words each and returns the shingles with minimum and maximum word hashes, calculated by the wordshingleMinHash function with the same input. Is case sensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashArg(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   word shingles each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashArg('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).', 1, 3) AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────┐
│ (('OLAP','database','analytical'),('online','oriented','processing')) │
└───────────────────────────────────────────────────────────────────────┘
## wordShingleMinHashArgCaseInsensitive [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashargcaseinsensitive)

Splits a ASCII string into parts (shingles) of `shinglesize` words each and returns the shingles with minimum and maximum word hashes, calculated by the wordShingleMinHashCaseInsensitive function with the same input. Is case insensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashArgCaseInsensitive(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   word shingles each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashArgCaseInsensitive('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).', 1, 3) AS Tuple Result:

┌─Tuple──────────────────────────────────────────────────────────────────┐
│ (('queries','database','analytical'),('oriented','processing','DBMS')) │
└────────────────────────────────────────────────────────────────────────┘
## wordShingleMinHashArgUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashargutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words each and returns the shingles with minimum and maximum word hashes, calculated by the wordShingleMinHashUTF8 function with the same input. Is case sensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashArgUTF8(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   word shingles each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashArgUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).', 1, 3) AS Tuple Result:

┌─Tuple─────────────────────────────────────────────────────────────────┐
│ (('OLAP','database','analytical'),('online','oriented','processing')) │
└───────────────────────────────────────────────────────────────────────┘
## wordShingleMinHashArgCaseInsensitiveUTF8 [¶](https://www.tinybird.co/docs/about:blank#wordshingleminhashargcaseinsensitiveutf8)

Splits a UTF-8 string into parts (shingles) of `shinglesize` words each and returns the shingles with minimum and maximum word hashes, calculated by the wordShingleMinHashCaseInsensitiveUTF8 function with the same input. Is case insensitive.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

wordShingleMinHashArgCaseInsensitiveUTF8(string[, shinglesize, hashnum])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `string`   : String. String.
- `shinglesize`   : The size of a word shingle. Optional. Possible values: any number from `1`   to `25`   . Default value: `3`   . UInt8.
- `hashnum`   : The number of minimum and maximum hashes used to calculate the result. Optional. Possible values: any number from `1`   to `25`   . Default value: `6`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple with two tuples with `hashnum`   word shingles each. Tuple(Tuple(String), Tuple(String)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT wordShingleMinHashArgCaseInsensitiveUTF8('Bazinga® is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).', 1, 3) AS Tuple Result:

┌─Tuple──────────────────────────────────────────────────────────────────┐
│ (('queries','database','analytical'),('oriented','processing','DBMS')) │
└────────────────────────────────────────────────────────────────────────┘
## sqidEncode [¶](https://www.tinybird.co/docs/about:blank#sqidencode)

Encodes numbers as a Sqid which is a YouTube-like ID string. The output alphabet is `abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789`. Do not use this function for hashing - the generated IDs can be decoded back into the original numbers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sqidEncode(number1, ...) Alias: `sqid`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- A variable number of UInt8, UInt16, UInt32 or UInt64 numbers.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A sqid String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT sqidEncode(1, 2, 3, 4, 5) ┌─sqidEncode(1, 2, 3, 4, 5)─┐
│ gXHfJ1C6dN                │
└───────────────────────────┘
## sqidDecode [¶](https://www.tinybird.co/docs/about:blank#sqiddecode)

Decodes a Sqid back into its original numbers. Returns an empty array in case the input string isn't a valid sqid.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sqidDecode(sqid)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- A sqid - String

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The sqid transformed to numbers Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT sqidDecode('gXHfJ1C6dN') ┌─sqidDecode('gXHfJ1C6dN')─┐
│ [1,2,3,4,5]              │
└──────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/geo-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Geo functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for geographical calculations."
---




# Geo functions [¶](https://www.tinybird.co/docs/about:blank#geo-functions)

Geo functions are used to perform geographical calculations on spatial data. These functions are essential for applications that require distance measurements, location-based queries, and other geospatial operations.

## greatCircleDistance [¶](https://www.tinybird.co/docs/about:blank#greatcircledistance)

Calculates the distance between two points on the Earth’s surface using the great-circle formula.

greatCircleDistance(lon1Deg, lat1Deg, lon2Deg, lat2Deg)
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

- `lon1Deg`   : Longitude of the first point in degrees. Range: `[-180°, 180°]`  .
- `lat1Deg`   : Latitude of the first point in degrees. Range: `[-90°, 90°]`  .
- `lon2Deg`   : Longitude of the second point in degrees. Range: `[-180°, 180°]`  .
- `lat2Deg`   : Latitude of the second point in degrees. Range: `[-90°, 90°]`  .

Positive values correspond to North latitude and East longitude, and negative values correspond to South latitude and West longitude.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The distance between two points on the Earth’s surface, in meters.

Generates an exception when the input parameter values fall outside of the range.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT greatCircleDistance(55.755831, 37.617673, -55.755831, -37.617673) AS greatCircleDistance ┌─greatCircleDistance─┐
│            14128352 │
└─────────────────────┘
## geoDistance [¶](https://www.tinybird.co/docs/about:blank#geodistance)

Similar to `greatCircleDistance` but calculates the distance on WGS-84 ellipsoid instead of sphere. This is more precise approximation of the Earth Geoid. The performance is the same as for `greatCircleDistance` (no performance drawback). It is recommended to use `geoDistance` to calculate the distances on Earth.

Technical note: for close enough points, you can calculate the distance using planar approximation with the metric on the tangent plane at the midpoint of the coordinates.

geoDistance(lon1Deg, lat1Deg, lon2Deg, lat2Deg)
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

- `lon1Deg`   : Longitude of the first point in degrees. Range: `[-180°, 180°]`  .
- `lat1Deg`   : Latitude of the first point in degrees. Range: `[-90°, 90°]`  .
- `lon2Deg`   : Longitude of the second point in degrees. Range: `[-180°, 180°]`  .
- `lat2Deg`   : Latitude of the second point in degrees. Range: `[-90°, 90°]`  .

Positive values correspond to North latitude and East longitude, and negative values correspond to South latitude and West longitude.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The distance between two points on the Earth’s surface, in meters.

Generates an exception when the input parameter values fall outside of the range.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT geoDistance(38.8976, -77.0366, 39.9496, -75.1503) AS geoDistance ┌─geoDistance─┐
│   212458.73 │
└─────────────┘
## greatCircleAngle [¶](https://www.tinybird.co/docs/about:blank#greatcircleangle)

Calculates the central angle between two points on the Earth’s surface using the great-circle formula.

greatCircleAngle(lon1Deg, lat1Deg, lon2Deg, lat2Deg)
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

- `lon1Deg`   : Longitude of the first point in degrees.
- `lat1Deg`   : Latitude of the first point in degrees.
- `lon2Deg`   : Longitude of the second point in degrees.
- `lat2Deg`   : Latitude of the second point in degrees.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The central angle between two points in degrees.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT greatCircleAngle(0, 0, 45, 0) AS arc ┌─arc─┐
│  45 │
└─────┘
## pointInEllipses [¶](https://www.tinybird.co/docs/about:blank#pointinellipses)

Checks whether the point belongs to at least one of the ellipses. Coordinates are geometric in the Cartesian coordinate system.

pointInEllipses(x, y, x₀, y₀, a₀, b₀,...,xₙ, yₙ, aₙ, bₙ)
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

- `x, y`   : Coordinates of a point on the plane.
- `xᵢ, yᵢ`   : Coordinates of the center of the `i`   -th ellipsis.
- `aᵢ, bᵢ`   : Axes of the `i`   -th ellipsis in units of x, y coordinates.

The input parameters must be `2+4⋅n` , where `n` is the number of ellipses.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

`1` if the point is inside at least one of the ellipses; `0` if it's not.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT pointInEllipses(10., 10., 10., 9.1, 1., 0.9999) ┌─pointInEllipses(10., 10., 10., 9.1, 1., 0.9999)─┐
│                                               1 │
└─────────────────────────────────────────────────┘
## pointInPolygon [¶](https://www.tinybird.co/docs/about:blank#pointinpolygon)

Checks whether the point belongs to the polygon on the plane.

pointInPolygon((x, y), [(a, b), (c, d) ...], ...)
### Input values [¶](https://www.tinybird.co/docs/about:blank#input-values)

- `(x, y)`   : Coordinates of a point on the plane. Data type: Tuple: A tuple of two numbers.
- `[(a, b), (c, d) ...]`   : Polygon vertices. Data type: Array. Each vertex is represented by a pair of coordinates `(a, b)`   . Vertices should be specified in a clockwise or counterclockwise order. The minimum number of vertices is 3. The polygon must be constant.
- The function also supports polygons with holes (cut out sections). In this case, add polygons that define the cut out sections using additional arguments of the function. The function doesn't support non-simply-connected polygons.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

`1` if the point is inside the polygon, `0` if it's not. If the point is on the polygon boundary, the function may return either 0 or 1.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT pointInPolygon((3., 3.), [(6, 0), (8, 4), (5, 8), (0, 2)]) AS res ┌─res─┐
│   1 │
└─────┘
## Geohash [¶](https://www.tinybird.co/docs/about:blank#geohash)

Geohash is the geocode system, which subdivides Earth’s surface into buckets of grid shape and encodes each cell into a short string of letters and digits. It is a hierarchical data structure, so the longer the geohash string is, the more precise the geographic location will be.

If you need to manually convert geographic coordinates to geohash strings, you can use geohash.org.

## geohashEncode [¶](https://www.tinybird.co/docs/about:blank#geohashencode)

Encodes latitude and longitude as a geohash-string.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

geohashEncode(longitude, latitude, [precision])
### Input values [¶](https://www.tinybird.co/docs/about:blank#input-values)

- `longitude`   : Longitude part of the coordinate you want to encode. Floating in range `[-180°, 180°]`   . Float.
- `latitude`   : Latitude part of the coordinate you want to encode. Floating in range `[-90°, 90°]`   . Float.
- `precision`   (optional): Length of the resulting encoded string. Defaults to `12`   . Integer in the range `[1, 12]`   . Int8.

- All coordinate parameters must be of the same type: either `Float32`   or `Float64`  .
- For the `precision`   parameter, any value less than `1`   or greater than `12`   is silently converted to `12`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Alphanumeric string of the encoded coordinate (modified version of the base32-encoding alphabet is used). String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT geohashEncode(-5.60302734375, 42.593994140625, 0) AS res Result:

┌─res──────────┐
│ ezs42d000000 │
└──────────────┘
## geohashDecode [¶](https://www.tinybird.co/docs/about:blank#geohashdecode)

Decodes any geohash-encoded string into longitude and latitude.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

geohashDecode(hash_str)
### Input values [¶](https://www.tinybird.co/docs/about:blank#input-values)

- `hash_str`   : Geohash-encoded string.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Tuple `(longitude, latitude)`   of `Float64`   values of longitude and latitude. Tuple(Float64)

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT geohashDecode('ezs42') AS res ┌─res─────────────────────────────┐
│ (-5.60302734375,42.60498046875) │
└─────────────────────────────────┘
## geohashesInBox [¶](https://www.tinybird.co/docs/about:blank#geohashesinbox)

Returns an array of geohash-encoded strings of given precision that fall inside and intersect boundaries of given box, basically a 2D grid flattened into array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

geohashesInBox(longitude_min, latitude_min, longitude_max, latitude_max, precision)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `longitude_min`   : Minimum longitude. Range: `[-180°, 180°]`   . Float.
- `latitude_min`   : Minimum latitude. Range: `[-90°, 90°]`   . Float.
- `longitude_max`   : Maximum longitude. Range: `[-180°, 180°]`   . Float.
- `latitude_max`   : Maximum latitude. Range: `[-90°, 90°]`   . Float.
- `precision`   : Geohash precision. Range: `[1, 12]`   . UInt8.

All coordinate parameters must be of the same type: either `Float32` or `Float64`.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of precision-long strings of geohash-boxes covering provided area, you should not rely on order of items. Array(String).
- `[]`   - Empty array if minimum latitude and longitude values aren’t less than corresponding maximum values.

Function throws an exception if resulting array is over 10’000’000 items long.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT geohashesInBox(24.48, 40.56, 24.785, 40.81, 4) AS thasos Result:

┌─thasos──────────────────────────────────────┐
│ ['sx1q','sx1r','sx32','sx1w','sx1x','sx38'] │
└─────────────────────────────────────────────┘
## H3 Index [¶](https://www.tinybird.co/docs/about:blank#h3-index)

H3 is a geographical indexing system where Earth’s surface divided into a grid of even hexagonal cells. This system is hierarchical, i. e. each hexagon on the top level ("parent") can be split into seven even but smaller ones ("children"), and so on.

The level of the hierarchy is called `resolution` and can receive a value from `0` till `15` , where `0` is the `base` level with the largest and coarsest cells.

A latitude and longitude pair can be transformed to a 64-bit H3 index, identifying a grid cell.

The H3 index is used primarily for bucketing locations and other geospatial manipulations.

The full description of the H3 system is available at the Uber Engineering site.

## h3IsValid [¶](https://www.tinybird.co/docs/about:blank#h3isvalid)

Verifies whether the number is a valid H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3IsValid(h3index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `h3index`   : Hexagon index number. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- 1: The number is a valid H3 index. UInt8.
- 0: The number isn't a valid H3 index. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3IsValid(630814730351855103) AS h3IsValid Result:

┌─h3IsValid─┐
│         1 │
└───────────┘
## h3GetResolution [¶](https://www.tinybird.co/docs/about:blank#h3getresolution)

Defines the resolution of the given H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetResolution(h3index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `h3index`   : Hexagon index number. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Index resolution. Range: `[0, 15]`   . UInt8.
- If the index isn't valid, the function returns a random value. Use h3IsValid to verify the index. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetResolution(639821929606596015) AS resolution Result:

┌─resolution─┐
│         14 │
└────────────┘
## h3EdgeAngle [¶](https://www.tinybird.co/docs/about:blank#h3edgeangle)

Calculates the average length of the H3 hexagon edge in grades.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3EdgeAngle(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. UInt8. Range: `[0, 15]`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The average length of the H3 hexagon edge in grades. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3EdgeAngle(10) AS edgeAngle Result:

┌───────h3EdgeAngle(10)─┐
│ 0.0005927224846720883 │
└───────────────────────┘
## h3EdgeLengthM [¶](https://www.tinybird.co/docs/about:blank#h3edgelengthm)

Calculates the average length of the H3 hexagon edge in meters.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3EdgeLengthM(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. UInt8. Range: `[0, 15]`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The average length of the H3 hexagon edge in meters. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3EdgeLengthM(15) AS edgeLengthM Result:

┌─edgeLengthM─┐
│ 0.509713273 │
└─────────────┘
## h3EdgeLengthKm [¶](https://www.tinybird.co/docs/about:blank#h3edgelengthkm)

Calculates the average length of the H3 hexagon edge in kilometers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3EdgeLengthKm(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. UInt8. Range: `[0, 15]`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The average length of the H3 hexagon edge in kilometers. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3EdgeLengthKm(15) AS edgeLengthKm Result:

┌─edgeLengthKm─┐
│  0.000509713 │
└──────────────┘
## geoToH3 [¶](https://www.tinybird.co/docs/about:blank#geotoh3)

Returns H3 point index `(lon, lat)` with specified resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

geoToH3(lon, lat, resolution)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lon`   : Longitude. Float64.
- `lat`   : Latitude. Float64.
- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Hexagon index number. UInt64.
- 0 in case of error. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT geoToH3(37.79506683, 55.71290588, 15) AS h3Index Result:

┌────────────h3Index─┐
│ 644325524701193974 │
└────────────────────┘
## h3ToGeo [¶](https://www.tinybird.co/docs/about:blank#h3togeo)

Returns the centroid longitude and latitude corresponding to the provided H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ToGeo(h3Index)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `h3Index`   : H3 Index. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- A tuple consisting of two values: `tuple(lon,lat)`  . `lon`   : Longitude. Float64. `lat`   : Latitude. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToGeo(644325524701193974) AS coordinates Result:

┌─coordinates───────────────────────────┐
│ (37.79506616830252,55.71290243145668) │
└───────────────────────────────────────┘
## h3ToGeoBoundary [¶](https://www.tinybird.co/docs/about:blank#h3togeoboundary)

Returns array of pairs `(lon, lat)` , which corresponds to the boundary of the provided H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ToGeoBoundary(h3Index)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `h3Index`   : H3 Index. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of pairs '(lon, lat)'. Array(Float64, Float64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToGeoBoundary(644325524701193974) AS coordinates Result:

┌─h3ToGeoBoundary(599686042433355775)────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [(37.2713558667319,-121.91508032705622),(37.353926450852256,-121.8622232890249),(37.42834118609435,-121.92354999630156),(37.42012867767779,-122.03773496427027),(37.33755608435299,-122.090428929044),(37.26319797461824,-122.02910130919001)] │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## h3kRing [¶](https://www.tinybird.co/docs/about:blank#h3kring)

Lists all the H3 hexagons in the raduis of `k` from the given hexagon in random order.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3kRing(h3index, k)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `h3index`   : Hexagon index number. UInt64.
- `k`   : Radius. integer

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of H3 indexes. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayJoin(h3kRing(644325529233966508, 1)) AS h3index Result:

┌────────────h3index─┐
│ 644325529233966508 │
│ 644325529233966497 │
│ 644325529233966510 │
│ 644325529233966504 │
│ 644325529233966509 │
│ 644325529233966355 │
│ 644325529233966354 │
└────────────────────┘
## h3GetBaseCell [¶](https://www.tinybird.co/docs/about:blank#h3getbasecell)

Returns the base cell number of the H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetBaseCell(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hexagon base cell number. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetBaseCell(612916788725809151) AS basecell Result:

┌─basecell─┐
│       12 │
└──────────┘
## h3HexAreaM2 [¶](https://www.tinybird.co/docs/about:blank#h3hexaream2)

Returns average hexagon area in square meters at the given resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3HexAreaM2(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Area in square meters. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3HexAreaM2(13) AS area Result:

┌─area─┐
│ 43.9 │
└──────┘
## h3HexAreaKm2 [¶](https://www.tinybird.co/docs/about:blank#h3hexareakm2)

Returns average hexagon area in square kilometers at the given resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3HexAreaKm2(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Area in square kilometers. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3HexAreaKm2(13) AS area Result:

┌──────area─┐
│ 0.0000439 │
└───────────┘
## h3IndexesAreNeighbors [¶](https://www.tinybird.co/docs/about:blank#h3indexesareneighbors)

Returns whether or not the provided H3 indexes are neighbors.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3IndexesAreNeighbors(index1, index2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `index1`   : Hexagon index number. UInt64.
- `index2`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : Indexes are neighbours. UInt8.
- `0`   : Indexes aren't neighbours. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3IndexesAreNeighbors(617420388351344639, 617420388352655359) AS n Result:

┌─n─┐
│ 1 │
└───┘
## h3ToChildren [¶](https://www.tinybird.co/docs/about:blank#h3tochildren)

Returns an array of child indexes for the given H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ToChildren(index, resolution)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `index`   : Hexagon index number. UInt64.
- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of the child H3-indexes. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToChildren(599405990164561919, 6) AS children Result:

┌─children───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [603909588852408319,603909588986626047,603909589120843775,603909589255061503,603909589389279231,603909589523496959,603909589657714687] │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## h3ToParent [¶](https://www.tinybird.co/docs/about:blank#h3toparent)

Returns the parent (coarser) index containing the given H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ToParent(index, resolution)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `index`   : Hexagon index number. UInt64.
- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Parent H3 index. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToParent(599405990164561919, 3) AS parent Result:

┌─────────────parent─┐
│ 590398848891879423 │
└────────────────────┘
## h3ToString [¶](https://www.tinybird.co/docs/about:blank#h3tostring)

Converts the `H3Index` representation of the index to the string representation.

h3ToString(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- String representation of the H3 index. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToString(617420388352917503) AS h3_string Result:

┌─h3_string───────┐
│ 89184926cdbffff │
└─────────────────┘
## stringToH3 [¶](https://www.tinybird.co/docs/about:blank#stringtoh3)

Converts the string representation to the `H3Index` (UInt64) representation.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stringToH3(index_str)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index_str`   : String representation of the H3 index. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Hexagon index number. Returns 0 on error. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT stringToH3('89184926cc3ffff') AS index Result:

┌──────────────index─┐
│ 617420388351344639 │
└────────────────────┘
## h3GetResolution [¶](https://www.tinybird.co/docs/about:blank#h3getresolution)

Returns the resolution of the H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetResolution(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Index resolution. Range: `[0, 15]`   . UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetResolution(617420388352917503) AS res Result:

┌─res─┐
│   9 │
└─────┘
## h3IsResClassIII [¶](https://www.tinybird.co/docs/about:blank#h3isresclassiii)

Returns whether H3 index has a resolution with Class III orientation.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3IsResClassIII(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : Index has a resolution with Class III orientation. UInt8.
- `0`   : Index doesn't have a resolution with Class III orientation. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3IsResClassIII(617420388352917503) AS res Result:

┌─res─┐
│   1 │
└─────┘
## h3IsPentagon [¶](https://www.tinybird.co/docs/about:blank#h3ispentagon)

Returns whether this H3 index represents a pentagonal cell.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3IsPentagon(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : Index represents a pentagonal cell. UInt8.
- `0`   : Index doesn't represent a pentagonal cell. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3IsPentagon(644721767722457330) AS pentagon Result:

┌─pentagon─┐
│        0 │
└──────────┘
## h3GetFaces [¶](https://www.tinybird.co/docs/about:blank#h3getfaces)

Returns icosahedron faces intersected by a given H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetFaces(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array containing icosahedron faces intersected by a given H3 index. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetFaces(599686042433355775) AS faces Result:

┌─faces─┐
│ [7]   │
└───────┘
## h3CellAreaM2 [¶](https://www.tinybird.co/docs/about:blank#h3cellaream2)

Returns the exact area of a specific cell in square meters corresponding to the given input H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3CellAreaM2(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Cell area in square meters. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3CellAreaM2(579205133326352383) AS area Result:

┌───────────────area─┐
│ 4106166334463.9233 │
└────────────────────┘
## h3CellAreaRads2 [¶](https://www.tinybird.co/docs/about:blank#h3cellarearads2)

Returns the exact area of a specific cell in square radians corresponding to the given input H3 index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3CellAreaRads2(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Cell area in square radians. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3CellAreaRads2(579205133326352383) AS area Result:

┌────────────────area─┐
│ 0.10116268528089567 │
└─────────────────────┘
## h3ToCenterChild [¶](https://www.tinybird.co/docs/about:blank#h3tocenterchild)

Returns the center child (finer) H3 index contained by given H3 at the given resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ToCenterChild(index, resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.
- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- H3 index of the center child contained by given H3 at the given resolution. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ToCenterChild(577023702256844799,1) AS centerToChild Result:

┌──────centerToChild─┐
│ 581496515558637567 │
└────────────────────┘
## h3ExactEdgeLengthM [¶](https://www.tinybird.co/docs/about:blank#h3exactedgelengthm)

Returns the exact edge length of the unidirectional edge represented by the input h3 index in meters.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ExactEdgeLengthM(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Exact edge length in meters. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ExactEdgeLengthM(1310277011704381439) AS exactEdgeLengthM Result:

┌───exactEdgeLengthM─┐
│ 195449.63163407316 │
└────────────────────┘
## h3ExactEdgeLengthKm [¶](https://www.tinybird.co/docs/about:blank#h3exactedgelengthkm)

Returns the exact edge length of the unidirectional edge represented by the input h3 index in kilometers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ExactEdgeLengthKm(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Exact edge length in kilometers. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ExactEdgeLengthKm(1310277011704381439) AS exactEdgeLengthKm Result:

┌──exactEdgeLengthKm─┐
│ 195.44963163407317 │
└────────────────────┘
## h3ExactEdgeLengthRads [¶](https://www.tinybird.co/docs/about:blank#h3exactedgelengthrads)

Returns the exact edge length of the unidirectional edge represented by the input h3 index in radians.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3ExactEdgeLengthRads(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Exact edge length in radians. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3ExactEdgeLengthRads(1310277011704381439) AS exactEdgeLengthRads Result:

┌──exactEdgeLengthRads─┐
│ 0.030677980118976447 │
└──────────────────────┘
## h3NumHexagons [¶](https://www.tinybird.co/docs/about:blank#h3numhexagons)

Returns the number of unique H3 indices at the given resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3NumHexagons(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of H3 indices. Int64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3NumHexagons(3) AS numHexagons Result:

┌─numHexagons─┐
│       41162 │
└─────────────┘
## h3PointDistM [¶](https://www.tinybird.co/docs/about:blank#h3pointdistm)

Returns the "great circle" or "haversine" distance between pairs of GeoCoord points (latitude/longitude) pairs in meters.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3PointDistM(lat1, lon1, lat2, lon2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lat1`  , `lon1`   : Latitude and Longitude of point1 in degrees. Float64.
- `lat2`  , `lon2`   : Latitude and Longitude of point2 in degrees. Float64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Haversine or great circle distance in meters.Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select h3PointDistM(-10.0 ,0.0, 10.0, 0.0) as h3PointDistM Result:

┌──────h3PointDistM─┐
│ 2223901.039504589 │
└───────────────────┘
## h3PointDistKm [¶](https://www.tinybird.co/docs/about:blank#h3pointdistkm)

Returns the "great circle" or "haversine" distance between pairs of GeoCoord points (latitude/longitude) pairs in kilometers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3PointDistKm(lat1, lon1, lat2, lon2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lat1`  , `lon1`   : Latitude and Longitude of point1 in degrees. Float64.
- `lat2`  , `lon2`   : Latitude and Longitude of point2 in degrees. Float64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Haversine or great circle distance in kilometers. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select h3PointDistKm(-10.0 ,0.0, 10.0, 0.0) as h3PointDistKm Result:

┌─────h3PointDistKm─┐
│ 2223.901039504589 │
└───────────────────┘
## h3PointDistRads [¶](https://www.tinybird.co/docs/about:blank#h3pointdistrads)

Returns the "great circle" or "haversine" distance between pairs of GeoCoord points (latitude/longitude) pairs in radians.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3PointDistRads(lat1, lon1, lat2, lon2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lat1`  , `lon1`   : Latitude and Longitude of point1 in degrees. Float64.
- `lat2`  , `lon2`   : Latitude and Longitude of point2 in degrees. Float64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Haversine or great circle distance in radians. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select h3PointDistRads(-10.0 ,0.0, 10.0, 0.0) as h3PointDistRads Result:

┌────h3PointDistRads─┐
│ 0.3490658503988659 │
└────────────────────┘
## h3GetRes0Indexes [¶](https://www.tinybird.co/docs/about:blank#h3getres0indexes)

Returns an array of all the resolution 0 H3 indexes.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetRes0Indexes()
### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of all the resolution 0 H3 indexes. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select h3GetRes0Indexes as indexes Result:

┌─indexes─────────────────────────────────────┐
│ [576495936675512319,576531121047601151,....]│
└─────────────────────────────────────────────┘
## h3GetPentagonIndexes [¶](https://www.tinybird.co/docs/about:blank#h3getpentagonindexes)

Returns all the pentagon H3 indexes at the specified resolution.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetPentagonIndexes(resolution)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `resolution`   : Index resolution. Range: `[0, 15]`   . UInt8.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of all pentagon H3 indexes. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetPentagonIndexes(3) AS indexes Result:

┌─indexes────────────────────────────────────────────────────────┐
│ [590112357393367039,590464201114255359,590816044835143679,...] │
└────────────────────────────────────────────────────────────────┘
## h3Line [¶](https://www.tinybird.co/docs/about:blank#h3line)

Returns the line of indices between the two indices that are provided.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3Line(start,end)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `start`   : Hexagon index number that represents a starting point. UInt64.
- `end`   : Hexagon index number that represents an ending point. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Array of h3 indexes representing the line of indices between the two provided indices. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3Line(590080540275638271,590103561300344831) as indexes Result:

┌─indexes────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [590080540275638271,590080471556161535,590080883873021951,590106516237844479,590104385934065663,590103630019821567,590103561300344831] │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## h3Distance [¶](https://www.tinybird.co/docs/about:blank#h3distance)

Returns the distance in grid cells between the two indices that are provided.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3Distance(start,end)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `start`   : Hexagon index number that represents a starting point. UInt64.
- `end`   : Hexagon index number that represents an ending point. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of grid cells. Int64.

Returns a negative number if finding the distance fails.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3Distance(590080540275638271,590103561300344831) as distance Result:

┌─distance─┐
│        7 │
└──────────┘
## h3HexRing [¶](https://www.tinybird.co/docs/about:blank#h3hexring)

Returns the indexes of the hexagonal ring centered at the provided origin h3Index and length k.

Returns 0 if no pentagonal distortion was encountered.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3HexRing(index, k)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number that represents the origin. UInt64.
- `k`   : Distance. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of H3 indexes. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3HexRing(590080540275638271, toUInt16(1)) AS hexRing Result:

┌─hexRing─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [590080815153545215,590080471556161535,590080677714591743,590077585338138623,590077447899185151,590079509483487231] │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## h3GetUnidirectionalEdge [¶](https://www.tinybird.co/docs/about:blank#h3getunidirectionaledge)

Returns a unidirectional edge H3 index based on the provided origin and destination and returns 0 on error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetUnidirectionalEdge(originIndex, destinationIndex)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `originIndex`   : Origin Hexagon index number. UInt64.
- `destinationIndex`   : Destination Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Unidirectional Edge Hexagon Index number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetUnidirectionalEdge(599686042433355775, 599686043507097599) as edge Result:

┌────────────────edge─┐
│ 1248204388774707199 │
└─────────────────────┘
## h3UnidirectionalEdgeIsValid [¶](https://www.tinybird.co/docs/about:blank#h3unidirectionaledgeisvalid)

Determines if the provided H3Index is a valid unidirectional edge index. Returns 1 if it's a unidirectional edge and 0 otherwise.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3UnidirectionalEdgeisValid(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- 1: The H3 index is a valid unidirectional edge. UInt8.
- 0: The H3 index isn't a valid unidirectional edge. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3UnidirectionalEdgeIsValid(1248204388774707199) as validOrNot Result:

┌─validOrNot─┐
│          1 │
└────────────┘
## h3GetOriginIndexFromUnidirectionalEdge [¶](https://www.tinybird.co/docs/about:blank#h3getoriginindexfromunidirectionaledge)

Returns the origin hexagon index from the unidirectional edge H3Index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetOriginIndexFromUnidirectionalEdge(edge)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `edge`   : Hexagon index number that represents a unidirectional edge. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Origin Hexagon Index number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetOriginIndexFromUnidirectionalEdge(1248204388774707197) as origin Result:

┌─────────────origin─┐
│ 599686042433355773 │
└────────────────────┘
## h3GetDestinationIndexFromUnidirectionalEdge [¶](https://www.tinybird.co/docs/about:blank#h3getdestinationindexfromunidirectionaledge)

Returns the destination hexagon index from the unidirectional edge H3Index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetDestinationIndexFromUnidirectionalEdge(edge)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `edge`   : Hexagon index number that represents a unidirectional edge. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Destination Hexagon Index number. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetDestinationIndexFromUnidirectionalEdge(1248204388774707197) as destination Result:

┌────────destination─┐
│ 599686043507097597 │
└────────────────────┘
## h3GetIndexesFromUnidirectionalEdge [¶](https://www.tinybird.co/docs/about:blank#h3getindexesfromunidirectionaledge)

Returns the origin and destination hexagon indexes from the given unidirectional edge H3Index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetIndexesFromUnidirectionalEdge(edge)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `edge`   : Hexagon index number that represents a unidirectional edge. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

A tuple consisting of two values `tuple(origin,destination)`:

- `origin`   : Origin Hexagon index number. UInt64.
- `destination`   : Destination Hexagon index number. UInt64.

Returns `(0,0)` if the provided input isn't valid.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetIndexesFromUnidirectionalEdge(1248204388774707199) as indexes Result:

┌─indexes─────────────────────────────────┐
│ (599686042433355775,599686043507097599) │
└─────────────────────────────────────────┘
## h3GetUnidirectionalEdgesFromHexagon [¶](https://www.tinybird.co/docs/about:blank#h3getunidirectionaledgesfromhexagon)

Provides all of the unidirectional edges from the provided H3Index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetUnidirectionalEdgesFromHexagon(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number that represents a unidirectional edge. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Array of h3 indexes representing each unidirectional edge. Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetUnidirectionalEdgesFromHexagon(1248204388774707199) as edges Result:

┌─edges─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ [1248204388774707199,1320261982812635135,1392319576850563071,1464377170888491007,1536434764926418943,1608492358964346879] │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## h3GetUnidirectionalEdgeBoundary [¶](https://www.tinybird.co/docs/about:blank#h3getunidirectionaledgeboundary)

Returns the coordinates defining the unidirectional edge.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

h3GetUnidirectionalEdgeBoundary(index)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `index`   : Hexagon index number that represents a unidirectional edge. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of pairs '(lon, lat)'. Array(Float64, Float64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT h3GetUnidirectionalEdgeBoundary(1248204388774707199) as boundary Result:

┌─boundary────────────────────────────────────────────────────────────────────────┐
│ [(37.42012867767779,-122.03773496427027),(37.33755608435299,-122.090428929044)] │
└─────────────────────────────────────────────────────────────────────────────────┘
## WKT [¶](https://www.tinybird.co/docs/about:blank#wkt)

Returns a WKT (Well Known Text) geometric object from various Geo data types. Supported WKT objects are:

- POINT
- POLYGON
- MULTIPOLYGON
- LINESTRING
- MULTILINESTRING

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

WKT(geo_data)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

`geo_data` can be one of the following Geo data types or their underlying primitive types:

- Point
- Ring
- Polygon
- MultiPolygon
- LineString
- MultiLineString

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- WKT geometric object `POINT`   is returned for a Point.
- WKT geometric object `POLYGON`   is returned for a Polygon
- WKT geometric object `MULTIPOLYGON`   is returned for a MultiPolygon.
- WKT geometric object `LINESTRING`   is returned for a LineString.
- WKT geometric object `MULTILINESTRING`   is returned for a MultiLineString.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

POINT from tuple:

SELECT wkt((0., 0.)) POINT(0 0) POLYGON from an array of tuples or an array of tuple arrays:

SELECT wkt([(0., 0.), (10., 0.), (10., 10.), (0., 10.)]) POLYGON((0 0,10 0,10 10,0 10)) MULTIPOLYGON from an array of multi-dimensional tuple arrays:

SELECT wkt([[[(0., 0.), (10., 0.), (10., 10.), (0., 10.)], [(4., 4.), (5., 4.), (5., 5.), (4., 5.)]], [[(-10., -10.), (-10., -9.), (-9., 10.)]]]) MULTIPOLYGON(((0 0,10 0,10 10,0 10,0 0),(4 4,5 4,5 5,4 5,4 4)),((-10 -10,-10 -9,-9 10,-10 -10)))
## readWKTMultiPolygon [¶](https://www.tinybird.co/docs/about:blank#readwktmultipolygon)

Converts a WKT (Well Known Text) MultiPolygon into a MultiPolygon type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toTypeName(readWKTMultiPolygon('MULTIPOLYGON(((2 0,10 0,10 10,0 10,2 0),(4 4,5 4,5 5,4 5,4 4)),((-10 -10,-10 -9,-9 10,-10 -10)))')) AS type,
    readWKTMultiPolygon('MULTIPOLYGON(((2 0,10 0,10 10,0 10,2 0),(4 4,5 4,5 5,4 5,4 4)),((-10 -10,-10 -9,-9 10,-10 -10)))') AS output FORMAT Markdown| type | output |
| --- | --- |
| MultiPolygon | [[[(2,0),(10,0),(10,10),(0,10),(2,0)],[(4,4),(5,4),(5,5),(4,5),(4,4)]],[[(-10,-10),(-10,-9),(-9,10),(-10,-10)]]] |

### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

String starting with `MULTIPOLYGON`

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## readWKTPolygon [¶](https://www.tinybird.co/docs/about:blank#readwktpolygon)

Converts a WKT (Well Known Text) MultiPolygon into a Polygon type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toTypeName(readWKTPolygon('POLYGON((2 0,10 0,10 10,0 10,2 0))')) AS type,
    readWKTPolygon('POLYGON((2 0,10 0,10 10,0 10,2 0))') AS output
FORMAT Markdown| type | output |
| --- | --- |
| Polygon | [[(2,0),(10,0),(10,10),(0,10),(2,0)]] |

### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

String starting with `POLYGON`

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Polygon

## readWKTPoint [¶](https://www.tinybird.co/docs/about:blank#readwktpoint)

The `readWKTPoint` function in Tinybird parses a Well-Known Text (WKT) representation of a Point geometry and returns a point in the internal format.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

readWKTPoint(wkt_string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `wkt_string`   : The input WKT string representing a Point geometry.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The function returns an internal representation of the Point geometry.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT readWKTPoint('POINT (1.2 3.4)') (1.2,3.4)
## readWKTLineString [¶](https://www.tinybird.co/docs/about:blank#readwktlinestring)

Parses a Well-Known Text (WKT) representation of a LineString geometry and returns it in the internal format.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

readWKTLineString(wkt_string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `wkt_string`   : The input WKT string representing a LineString geometry.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The function returns an internal representation of the linestring geometry.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT readWKTLineString('LINESTRING (1 1, 2 2, 3 3, 1 1)') [(1,1),(2,2),(3,3),(1,1)]
## readWKTMultiLineString [¶](https://www.tinybird.co/docs/about:blank#readwktmultilinestring)

Parses a Well-Known Text (WKT) representation of a MultiLineString geometry and returns it in the internal format.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

readWKTMultiLineString(wkt_string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `wkt_string`   : The input WKT string representing a MultiLineString geometry.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The function returns an internal representation of the multilinestring geometry.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT readWKTMultiLineString('MULTILINESTRING ((1 1, 2 2, 3 3), (4 4, 5 5, 6 6))') [[(1,1),(2,2),(3,3)],[(4,4),(5,5),(6,6)]]
## readWKTRing [¶](https://www.tinybird.co/docs/about:blank#readwktring)

Parses a Well-Known Text (WKT) representation of a Polygon geometry and returns a ring (closed linestring) in the internal format.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

readWKTRing(wkt_string)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `wkt_string`   : The input WKT string representing a Polygon geometry.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The function returns an internal representation of the ring (closed linestring) geometry.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT readWKTRing('POLYGON ((1 1, 2 2, 3 3, 1 1))') [(1,1),(2,2),(3,3),(1,1)]
## polygonsWithinSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonswithinspherical)

Returns true or false depending on whether or not one polygon lies completely inside another polygon. Reference https://www.boost.org/doc/libs/1_62_0/libs/geometry/doc/html/geometry/reference/algorithms/within/within_2.html

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select polygonsWithinSpherical([[[(4.3613577, 50.8651821), (4.349556, 50.8535879), (4.3602419, 50.8435626), (4.3830299, 50.8428851), (4.3904543, 50.8564867), (4.3613148, 50.8651279)]]], [[[(4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)]]]) 0
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

UInt8, 0 for false, 1 for true

## polygonsDistanceSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonsdistancespherical)

Calculates the minimal distance between two points where one point belongs to the first polygon and the second to another polygon. Spherical means that coordinates are interpreted as coordinates on a pure and ideal sphere, which isn't true for the Earth. Using this type of coordinate system speeds up execution, but of course isn't precise.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonsDistanceSpherical([[[(0, 0), (0, 0.1), (0.1, 0.1), (0.1, 0)]]], [[[(10., 10.), (10., 40.), (40., 40.), (40., 10.), (10., 10.)]]]) 0.24372872211133834
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Two polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Float64

## polygonsDistanceCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonsdistancecartesian)

Calculates distance between two polygons

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonsDistanceCartesian([[[(0, 0), (0, 0.1), (0.1, 0.1), (0.1, 0)]]], [[[(10., 10.), (10., 40.), (40., 40.), (40., 10.), (10., 10.)]]]) 14.000714267493642
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Two polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Float64

## polygonsEqualsCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonsequalscartesian)

Returns true if two polygons are equal

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonsEqualsCartesian([[[(1., 1.), (1., 4.), (4., 4.), (4., 1.)]]], [[[(1., 1.), (1., 4.), (4., 4.), (4., 1.), (1., 1.)]]]) 1
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Two polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

UInt8, 0 for false, 1 for true

## polygonsSymDifferenceSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonssymdifferencespherical)

Calculates the spatial set theoretic symmetric difference (XOR) between two polygons

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(arraySort(polygonsSymDifferenceSpherical([[(50., 50.), (50., -50.), (-50., -50.), (-50., 50.), (50., 50.)], [(10., 10.), (10., 40.), (40., 40.), (40., 10.), (10., 10.)], [(-10., -10.), (-10., -40.), (-40., -40.), (-40., -10.), (-10., -10.)]], [[(-20., -20.), (-20., 20.), (20., 20.), (20., -20.), (-20., -20.)]]))) MULTIPOLYGON(((-20 -10.3067,-10 -10,-10 -20.8791,-20 -20,-20 -10.3067)),((10 20.8791,20 20,20 10.3067,10 10,10 20.8791)),((50 50,50 -50,-50 -50,-50 50,50 50),(20 10.3067,40 10,40 40,10 40,10 20.8791,-20 20,-20 -10.3067,-40 -10,-40 -40,-10 -40,-10 -20.8791,20 -20,20 10.3067)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## polygonsSymDifferenceCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonssymdifferencecartesian)

The same as `polygonsSymDifferenceSpherical` , but the coordinates are in the Cartesian coordinate system; which is more close to the model of the real Earth.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(polygonsSymDifferenceCartesian([[[(0, 0), (0, 3), (1, 2.9), (2, 2.6), (2.6, 2), (2.9, 1), (3, 0), (0, 0)]]], [[[(1., 1.), (1., 4.), (4., 4.), (4., 1.), (1., 1.)]]])) MULTIPOLYGON(((1 2.9,1 1,2.9 1,3 0,0 0,0 3,1 2.9)),((1 2.9,1 4,4 4,4 1,2.9 1,2.6 2,2 2.6,1 2.9)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## polygonsIntersectionSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonsintersectionspherical)

Calculates the intersection (AND) between polygons, coordinates are spherical.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(arrayMap(a -> arrayMap(b -> arrayMap(c -> (round(c.1, 6), round(c.2, 6)), b), a), polygonsIntersectionSpherical([[[(4.3613577, 50.8651821), (4.349556, 50.8535879), (4.3602419, 50.8435626), (4.3830299, 50.8428851), (4.3904543, 50.8564867), (4.3613148, 50.8651279)]]], [[[(4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)]]]))) MULTIPOLYGON(((4.3666 50.8434,4.36024 50.8436,4.34956 50.8536,4.35268 50.8567,4.36794 50.8525,4.3666 50.8434)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## polygonsWithinCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonswithincartesian)

Returns true if the second polygon is within the first polygon.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonsWithinCartesian([[[(2., 2.), (2., 3.), (3., 3.), (3., 2.)]]], [[[(1., 1.), (1., 4.), (4., 4.), (4., 1.), (1., 1.)]]]) 1
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Two polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

UInt8, 0 for false, 1 for true

## polygonConvexHullCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonconvexhullcartesian)

Calculates a convex hull. Reference

Coordinates are in Cartesian coordinate system.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(polygonConvexHullCartesian([[[(0., 0.), (0., 5.), (5., 5.), (5., 0.), (2., 3.)]]])) POLYGON((0 0,0 5,5 5,5 0,0 0))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

MultiPolygon

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Polygon

## polygonAreaSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonareaspherical)

Calculates the surface area of a polygon.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT round(polygonAreaSpherical([[[(4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)]]]), 14) 9.387704e-8
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygon

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Float

## polygonsUnionSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonsunionspherical)

Calculates a union (OR).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(polygonsUnionSpherical([[[(4.3613577, 50.8651821), (4.349556, 50.8535879), (4.3602419, 50.8435626), (4.3830299, 50.8428851), (4.3904543, 50.8564867), (4.3613148, 50.8651279)]]], [[[(4.346693, 50.858306), (4.367945, 50.852455), (4.366227, 50.840809), (4.344961, 50.833264), (4.338074, 50.848677), (4.346693, 50.858306)]]])) MULTIPOLYGON(((4.36661 50.8434,4.36623 50.8408,4.34496 50.8333,4.33807 50.8487,4.34669 50.8583,4.35268 50.8567,4.36136 50.8652,4.36131 50.8651,4.39045 50.8565,4.38303 50.8429,4.36661 50.8434)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## polygonPerimeterSpherical [¶](https://www.tinybird.co/docs/about:blank#polygonperimeterspherical)

Calculates the perimeter of the polygon.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

## polygonsIntersectionCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonsintersectioncartesian)

Calculates the intersection of polygons.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(polygonsIntersectionCartesian([[[(0., 0.), (0., 3.), (1., 2.9), (2., 2.6), (2.6, 2.), (2.9, 1.), (3., 0.), (0., 0.)]]], [[[(1., 1.), (1., 4.), (4., 4.), (4., 1.), (1., 1.)]]])) MULTIPOLYGON(((1 2.9,2 2.6,2.6 2,2.9 1,1 1,1 2.9)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## polygonAreaCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonareacartesian)

Calculates the area of a polygon

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonAreaCartesian([[[(0., 0.), (0., 5.), (5., 5.), (5., 0.)]]]) 25
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygon

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Float64

## polygonPerimeterCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonperimetercartesian)

Calculates the perimeter of a polygon.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT polygonPerimeterCartesian([[[(0., 0.), (0., 5.), (5., 5.), (5., 0.)]]]) 15
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygon

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Float64

## polygonsUnionCartesian [¶](https://www.tinybird.co/docs/about:blank#polygonsunioncartesian)

Calculates the union of polygons.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT wkt(polygonsUnionCartesian([[[(0., 0.), (0., 3.), (1., 2.9), (2., 2.6), (2.6, 2.), (2.9, 1), (3., 0.), (0., 0.)]]], [[[(1., 1.), (1., 4.), (4., 4.), (4., 1.), (1., 1.)]]])) MULTIPOLYGON(((1 2.9,1 4,4 4,4 1,2.9 1,3 0,0 0,0 3,1 2.9)))
### Input parameters [¶](https://www.tinybird.co/docs/about:blank#input-parameters)

Polygons

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

MultiPolygon

## S2Index [¶](https://www.tinybird.co/docs/about:blank#s2index)

S2 is a geographical indexing system where all geographical data is represented on a three-dimensional sphere (similar to a globe).

In the S2 library points are represented as the S2 Index - a specific number which encodes internally a point on the surface of a unit sphere, unlike traditional (latitude, longitude) pairs. To get the S2 point index for a given point specified in the format (latitude, longitude) use the geoToS2 function. Also, you can use the s2ToGeo function for getting geographical coordinates corresponding to the specified S2 point index.

## geoToS2 [¶](https://www.tinybird.co/docs/about:blank#geotos2)

Returns S2 point index corresponding to the provided coordinates `(longitude, latitude)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

geoToS2(lon, lat)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `lon`   : Longitude. Float64.
- `lat`   : Latitude. Float64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- S2 point index. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT geoToS2(37.79506683, 55.71290588) AS s2Index Result:

┌─────────────s2Index─┐
│ 4704772434919038107 │
└─────────────────────┘
## s2ToGeo [¶](https://www.tinybird.co/docs/about:blank#s2togeo)

Returns geo coordinates `(longitude, latitude)` corresponding to the provided S2 point index.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2ToGeo(s2index)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2index`   : S2 Index. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- A tuple consisting of two values:
  - `lon`     . Float64.
  - `lat`     . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2ToGeo(4704772434919038107) AS s2Coodrinates Result:

┌─s2Coodrinates────────────────────────┐
│ (37.79506681471008,55.7129059052841) │
└──────────────────────────────────────┘
## s2GetNeighbors [¶](https://www.tinybird.co/docs/about:blank#s2getneighbors)

Returns S2 neighbor indexes corresponding to the provided S2. Each cell in the S2 system is a quadrilateral bounded by four geodesics. So, each cell has 4 neighbors.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2GetNeighbors(s2index)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2index`   : S2 Index. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array consisting of 4 neighbor indexes: `array[s2index1, s2index3, s2index2, s2index4]`   . Array(UInt64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2GetNeighbors(5074766849661468672) AS s2Neighbors Result:

┌─s2Neighbors───────────────────────────────────────────────────────────────────────┐
│ [5074766987100422144,5074766712222515200,5074767536856236032,5074767261978329088] │
└───────────────────────────────────────────────────────────────────────────────────┘
## s2CellsIntersect [¶](https://www.tinybird.co/docs/about:blank#s2cellsintersect)

Determines if the two provided S2 cells intersect or not.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2CellsIntersect(s2index1, s2index2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `siIndex1`  , `s2index2`   : S2 Index. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : If the cells intersect. UInt8.
- `0`   : If the cells don't intersect. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2CellsIntersect(9926595209846587392, 9926594385212866560) AS intersect Result:

┌─intersect─┐
│         1 │
└───────────┘
## s2CapContains [¶](https://www.tinybird.co/docs/about:blank#s2capcontains)

Determines if a cap contains a S2 point. A cap represents a part of the sphere that has been cut off by a plane. It is defined by a point on a sphere and a radius in degrees.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2CapContains(center, degrees, point)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `center`   : S2 point index corresponding to the cap. UInt64.
- `degrees`   : Radius of the cap in degrees. Float64.
- `point`   : S2 point index. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : If the cap contains the S2 point index. UInt8.
- `0`   : If the cap doesn't contain the S2 point index. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2CapContains(1157339245694594829, 1.0, 1157347770437378819) AS capContains Result:

┌─capContains─┐
│           1 │
└─────────────┘
## s2CapUnion [¶](https://www.tinybird.co/docs/about:blank#s2capunion)

Determines the smallest cap that contains the given two input caps. A cap represents a portion of the sphere that has been cut off by a plane. It is defined by a point on a sphere and a radius in degrees.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2CapUnion(center1, radius1, center2, radius2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `center1`  , `center2`   : S2 point indexes corresponding to the two input caps. UInt64.
- `radius1`  , `radius2`   : Radius of the two input caps in degrees. Float64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `center`   : S2 point index corresponding the center of the smallest cap containing the two input caps. UInt64.
- `radius`   : Radius of the smallest cap containing the two input caps. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2CapUnion(3814912406305146967, 1.0, 1157347770437378819, 1.0) AS capUnion Result:

┌─capUnion───────────────────────────────┐
│ (4534655147792050737,60.2088283994957) │
└────────────────────────────────────────┘
## s2RectAdd [¶](https://www.tinybird.co/docs/about:blank#s2rectadd)

Increases the size of the bounding rectangle to include the given S2 point. In the S2 system, a rectangle is represented by a type of S2Region called a `S2LatLngRect` that represents a rectangle in latitude-longitude space.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2RectAdd(s2pointLow, s2pointHigh, s2Point)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2PointLow`   : Low S2 point index corresponding to the rectangle. UInt64.
- `s2PointHigh`   : High S2 point index corresponding to the rectangle. UInt64.
- `s2Point`   : Target S2 point index that the bound rectangle should be grown to include. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `s2PointLow`   : Low S2 cell id corresponding to the grown rectangle. UInt64.
- `s2PointHigh`   : Height S2 cell id corresponding to the grown rectangle. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2RectAdd(5178914411069187297, 5177056748191934217, 5179056748191934217) AS rectAdd Result:

┌─rectAdd───────────────────────────────────┐
│ (5179062030687166815,5177056748191934217) │
└───────────────────────────────────────────┘
## s2RectContains [¶](https://www.tinybird.co/docs/about:blank#s2rectcontains)

Determines if a given rectangle contains a S2 point. In the S2 system, a rectangle is represented by a type of S2Region called a `S2LatLngRect` that represents a rectangle in latitude-longitude space.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2RectContains(s2PointLow, s2PointHi, s2Point)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2PointLow`   : Low S2 point index corresponding to the rectangle. UInt64.
- `s2PointHigh`   : High S2 point index corresponding to the rectangle. UInt64.
- `s2Point`   : Target S2 point index. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   : If the rectangle contains the given S2 point.
- `0`   : If the rectangle doesn't contain the given S2 point.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2RectContains(5179062030687166815, 5177056748191934217, 5177914411069187297) AS rectContains Result:

┌─rectContains─┐
│            0 │
└──────────────┘
## s2RectUnion [¶](https://www.tinybird.co/docs/about:blank#s2rectunion)

Returns the smallest rectangle containing the union of this rectangle and the given rectangle. In the S2 system, a rectangle is represented by a type of S2Region called a `S2LatLngRect` that represents a rectangle in latitude-longitude space.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2RectUnion(s2Rect1PointLow, s2Rect1PointHi, s2Rect2PointLow, s2Rect2PointHi)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2Rect1PointLow`  , `s2Rect1PointHi`   : Low and High S2 point indexes corresponding to the first rectangle. UInt64.
- `s2Rect2PointLow`  , `s2Rect2PointHi`   : Low and High S2 point indexes corresponding to the second rectangle. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `s2UnionRect2PointLow`   : Low S2 cell id corresponding to the union rectangle. UInt64.
- `s2UnionRect2PointHi`   : High S2 cell id corresponding to the union rectangle. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2RectUnion(5178914411069187297, 5177056748191934217, 5179062030687166815, 5177056748191934217) AS rectUnion Result:

┌─rectUnion─────────────────────────────────┐
│ (5179062030687166815,5177056748191934217) │
└───────────────────────────────────────────┘
## s2RectIntersection [¶](https://www.tinybird.co/docs/about:blank#s2rectintersection)

Returns the smallest rectangle containing the intersection of this rectangle and the given rectangle. In the S2 system, a rectangle is represented by a type of S2Region called a `S2LatLngRect` that represents a rectangle in latitude-longitude space.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

s2RectIntersection(s2Rect1PointLow, s2Rect1PointHi, s2Rect2PointLow, s2Rect2PointHi)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s2Rect1PointLow`  , `s2Rect1PointHi`   : Low and High S2 point indexes corresponding to the first rectangle. UInt64.
- `s2Rect2PointLow`  , `s2Rect2PointHi`   : Low and High S2 point indexes corresponding to the second rectangle. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `s2UnionRect2PointLow`   : Low S2 cell id corresponding to the rectangle containing the intersection of the given rectangles. UInt64.
- `s2UnionRect2PointHi`   : High S2 cell id corresponding to the rectangle containing the intersection of the given rectangles. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT s2RectIntersection(5178914411069187297, 5177056748191934217, 5179062030687166815, 5177056748191934217) AS rectIntersection Result:

┌─rectIntersection──────────────────────────┐
│ (5178914411069187297,5177056748191934217) │
└───────────────────────────────────────────┘
## Svg [¶](https://www.tinybird.co/docs/about:blank#svg)

Returns a string of select SVG element tags from Geo data.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

Svg(geometry,[style]) Aliases: `SVG`, `svg`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `geometry`   : Geo data. Geo.
- `style`   : Optional style name. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The SVG representation of the geometry. String.
  - SVG circle
  - SVG polygon
  - SVG path

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

### Circle [¶](https://www.tinybird.co/docs/about:blank#circle)

Query:

SELECT SVG((0., 0.)) Result:


### Polygon [¶](https://www.tinybird.co/docs/about:blank#polygon)

Query:

SELECT SVG([(0., 0.), (10, 0), (10, 10), (0, 10)]) Result:


### Path [¶](https://www.tinybird.co/docs/about:blank#path)

Query:

SELECT SVG([[(0., 0.), (10, 0), (10, 10), (0, 10)], [(4., 4.), (5, 4), (5, 5), (4, 5)]]) Result:



---

URL: https://www.tinybird.co/docs/sql-reference/functions/functions-for-nulls
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Functions for Nullable Values · Tinybird Docs"
theme-color: "#171612"
description: "Functions for handling nullable values."
---




# Functions for working with nullable values [¶](https://www.tinybird.co/docs/about:blank#functions-for-working-with-nullable-values)

## isNull [¶](https://www.tinybird.co/docs/about:blank#isnull)

Returns whether the argument is NULL.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isNull(x) Alias: `ISNULL`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : A value of non-compound data type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `x`   is `NULL`  .
- `0`   if `x`   isn't `NULL`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Table:

┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │    3 │
└───┴──────┘ Query:

SELECT x FROM t_null WHERE isNull(y)```

Result:

```text
┌─x─┐
│ 1 │
└───┘
## isNullable [¶](https://www.tinybird.co/docs/about:blank#isnullable)

Returns `1` if a column is Nullable (i.e allows `NULL` values), `0` otherwise.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isNullable(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : column.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `x`   allows `NULL`   values. UInt8.
- `0`   if `x`   doesn't allow `NULL`   values. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT isNullable(ordinary_col), isNullable(nullable_col) FROM 
(select c1::UInt32 as ordinary_col, c2::Nullable(UInt32) as nullable_col from values((1,1), (2, 2), (3,3))) Result:

┌───isNullable(ordinary_col)──┬───isNullable(nullable_col)──┐
│                           0 │                           1 │
│                           0 │                           1 │
│                           0 │                           1 │
└─────────────────────────────┴─────────────────────────────┘
## isNotNull [¶](https://www.tinybird.co/docs/about:blank#isnotnull)

Returns whether the argument isn't NULL.

isNotNull(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : A value of non-compound data type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `x`   isn't `NULL`  .
- `0`   if `x`   is `NULL`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Table:

┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │    3 │
└───┴──────┘ Query:

SELECT x FROM t_null WHERE isNotNull(y)```

Result:

```text
┌─x─┐
│ 2 │
└───┘
## isNotDistinctFrom [¶](https://www.tinybird.co/docs/about:blank#isnotdistinctfrom)

Performs null-safe comparison. Used to compare JOIN keys which contain NULL values in the JOIN ON section. This function will consider two `NULL` values as identical and will return `true` , which is distinct from the usual equals behavior where comparing two `NULL` values would return `NULL`.

This function is an internal function used by the implementation of JOIN ON. Please don't use it manually in queries.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isNotDistinctFrom(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first JOIN key.
- `y`   : second JOIN key.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `true`   when `x`   and `y`   are both `NULL`  .
- `false`   otherwise.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

For a complete example see: NULL values in JOIN keys.

## isZeroOrNull [¶](https://www.tinybird.co/docs/about:blank#iszeroornull)

Returns whether the argument is 0 (zero) or NULL.

isZeroOrNull(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : A value of non-compound data type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `1`   if `x`   is 0 (zero) or `NULL`  .
- `0`   else.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Table:

┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │    0 │
│ 3 │    3 │
└───┴──────┘ Query:

SELECT x FROM t_null WHERE isZeroOrNull(y) Result:

┌─x─┐
│ 1 │
│ 2 │
└───┘
## coalesce [¶](https://www.tinybird.co/docs/about:blank#coalesce)

Returns the leftmost non- `NULL` argument.

coalesce(x,...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- Any number of parameters of non-compound type. All parameters must be of mutually compatible data types.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The first non- `NULL`   argument
- `NULL`   , if all arguments are `NULL`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Consider a list of contacts that may specify multiple ways to contact a customer.

┌─name─────┬─mail─┬─phone─────┬──telegram─┐
│ client 1 │ ᴺᵁᴸᴸ │ 123-45-67 │       123 │
│ client 2 │ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ      │      ᴺᵁᴸᴸ │
└──────────┴──────┴───────────┴───────────┘ The `mail` and `phone` fields are of type String, but the `telegram` field is `UInt32` , so it needs to be converted to `String`.

Get the first available contact method for the customer from the contact list:

SELECT name, coalesce(mail, phone, CAST(telegram,'Nullable(String)')) FROM aBook```

```text
┌─name─────┬─coalesce(mail, phone, CAST(telegram, 'Nullable(String)'))─┐
│ client 1 │ 123-45-67                                                 │
│ client 2 │ ᴺᵁᴸᴸ                                                      │
└──────────┴───────────────────────────────────────────────────────────┘
## ifNull [¶](https://www.tinybird.co/docs/about:blank#ifnull)

Returns an alternative value if the argument is `NULL`.

ifNull(x, alt)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The value to check for `NULL`  .
- `alt`   : The value that the function returns if `x`   is `NULL`  .

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `x`   if `x`   isn't `NULL`  .
- `alt`   if `x`   is `NULL`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT ifNull('a', 'b')```

Result:

```text
┌─ifNull('a', 'b')─┐
│ a                │
└──────────────────┘ Query:

SELECT ifNull(NULL, 'b')```

Result:

```text
┌─ifNull(NULL, 'b')─┐
│ b                 │
└───────────────────┘
## nullIf [¶](https://www.tinybird.co/docs/about:blank#nullif)

Returns `NULL` if both arguments are equal.

nullIf(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`x`, `y` : Values to compare. Must be of compatible types.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- `NULL`   if the arguments are equal.
- `x`   if the arguments aren't equal.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT nullIf(1, 1)```

Result:

```text
┌─nullIf(1, 1)─┐
│         ᴺᵁᴸᴸ │
└──────────────┘ Query:

SELECT nullIf(1, 2)```

Result:

```text
┌─nullIf(1, 2)─┐
│            1 │
└──────────────┘
## assumeNotNull [¶](https://www.tinybird.co/docs/about:blank#assumenotnull)

Returns the corresponding non- `Nullable` value for a value of Nullable type. If the original value is `NULL` , an arbitrary result can be returned.

assumeNotNull(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The original value.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The input value as non- `Nullable`   type, if it's not `NULL`  .
- An arbitrary value, if the input value is `NULL`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Table:

┌─x─┬────y─┐
│ 1 │ ᴺᵁᴸᴸ │
│ 2 │    3 │
└───┴──────┘ Query:

SELECT assumeNotNull(y) FROM table```

Result:

```text
┌─assumeNotNull(y)─┐
│                0 │
│                3 │
└──────────────────┘ Query:

SELECT toTypeName(assumeNotNull(y)) FROM t_null```

Result:

```text
┌─toTypeName(assumeNotNull(y))─┐
│ Int8                         │
│ Int8                         │
└──────────────────────────────┘
## toNullable [¶](https://www.tinybird.co/docs/about:blank#tonullable)

Converts the argument type to `Nullable`.

toNullable(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : A value of non-compound type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The input value but of `Nullable`   type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT toTypeName(10)```

Result:

```text
┌─toTypeName(10)─┐
│ UInt8          │
└────────────────┘ Query:

SELECT toTypeName(toNullable(10))```

Result:

```text
┌─toTypeName(toNullable(10))─┐
│ Nullable(UInt8)            │
└────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/encryption-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Encryption functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for data encryption and decryption."
---




# Encryption functions [¶](https://www.tinybird.co/docs/about:blank#encryption-functions)

These following functions implement encryption and decryption of data with the AES algorithm.

Key length depends on encryption mode. It is 16, 24, and 32 bytes long for `-128-`, `-196-` , and `-256-` modes respectively.

Initialization vector length is always 16 bytes. Bytes in excess of 16 are ignored.

## encrypt [¶](https://www.tinybird.co/docs/about:blank#encrypt)

This function encrypts data using these modes:

- aes-128-ecb, aes-192-ecb, aes-256-ecb
- aes-128-cbc, aes-192-cbc, aes-256-cbc
- aes-128-ofb, aes-192-ofb, aes-256-ofb
- aes-128-gcm, aes-192-gcm, aes-256-gcm
- aes-128-ctr, aes-192-ctr, aes-256-ctr

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

encrypt('mode', 'plaintext', 'key' [, iv, aad])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mode`   : Encryption mode. String.
- `plaintext`   : Text that need to be encrypted. String.
- `key`   : Encryption key. String.
- `iv`   : Initialization vector. Required for `-gcm`   modes, optional for others. String.
- `aad`   : Additional authenticated data. It isn't encrypted, but it affects decryption. Works only in `-gcm`   modes, for others would throw an exception. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Ciphertext binary string. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Mock some data (please avoid storing the keys/ivs in the database as this undermines the whole concept of encryption), also storing 'hints' is unsafe too and used only for illustrative purposes:

Query:

SELECT comment, hex(secret) FROM
(
  select c1 as comment, c2 as secret from values(
    ('aes-256-ofb no IV', encrypt('aes-256-ofb', 'Secret', '12345678910121314151617181920212')),
    ('aes-256-ofb no IV, different key', encrypt('aes-256-ofb', 'Secret', 'keykeykeykeykeykeykeykeykeykeyke')),
    ('aes-256-ofb with IV', encrypt('aes-256-ofb', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv')),
    ('aes-256-cbc no IV', encrypt('aes-256-cbc', 'Secret', '12345678910121314151617181920212'))
  )
) Result:

┌─comment──────────────────────────┬─hex(secret)──────────────────────┐
│ aes-256-ofb no IV                │ B4972BDC4459                     │
│ aes-256-ofb no IV, different key │ 2FF57C092DC9                     │
│ aes-256-ofb with IV              │ 5E6CB398F653                     │
│ aes-256-cbc no IV                │ 1BC0629A92450D9E73A00E7D02CF4142 │
└──────────────────────────────────┴──────────────────────────────────┘ Example with `-gcm`:

Query:

SELECT comment, hex(secret) FROM
(
  select c1 as comment, c2 as secret from values(
    ('aes-256-ofb no IV', encrypt('aes-256-ofb', 'Secret', '12345678910121314151617181920212')),
    ('aes-256-ofb no IV, different key', encrypt('aes-256-ofb', 'Secret', 'keykeykeykeykeykeykeykeykeykeyke')),
    ('aes-256-ofb with IV', encrypt('aes-256-ofb', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv')),
    ('aes-256-cbc no IV', encrypt('aes-256-cbc', 'Secret', '12345678910121314151617181920212')),
    ('aes-256-gcm', encrypt('aes-256-gcm', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv')),
    ('aes-256-gcm with AAD', encrypt('aes-256-gcm', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv', 'aad'))
  )
)
WHERE comment LIKE '%gcm%' Result:

┌─comment──────────────┬─hex(secret)──────────────────────────────────┐
│ aes-256-gcm          │ A8A3CCBC6426CFEEB60E4EAE03D3E94204C1B09E0254 │
│ aes-256-gcm with AAD │ A8A3CCBC6426D9A1017A0A932322F1852260A4AD6837 │
└──────────────────────┴──────────────────────────────────────────────┘
## aes_encrypt_mysql [¶](https://www.tinybird.co/docs/about:blank#aes-encrypt-mysql)

Compatible with mysql encryption and resulting ciphertext can be decrypted with AES_DECRYPT function.

Will produce the same ciphertext as `encrypt` on equal inputs. But when `key` or `iv` are longer than they should normally be, `aes_encrypt_mysql` will stick to what MySQL's `aes_encrypt` does: 'fold' `key` and ignore excess bits of `iv`.

Supported encryption modes:

- aes-128-ecb, aes-192-ecb, aes-256-ecb
- aes-128-cbc, aes-192-cbc, aes-256-cbc
- aes-128-ofb, aes-192-ofb, aes-256-ofb

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

aes_encrypt_mysql('mode', 'plaintext', 'key' [, iv])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mode`   : Encryption mode. String.
- `plaintext`   : Text that needs to be encrypted. String.
- `key`   : Encryption key. If key is longer than required by mode, MySQL-specific key folding is performed. String.
- `iv`   : Initialization vector. Optional, only first 16 bytes are taken into account String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Ciphertext binary string. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Given equal input `encrypt` and `aes_encrypt_mysql` produce the same ciphertext:

Query:

SELECT encrypt('aes-256-ofb', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv') = aes_encrypt_mysql('aes-256-ofb', 'Secret', '12345678910121314151617181920212', 'iviviviviviviviv') AS ciphertexts_equal Result:

┌─ciphertexts_equal─┐
│                 1 │
└───────────────────┘ But `encrypt` fails when `key` or `iv` is longer than expected:

Query:

SELECT encrypt('aes-256-ofb', 'Secret', '123456789101213141516171819202122', 'iviviviviviviviv123') Result:

Received exception from server (version 22.6.1):
Code: 36. DB::Exception: Received from localhost:9000. DB::Exception: Invalid key size: 33 expected 32: While processing encrypt('aes-256-ofb', 'Secret', '123456789101213141516171819202122', 'iviviviviviviviv123'). While `aes_encrypt_mysql` produces MySQL-compatible output:

Query:

SELECT hex(aes_encrypt_mysql('aes-256-ofb', 'Secret', '123456789101213141516171819202122', 'iviviviviviviviv123')) AS ciphertext Result:

┌─ciphertext───┐
│ 24E9E4966469 │
└──────────────┘ Notice how supplying even longer `IV` produces the same result

Query:

SELECT hex(aes_encrypt_mysql('aes-256-ofb', 'Secret', '123456789101213141516171819202122', 'iviviviviviviviv123456')) AS ciphertext Result:

┌─ciphertext───┐
│ 24E9E4966469 │
└──────────────┘ Which is binary equal to what MySQL produces on same inputs:

mysql> SET  block_encryption_mode='aes-256-ofb'Query OK, 0 rows affected (0.00 sec)

mysql> SELECT aes_encrypt('Secret', '123456789101213141516171819202122', 'iviviviviviviviv123456') as ciphertext+------------------------+
| ciphertext             |
+------------------------+
| 0x24E9E4966469         |
+------------------------+
1 row in set (0.00 sec)
## decrypt [¶](https://www.tinybird.co/docs/about:blank#decrypt)

This function decrypts ciphertext into a plaintext using these modes:

- aes-128-ecb, aes-192-ecb, aes-256-ecb
- aes-128-cbc, aes-192-cbc, aes-256-cbc
- aes-128-ofb, aes-192-ofb, aes-256-ofb
- aes-128-gcm, aes-192-gcm, aes-256-gcm
- aes-128-ctr, aes-192-ctr, aes-256-ctr

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

decrypt('mode', 'ciphertext', 'key' [, iv, aad])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mode`   : Decryption mode. String.
- `ciphertext`   : Encrypted text that needs to be decrypted. String.
- `key`   : Decryption key. String.
- `iv`   : Initialization vector. Required for `-gcm`   modes, Optional for others. String.
- `aad`   : Additional authenticated data. Won't decrypt if this value is incorrect. Works only in `-gcm`   modes, for others would throw an exception. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Decrypted String. String.

## tryDecrypt [¶](https://www.tinybird.co/docs/about:blank#trydecrypt)

Similar to `decrypt` , but returns NULL if decryption fails because of using the wrong key.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Create a table where `user_id` is the unique user id, `encrypted` is an encrypted string field, `iv` is an initial vector for decrypt/encrypt. Assume that users know their id and the key to decrypt the encrypted field:

Query:

SELECT
    dt,
    user_id,
    tryDecrypt('aes-256-gcm', encrypted, 'keykeykeykeykeykeykeykeykeykey02', iv) AS value
FROM (
    select c1::DateTime as dt, c2::UInt32 as user_id, c3::String as encrypted, c4::String as iv from values(
        ('2022-08-02 00:00:00', 1, encrypt('aes-256-gcm', 'value1', 'keykeykeykeykeykeykeykeykeykey01', 'iv1'), 'iv1'),
        ('2022-09-02 00:00:00', 2, encrypt('aes-256-gcm', 'value2', 'keykeykeykeykeykeykeykeykeykey02', 'iv2'), 'iv2'),
        ('2022-09-02 00:00:01', 3, encrypt('aes-256-gcm', 'value3', 'keykeykeykeykeykeykeykeykeykey03', 'iv3'), 'iv3')
    )
)
ORDER BY user_id ASC Result:

┌──────────────────dt─┬─user_id─┬─value──┐
│ 2022-08-02 00:00:00 │       1 │ ᴺᵁᴸᴸ   │
│ 2022-09-02 00:00:00 │       2 │ value2 │
│ 2022-09-02 00:00:01 │       3 │ ᴺᵁᴸᴸ   │
└─────────────────────┴─────────┴────────┘
## aes_decrypt_mysql [¶](https://www.tinybird.co/docs/about:blank#aes-decrypt-mysql)

Compatible with mysql encryption and decrypts data encrypted with AES_ENCRYPT function.

Will produce same plaintext as `decrypt` on equal inputs. But when `key` or `iv` are longer than they should normally be, `aes_decrypt_mysql` will stick to what MySQL's `aes_decrypt` does: 'fold' `key` and ignore excess bits of `IV`.

Supported decryption modes:

- aes-128-ecb, aes-192-ecb, aes-256-ecb
- aes-128-cbc, aes-192-cbc, aes-256-cbc
- aes-128-cfb128
- aes-128-ofb, aes-192-ofb, aes-256-ofb

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

aes_decrypt_mysql('mode', 'ciphertext', 'key' [, iv])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `mode`   : Decryption mode. String.
- `ciphertext`   : Encrypted text that needs to be decrypted. String.
- `key`   : Decryption key. String.
- `iv`   : Initialization vector. Optional. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Decrypted String. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Decrypt data you've previously encrypted with MySQL:

mysql> SET  block_encryption_mode='aes-256-ofb'Query OK, 0 rows affected (0.00 sec)

mysql> SELECT aes_encrypt('Secret', '123456789101213141516171819202122', 'iviviviviviviviv123456') as ciphertext+------------------------+
| ciphertext             |
+------------------------+
| 0x24E9E4966469         |
+------------------------+
1 row in set (0.00 sec) Query:

SELECT aes_decrypt_mysql('aes-256-ofb', unhex('24E9E4966469'), '123456789101213141516171819202122', 'iviviviviviviviv123456') AS plaintext Result:

┌─plaintext─┐
│ Secret    │
└───────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/encoding-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Encoding functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for encoding and decoding data."
---




# Encoding functions [¶](https://www.tinybird.co/docs/about:blank#encoding-functions)

These functions are used to encode and decode data.

## char [¶](https://www.tinybird.co/docs/about:blank#char)

Returns the string with the length as the number of passed arguments and each byte has the value of corresponding argument. Accepts multiple arguments of numeric types. If the value of argument is out of range of UInt8 data type, it's converted to UInt8 with possible rounding and overflow.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

char(number_1, [number_2, ..., number_n])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `number_1, number_2, ..., number_n`   : Numerical arguments interpreted as integers. Types: Int, Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a string of given bytes. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT char(104.1, 101, 108.9, 108.9, 111) AS hello Result:

┌─hello─┐
│ hello │
└───────┘ You can construct a string of arbitrary encoding by passing the corresponding bytes. Here is example for UTF-8:

Query:

SELECT char(0xD0, 0xBF, 0xD1, 0x80, 0xD0, 0xB8, 0xD0, 0xB2, 0xD0, 0xB5, 0xD1, 0x82) AS hello Result:

┌─hello──┐
│ привет │
└────────┘ Query:

SELECT char(0xE4, 0xBD, 0xA0, 0xE5, 0xA5, 0xBD) AS hello Result:

┌─hello─┐
│ 你好  │
└───────┘
## hex [¶](https://www.tinybird.co/docs/about:blank#hex)

Returns a string containing the argument’s hexadecimal representation.

Alias: `HEX`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

hex(arg) The function is using uppercase letters `A-F` and not using any prefixes (like `0x` ) or suffixes (like `h` ).

For integer arguments, it prints hex digits (“nibbles”) from the most significant to least significant (big-endian or “human-readable” order). It starts with the most significant non-zero byte (leading zero bytes are omitted) but always prints both digits of every byte even if the leading digit is zero.

Values of type Date and DateTime are formatted as corresponding integers (the number of days since Epoch for Date and the value of Unix Timestamp for DateTime).

For String and FixedString, all bytes are simply encoded as two hexadecimal numbers. Zero bytes aren't omitted.

Values of Float and Decimal types are encoded as their representation in memory. They are encoded in little-endian. Zero leading/trailing bytes aren't omitted.

Values of UUID type are encoded as big-endian order string.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : A value to convert to hexadecimal. Types: String, UInt, Float, Decimal, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A string with the hexadecimal representation of the argument. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT hex(1) Result:

01 Query:

SELECT hex(toFloat32(number)) AS hex_presentation FROM numbers(15, 2) Result:

┌─hex_presentation─┐
│ 00007041         │
│ 00008041         │
└──────────────────┘ Query:

SELECT hex(toFloat64(number)) AS hex_presentation FROM numbers(15, 2) Result:

┌─hex_presentation─┐
│ 0000000000002E40 │
│ 0000000000003040 │
└──────────────────┘ Query:

SELECT lower(hex(toUUID('61f0c404-5cb3-11e7-907b-a6006ad3dba0'))) as uuid_hex Result:

┌─uuid_hex─────────────────────────┐
│ 61f0c4045cb311e7907ba6006ad3dba0 │
└──────────────────────────────────┘
## unhex [¶](https://www.tinybird.co/docs/about:blank#unhex)

Performs the opposite operation of hex. It interprets each pair of hexadecimal digits (in the argument) as a number and converts it to the byte represented by the number. The return value is a binary string (BLOB).

If you want to convert the result to a number, you can use the reverse and reinterpretAs functions.

Alias: `UNHEX`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

unhex(arg)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : A string containing any number of hexadecimal digits. String, FixedString.

Supports both uppercase and lowercase letters `A-F` . The number of hexadecimal digits doesn't have to be even. If it's odd, the last digit is interpreted as the least significant half of the `00-0F` byte. If the argument string contains anything other than hexadecimal digits, some implementation-defined result is returned (an exception isn’t thrown). For a numeric argument the inverse of hex(N) isn't performed by unhex().

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A binary string (BLOB). String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT unhex('303132'), UNHEX('4D7953514C') Result:

┌─unhex('303132')─┬─unhex('4D7953514C')─┐
│ 012             │ MySQL               │
└─────────────────┴─────────────────────┘ Query:

SELECT reinterpretAsUInt64(reverse(unhex('FFF'))) AS num Result:

┌──num─┐
│ 4095 │
└──────┘
## bin [¶](https://www.tinybird.co/docs/about:blank#bin)

Returns a string containing the argument’s binary representation.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bin(arg) Alias: `BIN`.

For integer arguments, it prints bin digits from the most significant to least significant (big-endian or “human-readable” order). It starts with the most significant non-zero byte (leading zero bytes are omitted) but always prints eight digits of every byte if the leading digit is zero.

Values of type Date and DateTime are formatted as corresponding integers (the number of days since Epoch for `Date` and the value of Unix Timestamp for `DateTime` ).

For String and FixedString, all bytes are simply encoded as eight binary numbers. Zero bytes aren't omitted.

Values of Float and Decimal types are encoded as their representation in memory. They are encoded in little-endian. Zero leading/trailing bytes aren't omitted.

Values of UUID type are encoded as big-endian order string.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : A value to convert to binary. String, FixedString, UInt, Float, Decimal, Date, or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A string with the binary representation of the argument. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT bin(14) Result:

┌─bin(14)──┐
│ 00001110 │
└──────────┘ Query:

SELECT bin(toFloat32(number)) AS bin_presentation FROM numbers(15, 2) Result:

┌─bin_presentation─────────────────┐
│ 00000000000000000111000001000001 │
│ 00000000000000001000000001000001 │
└──────────────────────────────────┘ Query:

SELECT bin(toFloat64(number)) AS bin_presentation FROM numbers(15, 2) Result:

┌─bin_presentation─────────────────────────────────────────────────┐
│ 0000000000000000000000000000000000000000000000000010111001000000 │
│ 0000000000000000000000000000000000000000000000000011000001000000 │
└─────────────────────────────────────────-────────────────────────┘ Query:

SELECT bin(toUUID('61f0c404-5cb3-11e7-907b-a6006ad3dba0')) as bin_uuid Result:

┌─bin_uuid─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 01100001111100001100010000000100010111001011001100010001111001111001000001111011101001100000000001101010110100111101101110100000 │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## unbin [¶](https://www.tinybird.co/docs/about:blank#unbin)

Interprets each pair of binary digits (in the argument) as a number and converts it to the byte represented by the number. The functions performs the opposite operation to bin.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

unbin(arg) Alias: `UNBIN`.

For a numeric argument `unbin()` doesn't return the inverse of `bin()` . If you want to convert the result to a number, you can use the reverse and reinterpretAs functions.

Supports binary digits `0` and `1` . The number of binary digits doesn't have to be multiples of eight. If the argument string contains anything other than binary digits, some implementation-defined result is returned (an exception isn’t thrown).

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : A string containing any number of binary digits. String, FixedString.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A binary string (BLOB). String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT unbin('303132'), UNBIN('4D7953514C') Result:

┌─unbin('303132')─┬─unbin('4D7953514C')─┐
│ 012             │ MySQL               │
└─────────────────┴─────────────────────┘
## bitAnd [¶](https://www.tinybird.co/docs/about:blank#bitand)

Computes the logical conjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAnd(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapAnd(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res─┐
│ [3] │
└─────┘
## bitmapOr [¶](https://www.tinybird.co/docs/about:blank#bitmapor)

Computes the logical disjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapOr(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapOr(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res─────────┐
│ [1,2,3,4,5] │
└─────────────┘
## bitmapXor [¶](https://www.tinybird.co/docs/about:blank#bitmapxor)

Xor-s two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapXor(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapXor(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res───────┐
│ [1,2,4,5] │
└───────────┘
## bitmapAndnot [¶](https://www.tinybird.co/docs/about:blank#bitmapandnot)

Computes the logical conjunction of two bitmaps and negates the result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAndnot(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapAndnot(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res───┐
│ [1,2] │
└───────┘
## bitmapAndCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapandcardinality)

Returns the cardinality of the logical conjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAndCardinality(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapAndCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   1 │
└─────┘
## bitmapOrCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmaporcardinality)

Returns the cardinality of the logical disjunction of two bitmaps.

bitmapOrCardinality(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapOrCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   5 │
└─────┘
## bitmapXorCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapxorcardinality)

Returns the cardinality of the XOR of two bitmaps.

bitmapXorCardinality(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapXorCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   4 │
└─────┘
## bitmapAndnotCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapandnotcardinality)

Returns the cardinality of the AND-NOT operation of two bitmaps.

bitmapAndnotCardinality(bitmap, bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapAndnotCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   2 │
└─────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/distance-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Distance functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for calculating distances."
---




# Distance functions [¶](https://www.tinybird.co/docs/about:blank#distance-functions)

The following functions are used to calculate distances between two points.

## L1Norm [¶](https://www.tinybird.co/docs/about:blank#l1norm)

Calculates the sum of absolute values of a vector.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L1Norm(vector) Alias: `normL1`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector`   : Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- L1-norm or taxicab geometry distance. UInt, Float or Decimal.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT L1Norm((1, 2)) Result:

┌─L1Norm((1, 2))─┐
│              3 │
└────────────────┘
## L2Norm [¶](https://www.tinybird.co/docs/about:blank#l2norm)

Calculates the square root of the sum of the squares of the vector values.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L2Norm(vector) Alias: `normL2`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector`   : Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- L2-norm or Euclidean distance. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L2Norm((1, 2)) Result:

┌───L2Norm((1, 2))─┐
│ 2.23606797749979 │
└──────────────────┘
## L2SquaredNorm [¶](https://www.tinybird.co/docs/about:blank#l2squarednorm)

Calculates the square root of the sum of the squares of the vector values (the L2Norm) squared.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L2SquaredNorm(vector) Alias: `normL2Squared`.

### *Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector`   : Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- L2-norm squared. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L2SquaredNorm((1, 2)) Result:

┌─L2SquaredNorm((1, 2))─┐
│                     5 │
└───────────────────────┘
## LinfNorm [¶](https://www.tinybird.co/docs/about:blank#linfnorm)

Calculates the maximum of absolute values of a vector.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LinfNorm(vector) Alias: `normLinf`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector`   : Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Linf-norm or the maximum absolute value. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LinfNorm((1, -2)) Result:

┌─LinfNorm((1, -2))─┐
│                 2 │
└───────────────────┘
## LpNorm [¶](https://www.tinybird.co/docs/about:blank#lpnorm)

Calculates the root of `p` -th power of the sum of the absolute values of a vector in the power of `p`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LpNorm(vector, p) Alias: `normLp`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector`   : Tuple or Array.
- `p`   : The power. Possible values: real number in `1; inf)`   . [UInt or Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Lp-norm#p-norm). Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LpNorm((1, -2), 2) Result:

┌─LpNorm((1, -2), 2)─┐
│   2.23606797749979 │
└────────────────────┘
## L1Distance [¶](https://www.tinybird.co/docs/about:blank#l1distance)

Calculates the distance between two points (the values of the vectors are the coordinates) in `L1` space (1-norm (taxicab geometry distance)).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L1Distance(vector1, vector2) Alias: `distanceL1`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First vector. Tuple or Array.
- `vector2`   : Second vector. Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- 1-norm distance. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L1Distance((1, 2), (2, 3)) Result:

┌─L1Distance((1, 2), (2, 3))─┐
│                          2 │
└────────────────────────────┘
## L2Distance [¶](https://www.tinybird.co/docs/about:blank#l2distance)

Calculates the distance between two points (the values of the vectors are the coordinates) in Euclidean space (Euclidean distance).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L2Distance(vector1, vector2) Alias: `distanceL2`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First vector. Tuple or Array.
- `vector2`   : Second vector. Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- 2-norm distance. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L2Distance((1, 2), (2, 3)) Result:

┌─L2Distance((1, 2), (2, 3))─┐
│         1.4142135623730951 │
└────────────────────────────┘
## L2SquaredDistance [¶](https://www.tinybird.co/docs/about:blank#l2squareddistance)

Calculates the sum of the squares of the difference between the corresponding elements of two vectors.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L2SquaredDistance(vector1, vector2) Alias: `distanceL2Squared`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First vector. Tuple or Array.
- `vector2`   : Second vector. Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Sum of the squares of the difference between the corresponding elements of two vectors. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L2SquaredDistance([1, 2, 3], [0, 0, 0]) Result:

┌─L2SquaredDistance([1, 2, 3], [0, 0, 0])─┐
│                                      14 │
└─────────────────────────────────────────┘
## LinfDistance [¶](https://www.tinybird.co/docs/about:blank#linfdistance)

Calculates the distance between two points (the values of the vectors are the coordinates) in `L_{inf}` space (maximum norm#Maximum_norm_(special_case_of:_infinity_norm,_uniform_norm,_or_supremum_norm))).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LinfDistance(vector1, vector2) Alias: `distanceLinf`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First vector. Tuple or Array.
- `vector1`   : Second vector. Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Infinity-norm distance. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LinfDistance((1, 2), (2, 3)) Result:

┌─LinfDistance((1, 2), (2, 3))─┐
│                            1 │
└──────────────────────────────┘
## LpDistance [¶](https://www.tinybird.co/docs/about:blank#lpdistance)

Calculates the distance between two points (the values of the vectors are the coordinates) in `Lp` space (p-norm distance#p-norm)).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LpDistance(vector1, vector2, p) Alias: `distanceLp`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First vector. Tuple or Array.
- `vector2`   : Second vector. Tuple or Array.
- `p`   : The power. Possible values: real number from `1; inf)`   . [UInt or Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- p-norm distance. Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LpDistance((1, 2), (2, 3), 3) Result:

┌─LpDistance((1, 2), (2, 3), 3)─┐
│            1.2599210498948732 │
└───────────────────────────────┘
## L1Normalize [¶](https://www.tinybird.co/docs/about:blank#l1normalize)

Calculates the unit vector of a given vector (the values of the tuple are the coordinates) in `L1` space (taxicab geometry).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L1Normalize(tuple) Alias: `normalizeL1`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `tuple`   : Tuple.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Unit vector. Tuple of Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L1Normalize((1, 2)) Result:

┌─L1Normalize((1, 2))─────────────────────┐
│ (0.3333333333333333,0.6666666666666666) │
└─────────────────────────────────────────┘
## L2Normalize [¶](https://www.tinybird.co/docs/about:blank#l2normalize)

Calculates the unit vector of a given vector (the values of the tuple are the coordinates) in Euclidean space (using Euclidean distance).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

L2Normalize(tuple) Alias: `normalizeL1`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `tuple`   : Tuple.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Unit vector. Tuple of Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT L2Normalize((3, 4)) Result:

┌─L2Normalize((3, 4))─┐
│ (0.6,0.8)           │
└─────────────────────┘
## LinfNormalize [¶](https://www.tinybird.co/docs/about:blank#linfnormalize)

Calculates the unit vector of a given vector (the values of the tuple are the coordinates) in `L_{inf}` space (using maximum norm#Maximum_norm_(special_case_of:_infinity_norm,_uniform_norm,_or_supremum_norm))).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LinfNormalize(tuple) Alias: `normalizeLinf`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `tuple`   : Tuple.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Unit vector. Tuple of Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LinfNormalize((3, 4)) Result:

┌─LinfNormalize((3, 4))─┐
│ (0.75,1)              │
└───────────────────────┘
## LpNormalize [¶](https://www.tinybird.co/docs/about:blank#lpnormalize)

Calculates the unit vector of a given vector (the values of the tuple are the coordinates) in `Lp` space (using p-norm#p-norm)).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LpNormalize(tuple, p) Alias: `normalizeLp`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `tuple`   : Tuple.
- `p`   : The power. Possible values: any number from 1;inf). [UInt or Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Unit vector. Tuple of Float.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT LpNormalize((3, 4),5) Result:

┌─LpNormalize((3, 4), 5)──────────────────┐
│ (0.7187302630182624,0.9583070173576831) │
└─────────────────────────────────────────┘
## cosineDistance [¶](https://www.tinybird.co/docs/about:blank#cosinedistance)

Calculates the cosine distance between two vectors (the values of the tuples are the coordinates). The smaller the returned value is, the more similar are the vectors.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

cosineDistance(vector1, vector2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `vector1`   : First tuple. Tuple or Array.
- `vector2`   : Second tuple. Tuple or Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Cosine of the angle between two vectors subtracted from one. Float.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT cosineDistance((1, 2), (2, 3)) Result:

┌─cosineDistance((1, 2), (2, 3))─┐
│           0.007722123286332261 │
└────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/date-time-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Date and Timefunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for manipulating dates and times."
---




# Functions for working with dates and times [¶](https://www.tinybird.co/docs/about:blank#functions-for-working-with-dates-and-times)

Most functions in this section accept an optional time zone argument, for example `Europe/Amsterdam` . In this case, the time zone is the specified one instead of the local or default one.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toDateTime('2016-06-15 23:00:00') AS time,
    toDate(time) AS date_local,
    toDate(time, 'Asia/Yekaterinburg') AS date_yekat,
    toString(time, 'US/Samoa') AS time_samoa ┌────────────────time─┬─date_local─┬─date_yekat─┬─time_samoa──────────┐
│ 2016-06-15 23:00:00 │ 2016-06-15 │ 2016-06-16 │ 2016-06-15 09:00:00 │
└─────────────────────┴────────────┴────────────┴─────────────────────┘
## makeDate [¶](https://www.tinybird.co/docs/about:blank#makedate)

Creates a Date

- from a year, month and day argument, or
- from a year and day of year argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

makeDate(year, month, day)makeDate(year, day_of_year) Alias:

- `MAKEDATE(year, month, day);`
- `MAKEDATE(year, day_of_year);`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `year`   : Year. Integer, Float or Decimal.
- `month`   : Month. Integer, Float or Decimal.
- `day`   : Day. Integer, Float or Decimal.
- `day_of_year`   : Day of the year. Integer, Float or Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A date created from the arguments. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Create a Date from a year, month and day:

SELECT makeDate(2023, 2, 28) AS Date Result:

┌───────date─┐
│ 2023-02-28 │
└────────────┘ Create a Date from a year and day of year argument:

SELECT makeDate(2023, 42) AS Date Result:

┌───────date─┐
│ 2023-02-11 │
└────────────┘
## makeDate32 [¶](https://www.tinybird.co/docs/about:blank#makedate32)

Creates a date of type Date32 from a year, month, day (or optionally a year and a day).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

makeDate32(year, [month,] day)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `year`   : Year. Integer, Float or Decimal.
- `month`   : Month (optional). Integer, Float or Decimal.
- `day`   : Day. Integer, Float or Decimal.

If `month` is omitted then `day` should take a value between `1` and `365` , otherwise it should take a value between `1` and `31`.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- A date created from the arguments. Date32.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Create a date from a year, month, and day:

Query:

SELECT makeDate32(2024, 1, 1) Result:

2024-01-01 Create a Date from a year and day of year:

Query:

SELECT makeDate32(2024, 100) Result:

2024-04-09
## makeDateTime [¶](https://www.tinybird.co/docs/about:blank#makedatetime)

Creates a DateTime from a year, month, day, hour, minute and second argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

makeDateTime(year, month, day, hour, minute, second[, timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `year`   : Year. Integer, Float or Decimal.
- `month`   : Month. Integer, Float or Decimal.
- `day`   : Day. Integer, Float or Decimal.
- `hour`   : Hour. Integer, Float or Decimal.
- `minute`   : Minute. Integer, Float or Decimal.
- `second`   : Second. Integer, Float or Decimal.
- `timezone`   : Timezone for the returned value (optional).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A date with time created from the arguments. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT makeDateTime(2023, 2, 28, 17, 12, 33) AS DateTime Result:

┌────────────DateTime─┐
│ 2023-02-28 17:12:33 │
└─────────────────────┘
## makeDateTime64 [¶](https://www.tinybird.co/docs/about:blank#makedatetime64)

Creates a DateTime64 data type value from its components: year, month, day, hour, minute, second. With optional sub-second precision.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

makeDateTime64(year, month, day, hour, minute, second[, precision])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `year`   : Year (0-9999). Integer, Float or Decimal.
- `month`   : Month (1-12). Integer, Float or Decimal.
- `day`   : Day (1-31). Integer, Float or Decimal.
- `hour`   : Hour (0-23). Integer, Float or Decimal.
- `minute`   : Minute (0-59). Integer, Float or Decimal.
- `second`   : Second (0-59). Integer, Float or Decimal.
- `precision`   : Optional precision of the sub-second component (0-9). Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A date and time created from the supplied arguments. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT makeDateTime64(2023, 5, 15, 10, 30, 45, 779, 5) ┌─makeDateTime64(2023, 5, 15, 10, 30, 45, 779, 5)─┐
│                       2023-05-15 10:30:45.00779 │
└─────────────────────────────────────────────────┘
## timestamp [¶](https://www.tinybird.co/docs/about:blank#timestamp)

Converts the first argument 'expr' to type DateTime64(6). If a second argument 'expr_time' is provided, it adds the specified time to the converted value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timestamp(expr[, expr_time]) Alias: `TIMESTAMP`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   - Date or date with time. String.
- `expr_time`   - Optional parameter. Time to add. String.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT timestamp('2023-12-31') as ts Result:

┌─────────────────────────ts─┐
│ 2023-12-31 00:00:00.000000 │
└────────────────────────────┘ SELECT timestamp('2023-12-31 12:00:00', '12:00:00.11') as ts Result:

┌─────────────────────────ts─┐
│ 2024-01-01 00:00:00.110000 │
└────────────────────────────┘
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- DateTime64(6)

## timeZone [¶](https://www.tinybird.co/docs/about:blank#timezone)

Returns the timezone of the current session, i.e. the value of setting session_timezone. If the function is executed in the context of a distributed table, then it generates a normal column with values relevant to each shard, otherwise it produces a constant value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeZone() Alias: `timezone`.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Timezone. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT timezone() Result:

┌─timezone()─────┐
│ America/Denver │
└────────────────┘
## serverTimeZone [¶](https://www.tinybird.co/docs/about:blank#servertimezone)

Returns the timezone of the server, i.e. the value of setting timezone. If the function is executed in the context of a distributed table, then it generates a normal column with values relevant to each shard. Otherwise, it produces a constant value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

serverTimeZone() Alias: `serverTimezone`.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Timezone. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT serverTimeZone() Result:

┌─serverTimeZone()─┐
│ UTC              │
└──────────────────┘
## toTimeZone [¶](https://www.tinybird.co/docs/about:blank#totimezone)

Converts a date or date with time to the specified time zone. Does not change the internal value (number of unix seconds) of the data, only the value's time zone attribute and the value's string representation changes.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toTimezone(value, timezone) Alias: `toTimezone`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Time or date and time. DateTime64.
- `timezone`   : Timezone for the returned value. String. This argument is a constant, because `toTimezone`   changes the timezone of a column (timezone is an attribute of `DateTime*`   types).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Date and time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDateTime('2019-01-01 00:00:00', 'UTC') AS time_utc,
    toTypeName(time_utc) AS type_utc,
    toInt32(time_utc) AS int32utc,
    toTimeZone(time_utc, 'Asia/Yekaterinburg') AS time_yekat,
    toTypeName(time_yekat) AS type_yekat,
    toInt32(time_yekat) AS int32yekat,
    toTimeZone(time_utc, 'US/Samoa') AS time_samoa,
    toTypeName(time_samoa) AS type_samoa,
    toInt32(time_samoa) AS int32samoa
FORMAT Vertical Result:

Row 1:
──────
time_utc:   2019-01-01 00:00:00
type_utc:   DateTime('UTC')
int32utc:   1546300800
time_yekat: 2019-01-01 05:00:00
type_yekat: DateTime('Asia/Yekaterinburg')
int32yekat: 1546300800
time_samoa: 2018-12-31 13:00:00
type_samoa: DateTime('US/Samoa')
int32samoa: 1546300800
## timeZoneOf [¶](https://www.tinybird.co/docs/about:blank#timezoneof)

Returns the timezone name of DateTime or DateTime64 data types.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeZoneOf(value) Alias: `timezoneOf`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Timezone name. String.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT timezoneOf(now()) Result:

┌─timezoneOf(now())─┐
│ Etc/UTC           │
└───────────────────┘
## timeZoneOffset [¶](https://www.tinybird.co/docs/about:blank#timezoneoffset)

Returns the timezone offset in seconds from UTC. The function daylight saving time and historical timezone changes at the specified date and time into account. The IANA timezone database is used to calculate the offset.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeZoneOffset(value) Alias: `timezoneOffset`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Offset from UTC in seconds. Int32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDateTime('2021-04-21 10:20:30', 'America/New_York') AS Time, toTypeName(Time) AS Type,
       timeZoneOffset(Time) AS Offset_in_seconds, (Offset_in_seconds / 3600) AS Offset_in_hours Result:

┌────────────────Time─┬─Type─────────────────────────┬─Offset_in_seconds─┬─Offset_in_hours─┐
│ 2021-04-21 10:20:30 │ DateTime('America/New_York') │            -14400 │              -4 │
└─────────────────────┴──────────────────────────────┴───────────────────┴─────────────────┘
## toYear [¶](https://www.tinybird.co/docs/about:blank#toyear)

Returns the year component (AD) of a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toYear(value) Alias: `YEAR`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The year of the given date/time. UInt16.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toYear(toDateTime('2023-04-21 10:20:30')) Result:

┌─toYear(toDateTime('2023-04-21 10:20:30'))─┐
│                                      2023 │
└───────────────────────────────────────────┘
## toQuarter [¶](https://www.tinybird.co/docs/about:blank#toquarter)

Returns the quarter (1-4) of a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toQuarter(value) Alias: `QUARTER`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The quarter of the year (1, 2, 3 or 4) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toQuarter(toDateTime('2023-04-21 10:20:30')) Result:

┌─toQuarter(toDateTime('2023-04-21 10:20:30'))─┐
│                                            2 │
└──────────────────────────────────────────────┘
## toMonth [¶](https://www.tinybird.co/docs/about:blank#tomonth)

Returns the month component (1-12) of a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toMonth(value) Alias: `MONTH`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The month of the year (1 - 12) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toMonth(toDateTime('2023-04-21 10:20:30')) Result:

┌─toMonth(toDateTime('2023-04-21 10:20:30'))─┐
│                                          4 │
└────────────────────────────────────────────┘
## toDayOfYear [¶](https://www.tinybird.co/docs/about:blank#todayofyear)

Returns the number of the day within the year (1-366) of a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toDayOfYear(value) Alias: `DAYOFYEAR`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The day of the year (1 - 366) of the given date/time. UInt16.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDayOfYear(toDateTime('2023-04-21 10:20:30')) Result:

┌─toDayOfYear(toDateTime('2023-04-21 10:20:30'))─┐
│                                            111 │
└────────────────────────────────────────────────┘
## toDayOfMonth [¶](https://www.tinybird.co/docs/about:blank#todayofmonth)

Returns the number of the day within the month (1-31) of a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toDayOfMonth(value) Aliases: `DAYOFMONTH`, `DAY`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The day of the month (1 - 31) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDayOfMonth(toDateTime('2023-04-21 10:20:30')) Result:

┌─toDayOfMonth(toDateTime('2023-04-21 10:20:30'))─┐
│                                              21 │
└─────────────────────────────────────────────────┘
## toDayOfWeek [¶](https://www.tinybird.co/docs/about:blank#todayofweek)

Returns the number of the day within the week of a date or date with time.

The two-argument form of `toDayOfWeek()` enables you to specify whether the week starts on Monday or Sunday, and whether the return value should be in the range from 0 to 6 or 1 to 7. If the mode argument is omitted, the default mode is 0. The time zone of the date can be specified as the third argument.

| Mode | First day of week | Range |
| --- | --- | --- |
| 0 | Monday | 1-7: Monday = 1, Tuesday = 2, ..., Sunday = 7 |
| 1 | Monday | 0-6: Monday = 0, Tuesday = 1, ..., Sunday = 6 |
| 2 | Sunday | 0-6: Sunday = 0, Monday = 1, ..., Saturday = 6 |
| 3 | Sunday | 1-7: Sunday = 1, Monday = 2, ..., Saturday = 7 |

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toDayOfWeek(t[, mode[, timezone]]) Alias: `DAYOFWEEK`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `t`   - a Date, Date32, DateTime or DateTime64
- `mode`   - determines what the first day of the week is. Possible values are 0, 1, 2 or 3. See the previous table for the differences.
- `timezone`   - optional parameter, it behaves like any other conversion function

The first argument can also be specified as String in a format supported by parseDateTime64BestEffort(). Support for string arguments exists only for reasons of compatibility with MySQL which is expected by certain 3rd party tools. As string argument support may in future be made dependent on new MySQL-compatibility settings and because string parsing is generally slow, it's recommended to not use it.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The day of the week (1-7), depending on the chosen mode, of the given date/time

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following date is April 21, 2023, which was a Friday:

SELECT
    toDayOfWeek(toDateTime('2023-04-21')),
    toDayOfWeek(toDateTime('2023-04-21'), 1) Result:

┌─toDayOfWeek(toDateTime('2023-04-21'))─┬─toDayOfWeek(toDateTime('2023-04-21'), 1)─┐
│                                     5 │                                        4 │
└───────────────────────────────────────┴──────────────────────────────────────────┘
## toHour [¶](https://www.tinybird.co/docs/about:blank#tohour)

Returns the hour component (0-24) of a date with time.

Assumes that if clocks are moved ahead, it's by one hour and occurs at 2 a.m., and if clocks are moved back, it's by one hour and occurs at 3 a.m. (which isn't always exactly when it occurs - it depends on the timezone).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toHour(value) Alias: `HOUR`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The hour of the day (0 - 23) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toHour(toDateTime('2023-04-21 10:20:30')) Result:

┌─toHour(toDateTime('2023-04-21 10:20:30'))─┐
│                                        10 │
└───────────────────────────────────────────┘
## toMinute [¶](https://www.tinybird.co/docs/about:blank#tominute)

Returns the minute component (0-59) a date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toMinute(value) Alias: `MINUTE`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The minute of the hour (0 - 59) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toMinute(toDateTime('2023-04-21 10:20:30')) Result:

┌─toMinute(toDateTime('2023-04-21 10:20:30'))─┐
│                                          20 │
└─────────────────────────────────────────────┘
## toSecond [¶](https://www.tinybird.co/docs/about:blank#tosecond)

Returns the second component (0-59) of a date with time. Leap seconds aren't considered.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toSecond(value) Alias: `SECOND`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The second in the minute (0 - 59) of the given date/time. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toSecond(toDateTime('2023-04-21 10:20:30')) Result:

┌─toSecond(toDateTime('2023-04-21 10:20:30'))─┐
│                                          30 │
└─────────────────────────────────────────────┘
## toMillisecond [¶](https://www.tinybird.co/docs/about:blank#tomillisecond)

Returns the millisecond component (0-999) of a date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toMillisecond(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - DateTime or DateTime64

Alias: `MILLISECOND`

SELECT toMillisecond(toDateTime64('2023-04-21 10:20:30.456', 3)) Result:

┌──toMillisecond(toDateTime64('2023-04-21 10:20:30.456', 3))─┐
│                                                        456 │
└────────────────────────────────────────────────────────────┘
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The millisecond in the minute (0 - 59) of the given date/time. UInt16.

## toUnixTimestamp [¶](https://www.tinybird.co/docs/about:blank#tounixtimestamp)

Converts a string, a date or a date with time to the Unix Timestamp in `UInt32` representation.

If the function is called with a string, it accepts an optional timezone argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toUnixTimestamp(date)
toUnixTimestamp(str, [timezone])
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the unix timestamp. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    '2017-11-05 08:07:47' AS dt_str,
    toUnixTimestamp(dt_str) AS from_str,
    toUnixTimestamp(dt_str, 'Asia/Tokyo') AS from_str_tokyo,
    toUnixTimestamp(toDateTime(dt_str)) AS from_datetime,
    toUnixTimestamp(toDateTime64(dt_str, 0)) AS from_datetime64,
    toUnixTimestamp(toDate(dt_str)) AS from_date,
    toUnixTimestamp(toDate32(dt_str)) AS from_date32
FORMAT Vertical Result:

Row 1:
──────
dt_str:          2017-11-05 08:07:47
from_str:        1509869267
from_str_tokyo:  1509836867
from_datetime:   1509869267
from_datetime64: 1509869267
from_date:       1509840000
from_date32:     1509840000 The return type of `toStartOf*`, `toLastDayOf*`, `toMonday`, `timeSlot` functions described below is determined by the configuration parameter enable_extended_results_for_datetime_functions which is `0` by default.

Behavior for

- `enable_extended_results_for_datetime_functions = 0`  :
  - Functions `toStartOfYear`    , `toStartOfISOYear`    , `toStartOfQuarter`    , `toStartOfMonth`    , `toStartOfWeek`    , `toLastDayOfWeek`    , `toLastDayOfMonth`    , `toMonday`     return `Date`     or `DateTime`    .
  - Functions `toStartOfDay`    , `toStartOfHour`    , `toStartOfFifteenMinutes`    , `toStartOfTenMinutes`    , `toStartOfFiveMinutes`    , `toStartOfMinute`    , `timeSlot`     return `DateTime`     . Though these functions can take values of the extended types `Date32`     and `DateTime64`     as an argument, passing them a time outside the normal range (year 1970 to 2149 for `Date`     / 2106 for `DateTime`     ) will produce wrong results.
- `enable_extended_results_for_datetime_functions = 1`  :
  - Functions `toStartOfYear`    , `toStartOfISOYear`    , `toStartOfQuarter`    , `toStartOfMonth`    , `toStartOfWeek`    , `toLastDayOfWeek`    , `toLastDayOfMonth`    , `toMonday`     return `Date`     or `DateTime`     if their argument is a `Date`     or `DateTime`     , and they return `Date32`     or `DateTime64`     if their argument is a `Date32`     or `DateTime64`    .
  - Functions `toStartOfDay`    , `toStartOfHour`    , `toStartOfFifteenMinutes`    , `toStartOfTenMinutes`    , `toStartOfFiveMinutes`    , `toStartOfMinute`    , `timeSlot`     return `DateTime`     if their argument is a `Date`     or `DateTime`     , and they return `DateTime64`     if their argument is a `Date32`     or `DateTime64`    .

## toStartOfYear [¶](https://www.tinybird.co/docs/about:blank#tostartofyear)

Rounds down a date or date with time to the first day of the year. Returns the date as a `Date` object.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfYear(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first day of the year of the input date/time. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toStartOfYear(toDateTime('2023-04-21 10:20:30')) Result:

┌─toStartOfYear(toDateTime('2023-04-21 10:20:30'))─┐
│                                       2023-01-01 │
└──────────────────────────────────────────────────┘
## toStartOfISOYear [¶](https://www.tinybird.co/docs/about:blank#tostartofisoyear)

Rounds down a date or date with time to the first day of the ISO year, which can be different than a "regular" year.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfISOYear(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first day of the year of the input date/time. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toStartOfISOYear(toDateTime('2023-04-21 10:20:30')) Result:

┌─toStartOfISOYear(toDateTime('2023-04-21 10:20:30'))─┐
│                                          2023-01-02 │
└─────────────────────────────────────────────────────┘
## toStartOfQuarter [¶](https://www.tinybird.co/docs/about:blank#tostartofquarter)

Rounds down a date or date with time to the first day of the quarter. The first day of the quarter is either 1 January, 1 April, 1 July, or 1 October. Returns the date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfQuarter(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first day of the quarter of the given date/time. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toStartOfQuarter(toDateTime('2023-04-21 10:20:30')) Result:

┌─toStartOfQuarter(toDateTime('2023-04-21 10:20:30'))─┐
│                                          2023-04-01 │
└─────────────────────────────────────────────────────┘
## toStartOfMonth [¶](https://www.tinybird.co/docs/about:blank#tostartofmonth)

Rounds down a date or date with time to the first day of the month. Returns the date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfMonth(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first day of the month of the given date/time. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toStartOfMonth(toDateTime('2023-04-21 10:20:30')) Result:

┌─toStartOfMonth(toDateTime('2023-04-21 10:20:30'))─┐
│                                        2023-04-01 │
└───────────────────────────────────────────────────┘ The behavior of parsing incorrect dates is implementation specific. The implementation may return zero date, throw an exception, or do "natural" overflow.

## toLastDayOfMonth [¶](https://www.tinybird.co/docs/about:blank#tolastdayofmonth)

Rounds a date or date with time to the last day of the month. Returns the date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toLastDayOfMonth(value) Alias: `LAST_DAY`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The last day of the month of the given date/time=. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toLastDayOfMonth(toDateTime('2023-04-21 10:20:30')) Result:

┌─toLastDayOfMonth(toDateTime('2023-04-21 10:20:30'))─┐
│                                          2023-04-30 │
└─────────────────────────────────────────────────────┘
## toMonday [¶](https://www.tinybird.co/docs/about:blank#tomonday)

Rounds down a date or date with time to the nearest Monday. Returns the date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toMonday(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The date of the nearest Monday on or prior to the given date. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toMonday(toDateTime('2023-04-21 10:20:30')), /* a Friday */
    toMonday(toDate('2023-04-24')), /* already a Monday */ Result:

┌─toMonday(toDateTime('2023-04-21 10:20:30'))─┬─toMonday(toDate('2023-04-24'))─┐
│                                  2023-04-17 │                     2023-04-24 │
└─────────────────────────────────────────────┴────────────────────────────────┘
## toStartOfWeek [¶](https://www.tinybird.co/docs/about:blank#tostartofweek)

Rounds a date or date with time down to the nearest Sunday or Monday. Returns the date. The mode argument works exactly like the mode argument in function `toWeek()` . If no mode is specified, it defaults to 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfWeek(t[, mode[, timezone]])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `t`   - a Date, Date32, DateTime or DateTime64
- `mode`   - determines the first day of the week as described in the toWeek() function
- `timezone`   - Optional parameter, it behaves like any other conversion function

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The date of the nearest Sunday or Monday on or prior to the given date, depending on the mode. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfWeek(toDateTime('2023-04-21 10:20:30')), /* a Friday */
    toStartOfWeek(toDateTime('2023-04-21 10:20:30'), 1), /* a Friday */
    toStartOfWeek(toDate('2023-04-24')), /* a Monday */
    toStartOfWeek(toDate('2023-04-24'), 1) /* a Monday */
FORMAT Vertical Result:

Row 1:
──────
toStartOfWeek(toDateTime('2023-04-21 10:20:30')):    2023-04-16
toStartOfWeek(toDateTime('2023-04-21 10:20:30'), 1): 2023-04-17
toStartOfWeek(toDate('2023-04-24')):                 2023-04-23
toStartOfWeek(toDate('2023-04-24'), 1):              2023-04-24
## toLastDayOfWeek [¶](https://www.tinybird.co/docs/about:blank#tolastdayofweek)

Rounds a date or date with time up to the nearest Saturday or Sunday. Returns the date. The mode argument works exactly like the mode argument in function `toWeek()` . If no mode is specified, mode is assumed as 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toLastDayOfWeek(t[, mode[, timezone]])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `t`   - a Date, Date32, DateTime or DateTime64
- `mode`   - determines the last day of the week as described in the toWeek function
- `timezone`   - Optional parameter, it behaves like any other conversion function

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The date of the nearest Sunday or Monday on or after the given date, depending on the mode. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toLastDayOfWeek(toDateTime('2023-04-21 10:20:30')), /* a Friday */
    toLastDayOfWeek(toDateTime('2023-04-21 10:20:30'), 1), /* a Friday */
    toLastDayOfWeek(toDate('2023-04-22')), /* a Saturday */
    toLastDayOfWeek(toDate('2023-04-22'), 1) /* a Saturday */
FORMAT Vertical Result:

Row 1:
──────
toLastDayOfWeek(toDateTime('2023-04-21 10:20:30')):    2023-04-22
toLastDayOfWeek(toDateTime('2023-04-21 10:20:30'), 1): 2023-04-23
toLastDayOfWeek(toDate('2023-04-22')):                 2023-04-22
toLastDayOfWeek(toDate('2023-04-22'), 1):              2023-04-23
## toStartOfDay [¶](https://www.tinybird.co/docs/about:blank#tostartofday)

Rounds down a date with time to the start of the day.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfDay(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the day of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toStartOfDay(toDateTime('2023-04-21 10:20:30')) Result:

┌─toStartOfDay(toDateTime('2023-04-21 10:20:30'))─┐
│                             2023-04-21 00:00:00 │
└─────────────────────────────────────────────────┘
## toStartOfHour [¶](https://www.tinybird.co/docs/about:blank#tostartofhour)

Rounds down a date with time to the start of the hour.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfHour(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a  DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the hour of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfHour(toDateTime('2023-04-21 10:20:30')),
    toStartOfHour(toDateTime64('2023-04-21', 6)) Result:

┌─toStartOfHour(toDateTime('2023-04-21 10:20:30'))─┬─toStartOfHour(toDateTime64('2023-04-21', 6))─┐
│                              2023-04-21 10:00:00 │                          2023-04-21 00:00:00 │
└──────────────────────────────────────────────────┴──────────────────────────────────────────────┘
## toStartOfMinute [¶](https://www.tinybird.co/docs/about:blank#tostartofminute)

Rounds down a date with time to the start of the minute.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfMinute(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a  DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the minute of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfMinute(toDateTime('2023-04-21 10:20:30')),
    toStartOfMinute(toDateTime64('2023-04-21 10:20:30.5300', 8))
FORMAT Vertical Result:

Row 1:
──────
toStartOfMinute(toDateTime('2023-04-21 10:20:30')):           2023-04-21 10:20:00
toStartOfMinute(toDateTime64('2023-04-21 10:20:30.5300', 8)): 2023-04-21 10:20:00
## toStartOfSecond [¶](https://www.tinybird.co/docs/about:blank#tostartofsecond)

Truncates sub-seconds.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfSecond(value, [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime64.
- `timezone`   : Timezone for the returned value (optional). If not specified, the function uses the timezone of the `value`   parameter. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Input value without sub-seconds. DateTime64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query without timezone:

WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64
SELECT toStartOfSecond(dt64) Result:

┌───toStartOfSecond(dt64)─┐
│ 2020-01-01 10:20:30.000 │
└─────────────────────────┘ Query with timezone:

WITH toDateTime64('2020-01-01 10:20:30.999', 3) AS dt64
SELECT toStartOfSecond(dt64, 'Asia/Istanbul') Result:

┌─toStartOfSecond(dt64, 'Asia/Istanbul')─┐
│                2020-01-01 13:20:30.000 │
└────────────────────────────────────────┘
## toStartOfMillisecond [¶](https://www.tinybird.co/docs/about:blank#tostartofmillisecond)

Rounds down a date with time to the start of the milliseconds.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfMillisecond(value, [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime64.
- `timezone`   : Timezone for the returned value (optional). If not specified, the function uses the timezone of the `value`   parameter. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Input value with sub-milliseconds. DateTime64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query without timezone:

WITH toDateTime64('2020-01-01 10:20:30.999999999', 9) AS dt64
SELECT toStartOfMillisecond(dt64) Result:

┌────toStartOfMillisecond(dt64)─┐
│ 2020-01-01 10:20:30.999000000 │
└───────────────────────────────┘ Query with timezone:

┌─toStartOfMillisecond(dt64, 'Asia/Istanbul')─┐
│               2020-01-01 12:20:30.999000000 │
└─────────────────────────────────────────────┘ Result:

┌─toStartOfMillisecond(dt64, 'Asia/Istanbul')─┐
│                     2020-01-01 12:20:30.999 │
└─────────────────────────────────────────────┘
## toStartOfMicrosecond [¶](https://www.tinybird.co/docs/about:blank#tostartofmicrosecond)

Rounds down a date with time to the start of the microseconds.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfMicrosecond(value, [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime64.
- `timezone`   : Timezone for the returned value (optional). If not specified, the function uses the timezone of the `value`   parameter. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Input value with sub-microseconds. DateTime64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query without timezone:

WITH toDateTime64('2020-01-01 10:20:30.999999999', 9) AS dt64
SELECT toStartOfMicrosecond(dt64) Result:

┌────toStartOfMicrosecond(dt64)─┐
│ 2020-01-01 10:20:30.999999000 │
└───────────────────────────────┘ Query with timezone:

WITH toDateTime64('2020-01-01 10:20:30.999999999', 9) AS dt64
SELECT toStartOfMicrosecond(dt64, 'Asia/Istanbul') Result:

┌─toStartOfMicrosecond(dt64, 'Asia/Istanbul')─┐
│               2020-01-01 12:20:30.999999000 │
└─────────────────────────────────────────────┘
## toStartOfNanosecond [¶](https://www.tinybird.co/docs/about:blank#tostartofnanosecond)

Rounds down a date with time to the start of the nanoseconds.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfNanosecond(value, [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Date and time. DateTime64.
- `timezone`   : Timezone for the returned value (optional). If not specified, the function uses the timezone of the `value`   parameter. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Input value with nanoseconds. DateTime64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query without timezone:

WITH toDateTime64('2020-01-01 10:20:30.999999999', 9) AS dt64
SELECT toStartOfNanosecond(dt64) Result:

┌─────toStartOfNanosecond(dt64)─┐
│ 2020-01-01 10:20:30.999999999 │
└───────────────────────────────┘ Query with timezone:

WITH toDateTime64('2020-01-01 10:20:30.999999999', 9) AS dt64
SELECT toStartOfNanosecond(dt64, 'Asia/Istanbul') Result:

┌─toStartOfNanosecond(dt64, 'Asia/Istanbul')─┐
│              2020-01-01 12:20:30.999999999 │
└────────────────────────────────────────────┘
## toStartOfFiveMinutes [¶](https://www.tinybird.co/docs/about:blank#tostartoffiveminutes)

Rounds down a date with time to the start of the five-minute interval.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfFiveMinutes(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a  DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the five-minute interval of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfFiveMinutes(toDateTime('2023-04-21 10:17:00')),
    toStartOfFiveMinutes(toDateTime('2023-04-21 10:20:00')),
    toStartOfFiveMinutes(toDateTime('2023-04-21 10:23:00'))
FORMAT Vertical Result:

Row 1:
──────
toStartOfFiveMinutes(toDateTime('2023-04-21 10:17:00')): 2023-04-21 10:15:00
toStartOfFiveMinutes(toDateTime('2023-04-21 10:20:00')): 2023-04-21 10:20:00
toStartOfFiveMinutes(toDateTime('2023-04-21 10:23:00')): 2023-04-21 10:20:00
## toStartOfTenMinutes [¶](https://www.tinybird.co/docs/about:blank#tostartoftenminutes)

Rounds down a date with time to the start of the ten-minute interval.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfTenMinutes(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a  DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the ten-minute interval of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfTenMinutes(toDateTime('2023-04-21 10:17:00')),
    toStartOfTenMinutes(toDateTime('2023-04-21 10:20:00')),
    toStartOfTenMinutes(toDateTime('2023-04-21 10:23:00'))
FORMAT Vertical Result:

Row 1:
──────
toStartOfTenMinutes(toDateTime('2023-04-21 10:17:00')): 2023-04-21 10:10:00
toStartOfTenMinutes(toDateTime('2023-04-21 10:20:00')): 2023-04-21 10:20:00
toStartOfTenMinutes(toDateTime('2023-04-21 10:23:00')): 2023-04-21 10:20:00
## toStartOfFifteenMinutes [¶](https://www.tinybird.co/docs/about:blank#tostartoffifteenminutes)

Rounds down the date with time to the start of the fifteen-minute interval.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfFifteenMinutes(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   - a  DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The start of the fifteen-minute interval of the given date/time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toStartOfFifteenMinutes(toDateTime('2023-04-21 10:17:00')),
    toStartOfFifteenMinutes(toDateTime('2023-04-21 10:20:00')),
    toStartOfFifteenMinutes(toDateTime('2023-04-21 10:23:00'))
FORMAT Vertical Result:

Row 1:
──────
toStartOfFifteenMinutes(toDateTime('2023-04-21 10:17:00')): 2023-04-21 10:15:00
toStartOfFifteenMinutes(toDateTime('2023-04-21 10:20:00')): 2023-04-21 10:15:00
toStartOfFifteenMinutes(toDateTime('2023-04-21 10:23:00')): 2023-04-21 10:15:00
## toStartOfInterval [¶](https://www.tinybird.co/docs/about:blank#tostartofinterval)

This function generalizes other `toStartOf*()` functions with `toStartOfInterval(date_or_date_with_time, INTERVAL x unit [, time_zone])` syntax. For example,

- `toStartOfInterval(t, INTERVAL 1 YEAR)`   returns the same as `toStartOfYear(t)`  ,
- `toStartOfInterval(t, INTERVAL 1 MONTH)`   returns the same as `toStartOfMonth(t)`  ,
- `toStartOfInterval(t, INTERVAL 1 DAY)`   returns the same as `toStartOfDay(t)`  ,
- `toStartOfInterval(t, INTERVAL 15 MINUTE)`   returns the same as `toStartOfFifteenMinutes(t)`  .

The calculation is performed relative to specific points in time:

| Interval | Start |
| --- | --- |
| YEAR | year 0 |
| QUARTER | 1900 Q1 |
| MONTH | 1900 January |
| WEEK | 1970, 1st week (01-05) |
| DAY | 1970-01-01 |
| HOUR | (*) |
| MINUTE | 1970-01-01 00:00:00 |
| SECOND | 1970-01-01 00:00:00 |
| MILLISECOND | 1970-01-01 00:00:00 |
| MICROSECOND | 1970-01-01 00:00:00 |
| NANOSECOND | 1970-01-01 00:00:00 |

(*) hour intervals are special: the calculation is always performed relative to 00:00:00 (midnight) of the current day. As a result, only hour values between 1 and 23 are useful.

If unit `WEEK` was specified, `toStartOfInterval` assumes that weeks start on Monday. Note that this behavior is different from that of function `toStartOfWeek` in which weeks start by default on Sunday.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toStartOfInterval(value, INTERVAL x unit[, time_zone])
toStartOfInterval(value, INTERVAL x unit[, origin[, time_zone]]) Aliases: `time_bucket`, `date_bin`.

The second overload emulates TimescaleDB's `time_bucket()` function, respectively PostgreSQL's `date_bin()` function, e.g.

SELECT toStartOfInterval(toDateTime('2023-01-01 14:45:00'), INTERVAL 1 MINUTE, toDateTime('2023-01-01 14:35:30')) Result:

┌───toStartOfInterval(...)─┐
│      2023-01-01 14:44:30 │
└─────��────────────────────┘
## toTime [¶](https://www.tinybird.co/docs/about:blank#totime)

Converts a date with time to a certain fixed date, while preserving the time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toTime(date[,timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date to convert to a time. Date/DateTime/DateTime64.
- `timezone`   (optional): Timezone for the returned value. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- DateTime with date equated to `1970-01-02`   while preserving the time. DateTime.

If the `date` input argument contained sub-second components, they will be dropped in the returned `DateTime` value with second-accuracy.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT toTime(toDateTime64('1970-12-10 01:20:30.3000',3)) AS result, toTypeName(result) Result:

┌──────────────result─┬─toTypeName(result)─┐
│ 1970-01-02 01:20:30 │ DateTime           │
└─────────────────────┴────────────────────┘
## toRelativeYearNum [¶](https://www.tinybird.co/docs/about:blank#torelativeyearnum)

Converts a date, or date with time, to the number of years elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeYearNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of years from a fixed reference point in the past. UInt16.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    toRelativeYearNum(toDate('2002-12-08')) AS y1,
    toRelativeYearNum(toDate('2010-10-26')) AS y2 Result:

┌───y1─┬───y2─┐
│ 2002 │ 2010 │
└──────┴──────┘
## toRelativeQuarterNum [¶](https://www.tinybird.co/docs/about:blank#torelativequarternum)

Converts a date, or date with time, to the number of quarters elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeQuarterNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of quarters from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeQuarterNum(toDate('1993-11-25')) AS q1,
  toRelativeQuarterNum(toDate('2005-01-05')) AS q2 Result:

┌───q1─┬───q2─┐
│ 7975 │ 8020 │
└──────┴──────┘
## toRelativeMonthNum [¶](https://www.tinybird.co/docs/about:blank#torelativemonthnum)

Converts a date, or date with time, to the number of months elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeMonthNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of months from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeMonthNum(toDate('2001-04-25')) AS m1,
  toRelativeMonthNum(toDate('2009-07-08')) AS m2 Result:

┌────m1─┬────m2─┐
│ 24016 │ 24115 │
└───────┴───────┘
## toRelativeWeekNum [¶](https://www.tinybird.co/docs/about:blank#torelativeweeknum)

Converts a date, or date with time, to the number of weeks elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeWeekNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of weeks from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeWeekNum(toDate('2000-02-29')) AS w1,
  toRelativeWeekNum(toDate('2001-01-12')) AS w2 Result:

┌───w1─┬───w2─┐
│ 1574 │ 1619 │
└──────┴──────┘
## toRelativeDayNum [¶](https://www.tinybird.co/docs/about:blank#torelativedaynum)

Converts a date, or date with time, to the number of days elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeDayNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of days from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeDayNum(toDate('1993-10-05')) AS d1,
  toRelativeDayNum(toDate('2000-09-20')) AS d2 Result:

┌───d1─┬────d2─┐
│ 8678 │ 11220 │
└──────┴───────┘
## toRelativeHourNum [¶](https://www.tinybird.co/docs/about:blank#torelativehournum)

Converts a date, or date with time, to the number of hours elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeHourNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of hours from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeHourNum(toDateTime('1993-10-05 05:20:36')) AS h1,
  toRelativeHourNum(toDateTime('2000-09-20 14:11:29')) AS h2 Result:

┌─────h1─┬─────h2─┐
│ 208276 │ 269292 │
└────────┴────────┘
## toRelativeMinuteNum [¶](https://www.tinybird.co/docs/about:blank#torelativeminutenum)

Converts a date, or date with time, to the number of minutes elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeMinuteNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of minutes from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeMinuteNum(toDateTime('1993-10-05 05:20:36')) AS m1,
  toRelativeMinuteNum(toDateTime('2000-09-20 14:11:29')) AS m2 Result:

┌───────m1─┬───────m2─┐
│ 12496580 │ 16157531 │
└──────────┴──────────┘
## toRelativeSecondNum [¶](https://www.tinybird.co/docs/about:blank#torelativesecondnum)

Converts a date, or date with time, to the number of the seconds elapsed since a certain fixed point in the past.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toRelativeSecondNum(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date/DateTime/DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The number of seconds from a fixed reference point in the past. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toRelativeSecondNum(toDateTime('1993-10-05 05:20:36')) AS s1,
  toRelativeSecondNum(toDateTime('2000-09-20 14:11:29')) AS s2 Result:

┌────────s1─┬────────s2─┐
│ 749794836 │ 969451889 │
└───────────┴───────────┘
## toISOYear [¶](https://www.tinybird.co/docs/about:blank#toisoyear)

Converts a date, or date with time, to the ISO year as a UInt16 number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toISOYear(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : The value with date or date with time. Date, Date32, DateTime or DateTime64

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The input value converted to a ISO year number. UInt16.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toISOYear(toDate('2024/10/02')) as year1,
  toISOYear(toDateTime('2024-10-02 01:30:00')) as year2 Result:

┌─year1─┬─year2─┐
│  2024 │  2024 │
└───────┴───────┘
## toISOWeek [¶](https://www.tinybird.co/docs/about:blank#toisoweek)

Converts a date, or date with time, to a UInt8 number containing the ISO Week number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toISOWeek(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : The value with date or date with time.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `value`   converted to the current ISO week number. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
  toISOWeek(toDate('2024/10/02')) AS week1,
  toISOWeek(toDateTime('2024/10/02 01:30:00')) AS week2 Response:

┌─week1─┬─week2─┐
│    40 │    40 │
└───────┴───────┘
## toWeek [¶](https://www.tinybird.co/docs/about:blank#toweek)

This function returns the week number for date or datetime. The two-argument form of `toWeek()` enables you to specify whether the week starts on Sunday or Monday and whether the return value should be in the range from 0 to 53 or from 1 to 53. If the mode argument is omitted, the default mode is 0.

`toISOWeek()` is a compatibility function that is equivalent to `toWeek(date,3)`.

The following table describes how the mode argument works.

| Mode | First day of week | Range | Week 1 is the first week ... |
| --- | --- | --- | --- |
| 0 | Sunday | 0-53 | with a Sunday in this year |
| 1 | Monday | 0-53 | with 4 or more days this year |
| 2 | Sunday | 1-53 | with a Sunday in this year |
| 3 | Monday | 1-53 | with 4 or more days this year |
| 4 | Sunday | 0-53 | with 4 or more days this year |
| 5 | Monday | 0-53 | with a Monday in this year |
| 6 | Sunday | 1-53 | with 4 or more days this year |
| 7 | Monday | 1-53 | with a Monday in this year |
| 8 | Sunday | 1-53 | contains January 1 |
| 9 | Monday | 1-53 | contains January 1 |

For mode values with a meaning of "with 4 or more days this year," weeks are numbered according to ISO 8601:1988:

- If the week containing January 1 has 4 or more days in the new year, it's week 1.
- Otherwise, it's the last week of the previous year, and the next week is week 1.

For mode values with a meaning of "contains January 1" weeks contain January 1 is week 1. It doesn't matter how many days in the new year the week contained, even if it contained only one day. I.e. if the last week of December contains January 1 of the next year, it will be week 1 of the next year.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toWeek(t[, mode[, time_zone]]) Alias: `WEEK`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `t`   – Date or DateTime.
- `mode`   – Optional parameter, Range of values is [0,9], default is 0.
- `Timezone`   – Optional parameter, it behaves like any other conversion function.

The first argument can also be specified as String in a format supported by parseDateTime64BestEffort(). Support for string arguments exists only for reasons of compatibility with MySQL which is expected by certain 3rd party tools. As string argument support may in future be made dependent on new MySQL-compatibility settings and because string parsing is generally slow, it's recommended to not use it.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDate('2016-12-27') AS date, toWeek(date) AS week0, toWeek(date,1) AS week1, toWeek(date,9) AS week9 ┌───────date─┬─week0─┬─week1─┬─week9─┐
│ 2016-12-27 │    52 │    52 │     1 │
└────────────┴───────┴───────┴───────┘
## toYearWeek [¶](https://www.tinybird.co/docs/about:blank#toyearweek)

Returns year and week for a date. The year in the result may be different from the year in the date argument for the first and the last week of the year.

The mode argument works like the mode argument to `toWeek()` . For the single-argument syntax, a mode value of 0 is used.

`toISOYear()` is a compatibility function that is equivalent to `intDiv(toYearWeek(date,3),100)`.

The week number returned by `toYearWeek()` can be different from what the `toWeek()` returns. `toWeek()` always returns week number in the context of the given year, and in case `toWeek()` returns `0`, `toYearWeek()` returns the value corresponding to the last week of previous year.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toYearWeek(t[, mode[, timezone]]) Alias: `YEARWEEK`

The first argument can also be specified as String in a format supported by parseDateTime64BestEffort(). Support for string arguments exists only for reasons of compatibility with MySQL which is expected by certain 3rd party tools. As string argument support may in future be made dependent on new MySQL-compatibility settings and because string parsing is generally slow, it's recommended to not use it.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDate('2016-12-27') AS date, toYearWeek(date) AS yearWeek0, toYearWeek(date,1) AS yearWeek1, toYearWeek(date,9) AS yearWeek9, toYearWeek(toDate('2022-01-01')) AS prev_yearWeek ┌───────date─┬─yearWeek0─┬─yearWeek1─┬─yearWeek9─┬─prev_yearWeek─┐
│ 2016-12-27 │    201652 │    201652 │    201701 │        202152 │
└────────────┴───────────┴───────────┴───────────┴───────────────┘
## toDaysSinceYearZero [¶](https://www.tinybird.co/docs/about:blank#todayssinceyearzero)

Returns for a given date, the number of days passed since 1 January 0000 in the proleptic Gregorian calendar defined by ISO 8601. The calculation is the same as in MySQL's `TO_DAYS()` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toDaysSinceYearZero(date[, time_zone]) Alias: `TO_DAYS`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : The date to calculate the number of days passed since year zero from. Date, Date32, DateTime or DateTime64.
- `time_zone`   : A String type const value or an expression represent the time zone. String types

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The number of days passed since date 0000-01-01. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toDaysSinceYearZero(toDate('2023-09-08')) Result:

┌─toDaysSinceYearZero(toDate('2023-09-08')))─┐
│                                     713569 │
└────────────────────────────────────────────┘
## fromDaysSinceYearZero [¶](https://www.tinybird.co/docs/about:blank#fromdayssinceyearzero)

Returns for a given number of days passed since 1 January 0000 the corresponding date in the proleptic Gregorian calendar defined by ISO 8601. The calculation is the same as in MySQL's `FROM_DAYS()` function.

The result is undefined if it can't be represented within the bounds of the Date type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

fromDaysSinceYearZero(days) Alias: `FROM_DAYS`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `days`   : The number of days passed since year zero.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The date corresponding to the number of days passed since year zero. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromDaysSinceYearZero(739136), fromDaysSinceYearZero(toDaysSinceYearZero(toDate('2023-09-08'))) Result:

┌─fromDaysSinceYearZero(739136)─┬─fromDaysSinceYearZero(toDaysSinceYearZero(toDate('2023-09-08')))─┐
│                    2023-09-08 │                                                       2023-09-08 │
└───────────────────────────────┴──────────────────────────────────────────────────────────────────┘
## fromDaysSinceYearZero32 [¶](https://www.tinybird.co/docs/about:blank#fromdayssinceyearzero32)

Like fromDaysSinceYearZero but returns a Date32.

## age [¶](https://www.tinybird.co/docs/about:blank#age)

Returns the `unit` component of the difference between `startdate` and `enddate` . The difference is calculated using a precision of 1 nanosecond. E.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for `day` unit, 0 months for `month` unit, 0 years for `year` unit.

For an alternative to `age` , see function `date_diff`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

age('unit', startdate, enddate, [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval for result. String.   Possible values:  

  - `nanosecond`    , `nanoseconds`    , `ns`
  - `microsecond`    , `microseconds`    , `us`    , `u`
  - `millisecond`    , `milliseconds`    , `ms`
  - `second`    , `seconds`    , `ss`    , `s`
  - `minute`    , `minutes`    , `mi`    , `n`
  - `hour`    , `hours`    , `hh`    , `h`
  - `day`    , `days`    , `dd`    , `d`
  - `week`    , `weeks`    , `wk`    , `ww`
  - `month`    , `months`    , `mm`    , `m`
  - `quarter`    , `quarters`    , `qq`    , `q`
  - `year`    , `years`    , `yyyy`    , `yy`
- `startdate`   : The first time value to subtract (the subtrahend). Date, Date32, DateTime or DateTime64.
- `enddate`   : The second time value to subtract from (the minuend). Date, Date32, DateTime or DateTime64.
- `timezone`   : Timezone name (optional). If specified, it's applied to both `startdate`   and `enddate`   . If not specified, timezones of `startdate`   and `enddate`   are used. If they aren't the same, the result is unspecified. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Difference between `enddate` and `startdate` expressed in `unit` . Int.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00')) Result:

┌─age('hour', toDateTime('2018-01-01 22:30:00'), toDateTime('2018-01-02 23:00:00'))─┐
│                                                                                24 │
└───────────────────────────────────────────────────────────────────────────────────┘ SELECT
    toDate('2022-01-01') AS e,
    toDate('2021-12-29') AS s,
    age('day', s, e) AS day_age,
    age('month', s, e) AS month__age,
    age('year', s, e) AS year_age Result:

┌──────────e─┬──────────s─┬─day_age─┬─month__age─┬─year_age─┐
│ 2022-01-01 │ 2021-12-29 │       3 │          0 │        0 │
└────────────┴─��──────────┴─────────┴────────────┴──────────┘
## date_diff [¶](https://www.tinybird.co/docs/about:blank#date-diff)

Returns the count of the specified `unit` boundaries crossed between the `startdate` and the `enddate`. The difference is calculated using relative units, e.g. the difference between `2021-12-29` and `2022-01-01` is 3 days for unit `day` (see toRelativeDayNum), 1 month for unit `month` (see toRelativeMonthNum) and 1 year for unit `year` (see toRelativeYearNum).

If unit `week` was specified, `date_diff` assumes that weeks start on Monday. Note that this behavior is different from that of function `toWeek()` in which weeks start by default on Sunday.

For an alternative to `date_diff` , see function `age`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

date_diff('unit', startdate, enddate, [timezone]) Aliases: `dateDiff`, `DATE_DIFF`, `timestampDiff`, `timestamp_diff`, `TIMESTAMP_DIFF`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval for result. String.   Possible values:  

  - `nanosecond`    , `nanoseconds`    , `ns`
  - `microsecond`    , `microseconds`    , `us`    , `u`
  - `millisecond`    , `milliseconds`    , `ms`
  - `second`    , `seconds`    , `ss`    , `s`
  - `minute`    , `minutes`    , `mi`    , `n`
  - `hour`    , `hours`    , `hh`    , `h`
  - `day`    , `days`    , `dd`    , `d`
  - `week`    , `weeks`    , `wk`    , `ww`
  - `month`    , `months`    , `mm`    , `m`
  - `quarter`    , `quarters`    , `qq`    , `q`
  - `year`    , `years`    , `yyyy`    , `yy`
- `startdate`   : The first time value to subtract (the subtrahend). Date, Date32, DateTime or DateTime64.
- `enddate`   : The second time value to subtract from (the minuend). Date, Date32, DateTime or DateTime64.
- `timezone`   : Timezone name (optional). If specified, it's applied to both `startdate`   and `enddate`   . If not specified, timezones of `startdate`   and `enddate`   are used. If they aren't the same, the result is unspecified. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Difference between `enddate` and `startdate` expressed in `unit` . Int.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00')) Result:

┌─dateDiff('hour', toDateTime('2018-01-01 22:00:00'), toDateTime('2018-01-02 23:00:00'))─┐
│                                                                                     25 │
└────────────────────────────────────────────────────────────────────────────────────────┘ SELECT
    toDate('2022-01-01') AS e,
    toDate('2021-12-29') AS s,
    dateDiff('day', s, e) AS day_diff,
    dateDiff('month', s, e) AS month__diff,
    dateDiff('year', s, e) AS year_diff Result:

┌──────────e─┬──────────s─┬─day_diff─┬─month__diff─┬─year_diff─┐
│ 2022-01-01 │ 2021-12-29 │        3 │           1 │         1 │
└────────────┴────────────┴──────────┴─────────────┴───────────┘
## date_trunc [¶](https://www.tinybird.co/docs/about:blank#date-trunc)

Truncates date and time data to the specified part of date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

date_trunc(unit, value[, timezone]) Alias: `dateTrunc`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval to truncate the result. String Literal.   Possible values:  

  - `nanosecond`     - Compatible only with DateTime64
  - `microsecond`     - Compatible only with DateTime64
  - `milisecond`     - Compatible only with DateTime64
  - `second`
  - `minute`
  - `hour`
  - `day`
  - `week`
  - `month`
  - `quarter`
  - `year`

`unit`   argument is case-insensitive.
- `value`   : Date and time. Date, Date32, DateTime or DateTime64.
- `timezone`   : Timezone name for the returned value (optional). If not specified, the function uses the timezone of the `value`   parameter. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Value, truncated to the specified part of date. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query without timezone:

SELECT now(), date_trunc('hour', now()) Result:

┌───────────────now()─┬─date_trunc('hour', now())─┐
│ 2020-09-28 10:40:45 │       2020-09-28 10:00:00 │
└─────────────────────┴───────────────────────────┘ Query with the specified timezone:

SELECT now(), date_trunc('hour', now(), 'Asia/Istanbul') Result:

┌───────────────now()─┬─date_trunc('hour', now(), 'Asia/Istanbul')─┐
│ 2020-09-28 10:46:26 │                        2020-09-28 13:00:00 │
└─────────────────────┴────────────────────────────────────────────┘
## date_add [¶](https://www.tinybird.co/docs/about:blank#date-add)

Adds the time interval or date interval to the provided date or date with time.

If the addition results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

date_add(unit, value, date) Alternative syntax:

date_add(date, INTERVAL value unit) Aliases: `dateAdd`, `DATE_ADD`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval to add. Note: This isn't a String and must therefore not be quoted.   Possible values:  

  - `second`
  - `minute`
  - `hour`
  - `day`
  - `week`
  - `month`
  - `quarter`
  - `year`
- `value`   : Value of interval to add. Int.
- `date`   : The date or date with time to which `value`   is added. Date, Date32, DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time obtained by adding `value` , expressed in `unit` , to `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT date_add(YEAR, 3, toDate('2018-01-01')) Result:

┌─plus(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                    2021-01-01 │
└───────────────────────────────────────────────┘ SELECT date_add(toDate('2018-01-01'), INTERVAL 3 YEAR) Result:

┌─plus(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                    2021-01-01 │
└───────────────────────────────────────────────┘
## date_sub [¶](https://www.tinybird.co/docs/about:blank#date-sub)

Subtracts the time interval or date interval from the provided date or date with time.

If the subtraction results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

date_sub(unit, value, date) Alternative syntax:

date_sub(date, INTERVAL value unit) Aliases: `dateSub`, `DATE_SUB`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval to subtract. Note: This isn't a String and must therefore not be quoted.  

  Possible values:  

  - `second`
  - `minute`
  - `hour`
  - `day`
  - `week`
  - `month`
  - `quarter`
  - `year`
- `value`   : Value of interval to subtract. Int.
- `date`   : The date or date with time from which `value`   is subtracted. Date, Date32, DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time obtained by subtracting `value` , expressed in `unit` , from `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT date_sub(YEAR, 3, toDate('2018-01-01')) Result:

┌─minus(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                     2015-01-01 │
└────────────────────────────────────────────────┘ SELECT date_sub(toDate('2018-01-01'), INTERVAL 3 YEAR) Result:

┌─minus(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                     2015-01-01 │
└────────────────────────────────────────────────┘
## timestamp_add [¶](https://www.tinybird.co/docs/about:blank#timestamp-add)

Adds the specified time value with the provided date or date time value.

If the addition results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timestamp_add(date, INTERVAL value unit) Aliases: `timeStampAdd`, `TIMESTAMP_ADD`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date, Date32, DateTime or DateTime64.
- `value`   : Value of interval to add. Int.
- `unit`   : The type of interval to add. String.   Possible values:  

  - `second`
  - `minute`
  - `hour`
  - `day`
  - `week`
  - `month`
  - `quarter`
  - `year`

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time with the specified `value` expressed in `unit` added to `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select timestamp_add(toDate('2018-01-01'), INTERVAL 3 MONTH) Result:

┌─plus(toDate('2018-01-01'), toIntervalMonth(3))─┐
│                                     2018-04-01 │
└────────────────────────────────────────────────┘
## timestamp_sub [¶](https://www.tinybird.co/docs/about:blank#timestamp-sub)

Subtracts the time interval from the provided date or date with time.

If the subtraction results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timestamp_sub(unit, value, date) Aliases: `timeStampSub`, `TIMESTAMP_SUB`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `unit`   : The type of interval to subtract. String.   Possible values:  

  - `second`
  - `minute`
  - `hour`
  - `day`
  - `week`
  - `month`
  - `quarter`
  - `year`
- `value`   : Value of interval to subtract. Int.
- `date`   : Date or date with time. Date, Date32, DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time obtained by subtracting `value` , expressed in `unit` , from `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

select timestamp_sub(MONTH, 5, toDateTime('2018-12-18 01:02:03')) Result:

┌─minus(toDateTime('2018-12-18 01:02:03'), toIntervalMonth(5))─┐
│                                          2018-07-18 01:02:03 │
└──────────────────────────────────────────────────────────────┘
## addDate [¶](https://www.tinybird.co/docs/about:blank#adddate)

Adds the time interval to the provided date, date with time or String-encoded date / date with time.

If the addition results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addDate(date, interval)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : The date or date with time to which `interval`   is added. Date, Date32, DateTime, DateTime64, or String
- `interval`   : Interval to add. Interval.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time obtained by adding `interval` to `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT addDate(toDate('2018-01-01'), INTERVAL 3 YEAR) Result:

┌─addDate(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                       2021-01-01 │
└──────────────────────────────────────────────────┘ Alias: `ADDDATE`

## subDate [¶](https://www.tinybird.co/docs/about:blank#subdate)

Subtracts the time interval from the provided date, date with time or String-encoded date / date with time.

If the subtraction results in a value outside the bounds of the data type, the result is undefined.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subDate(date, interval)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : The date or date with time from which `interval`   is subtracted. Date, Date32, DateTime, DateTime64, or String
- `interval`   : Interval to subtract. Interval.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Date or date with time obtained by subtracting `interval` from `date` . Date, Date32, DateTime or DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT subDate(toDate('2018-01-01'), INTERVAL 3 YEAR) Result:

┌─subDate(toDate('2018-01-01'), toIntervalYear(3))─┐
│                                       2015-01-01 │
└──────────────────────────────────────────────────┘ Alias: `SUBDATE`

## now [¶](https://www.tinybird.co/docs/about:blank#now)

Returns the current date and time at the moment of query analysis. The function is a constant expression.

Alias: `current_timestamp`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

now([timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timezone`   : Timezone name for the returned value (optional). String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Current date and time. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query without timezone:

SELECT now() Result:

┌───────────────now()─┐
│ 2020-10-17 07:42:09 │
└─────────────────────┘ Query with the specified timezone:

SELECT now('Asia/Istanbul') Result:

┌─now('Asia/Istanbul')─┐
│  2020-10-17 10:42:23 │
└──────────────────────┘
## now64 [¶](https://www.tinybird.co/docs/about:blank#now64)

Returns the current date and time with sub-second precision at the moment of query analysis. The function is a constant expression.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

now64([scale], [timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `scale`   - Tick size (precision): 10-precision seconds. Valid range: [ 0 : 9 ]. Typically, are used - 3 (default) (milliseconds), 6 (microseconds), 9 (nanoseconds).
- `timezone`   : Timezone name for the returned value (optional). String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Current date and time with sub-second precision. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT now64(), now64(9, 'Asia/Istanbul') Result:

┌─────────────────now64()─┬─────now64(9, 'Asia/Istanbul')─┐
│ 2022-08-21 19:34:26.196 │ 2022-08-21 22:34:26.196542766 │
└─────────────────────────┴───────────────────────────────┘
## nowInBlock [¶](https://www.tinybird.co/docs/about:blank#nowinblock)

Returns the current date and time at the moment of processing of each block of data. In contrast to the function now, it's not a constant expression, and the returned value will be different in different blocks for long-running queries.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

nowInBlock([timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `timezone`   : Timezone name for the returned value (optional). String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Current date and time at the moment of processing of each block of data. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    now(),
    nowInBlock(),
    sleep(1)
FROM numbers(3)
SETTINGS max_block_size = 1
FORMAT PrettyCompactMonoBlock Result:

┌───────────────now()─┬────────nowInBlock()─┬─sleep(1)─┐
│ 2022-08-21 19:41:19 │ 2022-08-21 19:41:19 │        0 │
│ 2022-08-21 19:41:19 │ 2022-08-21 19:41:20 │        0 │
│ 2022-08-21 19:41:19 │ 2022-08-21 19:41:21 │        0 │
└─────────────────────┴─────────────────────┴──────────┘
## today [¶](https://www.tinybird.co/docs/about:blank#today)

Returns the current date at moment of query analysis. It is the same as 'toDate(now())' and has aliases: `curdate`, `current_date`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

today()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- None

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Current date. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT today() AS today, curdate() AS curdate, current_date() AS current_date FORMAT Pretty
### Result [¶](https://www.tinybird.co/docs/about:blank#result)

Running the query above on the 3rd of March 2024 would have returned the following response:

┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃      today ┃    curdate ┃ current_date ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ 2024-03-03 │ 2024-03-03 │   2024-03-03 │
└────────────┴────────────┴──────────────┘
## yesterday [¶](https://www.tinybird.co/docs/about:blank#yesterday)

Accepts zero arguments and returns yesterday's date at one of the moments of query analysis. The same as 'today() - 1'.

## timeSlot [¶](https://www.tinybird.co/docs/about:blank#timeslot)

Round the time to the start of a half-an-hour length interval.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeSlot(time[, time_zone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `time`   : Time to round to the start of a half-an-hour length interval. DateTime/Date32/DateTime64.
- `time_zone`   : A String type const value or an expression representing the time zone. String.

Though this function can take values of the extended types `Date32` and `DateTime64` as an argument, passing it a time outside the normal range (year 1970 to 2149 for `Date` / 2106 for `DateTime` ) will produce wrong results.

### Return type [¶](https://www.tinybird.co/docs/about:blank#return-type)

- Returns the time rounded to the start of a half-an-hour length interval. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT timeSlot(toDateTime('2000-01-02 03:04:05', 'UTC')) Result:

┌─timeSlot(toDateTime('2000-01-02 03:04:05', 'UTC'))─┐
│                                2000-01-02 03:00:00 │
└────────────────────────────────────────────────────┘
## toYYYYMM [¶](https://www.tinybird.co/docs/about:blank#toyyyymm)

Converts a date or date with time to a UInt32 number containing the year and month number (YYYY * 100 + MM). Accepts a second optional timezone argument. If provided, the timezone must be a string constant.

This function is the opposite of function `YYYYMMDDToDate()`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    toYYYYMM(now(), 'US/Eastern') Result:

┌─toYYYYMM(now(), 'US/Eastern')─┐
│                        202303 │
└───────────────────────────────┘
## toYYYYMMDD [¶](https://www.tinybird.co/docs/about:blank#toyyyymmdd)

Converts a date or date with time to a UInt32 number containing the year and month number (YYYY * 10000 + MM * 100 + DD). Accepts a second optional timezone argument. If provided, the timezone must be a string constant.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toYYYYMMDD(now(), 'US/Eastern') Result:

┌─toYYYYMMDD(now(), 'US/Eastern')─┐
│                        20230302 │
└─────────────────────────────────┘
## toYYYYMMDDhhmmss [¶](https://www.tinybird.co/docs/about:blank#toyyyymmddhhmmss)

Converts a date or date with time to a UInt64 number containing the year and month number (YYYY * 10000000000 + MM * 100000000 + DD * 1000000 + hh * 10000 + mm * 100 + ss). Accepts a second optional timezone argument. If provided, the timezone must be a string constant.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toYYYYMMDDhhmmss(now(), 'US/Eastern') Result:

┌─toYYYYMMDDhhmmss(now(), 'US/Eastern')─┐
│                        20230302112209 │
└───────────────────────────────────────┘
## YYYYMMDDToDate [¶](https://www.tinybird.co/docs/about:blank#yyyymmddtodate)

Converts a number containing the year, month and day number to a Date.

This function is the opposite of function `toYYYYMMDD()`.

The output is undefined if the input doesn't encode a valid Date value.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

YYYYMMDDToDate(yyyymmdd)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `yyyymmdd`   - A number representing the year, month and day. Integer, Float or Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a date created from the arguments. Date.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT YYYYMMDDToDate(20230911) Result:

┌─toYYYYMMDD(20230911)─┐
│           2023-09-11 │
└──────────────────────┘
## YYYYMMDDToDate32 [¶](https://www.tinybird.co/docs/about:blank#yyyymmddtodate32)

Like function `YYYYMMDDToDate()` but produces a Date32.

## YYYYMMDDhhmmssToDateTime [¶](https://www.tinybird.co/docs/about:blank#yyyymmddhhmmsstodatetime)

Converts a number containing the year, month, day, hours, minute and second number to a DateTime.

The output is undefined if the input doesn't encode a valid DateTime value.

This function is the opposite of function `toYYYYMMDDhhmmss()`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

YYYYMMDDhhmmssToDateTime(yyyymmddhhmmss[, timezone])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `yyyymmddhhmmss`   - A number representing the year, month and day. Integer, Float or Decimal.
- `timezone`   - Timezone for the returned value (optional).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a date with time created from the arguments. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT YYYYMMDDToDateTime(20230911131415) Result:

┌──────YYYYMMDDhhmmssToDateTime(20230911131415)─┐
│                           2023-09-11 13:14:15 │
└───────────────────────────────────────────────┘
## YYYYMMDDhhmmssToDateTime64 [¶](https://www.tinybird.co/docs/about:blank#yyyymmddhhmmsstodatetime64)

Like function `YYYYMMDDhhmmssToDate()` but produces a DateTime64.

Accepts an additional, optional `precision` parameter after the `timezone` parameter.

## changeYear [¶](https://www.tinybird.co/docs/about:blank#changeyear)

Changes the year component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeYear(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the year. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The same type as `date_or_datetime`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeYear(toDate('1999-01-01'), 2000), changeYear(toDateTime64('1999-01-01 00:00:00.000', 3), 2000) Result:

┌─changeYear(toDate('1999-01-01'), 2000)─┬─changeYear(toDateTime64('1999-01-01 00:00:00.000', 3), 2000)─┐
│                             2000-01-01 │                                      2000-01-01 00:00:00.000 │
└────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘
## changeMonth [¶](https://www.tinybird.co/docs/about:blank#changemonth)

Changes the month component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeMonth(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the month. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value of same type as `date_or_datetime`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeMonth(toDate('1999-01-01'), 2), changeMonth(toDateTime64('1999-01-01 00:00:00.000', 3), 2) Result:

┌─changeMonth(toDate('1999-01-01'), 2)─┬─changeMonth(toDateTime64('1999-01-01 00:00:00.000', 3), 2)─┐
│                           1999-02-01 │                                    1999-02-01 00:00:00.000 │
└──────────────────────────────────────┴────────────────────────────────────────────────────────────┘
## changeDay [¶](https://www.tinybird.co/docs/about:blank#changeday)

Changes the day component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeDay(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the day. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value of same type as `date_or_datetime`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeDay(toDate('1999-01-01'), 5), changeDay(toDateTime64('1999-01-01 00:00:00.000', 3), 5) Result:

┌─changeDay(toDate('1999-01-01'), 5)─┬─changeDay(toDateTime64('1999-01-01 00:00:00.000', 3), 5)─┐
│                         1999-01-05 │                                  1999-01-05 00:00:00.000 │
└────────────────────────────────────┴──────────────────────────────────────────────────────────┘
## changeHour [¶](https://www.tinybird.co/docs/about:blank#changehour)

Changes the hour component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeHour(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the hour. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value of same type as `date_or_datetime`   . If the input is a Date, return DateTime. If the input is a Date32, return DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeHour(toDate('1999-01-01'), 14), changeHour(toDateTime64('1999-01-01 00:00:00.000', 3), 14) Result:

┌─changeHour(toDate('1999-01-01'), 14)─┬─changeHour(toDateTime64('1999-01-01 00:00:00.000', 3), 14)─┐
│                  1999-01-01 14:00:00 │                                    1999-01-01 14:00:00.000 │
└──────────────────────────────────────┴────────────────────────────────────────────────────────────┘
## changeMinute [¶](https://www.tinybird.co/docs/about:blank#changeminute)

Changes the minute component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeMinute(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the minute. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value of same type as `date_or_datetime`   . If the input is a Date, return DateTime. If the input is a Date32, return DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeMinute(toDate('1999-01-01'), 15), changeMinute(toDateTime64('1999-01-01 00:00:00.000', 3), 15) Result:

┌─changeMinute(toDate('1999-01-01'), 15)─┬─changeMinute(toDateTime64('1999-01-01 00:00:00.000', 3), 15)─┐
│                    1999-01-01 00:15:00 │                                      1999-01-01 00:15:00.000 │
└────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘
## changeSecond [¶](https://www.tinybird.co/docs/about:blank#changesecond)

Changes the second component of a date or date time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

changeSecond(date_or_datetime, value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_or_datetime`   - a Date, Date32, DateTime or DateTime64
- `value`   - a new value of the second. Integer.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a value of same type as `date_or_datetime`   . If the input is a Date, return DateTime. If the input is a Date32, return DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT changeSecond(toDate('1999-01-01'), 15), changeSecond(toDateTime64('1999-01-01 00:00:00.000', 3), 15) Result:

┌─changeSecond(toDate('1999-01-01'), 15)─┬─changeSecond(toDateTime64('1999-01-01 00:00:00.000', 3), 15)─┐
│                    1999-01-01 00:00:15 │                                      1999-01-01 00:00:15.000 │
└────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘
## addYears [¶](https://www.tinybird.co/docs/about:blank#addyears)

Adds a specified number of years to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addYears(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of years to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of years to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   years. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addYears(date, 1) AS add_years_with_date,
    addYears(date_time, 1) AS add_years_with_date_time,
    addYears(date_time_string, 1) AS add_years_with_date_time_string ┌─add_years_with_date─┬─add_years_with_date_time─┬─add_years_with_date_time_string─┐
│          2025-01-01 │      2025-01-01 00:00:00 │         2025-01-01 00:00:00.000 │
└─────────────────────┴──────────────────────────┴─────────────────────────────────┘
## addQuarters [¶](https://www.tinybird.co/docs/about:blank#addquarters)

Adds a specified number of quarters to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addQuarters(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of quarters to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of quarters to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   quarters. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addQuarters(date, 1) AS add_quarters_with_date,
    addQuarters(date_time, 1) AS add_quarters_with_date_time,
    addQuarters(date_time_string, 1) AS add_quarters_with_date_time_string ┌─add_quarters_with_date─┬─add_quarters_with_date_time─┬─add_quarters_with_date_time_string─┐
│             2024-04-01 │         2024-04-01 00:00:00 │            2024-04-01 00:00:00.000 │
└────────────────────────┴─────────────────────────────┴────────────────────────────────────┘
## addMonths [¶](https://www.tinybird.co/docs/about:blank#addmonths)

Adds a specified number of months to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addMonths(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of months to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of months to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   months. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addMonths(date, 6) AS add_months_with_date,
    addMonths(date_time, 6) AS add_months_with_date_time,
    addMonths(date_time_string, 6) AS add_months_with_date_time_string ┌─add_months_with_date─┬─add_months_with_date_time─┬─add_months_with_date_time_string─┐
│           2024-07-01 │       2024-07-01 00:00:00 │          2024-07-01 00:00:00.000 │
└──────────────────────┴───────────────────────────┴──────────────────────────────────┘
## addWeeks [¶](https://www.tinybird.co/docs/about:blank#addweeks)

Adds a specified number of weeks to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addWeeks(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of weeks to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of weeks to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   weeks. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addWeeks(date, 5) AS add_weeks_with_date,
    addWeeks(date_time, 5) AS add_weeks_with_date_time,
    addWeeks(date_time_string, 5) AS add_weeks_with_date_time_string ┌─add_weeks_with_date─┬─add_weeks_with_date_time─┬─add_weeks_with_date_time_string─┐
│          2024-02-05 │      2024-02-05 00:00:00 │         2024-02-05 00:00:00.000 │
└─────────────────────┴──────────────────────────┴─────────────────────────────────┘
## addDays [¶](https://www.tinybird.co/docs/about:blank#adddays)

Adds a specified number of days to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addDays(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of days to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of days to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   days. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addDays(date, 5) AS add_days_with_date,
    addDays(date_time, 5) AS add_days_with_date_time,
    addDays(date_time_string, 5) AS add_days_with_date_time_string ┌─add_days_with_date─┬─add_days_with_date_time─┬─add_days_with_date_time_string─┐
│         2024-01-06 │     2024-01-06 00:00:00 │        2024-01-06 00:00:00.000 │
└────────────────────┴─────────────────────────┴────────────────────────────────┘
## addHours [¶](https://www.tinybird.co/docs/about:blank#addhours)

Adds a specified number of days to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addHours(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of hours to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of hours to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

o

- Returns `date`   plus `num`   hours. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addHours(date, 12) AS add_hours_with_date,
    addHours(date_time, 12) AS add_hours_with_date_time,
    addHours(date_time_string, 12) AS add_hours_with_date_time_string ┌─add_hours_with_date─┬─add_hours_with_date_time─┬─add_hours_with_date_time_string─┐
│ 2024-01-01 12:00:00 │      2024-01-01 12:00:00 │         2024-01-01 12:00:00.000 │
└─────────────────────┴──────────────────────────┴─────────────────────────────────┘
## addMinutes [¶](https://www.tinybird.co/docs/about:blank#addminutes)

Adds a specified number of minutes to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addMinutes(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of minutes to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of minutes to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   minutes. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addMinutes(date, 20) AS add_minutes_with_date,
    addMinutes(date_time, 20) AS add_minutes_with_date_time,
    addMinutes(date_time_string, 20) AS add_minutes_with_date_time_string ┌─add_minutes_with_date─┬─add_minutes_with_date_time─┬─add_minutes_with_date_time_string─┐
│   2024-01-01 00:20:00 │        2024-01-01 00:20:00 │           2024-01-01 00:20:00.000 │
└───────────────────────┴────────────────────────────┴───────────────────────────────────┘
## addSeconds [¶](https://www.tinybird.co/docs/about:blank#addseconds)

Adds a specified number of seconds to a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addSeconds(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to add specified number of seconds to. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of seconds to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   plus `num`   seconds. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addSeconds(date, 30) AS add_seconds_with_date,
    addSeconds(date_time, 30) AS add_seconds_with_date_time,
    addSeconds(date_time_string, 30) AS add_seconds_with_date_time_string ┌─add_seconds_with_date─┬─add_seconds_with_date_time─┬─add_seconds_with_date_time_string─┐
│   2024-01-01 00:00:30 │        2024-01-01 00:00:30 │           2024-01-01 00:00:30.000 │
└───────────────────────┴────────────────────────────┴───────────────────────────────────┘
## addMilliseconds [¶](https://www.tinybird.co/docs/about:blank#addmilliseconds)

Adds a specified number of milliseconds to a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addMilliseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to add specified number of milliseconds to. DateTime/DateTime64, String.
- `num`   : Number of milliseconds to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   plus `num`   milliseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addMilliseconds(date_time, 1000) AS add_milliseconds_with_date_time,
    addMilliseconds(date_time_string, 1000) AS add_milliseconds_with_date_time_string ┌─add_milliseconds_with_date_time─┬─add_milliseconds_with_date_time_string─┐
│         2024-01-01 00:00:01.000 │                2024-01-01 00:00:01.000 │
└─────────────────────────────────┴────────────────────────────────────────┘
## addMicroseconds [¶](https://www.tinybird.co/docs/about:blank#addmicroseconds)

Adds a specified number of microseconds to a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addMicroseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to add specified number of microseconds to. DateTime/DateTime64, String.
- `num`   : Number of microseconds to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   plus `num`   microseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addMicroseconds(date_time, 1000000) AS add_microseconds_with_date_time,
    addMicroseconds(date_time_string, 1000000) AS add_microseconds_with_date_time_string ┌─add_microseconds_with_date_time─┬─add_microseconds_with_date_time_string─┐
│      2024-01-01 00:00:01.000000 │             2024-01-01 00:00:01.000000 │
└─────────────────────────────────┴────────────────────────────────────────┘
## addNanoseconds [¶](https://www.tinybird.co/docs/about:blank#addnanoseconds)

Adds a specified number of microseconds to a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addNanoseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to add specified number of nanoseconds to. DateTime/DateTime64, String.
- `num`   : Number of nanoseconds to add. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   plus `num`   nanoseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    addNanoseconds(date_time, 1000) AS add_nanoseconds_with_date_time,
    addNanoseconds(date_time_string, 1000) AS add_nanoseconds_with_date_time_string ┌─add_nanoseconds_with_date_time─┬─add_nanoseconds_with_date_time_string─┐
│  2024-01-01 00:00:00.000001000 │         2024-01-01 00:00:00.000001000 │
└────────────────────────────────┴───────────────────────────────────────┘
## addInterval [¶](https://www.tinybird.co/docs/about:blank#addinterval)

Adds an interval to another interval or tuple of intervals.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addInterval(interval_1, interval_2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `interval_1`   : First interval or tuple of intervals. interval, tuple(interval).
- `interval_2`   : Second interval to be added. interval.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of intervals. tuple(interval).

Intervals of the same type will be combined into a single interval. For instance if `toIntervalDay(1)` and `toIntervalDay(2)` are passed then the result will be `(3)` rather than `(1,1)`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT addInterval(INTERVAL 1 DAY, INTERVAL 1 MONTH)SELECT addInterval((INTERVAL 1 DAY, INTERVAL 1 YEAR), INTERVAL 1 MONTH)SELECT addInterval(INTERVAL 2 DAY, INTERVAL 1 DAY) Result:

┌─addInterval(toIntervalDay(1), toIntervalMonth(1))─┐
│ (1,1)                                             │
└───────────────────────────────────────────────────┘
┌─addInterval((toIntervalDay(1), toIntervalYear(1)), toIntervalMonth(1))─┐
│ (1,1,1)                                                                │
└────────────────────────────────────────────────────────────────────────┘
┌─addInterval(toIntervalDay(2), toIntervalDay(1))─┐
│ (3)                                             │
└─────────────────────────────────────────────────┘
## addTupleOfIntervals [¶](https://www.tinybird.co/docs/about:blank#addtupleofintervals)

Consecutively adds a tuple of intervals to a Date or a DateTime.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

addTupleOfIntervals(interval_1, interval_2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : First interval or interval of tuples. date/date32/datetime/datetime64.
- `intervals`   : Tuple of intervals to add to `date`   . tuple(interval).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   with added `intervals`   . date/date32/datetime/datetime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH toDate('2018-01-01') AS date
SELECT addTupleOfIntervals(date, (INTERVAL 1 DAY, INTERVAL 1 MONTH, INTERVAL 1 YEAR)) Result:

┌─addTupleOfIntervals(date, (toIntervalDay(1), toIntervalMonth(1), toIntervalYear(1)))─┐
│                                                                           2019-02-02 │
└──────────────────────────────────────────────────────────────────────────────────────┘
## subtractYears [¶](https://www.tinybird.co/docs/about:blank#subtractyears)

Subtracts a specified number of years from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractYears(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of years from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of years to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   years. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractYears(date, 1) AS subtract_years_with_date,
    subtractYears(date_time, 1) AS subtract_years_with_date_time,
    subtractYears(date_time_string, 1) AS subtract_years_with_date_time_string ┌─subtract_years_with_date─┬─subtract_years_with_date_time─┬─subtract_years_with_date_time_string─┐
│               2023-01-01 │           2023-01-01 00:00:00 │              2023-01-01 00:00:00.000 │
└──────────────────────────┴───────────────────────────────┴──────────────────────────────────────┘
## subtractQuarters [¶](https://www.tinybird.co/docs/about:blank#subtractquarters)

Subtracts a specified number of quarters from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractQuarters(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of quarters from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of quarters to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   quarters. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractQuarters(date, 1) AS subtract_quarters_with_date,
    subtractQuarters(date_time, 1) AS subtract_quarters_with_date_time,
    subtractQuarters(date_time_string, 1) AS subtract_quarters_with_date_time_string ┌─subtract_quarters_with_date─┬─subtract_quarters_with_date_time─┬─subtract_quarters_with_date_time_string─┐
│                  2023-10-01 │              2023-10-01 00:00:00 │                 2023-10-01 00:00:00.000 │
└─────────────────────────────┴──────────────────────────────────┴─────────────────────────────────────────┘
## subtractMonths [¶](https://www.tinybird.co/docs/about:blank#subtractmonths)

Subtracts a specified number of months from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractMonths(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of months from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of months to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   months. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractMonths(date, 1) AS subtract_months_with_date,
    subtractMonths(date_time, 1) AS subtract_months_with_date_time,
    subtractMonths(date_time_string, 1) AS subtract_months_with_date_time_string ┌─subtract_months_with_date─┬─subtract_months_with_date_time─┬─subtract_months_with_date_time_string─┐
│                2023-12-01 │            2023-12-01 00:00:00 │               2023-12-01 00:00:00.000 │
└───────────────────────────┴────────────────────────────────┴───────────────────────────────────────┘
## subtractWeeks [¶](https://www.tinybird.co/docs/about:blank#subtractweeks)

Subtracts a specified number of weeks from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractWeeks(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of weeks from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of weeks to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   weeks. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractWeeks(date, 1) AS subtract_weeks_with_date,
    subtractWeeks(date_time, 1) AS subtract_weeks_with_date_time,
    subtractWeeks(date_time_string, 1) AS subtract_weeks_with_date_time_string ┌─subtract_weeks_with_date─┬─subtract_weeks_with_date_time─┬─subtract_weeks_with_date_time_string─┐
 │               2023-12-25 │           2023-12-25 00:00:00 │              2023-12-25 00:00:00.000 │
 └──────────────────────────┴───────────────────────────────┴──────────────────────────────────────┘
## subtractDays [¶](https://www.tinybird.co/docs/about:blank#subtractdays)

Subtracts a specified number of days from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractDays(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of days from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of days to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   days. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractDays(date, 31) AS subtract_days_with_date,
    subtractDays(date_time, 31) AS subtract_days_with_date_time,
    subtractDays(date_time_string, 31) AS subtract_days_with_date_time_string ┌─subtract_days_with_date─┬─subtract_days_with_date_time─┬─subtract_days_with_date_time_string─┐
│              2023-12-01 │          2023-12-01 00:00:00 │             2023-12-01 00:00:00.000 │
└─────────────────────────┴──────────────────────────────┴─────────────────────────────────────┘
## subtractHours [¶](https://www.tinybird.co/docs/about:blank#subtracthours)

Subtracts a specified number of hours from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractHours(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of hours from. Date/Date32/Datetime/Datetime64, String.
- `num`   : Number of hours to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   hours. Date/Date32/Datetime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractHours(date, 12) AS subtract_hours_with_date,
    subtractHours(date_time, 12) AS subtract_hours_with_date_time,
    subtractHours(date_time_string, 12) AS subtract_hours_with_date_time_string ┌─subtract_hours_with_date─┬─subtract_hours_with_date_time─┬─subtract_hours_with_date_time_string─┐
│      2023-12-31 12:00:00 │           2023-12-31 12:00:00 │              2023-12-31 12:00:00.000 │
└──────────────────────────┴───────────────────────────────┴──────────────────────────────────────┘
## subtractMinutes [¶](https://www.tinybird.co/docs/about:blank#subtractminutes)

Subtracts a specified number of minutes from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractMinutes(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of minutes from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of minutes to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   minutes. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractMinutes(date, 30) AS subtract_minutes_with_date,
    subtractMinutes(date_time, 30) AS subtract_minutes_with_date_time,
    subtractMinutes(date_time_string, 30) AS subtract_minutes_with_date_time_string ┌─subtract_minutes_with_date─┬─subtract_minutes_with_date_time─┬─subtract_minutes_with_date_time_string─┐
│        2023-12-31 23:30:00 │             2023-12-31 23:30:00 │                2023-12-31 23:30:00.000 │
└────────────────────────────┴─────────────────────────────────┴────────────────────────────────────────┘
## subtractSeconds [¶](https://www.tinybird.co/docs/about:blank#subtractseconds)

Subtracts a specified number of seconds from a date, a date with time or a string-encoded date / date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractSeconds(date, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : Date / date with time to subtract specified number of seconds from. Date/Date32/DateTime/DateTime64, String.
- `num`   : Number of seconds to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   minus `num`   seconds. Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDate('2024-01-01') AS date,
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractSeconds(date, 60) AS subtract_seconds_with_date,
    subtractSeconds(date_time, 60) AS subtract_seconds_with_date_time,
    subtractSeconds(date_time_string, 60) AS subtract_seconds_with_date_time_string ┌─subtract_seconds_with_date─┬─subtract_seconds_with_date_time─┬─subtract_seconds_with_date_time_string─┐
│        2023-12-31 23:59:00 │             2023-12-31 23:59:00 │                2023-12-31 23:59:00.000 │
└────────────────────────────┴─────────────────────────────────┴────────────────────────────────────────┘
## subtractMilliseconds [¶](https://www.tinybird.co/docs/about:blank#subtractmilliseconds)

Subtracts a specified number of milliseconds from a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractMilliseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to subtract specified number of milliseconds from. DateTime/DateTime64, String.
- `num`   : Number of milliseconds to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   minus `num`   milliseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractMilliseconds(date_time, 1000) AS subtract_milliseconds_with_date_time,
    subtractMilliseconds(date_time_string, 1000) AS subtract_milliseconds_with_date_time_string ┌─subtract_milliseconds_with_date_time─┬─subtract_milliseconds_with_date_time_string─┐
│              2023-12-31 23:59:59.000 │                     2023-12-31 23:59:59.000 │
└──────────────────────────────────────┴─────────────────────────────────────────────┘
## subtractMicroseconds [¶](https://www.tinybird.co/docs/about:blank#subtractmicroseconds)

Subtracts a specified number of microseconds from a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractMicroseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to subtract specified number of microseconds from. DateTime/DateTime64, String.
- `num`   : Number of microseconds to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   minus `num`   microseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractMicroseconds(date_time, 1000000) AS subtract_microseconds_with_date_time,
    subtractMicroseconds(date_time_string, 1000000) AS subtract_microseconds_with_date_time_string ┌─subtract_microseconds_with_date_time─┬─subtract_microseconds_with_date_time_string─┐
│           2023-12-31 23:59:59.000000 │                  2023-12-31 23:59:59.000000 │
└──────────────────────────────────────┴─────────────────────────────────────────────┘
## subtractNanoseconds [¶](https://www.tinybird.co/docs/about:blank#subtractnanoseconds)

Subtracts a specified number of nanoseconds from a date with time or a string-encoded date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractNanoseconds(date_time, num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date_time`   : Date with time to subtract specified number of nanoseconds from. DateTime/DateTime64, String.
- `num`   : Number of nanoseconds to subtract. (U)Int*, Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date_time`   minus `num`   nanoseconds. DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH
    toDateTime('2024-01-01 00:00:00') AS date_time,
    '2024-01-01 00:00:00' AS date_time_string
SELECT
    subtractNanoseconds(date_time, 1000) AS subtract_nanoseconds_with_date_time,
    subtractNanoseconds(date_time_string, 1000) AS subtract_nanoseconds_with_date_time_string ┌─subtract_nanoseconds_with_date_time─┬─subtract_nanoseconds_with_date_time_string─┐
│       2023-12-31 23:59:59.999999000 │              2023-12-31 23:59:59.999999000 │
└─────────────────────────────────────┴────────────────────────────────────────────┘
## subtractInterval [¶](https://www.tinybird.co/docs/about:blank#subtractinterval)

Adds a negated interval to another interval or tuple of intervals.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractInterval(interval_1, interval_2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `interval_1`   : First interval or interval of tuples. interval, tuple(interval).
- `interval_2`   : Second interval to be negated. interval.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of intervals. tuple(interval).

Intervals of the same type will be combined into a single interval. For instance if `toIntervalDay(2)` and `toIntervalDay(1)` are passed then the result will be `(1)` rather than `(2,1)`

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT subtractInterval(INTERVAL 1 DAY, INTERVAL 1 MONTH)SELECT subtractInterval((INTERVAL 1 DAY, INTERVAL 1 YEAR), INTERVAL 1 MONTH)SELECT subtractInterval(INTERVAL 2 DAY, INTERVAL 1 DAY) Result:

┌─subtractInterval(toIntervalDay(1), toIntervalMonth(1))─┐
│ (1,-1)                                                 │
└────────────────────────────────────────────────────────┘
┌─subtractInterval((toIntervalDay(1), toIntervalYear(1)), toIntervalMonth(1))─┐
│ (1,1,-1)                                                                    │
└─────────────────────────────────────────────────────────────────────────────┘
┌─subtractInterval(toIntervalDay(2), toIntervalDay(1))─┐
│ (1)                                                  │
└──────────────────────────────────────────────────────┘
## subtractTupleOfIntervals [¶](https://www.tinybird.co/docs/about:blank#subtracttupleofintervals)

Consecutively subtracts a tuple of intervals from a Date or a DateTime.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subtractTupleOfIntervals(interval_1, interval_2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `date`   : First interval or interval of tuples. Date/Date32/DateTime/DateTime64.
- `intervals`   : Tuple of intervals to subtract from `date`   . tuple(interval).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `date`   with subtracted `intervals`   . Date/Date32/DateTime/DateTime64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH toDate('2018-01-01') AS date SELECT subtractTupleOfIntervals(date, (INTERVAL 1 DAY, INTERVAL 1 YEAR)) Result:

┌─subtractTupleOfIntervals(date, (toIntervalDay(1), toIntervalYear(1)))─┐
│                                                                           2016-12-31 │
└───────────────────────────────────────────────────────────────────────┘
## timeSlots [¶](https://www.tinybird.co/docs/about:blank#timeslots)

For a time interval starting at 'StartTime' and continuing for 'Duration' seconds, it returns an array of moments in time, consisting of points from this interval rounded down to the 'Size' in seconds. 'Size' is an optional parameter set to 1800 (30 minutes) by default. This is necessary, for example, when searching for pageviews in the corresponding session. Accepts DateTime and DateTime64 as 'StartTime' argument. For DateTime, 'Duration' and 'Size' arguments must be `UInt32` . For 'DateTime64' they must be `Decimal64`. Returns an array of DateTime/DateTime64 (return type matches the type of 'StartTime'). For DateTime64, the return value's scale can differ from the scale of 'StartTime' --- the highest scale among all given arguments is taken.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeSlots(StartTime, Duration,\[, Size\])
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600))SELECT timeSlots(toDateTime('1980-12-12 21:01:02', 'UTC'), toUInt32(600), 299)SELECT timeSlots(toDateTime64('1980-12-12 21:01:02.1234', 4, 'UTC'), toDecimal64(600.1, 1), toDecimal64(299, 0)) Result:

┌─timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600))─┐
│ ['2012-01-01 12:00:00','2012-01-01 12:30:00']               │
└─────────────────────────────────────────────────────────────┘
┌─timeSlots(toDateTime('1980-12-12 21:01:02', 'UTC'), toUInt32(600), 299)─┐
│ ['1980-12-12 20:56:13','1980-12-12 21:01:12','1980-12-12 21:06:11']     │
└─────────────────────────────────────────────────────────────────────────┘
┌─timeSlots(toDateTime64('1980-12-12 21:01:02.1234', 4, 'UTC'), toDecimal64(600.1, 1), toDecimal64(299, 0))─┐
│ ['1980-12-12 20:56:13.0000','1980-12-12 21:01:12.0000','1980-12-12 21:06:11.0000']                        │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┘
## formatDateTime [¶](https://www.tinybird.co/docs/about:blank#formatdatetime)

Formats a Time according to the given Format string. Format is a constant expression, so you can't have multiple formats for a single result column.

formatDateTime uses MySQL datetime format style, refer to https://dev.mysql.com/doc/refman/8.0/en/date-and-time-functions.html#function_date-format.

The opposite operation of this function is parseDateTime.

Alias: `DATE_FORMAT`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

formatDateTime(Time, Format[, Timezone])
### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Returns time and date values according to the determined format.

### Replacement fields [¶](https://www.tinybird.co/docs/about:blank#replacement-fields)

Using replacement fields, you can define a pattern for the resulting string. "Example" column shows formatting result for `2018-01-02 22:33:44`.

| Placeholder | Description | Example |
| --- | --- | --- |
| %a | abbreviated weekday name (Mon-Sun) | Mon |
| %b | abbreviated month name (Jan-Dec) | Jan |
| %c | month as an integer number (01-12) | 01 |
| %C | year divided by 100 and truncated to integer (00-99) | 20 |
| %d | day of the month, zero-padded (01-31) | 02 |
| %D | Short MM/DD/YY date, equivalent to %m/%d/%y | 01/02/18 |
| %e | day of the month, space-padded (1-31) | 2 |
| %f | fractional second | 1234560 |
| %F | short YYYY-MM-DD date, equivalent to %Y-%m-%d | 2018-01-02 |
| %g | two-digit year format, aligned to ISO 8601, abbreviated from four-digit notation | 18 |
| %G | four-digit year format for ISO week number, calculated from the week-based year defined by the ISO 8601 standard, normally useful only with %V | 2018 |
| %h | hour in 12h format (01-12) | 09 |
| %H | hour in 24h format (00-23) | 22 |
| %i | minute (00-59) | 33 |
| %I | hour in 12h format (01-12) | 10 |
| %j | day of the year (001-366) | 002 |
| %k | hour in 24h format (00-23) | 14 |
| %l | hour in 12h format (01-12) | 09 |
| %m | month as an integer number (01-12) | 01 |
| %M | full month name (January-December) | January |
| %n | new-line character (') |  |
| %p | AM or PM designation | PM |
| %Q | Quarter (1-4) | 1 |
| %r | 12-hour HH:MM AM/PM time, equivalent to %h:%i %p | 10:30 PM |
| %R | 24-hour HH:MM time, equivalent to %H:%i | 22:33 |
| %s | second (00-59) | 44 |
| %S | second (00-59) | 44 |
| %t | horizontal-tab character (') |  |
| %T | ISO 8601 time format (HH:MM:SS), equivalent to %H:%i:%S | 22:33:44 |
| %u | ISO 8601 weekday as number with Monday as 1 (1-7) | 2 |
| %V | ISO 8601 week number (01-53) | 01 |
| %w | weekday as a integer number with Sunday as 0 (0-6) | 2 |
| %W | full weekday name (Monday-Sunday) | Monday |
| %y | Year, last two digits (00-99) | 18 |
| %Y | Year | 2018 |
| %z | Time offset from UTC as +HHMM or -HHMM | -0500 |
| %% | a % sign | % |

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT formatDateTime(toDate('2010-01-04'), '%g') Result:

┌─formatDateTime(toDate('2010-01-04'), '%g')─┐
│ 10                                         │
└────────────────────────────────────────────┘ SELECT formatDateTime(toDateTime64('2010-01-04 12:34:56.123456', 7), '%f') Result:

┌─formatDateTime(toDateTime64('2010-01-04 12:34:56.123456', 7), '%f')─┐
│ 1234560                                                             │
└─────────────────────────────────────────────────────────────────────┘ Additionally, the `formatDateTime` function can take a third String argument containing the name of the time zone. Example: `Asia/Istanbul` . In this case, the time is formatted according to the specified time zone.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT
    now() AS ts,
    time_zone,
    formatDateTime(ts, '%T', time_zone) AS str_tz_time
FROM system.time_zones
WHERE time_zone LIKE 'Europe%'
LIMIT 10

┌──────────────────ts─┬─time_zone─────────┬─str_tz_time─┐
│ 2023-09-08 19:13:40 │ Europe/Amsterdam  │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Andorra    │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Astrakhan  │ 23:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Athens     │ 22:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Belfast    │ 20:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Belgrade   │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Berlin     │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Bratislava │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Brussels   │ 21:13:40    │
│ 2023-09-08 19:13:40 │ Europe/Bucharest  │ 22:13:40    │
└─────────────────────┴───────────────────┴─────────────┘
## formatDateTimeInJodaSyntax [¶](https://www.tinybird.co/docs/about:blank#formatdatetimeinjodasyntax)

Similar to formatDateTime, except that it formats datetime in Joda style instead of MySQL style.

The opposite operation of this function is parseDateTimeInJodaSyntax.

### Replacement fields [¶](https://www.tinybird.co/docs/about:blank#replacement-fields)

Using replacement fields, you can define a pattern for the resulting string.

| Placeholder | Description | Presentation | Examples |
| --- | --- | --- | --- |
| G | era | text | AD |
| C | century of era (>=0) | number | 20 |
| Y | year of era (>=0) | year | 1996 |
| x | weekyear (not supported yet) | year | 1996 |
| w | week of weekyear (not supported yet) | number | 27 |
| e | day of week | number | 2 |
| E | day of week | text | Tuesday; Tue |
| y | year | year | 1996 |
| D | day of year | number | 189 |
| M | month of year | month | July; Jul; 07 |
| d | day of month | number | 10 |
| a | halfday of day | text | PM |
| K | hour of halfday (0~11) | number | 0 |
| h | clockhour of halfday (1~12) | number | 12 |
| H | hour of day (0~23) | number | 0 |
| k | clockhour of day (1~24) | number | 24 |
| m | minute of hour | number | 30 |
| s | second of minute | number | 55 |
| S | fraction of second | number | 978 |
| z | time zone | text | Eastern Standard Time; EST |
| Z | time zone offset | zone | -0800; -0812 |
| ' | escape for text | delimiter |  |
| '' | single quote | literal | ' |

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT formatDateTimeInJodaSyntax(toDateTime('2010-01-04 12:34:56'), 'yyyy-MM-dd HH:mm:ss') Result:

┌─formatDateTimeInJodaSyntax(toDateTime('2010-01-04 12:34:56'), 'yyyy-MM-dd HH:mm:ss')─┐
│ 2010-01-04 12:34:56                                                                     │
└─────────────────────────────────────────────────────────────────────────────────────────┘
## dateName [¶](https://www.tinybird.co/docs/about:blank#datename)

Returns specified part of date.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

dateName(date_part, date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date_part`   : Date part. Possible values: 'year', 'quarter', 'month', 'week', 'dayofyear', 'day', 'weekday', 'hour', 'minute', 'second'. String.
- `date`   : Date. Date, Date32, DateTime or DateTime64.
- `timezone`   : Timezone. Optional. String.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The specified part of date. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH toDateTime('2021-04-14 11:22:33') AS date_value
SELECT
    dateName('year', date_value),
    dateName('month', date_value),
    dateName('day', date_value) Result:

┌─dateName('year', date_value)─┬─dateName('month', date_value)─┬─dateName('day', date_value)─┐
│ 2021                         │ April                         │ 14                          │
└──────────────────────────────┴───────────────────────────────┴─────────────────────────────┘
## monthName [¶](https://www.tinybird.co/docs/about:blank#monthname)

Returns name of the month.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

monthName(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date or date with time. Date, DateTime or DateTime64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The name of the month. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH toDateTime('2021-04-14 11:22:33') AS date_value
SELECT monthName(date_value) Result:

┌─monthName(date_value)─┐
│ April                 │
└───────────────────────┘
## fromUnixTimestamp [¶](https://www.tinybird.co/docs/about:blank#fromunixtimestamp)

This function converts a Unix timestamp to a calendar date and a time of a day.

It can be called in two ways:

When given a single argument of type Integer, it returns a value of type DateTime, i.e. behaves like toDateTime.

Alias: `FROM_UNIXTIME`.

### Example: [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromUnixTimestamp(423543535) Result:

┌─fromUnixTimestamp(423543535)─┐
│          1983-06-04 10:58:55 │
└──────────────────────────────┘ When given two or three arguments where the first argument is a value of type Integer, Date, Date32, DateTime or DateTime64, the second argument is a constant format string and the third argument is an optional constant time zone string, the function returns a value of type String, i.e. it behaves like formatDateTime. In this case, MySQL's datetime format style is used.

### Example: [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromUnixTimestamp(1234334543, '%Y-%m-%d %R:%S') AS DateTime Result:

┌─DateTime────────────┐
│ 2009-02-11 14:42:23 │
└─────────────────────┘
## fromUnixTimestampInJodaSyntax [¶](https://www.tinybird.co/docs/about:blank#fromunixtimestampinjodasyntax)

Same as fromUnixTimestamp but when called in the second way (two or three arguments), the formatting is performed using Joda style instead of MySQL style.

### Example: [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromUnixTimestampInJodaSyntax(1234334543, 'yyyy-MM-dd HH:mm:ss', 'UTC') AS DateTime Result:

┌─DateTime────────────┐
│ 2009-02-11 06:42:23 │
└─────────────────────┘
## toModifiedJulianDay [¶](https://www.tinybird.co/docs/about:blank#tomodifiedjulianday)

Converts a Proleptic Gregorian calendar date in text form `YYYY-MM-DD` to a Modified Julian Day number in Int32. This function supports date from `0000-01-01` to `9999-12-31` . It raises an exception if the argument can't be parsed as a date, or the date is invalid.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toModifiedJulianDay(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date in text form. String or FixedString.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Modified Julian Day number. Int32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toModifiedJulianDay('2020-01-01') Result:

┌─toModifiedJulianDay('2020-01-01')─┐
│                             58849 │
└───────────────────────────────────┘
## toModifiedJulianDayOrNull [¶](https://www.tinybird.co/docs/about:blank#tomodifiedjuliandayornull)

Similar to toModifiedJulianDay(), but instead of raising exceptions it returns `NULL`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toModifiedJulianDayOrNull(date)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `date`   : Date in text form. String or FixedString.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Modified Julian Day number. Nullable(Int32).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toModifiedJulianDayOrNull('2020-01-01') Result:

┌─toModifiedJulianDayOrNull('2020-01-01')─┐
│                                   58849 │
└─────────────────────────────────────────┘
## fromModifiedJulianDay [¶](https://www.tinybird.co/docs/about:blank#frommodifiedjulianday)

Converts a Modified Julian Day number to a Proleptic Gregorian calendar date in text form `YYYY-MM-DD` . This function supports day number from `-678941` to `2973483` (which represent 0000-01-01 and 9999-12-31 respectively). It raises an exception if the day number is outside of the supported range.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

fromModifiedJulianDay(day)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `day`   : Modified Julian Day number. Any integral types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Date in text form. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromModifiedJulianDay(58849) Result:

┌─fromModifiedJulianDay(58849)─┐
│ 2020-01-01                   │
└──────────────────────────────┘
## fromModifiedJulianDayOrNull [¶](https://www.tinybird.co/docs/about:blank#frommodifiedjuliandayornull)

Similar to fromModifiedJulianDayOrNull(), but instead of raising exceptions it returns `NULL`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

fromModifiedJulianDayOrNull(day)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `day`   : Modified Julian Day number. Any integral types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Date in text form. Nullable(String)

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromModifiedJulianDayOrNull(58849) Result:

┌─fromModifiedJulianDayOrNull(58849)─┐
│ 2020-01-01                         │
└────────────────────────────────────┘
## toUTCTimestamp [¶](https://www.tinybird.co/docs/about:blank#toutctimestamp)

Convert DateTime/DateTime64 type value from other time zone to UTC timezone timestamp. This function is mainly included for compatibility with Apache Spark and similar frameworks.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

toUTCTimestamp(time_val, time_zone)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `time_val`   : A DateTime/DateTime64 type const value or an expression . DateTime/DateTime64 types
- `time_zone`   : A String type const value or an expression represent the time zone. String types

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- DateTime/DateTime64 in text form

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT toUTCTimestamp(toDateTime('2023-03-16'), 'Asia/Shanghai') Result:

┌─toUTCTimestamp(toDateTime('2023-03-16'), 'Asia/Shanghai')┐
│                                     2023-03-15 16:00:00 │
└─────────────────────────────────────────────────────────┘
## fromUTCTimestamp [¶](https://www.tinybird.co/docs/about:blank#fromutctimestamp)

Convert DateTime/DateTime64 type value from UTC timezone to other time zone timestamp.  This function is mainly included for compatibility with Apache Spark and similar frameworks.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

fromUTCTimestamp(time_val, time_zone)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `time_val`   : A DateTime/DateTime64 type const value or an expression . DateTime/DateTime64 types
- `time_zone`   : A String type const value or an expression represent the time zone. String types

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- DateTime/DateTime64 in text form

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT fromUTCTimestamp(toDateTime64('2023-03-16 10:00:00', 3), 'Asia/Shanghai') Result:

┌─fromUTCTimestamp(toDateTime64('2023-03-16 10:00:00',3), 'Asia/Shanghai')─┐
│                                                 2023-03-16 18:00:00.000 │
└─────────────────────────────────────────────────────────────────────────┘
## UTCTimestamp [¶](https://www.tinybird.co/docs/about:blank#utctimestamp)

Returns the current date and time at the moment of query analysis. The function is a constant expression.

This function gives the same result that `now('UTC')` would. It was added only for MySQL support and `now` is the preferred usage.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

UTCTimestamp() Alias: `UTC_timestamp`.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the current date and time at the moment of query analysis. DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT UTCTimestamp() Result:

┌──────UTCTimestamp()─┐
│ 2024-05-28 08:32:09 │
└─────────────────────┘
## timeDiff [¶](https://www.tinybird.co/docs/about:blank#timediff)

Returns the difference between two dates or dates with time values. The difference is calculated in units of seconds. It is same as `dateDiff` and was added only for MySQL support. `dateDiff` is preferred.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

timeDiff(first_datetime, second_datetime)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `first_datetime`   : A DateTime/DateTime64 type const value or an expression . DateTime/DateTime64 types
- `second_datetime`   : A DateTime/DateTime64 type const value or an expression . DateTime/DateTime64 types

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The difference between two dates or dates with time values in seconds.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

timeDiff(toDateTime64('1927-01-01 00:00:00', 3), toDate32('1927-01-02')) **Result**:

┌─timeDiff(toDateTime64('1927-01-01 00:00:00', 3), toDate32('1927-01-02'))─┐
│                                                                    86400 │
└──────────────────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/conditional-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Conditionalfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for conditional logic."
---




# Conditional functions [¶](https://www.tinybird.co/docs/about:blank#conditional-functions)

The following functions are used to perform conditional logic.

## if [¶](https://www.tinybird.co/docs/about:blank#if)

Performs conditional branching.

If the condition `cond` evaluates to a non-zero value, the function returns the result of the expression `then` . If `cond` evaluates to zero or `NULL` , then the result of the `else` expression is returned.

Setting short_circuit_function_evaluation controls whether short-circuit evaluation is used. If enabled, the `then` expression is evaluated only on rows where `cond` is `true` and the `else` expression where `cond` is `false` . For example, with short-circuit evaluation, no division-by-zero exception is thrown when executing the query `SELECT if(number = 0, 0, intDiv(42, number)) FROM numbers(10)`.

`then` and `else` must be of a similar type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

if(cond, then, else) Alias: `cond ? then : else` (ternary operator)

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `cond`   – The evaluated condition. UInt8, Nullable(UInt8) or NULL.
- `then`   – The expression returned if `condition`   is true.
- `else`   – The expression returned if `condition`   is `false`   or NULL.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

The result of either the `then` and `else` expressions, depending on condition `cond`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT if(1, plus(2, 2), plus(2, 6)) Result:

┌─plus(2, 2)─┐
│          4 │
└────────────┘
## multiIf [¶](https://www.tinybird.co/docs/about:blank#multiif)

Allows to write the CASE operator more compactly in the query.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

multiIf(cond_1, then_1, cond_2, then_2, ..., else) Setting short_circuit_function_evaluation controls whether short-circuit evaluation is used. If enabled, the `then_i` expression is evaluated only on rows where `((NOT cond_1) AND (NOT cond_2) AND ... AND (NOT cond_{i-1}) AND cond_i)` is `true`, `cond_i` will be evaluated only on rows where `((NOT cond_1) AND (NOT cond_2) AND ... AND (NOT cond_{i-1}))` is `true` . For example, with short-circuit evaluation, no division-by-zero exception is thrown when executing the query `SELECT multiIf(number = 2, intDiv(1, number), number = 5) FROM numbers(10)`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function accepts `2N+1` parameters:

- `cond_N`   : The N-th evaluated condition which controls if `then_N`   is returned.
- `then_N`   : The result of the function when `cond_N`   is true.
- `else`   : The result of the function if none of conditions is true.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

The result of either any of the `then_N` or `else` expressions, depending on the conditions `cond_N`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Assuming this table:

┌─left─┬─right─┐
│ ᴺᵁᴸᴸ │     4 │
│    1 │     3 │
│    2 │     2 │
│    3 │     1 │
│    4 │  ᴺᵁᴸᴸ │
└──────┴───────┘ SELECT
    left,
    right,
    multiIf(left < right, 'left is smaller', left > right, 'left is greater', left = right, 'Both equal', 'Null value') AS result
FROM LEFT_RIGHT

┌─left─┬─right─┬─result──────────┐
│ ᴺᵁᴸᴸ │     4 │ Null value      │
│    1 │     3 │ left is smaller │
│    2 │     2 │ Both equal      │
│    3 │     1 │ left is greater │
│    4 │  ᴺᵁᴸᴸ │ Null value      │
└──────┴───────┴─────────────────┘
## Using Conditional Results Directly [¶](https://www.tinybird.co/docs/about:blank#using-conditional-results-directly)

Conditionals always result to `0`, `1` or `NULL` . So you can use conditional results directly like this:

SELECT left < right AS is_small
FROM LEFT_RIGHT

┌─is_small─┐
│     ᴺᵁᴸᴸ │
│        1 │
│        0 │
│        0 │
│     ᴺᵁᴸᴸ │
└──────────┘
## NULL Values in Conditionals [¶](https://www.tinybird.co/docs/about:blank#null-values-in-conditionals)

When `NULL` values are involved in conditionals, the result will also be `NULL`.

SELECT
    NULL < 1,
    2 < NULL,
    NULL < NULL,
    NULL = NULL

┌─less(NULL, 1)─┬─less(2, NULL)─┬─less(NULL, NULL)─┬─equals(NULL, NULL)─┐
│ ᴺᵁᴸᴸ          │ ᴺᵁᴸᴸ          │ ᴺᵁᴸᴸ             │ ᴺᵁᴸᴸ               │
└────────���──────┴───────────────┴──────────────────┴────────────────────┘ So you should construct your queries carefully if the types are `Nullable`.

The following example demonstrates this by failing to add equals condition to `multiIf`.

SELECT
    left,
    right,
    multiIf(left < right, 'left is smaller', left > right, 'right is smaller', 'Both equal') AS faulty_result
FROM LEFT_RIGHT

┌─left─┬─right─┬─faulty_result────┐
│ ᴺᵁᴸᴸ │     4 │ Both equal       │
│    1 │     3 │ left is smaller  │
│    2 │     2 │ Both equal       │
│    3 │     1 │ right is smaller │
│    4 │  ᴺᵁᴸᴸ │ Both equal       │
└──────┴───────┴──────────────────┘
## greatest [¶](https://www.tinybird.co/docs/about:blank#greatest)

Returns the greatest across a list of values.  All of the list members must be of comparable types.

Examples:

SELECT greatest(1, 2, toUInt8(3), 3.) result,  toTypeName(result) type ┌─result─┬─type────┐
│      3 │ Float64 │
└────────┴─────────┘ The type returned is a Float64 as the UInt8 must be promoted to 64 bit for the comparison.

SELECT greatest(['hello'], ['there'], ['world']) ┌─greatest(['hello'], ['there'], ['world'])─┐
│ ['world']                                 │
└───────────────────────────────────────────┘ SELECT greatest(toDateTime32(now() + toIntervalDay(1)), toDateTime64(now(), 3)) ┌─greatest(toDateTime32(plus(now(), toIntervalDay(1))), toDateTime64(now(), 3))─┐
│                                                       2023-05-12 01:16:59.000 │
└──---──────────────────────────────────────────────────────────────────────────┘ The type returned is a DateTime64 as the DataTime32 must be promoted to 64 bit for the comparison.

## least [¶](https://www.tinybird.co/docs/about:blank#least)

Returns the least across a list of values.  All of the list members must be of comparable types.

Examples:

SELECT least(1, 2, toUInt8(3), 3.) result,  toTypeName(result) type ┌─result─┬─type────┐
│      1 │ Float64 │
└────────┴─────────┘ The type returned is a Float64 as the UInt8 must be promoted to 64 bit for the comparison.

SELECT least(['hello'], ['there'], ['world']) ┌─least(['hello'], ['there'], ['world'])─┐
│ ['hello']                              │
└────────────────────────────────────────┘ SELECT least(toDateTime32(now() + toIntervalDay(1)), toDateTime64(now(), 3)) ┌─least(toDateTime32(plus(now(), toIntervalDay(1))), toDateTime64(now(), 3))─┐
│                                                    2023-05-12 01:16:59.000 │
└────────────────────────────────────────────────────────────────────────────┘ The type returned is a DateTime64 as the DataTime32 must be promoted to 64 bit for the comparison.

## clamp [¶](https://www.tinybird.co/docs/about:blank#clamp)

Constrain the return value between A and B.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

clamp(value, min, max)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   – Input value.
- `min`   – Limit the lower bound.
- `max`   – Limit the upper bound.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

If the value is less than the minimum value, return the minimum value; if it's greater than the maximum value, return the maximum value; otherwise, return the current value.

Examples:

SELECT clamp(1, 2, 3) result,  toTypeName(result) type ┌─result─┬─type────┐
│      2 │ Float64 │
└────────┴─────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/comparison-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Comparisonfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for comparing values."
---




# Comparison functions [¶](https://www.tinybird.co/docs/about:blank#comparison-functions)

The following functions are used to compare values. The comparison functions return 0 or 1 as Uint8.

The following types can be compared:

- Numbers
- Strings and fixed strings
- Dates
- Dates with times

Only values within the same group can be compared (e.g. UInt16 and UInt64) but not across groups (e.g. UInt16 and DateTime).

Strings are compared byte-by-byte. Note this may lead to unexpected results if one of the strings contains UTF-8 encoded multi-byte characters.

A string S1 which has another string S2 as prefix is considered longer than S2.

## equals operators [¶](https://www.tinybird.co/docs/about:blank#equals-operators)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

equals(a, b) Alias:

- `a = b`   (operator)
- `a == b`   (operator)

## notEquals operators [¶](https://www.tinybird.co/docs/about:blank#notequals-operators)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

notEquals(a, b) Alias:

- `a != b`   (operator)
- `a <> b`   (operator)

## less operator [¶](https://www.tinybird.co/docs/about:blank#less-operator)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

less(a, b) Alias:

- `a < b`   (operator)

## greater operator [¶](https://www.tinybird.co/docs/about:blank#greater-operator)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

greater(a, b) Alias:

- `a > b`   (operator)

## lessOrEquals operator [¶](https://www.tinybird.co/docs/about:blank#lessorequals-operator)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

lessOrEquals(a, b) Alias:

- `a <= b`   (operator)

## greaterOrEquals operator [¶](https://www.tinybird.co/docs/about:blank#greaterorequals-operator)

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

greaterOrEquals(a, b) Alias:

- `a >= b`   (operator)



---

URL: https://www.tinybird.co/docs/sql-reference/functions/bitmap-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Bitmapfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for working with bitmaps."
---




# Bitmap functions [¶](https://www.tinybird.co/docs/about:blank#bitmap-functions)

The following functions are used to work with bitmaps.

Bitmaps can be constructed in two ways. The first way is constructed by aggregation function groupBitmap with `-State` , the other way is to constructed a bitmap from an Array object.

## bitmapBuild [¶](https://www.tinybird.co/docs/about:blank#bitmapbuild)

Builds a bitmap from an unsigned integer array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapBuild(array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Unsigned integer array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapBuild([1, 2, 3, 4, 5]) AS res, toTypeName(res) ┌─res─┬─toTypeName(bitmapBuild([1, 2, 3, 4, 5]))─────┐
│     │ AggregateFunction(groupBitmap, UInt8)        │
└─────┴──────────────────────────────────────────────┘
## bitmapToArray [¶](https://www.tinybird.co/docs/about:blank#bitmaptoarray)

Converts bitmap to an integer array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapToArray(bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapBuild([1, 2, 3, 4, 5])) AS res Result:

┌─res─────────┐
│ [1,2,3,4,5] │
└─────────────┘
## bitmapSubsetInRange [¶](https://www.tinybird.co/docs/about:blank#bitmapsubsetinrange)

Returns the subset of a bitmap with bits within a value interval.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapSubsetInRange(bitmap, range_start, range_end)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.
- `range_start`   – Start of the range (inclusive). UInt32.
- `range_end`   – End of the range (exclusive). UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapSubsetInRange(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500]), toUInt32(30), toUInt32(200))) AS res Result:

┌─res───────────────┐
│ [30,31,32,33,100] │
└───────────────────┘
## bitmapSubsetLimit [¶](https://www.tinybird.co/docs/about:blank#bitmapsubsetlimit)

Returns a subset of a bitmap with smallest bit value `range_start` and at most `cardinality_limit` elements.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapSubsetLimit(bitmap, range_start, cardinality_limit)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.
- `range_start`   – Start of the range (inclusive). UInt32.
- `cardinality_limit`   – Maximum cardinality of the subset. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapSubsetLimit(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500]), toUInt32(30), toUInt32(200))) AS res Result:

┌─res───────────────────────┐
│ [30,31,32,33,100,200,500] │
└───────────────────────────┘
## subBitmap [¶](https://www.tinybird.co/docs/about:blank#subbitmap)

Returns a subset of the bitmap, starting from position `offset` . The maximum cardinality of the returned bitmap is `cardinality_limit`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

subBitmap(bitmap, offset, cardinality_limit)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – The bitmap. Bitmap object.
- `offset`   – The position of the first element of the subset. UInt32.
- `cardinality_limit`   – The maximum number of elements in the subset. UInt32.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(subBitmap(bitmapBuild([0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,100,200,500]), toUInt32(10), toUInt32(10))) AS res Result:

┌─res─────────────────────────────┐
│ [10,11,12,13,14,15,16,17,18,19] │
└─────────────────────────────────┘
## bitmapContains [¶](https://www.tinybird.co/docs/about:blank#bitmapcontains)

Checks whether the bitmap contains an element.

bitmapContains(bitmap, needle)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.
- `needle`   – Searched bit value. UInt32.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- 0: If `bitmap`   doesn't contain `needle`   . UInt8.
- 1: If `bitmap`   contains `needle`   . UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapContains(bitmapBuild([1,5,7,9]), toUInt32(9)) AS res Result:

┌─res─┐
│  1  │
└─────┘
## bitmapHasAny [¶](https://www.tinybird.co/docs/about:blank#bitmaphasany)

Checks whether two bitmaps intersect.

If `bitmap2` contains exactly one element, consider using bitmapContains instead as it works more efficiently.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapHasAny(bitmap1, bitmap2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap1`   – Bitmap object 1.
- `bitmap2`   – Bitmap object 2.

### Return values [¶](https://www.tinybird.co/docs/about:blank#return-values)

- `1`   , if `bitmap1`   and `bitmap2`   have at least one shared element.
- `0`   , otherwise.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapHasAny(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│  1  │
└─────┘
## bitmapHasAll [¶](https://www.tinybird.co/docs/about:blank#bitmaphasall)

Returns 1 if the first bitmap contains all elements of the second bitmap, otherwise 0. If the second bitmap is empty, returns 1.

Also see `hasAll(array, array)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapHasAll(bitmap1, bitmap2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap1`   – Bitmap object 1.
- `bitmap2`   – Bitmap object 2.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapHasAll(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│  0  │
└─────┘
## bitmapCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapcardinality)

Returns the cardinality of a bitmap.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapCardinality(bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapCardinality(bitmapBuild([1, 2, 3, 4, 5])) AS res Result:

┌─res─┐
│   5 │
└─────┘
## bitmapMin [¶](https://www.tinybird.co/docs/about:blank#bitmapmin)

Computes the smallest bit set in a bitmap, or UINT32_MAX if the bitmap is empty.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapMin(bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapMin(bitmapBuild([1, 2, 3, 4, 5])) AS res Result:

┌─res─┐
 │   1 │
 └─────┘
## bitmapMax [¶](https://www.tinybird.co/docs/about:blank#bitmapmax)

Computes the greatest bit set in a bitmap, or 0 if the bitmap is empty.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapMax(bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapMax(bitmapBuild([1, 2, 3, 4, 5])) AS res Result:

┌─res─┐
 │   5 │
 └─────┘
## bitmapTransform [¶](https://www.tinybird.co/docs/about:blank#bitmaptransform)

Replaces at most N bits in a bitmap. The old and new value of the i-th replaced bit is given by `from_array[i]` and `to_array[i]`.

The result depends on the array ordering if `from_array` and `to_array`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapTransform(bitmap, from_array, to_array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.
- `from_array`   – UInt32 array. For idx in range [0, from_array.size()), if bitmap contains from_array[idx], then replace it with to_array[idx].
- `to_array`   – UInt32 array with the same size as `from_array`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapTransform(bitmapBuild([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), cast([5,999,2] as Array(UInt32)), cast([2,888,20] as Array(UInt32)))) AS res Result:

┌─res───────────────────┐
 │ [1,3,4,6,7,8,9,10,20] │
 └───────────────────────┘
## bitmapAnd [¶](https://www.tinybird.co/docs/about:blank#bitmapand)

Computes the logical conjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAnd(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapAnd(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res─┐
│ [3] │
└─────┘
## bitmapOr [¶](https://www.tinybird.co/docs/about:blank#bitmapor)

Computes the logical disjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapOr(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapOr(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res─────────┐
│ [1,2,3,4,5] │
└─────────────┘
## bitmapXor [¶](https://www.tinybird.co/docs/about:blank#bitmapxor)

Xor-s two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapXor(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapXor(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res───────┐
│ [1,2,4,5] │
└───────────┘
## bitmapAndnot [¶](https://www.tinybird.co/docs/about:blank#bitmapandnot)

Computes the logical conjunction of two bitmaps and negates the result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAndnot(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapToArray(bitmapAndnot(bitmapBuild([1,2,3]),bitmapBuild([3,4,5]))) AS res Result:

┌─res───┐
│ [1,2] │
└───────┘
## bitmapAndCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapandcardinality)

Returns the cardinality of the logical conjunction of two bitmaps.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitmapAndCardinality(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapAndCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   1 │
└─────┘
## bitmapOrCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmaporcardinality)

Returns the cardinality of the logical disjunction of two bitmaps.

bitmapOrCardinality(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapOrCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   5 │
└─────┘
## bitmapXorCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapxorcardinality)

Returns the cardinality of the XOR of two bitmaps.

bitmapXorCardinality(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapXorCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   4 │
└─────┘
## bitmapAndnotCardinality [¶](https://www.tinybird.co/docs/about:blank#bitmapandnotcardinality)

Returns the cardinality of the AND-NOT operation of two bitmaps.

bitmapAndnotCardinality(bitmap,bitmap)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `bitmap`   – Bitmap object.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT bitmapAndnotCardinality(bitmapBuild([1,2,3]),bitmapBuild([3,4,5])) AS res Result:

┌─res─┐
│   2 │
└─────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/bit-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Bitfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for bitwise operations."
---




# Bit functions [¶](https://www.tinybird.co/docs/about:blank#bit-functions)

The following functions are used to perform bitwise operations.

Bit functions work for any pair of types from `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int8`, `Int16`, `Int32`, `Int64`, `Float32` , or `Float64` . Some functions support `String` and `FixedString` types.

The result type is an integer with bits equal to the maximum bits of its arguments. If at least one of the arguments is signed, the result is a signed number. If an argument is a floating-point number, it's cast to Int64.

- bitAnd(a, b)
- bitOr(a, b)
- bitXor(a, b)
- bitNot(a)

## bitShiftLeft(a, b) [¶](https://www.tinybird.co/docs/about:blank#bitshiftlefta-b)

The following functions shift the binary representation of a value to the left or right by a specified number of bit positions.

A `FixedString` or a `String` is treated as a single multibyte value.

Bits of a `FixedString` value are lost as they are shifted out. On the contrary, a `String` value is extended with additional bytes, so no bits are lost.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitShiftLeft(a, b)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `a`   : A value to shift. Integer types, String or FixedString.
- `b`   : The number of shift positions. Unsigned integer types, 64 bit types or less are allowed.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Shifted value.

The type of the returned value is the same as the type of the input value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

In the following queries bin and hex functions are used to show bits of shifted values.

SELECT 99 AS a, bin(a), bitShiftLeft(a, 2) AS a_shifted, bin(a_shifted)SELECT 'abc' AS a, hex(a), bitShiftLeft(a, 4) AS a_shifted, hex(a_shifted)SELECT toFixedString('abc', 3) AS a, hex(a), bitShiftLeft(a, 4) AS a_shifted, hex(a_shifted) Result:

┌──a─┬─bin(99)──┬─a_shifted─┬─bin(bitShiftLeft(99, 2))─┐
│ 99 │ 01100011 │       140 │ 10001100                 │
└────┴──────────┴───────────┴──────────────────────────┘
┌─a───┬─hex('abc')─┬─a_shifted─┬─hex(bitShiftLeft('abc', 4))─┐
│ abc │ 616263     │ &0        │ 06162630                    │
└─────┴────────────┴───────────┴─────────────────────────────┘
┌─a───┬─hex(toFixedString('abc', 3))─┬─a_shifted─┬─hex(bitShiftLeft(toFixedString('abc', 3), 4))─┐
│ abc │ 616263                       │ &0        │ 162630                                        │
└─────┴──────────────────────────────┴───────────┴───────────────────────────────────────────────┘
## bitShiftRight(a, b) [¶](https://www.tinybird.co/docs/about:blank#bitshiftrighta-b)

Shifts the binary representation of a value to the right by a specified number of bit positions.

A `FixedString` or a `String` is treated as a single multibyte value. Note that the length of a `String` value is reduced as bits are shifted out.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitShiftRight(a, b)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `a`   : A value to shift. Integer types, String or FixedString.
- `b`   : The number of shift positions. Unsigned integer types, 64 bit types or less are allowed.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Shifted value.

The type of the returned value is the same as the type of the input value.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT 101 AS a, bin(a), bitShiftRight(a, 2) AS a_shifted, bin(a_shifted)SELECT 'abc' AS a, hex(a), bitShiftRight(a, 12) AS a_shifted, hex(a_shifted)SELECT toFixedString('abc', 3) AS a, hex(a), bitShiftRight(a, 12) AS a_shifted, hex(a_shifted) Result:

┌───a─┬─bin(101)─┬─a_shifted─┬─bin(bitShiftRight(101, 2))─┐
│ 101 │ 01100101 │        25 │ 00011001                   │
└─────┴──────────┴───────────┴────────────────────────────┘
┌─a───┬─hex('abc')─┬─a_shifted─┬─hex(bitShiftRight('abc', 12))─┐
│ abc │ 616263     │           │ 0616                          │
└─────┴────────────┴───────────┴───────────────────────────────┘
┌─a───┬─hex(toFixedString('abc', 3))─┬─a_shifted─┬─hex(bitShiftRight(toFixedString('abc', 3), 12))─┐
│ abc │ 616263                       │           │ 000616                                          │
└─────┴──────────────────────────────┴───────────┴─────────────────────────────────────────────────┘
## bitRotateLeft(a, b) [¶](https://www.tinybird.co/docs/about:blank#bitrotatelefta-b)

## bitRotateRight(a, b) [¶](https://www.tinybird.co/docs/about:blank#bitrotaterighta-b)

## bitSlice(s, offset, length) [¶](https://www.tinybird.co/docs/about:blank#bitslices-offset-length)

Returns a substring starting with the bit from the ‘offset’ index that is ‘length’ bits long. bits indexing starts from 1

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitSlice(s, offset[, length])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `s`   : s is String or FixedString.
- `offset`   : The start index with bit, A positive value indicates an offset on the left, and a negative value is an indent on the right. Numbering of the bits begins with 1.
- `length`   : The length of substring with bit. If you specify a negative value, the function returns an open substring [offset, array_length - length]. If you omit the value, the function returns the substring [offset, the_end_string]. If length exceeds s, it will be truncate.If length isn't multiple of 8, will fill 0 on the right.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The substring. String

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select bin('Hello'), bin(bitSlice('Hello', 1, 8))
select bin('Hello'), bin(bitSlice('Hello', 1, 2))
select bin('Hello'), bin(bitSlice('Hello', 1, 9))
select bin('Hello'), bin(bitSlice('Hello', -4, 8)) Result:

┌─bin('Hello')─────────────────────────────┬─bin(bitSlice('Hello', 1, 8))─┐
│ 0100100001100101011011000110110001101111 │ 01001000                     │
└──────────────────────────────────────────┴──────────────────────────────┘
┌─bin('Hello')─────────────────────────────┬─bin(bitSlice('Hello', 1, 2))─┐
│ 0100100001100101011011000110110001101111 │ 01000000                     │
└──────────────────────────────────────────┴──────────────────────────────┘
┌─bin('Hello')─────────────────────────────┬─bin(bitSlice('Hello', 1, 9))─┐
│ 0100100001100101011011000110110001101111 │ 0100100000000000             │
└──────────────────────────────────────────┴──────────────────────────────┘
┌─bin('Hello')─────────────────────────────┬─bin(bitSlice('Hello', -4, 8))─┐
│ 0100100001100101011011000110110001101111 │ 11110000                      │
└──────────────────────────────────────────┴───────────────────────────────┘
## bitTest [¶](https://www.tinybird.co/docs/about:blank#bittest)

Takes any integer and converts it into binary form, returns the value of a bit at specified position. Counting is right-to-left, starting at 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT bitTest(number, index)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `number`   – Integer number.
- `index`   – Position of bit.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Value of the bit at the specified position. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

For example, the number 43 in base-2 (binary) numeral system is 101011.

Query:

SELECT bitTest(43, 1) Result:

┌─bitTest(43, 1)─┐
│              1 │
└────────────────┘ Another example:

Query:

SELECT bitTest(43, 2) Result:

┌─bitTest(43, 2)─┐
│              0 │
└────────────────┘
## bitTestAll [¶](https://www.tinybird.co/docs/about:blank#bittestall)

Returns result of logical conjunction (AND operator) of all bits at given positions. Counting is right-to-left, starting at 0.

The conjunction for bit-wise operations:

0 AND 0 = 0

0 AND 1 = 0

1 AND 0 = 0

1 AND 1 = 1

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT bitTestAll(number, index1, index2, index3, index4, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `number`   – Integer number.
- `index1`  , `index2`  , `index3`  , `index4`   – Positions of bit. For example, for set of positions ( `index1`  , `index2`  , `index3`  , `index4`   ) is true if and only if all of its positions are true ( `index1`   ⋀ `index2`   , ⋀ `index3`   ⋀ `index4`   ).

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Result of the logical conjunction. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

For example, the number 43 in base-2 (binary) numeral system is 101011.

Query:

SELECT bitTestAll(43, 0, 1, 3, 5) Result:

┌─bitTestAll(43, 0, 1, 3, 5)─┐
│                          1 │
└────────────────────────────┘ Another example:

Query:

SELECT bitTestAll(43, 0, 1, 3, 5, 2) Result:

┌─bitTestAll(43, 0, 1, 3, 5, 2)─┐
│                             0 │
└───────────────────────────────┘
## bitTestAny [¶](https://www.tinybird.co/docs/about:blank#bittestany)

Returns result of logical disjunction (OR operator) of all bits at given positions. Counting is right-to-left, starting at 0.

The disjunction for bit-wise operations:

0 OR 0 = 0

0 OR 1 = 1

1 OR 0 = 1

1 OR 1 = 1

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT bitTestAny(number, index1, index2, index3, index4, ...)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `number`   – Integer number.
- `index1`  , `index2`  , `index3`  , `index4`   – Positions of bit.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Result of the logical disjunction. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

For example, the number 43 in base-2 (binary) numeral system is 101011.

Query:

SELECT bitTestAny(43, 0, 2) Result:

┌─bitTestAny(43, 0, 2)─┐
│                    1 │
└──────────────────────┘ Another example:

Query:

SELECT bitTestAny(43, 4, 2) Result:

┌─bitTestAny(43, 4, 2)─┐
│                    0 │
└──────────────────────┘
## bitCount [¶](https://www.tinybird.co/docs/about:blank#bitcount)

Calculates the number of bits set to one in the binary representation of a number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitCount(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Integer or floating-point number. The function uses the value representation in memory. It allows supporting floating-point numbers.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Number of bits set to one in the input number. UInt8.

The function doesn't convert the input value to a larger type (sign extension). So, for example, `bitCount(toUInt8(-1)) = 8`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Take for example the number 333. Its binary representation: 0000000101001101.

Query:

SELECT bitCount(333) Result:

┌─bitCount(333)─┐
│             5 │
└───────────────┘
## bitHammingDistance [¶](https://www.tinybird.co/docs/about:blank#bithammingdistance)

Returns the Hamming Distance between the bit representations of two integer values. Can be used with SimHash functions for detection of semi-duplicate strings. The smaller is the distance, the more likely those strings are the same.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

bitHammingDistance(int1, int2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `int1`   : First integer value. Int64.
- `int2`   : Second integer value. Int64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The Hamming distance. UInt8.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT bitHammingDistance(111, 121) Result:

┌─bitHammingDistance(111, 121)─┐
│                            3 │
└──────────────────────────────┘ With SimHash:

SELECT bitHammingDistance(ngramSimHash('cat ate rat'), ngramSimHash('rat ate cat')) Result:

┌─bitHammingDistance(ngramSimHash('cat ate rat'), ngramSimHash('rat ate cat'))─┐
│                                                                            5 │
└──────────────────────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/array-join
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Array Join Function · Tinybird Docs"
theme-color: "#171612"
description: "Functions for joining arrays into rows."
---




# arrayJoin function [¶](https://www.tinybird.co/docs/about:blank#arrayjoin-function)

The `arrayJoin` function takes each row and generates a set of rows (unfold).

This function takes an array as an argument, and propagates the source row to multiple rows for the number of elements in the array. All the values in columns are simply copied, except the values in the column where this function is applied; it's replaced with the corresponding array value.

Example:

SELECT arrayJoin([1, 2, 3] AS src) AS dst, 'Hello', src ┌─dst─┬─\'Hello\'─┬─src─────┐
│   1 │ Hello     │ [1,2,3] │
│   2 │ Hello     │ [1,2,3] │
│   3 │ Hello     │ [1,2,3] │
└─────┴───────────┴─────────┘ The `arrayJoin` function affects all sections of the query, including the `WHERE` section. Notice the result 2, even though the subquery returned 1 row.

Example:

SELECT sum(1) AS impressions
FROM
(
    SELECT ['Istanbul', 'Berlin', 'Bobruisk'] AS cities
)
WHERE arrayJoin(cities) IN ['Istanbul', 'Berlin'] ┌─impressions─┐
│           2 │
└─────────────┘ A query can use multiple `arrayJoin` functions. In this case, the transformation is performed multiple times and the rows are multiplied.

Example:

SELECT
    sum(1) AS impressions,
    arrayJoin(cities) AS city,
    arrayJoin(browsers) AS browser
FROM
(
    SELECT
        ['Istanbul', 'Berlin', 'Bobruisk'] AS cities,
        ['Firefox', 'Chrome', 'Chrome'] AS browsers
)
GROUP BY
    2,
    3 ┌─impressions─┬─city─────┬─browser─┐
│           2 │ Istanbul │ Chrome  │
│           1 │ Istanbul │ Firefox │
│           2 │ Berlin   │ Chrome  │
│           1 │ Berlin   │ Firefox │
│           2 │ Bobruisk │ Chrome  │
│           1 │ Bobruisk │ Firefox │
└─────────────┴──────────┴─────────┘ Using multiple `arrayJoin` with same expression might not produce expected results due to optimizations. For those cases, consider modifying repeated array expression with extra operations that don't affect join result, for example `arrayJoin(arraySort(arr))`, `arrayJoin(arrayConcat(arr, []))`

Example:

SELECT
    arrayJoin(dice) as first_throw,
    /* arrayJoin(dice) as second_throw */ -- is technically correct, but will annihilate result set
    arrayJoin(arrayConcat(dice, [])) as second_throw -- intentionally changed expression to force re-evaluation
FROM (
    SELECT [1, 2, 3, 4, 5, 6] as dice
) Note the `ARRAY JOIN` syntax in the `SELECT` query, which provides broader possibilities. `ARRAY JOIN` allows you to convert multiple arrays with the same number of elements at a time.

Example:

SELECT
    sum(1) AS impressions,
    city,
    browser
FROM
(
    SELECT
        ['Istanbul', 'Berlin', 'Bobruisk'] AS cities,
        ['Firefox', 'Chrome', 'Chrome'] AS browsers
)
ARRAY JOIN
    cities AS city,
    browsers AS browser
GROUP BY
    2,
    3 ┌─impressions─┬─city─────┬─browser─┐
│           1 │ Istanbul │ Firefox │
│           1 │ Berlin   │ Chrome  │
│           1 │ Bobruisk │ Chrome  │
└─────────────┴──────────┴─────────┘ Or you can use Tuple. For example:

SELECT
    sum(1) AS impressions,
    (arrayJoin(arrayZip(cities, browsers)) AS t).1 AS city,
    t.2 AS browser
FROM
(
    SELECT
        ['Istanbul', 'Berlin', 'Bobruisk'] AS cities,
        ['Firefox', 'Chrome', 'Chrome'] AS browsers
)
GROUP BY
    2,
    3 ┌─impressions─┬─city─────┬─browser─┐
│           1 │ Istanbul │ Firefox │
│           1 │ Berlin   │ Chrome  │
│           1 │ Bobruisk │ Chrome  │
└─────────────┴──────────┴─────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/array-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Arrayfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for manipulating arrays."
---




# Array functions [¶](https://www.tinybird.co/docs/about:blank#array-functions)

The following functions are used to manipulate arrays.

## empty [¶](https://www.tinybird.co/docs/about:blank#empty)

Checks whether the input array is empty.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

empty([x]) An array is considered empty if it doesn't contain any elements.

Can be optimized by enabling the `optimize_functions_to_subcolumns` setting. With `optimize_functions_to_subcolumns = 1` the function reads only size0 subcolumn instead of reading and processing the whole array column. The query `SELECT empty(arr) FROM TABLE` transforms to `SELECT arr.size0 = 0 FROM TABLE`.

The function also works for strings or UUID.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `[x]`   : Input array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `1`   for an empty array or `0`   for a non-empty array. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT empty([]) Result:

┌─empty(array())─┐
│              1 │
└────────────────┘
## notEmpty [¶](https://www.tinybird.co/docs/about:blank#notempty)

Checks whether the input array is non-empty.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

notEmpty([x]) An array is considered non-empty if it contains at least one element.

Can be optimized by enabling the optimize_functions_to_subcolumns setting. With `optimize_functions_to_subcolumns = 1` the function reads only size0 subcolumn instead of reading and processing the whole array column. The query `SELECT notEmpty(arr) FROM table` transforms to `SELECT arr.size0 != 0 FROM TABLE`.

The function also works for strings or UUID.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `[x]`   : Input array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns `1`   for a non-empty array or `0`   for an empty array. UInt8.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT notEmpty([1,2]) Result:

┌─notEmpty([1, 2])─┐
│                1 │
└──────────────────┘
## length [¶](https://www.tinybird.co/docs/about:blank#length)

Returns the number of items in the array. The result type is UInt64. The function also works for strings.

Can be optimized by enabling the optimize_functions_to_subcolumns setting. With `optimize_functions_to_subcolumns = 1` the function reads only size0 subcolumn instead of reading and processing the whole array column. The query `SELECT length(arr) FROM table` transforms to `SELECT arr.size0 FROM TABLE`.

Alias: `OCTET_LENGTH`

## emptyArrayUInt8 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayuint8)

Returns an empty UInt8 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayUInt8()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayUInt8() Result:

[]
## emptyArrayUInt16 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayuint16)

Returns an empty UInt16 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayUInt16()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayUInt16() Result:

[]
## emptyArrayUInt32 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayuint32)

Returns an empty UInt32 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayUInt32()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayUInt32() Result:

[]
## emptyArrayUInt64 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayuint64)

Returns an empty UInt64 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayUInt64()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayUInt64() Result:

[]
## emptyArrayInt8 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayint8)

Returns an empty Int8 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayInt8()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayInt8() Result:

[]
## emptyArrayInt16 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayint16)

Returns an empty Int16 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayInt16()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayInt16() Result:

[]
## emptyArrayInt32 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayint32)

Returns an empty Int32 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayInt32()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayInt32() Result:

[]
## emptyArrayInt64 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayint64)

Returns an empty Int64 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayInt64()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayInt64() Result:

[]
## emptyArrayFloat32 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayfloat32)

Returns an empty Float32 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayFloat32()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayFloat32() Result:

[]
## emptyArrayFloat64 [¶](https://www.tinybird.co/docs/about:blank#emptyarrayfloat64)

Returns an empty Float64 array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayFloat64()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayFloat64() Result:

[]
## emptyArrayDate [¶](https://www.tinybird.co/docs/about:blank#emptyarraydate)

Returns an empty Date array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayDate()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayDate()
## emptyArrayDateTime [¶](https://www.tinybird.co/docs/about:blank#emptyarraydatetime)

Returns an empty DateTime array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

[]
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayDateTime() Result:

[]
## emptyArrayString [¶](https://www.tinybird.co/docs/about:blank#emptyarraystring)

Returns an empty String array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

emptyArrayString()
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

None.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An empty array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT emptyArrayString() Result:

[]
## emptyArrayToSingle [¶](https://www.tinybird.co/docs/about:blank#emptyarraytosingle)

Accepts an empty array and returns a one-element array that is equal to the default value.

## range(end), range([start, ] end [, step]) [¶](https://www.tinybird.co/docs/about:blank#rangeend-rangestart-end-step)

Returns an array of numbers from `start` to `end - 1` by `step` . The supported types are UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

range([start, ] end [, step])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start`   : The first element of the array. Optional, required if `step`   is used. Default value: 0.
- `end`   : The number before which the array is constructed. Required.
- `step`   : Determines the incremental step between each element in the array. Optional. Default value: 1.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of numbers from `start`   to `end - 1`   by `step`  .

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

- All arguments `start`  , `end`  , `step`   must be below data types: `UInt8`  , `UInt16`  , `UInt32`  , `UInt64`  , `Int8`  , `Int16`  , `Int32`  , `Int64`   , as well as elements of the returned array, which's type is a super type of all arguments.
- An exception is thrown if query results in arrays with a total length of more than number of elements specified by the function_range_max_elements_in_block setting.
- Returns Null if any argument has Nullable(Nothing) type. An exception is thrown if any argument has Null value (Nullable(T) type).

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT range(5), range(1, 5), range(1, 5, 2), range(-1, 5, 2) Result:

┌─range(5)────┬─range(1, 5)─┬─range(1, 5, 2)─┬─range(-1, 5, 2)─┐
│ [0,1,2,3,4] │ [1,2,3,4]   │ [1,3]          │ [-1,1,3]        │
└─────────────┴─────────────┴────────────────┴─────────────────┘
## array(x1, ...), operator [x1, ...] [¶](https://www.tinybird.co/docs/about:blank#arrayx1-operator-x1)

Creates an array from the function arguments. The arguments must be constants and have types that have the smallest common type. At least one argument must be passed, because otherwise it isn't clear which type of array to create. That is, you can't use this function to create an empty array (to do that, use the 'emptyArray*' function described above). Returns an 'Array(T)' type result, where 'T' is the smallest common type out of the passed arguments.

## arrayWithConstant(length, elem) [¶](https://www.tinybird.co/docs/about:blank#arraywithconstantlength-elem)

Creates an array of length `length` filled with the constant `elem`.

## arrayConcat [¶](https://www.tinybird.co/docs/about:blank#arrayconcat)

Combines arrays passed as arguments.

arrayConcat(arrays)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arrays`   – Arbitrary number of arguments of Array type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayConcat([1, 2], [3, 4], [5, 6]) AS res ┌─res───────────┐
│ [1,2,3,4,5,6] │
└───────────────┘
## arrayElement(arr, n), operator arr[n] [¶](https://www.tinybird.co/docs/about:blank#arrayelementarr-n-operator-arrn)

Get the element with the index `n` from the array `arr`. `n` must be any integer type. Indexes in an array begin from one.

Negative indexes are supported. In this case, it selects the corresponding element numbered from the end. For example, `arr[-1]` is the last item in the array.

If the index falls outside of the bounds of an array, it returns some default value (0 for numbers, an empty string for strings, etc.), except for the case with a non-constant array and a constant index 0 (in this case there will be an error `Array indices are 1-based` ).

## has(arr, elem) [¶](https://www.tinybird.co/docs/about:blank#hasarr-elem)

Checks whether the 'arr' array has the 'elem' element. Returns 0 if the element isn't in the array, or 1 if it's.

`NULL` is processed as a value.

SELECT has([1, 2, NULL], NULL) ┌─has([1, 2, NULL], NULL)─┐
│                       1 │
└─────────────────────────┘
## hasAll [¶](https://www.tinybird.co/docs/about:blank#hasall)

Checks whether one array is a subset of another.

hasAll(set, subset)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `set`   – Array of any type with a set of elements.
- `subset`   – Array of any type that shares a common supertype with `set`   containing elements that should be tested to be a subset of `set`  .

### Return values [¶](https://www.tinybird.co/docs/about:blank#return-values)

- `1`   , if `set`   contains all of the elements from `subset`  .
- `0`   , otherwise.

Raises an exception `NO_COMMON_TYPE` if the set and subset elements don't share a common supertype.

### Properties [¶](https://www.tinybird.co/docs/about:blank#properties)

- An empty array is a subset of any array.
- `Null`   processed as a value.
- Order of values in both of arrays doesn't matter.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

`SELECT hasAll([], [])` returns 1.

`SELECT hasAll([1, Null], [Null])` returns 1.

`SELECT hasAll([1.0, 2, 3, 4], [1, 3])` returns 1.

`SELECT hasAll(['a', 'b'], ['a'])` returns 1.

`SELECT hasAll([1], ['a'])` raises a `NO_COMMON_TYPE` exception.

`SELECT hasAll([[1, 2], [3, 4]], [[1, 2], [3, 5]])` returns 0.

## hasAny [¶](https://www.tinybird.co/docs/about:blank#hasany)

Checks whether two arrays have intersection by some elements.

hasAny(array1, array2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array1`   – Array of any type with a set of elements.
- `array2`   – Array of any type that shares a common supertype with `array1`  .

### Return values [¶](https://www.tinybird.co/docs/about:blank#return-values)

- `1`   , if `array1`   and `array2`   have one similar element at least.
- `0`   , otherwise.

Raises an exception `NO_COMMON_TYPE` if the array1 and array2 elements don't share a common supertype.

### Properties [¶](https://www.tinybird.co/docs/about:blank#properties)

- `Null`   processed as a value.
- Order of values in both of arrays doesn't matter.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

`SELECT hasAny([1], [])` returns `0`.

`SELECT hasAny([Null], [Null, 1])` returns `1`.

`SELECT hasAny([-128, 1., 512], [1])` returns `1`.

`SELECT hasAny([[1, 2], [3, 4]], ['a', 'c'])` raises a `NO_COMMON_TYPE` exception.

`SELECT hasAll([[1, 2], [3, 4]], [[1, 2], [1, 2]])` returns `1`.

## hasSubstr [¶](https://www.tinybird.co/docs/about:blank#hassubstr)

Checks whether all the elements of array2 appear in array1 in the same exact order. Therefore, the function will return 1, if and only if `array1 = prefix + array2 + suffix`.

hasSubstr(array1, array2) In other words, the functions will check whether all the elements of `array2` are contained in `array1` like the `hasAll` function. In addition, it will check that the elements are observed in the same order in both `array1` and `array2`.

For Example:

- `hasSubstr([1,2,3,4], [2,3])`   returns 1. However, `hasSubstr([1,2,3,4], [3,2])`   will return `0`  .
- `hasSubstr([1,2,3,4], [1,2,3])`   returns 1. However, `hasSubstr([1,2,3,4], [1,2,4])`   will return `0`  .

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array1`   – Array of any type with a set of elements.
- `array2`   – Array of any type with a set of elements.

### Return values [¶](https://www.tinybird.co/docs/about:blank#return-values)

- `1`   , if `array1`   contains `array2`  .
- `0`   , otherwise.

Raises an exception `NO_COMMON_TYPE` if the array1 and array2 elements don't share a common supertype.

### Properties [¶](https://www.tinybird.co/docs/about:blank#properties)

- The function will return `1`   if `array2`   is empty.
- `Null`   processed as a value. In other words `hasSubstr([1, 2, NULL, 3, 4], [2,3])`   will return `0`   . However, `hasSubstr([1, 2, NULL, 3, 4], [2,NULL,3])`   will return `1`
- Order of values in both of arrays does matter.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

`SELECT hasSubstr([], [])` returns 1.

`SELECT hasSubstr([1, Null], [Null])` returns 1.

`SELECT hasSubstr([1.0, 2, 3, 4], [1, 3])` returns 0.

`SELECT hasSubstr(['a', 'b'], ['a'])` returns 1.

`SELECT hasSubstr(['a', 'b' , 'c'], ['a', 'b'])` returns 1.

`SELECT hasSubstr(['a', 'b' , 'c'], ['a', 'c'])` returns 0.

`SELECT hasSubstr([[1, 2], [3, 4], [5, 6]], [[1, 2], [3, 4]])` returns 1. i `SELECT hasSubstr([1, 2, NULL, 3, 4], ['a'])` raises a `NO_COMMON_TYPE` exception.

## indexOf(arr, x) [¶](https://www.tinybird.co/docs/about:blank#indexofarr-x)

Returns the index of the first 'x' element (starting from 1) if it's in the array, or 0 if it's not.

Example:

SELECT indexOf([1, 3, NULL, NULL], NULL) ┌─indexOf([1, 3, NULL, NULL], NULL)─┐
│                                 3 │
└───────────────────────────────────┘ Elements set to `NULL` are handled as normal values.

## arrayCount([func,] arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraycountfunc-arr1)

Returns the number of elements for which `func(arr1[i], ..., arrN[i])` returns something other than 0. If `func` isn't specified, it returns the number of non-zero elements in the array.

Note that the `arrayCount` is a higher-order function. You can pass a lambda function to it as the first argument.

## arrayDotProduct [¶](https://www.tinybird.co/docs/about:blank#arraydotproduct)

Returns the dot product of two arrays.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayDotProduct(vector1, vector2) Alias: `scalarProduct`, `dotProduct`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `vector1`   : First vector. Array or Tuple of numeric values.
- `vector2`   : Second vector. Array or Tuple of numeric values.

The sizes of the two vectors must be equal. Arrays and Tuples may also contain mixed element types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The dot product of the two vectors. Numeric.

The return type is determined by the type of the arguments. If Arrays or Tuples contain mixed element types then the result type is the supertype.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayDotProduct([1, 2, 3], [4, 5, 6]) AS res, toTypeName(res) Result:

32	UInt16 Query:

SELECT dotProduct((1::UInt16, 2::UInt8, 3::Float32),(4::Int16, 5::Float32, 6::UInt8)) AS res, toTypeName(res) Result:

32	Float64
## countEqual(arr, x) [¶](https://www.tinybird.co/docs/about:blank#countequalarr-x)

Returns the number of elements in the array equal to x. Equivalent to arrayCount (elem -> elem = x, arr).

`NULL` elements are handled as separate values.

Example:

SELECT countEqual([1, 2, NULL, NULL], NULL) ┌─countEqual([1, 2, NULL, NULL], NULL)─┐
│                                    2 │
└──────────────────────────────────────┘
## arrayEnumerate(arr) [¶](https://www.tinybird.co/docs/about:blank#arrayenumeratearr)

Returns the array [1, 2, 3, ..., length (arr) ]

This function is normally used with ARRAY JOIN. It allows counting something just once for each array after applying ARRAY JOIN. Example:

SELECT
    count() AS Reaches,
    countIf(num = 1) AS Hits
FROM test.hits
ARRAY JOIN
    GoalsReached,
    arrayEnumerate(GoalsReached) AS num
WHERE CounterID = 160656
LIMIT 10 ┌─Reaches─┬──Hits─┐
│   95606 │ 31406 │
└─────────┴───────┘ In this example, Reaches is the number of conversions (the strings received after applying ARRAY JOIN), and Hits is the number of pageviews (strings before ARRAY JOIN). In this particular case, you can get the same result in an easier way:

SELECT
    sum(length(GoalsReached)) AS Reaches,
    count() AS Hits
FROM test.hits
WHERE (CounterID = 160656) AND notEmpty(GoalsReached) ┌─Reaches─┬──Hits─┐
│   95606 │ 31406 │
└─────────┴───────┘ This function can also be used in higher-order functions. For example, you can use it to get array indexes for elements that match a condition.

## arrayEnumerateUniq(arr, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayenumerateuniqarr)

Returns an array the same size as the source array, indicating for each element what its position is among elements with the same value. For example: arrayEnumerateUniq([10, 20, 10, 30]) = [1, 1, 2, 1].

This function is useful when using ARRAY JOIN and aggregation of array elements. Example:

SELECT
    Goals.ID AS GoalID,
    sum(Sign) AS Reaches,
    sumIf(Sign, num = 1) AS Visits
FROM test.visits
ARRAY JOIN
    Goals,
    arrayEnumerateUniq(Goals.ID) AS num
WHERE CounterID = 160656
GROUP BY GoalID
ORDER BY Reaches DESC
LIMIT 10 ┌──GoalID─┬─Reaches─┬─Visits─┐
│   53225 │    3214 │   1097 │
│ 2825062 │    3188 │   1097 │
│   56600 │    2803 │    488 │
│ 1989037 │    2401 │    365 │
│ 2830064 │    2396 │    910 │
│ 1113562 │    2372 │    373 │
│ 3270895 │    2262 │    812 │
│ 1084657 │    2262 │    345 │
│   56599 │    2260 │    799 │
│ 3271094 │    2256 │    812 │
└─────────┴─────────┴────────┘ In this example, each goal ID has a calculation of the number of conversions, each element in the Goals nested data structure is a goal that was reached, which it's referred to as a conversion, and the number of sessions. Without ARRAY JOIN, you would have counted the number of sessions as sum(Sign), but in this particular case, the rows were multiplied by the nested Goals structure, so to count each session one time after this, you can apply a condition to the value of the arrayEnumerateUniq(Goals.ID) function.

The arrayEnumerateUniq function can take multiple arrays of the same size as arguments. In this case, uniqueness is considered for tuples of elements in the same positions in all the arrays.

SELECT arrayEnumerateUniq([1, 1, 1, 2, 2, 2], [1, 1, 2, 1, 1, 2]) AS res ┌─res───────────┐
│ [1,2,1,1,2,1] │
└───────────────┘ This is necessary when using ARRAY JOIN with a nested data structure and further aggregation across multiple elements in this structure.

## arrayEnumerateUniqRanked [¶](https://www.tinybird.co/docs/about:blank#arrayenumerateuniqranked)

Returns an array the same size as the source array, indicating for each element what its position is among elements with the same value. It allows for enumeration of a multidimensional array with the ability to specify how deep to look inside the array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayEnumerateUniqRanked(clear_depth, arr, max_array_depth)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `clear_depth`   : Enumerate elements at the specified level separately. Positive Integer less than or equal to `max_arr_depth`  .
- `arr`   : N-dimensional array to enumerate. Array.
- `max_array_depth`   : The maximum effective depth. Positive Integer less than or equal to the depth of `arr`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

With `clear_depth=1` and `max_array_depth=1` , the result of `arrayEnumerateUniqRanked` is identical to that which `arrayEnumerateUniq` would give for the same array.

Query:

SELECT arrayEnumerateUniqRanked(1, [1,2,1], 1) Result:

[1,1,2] In this example, `arrayEnumerateUniqRanked` is used to obtain an array indicating, for each element of the multidimensional array, what its position is among elements of the same value. For the first row of the passed array, `[1,2,3]` , the corresponding result is `[1,1,1]` , indicating that this is the first time `1`, `2` and `3` are encountered. For the second row of the provided array, `[2,2,1]` , the corresponding result is `[2,3,3]` , indicating that `2` is encountered for a second and third time, and `1` is encountered for the second time. Likewise, for the third row of the provided array `[3]` the corresponding result is `[2]` indicating that `3` is encountered for the second time.

Query:

SELECT arrayEnumerateUniqRanked(1, [[1,2,3],[2,2,1],[3]], 2) Result:

[[1,1,1],[2,3,2],[2]] Changing `clear_depth=2` , results in elements being enumerated separately for each row.

Query:

SELECT arrayEnumerateUniqRanked(2, [[1,2,3],[2,2,1],[3]], 2) Result:

[[1,1,1],[1,2,1],[1]]
## arrayPopBack [¶](https://www.tinybird.co/docs/about:blank#arraypopback)

Removes the last item from the array.

arrayPopBack(array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayPopBack([1, 2, 3]) AS res ┌─res───┐
│ [1,2] │
└───────┘
## arrayPopFront [¶](https://www.tinybird.co/docs/about:blank#arraypopfront)

Removes the first item from the array.

arrayPopFront(array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayPopFront([1, 2, 3]) AS res ┌─res───┐
│ [2,3] │
└───────┘
## arrayPushBack [¶](https://www.tinybird.co/docs/about:blank#arraypushback)

Adds one item to the end of the array.

arrayPushBack(array, single_value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.
- `single_value`   – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. Can be `NULL`   . The function adds a `NULL`   element to an array, and the type of array elements converts to `Nullable`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayPushBack(['a'], 'b') AS res ┌─res───────┐
│ ['a','b'] │
└───────────┘
## arrayPushFront [¶](https://www.tinybird.co/docs/about:blank#arraypushfront)

Adds one element to the beginning of the array.

arrayPushFront(array, single_value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.
- `single_value`   – A single value. Only numbers can be added to an array with numbers, and only strings can be added to an array of strings. Can be `NULL`   . The function adds a `NULL`   element to an array, and the type of array elements converts to `Nullable`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arrayPushFront(['b'], 'a') AS res ┌─res───────┐
│ ['a','b'] │
└───────────┘
## arrayResize [¶](https://www.tinybird.co/docs/about:blank#arrayresize)

Changes the length of the array.

arrayResize(array, size[, extender])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   : Array.
- `size`   : Required length of the array.
  - If `size`     is less than the original size of the array, the array is truncated from the right.
- If `size`   is larger than the initial size of the array, the array is extended to the right with `extender`   values or default values for the data type of the array items.
- `extender`   : Value for extending an array. Can be `NULL`  .

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

An array of length `size`.

### Examples of calls [¶](https://www.tinybird.co/docs/about:blank#examples-of-calls)

SELECT arrayResize([1], 3) ┌─arrayResize([1], 3)─┐
│ [1,0,0]             │
└─────────────────────┘ SELECT arrayResize([1], 3, NULL) ┌─arrayResize([1], 3, NULL)─┐
│ [1,NULL,NULL]             │
└───────────────────────────┘
## arraySlice [¶](https://www.tinybird.co/docs/about:blank#arrayslice)

Returns a slice of the array.

arraySlice(array, offset[, length])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array of data.
- `offset`   – Indent from the edge of the array. A positive value indicates an offset on the left, and a negative value is an indent on the right. Numbering of the array items begins with 1.
- `length`   – The length of the required slice. If you specify a negative value, the function returns an open slice `[offset, array_length - length]`   . If you omit the value, the function returns the slice `[offset, the_end_of_array]`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT arraySlice([1, 2, NULL, 4, 5], 2, 3) AS res ┌─res────────┐
│ [2,NULL,4] │
└────────────┘ Array elements set to `NULL` are handled as normal values.

## arrayShingles [¶](https://www.tinybird.co/docs/about:blank#arrayshingles)

Generates an array of "shingles", i.e. consecutive sub-arrays with specified length of the input array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayShingles(array, length)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   : Input array Array.
- `length`   : The length of each shingle.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array of generated shingles. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayShingles([1,2,3,4], 3) as res Result:

┌─res───────────────┐
│ [[1,2,3],[2,3,4]] │
└───────────────────┘
## arraySort([func,] arr, ...) [¶](https://www.tinybird.co/docs/about:blank#arraysortfunc-arr)

Sorts the elements of the `arr` array in ascending order. If the `func` function is specified, sorting order is determined by the result of the `func` function applied to the elements of the array. If `func` accepts multiple arguments, the `arraySort` function is passed several arrays that the arguments of `func` will correspond to. Detailed examples are shown at the end of `arraySort` description.

Example of integer values sorting:

SELECT arraySort([1, 3, 3, 0]) ┌─arraySort([1, 3, 3, 0])─┐
│ [0,1,3,3]               │
└─────────────────────────┘ Example of string values sorting:

SELECT arraySort(['hello', 'world', '!']) ┌─arraySort(['hello', 'world', '!'])─┐
│ ['!','hello','world']              │
└────────────────────────────────────┘ Consider the following sorting order for the `NULL`, `NaN` and `Inf` values:

SELECT arraySort([1, nan, 2, NULL, 3, nan, -4, NULL, inf, -inf]) ┌─arraySort([1, nan, 2, NULL, 3, nan, -4, NULL, inf, -inf])─┐
│ [-inf,-4,1,2,3,inf,nan,nan,NULL,NULL]                     │
└───────────────────────────────────────────────────────────┘
- `-Inf`   values are first in the array.
- `NULL`   values are last in the array.
- `NaN`   values are right before `NULL`  .
- `Inf`   values are right before `NaN`  .

Note that `arraySort` is a higher-order function. You can pass a lambda function to it as the first argument. In this case, sorting order is determined by the result of the lambda function applied to the elements of the array.

Let's consider the following example:

SELECT arraySort((x) -> -x, [1, 2, 3]) as res ┌─res─────┐
│ [3,2,1] │
└─────────┘ For each element of the source array, the lambda function returns the sorting key, that is, [1 –> -1, 2 –> -2, 3 –> -3]. Since the `arraySort` function sorts the keys in ascending order, the result is [3, 2, 1]. Thus, the `(x) –> -x` lambda function sets the descending order in a sorting.

The lambda function can accept multiple arguments. In this case, you need to pass the `arraySort` function several arrays of identical length that the arguments of lambda function will correspond to. The resulting array will consist of elements from the first input array elements from the next input array(s) specify the sorting keys. For example:

SELECT arraySort((x, y) -> y, ['hello', 'world'], [2, 1]) as res ┌─res────────────────┐
│ ['world', 'hello'] │
└────────────────────┘ Here, the elements that are passed in the second array ([2, 1]) define a sorting key for the corresponding element from the source array (['hello', 'world']), that is, ['hello' –> 2, 'world' –> 1]. Since the lambda function doesn't use `x` , actual values of the source array don't affect the order in the result. So, 'hello' will be the second element in the result, and 'world' will be the first.

Other examples are shown below.

SELECT arraySort((x, y) -> y, [0, 1, 2], ['c', 'b', 'a']) as res ┌─res─────┐
│ [2,1,0] │
└─────────┘ SELECT arraySort((x, y) -> -y, [0, 1, 2], [1, 2, 3]) as res ┌─res─────┐
│ [2,1,0] │
└─────────┘ To improve sorting efficiency, the Schwartzian transform is used.

## arrayPartialSort([func,] limit, arr, ...) [¶](https://www.tinybird.co/docs/about:blank#arraypartialsortfunc-limit-arr)

Same as `arraySort` with additional `limit` argument allowing partial sorting. Returns an array of the same size as the original array where elements in range `[1..limit]` are sorted in ascending order. Remaining elements `(limit..N]` shall contain elements in unspecified order.

## arrayReverseSort [¶](https://www.tinybird.co/docs/about:blank#arrayreversesort)

Sorts the elements of the `arr` array in descending order. If the `func` function is specified, `arr` is sorted according to the result of the `func` function applied to the elements of the array, and then the sorted array is reversed. If `func` accepts multiple arguments, the `arrayReverseSort` function is passed several arrays that the arguments of `func` will correspond to. Detailed examples are shown at the end of `arrayReverseSort` description.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayReverseSort([func,] arr, ...) Example of integer values sorting:

SELECT arrayReverseSort([1, 3, 3, 0]) ┌─arrayReverseSort([1, 3, 3, 0])─┐
│ [3,3,1,0]                      │
└────────────────────────────────┘ Example of string values sorting:

SELECT arrayReverseSort(['hello', 'world', '!']) ┌─arrayReverseSort(['hello', 'world', '!'])─┐
│ ['world','hello','!']                     │
└───────────────────────────────────────────┘ Consider the following sorting order for the `NULL`, `NaN` and `Inf` values:

SELECT arrayReverseSort([1, nan, 2, NULL, 3, nan, -4, NULL, inf, -inf]) as res ┌─res───────────────────────────────────┐
│ [inf,3,2,1,-4,-inf,nan,nan,NULL,NULL] │
└───────────────────────────────────────┘
- `Inf`   values are first in the array.
- `NULL`   values are last in the array.
- `NaN`   values are right before `NULL`  .
- `-Inf`   values are right before `NaN`  .

Note that the `arrayReverseSort` is a higher-order function. You can pass a lambda function to it as the first argument. Example is shown below.

SELECT arrayReverseSort((x) -> -x, [1, 2, 3]) as res ┌─res─────┐
│ [1,2,3] │
└─────────┘ The array is sorted in the following way:

1. At first, the source array ([1, 2, 3]) is sorted according to the result of the lambda function applied to the elements of the array. The result is an array [3, 2, 1].
2. Array that is obtained on the previous step, is reversed. So, the final result is [1, 2, 3].

The lambda function can accept multiple arguments. In this case, you need to pass the `arrayReverseSort` function several arrays of identical length that the arguments of lambda function will correspond to. The resulting array will consist of elements from the first input array; elements from the next input array(s) specify the sorting keys. For example:

SELECT arrayReverseSort((x, y) -> y, ['hello', 'world'], [2, 1]) as res ┌─res───────────────┐
│ ['hello','world'] │
└───────────────────┘ In this example, the array is sorted in the following way:

1. At first, the source array (['hello', 'world'] is sorted according to the result of the lambda function applied to the elements of the arrays. The elements that are passed in the second array ([2, 1]), define the sorting keys for corresponding elements from the source array. The result is an array ['world', 'hello'].
2. Array that was sorted on the previous step, is reversed. So, the final result is ['hello', 'world'].

Other examples are shown below.

SELECT arrayReverseSort((x, y) -> y, [4, 3, 5], ['a', 'b', 'c']) AS res ┌─res─────┐
│ [5,3,4] │
└─────────┘ SELECT arrayReverseSort((x, y) -> -y, [4, 3, 5], [1, 2, 3]) AS res ┌─res─────┐
│ [4,3,5] │
└─────────┘
## arrayPartialReverseSort([func,] limit, arr, ...) [¶](https://www.tinybird.co/docs/about:blank#arraypartialreversesortfunc-limit-arr)

Same as `arrayReverseSort` with additional `limit` argument allowing partial sorting. Returns an array of the same size as the original array where elements in range `[1..limit]` are sorted in descending order. Remaining elements `(limit..N]` shall contain elements in unspecified order.

## arrayShuffle [¶](https://www.tinybird.co/docs/about:blank#arrayshuffle)

Returns an array of the same size as the original array containing the elements in shuffled order. Elements are reordered in such a way that each possible permutation of those elements has equal probability of appearance.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayShuffle(arr[, seed])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `arr`   : The array to partially shuffle. Array.
- `seed`   (optional): seed to be used with random number generation. If not provided a random one is used. UInt or Int.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array with elements shuffled.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

This function will not materialize constants.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

In this example, `arrayShuffle` is used without providing a `seed` and will therefore generate one randomly itself.

Query:

SELECT arrayShuffle([1, 2, 3, 4]) Result:

[1,4,2,3] In this example, `arrayShuffle` is provided a `seed` and will produce stable results.

Query:

SELECT arrayShuffle([1, 2, 3, 4], 41) Result:

[3,2,1,4]
## arrayPartialShuffle [¶](https://www.tinybird.co/docs/about:blank#arraypartialshuffle)

Given an input array of cardinality `N` , returns an array of size N where elements in the range `[1...limit]` are shuffled and the remaining elements in the range `(limit...n]` are unshuffled.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayPartialShuffle(arr[, limit[, seed]])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `arr`   : The array size `N`   to partially shuffle. Array.
- `limit`   (optional): The number to limit element swaps to, in the range `[1..N]`   . UInt or Int.
- `seed`   (optional): The seed value to be used with random number generation. If not provided a random one is used. UInt or Int

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array with elements partially shuffled.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

This function will not materialize constants.

The value of `limit` should be in the range `[1..N]` . Values outside of that range are equivalent to performing full arrayShuffle.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayPartialShuffle([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 1) Result:

The order of elements is preserved ( `[2,3,4,5], [7,8,9,10]` ) except for the two shuffled elements `[1, 6]` . No `seed` is provided so the function selects its own randomly.

[6,2,3,4,5,1,7,8,9,10] In this example, the `limit` is increased to `2` and a `seed` value is provided. The order

Query:

SELECT arrayPartialShuffle([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 2) The order of elements is preserved ( `[4, 5, 6, 7, 8], [10]` ) except for the four shuffled elements `[1, 2, 3, 9]`.

Result:

[3,9,1,4,5,6,7,8,2,10]
## arrayUniq(arr, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayuniqarr)

If one argument is passed, it counts the number of different elements in the array. If multiple arguments are passed, it counts the number of different tuples of elements at corresponding positions in multiple arrays.

If you want to get a list of unique items in an array, you can use arrayReduce('groupUniqArray', arr).

## arrayDifference [¶](https://www.tinybird.co/docs/about:blank#arraydifference)

Calculates an array of differences between adjacent array elements. The first element of the result array will be 0, the second `a[1] - a[0]` , the third `a[2] - a[1]` , etc. The type of elements in the result array is determined by the type inference rules for subtraction (e.g. `UInt8` - `UInt8` = `Int16` ).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayDifference(array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Returns an array of differences between adjacent array elements. UInt*, Int*, Float*.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayDifference([1, 2, 3, 4]) Result:

┌─arrayDifference([1, 2, 3, 4])─┐
│ [0,1,1,1]                     │
└───────────────────────────────┘ Example of the overflow due to result type Int64:

Query:

SELECT arrayDifference([0, 10000000000000000000]) Result:

┌─arrayDifference([0, 10000000000000000000])─┐
│ [0,-8446744073709551616]                   │
└────────────────────────────────────────────┘
## arrayDistinct [¶](https://www.tinybird.co/docs/about:blank#arraydistinct)

Takes an array, returns an array containing the distinct elements only.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayDistinct(array)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `array`   – Array.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Returns an array containing the distinct elements.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayDistinct([1, 2, 2, 3, 1]) Result:

┌─arrayDistinct([1, 2, 2, 3, 1])─┐
│ [1,2,3]                        │
└────────────────────────────────┘
## arrayEnumerateDense [¶](https://www.tinybird.co/docs/about:blank#arrayenumeratedense)

Returns an array of the same size as the source array, indicating where each element first appears in the source array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayEnumerateDense(arr)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayEnumerateDense([10, 20, 10, 30]) Result:

┌─arrayEnumerateDense([10, 20, 10, 30])─┐
│ [1,2,1,3]                             │
└───────────────────────────────────────┘
## arrayEnumerateDenseRanked [¶](https://www.tinybird.co/docs/about:blank#arrayenumeratedenseranked)

Returns an array the same size as the source array, indicating where each element first appears in the source array. It allows for enumeration of a multidimensional array with the ability to specify how deep to look inside the array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayEnumerateDenseRanked(clear_depth, arr, max_array_depth)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `clear_depth`   : Enumerate elements at the specified level separately. Positive Integer less than or equal to `max_arr_depth`  .
- `arr`   : N-dimensional array to enumerate. Array.
- `max_array_depth`   : The maximum effective depth. Positive Integer less than or equal to the depth of `arr`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

With `clear_depth=1` and `max_array_depth=1` , the result is identical to what arrayEnumerateDense would give.

Query:

SELECT arrayEnumerateDenseRanked(1,[10, 20, 10, 30],1) Result:

[1,2,1,3] In this example, `arrayEnumerateDenseRanked` is used to obtain an array indicating, for each element of the multidimensional array, what its position is among elements of the same value. For the first row of the passed array, `[10,10,30,20]` , the corresponding first row of the result is `[1,1,2,3]` , indicating that `10` is the first number encountered in position 1 and 2, `30` the second number encountered in position 3 and `20` is the third number encountered in position 4. For the second row, `[40, 50, 10, 30]` , the corresponding second row of the result is `[4,5,1,2]` , indicating that `40` and `50` are the fourth and fifth numbers encountered in position 1 and 2 of that row, that another `10` (the first encountered number) is in position 3 and `30` (the second number encountered) is in the last position.

Query:

SELECT arrayEnumerateDenseRanked(1,[[10,10,30,20],[40,50,10,30]],2) Result:

[[1,1,2,3],[4,5,1,2]] Changing `clear_depth=2` results in the enumeration occurring separately for each row anew.

Query:

SELECT arrayEnumerateDenseRanked(2,[[10,10,30,20],[40,50,10,30]],2) Result:

[[1,1,2,3],[1,2,3,4]]
## arrayIntersect(arr) [¶](https://www.tinybird.co/docs/about:blank#arrayintersectarr)

Takes multiple arrays, returns an array with elements that are present in all source arrays.

Example:

SELECT
    arrayIntersect([1, 2], [1, 3], [2, 3]) AS no_intersect,
    arrayIntersect([1, 2], [1, 3], [1, 4]) AS intersect ┌─no_intersect─┬─intersect─┐
│ []           │ [1]       │
└──────────────┴───────────┘
## arrayJaccardIndex [¶](https://www.tinybird.co/docs/about:blank#arrayjaccardindex)

Returns the Jaccard index of two arrays.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayJaccardIndex([1, 2], [2, 3]) AS res Result:

┌─res────────────────┐
│ 0.3333333333333333 │
└────────────────────┘
## arrayReduce [¶](https://www.tinybird.co/docs/about:blank#arrayreduce)

Applies an aggregate function to array elements and returns its result. The name of the aggregation function is passed as a string in single quotes `'max'`, `'sum'` . When using parametric aggregate functions, the parameter is indicated after the function name in parentheses `'uniqUpTo(6)'`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayReduce(agg_func, arr1, arr2, ..., arrN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `agg_func`   : The name of an aggregate function which should be a constant string.
- `arr`   : Any number of array type columns as the parameters of the aggregation function.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayReduce('max', [1, 2, 3]) Result:

┌─arrayReduce('max', [1, 2, 3])─┐
│                             3 │
└───────────────────────────────┘ If an aggregate function takes multiple arguments, then this function must be applied to multiple arrays of the same size.

Query:

SELECT arrayReduce('maxIf', [3, 5], [1, 0]) Result:

┌─arrayReduce('maxIf', [3, 5], [1, 0])─┐
│                                    3 │
└──────────────────────────────────────┘ Example with a parametric aggregate function:

Query:

SELECT arrayReduce('uniqUpTo(3)', [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]) Result:

┌─arrayReduce('uniqUpTo(3)', [1, 2, 3, 4, 5, 6, 7, 8, 9, 10])─┐
│                                                           4 │
└─────────────────────────────────────────────────────────────┘
## arrayReduceInRanges [¶](https://www.tinybird.co/docs/about:blank#arrayreduceinranges)

Applies an aggregate function to array elements in given ranges and returns an array containing the result corresponding to each range. The function will return the same result as multiple `arrayReduce(agg_func, arraySlice(arr1, index, length), ...)`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayReduceInRanges(agg_func, ranges, arr1, arr2, ..., arrN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `agg_func`   : The name of an aggregate function which should be a constant string.
- `ranges`   : The ranges to aggregate which should be an array of tuples which containing the index and the length of each range.
- `arr`   : Any number of Array type columns as the parameters of the aggregation function.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array containing results of the aggregate function over specified ranges. Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayReduceInRanges(
    'sum',
    [(1, 5), (2, 3), (3, 4), (4, 4)],
    [1000000, 200000, 30000, 4000, 500, 60, 7]
) AS res Result:

┌─res─────────────────────────┐
│ [1234500,234000,34560,4567] │
└─────────────────────────────┘
## arrayFold [¶](https://www.tinybird.co/docs/about:blank#arrayfold)

Applies a lambda function to one or more equally-sized arrays and collects the result in an accumulator.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayFold(lambda_function, arr1, arr2, ..., accumulator)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayFold( acc,x -> acc + x*2,  [1, 2, 3, 4], toInt64(3)) AS res Result:

┌─res─┐
│  23 │
└─────┘
### Example with the Fibonacci sequence [¶](https://www.tinybird.co/docs/about:blank#example-with-the-fibonacci-sequence)

SELECT arrayFold( acc,x -> (acc.2, acc.2 + acc.1), range(number), (1::Int64, 0::Int64)).1 AS fibonacci
FROM numbers(1,10)
┌─fibonacci─┐
│         0 │
│         1 │
│         1 │
│         2 │
│         3 │
│         5 │
│         8 │
│        13 │
│        21 │
│        34 │
└───────────┘
## arrayReverse [¶](https://www.tinybird.co/docs/about:blank#arrayreverse)

Returns an array of the same size as the original array containing the elements in reverse order.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayReverse(arr) Example:

SELECT arrayReverse([1, 2, 3]) ┌─arrayReverse([1, 2, 3])─┐
│ [3,2,1]                 │
└─────────────────────────┘
## reverse(arr) [¶](https://www.tinybird.co/docs/about:blank#reversearr)

Synonym for "arrayReverse"

## arrayFlatten [¶](https://www.tinybird.co/docs/about:blank#arrayflatten)

Converts an array of arrays to a flat array.

Function:

- Applies to any depth of nested arrays.
- Does not change arrays that are already flat.

The flattened array contains all the elements from all source arrays.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

flatten(array_of_arrays) Alias: `flatten`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `array_of_arrays`   : Array of arrays. For example, `[[1,2,3], [4,5]]`  .

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT flatten([[[1]], [[2], [3]]]) ┌─flatten(array(array([1]), array([2], [3])))─┐
│ [1,2,3]                                     │
└─────────────────────────────────────────────┘
## arrayCompact [¶](https://www.tinybird.co/docs/about:blank#arraycompact)

Removes consecutive duplicate elements from an array. The order of result values is determined by the order in the source array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayCompact(arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`arr` : The array to inspect.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The array without duplicate. Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayCompact([1, 1, nan, nan, 2, 3, 3, 3]) Result:

┌─arrayCompact([1, 1, nan, nan, 2, 3, 3, 3])─┐
│ [1,nan,nan,2,3]                            │
└────────────────────────────────────────────┘
## arrayZip [¶](https://www.tinybird.co/docs/about:blank#arrayzip)

Combines multiple arrays into a single array. The resulting array contains the corresponding elements of the source arrays grouped into tuples in the listed order of arguments.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayZip(arr1, arr2, ..., arrN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arrN`   : Array.

The function can take any number of arrays of different types. All the input arrays must be of equal size.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array with elements from the source arrays grouped into tuples. Data types in the tuple are the same as types of the input arrays and in the same order as arrays are passed. Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayZip(['a', 'b', 'c'], [5, 2, 1]) Result:

┌─arrayZip(['a', 'b', 'c'], [5, 2, 1])─┐
│ [('a',5),('b',2),('c',1)]            │
└──────────────────────────────────────┘
## arrayZipUnaligned [¶](https://www.tinybird.co/docs/about:blank#arrayzipunaligned)

Combines multiple arrays into a single array, allowing for unaligned arrays. The resulting array contains the corresponding elements of the source arrays grouped into tuples in the listed order of arguments.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayZipUnaligned(arr1, arr2, ..., arrN)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arrN`   : Array.

The function can take any number of arrays of different types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array with elements from the source arrays grouped into tuples. Data types in the tuple are the same as types of the input arrays and in the same order as arrays are passed. Array. If the arrays have different sizes, the shorter arrays will be padded with `null`   values.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayZipUnaligned(['a'], [1, 2, 3]) Result:

┌─arrayZipUnaligned(['a'], [1, 2, 3])─┐
│ [('a',1),(NULL,2),(NULL,3)]         │
└─────────────────────────────────────┘
## arrayAUC [¶](https://www.tinybird.co/docs/about:blank#arrayauc)

Calculate AUC (Area Under the Curve, which is a concept in machine learning, see more details: [https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve) ).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayAUC(arr_scores, arr_labels[, scale])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr_scores`   : scores prediction model gives.
- `arr_labels`   : labels of samples, usually 1 for positive sample and 0 for negative sample.
- `scale`   - Optional. Wether to return the normalized area. Default value: true. [Bool]

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Returns AUC value with type Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select arrayAUC([0.1, 0.4, 0.35, 0.8], [0, 0, 1, 1]) Result:

┌─arrayAUC([0.1, 0.4, 0.35, 0.8], [0, 0, 1, 1])─┐
│                                          0.75 │
└───────────────────────────────────────────────┘
## arrayMap(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraymapfunc-arr1)

Returns an array obtained from the original arrays by application of `func(arr1[i], ..., arrN[i])` for each element. Arrays `arr1` ... `arrN` must have the same number of elements.

Examples:

SELECT arrayMap(x -> (x + 2), [1, 2, 3]) as res ┌─res─────┐
│ [3,4,5] │
└─────────┘ The following example shows how to create a tuple of elements from different arrays:

SELECT arrayMap((x, y) -> (x, y), [1, 2, 3], [4, 5, 6]) AS res ┌─res─────────────────┐
│ [(1,4),(2,5),(3,6)] │
└─────────────────────┘ Note that the `arrayMap` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayFilter(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayfilterfunc-arr1)

Returns an array containing only the elements in `arr1` for which `func(arr1[i], ..., arrN[i])` returns something other than 0.

Examples:

SELECT arrayFilter(x -> x LIKE '%World%', ['Hello', 'abc World']) AS res ┌─res───────────┐
│ ['abc World'] │
└───────────────┘ SELECT
    arrayFilter(
        (i, x) -> x LIKE '%World%',
        arrayEnumerate(arr),
        ['Hello', 'abc World'] AS arr)
    AS res ┌─res─┐
│ [2] │
└─────┘ Note that the `arrayFilter` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayFill(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayfillfunc-arr1)

Scan through `arr1` from the first element to the last element and replace `arr1[i]` by `arr1[i - 1]` if `func(arr1[i], ..., arrN[i])` returns 0. The first element of `arr1` will not be replaced.

Examples:

SELECT arrayFill(x -> not isNull(x), [1, null, 3, 11, 12, null, null, 5, 6, 14, null, null]) AS res ┌─res──────────────────────────────┐
│ [1,1,3,11,12,12,12,5,6,14,14,14] │
└──────────────────────────────────┘ Note that the `arrayFill` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayReverseFill(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayreversefillfunc-arr1)

Scan through `arr1` from the last element to the first element and replace `arr1[i]` by `arr1[i + 1]` if `func(arr1[i], ..., arrN[i])` returns 0. The last element of `arr1` will not be replaced.

Examples:

SELECT arrayReverseFill(x -> not isNull(x), [1, null, 3, 11, 12, null, null, 5, 6, 14, null, null]) AS res ┌─res────────────────────────────────┐
│ [1,3,3,11,12,5,5,5,6,14,NULL,NULL] │
└────────────────────────────────────┘ Note that the `arrayReverseFill` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arraySplit(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraysplitfunc-arr1)

Split `arr1` into multiple arrays. When `func(arr1[i], ..., arrN[i])` returns something other than 0, the array will be split on the left hand side of the element. The array will not be split before the first element.

Examples:

SELECT arraySplit((x, y) -> y, [1, 2, 3, 4, 5], [1, 0, 0, 1, 0]) AS res ┌─res─────────────┐
│ [[1,2,3],[4,5]] │
└─────────────────┘ Note that the `arraySplit` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayReverseSplit(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayreversesplitfunc-arr1)

Split `arr1` into multiple arrays. When `func(arr1[i], ..., arrN[i])` returns something other than 0, the array will be split on the right hand side of the element. The array will not be split after the last element.

Examples:

SELECT arrayReverseSplit((x, y) -> y, [1, 2, 3, 4, 5], [1, 0, 0, 1, 0]) AS res ┌─res───────────────┐
│ [[1],[2,3,4],[5]] │
└───────────────────┘ Note that the `arrayReverseSplit` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayExists([func,] arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayexistsfunc-arr1)

Returns 1 if there is at least one element in `arr` for which `func(arr1[i], ..., arrN[i])` returns something other than 0. Otherwise, it returns 0.

Note that the `arrayExists` is a higher-order function. You can pass a lambda function to it as the first argument.

## arrayAll([func,] arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayallfunc-arr1)

Returns 1 if `func(arr1[i], ..., arrN[i])` returns something other than 0 for all the elements in arrays. Otherwise, it returns 0.

Note that the `arrayAll` is a higher-order function. You can pass a lambda function to it as the first argument.

## arrayFirst(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayfirstfunc-arr1)

Returns the first element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0.

## arrayFirstOrNull [¶](https://www.tinybird.co/docs/about:blank#arrayfirstornull)

Returns the first element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0, otherwise it returns `NULL`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayFirstOrNull(func, arr1, ...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `func`   : Lambda function. Lambda function.
- `arr1`   : Array to operate on. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The first element in the passed array.
- Otherwise, returns `NULL`

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

Note that the `arrayFirstOrNull` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayFirstOrNull(x -> x >= 2, [1, 2, 3]) Result:

2 Query:

SELECT arrayFirstOrNull(x -> x >= 2, emptyArrayUInt8()) Result:

\N Query:

SELECT arrayLastOrNull((x,f) -> f, [1,2,3,NULL], [0,1,0,1]) Result:

\N
## arrayLast(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraylastfunc-arr1)

Returns the last element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0.

Note that the `arrayLast` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayLastOrNull [¶](https://www.tinybird.co/docs/about:blank#arraylastornull)

Returns the last element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0, otherwise returns `NULL`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayLastOrNull(func, arr1, ...)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `func`   : Lambda function. Lambda function.
- `arr1`   : Array to operate on. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The last element in the passed array.
- Otherwise, returns `NULL`

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

Note that the `arrayLastOrNull` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayLastOrNull(x -> x >= 2, [1, 2, 3]) Result:

3 Query:

SELECT arrayLastOrNull(x -> x >= 2, emptyArrayUInt8()) Result:

\N
## arrayFirstIndex(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arrayfirstindexfunc-arr1)

Returns the index of the first element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0.

Note that the `arrayFirstIndex` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayLastIndex(func, arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraylastindexfunc-arr1)

Returns the index of the last element in the `arr1` array for which `func(arr1[i], ..., arrN[i])` returns something other than 0.

Note that the `arrayLastIndex` is a higher-order function. You must pass a lambda function to it as the first argument, and it can't be omitted.

## arrayMin [¶](https://www.tinybird.co/docs/about:blank#arraymin)

Returns the minimum of elements in the source array.

If the `func` function is specified, returns the mininum of elements converted by this function.

Note that the `arrayMin` is a higher-order function. You can pass a lambda function to it as the first argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayMin([func,] arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `func`   : Function. Expression.
- `arr`   : Array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The minimum of function values (or the array minimum).

If `func` is specified, then the return type matches the return value type of `func` , otherwise it matches the type of the array elements.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayMin([1, 2, 4]) AS res Result:

┌─res─┐
│   1 │
└─────┘ Query:

SELECT arrayMin(x -> (-x), [1, 2, 4]) AS res Result:

┌─res─┐
│  -4 │
└─────┘
## arrayMax [¶](https://www.tinybird.co/docs/about:blank#arraymax)

Returns the maximum of elements in the source array.

If the `func` function is specified, returns the maximum of elements converted by this function.

Note that the `arrayMax` is a higher-order function. You can pass a lambda function to it as the first argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayMax([func,] arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `func`   : Function. Expression.
- `arr`   : Array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The maximum of function values (or the array maximum).

if `func` is specified then the return type matches the return value type of `func` , otherwise it matches the type of the array elements.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayMax([1, 2, 4]) AS res Result:

┌─res─┐
│   4 │
└─────┘ Query:

SELECT arrayMax(x -> (-x), [1, 2, 4]) AS res Result:

┌─res─┐
│  -1 │
└─────┘
## arraySum [¶](https://www.tinybird.co/docs/about:blank#arraysum)

Returns the sum of elements in the source array.

If the `func` function is specified, returns the sum of elements converted by this function.

Note that the `arraySum` is a higher-order function. You can pass a lambda function to it as the first argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arraySum([func,] arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `func`   : Function. Expression.
- `arr`   : Array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The sum of the function values (or the array sum).

Return type:

- For decimal numbers in the source array (or for converted values, if `func`   is specified): Decimal128.
- For floating point numbers: Float64.
- For numeric unsigned: UInt64.
- For numeric signed: Int64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arraySum([2, 3]) AS res Result:

┌─res─┐
│   5 │
└─────┘ Query:

SELECT arraySum(x -> x*x, [2, 3]) AS res Result:

┌─res─┐
│  13 │
└─────┘
## arrayAvg [¶](https://www.tinybird.co/docs/about:blank#arrayavg)

Returns the average of elements in the source array.

If the `func` function is specified, returns the average of elements converted by this function.

Note that the `arrayAvg` is a higher-order function. You can pass a lambda function to it as the first argument.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayAvg([func,] arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `func`   : Function. Expression.
- `arr`   : Array. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The average of function values (or the array average). Float64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayAvg([1, 2, 4]) AS res Result:

┌────────────────res─┐
│ 2.3333333333333335 │
└────────────────────┘ Query:

SELECT arrayAvg(x -> (x * x), [2, 4]) AS res Result:

┌─res─┐
│  10 │
└─────┘
## arrayCumSum([func,] arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraycumsumfunc-arr1)

Returns an array of the partial (running) sums of the elements in the source array `arr1` . If `func` is specified, then the sum is computed from applying `func` to `arr1`, `arr2` , ..., `arrN` , i.e. `func(arr1[i], ..., arrN[i])`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayCumSum(arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array of numeric values.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array of the partial sums of the elements in the source array. UInt*, Int*, Float*.

Example:

SELECT arrayCumSum([1, 1, 1, 1]) AS res ┌─res──────────┐
│ [1, 2, 3, 4] │
└──────────────┘ Note that the `arrayCumSum` is a higher-order function. You can pass a lambda function to it as the first argument.

## arrayCumSumNonNegative([func,] arr1, ...) [¶](https://www.tinybird.co/docs/about:blank#arraycumsumnonnegativefunc-arr1)

Same as `arrayCumSum` , returns an array of the partial (running) sums of the elements in the source array. If `func` is specified, then the sum is computed from applying `func` to `arr1`, `arr2` , ..., `arrN` , i.e. `func(arr1[i], ..., arrN[i])` . Unlike `arrayCumSum` , if the current running sum is smaller than `0` , it's replaced by `0`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayCumSumNonNegative(arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array of numeric values.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns an array of non-negative partial sums of elements in the source array. UInt*, Int*, Float*.

SELECT arrayCumSumNonNegative([1, 1, -4, 1]) AS res ┌─res───────┐
│ [1,2,0,1] │
└───────────┘ Note that the `arraySumNonNegative` is a higher-order function. You can pass a lambda function to it as the first argument.

## arrayProduct [¶](https://www.tinybird.co/docs/about:blank#arrayproduct)

Multiplies elements of an array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayProduct(arr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array of numeric values.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A product of array's elements. Float64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayProduct([1,2,3,4,5,6]) as res Result:

┌─res───┐
│ 720   │
└───────┘ Query:

SELECT arrayProduct([toDecimal64(1,8), toDecimal64(2,8), toDecimal64(3,8)]) as res, toTypeName(res) Return value type is always Float64. Result:

┌─res─┬─toTypeName(arrayProduct(array(toDecimal64(1, 8), toDecimal64(2, 8), toDecimal64(3, 8))))─┐
│ 6   │ Float64                                                                                  │
└─────┴──────────────────────────────────────────────────────────────────────────────────────────┘
## arrayRotateLeft [¶](https://www.tinybird.co/docs/about:blank#arrayrotateleft)

Rotates an array to the left by the specified number of elements. If the number of elements is negative, the array is rotated to the right.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayRotateLeft(arr, n)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array.
- `n`   : Number of elements to rotate.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array rotated to the left by the specified number of elements. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayRotateLeft([1,2,3,4,5,6], 2) as res Result:

┌─res───────────┐
│ [3,4,5,6,1,2] │
└───────────────┘ Query:

SELECT arrayRotateLeft([1,2,3,4,5,6], -2) as res Result:

┌─res───────────┐
│ [5,6,1,2,3,4] │
└───────────────┘ Query:

SELECT arrayRotateLeft(['a','b','c','d','e'], 3) as res Result:

┌─res───────────────────┐
│ ['d','e','a','b','c'] │
└───────────────────────┘
## arrayRotateRight [¶](https://www.tinybird.co/docs/about:blank#arrayrotateright)

Rotates an array to the right by the specified number of elements. If the number of elements is negative, the array is rotated to the left.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayRotateRight(arr, n)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array.
- `n`   : Number of elements to rotate.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array rotated to the right by the specified number of elements. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayRotateRight([1,2,3,4,5,6], 2) as res Result:

┌─res───────────┐
│ [5,6,1,2,3,4] │
└───────────────┘ Query:

SELECT arrayRotateRight([1,2,3,4,5,6], -2) as res Result:

┌─res───────────┐
│ [3,4,5,6,1,2] │
└───────────────┘ Query:

SELECT arrayRotateRight(['a','b','c','d','e'], 3) as res Result:

┌─res───────────────────┐
│ ['c','d','e','a','b'] │
└───────────────────────┘
## arrayShiftLeft [¶](https://www.tinybird.co/docs/about:blank#arrayshiftleft)

Shifts an array to the left by the specified number of elements. New elements are filled with the provided argument or the default value of the array element type. If the number of elements is negative, the array is shifted to the right.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayShiftLeft(arr, n[, default])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array.
- `n`   : Number of elements to shift.
- `default`   : Optional. Default value for new elements.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array shifted to the left by the specified number of elements. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayShiftLeft([1,2,3,4,5,6], 2) as res Result:

┌─res───────────┐
│ [3,4,5,6,0,0] │
└───────────────┘ Query:

SELECT arrayShiftLeft([1,2,3,4,5,6], -2) as res Result:

┌─res───────────┐
│ [0,0,1,2,3,4] │
└───────────────┘ Query:

SELECT arrayShiftLeft([1,2,3,4,5,6], 2, 42) as res Result:

┌─res─────────────┐
│ [3,4,5,6,42,42] │
└─────────────────┘ Query:

SELECT arrayShiftLeft(['a','b','c','d','e','f'], 3, 'foo') as res Result:

┌─res─────────────────────────────┐
│ ['d','e','f','foo','foo','foo'] │
└─────────────────────────────────┘ Query:

SELECT arrayShiftLeft([1,2,3,4,5,6] :: Array(UInt16), 2, 4242) as res Result:

┌─res─────────────────┐
│ [3,4,5,6,4242,4242] │
└─────────────────────┘
## arrayShiftRight [¶](https://www.tinybird.co/docs/about:blank#arrayshiftright)

Shifts an array to the right by the specified number of elements. New elements are filled with the provided argument or the default value of the array element type. If the number of elements is negative, the array is shifted to the left.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayShiftRight(arr, n[, default])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : Array.
- `n`   : Number of elements to shift.
- `default`   : Optional. Default value for new elements.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array shifted to the right by the specified number of elements. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayShiftRight([1,2,3,4,5,6], 2) as res Result:

┌─res───────────┐
│ [0,0,1,2,3,4] │
└───────────────┘ Query:

SELECT arrayShiftRight([1,2,3,4,5,6], -2) as res Result:

┌─res───────────┐
│ [3,4,5,6,0,0] │
└───────────────┘ Query:

SELECT arrayShiftRight([1,2,3,4,5,6], 2, 42) as res Result:

┌─res─────────────┐
│ [42,42,1,2,3,4] │
└─────────────────┘ Query:

SELECT arrayShiftRight(['a','b','c','d','e','f'], 3, 'foo') as res Result:

┌─res─────────────────────────────┐
│ ['foo','foo','foo','a','b','c'] │
└─────────────────────────────────┘ Query:

SELECT arrayShiftRight([1,2,3,4,5,6] :: Array(UInt16), 2, 4242) as res Result:

┌─res─────────────────┐
│ [4242,4242,1,2,3,4] │
└─────────────────────┘
## arrayRandomSample [¶](https://www.tinybird.co/docs/about:blank#arrayrandomsample)

Function `arrayRandomSample` returns a subset with `samples` -many random elements of an input array. If `samples` exceeds the size of the input array, the sample size is limited to the size of the array, i.e. all array elements are returned but their order isn't guaranteed. The function can handle both flat arrays and nested arrays.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

arrayRandomSample(arr, samples)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arr`   : The input array from which to sample elements. (Array(T))
- `samples`   : The number of elements to include in the random sample (UInt*)

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array containing a random sample of elements from the input array. Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT arrayRandomSample(['apple', 'banana', 'cherry', 'date'], 2) as res Result:

┌─res────────────────┐
│ ['cherry','apple'] │
└────────────────────┘ Query:

SELECT arrayRandomSample([[1, 2], [3, 4], [5, 6]], 2) as res Result:

┌─res───────────┐
│ [[3,4],[5,6]] │
└───────────────┘ Query:

SELECT arrayRandomSample([1, 2, 3], 5) as res Result:

┌─res─────┐
│ [3,1,2] │
└─────────┘
## Distance functions [¶](https://www.tinybird.co/docs/about:blank#distance-functions)

All supported functions are described in distance functions documentation.



---

URL: https://www.tinybird.co/docs/sql-reference/functions/arithmetic-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Arithmeticfunctions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for performing arithmetic operations."
---




# Arithmetic functions [¶](https://www.tinybird.co/docs/about:blank#arithmetic-functions)

The following functions are used to perform arithmetic operations.

Arithmetic functions work for any two operands of type `UInt8`, `UInt16`, `UInt32`, `UInt64`, `Int8`, `Int16`, `Int32`, `Int64`, `Float32` , or `Float64`.

Before performing the operation, both operands are cast to the result type. The result type is determined as follows:

- If both operands are up to 32 bits wide, the size of the result type will be the size of the next bigger type following the bigger of the   two operands (integer size promotion). For example, `UInt8 + UInt16 = UInt32`   or `Float32 * Float32 = Float64`  .
- If one of the operands has 64 or more bits, the size of the result type will be the same size as the bigger of the two operands. For   example, `UInt32 + UInt128 = UInt128`   or `Float32 * Float64 = Float64`  .
- If one of the operands is signed, the result type will also be signed, otherwise it will be signed. For example, `UInt32 * Int32 = Int64`  .

These rules make sure that the result type will be the smallest type which can represent all possible results. While this introduces a risk of overflows around the value range boundary, it ensures that calculations are performed quickly using the maximum native integer width of 64 bit. This behavior also guarantees compatibility with many other databases which provide 64 bit integers (BIGINT) as the biggest integer type.

For example:

SELECT toTypeName(0), toTypeName(0 + 0), toTypeName(0 + 0 + 0), toTypeName(0 + 0 + 0 + 0) ┌─toTypeName(0)─┬─toTypeName(plus(0, 0))─┬─toTypeName(plus(plus(0, 0), 0))─┬─toTypeName(plus(plus(plus(0, 0), 0), 0))─┐
│ UInt8         │ UInt16                 │ UInt32                          │ UInt64                                   │
└───────────────┴────────────────────────┴─────────────────────────────────┴──────────────────────────────────────────┘ Overflows are produced the same way as in C++.

## plus [¶](https://www.tinybird.co/docs/about:blank#plus)

Calculates the sum of two values `a` and `b`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

plus(a, b) It is possible to add an integer and a date or date with time. The former operation increments the number of days in the date, the latter operation increments the number of seconds in the date with time.

Alias: `a + b` (operator)

## minus [¶](https://www.tinybird.co/docs/about:blank#minus)

Calculates the difference of two values `a` and `b` . The result is always signed.

Similar to `plus` , it's possible to subtract an integer from a date or date with time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

minus(a, b) Alias: `a - b` (operator)

## multiply [¶](https://www.tinybird.co/docs/about:blank#multiply)

Calculates the product of two values `a` and `b`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

multiply(a, b) Alias: `a * b` (operator)

## divide [¶](https://www.tinybird.co/docs/about:blank#divide)

Calculates the quotient of two values `a` and `b` . The result type is always Float64. Integer division is provided by the `intDiv` function.

Division by 0 returns `inf`, `-inf` , or `nan`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

divide(a, b) Alias: `a / b` (operator)

## intDiv [¶](https://www.tinybird.co/docs/about:blank#intdiv)

Performs an integer division of two values `a` by `b` , i.e. computes the quotient rounded down to the next smallest integer.

The result has the same width as the dividend (the first parameter).

An exception is thrown when dividing by zero, when the quotient doesn't fit in the range of the dividend, or when dividing a minimal negative number by minus one.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intDiv(a, b)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    intDiv(toFloat64(1), 0.001) AS res,
    toTypeName(res) ┌──res─┬─toTypeName(intDiv(toFloat64(1), 0.001))─┐
│ 1000 │ Int64                                   │
└──────┴─────────────────────────────────────────┘ SELECT
    intDiv(1, 0.001) AS res,
    toTypeName(res) Received exception from server (version 23.2.1):
Code: 153. DB::Exception: Received from localhost:9000. DB::Exception: Cannot perform integer division, because it will produce infinite or too large number: While processing intDiv(1, 0.001) AS res, toTypeName(res). (ILLEGAL_DIVISION)
## intDivOrZero [¶](https://www.tinybird.co/docs/about:blank#intdivorzero)

Same as `intDiv` but returns zero when dividing by zero or when dividing a minimal negative number by minus one.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intDivOrZero(a, b)
## isFinite [¶](https://www.tinybird.co/docs/about:blank#isfinite)

Returns 1 if the Float32 or Float64 argument not infinite and not a NaN, otherwise this function returns 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isFinite(x)
## isInfinite [¶](https://www.tinybird.co/docs/about:blank#isinfinite)

Returns 1 if the Float32 or Float64 argument is infinite, otherwise this function returns 0. Note that 0 is returned for a NaN.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isInfinite(x)
## ifNotFinite [¶](https://www.tinybird.co/docs/about:blank#ifnotfinite)

Checks whether a floating point value is finite.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

ifNotFinite(x,y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Value to check for infinity. Float*.
- `y`   : Fallback value. Float*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `x`   if `x`   is finite.
- `y`   if `x`   isn't finite.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT 1/0 as infimum, ifNotFinite(infimum,42)

Result:

┌─infimum─┬─ifNotFinite(divide(1, 0), 42)─┐ │     inf │                            42 │ └─────────┴───────────────────────────────┘

You can get similar result by using the ternary operator: `isFinite(x) ? x : y`.

## isNaN [¶](https://www.tinybird.co/docs/about:blank#isnan)

Returns 1 if the Float32 and Float64 argument is NaN, otherwise this function 0.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

isNaN(x)
## modulo [¶](https://www.tinybird.co/docs/about:blank#modulo)

Calculates the remainder of the division of two values `a` by `b`.

The result type is an integer if both inputs are integers. If one of the inputs is a floating-point number, the result type is Float64.

The remainder is computed like in C++. Truncated division is used for negative numbers.

An exception is thrown when dividing by zero or when dividing a minimal negative number by minus one.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

modulo(a, b) Alias: `a % b` (operator)

## moduloOrZero [¶](https://www.tinybird.co/docs/about:blank#moduloorzero)

Like modulo but returns zero when the divisor is zero.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

moduloOrZero(a, b)
## positiveModulo(a, b) [¶](https://www.tinybird.co/docs/about:blank#positivemoduloa-b)

Like modulo but always returns a non-negative number.

This function is 4-5 times slower than `modulo`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

positiveModulo(a, b) Alias:

- `positive_modulo(a, b)`
- `pmod(a, b)`

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT positiveModulo(-1, 10) Result:

┌─positiveModulo(-1, 10)─┐
│                      9 │
└────────────────────────┘
## negate [¶](https://www.tinybird.co/docs/about:blank#negate)

Negates a value `a` . The result is always signed.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

negate(a) Alias: `-a`

## abs [¶](https://www.tinybird.co/docs/about:blank#abs)

Calculates the absolute value of `a` . Has no effect if `a` is of an unsigned type. If `a` is of a signed type, it returns an unsigned number.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

abs(a)
## gcd [¶](https://www.tinybird.co/docs/about:blank#gcd)

Returns the greatest common divisor of two values `a` and `b`.

An exception is thrown when dividing by zero or when dividing a minimal negative number by minus one.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

gcd(a, b)
## lcm(a, b) [¶](https://www.tinybird.co/docs/about:blank#lcma-b)

Returns the least common multiple of two values `a` and `b`.

An exception is thrown when dividing by zero or when dividing a minimal negative number by minus one.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

lcm(a, b)
## max2 [¶](https://www.tinybird.co/docs/about:blank#max2)

Returns the bigger of two values `a` and `b` . The returned value is of type Float64.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

max2(a, b)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT max2(-1, 2) Result:

┌─max2(-1, 2)─┐
│           2 │
└─────────────┘
## min2 [¶](https://www.tinybird.co/docs/about:blank#min2)

Returns the smaller of two values `a` and `b` . The returned value is of type Float64.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

min2(a, b)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT min2(-1, 2) Result:

┌─min2(-1, 2)─┐
│          -1 │
└─────────────┘
## multiplyDecimal [¶](https://www.tinybird.co/docs/about:blank#multiplydecimal)

Multiplies two decimals `a` and `b` . The result value will be of type Decimal256.

The scale of the result can be explicitly specified by `result_scale` . If `result_scale` isn't specified, it's assumed to be the maximum scale of the input values.

This function work significantly slower than usual `multiply` . In case no control over the result precision is needed and/or fast computation is desired, consider using `multiply`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

multiplyDecimal(a, b[, result_scale])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `a`   : First value. Decimal.
- `b`   : Second value. Decimal.
- `result_scale`   : Scale of result. Int/UInt.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The result of multiplication with given scale. Decimal256.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

┌─multiplyDecimal(toDecimal256(-12, 0), toDecimal32(-2.1, 1), 1)─┐
│                                                           25.2 │
└────────────────────────────────────────────────────────────────┘
### Differences compared to regular multiplication [¶](https://www.tinybird.co/docs/about:blank#differences-compared-to-regular-multiplication)

SELECT toDecimal64(-12.647, 3) * toDecimal32(2.1239, 4)
SELECT toDecimal64(-12.647, 3) as a, toDecimal32(2.1239, 4) as b, multiplyDecimal(a, b) Result:

┌─multiply(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
│                                               -26.8609633 │
└───────────────────────────────────────────────────────────┘
┌───────a─┬──────b─┬─multiplyDecimal(toDecimal64(-12.647, 3), toDecimal32(2.1239, 4))─┐
│ -12.647 │ 2.1239 │                                                         -26.8609 │
└─────────┴────────┴──────────────────────────────────────────────────────────────────┘ SELECT
    toDecimal64(-12.647987876, 9) AS a,
    toDecimal64(123.967645643, 9) AS b,
    multiplyDecimal(a, b)

SELECT
    toDecimal64(-12.647987876, 9) AS a,
    toDecimal64(123.967645643, 9) AS b,
    a * b Result:

┌─────────────a─┬─────────────b─┬─multiplyDecimal(toDecimal64(-12.647987876, 9), toDecimal64(123.967645643, 9))─┐
│ -12.647987876 │ 123.967645643 │                                                               -1567.941279108 │
└───────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────┘

Received exception from server (version 22.11.1):
Code: 407. DB::Exception: Received from localhost:9000. DB::Exception: Decimal math overflow: While processing toDecimal64(-12.647987876, 9) AS a, toDecimal64(123.967645643, 9) AS b, a * b. (DECIMAL_OVERFLOW)
## divideDecimal [¶](https://www.tinybird.co/docs/about:blank#dividedecimal)

Divides two decimals `a` and `b` . The result value will be of type Decimal256.

The scale of the result can be explicitly specified by `result_scale` . If `result_scale` isn't specified, it's assumed to be the maximum scale of the input values.

This function work significantly slower than usual `divide` . In case no control over the result precision is needed and/or fast computation is desired, consider using `divide`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

divideDecimal(a, b[, result_scale])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `a`   : First value: Decimal.
- `b`   : Second value: Decimal.
- `result_scale`   : Scale of result: Int/UInt.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The result of division with given scale. Decimal256.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

┌─divideDecimal(toDecimal256(-12, 0), toDecimal32(2.1, 1), 10)─┐
│                                                -5.7142857142 │
└──────────────────────────────────────────────────────────────┘
### Differences compared to regular division [¶](https://www.tinybird.co/docs/about:blank#differences-compared-to-regular-division)

SELECT toDecimal64(-12, 1) / toDecimal32(2.1, 1)
SELECT toDecimal64(-12, 1) as a, toDecimal32(2.1, 1) as b, divideDecimal(a, b, 1), divideDecimal(a, b, 5) Result:

┌─divide(toDecimal64(-12, 1), toDecimal32(2.1, 1))─┐
│                                             -5.7 │
└──────────────────────────────────────────────────┘

┌───a─┬───b─┬─divideDecimal(toDecimal64(-12, 1), toDecimal32(2.1, 1), 1)─┬─divideDecimal(toDecimal64(-12, 1), toDecimal32(2.1, 1), 5)─┐
│ -12 │ 2.1 │                                                       -5.7 │                                                   -5.71428 │
└─────┴─────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘ SELECT toDecimal64(-12, 0) / toDecimal32(2.1, 1)
SELECT toDecimal64(-12, 0) as a, toDecimal32(2.1, 1) as b, divideDecimal(a, b, 1), divideDecimal(a, b, 5) Result:

DB::Exception: Decimal result's scale is less than argument's one: While processing toDecimal64(-12, 0) / toDecimal32(2.1, 1). (ARGUMENT_OUT_OF_BOUND)

┌───a─┬───b─┬─divideDecimal(toDecimal64(-12, 0), toDecimal32(2.1, 1), 1)─┬─divideDecimal(toDecimal64(-12, 0), toDecimal32(2.1, 1), 5)─┐
│ -12 │ 2.1 │                                                       -5.7 │                                                   -5.71428 │
└─────┴─────┴────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
## byteSwap [¶](https://www.tinybird.co/docs/about:blank#byteswap)

Reverses the bytes of an integer, i.e. changes its endianness.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

byteSwap(a)
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

byteSwap(3351772109) Result:

┌─byteSwap(3351772109)─┐
│           3455829959 │
└──────────────────────┘ The previous example can be followed in the following manner:

1. Convert the base-10 integer to its equivalent hexadecimal format in big-endian format, for example `3351772109 -> C7 C7 FB CD (4 bytes)`  .
2. Reverse the bytes, for example `C7 C7 FB CD -> CD FB C7 C7`  .
3. Convert the result back to an integer assuming big-endian, for example `CD FB C7 C7  -> 3455829959`  .

A use case of this function is reversing IPv4s:

┌─toIPv4(byteSwap(toUInt32(toIPv4('205.251.199.199'))))─┐
│ 199.199.251.205                                       │
└───────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/functions/aggregate-functions
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Aggregate functions · Tinybird Docs"
theme-color: "#171612"
description: "Functions for performing calculations on a set of values and return a single value."
---




# Aggregate functions [¶](https://www.tinybird.co/docs/about:blank#aggregate-functions)

Aggregate functions perform a calculation on a set of values and return a single value. They are commonly used in data analysis and reporting to summarize large datasets. These functions can be used to compute sums, averages, counts, and other statistical measures.

## Aggregate function combinators [¶](https://www.tinybird.co/docs/about:blank#aggregate-function-combinators)

The name of an aggregate function can have a suffix appended to it. This changes the way the aggregate function works.

### -If [¶](https://www.tinybird.co/docs/about:blank#if)

The suffix -If can be appended to the name of any aggregate function. In this case, the aggregate function accepts an extra argument – a condition (Uint8 type). The aggregate function processes only the rows that trigger the condition. If the condition was not triggered even once, it returns a default value (usually zeros or empty strings).

Examples: `sumIf(column, cond)`, `countIf(cond)`, `avgIf(x, cond)`, `quantilesTimingIf(level1, level2)(x, cond)`, `argMinIf(arg, val, cond)` and so on.

With conditional aggregate functions, you can calculate aggregates for several conditions at once, without using subqueries and `JOIN` s. For example, conditional aggregate functions can be used to implement the segment comparison functionality.

### -Array [¶](https://www.tinybird.co/docs/about:blank#array)

The -Array suffix can be appended to any aggregate function. In this case, the aggregate function takes arguments of the 'Array(T)' type (arrays) instead of 'T' type arguments. If the aggregate function accepts multiple arguments, this must be arrays of equal lengths. When processing arrays, the aggregate function works like the original aggregate function across all array elements.

Example 1: `sumArray(arr)` - Totals all the elements of all 'arr' arrays. In this example, it could have been written more simply: `sum(arraySum(arr))`.

Example 2: `uniqArray(arr)` – Counts the number of unique elements in all 'arr' arrays. This could be done an easier way: `uniq(arrayJoin(arr))` , but it's not always possible to add 'arrayJoin' to a query.

-If and -Array can be combined. However, 'Array' must come first, then 'If'. Examples: `uniqArrayIf(arr, cond)`, `quantilesTimingArrayIf(level1, level2)(arr, cond)` . Due to this order, the 'cond' argument won't be an array.

### -Map [¶](https://www.tinybird.co/docs/about:blank#map)

The -Map suffix can be appended to any aggregate function. This will create an aggregate function which gets Map type as an argument, and aggregates values of each key of the map separately using the specified aggregate function. The result is also of a Map type.

#### Example [¶](https://www.tinybird.co/docs/about:blank#example)

WITH map_map AS (
  SELECT c1::Date AS date, c2::DateTime AS timeslot, c3::Map(String, UInt16) AS status FROM values (
    ('2000-01-01', '2000-01-01 00:00:00', (['a', 'b', 'c'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:00:00', (['c', 'd', 'e'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:01:00', (['d', 'e', 'f'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:01:00', (['f', 'g', 'g'], [10, 10, 10]))
))
SELECT
    timeslot,
    sumMap(status),
    avgMap(status),
    minMap(status)
FROM map_map
GROUP BY timeslot

┌────────────timeslot─┬─sumMap(status)───────────────────────┬─avgMap(status)───────────────────────┬─minMap(status)───────────────────────┐
│ 2000-01-01 00:00:00 │ {'a':10,'b':10,'c':20,'d':10,'e':10} │ {'a':10,'b':10,'c':10,'d':10,'e':10} │ {'a':10,'b':10,'c':10,'d':10,'e':10} │
│ 2000-01-01 00:01:00 │ {'d':10,'e':10,'f':20,'g':20}        │ {'d':10,'e':10,'f':10,'g':10}        │ {'d':10,'e':10,'f':10,'g':10}        │
└─────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┘
### -SimpleState [¶](https://www.tinybird.co/docs/about:blank#simplestate)

If you apply this combinator, the aggregate function returns the same value but with a different type. This is a SimpleAggregateFunction(...) that can be stored in a table to work with AggregatingMergeTree tables.

#### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

SimpleState(x)
#### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Aggregate function parameters.

#### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

The value of an aggregate function with the `SimpleAggregateFunction(...)` type.

#### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH anySimpleState(number) AS c SELECT toTypeName(c), c FROM numbers(1) Result:

┌─toTypeName(c)────────────────────────┬─c─┐
│ SimpleAggregateFunction(any, UInt64) │ 0 │
└──────────────────────────────────────┴───┘
### -State [¶](https://www.tinybird.co/docs/about:blank#state)

If you apply this combinator, the aggregate function doesn't return the resulting value (such as the number of unique values for the uniq function), but an intermediate state of the aggregation (for `uniq` , this is the hash table for calculating the number of unique values). This is an `AggregateFunction(...)` that can be used for further processing or stored in a table to finish aggregating later.

`-MapState` isn't an invariant for the same data because the order of data in intermediate state can change, though it doesn't impact ingestion of this data.

To work with these states, use:

- AggregatingMergeTree table engine.
- finalizeAggregation function.
- runningAccumulate function.
- -Merge combinator.
- -MergeState combinator.

### -Merge [¶](https://www.tinybird.co/docs/about:blank#merge)

If you apply this combinator, the aggregate function takes the intermediate aggregation state as an argument, combines the states to finish aggregation, and returns the resulting value.

### -MergeState [¶](https://www.tinybird.co/docs/about:blank#mergestate)

Merges the intermediate aggregation states in the same way as the -Merge combinator. However, it doesn't return the resulting value, but an intermediate aggregation state, similar to the -State combinator.

### -ForEach [¶](https://www.tinybird.co/docs/about:blank#foreach)

Converts an aggregate function for tables into an aggregate function for arrays that aggregates the corresponding array items and returns an array of results. For example, `sumForEach` for the arrays `[1, 2]`, `[3, 4, 5]` and `[6, 7]` returns the result `[10, 13, 5]` after adding together the corresponding array items.

### -Distinct [¶](https://www.tinybird.co/docs/about:blank#distinct)

Every unique combination of arguments will be aggregated only once. Repeating values are ignored. Examples: `sum(DISTINCT x)` (or `sumDistinct(x)` ), `groupArray(DISTINCT x)` (or `groupArrayDistinct(x)` ), `corrStable(DISTINCT x, y)` (or `corrStableDistinct(x, y)` ) and so on.

### -OrDefault [¶](https://www.tinybird.co/docs/about:blank#ordefault)

Changes behavior of an aggregate function.

If an aggregate function doesn't have input values, with this combinator it returns the default value for its return data type. Applies to the aggregate functions that can take empty input data.

`-OrDefault` can be used with other combinators.

#### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

OrDefault(x)
#### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Aggregate function parameters.

#### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Returns the default value of an aggregate function's return type if there is nothing to aggregate.

Type depends on the aggregate function used.

#### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT avg(number), avgOrDefault(number) FROM numbers(0) Result:

┌─avg(number)─-─avgOrDefault(number)─┐
│         nan │                    0 │
└─────────────┴──────────────────────┘ Also `-OrDefault` can be used with another combinators. It is useful when the aggregate function doesn't accept the empty input.

Query:

SELECT avgOrDefaultIf(x, x > 10)
FROM
(
    SELECT toDecimal32(1.23, 2) AS x
) Result:

┌─avgOrDefaultIf(x, greater(x, 10))─┐
│                              0.00 │
└───────────────────────────────────┘
### -OrNull [¶](https://www.tinybird.co/docs/about:blank#ornull)

Changes behavior of an aggregate function.

This combinator converts a result of an aggregate function to the Nullable data type. If the aggregate function doesn't have values to calculate it returns NULL.

`-OrNull` can be used with other combinators.

#### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

OrNull(x)
#### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Aggregate function parameters.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The result of the aggregate function, converted to the `Nullable`   data type.
- `NULL`   , if there is nothing to aggregate.

Type: `Nullable(aggregate function return type)`.

#### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Add `-orNull` to the end of aggregate function.

Query:

SELECT sumOrNull(number), toTypeName(sumOrNull(number)) FROM numbers(10) WHERE number > 10 Result:

┌─sumOrNull(number)─┬─toTypeName(sumOrNull(number))─┐
│              ᴺᵁᴸᴸ │ Nullable(UInt64)              │
└───────────────────┴───────────────────────────────┘ Also `-OrNull` can be used with another combinators. It is useful when the aggregate function doesn't accept the empty input.

Query:

SELECT avgOrNullIf(x, x > 10)
FROM
(
    SELECT toDecimal32(1.23, 2) AS x
) Result:

┌─avgOrNullIf(x, greater(x, 10))─┐
│                           ᴺᵁᴸᴸ │
└────────────────────────────────┘
### -Resample [¶](https://www.tinybird.co/docs/about:blank#resample)

Lets you divide data into groups, and then separately aggregates the data in those groups. Groups are created by splitting the values from one column into intervals.

Resample(start, end, step)(, resampling_key)
#### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start`   : Starting value of the whole required interval for `resampling_key`   values.
- `stop`   : Ending value of the whole required interval for `resampling_key`   values. The whole interval doesn't include the `stop`   value `[start, stop)`  .
- `step`   : Step for separating the whole interval into subintervals. The `aggFunction`   is executed over each of those subintervals independently.
- `resampling_key`   : Column whose values are used for separating data into intervals.
- `aggFunction_params`  : `aggFunction`   parameters.

#### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of `aggFunction`   results for each subinterval.

#### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Consider the `people` table with the following data:

┌─name───┬─age─┬─wage─┐
│ John   │  16 │   10 │
│ Alice  │  30 │   15 │
│ Mary   │  35 │    8 │
│ Evelyn │  48 │ 11.5 │
│ David  │  62 │  9.9 │
│ Brian  │  60 │   16 │
└────────┴─────┴──────┘ Let's get the names of the people whose age lies in the intervals of `[30,60)` and `[60,75)` . Since we use integer representation for age, we get ages in the `[30, 59]` and `[60,74]` intervals.

To aggregate names in an array, we use the groupArray aggregate function. It takes one argument. In our case, it's the `name` column. The `groupArrayResample` function should use the `age` column to aggregate names by age. To define the required intervals, we pass the `30, 75, 30` arguments into the `groupArrayResample` function.

SELECT groupArrayResample(30, 75, 30)(name, age) FROM people ┌─groupArrayResample(30, 75, 30)(name, age)─────┐
│ [['Alice','Mary','Evelyn'],['David','Brian']] │
└───────────────────────────────────────────────┘ Consider the results.

`John` is out of the sample because he's too young. Other people are distributed according to the specified age intervals.

Now let's count the total number of people and their average wage in the specified age intervals.

SELECT
    countResample(30, 75, 30)(name, age) AS amount,
    avgResample(30, 75, 30)(wage, age) AS avg_wage
FROM people ┌─amount─┬─avg_wage──────────────────┐
│ [3,2]  │ [11.5,12.949999809265137] │
└────────┴───────────────────────────┘
### -ArgMin [¶](https://www.tinybird.co/docs/about:blank#argmin)

The suffix -ArgMin can be appended to the name of any aggregate function. In this case, the aggregate function accepts an additional argument, which should be any comparable expression. The aggregate function processes only the rows that have the minimum value for the specified extra expression.

Examples: `sumArgMin(column, expr)`, `countArgMin(expr)`, `avgArgMin(x, expr)` and so on.

### -ArgMax [¶](https://www.tinybird.co/docs/about:blank#argmax)

Similar to suffix -ArgMin but processes only the rows that have the maximum value for the specified extra expression.

## aggThrow [¶](https://www.tinybird.co/docs/about:blank#aggthrow)

This function can be used for the purpose of testing exception safety. It will throw an exception on creation with the specified probability.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

aggThrow(throw_prob)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `throw_prob`   : Probability to throw on creation. Float64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An exception: `Code: 503. DB::Exception: Aggregate function aggThrow has thrown exception successfully`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT number % 2 AS even, aggThrow(number) FROM numbers(10) GROUP BY even Result:

Received exception:
Code: 503. DB::Exception: Aggregate function aggThrow has thrown exception successfully: While executing AggregatingTransform. (AGGREGATE_FUNCTION_THROW)
## analysisOfVariance [¶](https://www.tinybird.co/docs/about:blank#analysisofvariance)

Provides a statistical test for one-way analysis of variance (ANOVA test). It is a test over several groups of normally distributed observations to find out whether all groups have the same mean or not.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

analysisOfVariance(val, group_no) Aliases: `anova`

**Parameters**

- `val`   : value.
- `group_no`   : group number that `val`   belongs to.

Groups are enumerated starting from 0 and there should be at least two groups to perform a test. There should be at least one group with the number of observations greater than one.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `(f_statistic, p_value)`   . Tuple(Float64, Float64).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT analysisOfVariance(number, number % 2) FROM numbers(1048575) Result:

┌─analysisOfVariance(number, modulo(number, 2))─┐
│ (0,1)                                         │
└───────────────────────────────────────────────┘
## any [¶](https://www.tinybird.co/docs/about:blank#any)

Selects the first encountered value of a column.

As a query can be executed in arbitrary order, the result of this function is non-deterministic. If you need an arbitrary but deterministic result, use functions `min` or `max`.

By default, the function never returns NULL, i.e. ignores NULL values in the input column. However, if the function is used with the `RESPECT NULLS` modifier, it returns the first value reads no matter if NULL or not.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

any(column) [RESPECT NULLS] Aliases `any(column)` (without `RESPECT NULLS` )

- `any_value`  , `first_value`  .

Alias for `any(column) RESPECT NULLS`

- `any_respect_nulls`  , `first_value_respect_nulls`  , `any_value_respect_nulls`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `column`   : The column name.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The first value encountered.

The return type of the function is the same as the input, except for LowCardinality which is discarded. This means that given no rows as input it will return the default value of that type (0 for integers, or Null for a Nullable() column). You might use the `-OrNull` combinator ) to modify this behaviour.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH cte AS (SELECT arrayJoin([NULL, 'Amsterdam', 'New York', 'Tokyo', 'Valencia', NULL]) as city)
SELECT any(city), any_respect_nulls(city) FROM cte ┌─any(city)─┬─any_respect_nulls(city)─┐
│ Amsterdam │ ᴺᵁᴸᴸ                    │
└───────────┴─────────────────────────┘
## anyHeavy [¶](https://www.tinybird.co/docs/about:blank#anyheavy)

Selects a frequently occurring value using the heavy hitters algorithm. If there is a value that occurs more than in half the cases in each of the query's execution threads, this value is returned. Normally, the result is nondeterministic.

anyHeavy(column)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   – The column name.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH cte AS (SELECT arrayJoin([2,1,1,1,3,1,1,2,2]) as n)
SELECT any(n), anyHeavy(n) FROM cte ┌─any(n)─┬─anyHeavy(n)─┐
│     2  │           1 │
└────────┴─────────────┘
## anyLast [¶](https://www.tinybird.co/docs/about:blank#anylast)

Selects the last encountered value of a column.

As a query can be executed in arbitrary order, the result of this function is non-deterministic. If you need an arbitrary but deterministic result, use functions `min` or `max`.

By default, the function never returns NULL, i.e. ignores NULL values in the input column. However, if the function is used with the `RESPECT NULLS` modifier, it returns the first value reads no matter if NULL or not.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

anyLast(column) [RESPECT NULLS] Alias `anyLast(column)` (without `RESPECT NULLS` )

- `last_value`  .

Aliases for `anyLast(column) RESPECT NULLS`

- `anyLast_respect_nulls`  , `last_value_respect_nulls`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `column`   : The column name.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The last value encountered.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

WITH cte AS (SELECT arrayJoin([NULL, 'Amsterdam', 'New York', 'Tokyo', 'Valencia', NULL]) as city)
SELECT anyLast(city), anyLast_respect_nulls(city) FROM cte ┌─anyLast(city)─┬─anyLast_respect_nulls(city)─┐
│ Valencia      │ ᴺᵁᴸᴸ                        │
└───────────────┴─────────────────────────────┘
## approx_top_k [¶](https://www.tinybird.co/docs/about:blank#approx-top-k)

Returns an array of the approximately most frequent values and their counts in the specified column. The resulting array is sorted in descending order of approximate frequency of values (not by the values themselves).

approx_top_k(N)(column)
approx_top_k(N, reserved)(column) This function doesn't provide a guaranteed result. In certain situations, errors might occur and it might return frequent values that aren't the most frequent values.

We recommend using the `N < 10` value; performance is reduced with large `N` values. Maximum value of `N = 65536`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `N`   : The number of elements to return. Optional. Default value: 10.
- `reserved`   : Defines, how many cells reserved for values. If uniq(column) > reserved, result of topK function will be approximate. Optional. Default value: N * 3.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : The value to calculate frequency.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT approx_top_k(2)(k)
FROM values('k Char, w UInt64', ('y', 1), ('y', 1), ('x', 5), ('y', 1), ('z', 10)) Result:

┌─approx_top_k(2)(k)────┐
│ [('y',3,0),('x',1,0)] │
└───────────────────────┘
## approx_top_count [¶](https://www.tinybird.co/docs/about:blank#approx-top-count)

Is an alias to `approx_top_k` function

## approx_top_sum [¶](https://www.tinybird.co/docs/about:blank#approx-top-sum)

Returns an array of the approximately most frequent values and their counts in the specified column. The resulting array is sorted in descending order of approximate frequency of values (not by the values themselves). Additionally, the weight of the value is taken into account.

approx_top_sum(N)(column, weight)
approx_top_sum(N, reserved)(column, weight) This function doesn't provide a guaranteed result. In certain situations, errors might occur and it might return frequent values that aren't the most frequent values.

We recommend using the `N < 10` value; performance is reduced with large `N` values. Maximum value of `N = 65536`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `N`   : The number of elements to return. Optional. Default value: 10.
- `reserved`   : Defines, how many cells reserved for values. If uniq(column) > reserved, result of topK function will be approximate. Optional. Default value: N * 3.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : The value to calculate frequency.
- `weight`   : The weight. Every value is accounted `weight`   times for frequency calculation. UInt64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT approx_top_sum(2)(k, w)
FROM values('k Char, w UInt64', ('y', 1), ('y', 1), ('x', 5), ('y', 1), ('z', 10)) Result:

┌─approx_top_sum(2)(k, w)─┐
│ [('z',10,0),('x',5,0)]  │
└─────────────────────────┘
## argMax [¶](https://www.tinybird.co/docs/about:blank#argmax)

Calculates the `arg` value for a maximum `val` value. If there are multiple rows with equal `val` being the maximum, which of the associated `arg` is returned isn't deterministic. Both parts the `arg` and the `max` behave as aggregate functions, they both skip `Null` during processing and return not `Null` values if not `Null` values are available.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

argMax(arg, val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : Argument.
- `val`   : Value.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `arg`   value that corresponds to maximum `val`   value.

Type: matches `arg` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─user─────┬─salary─┐
│ director │   5000 │
│ manager  │   3000 │
│ worker   │   1000 │
└──────────┴────────┘ Query:

SELECT argMax(user, salary) FROM salary Result:

┌─argMax(user, salary)─┐
│ director             │
└──────────────────────┘
### Extended example [¶](https://www.tinybird.co/docs/about:blank#extended-example)

WITH test AS (
  SELECT c1 AS a, c2 AS b FROM values(('a', 1), ('b', 2), ('c', 2), (NULL, 3), (NULL, NULL), ('d', NULL))
)

SELECT * FROM test
┌─a─────┬─b────┐
│ a     │ 1    │
│ b     │ 2    │
│ c     │ 2    │
│ ᴺᵁᴸᴸ  │ 3    │
│ ᴺᵁᴸᴸ  │ ᴺᵁᴸᴸ │
│ d     │ ᴺᵁᴸᴸ │
└───────┴──────┘


SELECT argMax(a, b), max(b) FROM test
┌─argMax(a, b)─┬─max(b)─┐
│ b            │      3 │ -- argMax = 'b' because it the first not Null value, max(b) is from another row!
└──────────────┴────────┘


SELECT argMax(tuple(a), b) FROM test
┌─argMax(tuple(a), b)─┐
│ (NULL)              │ -- The a `Tuple` that contains only a `NULL` value isn't `NULL`, so the aggregate functions won't skip that row because of that `NULL` value
└─────────────────────┘


SELECT (argMax((a, b), b) as t).1 argMaxA, t.2 argMaxB FROM test
┌─argMaxA─┬─argMaxB─┐
│ ᴺᵁᴸᴸ    │       3 │ -- you can use Tuple and get both (all - tuple(*)) columns for the according max(b)
└─────────┴─────────┘


SELECT argMax(a, b), max(b) FROM test WHERE a IS NULL AND b IS NULL
┌─argMax(a, b)─┬─max(b)─┐
│ ᴺᵁᴸᴸ         │   ᴺᵁᴸᴸ │ -- All aggregated rows contains at least one `NULL` value because of the filter, so all rows are skipped, therefore the result will be `NULL`
└──────────────┴────────┘


SELECT argMax(a, (b,a)) FROM test
┌─argMax(a, tuple(b, a))─┐
│ c                      │ -- There are two rows with b=2, `Tuple` in the `Max` allows to get not the first `arg`
└────────────────────────┘


SELECT argMax(a, tuple(b)) FROM test
┌─argMax(a, tuple(b))─┐
│ b                   │ -- `Tuple` can be used in `Max` to not skip Nulls in `Max`
└─────────────────────┘
## argMin [¶](https://www.tinybird.co/docs/about:blank#argmin)

Calculates the `arg` value for a minimum `val` value. If there are multiple rows with equal `val` being the maximum, which of the associated `arg` is returned isn't deterministic. Both parts the `arg` and the `min` behave as aggregate functions, they both skip `Null` during processing and return not `Null` values if not `Null` values are available.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

argMin(arg, val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `arg`   : Argument.
- `val`   : Value.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `arg`   value that corresponds to minimum `val`   value.

Type: matches `arg` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─user─────┬─salary─┐
│ director │   5000 │
│ manager  │   3000 │
│ worker   │   1000 │
└──────────┴────────┘ Query:

SELECT argMin(user, salary) FROM salary Result:

┌─argMin(user, salary)─┐
│ worker               │
└──────────────────────┘
### Extended example [¶](https://www.tinybird.co/docs/about:blank#extended-example)

WITH test AS (
  SELECT c1 AS a, c2 AS b FROM values((NULL, 0), ('a', 1), ('b', 2), ('c', 2), (NULL, NULL), ('d', NULL))
)

SELECT * FROM test
┌─a────┬────b─┐
│ ᴺᵁᴸᴸ │    0 │
│ a    │    1 │
│ b    │    2 │
│ c    │    2 │
│ ᴺᵁᴸᴸ │ ᴺᵁᴸᴸ │
│ d    │ ᴺᵁᴸᴸ │
└──────┴──────┘


SELECT argMin(a, b), min(b) FROM test
┌─argMin(a, b)─┬─min(b)─┐
│ a            │      0 │ -- argMin = a because it the first not `NULL` value, min(b) is from another row!
└──────────────┴────────┘


SELECT argMin(tuple(a), b) FROM test
┌─argMin(tuple(a), b)─┐
│ (NULL)              │ -- The a `Tuple` that contains only a `NULL` value isn't `NULL`, so the aggregate functions won't skip that row because of that `NULL` value
└─────────────────────┘


SELECT (argMin((a, b), b) as t).1 argMinA, t.2 argMinB from test
┌─argMinA─┬─argMinB─┐
│ ᴺᵁᴸᴸ    │       0 │ -- you can use `Tuple` and get both (all - tuple(*)) columns for the according max(b)
└─────────┴─────────┘


SELECT argMin(a, b), min(b) FROM test WHERE a IS NULL and b IS NULL
┌─argMin(a, b)─┬─min(b)─┐
│ ᴺᵁᴸᴸ         │   ᴺᵁᴸᴸ │ -- All aggregated rows contains at least one `NULL` value because of the filter, so all rows are skipped, therefore the result will be `NULL`
└──────────────┴────────┘


SELECT argMin(a, (b, a)), min(tuple(b, a)) FROM test
┌─argMin(a, tuple(b, a))─┬─min(tuple(b, a))─┐
│ d                      │ (NULL,NULL)      │ -- 'd' is the first not `NULL` value for the min
└────────────────────────┴──────────────────┘


SELECT argMin((a, b), (b, a)), min(tuple(b, a)) FROM test
┌─argMin(tuple(a, b), tuple(b, a))─┬─min(tuple(b, a))─┐
│ (NULL,NULL)                      │ (NULL,NULL)      │ -- argMin returns (NULL,NULL) here because `Tuple` allows to don't skip `NULL` and min(tuple(b, a)) in this case is minimal value for this dataset
└──────────────────────────────────┴──────────────────┘


SELECT argMin(a, tuple(b)) FROM test
┌─argMin(a, tuple(b))─┐
│ d                   │ -- `Tuple` can be used in `min` to not skip rows with `NULL` values as b.
└─────────────────────┘
## array_concat_agg [¶](https://www.tinybird.co/docs/about:blank#array-concat-agg)

- Alias of `groupArrayArray`   . The function is case insensitive.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT *
FROM t

┌─a───────┐
│ [1,2,3] │
│ [4,5]   │
│ [6]     │
└─────────┘ Query:

SELECT array_concat_agg(a) AS a
FROM t

┌─a─────────────┐
│ [1,2,3,4,5,6] │
└───────────────┘
## avg [¶](https://www.tinybird.co/docs/about:blank#avg)

Calculates the arithmetic mean.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

avg(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : input values, must be Integer, Float, or Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The arithmetic mean, always as Float64.
- `NaN`   if the input parameter `x`   is empty.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT avg(x) FROM values('x Int8', 0, 1, 2, 3, 4, 5) Result:

┌─avg(x)─┐
│    2.5 │
└────────┘
## avgWeighted [¶](https://www.tinybird.co/docs/about:blank#avgweighted)

Calculates the weighted arithmetic mean.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

avgWeighted(x, weight)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Values.
- `weight`   : Weights of the values.

`x` and `weight` must both be Integer or floating-point, but may have different types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- `NaN`   if all the weights are equal to 0 or the supplied weights parameter is empty.
- Weighted mean otherwise.

**Return type** is always Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT avgWeighted(x, w)
FROM values('x Int8, w Int8', (4, 1), (1, 0), (10, 2)) Result:

┌─avgWeighted(x, weight)─┐
│                      8 │
└────────────────────────┘
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT avgWeighted(x, w)
FROM values('x Int8, w Float64', (4, 1), (1, 0), (10, 2)) Result:

┌─avgWeighted(x, weight)─┐
│                      8 │
└────────────────────────┘
### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT avgWeighted(x, w)
FROM values('x Int8, w Int8', (0, 0), (1, 0), (10, 0)) Result:

┌─avgWeighted(x, weight)─┐
│                    nan │
└────────────────────────┘
## boundingRatio [¶](https://www.tinybird.co/docs/about:blank#boundingratio)

Aggregate function that calculates the slope between the leftmost and rightmost points across a group of values.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Sample data:

SELECT
    number,
    number * 1.5
FROM numbers(10) ┌─number─┬─multiply(number, 1.5)─┐
│      0 │                     0 │
│      1 │                   1.5 │
│      2 │                     3 │
│      3 │                   4.5 │
│      4 │                     6 │
│      5 │                   7.5 │
│      6 │                     9 │
│      7 │                  10.5 │
│      8 │                    12 │
│      9 │                  13.5 │
└────────┴───────────────────────┘ The boundingRatio() function returns the slope of the line between the leftmost and rightmost points, in the above data these points are `(0,0)` and `(9,13.5)`.

SELECT boundingRatio(number, number * 1.5)
FROM numbers(10) ┌─boundingRatio(number, multiply(number, 1.5))─┐
│                                          1.5 │
└──────────────────────────────────────────────┘ Calculates the value of `(P(tag = 1) - P(tag = 0))(log(P(tag = 1)) - log(P(tag = 0)))` for each category.

categoricalInformationValue(category1, category2, ..., tag) The result indicates how a discrete (categorical) feature `[category1, category2, ...]` contribute to a learning model which predicting the value of `tag`.

## contingency [¶](https://www.tinybird.co/docs/about:blank#contingency)

The `contingency` function calculates the contingency coefficient, a value that measures the association between two columns in a table. The computation is similar to the `cramersV` function but with a different denominator in the square root.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

contingency(column1, column2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column1`   and `column2`   are the columns to be compared

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a value between 0 and 1. The larger the result, the closer the association of the two columns.

**Return type** is always Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The two columns being compared below have a small association with each other. We have included the result of `cramersV` also (as a comparison):

SELECT
    cramersV(a, b),
    contingency(a ,b)
FROM
    (
        SELECT
            number % 10 AS a,
            number % 4 AS b
        FROM
            numbers(150)
    ) Result:

┌──────cramersV(a, b)─┬───contingency(a, b)─┐
│ 0.41171788506213564 │ 0.05812725261759165 │
└─────────────────────┴─────────────────────┘
## corr [¶](https://www.tinybird.co/docs/about:blank#corr)

Calculates the Pearson correlation coefficient: `Σ((x - x̄)(y - ȳ)) / √(Σ(x - x̄)²*Σ(y - ȳ)²)`

Σ ( x − x ˉ ) ( y − y ˉ ) Σ ( x − x ˉ ) 2 ∗ Σ ( y − y ˉ ) 2 \frac{\Sigma{(x - \bar{x})(y - \bar{y})}}{\sqrt{\Sigma{(x - \bar{x})^2} * \Sigma{(y - \bar{y})^2}}} Σ ( x − x ˉ ) 2 ∗ Σ ( y − y ˉ ​ ) 2 ​ Σ ( x − x ˉ ) ( y − y ˉ ​ ) ​

This function uses a numerically unstable algorithm. If you need numerical stability in calculations, use the `corrStable` function. It is slower but provides a more accurate result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

corr(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The Pearson correlation coefficient. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT corr(c2, c3) 
FROM values((1, 5.6, -4.4),(2, -9.6, 3),(3, -1.3, -4),(4, 5.3, 9.7),(5, 4.4, 0.037),(6, -8.6, -7.8),(7, 5.1, 9.3),(8, 7.9, -3.6),(9, -8.2, 0.62),(10, -3, 7.3)) Result:

┌─corr(c2, c3)──────────┐
│    0.1730265755453256 │
└───────────────────────┘
## corrMatrix [¶](https://www.tinybird.co/docs/about:blank#corrmatrix)

Computes the correlation matrix over N variables.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

corrMatrix(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : a variable number of parameters. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Correlation matrix. Array(Array(Float64)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT 
  arrayMap(x -> round(x, 3), arrayJoin(corrMatrix(c1, c2, c3, c4))) AS corrMatrix
FROM values ((1, 5.6, -4.4, 2.6), (2, -9.6, 3, 3.3), (3, -1.3, -4, 1.2), (4, 5.3, 9.7, 2.3), (5, 4.4, 0.037, 1.222), (6, -8.6, -7.8, 2.1233), (7, 5.1, 9.3, 8.1222), (8, 7.9, -3.6, 9.837), (9, -8.2, 0.62, 8.43555), (10, -3, 7.3, 6.762)) Result:

┌─corrMatrix─────────────┐
│ [1,-0.096,0.243,0.746] │
│ [-0.096,1,0.173,0.106] │
│ [0.243,0.173,1,0.258]  │
│ [0.746,0.106,0.258,1]  │
└────────────────────────┘
## corrStable [¶](https://www.tinybird.co/docs/about:blank#corrstable)

Calculates the Pearson correlation coefficient: `Σ((x - x̄)(y - ȳ)) / √(Σ(x - x̄)²*Σ(y - ȳ)²)`

Σ ( x − x ˉ ) ( y − y ˉ ) Σ ( x − x ˉ ) 2 ∗ Σ ( y − y ˉ ) 2 \frac{\Sigma{(x - \bar{x})(y - \bar{y})}}{\sqrt{\Sigma{(x - \bar{x})^2} * \Sigma{(y - \bar{y})^2}}} Σ ( x − x ˉ ) 2 ∗ Σ ( y − y ˉ ​ ) 2 ​ Σ ( x − x ˉ ) ( y − y ˉ ​ ) ​

Similar to the `corr` function, but uses a numerically stable algorithm. As a result, `corrStable` is slower than `corr` but produces a more accurate result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

corrStable(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The Pearson correlation coefficient. Float64.

### *Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT corrStable(c2, c3)
FROM values((1, 5.6, -4.4),(2, -9.6, 3),(3, -1.3, -4),(4, 5.3, 9.7),(5, 4.4, 0.037),(6, -8.6, -7.8),(7, 5.1, 9.3),(8, 7.9, -3.6),(9, -8.2, 0.62),(10, -3, 7.3)) Result:

┌─corrStable(c2, c3)─────┐
│    0.17302657554532558 │
└────────────────────────┘
## count [¶](https://www.tinybird.co/docs/about:blank#count)

Counts the number of rows or not-NULL values.

Use the following syntaxes for `count`:

- `count(expr)`   or `COUNT(DISTINCT expr)`  .
- `count()`   or `COUNT(*)`  .

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function can take:

- Zero parameters.
- One expression.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- If the function is called without parameters it counts the number of rows.
- If the expression is passed, then the function counts how many times this expression returned not null. If the expression returns a Nullable-type value, then the result of `count`   stays not `Nullable`   . The function returns 0 if the expression returned `NULL`   for all the rows.

In both cases the type of the returned value is UInt64.

## covarPop [¶](https://www.tinybird.co/docs/about:blank#covarpop)

Calculates the population covariance: `Σ(x - x̄)(y - ȳ) / n`

Σ ( x − x ˉ ) ( y − y ˉ ) n \frac{\Sigma{(x - \bar{x})(y - \bar{y})}}{n} n Σ ( x − x ˉ ) ( y − y ˉ ​ ) ​

This function uses a numerically unstable algorithm. If you need numerical stability in calculations, use the `covarPopStable` function. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarPop(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The population covariance between `x`   and `y`   . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT covarPop(c2, c3)
FROM values ((1, 5.6, -4.4),(2, -9.6, 3),(3, -1.3, -4),(4, 5.3, 9.7),(5, 4.4, 0.037),(6, -8.6, -7.8),(7, 5.1, 9.3),(8, 7.9, -3.6),(9, -8.2, 0.62),(10, -3, 7.3)) Result:

┌─covarPop(c2, c3)───┐
│           6.485648 │
└────────────────────┘
## covarPopMatrix [¶](https://www.tinybird.co/docs/about:blank#covarpopmatrix)

Returns the population covariance matrix over N variables.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarPopMatrix(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : a variable number of parameters. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Population covariance matrix. Array(Array(Float64)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayMap(x -> round(x, 3), arrayJoin(covarPopMatrix(c1, c2, c3, c4))) AS covarPopMatrix
FROM values ((1, 5.6, -4.4, 2.6), (2, -9.6, 3, 3.3), (3, -1.3, -4, 1.2), (4, 5.3, 9.7, 2.3), (5, 4.4, 0.037, 1.222), (6, -8.6, -7.8, 2.1233), (7, 5.1, 9.3, 8.1222), (8, 7.9, -3.6, 9.837), (9, -8.2, 0.62, 8.43555), (10, -3, 7.3, 6.762)) Result:

┌─covarPopMatrix────────────┐
│ [8.25,-1.76,4.08,6.748]   │
│ [-1.76,41.07,6.486,2.132] │
│ [4.08,6.486,34.21,4.755]  │
│ [6.748,2.132,4.755,9.93]  │
└───────────────────────────┘
## covarPopStable [¶](https://www.tinybird.co/docs/about:blank#covarpopstable)

Calculates the value of the population covariance: `Σ(x - x̄)(y - ȳ) / n`

Σ ( x − x ˉ ) ( y − y ˉ ) n \frac{\Sigma{(x - \bar{x})(y - \bar{y})}}{n} n Σ ( x − x ˉ ) ( y − y ˉ ​ ) ​

It is similar to the covarPop function, but uses a numerically stable algorithm. As a result, `covarPopStable` is slower than `covarPop` but produces a more accurate result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarPop(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The population covariance between `x`   and `y`   . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT covarPopStable(c2, c3)
FROM values ((1, 5.6,-4.4),(2, -9.6,3),(3, -1.3,-4),(4, 5.3,9.7),(5, 4.4,0.037),(6, -8.6,-7.8),(7, 5.1,9.3),(8, 7.9,-3.6),(9, -8.2,0.62),(10, -3,7.3)) Result:

┌─covarPopStable(c2, c3)─┐
│               6.485648 │
└────────────────────────┘
## covarSamp [¶](https://www.tinybird.co/docs/about:blank#covarsamp)

Calculates the value of `Σ((x - x̅)(y - y̅)) / (n - 1)`.

This function uses a numerically unstable algorithm. If you need numerical stability in calculations, use the `covarSampStable` function. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarSamp(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The sample covariance between `x`   and `y`   . For `n <= 1`  , `nan`   is returned. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT covarSamp(c2, c3)
FROM values ((1, 5.6,-4.4),(2, -9.6,3),(3, -1.3,-4),(4, 5.3,9.7),(5, 4.4,0.037),(6, -8.6,-7.8),(7, 5.1,9.3),(8, 7.9,-3.6),(9, -8.2,0.62),(10, -3,7.3)) Result:

┌─covarSamp(c2, c3)───┐
│   7.206275555555556 │
└─────────────────────┘ Query:

SELECT covarSamp(c2, c3)
FROM values ((1, 5.6,-4.4)) Result:

┌─covarSamp(c2, c3)───┐
│                 nan │
└─────────────────────┘
## covarSampMatrix [¶](https://www.tinybird.co/docs/about:blank#covarsampmatrix)

Returns the sample covariance matrix over N variables.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarSampMatrix(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : a variable number of parameters. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Sample covariance matrix. Array(Array(Float64)).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT arrayMap(x -> round(x, 3), arrayJoin(covarSampMatrix(c1, c2, c3, c4))) AS covarSampMatrix
FROM values ((1, 5.6, -4.4, 2.6), (2, -9.6, 3, 3.3), (3, -1.3, -4, 1.2), (4, 5.3, 9.7, 2.3), (5, 4.4, 0.037, 1.222), (6, -8.6, -7.8, 2.1233), (7, 5.1, 9.3, 8.1222), (8, 7.9, -3.6, 9.837), (9, -8.2, 0.62, 8.43555), (10, -3, 7.3, 6.762)) Result:

┌─covarSampMatrix─────────────┐
│ [9.167,-1.956,4.534,7.498]  │
│ [-1.956,45.634,7.206,2.369] │
│ [4.534,7.206,38.011,5.283]  │
│ [7.498,2.369,5.283,11.034]  │
└─────────────────────────────┘
## covarSampStable [¶](https://www.tinybird.co/docs/about:blank#covarsampstable)

Calculates the value of `Σ((x - x̅)(y - y̅)) / (n - 1)` . Similar to covarSamp but works slower while providing a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

covarSampStable(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : first variable. (U)Int*, Float*, Decimal.
- `y`   : second variable. (U)Int*, Float*, Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The sample covariance between `x`   and `y`   . For `n <= 1`  , `inf`   is returned. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT covarSampStable(c2, c3)
FROM values ((1, 5.6,-4.4),(2, -9.6,3),(3, -1.3,-4),(4, 5.3,9.7),(5, 4.4,0.037),(6, -8.6,-7.8),(7, 5.1,9.3),(8, 7.9,-3.6),(9, -8.2,0.62),(10, -3,7.3)) Result:

┌─covarSampStable(x_value, y_value)─┐
│                 7.206275555555556 │
└───────────────────────────────────┘ Query:

SELECT covarSampStable(c2, c3)
FROM values ((1, 5.6,-4.4)) Result:

┌─covarSampStable(c2, c3)─┐
│                     inf │
└─────────────────────────┘
## cramersV [¶](https://www.tinybird.co/docs/about:blank#cramersv)

Cramer's V (sometimes referred to as Cramer's phi) is a measure of association between two columns in a table. The result of the `cramersV` function ranges from 0 (corresponding to no association between the variables) to 1 and can reach 1 only when each value is completely determined by the other. It may be viewed as the association between two variables as a percentage of their maximum possible variation.

For a bias corrected version of Cramer's V see: cramersVBiasCorrected

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

cramersV(column1, column2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `column1`   : first column to be compared.
- `column2`   : second column to be compared.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a value between 0 (corresponding to no association between the columns' values) to 1 (complete association).

Type: always Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following two columns being compared below have no association with each other, so the result of `cramersV` is 0:

Query:

SELECT
    cramersV(a, b)
FROM
    (
        SELECT
            number % 3 AS a,
            number % 5 AS b
        FROM
            numbers(150)
    ) Result:

┌─cramersV(a, b)─┐
│              0 │
└────────────────┘ The following two columns below have a fairly close association, so the result of `cramersV` is a high value:

SELECT
    cramersV(a, b)
FROM
    (
        SELECT
            number % 10 AS a,
            number % 5 AS b
        FROM
            numbers(150)
    ) Result:

┌─────cramersV(a, b)─┐
│ 0.8944271909999159 │
└────────────────────┘
## cramersVBiasCorrected [¶](https://www.tinybird.co/docs/about:blank#cramersvbiascorrected)

Cramer's V is a measure of association between two columns in a table. The result of the `cramersV` function ranges from 0 (corresponding to no association between the variables) to 1 and can reach 1 only when each value is completely determined by the other. The function can be heavily biased, so this version of Cramer's V uses the bias correction.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

cramersVBiasCorrected(column1, column2)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `column1`   : first column to be compared.
- `column2`   : second column to be compared.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a value between 0 (corresponding to no association between the columns' values) to 1 (complete association).

Type: always Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following two columns being compared below have a small association with each other. Notice the result of `cramersVBiasCorrected` is smaller than the result of `cramersV`:

Query:

SELECT
    cramersV(a, b),
    cramersVBiasCorrected(a ,b)
FROM
    (
        SELECT
            number % 10 AS a,
            number % 4 AS b
        FROM
            numbers(150)
    ) Result:

┌──────cramersV(a, b)─┬─cramersVBiasCorrected(a, b)─┐
│ 0.41171788506213564 │         0.33369281784141364 │
└─────────────────────┴─────────────────────────────┘
## deltaSum [¶](https://www.tinybird.co/docs/about:blank#deltasum)

Sums the arithmetic difference between consecutive rows. If the difference is negative, it's ignored.

The underlying data must be sorted for this function to work properly. If you would like to use this function in a materialized view, you most likely want to use the deltaSumTimestamp method instead.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

deltaSum(value)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Input values, must be Integer or Float type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A gained arithmetic difference of the `Integer`   or `Float`   type.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT deltaSum(arrayJoin([1, 2, 3])) Result:

┌─deltaSum(arrayJoin([1, 2, 3]))─┐
│                              2 │
└────────────────────────────────┘ Query:

SELECT deltaSum(arrayJoin([1, 2, 3, 0, 3, 4, 2, 3])) Result:

┌─deltaSum(arrayJoin([1, 2, 3, 0, 3, 4, 2, 3]))─┐
│                                             7 │
└───────────────────────────────────────────────┘ Query:

SELECT deltaSum(arrayJoin([2.25, 3, 4.5])) Result:

┌─deltaSum(arrayJoin([2.25, 3, 4.5]))─┐
│                                2.25 │
└─────────────────────────────────────┘
## deltaSumTimestamp [¶](https://www.tinybird.co/docs/about:blank#deltasumtimestamp)

Adds the difference between consecutive rows. If the difference is negative, it's ignored.

This function is primarily for materialized views that store data ordered by some time bucket-aligned timestamp, for example, a `toStartOfMinute` bucket. Because the rows in such a materialized view will all have the same timestamp, it's impossible for them to be merged in the correct order, without storing the original, unrounded timestamp value. The `deltaSumTimestamp` function keeps track of the original `timestamp` of the values it's seen, so the values (states) of the function are correctly computed during merging of parts.

To calculate the delta sum across an ordered collection you can simply use the deltaSum function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

deltaSumTimestamp(value, timestamp)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Input values, must be some Integer type or Float type or a Date or DateTime.
- `timestamp`   : The parameter for order values, must be some Integer type or Float type or a Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Accumulated differences between consecutive values, ordered by the `timestamp`   parameter.

Type: Integer or Float or Date or DateTime.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT deltaSumTimestamp(value, timestamp)
FROM (SELECT number AS timestamp, [0, 4, 8, 3, 0, 0, 0, 1, 3, 5][number] AS value FROM numbers(1, 10)) Result:

┌─deltaSumTimestamp(value, timestamp)─┐
│                                  13 │
└─────────────────────────────────────┘
## entropy [¶](https://www.tinybird.co/docs/about:blank#entropy)

Calculates Shannon entropy) of a column of values.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

entropy(val)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `val`   : Column of values of any type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Shannon entropy.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select entropy(c1), entropy(c2) from values((1, 'A'), (1, 'A'), (1,'A'), (1,'A'), (2,'B'), (2,'B'), (2,'C'), (2,'D')) Result:

┌─entropy(c1)─┬─entropy(c2)─┐
│           1 │        1.75 │
└─────────────┴─────────────┘
## exponentialMovingAverage [¶](https://www.tinybird.co/docs/about:blank#exponentialmovingaverage)

Calculates the exponential moving average of values for the determined time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exponentialMovingAverage(x)(value, timeunit) Each `value` corresponds to the determinate `timeunit` . The half-life `x` is the time lag at which the exponential weights decay by one-half. The function returns a weighted average: the older the time point, the less weight the corresponding value is considered to be.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Value. Integer, Float or Decimal.
- `timeunit`   : Timeunit. Integer, Float or Decimal. Timeunit isn't timestamp (seconds), it's -- an index of the time interval. Can be calculated using intDiv.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Half-life period. Integer, Float or Decimal.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns an exponentially smoothed moving average of the values for the past `x`   time at the latest point of time.

Type: Float64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Input table:

┌──temperature─┬─timestamp──┐
│          95  │         1  │
│          95  │         2  │
│          95  │         3  │
│          96  │         4  │
│          96  │         5  │
│          96  │         6  │
│          96  │         7  │
│          97  │         8  │
│          97  │         9  │
│          97  │        10  │
│          97  │        11  │
│          98  │        12  │
│          98  │        13  │
│          98  │        14  │
│          98  │        15  │
│          99  │        16  │
│          99  │        17  │
│          99  │        18  │
│         100  │        19  │
│         100  │        20  │
└──────────────┴────────────┘ Query:

SELECT exponentialMovingAverage(5)(temperature, timestamp) Result:

┌──exponentialMovingAverage(5)(temperature, timestamp)──┐
│                                    92.25779635374204  │
└───────────────────────────────────────────────────────┘ Query:

SELECT
    value,
    time,
    round(exp_smooth, 3),
    bar(exp_smooth, 0, 1, 50) AS bar
FROM
(
    SELECT
        (number = 0) OR (number >= 25) AS value,
        number AS time,
        exponentialMovingAverage(10)(value, time) OVER (Rows BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS exp_smooth
    FROM numbers(50)
) Result:

┌─value─┬─time─┬─round(exp_smooth, 3)─┬─bar────────────────────────────────────────┐
│     1 │    0 │                0.067 │ ███▎                                       │
│     0 │    1 │                0.062 │ ███                                        │
│     0 │    2 │                0.058 │ ██▊                                        │
│     0 │    3 │                0.054 │ ██▋                                        │
│     0 │    4 │                0.051 │ ██▌                                        │
│     0 │    5 │                0.047 │ ██▎                                        │
│     0 │    6 │                0.044 │ ██▏                                        │
│     0 │    7 │                0.041 │ ██                                         │
│     0 │    8 │                0.038 │ █▊                                         │
│     0 │    9 │                0.036 │ █▋                                         │
│     0 │   10 │                0.033 │ █▋                                         │
│     0 │   11 │                0.031 │ █▌                                         │
│     0 │   12 │                0.029 │ █▍                                         │
│     0 │   13 │                0.027 │ █▎                                         │
│     0 │   14 │                0.025 │ █▎                                         │
│     0 │   15 │                0.024 │ █▏                                         │
│     0 │   16 │                0.022 │ █                                          │
│     0 │   17 │                0.021 │ █                                          │
│     0 │   18 │                0.019 │ ▊                                          │
│     0 │   19 │                0.018 │ ▊                                          │
│     0 │   20 │                0.017 │ ▋                                          │
│     0 │   21 │                0.016 │ ▋                                          │
│     0 │   22 │                0.015 │ ▋                                          │
│     0 │   23 │                0.014 │ ▋                                          │
│     0 │   24 │                0.013 │ ▋                                          │
│     1 │   25 │                0.079 │ ███▊                                       │
│     1 │   26 │                 0.14 │ ███████                                    │
│     1 │   27 │                0.198 │ █████████▊                                 │
│     1 │   28 │                0.252 │ ████████████▌                              │
│     1 │   29 │                0.302 │ ███████████████                            │
│     1 │   30 │                0.349 │ █████████████████▍                         │
│     1 │   31 │                0.392 │ ███████████████████▌                       │
│     1 │   32 │                0.433 │ █████████████████████▋                     │
│     1 │   33 │                0.471 │ ███████████████████████▌                   │
│     1 │   34 │                0.506 │ █████████████████████████▎                 │
│     1 │   35 │                0.539 │ ██████████████████████████▊                │
│     1 │   36 │                 0.57 │ ████████████████████████████▌              │
│     1 │   37 │                0.599 │ █████████████████████████████▊             │
│     1 │   38 │                0.626 │ ███████████████████████████████▎           │
│     1 │   39 │                0.651 │ ████████████████████████████████▌          │
│     1 │   40 │                0.674 │ █████████████████████████████████▋         │
│     1 │   41 │                0.696 │ ██████████████████████████████████▋        │
│     1 │   42 │                0.716 │ ███████████████████████████████████▋       │
│     1 │   43 │                0.735 │ ████████████████████████████████████▋      │
│     1 │   44 │                0.753 │ █████████████████████████████████████▋     │
│     1 │   45 │                 0.77 │ ██████████████████████████████████████▍    │
│     1 │   46 │                0.785 │ ███████████████████████████████████████▎   │
│     1 │   47 │                  0.8 │ ███████████████████████████████████████▊   │  
│     1 │   48 │                0.813 │ ████████████████████████████████████████▋  │
│     1 │   49 │                0.825 │ █████████████████████████████████████████▎ │
└───────┴──────┴──────────────────────┴────────────────────────────────────────────┘ -- Calculate timeunit using intDiv
SELECT
    value,
    time,
    exponentialMovingAverage(1)(value, intDiv(toUInt32(time), 3600)) OVER (ORDER BY time ASC) AS res,
    intDiv(toUInt32(time), 3600) AS timeunit
FROM (SELECT
    10 AS value,
    toDateTime('2020-01-01') + (3600 * number) AS time
FROM numbers_mt(10))
ORDER BY time ASC

┌─value─┬────────────────time─┬─────────res─┬─timeunit─┐
│    10 │ 2020-01-01 00:00:00 │           5 │   438288 │
│    10 │ 2020-01-01 01:00:00 │         7.5 │   438289 │
│    10 │ 2020-01-01 02:00:00 │        8.75 │   438290 │
│    10 │ 2020-01-01 03:00:00 │       9.375 │   438291 │
│    10 │ 2020-01-01 04:00:00 │      9.6875 │   438292 │
│    10 │ 2020-01-01 05:00:00 │     9.84375 │   438293 │
│    10 │ 2020-01-01 06:00:00 │    9.921875 │   438294 │
│    10 │ 2020-01-01 07:00:00 │   9.9609375 │   438295 │
│    10 │ 2020-01-01 08:00:00 │  9.98046875 │   438296 │
│    10 │ 2020-01-01 09:00:00 │ 9.990234375 │   438297 │
└───────┴─────────────────────┴─────────────┴──────────┘


-- Calculate timeunit using toRelativeHourNum
SELECT
    value,
    time,
    exponentialMovingAverage(1)(value, toRelativeHourNum(time)) OVER (ORDER BY time ASC) AS res,
    toRelativeHourNum(time) AS timeunit
FROM (SELECT
    10 AS value,
    toDateTime('2020-01-01') + (3600 * number) AS time
FROM numbers_mt(10))
ORDER BY time ASC

┌─value─┬────────────────time─┬─────────res─┬─timeunit─┐
│    10 │ 2020-01-01 00:00:00 │           5 │   438288 │
│    10 │ 2020-01-01 01:00:00 │         7.5 │   438289 │
│    10 │ 2020-01-01 02:00:00 │        8.75 │   438290 │
│    10 │ 2020-01-01 03:00:00 │       9.375 │   438291 │
│    10 │ 2020-01-01 04:00:00 │      9.6875 │   438292 │
│    10 │ 2020-01-01 05:00:00 │     9.84375 │   438293 │
│    10 │ 2020-01-01 06:00:00 │    9.921875 │   438294 │
│    10 │ 2020-01-01 07:00:00 │   9.9609375 │   438295 │
│    10 │ 2020-01-01 08:00:00 │  9.98046875 │   438296 │
│    10 │ 2020-01-01 09:00:00 │ 9.990234375 │   438297 │
└───────┴─────────────────────┴─────────────┴──────────┘
## exponentialTimeDecayedAvg [¶](https://www.tinybird.co/docs/about:blank#exponentialtimedecayedavg)

Returns the exponentially smoothed weighted moving average of values of a time series at point `t` in time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exponentialTimeDecayedAvg(x)(v, t)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `v`   : Value. Integer, Float or Decimal.
- `t`   : Time. Integer, Float or Decimal, DateTime, DateTime64.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Half-life period. Integer, Float or Decimal.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns an exponentially smoothed weighted moving average at index `t`   in time. Float64.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Query:

SELECT
    value,
    time,
    round(exp_smooth, 3),
    bar(exp_smooth, 0, 5, 50) AS bar
FROM
    (
    SELECT
    (number = 0) OR (number >= 25) AS value,
    number AS time,
    exponentialTimeDecayedAvg(10)(value, time) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS exp_smooth
    FROM numbers(50)
    ) Response:

┌─value─┬─time─┬─round(exp_smooth, 3)─┬─bar────────┐
1. │     1 │    0 │                    1 │ ██████████ │
2. │     0 │    1 │                0.475 │ ████▊      │
3. │     0 │    2 │                0.301 │ ███        │
4. │     0 │    3 │                0.214 │ ██▏        │
5. │     0 │    4 │                0.162 │ █▌         │
6. │     0 │    5 │                0.128 │ █▎         │
7. │     0 │    6 │                0.104 │ █          │
8. │     0 │    7 │                0.086 │ ▊          │
9. │     0 │    8 │                0.072 │ ▋          │
0. │     0 │    9 │                0.061 │ ▌          │
1. │     0 │   10 │                0.052 │ ▌          │
2. │     0 │   11 │                0.045 │ ▍          │
3. │     0 │   12 │                0.039 │ ▍          │
4. │     0 │   13 │                0.034 │ ▎          │
5. │     0 │   14 │                 0.03 │ ▎          │
6. │     0 │   15 │                0.027 │ ▎          │
7. │     0 │   16 │                0.024 │ ▏          │
8. │     0 │   17 │                0.021 │ ▏          │
9. │     0 │   18 │                0.018 │ ▏          │
0. │     0 │   19 │                0.016 │ ▏          │
1. │     0 │   20 │                0.015 │ ▏          │
2. │     0 │   21 │                0.013 │ ▏          │
3. │     0 │   22 │                0.012 │            │
4. │     0 │   23 │                 0.01 │            │
5. │     0 │   24 │                0.009 │            │
6. │     1 │   25 │                0.111 │ █          │
7. │     1 │   26 │                0.202 │ ██         │
8. │     1 │   27 │                0.283 │ ██▊        │
9. │     1 │   28 │                0.355 │ ███▌       │
0. │     1 │   29 │                 0.42 │ ████▏      │
1. │     1 │   30 │                0.477 │ ████▊      │
2. │     1 │   31 │                0.529 │ █████▎     │
3. │     1 │   32 │                0.576 │ █████▊     │
4. │     1 │   33 │                0.618 │ ██████▏    │
5. │     1 │   34 │                0.655 │ ██████▌    │
6. │     1 │   35 │                0.689 │ ██████▉    │
7. │     1 │   36 │                0.719 │ ███████▏   │
8. │     1 │   37 │                0.747 │ ███████▍   │
9. │     1 │   38 │                0.771 │ ███████▋   │
0. │     1 │   39 │                0.793 │ ███████▉   │
1. │     1 │   40 │                0.813 │ ████████▏  │
2. │     1 │   41 │                0.831 │ ████████▎  │
3. │     1 │   42 │                0.848 │ ████████▍  │
4. │     1 │   43 │                0.862 │ ████████▌  │
5. │     1 │   44 │                0.876 │ ████████▊  │
6. │     1 │   45 │                0.888 │ ████████▉  │
7. │     1 │   46 │                0.898 │ ████████▉  │
8. │     1 │   47 │                0.908 │ █████████  │
9. │     1 │   48 │                0.917 │ █████████▏ │
0. │     1 │   49 │                0.925 │ █████████▏ │
   └───────┴──────┴──────────────────────┴────────────┘
## exponentialTimeDecayedCount [¶](https://www.tinybird.co/docs/about:blank#exponentialtimedecayedcount)

Returns the cumulative exponential decay over a time series at the index `t` in time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exponentialTimeDecayedCount(x)(t)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `t`   : Time. Integer, Float or Decimal, DateTime, DateTime64.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Half-life period. Integer, Float or Decimal.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns the cumulative exponential decay at the given point in time. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    value,
    time,
    round(exp_smooth, 3),
    bar(exp_smooth, 0, 20, 50) AS bar
FROM
(
    SELECT
        (number % 5) = 0 AS value,
        number AS time,
        exponentialTimeDecayedCount(10)(time) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS exp_smooth
    FROM numbers(50)
) Result:

┌─value─┬─time─┬─round(exp_smooth, 3)─┬─bar────────────────────────┐
 1. │     1 │    0 │                    1 │ ██▌                        │
 2. │     0 │    1 │                1.905 │ ████▊                      │
 3. │     0 │    2 │                2.724 │ ██████▊                    │
 4. │     0 │    3 │                3.464 │ ████████▋                  │
 5. │     0 │    4 │                4.135 │ ██████████▎                │
 6. │     1 │    5 │                4.741 │ ███████████▊               │
 7. │     0 │    6 │                 5.29 │ █████████████▏             │
 8. │     0 │    7 │                5.787 │ ██████████████▍            │
 9. │     0 │    8 │                6.236 │ ███████████████▌           │
10. │     0 │    9 │                6.643 │ ████████████████▌          │
11. │     1 │   10 │                 7.01 │ █████████████████▌         │
12. │     0 │   11 │                7.343 │ ██████████████████▎        │
13. │     0 │   12 │                7.644 │ ███████████████████        │
14. │     0 │   13 │                7.917 │ ███████████████████▊       │
15. │     0 │   14 │                8.164 │ ████████████████████▍      │
16. │     1 │   15 │                8.387 │ ████████████████████▉      │
17. │     0 │   16 │                8.589 │ █████████████████████▍     │
18. │     0 │   17 │                8.771 │ █████████████████████▉     │
19. │     0 │   18 │                8.937 │ ██████████████████████▎    │
20. │     0 │   19 │                9.086 │ ██████████████████████▋    │
21. │     1 │   20 │                9.222 │ ███████████████████████    │
22. │     0 │   21 │                9.344 │ ███████████████████████▎   │
23. │     0 │   22 │                9.455 │ ███████████████████████▋   │
24. │     0 │   23 │                9.555 │ ███████████████████████▉   │
25. │     0 │   24 │                9.646 │ ████████████████████████   │
26. │     1 │   25 │                9.728 │ ████████████████████████▎  │
27. │     0 │   26 │                9.802 │ ████████████████████████▌  │
28. │     0 │   27 │                9.869 │ ████████████████████████▋  │
29. │     0 │   28 │                 9.93 │ ████████████████████████▊  │
30. │     0 │   29 │                9.985 │ ████████████████████████▉  │
31. │     1 │   30 │               10.035 │ █████████████████████████  │
32. │     0 │   31 │                10.08 │ █████████████████████████▏ │
33. │     0 │   32 │               10.121 │ █████████████████████████▎ │
34. │     0 │   33 │               10.158 │ █████████████████████████▍ │
35. │     0 │   34 │               10.191 │ █████████████████████████▍ │
36. │     1 │   35 │               10.221 │ █████████████████████████▌ │
37. │     0 │   36 │               10.249 │ █████████████████████████▌ │
38. │     0 │   37 │               10.273 │ █████████████████████████▋ │
39. │     0 │   38 │               10.296 │ █████████████████████████▋ │
40. │     0 │   39 │               10.316 │ █████████████████████████▊ │
41. │     1 │   40 │               10.334 │ █████████████████████████▊ │
42. │     0 │   41 │               10.351 │ █████████████████████████▉ │
43. │     0 │   42 │               10.366 │ █████████████████████████▉ │
44. │     0 │   43 │               10.379 │ █████████████████████████▉ │
45. │     0 │   44 │               10.392 │ █████████████████████████▉ │
46. │     1 │   45 │               10.403 │ ██████████████████████████ │
47. │     0 │   46 │               10.413 │ ██████████████████████████ │
48. │     0 │   47 │               10.422 │ ██████████████████████████ │
49. │     0 │   48 │                10.43 │ ██████████████████████████ │
50. │     0 │   49 │               10.438 │ ██████████████████████████ │
    └───────┴──────┴──────────────────────┴────────────────────────────┘
## exponentialTimeDecayedMax [¶](https://www.tinybird.co/docs/about:blank#exponentialtimedecayedmax)

Returns the maximum of the computed exponentially smoothed moving average at index `t` in time with that at `t-1`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exponentialTimeDecayedMax(x)(value, timeunit)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `value`   : Value. Integer, Float or Decimal.
- `timeunit`   : Timeunit. Integer, Float or Decimal, DateTime, DateTime64.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Half-life period. Integer, Float or Decimal.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns the maximum of the exponentially smoothed weighted moving average at `t`   and `t-1`   . Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    value,
    time,
    round(exp_smooth, 3),
    bar(exp_smooth, 0, 5, 50) AS bar
FROM
    (
    SELECT
    (number = 0) OR (number >= 25) AS value,
    number AS time,
    exponentialTimeDecayedMax(10)(value, time) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS exp_smooth
    FROM numbers(50)
    ) Result:

┌─value─┬─time─┬─round(exp_smooth, 3)─┬─bar────────┐
 1. │     1 │    0 │                    1 │ ██████████ │
 2. │     0 │    1 │                0.905 │ █████████  │
 3. │     0 │    2 │                0.819 │ ████████▏  │
 4. │     0 │    3 │                0.741 │ ███████▍   │
 5. │     0 │    4 │                 0.67 │ ██████▋    │
 6. │     0 │    5 │                0.607 │ ██████     │
 7. │     0 │    6 │                0.549 │ █████▍     │
 8. │     0 │    7 │                0.497 │ ████▉      │
 9. │     0 │    8 │                0.449 │ ████▍      │
10. │     0 │    9 │                0.407 │ ████       │
11. │     0 │   10 │                0.368 │ ███▋       │
12. │     0 │   11 │                0.333 │ ███▎       │
13. │     0 │   12 │                0.301 │ ███        │
14. │     0 │   13 │                0.273 │ ██▋        │
15. │     0 │   14 │                0.247 │ ██▍        │
16. │     0 │   15 │                0.223 │ ██▏        │
17. │     0 │   16 │                0.202 │ ██         │
18. │     0 │   17 │                0.183 │ █▊         │
19. │     0 │   18 │                0.165 │ █▋         │
20. │     0 │   19 │                 0.15 │ █▍         │
21. │     0 │   20 │                0.135 │ █▎         │
22. │     0 │   21 │                0.122 │ █▏         │
23. │     0 │   22 │                0.111 │ █          │
24. │     0 │   23 │                  0.1 │ █          │
25. │     0 │   24 │                0.091 │ ▉          │
26. │     1 │   25 │                    1 │ ██████████ │
27. │     1 │   26 │                    1 │ ██████████ │
28. │     1 │   27 │                    1 │ ██████████ │
29. │     1 │   28 │                    1 │ ██████████ │
30. │     1 │   29 │                    1 │ ██████████ │
31. │     1 │   30 │                    1 │ ██████████ │
32. │     1 │   31 │                    1 │ ██████████ │
33. │     1 │   32 │                    1 │ ██████████ │
34. │     1 │   33 │                    1 │ ██████████ │
35. │     1 │   34 │                    1 │ ██████████ │
36. │     1 │   35 │                    1 │ ██████████ │
37. │     1 │   36 │                    1 │ ██████████ │
38. │     1 │   37 │                    1 │ ██████████ │
39. │     1 │   38 │                    1 │ ██████████ │
40. │     1 │   39 │                    1 │ ██████████ │
41. │     1 │   40 │                    1 │ ██████████ │
42. │     1 │   41 │                    1 │ ██████████ │
43. │     1 │   42 │                    1 │ ██████████ │
44. │     1 │   43 │                    1 │ ██████████ │
45. │     1 │   44 │                    1 │ ██████████ │
46. │     1 │   45 │                    1 │ ██████████ │
47. │     1 │   46 │                    1 │ ██████████ │
48. │     1 │   47 │                    1 │ ██████████ │
49. │     1 │   48 │                    1 │ ██████████ │
50. │     1 │   49 │                    1 │ ██████████ │
    └───────┴──────┴──────────────────────┴────────────┘
## exponentialTimeDecayedSum [¶](https://www.tinybird.co/docs/about:blank#exponentialtimedecayedsum)

Returns the sum of exponentially smoothed moving average values of a time series at the index `t` in time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

exponentialTimeDecayedSum(x)(v, t)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `v`   : Value. Integer, Float or Decimal.
- `t`   : Time. Integer, Float or Decimal, DateTime, DateTime64.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Half-life period. Integer, Float or Decimal.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns the sum of exponentially smoothed moving average values at the given point in time. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT
    value,
    time,
    round(exp_smooth, 3),
    bar(exp_smooth, 0, 10, 50) AS bar
FROM
    (
    SELECT
    (number = 0) OR (number >= 25) AS value,
    number AS time,
    exponentialTimeDecayedSum(10)(value, time) OVER (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS exp_smooth
    FROM numbers(50)
    ) Result:

┌─value─┬─time─┬─round(exp_smooth, 3)─┬─bar───────────────────────────────────────────────┐
 1. │     1 │    0 │                    1 │ █████                                             │
 2. │     0 │    1 │                0.905 │ ████▌                                             │
 3. │     0 │    2 │                0.819 │ ████                                              │
 4. │     0 │    3 │                0.741 │ ███▋                                              │
 5. │     0 │    4 │                 0.67 │ ███▎                                              │
 6. │     0 │    5 │                0.607 │ ███                                               │
 7. │     0 │    6 │                0.549 │ ██▋                                               │
 8. │     0 │    7 │                0.497 │ ██▍                                               │
 9. │     0 │    8 │                0.449 │ ██▏                                               │
10. │     0 │    9 │                0.407 │ ██                                                │
11. │     0 │   10 │                0.368 │ █▊                                                │
12. │     0 │   11 │                0.333 │ █▋                                                │
13. │     0 │   12 │                0.301 │ █▌                                                │
14. │     0 │   13 │                0.273 │ █▎                                                │
15. │     0 │   14 │                0.247 │ █▏                                                │
16. │     0 │   15 │                0.223 │ █                                                 │
17. │     0 │   16 │                0.202 │ █                                                 │
18. │     0 │   17 │                0.183 │ ▉                                                 │
19. │     0 │   18 │                0.165 │ ▊                                                 │
20. │     0 │   19 │                 0.15 │ ▋                                                 │
21. │     0 │   20 │                0.135 │ ▋                                                 │
22. │     0 │   21 │                0.122 │ ▌                                                 │
23. │     0 │   22 │                0.111 │ ▌                                                 │
24. │     0 │   23 │                  0.1 │ ▌                                                 │
25. │     0 │   24 │                0.091 │ ▍                                                 │
26. │     1 │   25 │                1.082 │ █████▍                                            │
27. │     1 │   26 │                1.979 │ █████████▉                                        │
28. │     1 │   27 │                2.791 │ █████████████▉                                    │
29. │     1 │   28 │                3.525 │ █████████████████▋                                │
30. │     1 │   29 │                 4.19 │ ████████████████████▉                             │
31. │     1 │   30 │                4.791 │ ███████████████████████▉                          │
32. │     1 │   31 │                5.335 │ ██████████████████████████▋                       │
33. │     1 │   32 │                5.827 │ █████████████████████████████▏                    │
34. │     1 │   33 │                6.273 │ ███████████████████████████████▎                  │
35. │     1 │   34 │                6.676 │ █████████████████████████████████▍                │
36. │     1 │   35 │                7.041 │ ███████████████████████████████████▏              │
37. │     1 │   36 │                7.371 │ ████████████████████████████████████▊             │
38. │     1 │   37 │                7.669 │ ██████████████████████████████████████▎           │
39. │     1 │   38 │                7.939 │ ███████████████████████████████████████▋          │
40. │     1 │   39 │                8.184 │ ████████████████████████████████████████▉         │
41. │     1 │   40 │                8.405 │ ██████████████████████████████████████████        │
42. │     1 │   41 │                8.605 │ ███████████████████████████████████████████       │
43. │     1 │   42 │                8.786 │ ███████████████████████████████████████████▉      │
44. │     1 │   43 │                 8.95 │ ████████████████████████████████████████████▊     │
45. │     1 │   44 │                9.098 │ █████████████████████████████████████████████▍    │
46. │     1 │   45 │                9.233 │ ██████████████████████████████████████████████▏   │
47. │     1 │   46 │                9.354 │ ██████████████████████████████████████████████▊   │
48. │     1 │   47 │                9.464 │ ███████████████████████████████████████████████▎  │
49. │     1 │   48 │                9.563 │ ███████████████████████████████████████████████▊  │
50. │     1 │   49 │                9.653 │ ████████████████████████████████████████████████▎ │
    └───────┴──────┴──────────────────────┴───────────────────────────────────────────────────┘
## first_value [¶](https://www.tinybird.co/docs/about:blank#first-value)

It is an alias for `any` but it was introduced for compatibility with Window Functions, where sometimes it's necessary to process `NULL` values.

It supports declaring a modifier to respect nulls ( `RESPECT NULLS` ), both under Window Functions and in normal aggregations.

As with `any` , without Window Functions the result will be random if the source stream isn't ordered and the return type matches the input type (Null is only returned if the input is Nullable or -OrNull combinator is added).

## flameGraph [¶](https://www.tinybird.co/docs/about:blank#flamegraph)

Aggregate function which builds a [flamegraph](https://www.brendangregg.com/flamegraphs.html) using the list of stacktraces. Outputs an array of strings which can be used by [flamegraph.pl utility](https://github.com/brendangregg/FlameGraph) to render an SVG of the flamegraph.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

flameGraph(traces, [size], [ptr])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `traces`   : a stacktrace. Array(UInt64).
- `size`   : an allocation size for memory profiling. (optional - default `1`   ). UInt64.
- `ptr`   : an allocation address. (optional - default `0`   ). UInt64.

In the case where `ptr != 0` , a flameGraph will map allocations (size > 0) and deallocations (size < 0) with the same size and ptr. Only allocations which were not freed are shown. Non mapped deallocations are ignored.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- An array of strings for use with[  flamegraph.pl utility](https://github.com/brendangregg/FlameGraph)   . Array(String).

## groupArray [¶](https://www.tinybird.co/docs/about:blank#grouparray)

Syntax: `groupArray(x)` or `groupArray(max_size)(x)`

Creates an array of argument values. Values can be added to the array in any (indeterminate) order.

The second version (with the `max_size` parameter) limits the size of the resulting array to `max_size` elements. For example, `groupArray(1)(x)` is equivalent to `[any (x)]`.

In some cases, you can still rely on the order of execution. This applies to cases when `SELECT` comes from a subquery that uses `ORDER BY` if the subquery result is small enough.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT * FROM default.ck
┌─id─┬─name─────┐
│  1 │ zhangsan │
│  1 │ ᴺᵁᴸᴸ     │
│  1 │ lisi     │
│  2 │ wangwu   │
└────┴──────────┘ Query:

select id, groupArray(10)(name) from default.ck group by id Result:

┌─id─┬─groupArray(10)(name)─┐
│  1 │ ['zhangsan','lisi']  │
│  2 │ ['wangwu']           │
└────┴──────────────────────┘ The groupArray function will remove ᴺᵁᴸᴸ value based on the above results.

- Alias: `array_agg`  .

## groupArrayInsertAt [¶](https://www.tinybird.co/docs/about:blank#grouparrayinsertat)

Inserts a value into the array at the specified position.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

groupArrayInsertAt(default_x, size)(x, pos) If in one query several values are inserted into the same position, the function behaves in the following ways:

- If a query is executed in a single thread, the first one of the inserted values is used.
- If a query is executed in multiple threads, the resulting value is an undetermined one of the inserted values.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Value to be inserted. Expression resulting in one of the supported data types.
- `pos`   : Position at which the specified element `x`   is to be inserted. Index numbering in the array starts from zero. UInt32.
- `default_x`   : Default value for substituting in empty positions. Optional parameter. Expression resulting in the data type configured for the `x`   parameter. If `default_x`   isn't defined, the [default values are used.
- `size`   : Length of the resulting array. Optional parameter. When using this parameter, the default value `default_x`   must be specified. UInt32.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array with inserted values.

Type: Array.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT groupArrayInsertAt(toString(number), number * 2) FROM numbers(5) Result:

┌─groupArrayInsertAt(toString(number), multiply(number, 2))─┐
│ ['0','','1','','2','','3','','4']                         │
└───────────────────────────────────────────────────────────┘ Query:

SELECT groupArrayInsertAt('-')(toString(number), number * 2) FROM numbers(5) Result:

┌─groupArrayInsertAt('-')(toString(number), multiply(number, 2))─┐
│ ['0','-','1','-','2','-','3','-','4']                          │
└────────────────────────────────────────────────────────────────┘ Query:

SELECT groupArrayInsertAt('-', 5)(toString(number), number * 2) FROM numbers(5) Result:

┌─groupArrayInsertAt('-', 5)(toString(number), multiply(number, 2))─┐
│ ['0','-','1','-','2']                                             │
└───────────────────────────────────────────────────────────────────┘ Multi-threaded insertion of elements into one position.

Query:

SELECT groupArrayInsertAt(number, 0) FROM numbers_mt(10) SETTINGS max_block_size = 1```

As a result of this query you get random integer in the `[0,9]` range. For example:

``` text
┌─groupArrayInsertAt(number, 0)─┐
│ [7]                           │
└───────────────────────────────┘
## groupArrayIntersect [¶](https://www.tinybird.co/docs/about:blank#grouparrayintersect)

Return an intersection of given arrays (Return all items of arrays, that are in all given arrays).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

groupArrayIntersect(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Argument (column name or expression).

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array that contains elements that are in all arrays.

Type: Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Consider table `numbers`:

┌─a──────────────┐
│ [1,2,4]        │
│ [1,5,2,8,-1,0] │
│ [1,5,7,5,8,2]  │
└────────────────┘ Query with column name as argument:

SELECT groupArrayIntersect(a) as intersection FROM numbers Result:

┌─intersection──────┐
│ [1, 2]            │
└───────────────────┘
## groupArrayLast [¶](https://www.tinybird.co/docs/about:blank#grouparraylast)

Syntax: `groupArrayLast(max_size)(x)`

Creates an array of last argument values. For example, `groupArrayLast(1)(x)` is equivalent to `[anyLast (x)]`.

In some cases, you can still rely on the order of execution. This applies to cases when `SELECT` comes from a subquery that uses `ORDER BY` if the subquery result is small enough.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

select groupArrayLast(2)(number+1) numbers from numbers(10) Result:

┌─numbers─┐
│ [9,10]  │
└─────────┘ In compare to `groupArray`:

select groupArray(2)(number+1) numbers from numbers(10) ┌─numbers─┐
│ [1,2]   │
└─────────┘
## groupArrayMovingAvg [¶](https://www.tinybird.co/docs/about:blank#grouparraymovingavg)

Calculates the moving average of input values.

groupArrayMovingAvg(numbers_for_summing)
groupArrayMovingAvg(window_size)(numbers_for_summing) The function can take the window size as a parameter. If left unspecified, the function takes the window size equal to the number of rows in the column.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `numbers_for_summing`   : Expression resulting in a numeric data type value.
- `window_size`   : Size of the calculation window.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of the same size and type as the input data.

The function uses rounding towards zero. It truncates the decimal places insignificant for the resulting data type.

## groupArrayMovingSum [¶](https://www.tinybird.co/docs/about:blank#grouparraymovingsum)

Calculates the moving sum of input values.

groupArrayMovingSum(numbers_for_summing)
groupArrayMovingSum(window_size)(numbers_for_summing) The function can take the window size as a parameter. If left unspecified, the function takes the window size equal to the number of rows in the column.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `numbers_for_summing`   : Expression resulting in a numeric data type value.
- `window_size`   : Size of the calculation window.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of the same size and type as the input data.

## groupArraySample [¶](https://www.tinybird.co/docs/about:blank#grouparraysample)

Creates an array of sample argument values. The size of the resulting array is limited to `max_size` elements. Argument values are selected and added to the array randomly.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

groupArraySample(max_size[, seed])(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `max_size`   : Maximum size of the resulting array. UInt64.
- `seed`   : Seed for the random number generator. Optional. UInt64. Default value: `123456`  .
- `x`   : Argument (column name or expression).

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Array of randomly selected `x`   arguments.

Type: Array.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Consider table `colors`:

┌─id─┬─color──┐
│  1 │ red    │
│  2 │ blue   │
│  3 │ green  │
│  4 │ white  │
│  5 │ orange │
└────┴────────┘ Query with column name as argument:

SELECT groupArraySample(3)(color) as newcolors FROM colors Result:

┌─newcolors──────────────────┐
│ ['white','blue','green']   │
└────────────────────────────┘ Query with column name and different seed:

SELECT groupArraySample(3, 987654321)(color) as newcolors FROM colors Result:

┌─newcolors──────────────────┐
│ ['red','orange','green']   │
└────────────────────────────┘ Query with expression as argument:

SELECT groupArraySample(3)(concat('light-', color)) as newcolors FROM colors Result:

┌─newcolors───────────────────────────────────┐
│ ['light-blue','light-orange','light-green'] │
└─────────────────────────────────────────────┘
## groupArraySorted {#groupArraySorted} [¶](https://www.tinybird.co/docs/about:blank#grouparraysorted-grouparraysorted)

Returns an array with the first N items in ascending order.

groupArraySorted(N)(column) **Arguments**

- `N`   – The number of elements to return.
- `column`   – The value (Integer, String, Float and other Generic types).

**Example**

Gets the first 10 numbers:

SELECT groupArraySorted(10)(number) FROM numbers(100) ┌─groupArraySorted(10)(number)─┐
│ [0,1,2,3,4,5,6,7,8,9]        │
└──────────────────────────────┘ Gets all the String implementations of all numbers in column:

SELECT groupArraySorted(5)(str) FROM (SELECT toString(number) as str FROM numbers(5)) ┌─groupArraySorted(5)(str)─┐
│ ['0','1','2','3','4']    │
└──────────────────────────┘
## groupBitAnd [¶](https://www.tinybird.co/docs/about:blank#groupbitand)

Applies bit-wise `AND` for series of numbers.

groupBitAnd(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `UInt*` or `Int*` type.

### Return value [¶](https://www.tinybird.co/docs/about:blank#return-value)

Value of the `UInt*` or `Int*` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Test data:

binary     decimal
00101100 = 44
00011100 = 28
00001101 = 13
01010101 = 85 Query:

SELECT groupBitAnd(num) FROM t Where `num` is the column with the test data.

Result:

binary     decimal
00000100 = 4
## groupBitmap [¶](https://www.tinybird.co/docs/about:blank#groupbitmap)

Bitmap or Aggregate calculations from a unsigned integer column, return cardinality of type UInt64, if add suffix -State, then return bitmap object.

groupBitmap(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `UInt*` type.

### Return value [¶](https://www.tinybird.co/docs/about:blank#return-value)

Value of the `UInt64` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Test data:

UserID
1
1
2
3 Query:

SELECT groupBitmap(UserID) as num FROM t Result:

num
3
## groupBitmapAnd [¶](https://www.tinybird.co/docs/about:blank#groupbitmapand)

Calculations the AND of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object.

groupBitmapAnd(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `AggregateFunction(groupBitmap, UInt*)` type.

### Return value [¶](https://www.tinybird.co/docs/about:blank#return-value)

Value of the `UInt64` type.

## groupBitmapOr [¶](https://www.tinybird.co/docs/about:blank#groupbitmapor)

Calculations the OR of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object. This is equivalent to `groupBitmapMerge`.

groupBitmapOr(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `AggregateFunction(groupBitmap, UInt*)` type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Value of the `UInt64` type.

## groupBitmapXor [¶](https://www.tinybird.co/docs/about:blank#groupbitmapxor)

Calculations the XOR of a bitmap column, return cardinality of type UInt64, if add suffix -State, then return bitmap object.

groupBitmapOr(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `AggregateFunction(groupBitmap, UInt*)` type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Value of the `UInt64` type.

## groupBitOr [¶](https://www.tinybird.co/docs/about:blank#groupbitor)

Applies bit-wise `OR` for series of numbers.

groupBitOr(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `UInt*` or `Int*` type.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Value of the `UInt*` or `Int*` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Test data:

binary     decimal
00101100 = 44
00011100 = 28
00001101 = 13
01010101 = 85 Query:

SELECT groupBitOr(num) FROM t Where `num` is the column with the test data.

Result:

binary     decimal
01111101 = 125
## groupBitXor [¶](https://www.tinybird.co/docs/about:blank#groupbitxor)

Applies bit-wise `XOR` for series of numbers.

groupBitXor(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` – An expression that results in `UInt*` or `Int*` type.

### Return value [¶](https://www.tinybird.co/docs/about:blank#return-value)

Value of the `UInt*` or `Int*` type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Test data:

binary     decimal
00101100 = 44
00011100 = 28
00001101 = 13
01010101 = 85 Query:

SELECT groupBitXor(num) FROM t Where `num` is the column with the test data.

Result:

binary     decimal
01101000 = 104
## groupConcat [¶](https://www.tinybird.co/docs/about:blank#groupconcat)

Calculates a concatenated string from a group of strings, optionally separated by a delimiter, and optionally limited by a maximum number of elements.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

groupConcat[(delimiter [, limit])](expression)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expression`   : The expression or column name that outputs strings to be concatenated..
- `delimiter`   : A string that will be used to separate concatenated values. This parameter is optional and defaults to an empty string if not specified.
- `limit`   : A positive integer specifying the maximum number of elements to concatenate. If more elements are present, excess elements are ignored. This parameter is optional.

If delimiter is specified without limit, it must be the first parameter. If both delimiter and limit are specified, delimiter must precede limit.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a string consisting of the concatenated values of the column or expression. If the group has no elements or only null elements, and the function doesn't specify a handling for only null values, the result is a nullable string with a null value.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

Input table:

┌─id─┬─name─┐
│ 1  │  John│
│ 2  │  Jane│
│ 3  │   Bob│
└────┴──────┘
1. Basic usage without a delimiter:

Query:

SELECT groupConcat(Name) FROM Employees Result:

JohnJaneBob This concatenates all names into one continuous string without any separator.

1. Using comma as a delimiter:

Query:

SELECT groupConcat(', ')(Name)  FROM Employees Result:

John, Jane, Bob This output shows the names separated by a comma followed by a space.

1. Limiting the number of concatenated elements

Query:

SELECT groupConcat(', ', 2)(Name) FROM Employees Result:

John, Jane This query limits the output to the first two names, even though there are more names in the table.

## groupUniqArray [¶](https://www.tinybird.co/docs/about:blank#groupuniqarray)

**Syntax:** `groupUniqArray(x)` or `groupUniqArray(max_size)(x)`

Creates an array from different argument values. Memory consumption is the same as for the uniqExact function.

The second version (with the `max_size` parameter) limits the size of the resulting array to `max_size` elements. For example, `groupUniqArray(1)(x)` is equivalent to `[any(x)]`.

## intervalLengthSum [¶](https://www.tinybird.co/docs/about:blank#intervallengthsum)

Calculates the total length of union of all ranges (segments on numeric axis).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

intervalLengthSum(start, end)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start`   : The starting value of the interval. Int32, Int64, UInt32, UInt64, Float32, Float64, DateTime or Date.
- `end`   : The ending value of the interval. Int32, Int64, UInt32, UInt64, Float32, Float64, DateTime or Date.

Arguments must be of the same data type. Otherwise, an exception will be thrown.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Total length of union of all ranges (segments on numeric axis). Depending on the type of the argument, the return value may be UInt64 or Float64 type.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

1. Input table:

┌─id─┬─start─┬─end─┐
│ a  │   1.1 │ 2.9 │
│ a  │   2.5 │ 3.2 │
│ a  │     4 │   5 │
└────┴───────┴─────┘ In this example, the arguments of the Float32 type are used. The function returns a value of the Float64 type.

Result is the sum of lengths of intervals `[1.1, 3.2]` (union of `[1.1, 2.9]` and `[2.5, 3.2]` ) and `[4, 5]`

Query:

SELECT id, intervalLengthSum(start, end), toTypeName(intervalLengthSum(start, end)) FROM fl_interval GROUP BY id ORDER BY id Result:

┌─id─┬─intervalLengthSum(start, end)─┬─toTypeName(intervalLengthSum(start, end))─┐
│ a  │                           3.1 │ Float64                                   │
└────┴───────────────────────────────┴───────────────────────────────────────────┘
1. Input table:

┌─id─┬───────────────start─┬─────────────────end─┐
│ a  │ 2020-01-01 01:12:30 │ 2020-01-01 02:10:10 │
│ a  │ 2020-01-01 02:05:30 │ 2020-01-01 02:50:31 │
│ a  │ 2020-01-01 03:11:22 │ 2020-01-01 03:23:31 │
└────┴─────────────────────┴─────────────────────┘ In this example, the arguments of the DateTime type are used. The function returns a value in seconds.

Query:

SELECT id, intervalLengthSum(start, end), toTypeName(intervalLengthSum(start, end)) FROM dt_interval GROUP BY id ORDER BY id Result:

┌─id─┬─intervalLengthSum(start, end)─┬─toTypeName(intervalLengthSum(start, end))─┐
│ a  │                          6610 │ UInt64                                    │
└────┴───────────────────────────────┴───────────────────────────────────────────┘
1. Input table:

┌─id─┬──────start─┬────────end─┐
│ a  │ 2020-01-01 │ 2020-01-04 │
│ a  │ 2020-01-12 │ 2020-01-18 │
└────┴────────────┴────────────┘ In this example, the arguments of the Date type are used. The function returns a value in days.

Query:

SELECT id, intervalLengthSum(start, end), toTypeName(intervalLengthSum(start, end)) FROM date_interval GROUP BY id ORDER BY id Result:

┌─id─┬─intervalLengthSum(start, end)─┬─toTypeName(intervalLengthSum(start, end))─┐
│ a  │                             9 │ UInt64                                    │
└────┴───────────────────────────────┴───────────────────────────────────────────┘
## kolmogorovSmirnovTest [¶](https://www.tinybird.co/docs/about:blank#kolmogorovsmirnovtest)

Applies Kolmogorov-Smirnov's test to samples from two populations.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

kolmogorovSmirnovTest([alternative, computation_method])(sample_data, sample_index) Values of both samples are in the `sample_data` column. If `sample_index` equals to 0 then the value in that row belongs to the sample from the first population. Otherwise it belongs to the sample from the second population. Samples must belong to continuous, one-dimensional probability distributions.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `sample_data`   : Sample data. Integer, Float or Decimal.
- `sample_index`   : Sample index. Integer.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `alternative`   : alternative hypothesis. (Optional, default: `'two-sided'`   .) String.   Let F(x) and G(x) be the CDFs of the first and second distributions respectively.
  - `'two-sided'`     The null hypothesis is that samples come from the same distribution, e.g. F(x) = G(x) for all x.     And the alternative is that the distributions aren't identical.
  - `'greater'`     The null hypothesis is that values in the first sample are*    stochastically smaller*     than those in the second one,     e.g. the CDF of first distribution lies above and hence to the left of that for the second one.     Which in fact means that F(x) >= G(x) for all x. And the alternative in this case is that F(x) < G(x) for at least one x.
  - `'less'`    .     The null hypothesis is that values in the first sample are*    stochastically greater*     than those in the second one,     e.g. the CDF of first distribution lies below and hence to the right of that for the second one.     Which in fact means that F(x) <= G(x) for all x. And the alternative in this case is that F(x) > G(x) for at least one x.
- `computation_method`   : the method used to compute p-value. (Optional, default: `'auto'`   .) String.
  - `'exact'`     - calculation is performed using precise probability distribution of the test statistics. Compute intensive and wasteful except for small samples.
  - `'asymp'`     ( `'asymptotic'`     ) - calculation is performed using an approximation. For large sample sizes, the exact and asymptotic p-values are very similar.
  - `'auto'`     - the `'exact'`     method is used when a maximum number of samples is less than 10'000.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Tuple with two elements:

- calculated statistic. Float64.
- calculated p-value. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT kolmogorovSmirnovTest('less', 'exact')(value, num)
FROM
(
    SELECT
        randNormal(0, 10) AS value,
        0 AS num
    FROM numbers(10000)
    UNION ALL
    SELECT
        randNormal(0, 10) AS value,
        1 AS num
    FROM numbers(10000)
) Result:

┌─kolmogorovSmirnovTest('less', 'exact')(value, num)─┐
│ (0.009899999999999996,0.37528595205132287)         │
└────────────────────────────────────────────────────┘ Note: P-value is bigger than 0.05 (for confidence level of 95%), so null hypothesis isn't rejected.

Query:

SELECT kolmogorovSmirnovTest('two-sided', 'exact')(value, num)
FROM
(
    SELECT
        randStudentT(10) AS value,
        0 AS num
    FROM numbers(100)
    UNION ALL
    SELECT
        randNormal(0, 10) AS value,
        1 AS num
    FROM numbers(100)
) Result:

┌─kolmogorovSmirnovTest('two-sided', 'exact')(value, num)─┐
│ (0.4100000000000002,6.61735760482795e-8)                │
└─────────────────────────────────────────────────────────┘ Note: P-value is less than 0.05 (for confidence level of 95%), so null hypothesis is rejected.

## kurtPop [¶](https://www.tinybird.co/docs/about:blank#kurtpop)

Computes the kurtosis of a sequence.

kurtPop(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` : Expression returning a number.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The kurtosis of the given distribution. Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT kurtPop(value) FROM series_with_value_column
## kurtSamp [¶](https://www.tinybird.co/docs/about:blank#kurtsamp)

Computes the sample kurtosis of a sequence.

It represents an unbiased estimate of the kurtosis of a random variable if passed values form its sample.

kurtSamp(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` : Expression returning a number.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The kurtosis of the given distribution. Type: Float64. If `n <= 1` ( `n` is a size of the sample), then the function returns `nan`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT kurtSamp(value) FROM series_with_value_column
## largestTriangleThreeBuckets [¶](https://www.tinybird.co/docs/about:blank#largesttrianglethreebuckets)

Applies the Largest-Triangle-Three-Buckets algorithm to the input data. The algorithm is used for downsampling time series data for visualization. It is designed to operate on series sorted by x coordinate. It works by dividing the sorted series into buckets and then finding the largest triangle in each bucket. The number of buckets is equal to the number of points in the resulting series. the function will sort data by `x` and then apply the downsampling algorithm to the sorted data.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

largestTriangleThreeBuckets(n)(x, y) Alias: `lttb`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : x coordinate. Integer, Float, Decimal, Date, Date32, DateTime, DateTime64.
- `y`   : y coordinate. Integer, Float, Decimal, Date, Date32, DateTime, DateTime64.

NaNs are ignored in the provided series, meaning that any NaN values will be excluded from the analysis. This ensures that the function operates only on valid numerical data.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `n`   : number of points in the resulting series. UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Array of Tuple with two elements:

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─────x───────┬───────y──────┐
│ 1.000000000 │ 10.000000000 │
│ 2.000000000 │ 20.000000000 │
│ 3.000000000 │ 15.000000000 │
│ 8.000000000 │ 60.000000000 │
│ 9.000000000 │ 55.000000000 │
│ 10.00000000 │ 70.000000000 │
│ 4.000000000 │ 30.000000000 │
│ 5.000000000 │ 40.000000000 │
│ 6.000000000 │ 35.000000000 │
│ 7.000000000 │ 50.000000000 │
└─────────────┴──────────────┘ Query:

SELECT largestTriangleThreeBuckets(4)(x, y) FROM largestTriangleThreeBuckets_test Result:

┌────────largestTriangleThreeBuckets(4)(x, y)───────────┐
│           [(1,10),(3,15),(9,55),(10,70)]              │
└───────────────────────────────────────────────────────┘
## last_value [¶](https://www.tinybird.co/docs/about:blank#last-value)

Selects the last encountered value, similar to `anyLast` , but could accept NULL. Mostly it should be used with Window Functions. Without Window Functions the result will be random if the source stream isn't ordered.

## mannWhitneyUTest [¶](https://www.tinybird.co/docs/about:blank#mannwhitneyutest)

Applies the Mann-Whitney rank test to samples from two populations.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

mannWhitneyUTest[(alternative[, continuity_correction])](sample_data, sample_index) Values of both samples are in the `sample_data` column. If `sample_index` equals to 0 then the value in that row belongs to the sample from the first population. Otherwise it belongs to the sample from the second population. The null hypothesis is that two populations are stochastically equal. Also one-sided hypothesises can be tested. This test doesn't assume that data have normal distribution.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `sample_data`   : sample data. Integer, Float or Decimal.
- `sample_index`   : sample index. Integer.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `alternative`   : alternative hypothesis. (Optional, default: `'two-sided'`   .) String.
  - `'two-sided'`     - `'greater'`     - `'less'`    .
- `continuity_correction`   : if not 0 then continuity correction in the normal approximation for the p-value is applied. (Optional, default: 1.) UInt64.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Tuple with two elements:

- calculated U-statistic. Float64.
- calculated p-value. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─sample_data─┬─sample_index─┐
│          10 │            0 │
│          11 │            0 │
│          12 │            0 │
│           1 │            1 │
│           2 │            1 │
│           3 │            1 │
└─────────────┴──────────────┘ Query:

SELECT mannWhitneyUTest('greater')(sample_data, sample_index) FROM mww_ttest Result:

┌─mannWhitneyUTest('greater')(sample_data, sample_index)─┐
│ (9,0.04042779918503192)                                │
└────────────────────────────────────────────────────────┘
## max [¶](https://www.tinybird.co/docs/about:blank#max)

Aggregate function that calculates the maximum across a group of values.

### Syntax: [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT max(salary) FROM employees SELECT department, max(salary) FROM employees GROUP BY department If you need non-aggregate function to choose a maximum of two values, see `greatest`:

SELECT greatest(a, b) FROM table
## maxIntersections [¶](https://www.tinybird.co/docs/about:blank#maxintersections)

Aggregate function that calculates the maximum number of times that a group of intervals intersects each other (if all the intervals intersect at least once).

The syntax is:

maxIntersections(start_column, end_column)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start_column`   – the numeric column that represents the start of each interval. If `start_column`   is `NULL`   or 0 then the interval will be skipped.
- `end_column`   - the numeric column that represents the end of each interval. If `end_column`   is `NULL`   or 0 then the interval will be skipped.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Returns the maximum number of intersected intervals.

## maxIntersectionsPosition [¶](https://www.tinybird.co/docs/about:blank#maxintersectionsposition)

Aggregate function that calculates the positions of the occurrences of the `maxIntersections` function.

The syntax is:

maxIntersectionsPosition(start_column, end_column)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `start_column`   – the numeric column that represents the start of each interval. If `start_column`   is `NULL`   or 0 then the interval will be skipped.
- `end_column`   - the numeric column that represents the end of each interval. If `end_column`   is `NULL`   or 0 then the interval will be skipped.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Returns the start positions of the maximum number of intersected intervals.

## maxMap [¶](https://www.tinybird.co/docs/about:blank#maxmap)

Calculates the maximum from `value` array according to the keys specified in the `key` array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

maxMap(key, value) or

maxMap(Tuple(key, value)) Alias: `maxMappedArrays`

- Passing a tuple of keys and value arrays is identical to passing two arrays of keys and values.
- The number of elements in `key`   and `value`   must be the same for each row that is totaled.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `key`   : Array of keys. Array.
- `value`   : Array of values. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of two arrays: keys in sorted order, and values calculated for the corresponding keys. Tuple(Array, Array).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT maxMap(a, b)
FROM values('a Array(Char), b Array(Int64)', (['x', 'y'], [2, 2]), (['y', 'z'], [3, 1])) Result:

┌─maxMap(a, b)───────────┐
│ [['x','y','z'],[2,3,1]]│
└────────────────────────┘
## meanZTest [¶](https://www.tinybird.co/docs/about:blank#meanztest)

Applies mean z-test to samples from two populations.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

meanZTest(population_variance_x, population_variance_y, confidence_level)(sample_data, sample_index) Values of both samples are in the `sample_data` column. If `sample_index` equals to 0 then the value in that row belongs to the sample from the first population. Otherwise it belongs to the sample from the second population. The null hypothesis is that means of populations are equal. Normal distribution is assumed. Populations may have unequal variance and the variances are known.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `sample_data`   : Sample data. Integer, Float or Decimal.
- `sample_index`   : Sample index. Integer.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `population_variance_x`   : Variance for population x. Float.
- `population_variance_y`   : Variance for population y. Float.
- `confidence_level`   : Confidence level in order to calculate confidence intervals. Float.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Tuple with four elements:

- calculated t-statistic. Float64.
- calculated p-value. Float64.
- calculated confidence-interval-low. Float64.
- calculated confidence-interval-high. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─sample_data─┬─sample_index─┐
│        20.3 │            0 │
│        21.9 │            0 │
│        22.1 │            0 │
│        18.9 │            1 │
│          19 │            1 │
│        20.3 │            1 │
└─────────────┴──────────────┘ Query:

SELECT meanZTest(0.7, 0.45, 0.95)(sample_data, sample_index) FROM mean_ztest Result:

┌─meanZTest(0.7, 0.45, 0.95)(sample_data, sample_index)────────────────────────────┐
│ (3.2841296025548123,0.0010229786769086013,0.8198428246768334,3.2468238419898365) │
└──────────────────────────────────────────────────────────────────────────────────┘
## median [¶](https://www.tinybird.co/docs/about:blank#median)

The `median*` functions are the aliases for the corresponding `quantile*` functions. They calculate median of a numeric data sample.

Functions:

- `median`   : Alias for `quantile`  .
- `medianDeterministic`   : Alias for `quantileDeterministic`  .
- `medianExact`   : Alias for `quantileExact`  .
- `medianExactWeighted`   : Alias for `quantileExactWeighted`  .
- `medianTiming`   : Alias for `quantileTiming`  .
- `medianTimingWeighted`   : Alias for `quantileTimingWeighted`  .
- `medianTDigest`   : Alias for `quantileTDigest`  .
- `medianTDigestWeighted`   : Alias for `quantileTDigestWeighted`  .
- `medianBFloat16`   : Alias for `quantileBFloat16`  .
- `medianDD`   : Alias for `quantileDD`  .

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─val─┐
│   1 │
│   1 │
│   2 │
│   3 │
└─────┘ Query:

SELECT medianDeterministic(val, 1) FROM t Result:

┌─medianDeterministic(val, 1)─┐
│                         1.5 │
└─────────────────────────────┘
## min [¶](https://www.tinybird.co/docs/about:blank#min)

Aggregate function that calculates the minimum across a group of values.

### Syntax: [¶](https://www.tinybird.co/docs/about:blank#syntax)

SELECT min(salary) FROM employees SELECT department, min(salary) FROM employees GROUP BY department If you need non-aggregate function to choose a minimum of two values, see `least`:

SELECT least(a, b) FROM table
## minMap [¶](https://www.tinybird.co/docs/about:blank#minmap)

Calculates the minimum from `value` array according to the keys specified in the `key` array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

`minMap(key, value)` or

minMap(Tuple(key, value)) Alias: `minMappedArrays`

- Passing a tuple of keys and value arrays is identical to passing an array of keys and an array of values.
- The number of elements in `key`   and `value`   must be the same for each row that is totaled.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `key`   : Array of keys. Array.
- `value`   : Array of values. Array.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns a tuple of two arrays: keys in sorted order, and values calculated for the corresponding keys. Tuple(Array, Array).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT minMap(a, b)
FROM values('a Array(Int32), b Array(Int64)', ([1, 2], [2, 2]), ([2, 3], [1, 1])) Result:

┌─minMap(a, b)──────┐
│ ([1,2,3],[2,1,1]) │
└───────────────────┘
## quantile [¶](https://www.tinybird.co/docs/about:blank#quantile)

Computes an approximate quantile of a numeric data sequence.

This function applies reservoir sampling with a reservoir size up to 8192 and a random number generator for sampling. The result is non-deterministic. To get an exact quantile, use the `quantileExact` function.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

Note that for an empty numeric sequence, `quantile` will return NaN, but its `quantile*` variants will return either NaN or a default value for the sequence type, depending on the variant.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantile(level)(expr) Alias: `median`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─val─┐
│   1 │
│   1 │
│   2 │
│   3 │
└─────┘ Query:

SELECT quantile(val) FROM t Result:

┌─quantile(val)─┐
│           1.5 │
└───────────────┘
## quantileBFloat16 [¶](https://www.tinybird.co/docs/about:blank#quantilebfloat16)

Computes an approximate quantile of a sample consisting of bfloat16 numbers. `bfloat16` is a floating-point data type with 1 sign bit, 8 exponent bits and 7 fraction bits. The function converts input values to 32-bit floats and takes the most significant 16 bits. Then it calculates `bfloat16` quantile value and converts the result to a 64-bit float by appending zero bits. The function is a fast quantile estimator with a relative error no more than 0.390625%.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileBFloat16[(level)](https://www.tinybird.co/docs/expr) Alias: `medianBFloat16`

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Column with numeric data. Integer, Float.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `level`   : Level of quantile. Optional. Possible values are in the range from 0 to 1. Default value: 0.5. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table has an integer and a float columns:

┌─a─┬─────b─┐
│ 1 │ 1.001 │
│ 2 │ 1.002 │
│ 3 │ 1.003 │
│ 4 │ 1.004 │
└───┴───────┘ Query to calculate 0.75-quantile (third quartile):

SELECT quantileBFloat16(0.75)(a), quantileBFloat16(0.75)(b) FROM example_table Result:

┌─quantileBFloat16(0.75)(a)─┬─quantileBFloat16(0.75)(b)─┐
│                         3 │                         1 │
└───────────────────────────┴───────────────────────────┘ Note that all floating point values in the example are truncated to 1.0 when converting to `bfloat16`.

## quantileBFloat16Weighted [¶](https://www.tinybird.co/docs/about:blank#quantilebfloat16weighted)

Like `quantileBFloat16` but takes into account the weight of each sequence member.

## quantileDD [¶](https://www.tinybird.co/docs/about:blank#quantiledd)

Computes an approximate quantile of a sample with relative-error guarantees. It works by building a [DD](https://www.vldb.org/pvldb/vol12/p2195-masson.pdf).

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileDD(relative_accuracy, [level])(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Column with numeric data. Integer, Float.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `relative_accuracy`   : Relative accuracy of the quantile. Possible values are in the range from 0 to 1. Float. The size of the sketch depends on the range of the data and the relative accuracy. The larger the range and the smaller the relative accuracy, the larger the sketch. The rough memory size of the of the sketch is `log(max_value/min_value)/relative_accuracy`   . The recommended value is 0.001 or higher.
- `level`   : Level of quantile. Optional. Possible values are in the range from 0 to 1. Default value: 0.5. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table has an integer and a float columns:

┌─a─┬─────b─┐
│ 1 │ 1.001 │
│ 2 │ 1.002 │
│ 3 │ 1.003 │
│ 4 │ 1.004 │
└───┴───────┘ Query to calculate 0.75-quantile (third quartile):

SELECT quantileDD(0.01, 0.75)(a), quantileDD(0.01, 0.75)(b) FROM example_table Result:

┌─quantileDD(0.01, 0.75)(a)─┬─quantileDD(0.01, 0.75)(b)─┐
│               2.974233423476717 │                            1.01 │
└─────────────────────────────────┴─────────────────────────────────┘
## quantileDeterministic [¶](https://www.tinybird.co/docs/about:blank#quantiledeterministic)

Computes an approximate quantile of a numeric data sequence.

This function applies reservoir sampling with a reservoir size up to 8192 and deterministic algorithm of sampling. The result is deterministic. To get an exact quantile, use the `quantileExact` function.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileDeterministic(level)(expr, determinator) Alias: `medianDeterministic`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.
- `determinator`   : Number whose hash is used instead of a random number generator in the reservoir sampling algorithm to make the result of sampling deterministic. As a determinator you can use any deterministic positive number, for example, a user id or an event id. If the same determinator value occurs too often, the function works incorrectly.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─val─┐
│   1 │
│   1 │
│   2 │
│   3 │
└─────┘ Query:

SELECT quantileDeterministic(val, 1) FROM t Result:

┌─quantileDeterministic(val, 1)─┐
│                           1.5 │
└───────────────────────────────┘
## quantileExact [¶](https://www.tinybird.co/docs/about:blank#quantileexact)

Exactly computes the quantile of a numeric data sequence.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes `O(n)` memory, where `n` is a number of values that were passed. However, for a small number of values, the function is very effective.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExact(level)(expr) Alias: `medianExact`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantileExact(number) FROM numbers(10) Result:

┌─quantileExact(number)─┐
│                     5 │
└───────────────────────┘
## quantileExactLow [¶](https://www.tinybird.co/docs/about:blank#quantileexactlow)

Similar to `quantileExact` , this computes the exact quantile of a numeric data sequence.

To get the exact value, all the passed values are combined into an array, which is then fully sorted.  The sorting algorithm's complexity is `O(N·log(N))` , where `N = std::distance(first, last)` comparisons.

The return value depends on the quantile level and the number of elements in the selection, i.e. if the level is 0.5, then the function returns the lower median value for an even number of elements and the middle median value for an odd number of elements. Median is calculated similarly to the [median_low](https://docs.python.org/3/library/statistics.html#statistics.median_low) implementation which is used in python.

For all other levels, the element at the index corresponding to the value of `level * size_of_array` is returned. For example:

SELECT quantileExactLow(0.1)(number) FROM numbers(10)

┌─quantileExactLow(0.1)(number)─┐
│                             1 │
└───────────────────────────────┘ When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactLow(level)(expr) Alias: `medianExactLow`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantileExactLow(number) FROM numbers(10) Result:

┌─quantileExactLow(number)─┐
│                        4 │
└──────────────────────────┘
## quantileExactHigh [¶](https://www.tinybird.co/docs/about:blank#quantileexacthigh)

Similar to `quantileExact` , this computes the exact quantile of a numeric data sequence.

All the passed values are combined into an array, which is then fully sorted, to get the exact value.  The sorting algorithm's complexity is `O(N·log(N))` , where `N = std::distance(first, last)` comparisons.

The return value depends on the quantile level and the number of elements in the selection, i.e. if the level is 0.5, then the function returns the higher median value for an even number of elements and the middle median value for an odd number of elements. Median is calculated similarly to the [median_high](https://docs.python.org/3/library/statistics.html#statistics.median_high) implementation which is used in python. For all other levels, the element at the index corresponding to the value of `level * size_of_array` is returned.

This implementation behaves exactly similar to the current `quantileExact` implementation.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactHigh(level)(expr) Alias: `medianExactHigh`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantileExactHigh(number) FROM numbers(10) Result:

┌─quantileExactHigh(number)─┐
│                         5 │
└───────────────────────────┘
## quantileExactExclusive [¶](https://www.tinybird.co/docs/about:blank#quantileexactexclusive)

Exactly computes the quantile of a numeric data sequence.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes `O(n)` memory, where `n` is a number of values that were passed. However, for a small number of values, the function is very effective.

This function is equivalent to [PERCENTILE.EXC](https://support.microsoft.com/en-us/office/percentile-exc-function-bbaa7204-e9e1-4010-85bf-c31dc5dce4ba) Excel function, ( [type R6](https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample) ).

When using multiple `quantileExactExclusive` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantilesExactExclusive` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactExclusive(level)(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `level`   : Level of quantile. Optional. Possible values: (0, 1): bounds not included. Default value: 0.5. At `level=0.5`   the function calculates median. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

## quantileExactInclusive [¶](https://www.tinybird.co/docs/about:blank#quantileexactinclusive)

Exactly computes the quantile of a numeric data sequence.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes `O(n)` memory, where `n` is a number of values that were passed. However, for a small number of values, the function is very effective.

This function is equivalent to [PERCENTILE.INC](https://support.microsoft.com/en-us/office/percentile-inc-function-680f9539-45eb-410b-9a5e-c1355e5fe2ed) Excel function, ( [type R7](https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample) ).

When using multiple `quantileExactInclusive` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantilesExactInclusive` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactInclusive(level)(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `level`   : Level of quantile. Optional. Possible values: [0, 1]: bounds included. Default value: 0.5. At `level=0.5`   the function calculates median. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

## quantileExactWeighted [¶](https://www.tinybird.co/docs/about:blank#quantileexactweighted)

Exactly computes the [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence, taking into account the weight of each element.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Each value is counted with its weight, as if it's present `weight` times. A hash table is used in the algorithm. Because of this, if the passed values ​​are frequently repeated, the function consumes less RAM than quantileExact. You can use this function instead of `quantileExact` and specify the weight 1.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactWeighted(level)(expr, weight) Alias: `medianExactWeighted`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.
- `weight`   : Column with weights of sequence members. Weight is a number of value occurrences with Unsigned integer types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─n─┬─val─┐
│ 0 │   3 │
│ 1 │   2 │
│ 2 │   1 │
│ 5 │   4 │
└───┴─────┘ Query:

SELECT quantileExactWeighted(n, val) FROM t Result:

┌─quantileExactWeighted(n, val)─┐
│                             1 │
└───────────────────────────────┘
## quantileExactWeightedInterpolated [¶](https://www.tinybird.co/docs/about:blank#quantileexactweightedinterpolated)

Computes [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence using linear interpolation, taking into account the weight of each element.

To get the interpolated value, all the passed values are combined into an array, which are then sorted by their corresponding weights. Quantile interpolation is then performed using the [weighted percentile method](https://en.wikipedia.org/wiki/Percentile#The_weighted_percentile_method) by building a cumulative distribution based on weights and then a linear interpolation is performed using the weights and the values to compute the quantiles.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

We strongly recommend using `quantileExactWeightedInterpolated` instead of `quantileInterpolatedWeighted` because `quantileExactWeightedInterpolated` is more accurate than `quantileInterpolatedWeighted` . Here is an example:

SELECT
    quantileExactWeightedInterpolated(0.99)(number, 1),
    quantile(0.99)(number),
    quantileInterpolatedWeighted(0.99)(number, 1)
FROM numbers(9)


┌─quantileExactWeightedInterpolated(0.99)(number, 1)─┬─quantile(0.99)(number)─┬─quantileInterpolatedWeighted(0.99)(number, 1)─┐
│                                               7.92 │                   7.92 │                                             8 │
└────────────────────────────────────────────────────┴────────────────────────┴───────────────────────────────────────────────┘
### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileExactWeightedInterpolated(level)(expr, weight) Alias: `medianExactWeightedInterpolated`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates[  median](https://en.wikipedia.org/wiki/Median)  .
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.
- `weight`   : Column with weights of sequence members. Weight is a number of value occurrences with Unsigned integer types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─n─┬─val─┐
│ 0 │   3 │
│ 1 │   2 │
│ 2 │   1 │
│ 5 │   4 │
└───┴─────┘ Result:

┌─quantileExactWeightedInterpolated(n, val)─┐
│                                       1.5 │
└───────────────────────────────────────────┘
## quantileGK [¶](https://www.tinybird.co/docs/about:blank#quantilegk)

Computes the [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence using the [Greenwald-Khanna](http://infolab.stanford.edu/~datar/courses/cs361a/papers/quantiles.pdf) algorithm. The Greenwald-Khanna algorithm is an algorithm used to compute quantiles on a stream of data in a highly efficient manner. It was introduced by Michael Greenwald and Sanjeev Khanna in 2001. It is widely used in databases and big data systems where computing accurate quantiles on a large stream of data in real-time is necessary. The algorithm is highly efficient, taking only O(log n) space and O(log log n) time per item (where n is the size of the input). It is also highly accurate, providing an approximate quantile value with high probability.

`quantileGK` is different from other quantile functions, because it enables user to control the accuracy of the approximate quantile result.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileGK(accuracy, level)(expr) Alias: `medianGK`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `accuracy`   : Accuracy of quantile. Constant positive integer. Larger accuracy value means less error. For example, if the accuracy argument is set to 100, the computed quantile will have an error no greater than 1% with high probability. There is a trade-off between the accuracy of the computed quantiles and the computational complexity of the algorithm. A larger accuracy requires more memory and computational resources to compute the quantile accurately, while a smaller accuracy argument allows for a faster and more memory-efficient computation but with a slightly lower accuracy.
- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level and accuracy.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT quantileGK(1, 0.25)(number + 1)
FROM numbers(1000)

┌─quantileGK(1, 0.25)(plus(number, 1))─┐
│                                    1 │
└──────────────────────────────────────┘

SELECT quantileGK(10, 0.25)(number + 1)
FROM numbers(1000)

┌─quantileGK(10, 0.25)(plus(number, 1))─┐
│                                   156 │
└───────────────────────────────────────┘

SELECT quantileGK(100, 0.25)(number + 1)
FROM numbers(1000)

┌─quantileGK(100, 0.25)(plus(number, 1))─┐
│                                    251 │
└────────────────────────────────────────┘

SELECT quantileGK(1000, 0.25)(number + 1)
FROM numbers(1000)

┌─quantileGK(1000, 0.25)(plus(number, 1))─┐
│                                     249 │
└─────────────────────────────────────────┘
## quantileInterpolatedWeighted [¶](https://www.tinybird.co/docs/about:blank#quantileinterpolatedweighted)

Computes [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence using linear interpolation, taking into account the weight of each element.

To get the interpolated value, all the passed values are combined into an array, which are then sorted by their corresponding weights. Quantile interpolation is then performed using the [weighted percentile method](https://en.wikipedia.org/wiki/Percentile#The_weighted_percentile_method) by building a cumulative distribution based on weights and then a linear interpolation is performed using the weights and the values to compute the quantiles.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileInterpolatedWeighted(level)(expr, weight) Alias: `medianInterpolatedWeighted`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.
- `weight`   : Column with weights of sequence members. Weight is a number of value occurrences.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─n─┬─val─┐
│ 0 │   3 │
│ 1 │   2 │
│ 2 │   1 │
│ 5 │   4 │
└───┴─────┘ Query:

SELECT quantileInterpolatedWeighted(n, val) FROM t Result:

┌─quantileInterpolatedWeighted(n, val)─┐
│                                    1 │
└──────────────────────────────────────┘
## quantiles [¶](https://www.tinybird.co/docs/about:blank#quantiles)

Syntax: `quantiles(level1, level2, ...)(x)`

All the quantile functions also have corresponding quantiles functions: `quantiles`, `quantilesDeterministic`, `quantilesTiming`, `quantilesTimingWeighted`, `quantilesExact`, `quantilesExactWeighted`, `quantileExactWeightedInterpolated`, `quantileInterpolatedWeighted`, `quantilesTDigest`, `quantilesBFloat16`, `quantilesDD` . These functions calculate all the quantiles of the listed levels in one pass, and return an array of the resulting values.

## quantilesExactExclusive [¶](https://www.tinybird.co/docs/about:blank#quantilesexactexclusive)

Exactly computes the [quantiles](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes `O(n)` memory, where `n` is a number of values that were passed. However, for a small number of values, the function is very effective.

This function is equivalent to [PERCENTILE.EXC](https://support.microsoft.com/en-us/office/percentile-exc-function-bbaa7204-e9e1-4010-85bf-c31dc5dce4ba) Excel function, ( [type R6](https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample) ).

Works more efficiently with sets of levels than `quantileExactExclusive`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantilesExactExclusive(level1, level2, ...)(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `level`   : Levels of quantiles. Possible values: (0, 1): bounds not included. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of quantiles of the specified levels.

Type of array values:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

## quantilesExactInclusive [¶](https://www.tinybird.co/docs/about:blank#quantilesexactinclusive)

Exactly computes the [quantiles](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence.

To get exact value, all the passed values ​​are combined into an array, which is then partially sorted. Therefore, the function consumes `O(n)` memory, where `n` is a number of values that were passed. However, for a small number of values, the function is very effective.

This function is equivalent to [PERCENTILE.INC](https://support.microsoft.com/en-us/office/percentile-inc-function-680f9539-45eb-410b-9a5e-c1355e5fe2ed) Excel function, ( [type R7](https://en.wikipedia.org/wiki/Quantile#Estimating_quantiles_from_a_sample) ).

Works more efficiently with sets of levels than `quantileExactInclusive`.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantilesExactInclusive(level1, level2, ...)(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `level`   : Levels of quantiles. Possible values: 0, 1: bounds included. Float.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of quantiles of the specified levels.

Type of array values:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

## quantilesGK [¶](https://www.tinybird.co/docs/about:blank#quantilesgk)

`quantilesGK` works similarly with `quantileGK` but allows us to calculate quantities at different levels simultaneously and returns an array.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantilesGK(accuracy, level1, level2, ...)(expr)
### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Array of quantiles of the specified levels.

Type of array values:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantilesGK(1, 0.25, 0.5, 0.75)(number + 1)
FROM numbers(1000)

┌─quantilesGK(1, 0.25, 0.5, 0.75)(plus(number, 1))─┐
│ [1,1,1]                                          │
└──────────────────────────────────────────────────┘

SELECT quantilesGK(10, 0.25, 0.5, 0.75)(number + 1)
FROM numbers(1000)

┌─quantilesGK(10, 0.25, 0.5, 0.75)(plus(number, 1))─┐
│ [156,413,659]                                     │
└───────────────────────────────────────────────────┘


SELECT quantilesGK(100, 0.25, 0.5, 0.75)(number + 1)
FROM numbers(1000)

┌─quantilesGK(100, 0.25, 0.5, 0.75)(plus(number, 1))─┐
│ [251,498,741]                                      │
└────────────────────────────────────────────────────┘

SELECT quantilesGK(1000, 0.25, 0.5, 0.75)(number + 1)
FROM numbers(1000)

┌─quantilesGK(1000, 0.25, 0.5, 0.75)(plus(number, 1))─┐
│ [249,499,749]                                       │
└─────────────────────────────────────────────────────┘
## quantileTDigest [¶](https://www.tinybird.co/docs/about:blank#quantiletdigest)

Computes an approximate [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence using the [t-digest](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf) algorithm.

Memory consumption is `log(n)` , where `n` is a number of values. The result depends on the order of running the query, and is nondeterministic.

The performance of the function is lower than performance of `quantile` or `quantileTiming` . In terms of the ratio of State size to precision, this function is much better than `quantile`.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileTDigest(level)(expr) Alias: `medianTDigest`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates[  median](https://en.wikipedia.org/wiki/Median)  .
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantileTDigest(number) FROM numbers(10) Result:

┌─quantileTDigest(number)─┐
│                     4.5 │
└─────────────────────────┘
## quantileTDigestWeighted [¶](https://www.tinybird.co/docs/about:blank#quantiletdigestweighted)

Computes an approximate [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence using the [t-digest](https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf) algorithm. The function takes into account the weight of each sequence member. The maximum error is 1%. Memory consumption is `log(n)` , where `n` is a number of values.

The performance of the function is lower than performance of `quantile` or `quantileTiming` . In terms of the ratio of State size to precision, this function is much better than `quantile`.

The result depends on the order of running the query, and is nondeterministic.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileTDigestWeighted(level)(expr, weight) Alias: `medianTDigestWeighted`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates median.
- `expr`   : Expression over the column values resulting in numeric data types, Date or DateTime.
- `weight`   : Column with weights of sequence elements. Weight is a number of value occurrences.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Approximate quantile of the specified level.

Type:

- Float64 for numeric data type input.
- Date if input values have the Date type.
- DateTime if input values have the DateTime type.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT quantileTDigestWeighted(number, 1) FROM numbers(10) Result:

┌─quantileTDigestWeighted(number, 1)─┐
│                                4.5 │
└────────────────────────────────────┘
## quantileTiming [¶](https://www.tinybird.co/docs/about:blank#quantiletiming)

With the determined precision computes the [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence.

The result is deterministic (it doesn't depend on the query processing order). The function is optimized for working with sequences which describe distributions like loading web pages times or backend response times.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileTiming(level)(expr) Alias: `medianTiming`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates[  median](https://en.wikipedia.org/wiki/Median)  .
- `expr`   : Expression over a column values returning a Float* type number.  

  - If negative values are passed to the function, the behavior is undefined.
  - If the value is greater than 30,000 (a page loading time of more than 30 seconds), it's assumed to be 30,000.

### Accuracy [¶](https://www.tinybird.co/docs/about:blank#accuracy)

The calculation is accurate if:

- Total number of values doesn't exceed 5670.
- Total number of values exceeds 5670, but the page loading time is less than 1024ms.

Otherwise, the result of the calculation is rounded to the nearest multiple of 16 ms.

For calculating page loading time quantiles, this function is more effective and accurate than `quantile`.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type: `Float32`.

If no values are passed to the function (when using `quantileTimingIf` ), NaN is returned. The purpose of this is to differentiate these cases from cases that result in zero.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─response_time─┐
│            72 │
│           112 │
│           126 │
│           145 │
│           104 │
│           242 │
│           313 │
│           168 │
│           108 │
└───────────────┘ Query:

SELECT quantileTiming(response_time) FROM t Result:

┌─quantileTiming(response_time)─┐
│                           126 │
└───────────────────────────────┘
## quantileTimingWeighted [¶](https://www.tinybird.co/docs/about:blank#quantiletimingweighted)

With the determined precision computes the [quantile](https://en.wikipedia.org/wiki/Quantile) of a numeric data sequence according to the weight of each sequence member.

The result is deterministic (it doesn't depend on the query processing order). The function is optimized for working with sequences which describe distributions like loading web pages times or backend response times.

When using multiple `quantile*` functions with different levels in a query, the internal states aren't combined (that is, the query works less efficiently than it could). In this case, use the `quantiles` function.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

quantileTimingWeighted(level)(expr, weight) Alias: `medianTimingWeighted`.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `level`   : Level of quantile. Optional parameter. Constant floating-point number from 0 to 1. We recommend using a `level`   value in the range of `[0.01, 0.99]`   . Default value: 0.5. At `level=0.5`   the function calculates[  median](https://en.wikipedia.org/wiki/Median)  .
- `expr`   : Expression over a column values returning a Float* type number.  

  - If negative values are passed to the function, the behavior is undefined.   - If the value is greater than 30,000 (a page loading time of more than 30 seconds), it's assumed to be 30,000.
- `weight`   : Column with weights of sequence elements. Weight is a number of value occurrences.

### Accuracy [¶](https://www.tinybird.co/docs/about:blank#accuracy)

The calculation is accurate if:

- Total number of values doesn't exceed 5670.
- Total number of values exceeds 5670, but the page loading time is less than 1024ms.

Otherwise, the result of the calculation is rounded to the nearest multiple of 16 ms.

For calculating page loading time quantiles, this function is more effective and accurate than `quantile`.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Quantile of the specified level.

Type: `Float32`.

If no values are passed to the function (when using `quantileTimingIf` ), NaN is returned. The purpose of this is to differentiate these cases from cases that result in zero.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─response_time─┬─weight─┐
│            68 │      1 │
│           104 │      2 │
│           112 │      3 │
│           126 │      2 │
│           138 │      1 │
│           162 │      1 │
└───────────────┴────────┘ Query:

SELECT quantileTimingWeighted(response_time, weight) FROM t Result:

┌─quantileTimingWeighted(response_time, weight)─┐
│                                           112 │
└───────────────────────────────────────────────┘
## quantilesTimingWeighted [¶](https://www.tinybird.co/docs/about:blank#quantilestimingweighted)

Same as `quantileTimingWeighted` , but accept multiple parameters with quantile levels and return an Array filled with many values of that quantiles.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─response_time─┬─weight─┐
│            68 │      1 │
│           104 │      2 │
│           112 │      3 │
│           126 │      2 │
│           138 │      1 │
│           162 │      1 │
└───────────────┴────────┘ Query:

SELECT quantilesTimingWeighted(0,5, 0.99)(response_time, weight) FROM t Result:

┌─quantilesTimingWeighted(0.5, 0.99)(response_time, weight)─┐
│ [112,162]                                                 │
└───────────────────────────────────────────────────────────┘
## rankCorr [¶](https://www.tinybird.co/docs/about:blank#rankcorr)

Computes a rank correlation coefficient.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

rankCorr(x, y)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Arbitrary value. Float32 or Float64.
- `y`   : Arbitrary value. Float32 or Float64.

### Returned value(s) [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- Returns a rank correlation coefficient of the ranks of x and y. The value of the correlation coefficient ranges from -1 to +1. If less than two arguments are passed, the function will return an exception. The value close to +1 denotes a high linear relationship, and with an increase of one random variable, the second random variable also increases. The value close to -1 denotes a high linear relationship, and with an increase of one random variable, the second random variable decreases. The value close or equal to 0 denotes no relationship between the two random variables.

Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT rankCorr(number, number) FROM numbers(100) Result:

┌─rankCorr(number, number)─┐
│                        1 │
└──────────────────────────┘ Query:

SELECT roundBankers(rankCorr(exp(number), sin(number)), 3) FROM numbers(100) Result:

┌─roundBankers(rankCorr(exp(number), sin(number)), 3)─┐
│                                              -0.037 │
└─────────────────────────────────────────────────────┘
## simpleLinearRegression [¶](https://www.tinybird.co/docs/about:blank#simplelinearregression)

Performs simple (unidimensional) linear regression.

simpleLinearRegression(x, y) Parameters:

- `x`   : Column with explanatory variable values.
- `y`   : Column with dependent variable values.

Returned values:

Constants `(k, b)` of the resulting line `y = k*x + b`.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

SELECT arrayReduce('simpleLinearRegression', [0, 1, 2, 3], [0, 1, 2, 3]) ┌─arrayReduce('simpleLinearRegression', [0, 1, 2, 3], [0, 1, 2, 3])─┐
│ (1,0)                                                             │
└───────────────────────────────────────────────────────────────────┘ SELECT arrayReduce('simpleLinearRegression', [0, 1, 2, 3], [3, 4, 5, 6]) ┌─arrayReduce('simpleLinearRegression', [0, 1, 2, 3], [3, 4, 5, 6])─┐
│ (1,3)                                                             │
└───────────────────────────────────────────────────────────────────┘
## singleValueOrNull [¶](https://www.tinybird.co/docs/about:blank#singlevalueornull)

The aggregate function `singleValueOrNull` is used to implement subquery operators, such as `x = ALL (SELECT ...)` . It checks if there is only one unique non-NULL value in the data. If there is only one unique value, it returns it. If there are zero or at least two distinct values, it returns NULL.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

singleValueOrNull(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Column of any data type (except Map, Array or Tuple which can't be of type Nullable).

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

- The unique value, if there is only one unique non-NULL value in `x`  .
- `NULL`   , if there are zero or at least two distinct values.

## skewPop [¶](https://www.tinybird.co/docs/about:blank#skewpop)

Computes the [skewness](https://en.wikipedia.org/wiki/Skewness) of a sequence.

skewPop(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

`expr` : Expression returning a number.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The skewness of the given distribution. Type: Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT skewPop(value) FROM series_with_value_column
## skewSamp [¶](https://www.tinybird.co/docs/about:blank#skewsamp)

Computes the [sample skewness](https://en.wikipedia.org/wiki/Skewness) of a sequence.

It represents an unbiased estimate of the skewness of a random variable if passed values form its sample.

skewSamp(expr)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `expr`   : Expression returning a number.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

The skewness of the given distribution. Type: Float64. If `n <= 1` ( `n` is the size of the sample), then the function returns `nan`.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

SELECT skewSamp(value) FROM series_with_value_column
## sparkbar [¶](https://www.tinybird.co/docs/about:blank#sparkbar)

The function plots a frequency histogram for values `x` and the repetition rate `y` of these values over the interval `[min_x, max_x]`. Repetitions for all `x` falling into the same bucket are averaged, so data should be pre-aggregated. Negative repetitions are ignored.

If no interval is specified, then the minimum `x` is used as the interval start, and the maximum `x` : as the interval end. Otherwise, values outside the interval are ignored.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sparkbar(buckets[, min_x, max_x])(x, y)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `buckets`   : The number of segments. Type: Integer.
- `min_x`   : The interval start. Optional parameter.
- `max_x`   : The interval end. Optional parameter.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : The field with values.
- `y`   : The field with the frequency of values.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The frequency histogram.

## stddevPop [¶](https://www.tinybird.co/docs/about:blank#stddevpop)

The result is equal to the square root of `varPop`.

Aliases: `STD`, `STDDEV_POP`.

This function uses a numerically unstable algorithm. If you need numerical stability in calculations, use the `stddevPopStable` function. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stddevPop(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Population of values to find the standard deviation of. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Square root of standard deviation of `x`   . Float64.

## stddevPopStable [¶](https://www.tinybird.co/docs/about:blank#stddevpopstable)

The result is equal to the square root of `varPop` . Unlike `stddevPop` , this function uses a numerically stable algorithm. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stddevPopStable(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Population of values to find the standard deviation of. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Square root of standard deviation of `x` . Float64.

## stddevSamp [¶](https://www.tinybird.co/docs/about:blank#stddevsamp)

The result is equal to the square root of `varSamp`.

Alias: `STDDEV_SAMP`.

This function uses a numerically unstable algorithm. If you need numerical stability in calculations, use the `stddevSampStable` function. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stddevSamp(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Values for which to find the square root of sample variance. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Square root of sample variance of `x` . Float64.

## stddevSampStable [¶](https://www.tinybird.co/docs/about:blank#stddevsampstable)

The result is equal to the square root of `varSamp` . Unlike `stddevSamp` , this function uses a numerically stable algorithm. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

stddevSampStable(x)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Values for which to find the square root of sample variance. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Square root of sample variance of `x` . Float64.

## stochasticLinearRegression {#agg_functions_stochasticlinearregression_parameters} [¶](https://www.tinybird.co/docs/about:blank#stochasticlinearregression-agg-functions-stochasticlinearregression-parameters)

This function implements stochastic linear regression. It supports custom parameters for learning rate, L2 regularization coefficient, mini-batch size, and has a few methods for updating weight: [Adam](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam) (used by default), [simple SGD](https://en.wikipedia.org/wiki/Stochastic_gradient_descent), [Momentum](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Momentum) , and Nesterov.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

There are 4 customizable parameters. They are passed to the function sequentially, but there is no need to pass all four - default values will be used, however good model required some parameter tuning.

stochasticLinearRegression(0.00001, 0.1, 15, 'Adam')
1. `learning rate`   is the coefficient on step length, when the gradient descent step is performed. A learning rate that is too big may cause infinite weights of the model. Default is `0.00001`  .
2. `l2 regularization coefficient`   which may help to prevent overfitting. Default is `0.1`  .
3. `mini-batch size`   sets the number of elements, which gradients will be computed and summed to perform one step of gradient descent. Pure stochastic descent uses one element, however, having small batches (about 10 elements) makes gradient steps more stable. Default is `15`  .
4. `method for updating weights`   , they are: `Adam`   (by default), `SGD`  , `Momentum`   , and `Nesterov`  . `Momentum`   and `Nesterov`   require a little bit more computations and memory, however, they happen to be useful in terms of speed of convergence and stability of stochastic gradient methods.

## stochasticLogisticRegression [¶](https://www.tinybird.co/docs/about:blank#stochasticlogisticregression)

This function implements stochastic logistic regression. It can be used for binary classification problem, supports the same custom parameters as stochasticLinearRegression and works the same way.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

Parameters are exactly the same as in stochasticLinearRegression: `learning rate`, `l2 regularization coefficient`, `mini-batch size`, `method for updating weights`. For more information see parameters.

stochasticLogisticRegression(1.0, 1.0, 10, 'SGD')
## studentTTest [¶](https://www.tinybird.co/docs/about:blank#studentttest)

Applies Student's t-test to samples from two populations.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

studentTTest([confidence_level])(sample_data, sample_index) Values of both samples are in the `sample_data` column. If `sample_index` equals to 0 then the value in that row belongs to the sample from the first population. Otherwise it belongs to the sample from the second population. The null hypothesis is that means of populations are equal. Normal distribution with equal variances is assumed.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `sample_data`   : Sample data. Integer, Float or Decimal.
- `sample_index`   : Sample index. Integer.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `confidence_level`   : Confidence level in order to calculate confidence intervals. Float.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Tuple with two or four elements (if the optional `confidence_level` is specified):

- calculated t-statistic. Float64.
- calculated p-value. Float64.
- calculated confidence-interval-low. Float64.
- calculated confidence-interval-high. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─sample_data─┬─sample_index─┐
│        20.3 │            0 │
│        21.1 │            0 │
│        21.9 │            1 │
│        21.7 │            0 │
│        19.9 │            1 │
│        21.8 │            1 │
└─────────────┴──────────────┘ Query:

SELECT studentTTest(sample_data, sample_index) FROM student_ttest Result:

┌─studentTTest(sample_data, sample_index)───┐
│ (-0.21739130434783777,0.8385421208415731) │
└───────────────────────────────────────────┘
## sum [¶](https://www.tinybird.co/docs/about:blank#sum)

Calculates the sum. Only works for numbers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sum(num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `num`   : Column of numeric values. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The sum of the values. (U)Int*, Float*, Decimal*.

## sumCount [¶](https://www.tinybird.co/docs/about:blank#sumcount)

Calculates the sum of the numbers and counts the number of rows at the same time.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sumCount(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Input value, must be Integer, Float, or Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Tuple `(sum, count)`   , where `sum`   is the sum of numbers and `count`   is the number of rows with not-NULL values.

Type: Tuple.

## sumKahan [¶](https://www.tinybird.co/docs/about:blank#sumkahan)

Calculates the sum of the numbers with [Kahan compensated summation algorithm](https://en.wikipedia.org/wiki/Kahan_summation_algorithm) Slower than `sum` function. The compensation works only for Float types.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sumKahan(x)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `x`   : Input value, must be Integer, Float, or Decimal.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- the sum of numbers, with type Integer, Float, or Decimal depends on type of input arguments

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT sum(0.1), sumKahan(0.1) FROM numbers(10) Result:

┌───────────sum(0.1)─┬─sumKahan(0.1)─┐
│ 0.9999999999999999 │             1 │
└────────────────────┴───────────────┘
## sumMap [¶](https://www.tinybird.co/docs/about:blank#summap)

Totals a `value` array according to the keys specified in the `key` array. Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys without overflow.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

- `sumMap(key , value )`   Array type.
- `sumMap(Tuple(key , value ))`   Tuple type.

Alias: `sumMappedArrays`.

**Arguments**

- `key`   : Array of keys.
- `value`   : Array of values.

Passing a tuple of key and value arrays is a synonym to passing separately an array of keys and an array of values.

The number of elements in `key` and `value` must be the same for each row that is totaled.

**Returned Value**

- Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys.

## sumMapWithOverflow [¶](https://www.tinybird.co/docs/about:blank#summapwithoverflow)

Totals a `value` array according to the keys specified in the `key` array. Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys. It differs from the `sumMap` function in that it does summation with overflow - i.e. returns the same data type for the summation as the argument data type.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

- `sumMapWithOverflow(key , value )`   Array type.
- `sumMapWithOverflow(Tuple(key , value ))`   Tuple type.

**Arguments**

- `key`   : Array of keys.
- `value`   : Array of values.

Passing a tuple of key and value arrays is a synonym to passing separately an array of keys and an array of values.

The number of elements in `key` and `value` must be the same for each row that is totaled.

**Returned Value**

- Returns a tuple of two arrays: keys in sorted order, and values ​​summed for the corresponding keys.

## sumWithOverflow [¶](https://www.tinybird.co/docs/about:blank#sumwithoverflow)

Computes the sum of the numbers, using the same data type for the result as for the input parameters. If the sum exceeds the maximum value for this data type, it's calculated with overflow.

Only works for numbers.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

sumWithOverflow(num)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `num`   : Column of numeric values. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- The sum of the values. (U)Int*, Float*, Decimal*.

## theilsU [¶](https://www.tinybird.co/docs/about:blank#theilsu)

The `theilsU` function calculates the [Theil's U uncertainty coefficient](https://en.wikipedia.org/wiki/Contingency_table#Uncertainty_coefficient) , a value that measures the association between two columns in a table. Its values range from −1.0 (100% negative association, or perfect inversion) to +1.0 (100% positive association, or perfect agreement). A value of 0.0 indicates the absence of association.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

theilsU(column1, column2)
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column1`   and `column2`   are the columns to be compared

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- a value between -1 and 1

**Return type** is always Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

The following two columns being compared below have a small association with each other, so the value of `theilsU` is negative:

SELECT
    theilsU(a ,b)
FROM
    (
        SELECT
            number % 10 AS a,
            number % 4 AS b
        FROM
            numbers(150)
    ) Result:

┌────────theilsU(a, b)─┐
│ -0.30195720557678846 │
└──────────────────────┘
## topK [¶](https://www.tinybird.co/docs/about:blank#topk)

Returns an array of the approximately most frequent values in the specified column. The resulting array is sorted in descending order of approximate frequency of values (not by the values themselves).

Implements the [Filtered Space-Saving](https://doi.org/10.1016/j.ins.2010.08.024) algorithm for analyzing TopK, based on the reduce-and-combine algorithm from [Parallel Space Saving](https://doi.org/10.1016/j.ins.2015.09.003).

topK(N)(column)
topK(N, load_factor)(column)
topK(N, load_factor, 'counts')(column) This function doesn't provide a guaranteed result. In certain situations, errors might occur and it might return frequent values that aren’t the most frequent values.

We recommend using the `N < 10` value; performance is reduced with large `N` values. Maximum value of `N = 65536`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `N`   : The number of elements to return. Optional. Default value: 10.
- `load_factor`   : Defines, how many cells reserved for values. If uniq(column) > N * load_factor, result of topK function will be approximate. Optional. Default value: 3.
- `counts`   : Defines, should result contain approximate count and error value.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : The value to calculate frequency.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Take the OnTime data set and select the three most frequently occurring values in the `AirlineID` column.

SELECT topK(3)(AirlineID) AS res
FROM ontime ┌─res─────────────────┐
│ [19393,19790,19805] │
└─────────────────────┘
## topKWeighted [¶](https://www.tinybird.co/docs/about:blank#topkweighted)

Returns an array of the approximately most frequent values in the specified column. The resulting array is sorted in descending order of approximate frequency of values (not by the values themselves). Additionally, the weight of the value is taken into account.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

topKWeighted(N)(column, weight)
topKWeighted(N, load_factor)(column, weight)
topKWeighted(N, load_factor, 'counts')(column, weight)
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `N`   : The number of elements to return. Optional. Default value: 10.
- `load_factor`   : Defines, how many cells reserved for values. If uniq(column) > N * load_factor, result of topK function will be approximate. Optional. Default value: 3.
- `counts`   : Defines, should result contain approximate count and error value.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `column`   : The value.
- `weight`   : The weight. Every value is accounted `weight`   times for frequency calculation. UInt64.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

Returns an array of the values with maximum approximate sum of weights.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT topKWeighted(2)(k, w) FROM
VALUES('k Char, w UInt64', ('y', 1), ('y', 1), ('x', 5), ('y', 1), ('z', 10)) Result:

┌─topKWeighted(2)(k, w)──┐
│ ['z','x']              │
└────────────────────────┘ Query:

SELECT topKWeighted(2, 10, 'counts')(k, w)
FROM VALUES('k Char, w UInt64', ('y', 1), ('y', 1), ('x', 5), ('y', 1), ('z', 10)) Result:

┌─topKWeighted(2, 10, 'counts')(k, w)─┐
│ [('z',10,0),('x',5,0)]              │
└─────────────────────────────────────┘
## uniq [¶](https://www.tinybird.co/docs/about:blank#uniq)

Calculates the approximate number of different values of the argument.

uniq(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of parameters. Parameters can be `Tuple`, `Array`, `Date`, `DateTime`, `String` , or numeric types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A UInt64-type number.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

Function:

- Calculates a hash for all parameters in the aggregate, then uses it in calculations.
- Uses an adaptive sampling algorithm. For the calculation state, the function uses a sample of element hash values up to 65536. This algorithm is very accurate and very efficient on the CPU. When the query contains several of these functions, using `uniq`   is almost as fast as using other aggregate functions.
- Provides the result deterministically (it doesn't depend on the query processing order).

We recommend using this function in almost all scenarios.

## uniqCombined [¶](https://www.tinybird.co/docs/about:blank#uniqcombined)

Calculates the approximate number of different argument values.

uniqCombined(HLL_precision)(x[, ...]) The `uniqCombined` function is a good choice for calculating the number of different values.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `HLL_precision`   : The base-2 logarithm of the number of cells in[  HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog)   . Optional, you can use the function as `uniqCombined(x[, ...])`   . The default value for `HLL_precision`   is 17, which is effectively 96 KiB of space (2^17 cells, 6 bits each).
- `X`   : A variable number of parameters. Parameters can be `Tuple`  , `Array`  , `Date`  , `DateTime`  , `String`   , or numeric types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A number UInt64-type number.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

The `uniqCombined` function:

- Calculates a hash (64-bit hash for `String`   and 32-bit otherwise) for all parameters in the aggregate, then uses it in calculations.
- Uses a combination of three algorithms: array, hash table, and HyperLogLog with an error correction table.
  - For a small number of distinct elements, an array is used.
  - When the set size is larger, a hash table is used.
  - For a larger number of elements, HyperLogLog is used, which will occupy a fixed amount of memory.
- Provides the result deterministically (it doesn't depend on the query processing order).

Since it uses a 32-bit hash for non- `String` types, the result will have very high error for cardinalities significantly larger than `UINT_MAX` (error will raise quickly after a few tens of billions of distinct values), hence in this case you should use uniqCombined64.

Compared to the uniq function, the uniqCombined function:

- Consumes several times less memory.
- Calculates with several times higher accuracy.
- Usually has slightly lower performance. In some scenarios, uniqCombined can perform better than uniq, for example, with distributed queries that transmit a large number of aggregation states over the network.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Query:

SELECT uniqCombined(number) FROM numbers(1e6) Result:

┌─uniqCombined(number)─┐
│              1001148 │ -- 1.00 million
└──────────────────────┘ See the example section of uniqCombined64 for an example of the difference between uniqCombined and uniqCombined64 for much larger inputs.

## uniqCombined64 [¶](https://www.tinybird.co/docs/about:blank#uniqcombined64)

Calculates the approximate number of different argument values. It is the same as uniqCombined, but uses a 64-bit hash for all data types rather than just for the String data type.

uniqCombined64(HLL_precision)(x[, ...])
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `HLL_precision`   : The base-2 logarithm of the number of cells in HyperLogLog. Optionally, you can use the function as `uniqCombined64(x[, ...])`   . The default value for `HLL_precision`   is 17, which is effectively 96 KiB of space (2^17 cells, 6 bits each).
- `X`   : A variable number of parameters. Parameters can be `Tuple`  , `Array`  , `Date`  , `DateTime`  , `String`   , or numeric types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A number UInt64-type number.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

The `uniqCombined64` function:

- Calculates a hash (64-bit hash for all data types) for all parameters in the aggregate, then uses it in calculations.
- Uses a combination of three algorithms: array, hash table, and HyperLogLog with an error correction table.
  - For a small number of distinct elements, an array is used.
  - When the set size is larger, a hash table is used.
  - For a larger number of elements, HyperLogLog is used, which will occupy a fixed amount of memory.
- Provides the result deterministically (it doesn't depend on the query processing order).

Since it uses 64-bit hash for all types, the result doesn't suffer from very high error for cardinalities significantly larger than `UINT_MAX` like uniqCombined does, which uses a 32-bit hash for non- `String` types.

Compared to the uniq function, the uniqCombined64 function:

- Consumes several times less memory.
- Calculates with several times higher accuracy.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

In the example below `uniqCombined64` is run on `1e10` different numbers returning a very close approximation of the number of different argument values.

Query:

SELECT uniqCombined64(number) FROM numbers(1e10) Result:

┌─uniqCombined64(number)─┐
│             9998568925 │ -- 10.00 billion
└────────────────────────┘ By comparison the `uniqCombined` function returns a rather poor approximation for an input this size.

Query:

SELECT uniqCombined(number) FROM numbers(1e10) Result:

┌─uniqCombined(number)─┐
│           5545308725 │ -- 5.55 billion
└──────────────────────┘
## uniqExact [¶](https://www.tinybird.co/docs/about:blank#uniqexact)

Calculates the exact number of different argument values.

uniqExact(x[, ...]) Use the `uniqExact` function if you absolutely need an exact result. Otherwise use the uniq function.

The `uniqExact` function uses more memory than `uniq` , because the size of the state has unbounded growth as the number of different values increases.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of parameters. Parameters can be `Tuple`, `Array`, `Date`, `DateTime`, `String` , or numeric types.

## uniqHLL12 [¶](https://www.tinybird.co/docs/about:blank#uniqhll12)

Calculates the approximate number of different argument values, using the [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) algorithm.

uniqHLL12(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of parameters. Parameters can be `Tuple`, `Array`, `Date`, `DateTime`, `String` , or numeric types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A UInt64-type number.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

Function:

- Calculates a hash for all parameters in the aggregate, then uses it in calculations.
- Uses the HyperLogLog algorithm to approximate the number of different argument values.  

  2^12 5-bit cells are used. The size of the state is slightly more than 2.5 KB. The result isn't very accurate (up to ~10% error) for small data sets (<10K elements). However, the result is fairly accurate for high-cardinality data sets (10K-100M), with a maximum error of ~1.6%. Starting from 100M, the estimation error increases, and the function will return very inaccurate results for data sets with extremely high cardinality (1B+ elements).
- Provides the determinate result (it doesn't depend on the query processing order).

We don't recommend using this function. In most cases, use the uniq or uniqCombined function.

## uniqTheta [¶](https://www.tinybird.co/docs/about:blank#uniqtheta)

Calculates the approximate number of different argument values, using the Theta Sketch Framework.

uniqTheta(x[, ...])
### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

The function takes a variable number of parameters. Parameters can be `Tuple`, `Array`, `Date`, `DateTime`, `String` , or numeric types.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- A UInt64-type number.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

Function:

- Calculates a hash for all parameters in the aggregate, then uses it in calculations.
- Uses the[  KMV](https://datasketches.apache.org/docs/Theta/InverseEstimate.html)   algorithm to approximate the number of different argument values.  

  4096(2^12) 64-bit sketch are used. The size of the state is about 41 KB.
- The relative error is 3.125% (95% confidence), see the[  relative error table](https://datasketches.apache.org/docs/Theta/ThetaErrorTable.html)   for detail.

## varPop [¶](https://www.tinybird.co/docs/about:blank#varpop)

Calculates the population variance.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

varPop(x) Alias: `VAR_POP`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Population of values to find the population variance of. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the population variance of `x`   . Float64.

## varPopStable [¶](https://www.tinybird.co/docs/about:blank#varpopstable)

Returns the population variance. Unlike varPop, this function uses a numerically stable algorithm. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

varPopStable(x) Alias: `VAR_POP_STABLE`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : Population of values to find the population variance of. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the population variance of `x`   . Float64.

## varSamp [¶](https://www.tinybird.co/docs/about:blank#varsamp)

Calculate the sample variance of a data set.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

varSamp(x) Alias: `VAR_SAMP`.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : The population for which you want to calculate the sample variance. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the sample variance of the input data set `x`   . Float64.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

The `varSamp` function calculates the sample variance using the following formula:

$$ \sum\frac{(x - \text{mean}(x))^2}{(n - 1)} $$

Where:

- `x`   is each individual data point in the data set.
- `mean(x)`   is the arithmetic mean of the data set.
- `n`   is the number of data points in the data set.

The function assumes that the input data set represents a sample from a larger population. If you want to calculate the variance of the entire population (when you have the complete data set), you should use `varPop` instead.

## varSampStable [¶](https://www.tinybird.co/docs/about:blank#varsampstable)

Calculate the sample variance of a data set. Unlike varSamp, this function uses a numerically stable algorithm. It works slower but provides a lower computational error.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

varSampStable(x) Alias: `VAR_SAMP_STABLE`

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `x`   : The population for which you want to calculate the sample variance. (U)Int*, Float*, Decimal*.

### Returned value [¶](https://www.tinybird.co/docs/about:blank#returned-value)

- Returns the sample variance of the input data set. Float64.

### Implementation details [¶](https://www.tinybird.co/docs/about:blank#implementation-details)

The `varSampStable` function calculates the sample variance using the same formula as the `varSamp`:

$$ \sum\frac{(x - \text{mean}(x))^2}{(n - 1)} $$

Where:

- `x`   is each individual data point in the data set.
- `mean(x)`   is the arithmetic mean of the data set.
- `n`   is the number of data points in the data set.

## welchTTest [¶](https://www.tinybird.co/docs/about:blank#welchttest)

Applies Welch's t-test to samples from two populations.

### Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

welchTTest([confidence_level])(sample_data, sample_index) Values of both samples are in the `sample_data` column. If `sample_index` equals to 0 then the value in that row belongs to the sample from the first population. Otherwise it belongs to the sample from the second population. The null hypothesis is that means of populations are equal. Normal distribution is assumed. Populations may have unequal variance.

### Arguments [¶](https://www.tinybird.co/docs/about:blank#arguments)

- `sample_data`   : Sample data. Integer, Float or Decimal.
- `sample_index`   : Sample index. Integer.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `confidence_level`   : Confidence level in order to calculate confidence intervals. Float.

### Returned values [¶](https://www.tinybird.co/docs/about:blank#returned-values)

Tuple with two or four elements (if the optional `confidence_level` is specified)

- calculated t-statistic. Float64.
- calculated p-value. Float64.
- calculated confidence-interval-low. Float64.
- calculated confidence-interval-high. Float64.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Input table:

┌─sample_data─┬─sample_index─┐
│        20.3 │            0 │
│        22.1 │            0 │
│        21.9 │            0 │
│        18.9 │            1 │
│        20.3 │            1 │
│          19 │            1 │
└─────────────┴──────────────┘ Query:

SELECT welchTTest(sample_data, sample_index) FROM welch_ttest Result:

┌─welchTTest(sample_data, sample_index)─────┐
│ (2.7988719532211235,0.051807360348581945) │
└───────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/uuid
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "UUID data type · Tinybird Docs"
theme-color: "#171612"
description: "Universally Unique Identifier (UUID)."
---




# UUID [¶](https://www.tinybird.co/docs/about:blank#uuid)

A Universally Unique Identifier (UUID) is a 16-byte value used to identify records. For detailed information about UUIDs, see Wikipedia.

While different UUID variants exist, Tinybird doesn't validate that inserted UUIDs conform to a particular variant. UUIDs are internally treated as a sequence of 16 random bytes with 8-4-4-4-12 representation at SQL level.

Example UUID value:

61f0c404-5cb3-11e7-907b-a6006ad3dba0 The default UUID is all-zero. It is used, for example, when a new record is inserted but no value for a UUID column is specified:

00000000-0000-0000-0000-000000000000 Due to historical reasons, UUIDs are sorted by their second half. UUIDs should therefore not be used directly in a primary key, sorting key, or partition key of a table.

Example:

SELECT * FROM tab ORDER BY uuid Result:

┌─uuid─────────────────────────────────┐
│ 36a0b67c-b74a-4640-803b-e44bb4547e3c │
│ 3a00aeb8-2605-4eec-8215-08c0ecb51112 │
│ 3fda7c49-282e-421a-85ab-c5684ef1d350 │
│ 16ab55a7-45f6-44a8-873c-7a0b44346b3e │
│ e3776711-6359-4f22-878d-bf290d052c85 │
│                [...]                 │
│ 9eceda2f-6946-40e3-b725-16f2709ca41a │
│ 03644f74-47ba-4020-b865-be5fd4c8c7ff │
│ ce3bc93d-ab19-4c74-b8cc-737cb9212099 │
│ b7ad6c91-23d6-4b5e-b8e4-a52297490b56 │
│ 06892f64-cc2d-45f3-bf86-f5c5af5768a9 │
└──────────────────────────────────────┘ As a workaround, the UUID can be converted to a type with an intuitive sort order.

Example using conversion to UInt128:

SELECT * FROM tab ORDER BY toUInt128(uuid) Result:

┌─uuid─────────────────────────────────┐
│ 018b81cd-aca1-4e9c-9e56-a84a074dc1a8 │
│ 02380033-c96a-438e-913f-a2c67e341def │
│ 057cf435-7044-456a-893b-9183a4475cea │
│ 0a3c1d4c-f57d-44cc-8567-60cb0c46f76e │
│ 0c15bf1c-8633-4414-a084-7017eead9e41 │
│                [...]                 │
│ f808cf05-ea57-4e81-8add-29a195bde63d │
│ f859fb5d-764b-4a33-81e6-9e4239dae083 │
│ fb1b7e37-ab7b-421a-910b-80e60e2bf9eb │
│ fc3174ff-517b-49b5-bfe2-9b369a5c506d │
│ fece9bf6-3832-449a-b058-cd1d70a02c8b │
└──────────────────────────────────────┘
## Generating UUIDs [¶](https://www.tinybird.co/docs/about:blank#generating-uuids)

The generateUUIDv4 function generates random UUID version 4 values.

## Usage examples [¶](https://www.tinybird.co/docs/about:blank#usage-examples)

**Example 1**

This example demonstrates the creation of a table with a UUID column and the insertion of a value into the table.

SELECT * FROM t_uuid Result:

┌────────────────────────────────────x─┬─y─────────┐
│ 417ddc5d-e556-4d27-95dd-a34d84e46a50 │ Example 1 │
└──────────────────────────────────────┴───────────┘ **Example 2**

In this example, no UUID column value is specified when the record is inserted, i.e. the default UUID value is inserted:

SELECT * FROM t_uuid ┌────────────────────────────────────x─┬─y─────────┐
│ 417ddc5d-e556-4d27-95dd-a34d84e46a50 │ Example 1 │
│ 00000000-0000-0000-0000-000000000000 │ Example 2 │
└──────────────────────────────────────┴───────────┘
## Restrictions [¶](https://www.tinybird.co/docs/about:blank#restrictions)

The UUID data type only supports functions which String data type also supports (for example, min, max, and count).

The UUID data type isn't supported by arithmetic operations (for example, abs) or aggregate functions, such as sum and avg.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/tuple
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Tuple data type · Tinybird Docs"
theme-color: "#171612"
description: "A tuple of elements with individual types."
---




# Tuple(T1, T2, ...) [¶](https://www.tinybird.co/docs/about:blank#tuplet1-t2)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

A tuple of elements, each having an individual type. Tuple must contain at least one element.

Tuples are used for temporary column grouping. Columns can be grouped when an IN expression is used in a query, and for specifying certain formal parameters of lambda functions. For more information, see the sections IN operators and Higher order functions.

Tuples can be the result of a query. In this case, for text formats other than JSON, values are comma-separated in brackets. In JSON formats, tuples are output as arrays (in square brackets).

## Creating tuples [¶](https://www.tinybird.co/docs/about:blank#creating-tuples)

You can use a function to create a tuple:

tuple(T1, T2, ...) Example of creating a tuple:

SELECT tuple(1, 'a') AS x, toTypeName(x) ┌─x───────┬─toTypeName(tuple(1, 'a'))─┐
│ (1,'a') │ Tuple(UInt8, String)      │
└─────────┴───────────────────────────┘ A Tuple can contain a single element

Example:

SELECT tuple('a') AS x ┌─x─────┐
│ ('a') │
└───────┘ Syntax `(tuple_element1, tuple_element2)` may be used to create a tuple of several elements without calling the `tuple()` function.

Example:

SELECT (1, 'a') AS x, (today(), rand(), 'someString') AS y, ('a') AS not_a_tuple ┌─x───────┬─y──────────────────────────────────────┬─not_a_tuple─┐
│ (1,'a') │ ('2022-09-21',2006973416,'someString') │ a           │
└─────────┴────────────────────────────────────────┴─────────────┘
## Data type detection [¶](https://www.tinybird.co/docs/about:blank#data-type-detection)

When creating tuples on the fly, Tinybird interferes the type of the tuples arguments as the smallest types which can hold the provided argument value. If the value is NULL, the interfered type is Nullable.

Example of automatic data type detection:

SELECT tuple(1, NULL) AS x, toTypeName(x) ┌─x─────────┬─toTypeName(tuple(1, NULL))──────┐
│ (1, NULL) │ Tuple(UInt8, Nullable(Nothing)) │
└───────────┴─────────────────────────────────┘
## Referring to tuple elements [¶](https://www.tinybird.co/docs/about:blank#referring-to-tuple-elements)

Tuple elements can be referred to by name or by index:

SELECT a.s FROM named_tuples; -- by name
SELECT a.2 FROM named_tuples; -- by index Result:

┌─a.s─┐
│ y   │
│ x   │
└─────┘

┌─tupleElement(a, 2)─┐
│                 10 │
│                -10 │
└────────────────────┘
## Comparison operations with tuple [¶](https://www.tinybird.co/docs/about:blank#comparison-operations-with-tuple)

Two tuples are compared by sequentially comparing their elements from the left to the right. If first tuples element is greater (smaller) than the second tuples corresponding element, then the first tuple is greater (smaller) than the second, otherwise (both elements are equal), the next element is compared.

Example:

SELECT (1, 'z') > (1, 'a') c1, (2022, 01, 02) > (2023, 04, 02) c2, (1,2,3) = (3,2,1) c3 ┌─c1─┬─c2─┬─c3─┐
│  1 │  0 │  0 │
└────┴────┴────┘ Real world examples:

SELECT *
FROM values((2022, 12, 31), (2000, 1, 1))

SELECT * FROM test

┌─year─┬─month─┬─day─┐
│ 2022 │    12 │  31 │
│ 2000 │     1 │   1 │
└──────┴───────┴─────┘

SELECT *
FROM test
WHERE (year, month, day) > (2010, 1, 1)

┌─year─┬─month─┬─day─┐
│ 2022 │    12 │  31 │
└──────┴───────┴─────┘


SELECT *
FROM values((1, 42, 66.5), (1, 42, 70), (2, 1, 10), (2, 2, 0))

SELECT * FROM test

┌─key─┬─duration─┬─value─┐
│   1 │       42 │  66.5 │
│   1 │       42 │    70 │
│   2 │        1 │    10 │
│   2 │        2 │     0 │
└─────┴──────────┴───────┘

-- Let's find a value for each key with the biggest duration, if durations are equal, select the biggest value

SELECT
    key,
    max(duration),
    argMax(value, (duration, value))
FROM test
GROUP BY key
ORDER BY key ASC

┌─key─┬─max(duration)─┬─argMax(value, tuple(duration, value))─┐
│   1 │            42 │                                    70 │
│   2 │             2 │                                     0 │
└─────┴───────────────┴───────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/string
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "String data type · Tinybird Docs"
theme-color: "#171612"
description: "Strings of arbitrary length."
---




# String [¶](https://www.tinybird.co/docs/about:blank#string)

Strings of an arbitrary length. Length isn't limited. The value can contain an arbitrary set of bytes, including null bytes. The String type replaces the types VARCHAR, BLOB, CLOB, and others from other DBMSs.

When creating tables, numeric parameters for string fields can be set (for example, `VARCHAR(255)` ).

Aliases:

- `String`  : `LONGTEXT`  , `MEDIUMTEXT`  , `TINYTEXT`  , `TEXT`  , `LONGBLOB`  , `MEDIUMBLOB`  , `TINYBLOB`  , `BLOB`  , `VARCHAR`  , `CHAR`  , `CHAR LARGE OBJECT`  , `CHAR VARYING`  , `CHARACTER LARGE OBJECT`  , `CHARACTER VARYING`  , `NCHAR LARGE OBJECT`  , `NCHAR VARYING`  , `NATIONAL CHARACTER LARGE OBJECT`  , `NATIONAL CHARACTER VARYING`  , `NATIONAL CHAR VARYING`  , `NATIONAL CHARACTER`  , `NATIONAL CHAR`  , `BINARY LARGE OBJECT`  , `BINARY VARYING`  ,

## Encodings [¶](https://www.tinybird.co/docs/about:blank#encodings)

Tinybird doesn't have the concept of encodings. Strings can contain an arbitrary set of bytes, which are stored and output as-is. If you need to store texts, use UTF-8 encoding. At the very least, if your terminal uses UTF-8, you can read and write your values without making conversions.

Certain functions for working with strings have separate variations that work under the assumption that the string contains a set of bytes representing a UTF-8 encoded text. For example, the length function calculates the string length in bytes, while the lengthUTF8 function calculates the string length in Unicode code points, assuming that the value is UTF-8 encoded.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/simpleaggregatefunction
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "SimpleAggregateFunction data type · Tinybird Docs"
theme-color: "#171612"
description: "Stores current value of aggregate functions."
---




# SimpleAggregateFunction [¶](https://www.tinybird.co/docs/about:blank#simpleaggregatefunction)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

`SimpleAggregateFunction(name, types_of_arguments...)` data type stores current value (intermediate state) of the aggregate function, but not its full state as `AggregateFunction` does.

This optimization can be applied to functions for which the following property holds: the result of applying a function `f` to a row set `S1 UNION ALL S2` can be obtained by applying `f` to parts of the row set separately, and then again applying `f` to the results: `f(S1 UNION ALL S2) = f(f(S1) UNION ALL f(S2))` . This property guarantees that partial aggregation results are enough to compute the combined one, so you don't have to store and process additional data.

The common way to produce an aggregate function value is by calling the aggregate function with the -SimpleState suffix.

The following aggregate functions are supported:

- `any`
- `anyLast`
- `min`
- `max`
- `sum`
- `sumWithOverflow`
- `groupBitAnd`
- `groupBitOr`
- `groupBitXor`
- `groupArrayArray`
- `groupUniqArrayArray`
- `sumMap`
- `minMap`
- `maxMap`

Values of the `SimpleAggregateFunction(func, Type)` look and stored the same way as `Type` , so you don't need to apply functions with `-Merge` / `-State` suffixes.

`SimpleAggregateFunction` has better performance than `AggregateFunction` with same aggregation function.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- Name of the aggregate function.
- Types of the aggregate function arguments.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/nullable
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Nullable data type · Tinybird Docs"
theme-color: "#171612"
description: "Allows storage of NULL values alongside normal values."
---




# Nullable(T) [¶](https://www.tinybird.co/docs/about:blank#nullablet)

Allows to store special marker (NULL) that denotes “missing value” alongside normal values allowed by `T` . For example, a `Nullable(Int8)` type column can store `Int8` type values, and the rows that don't have a value will store `NULL`.

`T` can’t be any of the composite data types Array, Map and Tuple but composite data types can contain `Nullable` type values, e.g. `Array(Nullable(Int8))`.

A `Nullable` type field can’t be included in table indexes.

`NULL` is the default value for any `Nullable` type.

## Finding NULL [¶](https://www.tinybird.co/docs/about:blank#finding-null)

You can find `NULL` values in a column by using `null` subcolumn without reading the whole column. It returns `1` if the corresponding value is `NULL` and `0` otherwise.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/nothing
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Nothing data type · Tinybird Docs"
theme-color: "#171612"
description: "Represents cases where a value isn't expected."
---




# Nothing [¶](https://www.tinybird.co/docs/about:blank#nothing)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

The only purpose of this data type is to represent cases where a value isn't expected. So you can’t create a `Nothing` type value.

For example, literal NULL has type of `Nullable(Nothing)`.

The `Nothing` type can also used to denote empty arrays:

SELECT toTypeName(array()) ┌─toTypeName(array())─┐
│ Array(Nothing)      │
└─────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/map
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "Map data type · Tinybird Docs"
theme-color: "#171612"
description: "Stores key-value pairs."
---




# Map(K, V) [¶](https://www.tinybird.co/docs/about:blank#mapk-v)

Data type `Map(K, V)` stores key-value pairs.

Unlike other databases, maps aren't unique in Tinybird: a map can contain two elements with the same key. The reason for that is that maps are internally implemented as `Array(Tuple(K, V))`.

You can use the syntax `m[k]` to obtain the value for key `k` in map `m` . Also, `m[k]` scans the map, for example the runtime of the operation is linear in the size of the map.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `K`   is the type of the Map keys. Arbitrary type except Nullable and LowCardinality nested with Nullable types.
- `V`   is the type of the Map values. Arbitrary type.

## Convert tuple to map [¶](https://www.tinybird.co/docs/about:blank#convert-tuple-to-map)

Values of type `Tuple()` can be cast to values of type `Map()` using function CAST:

**Example**

Query:

SELECT CAST(([1, 2, 3], ['Ready', 'Steady', 'Go']), 'Map(UInt8, String)') AS map Result:

┌─map───────────────────────────┐
│ {1:'Ready',2:'Steady',3:'Go'} │
└───────────────────────────────┘
## Reading subcolumns of map [¶](https://www.tinybird.co/docs/about:blank#reading-subcolumns-of-map)

To avoid reading the entire map, you can use subcolumns `keys` and `values` in some cases.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/lowcardinality
Last update: 2025-01-10T11:07:31.000Z
Content:
---
title: "LowCardinality data type · Tinybird Docs"
theme-color: "#171612"
description: "Dictionary-encoded representation of data types."
---




# LowCardinality(T) [¶](https://www.tinybird.co/docs/about:blank#lowcardinalityt)

Changes the internal representation of other data types to be dictionary-encoded.

## Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

LowCardinality(data_type)
##### lc_s.datasource

SCHEMA >
    `s` LowCardinality(String)

ENGINE "MergeTree"
### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- `data_type`   : String, FixedString, Date, DateTime, and numbers excepting Decimal. `LowCardinality`   isn't efficient for some data types, see the allow_suspicious_low_cardinality_types setting description.

## Description [¶](https://www.tinybird.co/docs/about:blank#description)

`LowCardinality` is a superstructure that changes a data storage method and rules of data processing. Tinybird applies dictionary coding to `LowCardinality` -columns. Operating with dictionary encoded data significantly increases performance of SELECT queries for many applications.

The efficiency of using `LowCardinality` data type depends on data diversity. If a dictionary contains less than 10,000 distinct values, then Tinybird mostly shows higher efficiency of data reading and storing. If a dictionary contains more than 100,000 distinct values, then Tinybird can perform worse in comparison with using ordinary data types.

Consider using `LowCardinality` instead of Enum when working with strings. `LowCardinality` provides more flexibility in use and often reveals the same or higher efficiency.

## Related functions [¶](https://www.tinybird.co/docs/about:blank#related-functions)

- [  toLowCardinality](https://www.tinybird.co/docs/docs/sql-reference/functions/type-conversion-functions#tolowcardinality)
- [  lowCardinalityKeys](https://www.tinybird.co/docs/docs/sql-reference/functions/other-functions#lowcardinalitykeys)
- [  lowCardinalityIndices](https://www.tinybird.co/docs/docs/sql-reference/functions/other-functions#lowcardinalityindices)



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/json
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "JSON data type · Tinybird Docs"
theme-color: "#171612"
description: "Stores JSON documents in a single column."
---




# JSON data type BETA

[¶](https://www.tinybird.co/docs/about:blank#json-data-type)

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

Stores JavaScript Object Notation (JSON) documents in a single column.

To declare a column of `JSON` type, use the following syntax:

 JSON([max_dynamic_paths=N, max_dynamic_types=M, some.path TypeName, SKIP path.to.skip, SKIP REGEXP 'paths_regexp']) [jsonPath] Where:

- `some.path TypeName`   is an optional type hint for particular path in the JSON. Such paths will be always stored as subcolumns with specified type.
- `SKIP path.to.skip`   is an optional hint for particular path that should be skipped during JSON parsing. Such paths will never be stored in the JSON column. If specified path is a nested JSON object, the whole nested object will be skipped.
- `SKIP REGEXP 'path_regexp'`   is an optional hint with a regular expression that is used to skip paths during JSON parsing. All paths that match this regular expression will never be stored in the JSON column.
- `max_dynamic_paths`   is an optional parameter indicating how many paths can be stored separately as subcolumns across single block of data that is stored separately (for example across single data part for MergeTree table). If this limit is exceeded, all other paths will be stored together in a single structure. Default value of `max_dynamic_paths`   is `1024`  .
- `max_dynamic_types`   is an optional parameter between `1`   and `255`   indicating how many different data types can be stored inside a single path column with type `Dynamic`   across single block of data that is stored separately (for example across single data part for MergeTree table). If this limit is exceeded, all new types will be converted to type `String`   . Default value of `max_dynamic_types`   is `32`  .

## Creating a Data Source with JSON columns [¶](https://www.tinybird.co/docs/about:blank#creating-a-data-source-with-json-columns)

You can create a data source with JSON columns by using the `JSON` data type.

##### test.datasource

SCHEMA >
    `json` JSON `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" tb push datasources/test.datasource
echo '{"a" : {"b" : 42}, "c" : [1, 2, 3]}\n{"f" : "Hello, World!"}\n{"a" : {"b" : 43, "e" : 10}, "c" : [4, 5, 6]}' > test.ndjson
tb datasource append test test.ndjson
tb sql 'select * from test'
---------------------------------------------
| json                                      |
---------------------------------------------
| {'a': {'b': 42}, 'c': [1, 2, 3]}          |
| {'f': 'Hello, World!'}                    |
| {'a': {'b': 43, 'e': 10}, 'c': [4, 5, 6]} |
--------------------------------------------- Specifying the paths will force the data source to have them:

##### test.datasource

SCHEMA >
    `json` JSON(a.b UInt32, SKIP a.e) `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" tb push datasources/test.datasource
echo '{"a" : {"b" : 42}, "c" : [1, 2, 3]}\n{"f" : "Hello, World!"}\n{"a" : {"b" : 43, "e" : 10}, "c" : [4, 5, 6]}' > test.ndjson
tb datasource append test test.ndjson
tb sql 'select * from test'
-----------------------------------------
| json                                  |
-----------------------------------------
| {'a': {'b': 42}, 'c': [1, 2, 3]}      |
| {'a': {'b': 0}, 'f': 'Hello, World!'} |
| {'a': {'b': 43}, 'c': [4, 5, 6]}      |
----------------------------------------- For more details, check [Ingesting NDJSON data](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#json-data-type).

## Creating JSON in a Query [¶](https://www.tinybird.co/docs/about:blank#creating-json-in-a-query)

Using CAST from `String`:

SELECT '{"a" : {"b" : 42},"c" : [1, 2, 3], "d" : "Hello, World!"}'::JSON AS json ┌─json───────────────────────────────────────────┐
│ {"a":{"b":42},"c":[1,2,3],"d":"Hello, World!"} │
└────────────────────────────────────────────────┘ To CAST from `Tuple` or `Map` to `JSON` is you need to CAST the column into `String` column containing JSON objects and deserializing it back to `JSON` type column.

## Reading JSON paths as subcolumns [¶](https://www.tinybird.co/docs/about:blank#reading-json-paths-as-subcolumns)

JSON type supports reading every path as a separate subcolumn. If type of the requested path was not specified in the JSON type declaration, the subcolumn of the path will always have type Dynamic.

##### test.datasource

SCHEMA >
    `json` JSON(a.b UInt32, SKIP a.e) `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" tb push datasources/test.datasource
echo '{"a" : {"b" : 42, "g" : 42.42}, "c" : [1, 2, 3], "d" : "2020-01-01"}\n{"f" : "Hello, World!", "d" : "2020-01-02"}\n{"a" : {"b" : 43, "e" : 10, "g" : 43.43}, "c" : [4, 5, 6]}' > test.ndjson
tb datasource append test test.ndjson For example:

SELECT json.a.b, json.a.g, json.c, json.d FROM test ┌─json.a.b─┬─json.a.g─┬─json.c──┬─json.d─────┐
│       42 │ 42.42    │ [1,2,3] │ 2020-01-01 │
│        0 │ ᴺᵁᴸᴸ     │ ᴺᵁᴸᴸ    │ 2020-01-02 │
│       43 │ 43.43    │ [4,5,6] │ ᴺᵁᴸᴸ       │
└──────────┴──────────┴─────────┴────────────┘ If you ever find any issue with dot notation, try `getSubcloumn(json,'path')` syntax:

See getSubcloumn() example:

SELECT getSubcolumn(json, 'a.b'), getSubcolumn(json, 'a.g'), getSubcolumn(json, 'c'), getSubcolumn(json, 'd') FROM test ┌─getSubcolumn(json, 'a.b')─┬─getSubcolumn(json, 'a.g')─┬─getSubcolumn(json, 'c')─┬─getSubcolumn(json, 'd')─────┐
│                        42 │                     42.42 │                 [1,2,3] │                  2020-01-01 │
│                         0 │                  ᴺᵁᴸᴸ     │                 ᴺᵁᴸᴸ    │                  2020-01-02 │
│                        43 │                  43.43    │                 [4,5,6] │                  ᴺᵁᴸᴸ       │
└───────────────────────────┴───────────────────────────┴─────────────────────────┴─────────────────────────────┘ If the requested path wasn't found in the data, it's filled with `NULL` values:

SELECT json.non.existing.path FROM test ┌─json.non.existing.path─┐
│ ᴺᵁᴸᴸ                   │
│ ᴺᵁᴸᴸ                   │
│ ᴺᵁᴸᴸ                   │
└────────────────────────┘ You can check the data types of returned subcolumns:

SELECT toTypeName(json.a.b), toTypeName(json.a.g), toTypeName(json.c), toTypeName(json.d) FROM test ┌─toTypeName(json.a.b)─┬─toTypeName(json.a.g)─┬─toTypeName(json.c)─┬─toTypeName(json.d)─┐
│ UInt32               │ Dynamic              │ Dynamic            │ Dynamic            │
│ UInt32               │ Dynamic              │ Dynamic            │ Dynamic            │
│ UInt32               │ Dynamic              │ Dynamic            │ Dynamic            │
└──────────────────────┴──────────────────────┴────────────────────┴────────────────────┘ For `a.b` the type is `UInt32` as specified in the JSON type declaration, and for all other subcolumns the type is `Dynamic`.

You can also read subcolumns of a `Dynamic` type using special syntax `json.some.path.:TypeName`:

select json.a.g.:Float64, dynamicType(json.a.g), json.d.:Date, dynamicType(json.d) FROM test ┌─json.a.g.:`Float64`─┬─dynamicType(json.a.g)─┬─json.d.:`Date`─┬─dynamicType(json.d)─┐
│               42.42 │ Float64               │     2020-01-01 │ Date                │
│                ᴺᵁᴸᴸ │ None                  │     2020-01-02 │ Date                │
│               43.43 │ Float64               │           ᴺᵁᴸᴸ │ None                │
└─────────────────────┴───────────────────────┴────────────────┴─────────────────────┘ `Dynamic` subcolumns can be cast to any data type. In this case the exception will be thrown if internal type inside `Dynamic` can't be cast to the requested type:

select json.a.g::UInt64 as uint FROM test ┌─uint─┐
│   42 │
│    0 │
│   43 │
└──────┘ select json.a.g::UUID as float FROM test [Error] Conversion between numeric types and UUID isn't supported. Probably the passed UUID is unquoted: while executing 'FUNCTION CAST(json.a.g :: 0, 'UUID' :: 1) -> CAST(json.a.g, 'UUID') UUID : 2'. (NOT_IMPLEMENTED)
## Reading JSON sub-objects as subcolumns [¶](https://www.tinybird.co/docs/about:blank#reading-json-sub-objects-as-subcolumns)

JSON type supports reading nested objects as subcolumns with type `JSON` using special syntax `json.^some.path`:

##### test.datasource

SCHEMA >
    `json` JSON `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" tb push datasources/test.datasource
echo '{"a" : {"b" : {"c" : 42, "g" : 42.42}}, "c" : [1, 2, 3], "d" : {"e" : {"f" : {"g" : "Hello, World", "h" : [1, 2, 3]}}}}\n{"f" : "Hello, World!", "d" : {"e" : {"f" : {"h" : [4, 5, 6]}}}}\n{"a" : {"b" : {"c" : 43, "e" : 10, "g" : 43.43}}, "c" : [4, 5, 6]}' > test.ndjson
tb datasource append test test.ndjson For example:

SELECT json.^a.b, json.^d.e.f FROM test ┌─json.^`a`.b───────────────┬─json.^`d`.e.f────────────────────┐
│ {"c":42,"g":42.42}        │ {"g":"Hello, World","h":[1,2,3]} │
│ {}                        │ {"h":[4,5,6]}                    │
│ {"c":43,"e":10,"g":43.43} │ {}                               │
└───────────────────────────┴──────────────────────────────────┘ Reading sub-objects as subcolumns may be inefficient, as this may require almost full scan of the JSON data.

## Handling arrays of JSON objects [¶](https://www.tinybird.co/docs/about:blank#handling-arrays-of-json-objects)

JSON paths that contains an array of objects are parsed as type `Array(JSON)` and inserted into `Dynamic` column for this path. To read an array of objects you can extract it from `Dynamic` column as a subcolumn:

##### test.datasource

SCHEMA >
    `json` JSON `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" tb push datasources/test.datasource
echo '{"a" : {"b" : [{"c" : 42, "d" : "Hello", "f" : [[{"g" : 42.42}]], "k" : {"j" : 1000}}, {"c" : 43}, {"e" : [1, 2, 3], "d" : "My", "f" : [[{"g" : 43.43, "h" : "2020-01-01"}]],  "k" : {"j" : 2000}}]}}\n{"a" : {"b" : [1, 2, 3]}}\n{"a" : {"b" : [{"c" : 44, "f" : [[{"h" : "2020-01-02"}]]}, {"e" : [4, 5, 6], "d" : "World", "f" : [[{"g" : 44.44}]],  "k" : {"j" : 3000}}]}}' > test.ndjson
tb datasource append test test.ndjson For example:

SELECT json.a.b, dynamicType(json.a.b) FROM test ┌─json.a.b──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─dynamicType(json.a.b)────────────────────────────────────┐
│ ['{"c":"42","d":"Hello","f":[[{"g":42.42}]],"k":{"j":"1000"}}','{"c":"43"}','{"d":"My","e":["1","2","3"],"f":[[{"g":43.43,"h":"2020-01-01"}]],"k":{"j":"2000"}}'] │ Array(JSON(max_dynamic_types=16, max_dynamic_paths=256)) │
│ [1,2,3]                                                                                                                                                           │ Array(Nullable(Int64))                                   │
│ ['{"c":"44","f":[[{"h":"2020-01-02"}]]}','{"d":"World","e":["4","5","6"],"f":[[{"g":44.44}]],"k":{"j":"3000"}}']                                                  │ Array(JSON(max_dynamic_types=16, max_dynamic_paths=256)) │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────┘ The `max_dynamic_types/max_dynamic_paths` parameters of the nested `JSON` type were reduced compared to the default values. It's needed to avoid number of subcolumns to grow uncontrolled on nested arrays of JSON objects.

When reading subcolumns from a nested `JSON` column:

SELECT json.a.b.:`Array(JSON)`.c, json.a.b.:`Array(JSON)`.f, json.a.b.:`Array(JSON)`.d FROM test ┌─json.a.b.:`Array(JSON)`.c─┬─json.a.b.:`Array(JSON)`.f───────────────────────────────────┬─json.a.b.:`Array(JSON)`.d─┐
│ [42,43,NULL]              │ [[['{"g":42.42}']],NULL,[['{"g":43.43,"h":"2020-01-01"}']]] │ ['Hello',NULL,'My']       │
│ []                        │ []                                                          │ []                        │
│ [44,NULL]                 │ [[['{"h":"2020-01-02"}']],[['{"g":44.44}']]]                │ [NULL,'World']            │
└───────────────────────────┴─────────────────────────────────────────────────────────────┴───────────────────────────┘ You can avoid writing the `Array(JSON)` subcolumn name using the following syntax:

SELECT json.a.b[].c, json.a.b[].f, json.a.b[].d FROM test ┌─json.a.b.:`Array(JSON)`.c─┬─json.a.b.:`Array(JSON)`.f───────────────────────────────────┬─json.a.b.:`Array(JSON)`.d─┐
│ [42,43,NULL]              │ [[['{"g":42.42}']],NULL,[['{"g":43.43,"h":"2020-01-01"}']]] │ ['Hello',NULL,'My']       │
│ []                        │ []                                                          │ []                        │
│ [44,NULL]                 │ [[['{"h":"2020-01-02"}']],[['{"g":44.44}']]]                │ [NULL,'World']            │
└───────────────────────────┴─────────────────────────────────────────────────────────────┴───────────────────────────┘ The number of `[]` after path indicates the array level. `json.path[][]` is transformed to `json.path.:Array(Array(JSON))`

To read subcolumns from `Array(JSON)` column:

SELECT json.a.b[].c.:Int64, json.a.b[].f[][].g.:Float64, json.a.b[].f[][].h.:Date FROM test ┌─json.a.b.:`Array(JSON)`.c.:`Int64`─┬─json.a.b.:`Array(JSON)`.f.:`Array(Array(JSON))`.g.:`Float64`─┬─json.a.b.:`Array(JSON)`.f.:`Array(Array(JSON))`.h.:`Date`─┐
│ [42,43,NULL]                       │ [[[42.42]],[],[[43.43]]]                                     │ [[[NULL]],[],[['2020-01-01']]]                            │
│ []                                 │ []                                                           │ []                                                        │
│ [44,NULL]                          │ [[[NULL]],[[44.44]]]                                         │ [[['2020-01-02']],[[NULL]]]                               │
└────────────────────────────────────┴──────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┘ You can also read sub-object subcolumns from nested `JSON` column:

SELECT json.a.b[].^k FROM test ┌─json.a.b.:`Array(JSON)`.^`k`─────────┐
│ ['{"j":"1000"}','{}','{"j":"2000"}'] │
│ []                                   │
│ ['{}','{"j":"3000"}']                │
└──────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/ipv6
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "IPv6 data type · Tinybird Docs"
theme-color: "#171612"
description: "Represents IPv6 addresses."
---




# IPv6 [¶](https://www.tinybird.co/docs/about:blank#ipv6)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

IPv6 addresses. Stored in 16 bytes as UInt128 big-endian.

WITH (SELECT [('https://wikipedia.org', '::ffff:185.15.58.224'),('https://tinybird.co','::ffff:66.33.60.67'),('https://vercel.com','::ffff:64.239.109.193')]) AS ips
SELECT arrayJoin(ips).1 as url, (arrayJoin(ips).2)::IPv6 as ipv6 -------------------------------------------------
| url                   | ipv6                  |
-------------------------------------------------
| https://wikipedia.org | ::ffff:185.15.58.224  |
| https://tinybird.co   | ::ffff:66.33.60.67    |
| https://vercel.com    | ::ffff:64.239.109.193 |
------------------------------------------------- Values are stored in compact binary form:

WITH (SELECT [('https://wikipedia.org', '::ffff:185.15.58.224'),('https://tinybird.co','::ffff:66.33.60.67'),('https://vercel.com','::ffff:64.239.109.193')]) AS ips
SELECT (arrayJoin(ips).2)::IPv6 as ipv6, toTypeName(ipv6), hex(ipv6) -------------------------------------------------------------------------------
| ipv6                  | toTypeName(ipv6) | hex(ipv6)                        |
-------------------------------------------------------------------------------
| ::ffff:185.15.58.224  | IPv6             | 00000000000000000000FFFFB90F3AE0 |
| ::ffff:66.33.60.67    | IPv6             | 00000000000000000000FFFF42213C43 |
| ::ffff:64.239.109.193 | IPv6             | 00000000000000000000FFFF40EF6DC1 |
------------------------------------------------------------------------------- IPv6 addresses can be directly compared to IPv4 addresses:

SELECT toIPv4('127.0.0.1') = toIPv6('::ffff:127.0.0.1') -----------------------------------------------------------
| equals(toIPv4('127.0.0.1'), toIPv6('::ffff:127.0.0.1')) |
-----------------------------------------------------------
|                                                       1 |
-----------------------------------------------------------

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/ipv4
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "IPv4 data type · Tinybird Docs"
theme-color: "#171612"
description: "Represents IPv4 addresses."
---




# IPv4 [¶](https://www.tinybird.co/docs/about:blank#ipv4)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

IPv4 addresses. Stored in 4 bytes as UInt32.

WITH (SELECT [('https://wikipedia.org', '185.15.58.224'),('https://tinybird.co','66.33.60.67'),('https://vercel.com','64.239.109.193')]) AS ips
SELECT arrayJoin(ips).1 as url, (arrayJoin(ips).2)::IPv4 as ipv4 ------------------------------------------
| url                   | ipv4           |
------------------------------------------
| https://wikipedia.org | 185.15.58.224  |
| https://tinybird.co   | 66.33.60.67    |
| https://vercel.com    | 64.239.109.193 |
------------------------------------------ Values are stored in compact binary form:

WITH (SELECT [('https://wikipedia.org', '185.15.58.224'),('https://tinybird.co','66.33.60.67'),('https://vercel.com','64.239.109.193')]) AS ips
SELECT (arrayJoin(ips).2)::IPv4 as ipv4, toTypeName(ipv4), hex(ipv4) -------------------------------------------------
| ipv4           | toTypeName(ipv4) | hex(ipv4) |
-------------------------------------------------
| 185.15.58.224  | IPv4             | B90F3AE0  |
| 66.33.60.67    | IPv4             | 42213C43  |
| 64.239.109.193 | IPv4             | 40EF6DC1  |
------------------------------------------------- IPv4 addresses can be directly compared to IPv6 addresses:

SELECT toIPv4('127.0.0.1') = toIPv6('::ffff:127.0.0.1') -----------------------------------------------------------
| equals(toIPv4('127.0.0.1'), toIPv6('::ffff:127.0.0.1')) |
-----------------------------------------------------------
|                                                       1 |
-----------------------------------------------------------

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/interval
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Interval data type · Tinybird Docs"
theme-color: "#171612"
description: "Represents time and date intervals."
---




# Interval [¶](https://www.tinybird.co/docs/about:blank#interval)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

The family of data types representing time and date intervals. The resulting types of the INTERVAL operator.

Structure:

- Time interval as an unsigned integer value.
- Type of an interval.

Supported interval types:

- `NANOSECOND`
- `MICROSECOND`
- `MILLISECOND`
- `SECOND`
- `MINUTE`
- `HOUR`
- `DAY`
- `WEEK`
- `MONTH`
- `QUARTER`
- `YEAR`

For each interval type, there is a separate data type. For example, the `DAY` interval corresponds to the `IntervalDay` data type:

SELECT toTypeName(INTERVAL 4 DAY) ┌─toTypeName(toIntervalDay(4))─┐
│ IntervalDay                  │
└──────────────────────────────┘
## Considerations [¶](https://www.tinybird.co/docs/about:blank#considerations)

You can use `Interval` -type values in arithmetical operations with Date and DateTime-type values. For example, you can add 4 days to the current time:

SELECT now() as current_date_time, current_date_time + INTERVAL 4 DAY ┌───current_date_time─┬─plus(now(), toIntervalDay(4))─┐
│ 2019-10-23 10:58:45 │           2019-10-27 10:58:45 │
└─────────────────────┴───────────────────────────────┘ You can use multiple intervals simultaneously:

SELECT now() AS current_date_time, current_date_time + (INTERVAL 4 DAY + INTERVAL 3 HOUR) ┌───current_date_time─┬─plus(current_date_time, plus(toIntervalDay(4), toIntervalHour(3)))─┐
│ 2024-08-08 18:31:39 │                                                2024-08-12 21:31:39 │
└─────────────────────┴────────────────────────────────────────────────────────────────────┘ And to compare values with different intervals:

SELECT toIntervalMicrosecond(3600000000) = toIntervalHour(1) ┌─less(toIntervalMicrosecond(179999999), toIntervalMinute(3))─┐
│                                                           1 │
└─────────────────────────────────────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/int-uint
Last update: 2024-12-16T08:49:56.000Z
Content:
---
title: "Integer and Unsigned Integer data types · Tinybird Docs"
theme-color: "#171612"
description: "Fixed-length integers with or without a sign."
---




# Integer [¶](https://www.tinybird.co/docs/about:blank#integer)

Include UInt8, UInt16, UInt32, UInt64, UInt128, UInt256, Int8, Int16, Int32, Int64, Int128, Int256.

Fixed-length integers, with or without a sign.

## Int ranges [¶](https://www.tinybird.co/docs/about:blank#int-ranges)

- `Int8`   : [-128 : 127]
- `Int16`   : [-32768 : 32767]
- `Int32`   : [-2147483648 : 2147483647]
- `Int64`   : [-9223372036854775808 : 9223372036854775807]
- `Int128`   : [-170141183460469231731687303715884105728 : 170141183460469231731687303715884105727]
- `Int256`   : [-57896044618658097711785492504343953926634992332820282019728792003956564819968 : 57896044618658097711785492504343953926634992332820282019728792003956564819967]

Aliases:

- `Int8`  : `TINYINT`  , `INT1`  , `BYTE`  , `TINYINT SIGNED`  , `INT1 SIGNED`  .
- `Int16`  : `SMALLINT`  , `SMALLINT SIGNED`  .
- `Int32`  : `INT`  , `INTEGER`  , `MEDIUMINT`  , `MEDIUMINT SIGNED`  , `INT SIGNED`  , `INTEGER SIGNED`  .
- `Int64`  : `BIGINT`  , `SIGNED`  , `BIGINT SIGNED`  , `TIME`  .

## UInt ranges [¶](https://www.tinybird.co/docs/about:blank#uint-ranges)

- `UInt8`   : [0 : 255]
- `UInt16`   : [0 : 65535]
- `UInt32`   : [0 : 4294967295]
- `UInt64`   : [0 : 18446744073709551615]
- `UInt128`   : [0 : 340282366920938463463374607431768211455]
- `UInt256`   : [0 : 115792089237316195423570985008687907853269984665640564039457584007913129639935]

Aliases:

- `UInt8`  : `TINYINT UNSIGNED`  , `INT1 UNSIGNED`  .
- `UInt16`  : `SMALLINT UNSIGNED`  .
- `UInt32`  : `MEDIUMINT UNSIGNED`  , `INT UNSIGNED`  , `INTEGER UNSIGNED`
- `UInt64`  : `UNSIGNED`  , `BIGINT UNSIGNED`  , `BIT`  , `SET`



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/geo
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Geographical data types · Tinybird Docs"
theme-color: "#171612"
description: "Data types for representing geographical objects."
---




# Geographical data types [¶](https://www.tinybird.co/docs/about:blank#geographical-data-types)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

Supports data types for representing geographical objects: locations, lands, etc.

## Point [¶](https://www.tinybird.co/docs/about:blank#point)

`Point` is represented by its X and Y coordinates, stored as a Tuple(Float64, Float64).

## Ring [¶](https://www.tinybird.co/docs/about:blank#ring)

`Ring` is a simple polygon without holes stored as an array of points: Array(Point).

## LineString [¶](https://www.tinybird.co/docs/about:blank#linestring)

`LineString` is a line stored as an array of points: Array(Point).

## MultiLineString [¶](https://www.tinybird.co/docs/about:blank#multilinestring)

`MultiLineString` is multiple lines stored as an array of `LineString` : Array(LineString).

## Polygon [¶](https://www.tinybird.co/docs/about:blank#polygon)

`Polygon` is a polygon with holes stored as an array of rings: Array(Ring). First element of outer array is the outer shape of polygon and all the following elements are holes.

## MultiPolygon [¶](https://www.tinybird.co/docs/about:blank#multipolygon)

`MultiPolygon` consists of multiple polygons and is stored as an array of polygons: Array(Polygon).



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/float
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Float data types · Tinybird Docs"
theme-color: "#171612"
description: "Floating point numbers for various precisions."
---




# Float [¶](https://www.tinybird.co/docs/about:blank#float)

Includes Float32, Float64, BFloat16.

Equivalent types:

- `Float32`  : `float`  .
- `Float64`  : `double`  .

Float types have the following aliases:

- `Float32`  : `FLOAT`  , `REAL`  , `SINGLE`  .
- `Float64`  : `DOUBLE`  , `DOUBLE PRECISION`  .

## Using floating-point numbers [¶](https://www.tinybird.co/docs/about:blank#using-floating-point-numbers)

- Computations with floating-point numbers might produce a rounding error.

SELECT 1 - 0.9 ┌───────minus(1, 0.9)─┐
│ 0.09999999999999998 │
└─────────────────────┘
- The result of the calculation depends on the calculation method (the processor type and architecture of the computer system).
- Floating-point calculations might result in numbers such as infinity ( `Inf`   ) and “not-a-number” ( `NaN`   ). This should be taken into account when processing the results of calculations.
- When parsing floating-point numbers from text, the result might not be the nearest machine-representable number.

## NaN and Inf [¶](https://www.tinybird.co/docs/about:blank#nan-and-inf)

In contrast to standard SQL, the following categories of floating-point numbers are supported:

- `Inf`   – Infinity.



SELECT 0.5 / 0 ┌─divide(0.5, 0)─┐
│            inf │
└────────────────┘
- `-Inf`   : Negative infinity.



SELECT -0.5 / 0 ┌─divide(-0.5, 0)─┐
│            -inf │
└─────────────────┘
- `NaN`   : Not a number.



SELECT 0 / 0 ┌─divide(0, 0)─┐
│          nan │
└──────────────┘ See the rules for `NaN` sorting in the section ORDER BY clause.

## BFloat16 [¶](https://www.tinybird.co/docs/about:blank#bfloat16)

`BFloat16` is a 16-bit floating point data type with 8-bit exponent, sign, and 7-bit mantissa.

It is useful for machine learning and AI applications.

Supports conversions between `Float32` and `BFloat16` . Most of other operations aren't supported.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/fixedstring
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "FixedString data type · Tinybird Docs"
theme-color: "#171612"
description: "Fixed-length string of N bytes."
---




# FixedString(N) [¶](https://www.tinybird.co/docs/about:blank#fixedstringn)

A fixed-length string of `N` bytes (neither characters nor code points).

To declare a column of `FixedString` type, use the following syntax:

 FixedString(N) Where `N` is a natural number.

The `FixedString` type is efficient when data has the length of precisely `N` bytes. In all other cases, it's likely to reduce efficiency.

Examples of the values that can be efficiently stored in `FixedString` -typed columns:

- The binary representation of IP addresses ( `FixedString(16)`   for IPv6).
- Language codes, like `en_US`  .
- Currency codes, like `USD`   or `EUR`  .
- Binary representation of hashes: `FixedString(16)`   for MD5, `FixedString(32)`   for SHA256.

To store UUID values, use the UUID data type.

When inserting the data, the database:

- Complements a string with null bytes if the string contains fewer than `N`   bytes.
- Throws the `Too large value for FixedString(N)`   exception if the string contains more than `N`   bytes.

When selecting the data, the database doesn't remove the null bytes at the end of the string. If you use the `WHERE` clause, you should add null bytes manually to match the `FixedString` value. The following example illustrates how to use the `WHERE` clause with `FixedString`.

Let’s consider the following table with the single `FixedString(2)` column:

┌─name──┐
│ b     │
└───────┘ The query `SELECT * FROM FixedStringTable WHERE a = 'b'` doesn't return any data as a result. We should complement the filter pattern with null bytes.

SELECT * FROM FixedStringTable
WHERE a = 'b\0' ┌─a─┐
│ b │
└───┘ This behavior differs from MySQL for the `CHAR` type, where strings are padded with spaces, and the spaces are removed for output.

The length of the `FixedString(N)` value is constant. The length function returns `N` even if the `FixedString(N)` value is filled only with null bytes, but the empty function returns `1` in this case.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/enum
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Enum data type · Tinybird Docs"
theme-color: "#171612"
description: "Enumerated type with named values."
---




# Enum [¶](https://www.tinybird.co/docs/about:blank#enum)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

Enumerated type consisting of named values.

Named values can be declared as `'string' = integer` pairs or `'string'` names . Only numbers are stored, but supports operations with the values through their names.

Supported:

- 8-bit `Enum`   . It can contain up to 256 values enumerated in the `[-128, 127]`   range.
- 16-bit `Enum`   . It can contain up to 65536 values enumerated in the `[-32768, 32767]`   range.

The type of `Enum` is chosen automatically when data is inserted. You can also use `Enum8` or `Enum16` types to be sure in the size of storage.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/decimal
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Decimal · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Decimal data type."
---




# Decimal [¶](https://www.tinybird.co/docs/about:blank#decimal)

Signed fixed-point numbers that keep precision during add, subtract and multiply operations. For division least significant digits are discarded (not rounded).

Includes Decimal, Decimal(P), Decimal(P, S), Decimal32(S), Decimal64(S), Decimal128(S), Decimal256(S).

## Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- P is precision. Valid range: [ 1 : 76 ]. Determines how many decimal digits number can have (including fraction). By default, the precision is 10.
- S is scale. Valid range: [ 0 : P ]. Determines how many decimal digits fraction can have.

Decimal(P) is equivalent to Decimal(P, 0). Similarly, the syntax Decimal is equivalent to Decimal(10, 0).

Depending on P parameter value Decimal(P, S) is a synonym for:

- P from [ 1 : 9 ] - for Decimal32(S)
- P from [ 10 : 18 ] - for Decimal64(S)
- P from [ 19 : 38 ] - for Decimal128(S)
- P from [ 39 : 76 ] - for Decimal256(S)

## Decimal Value Ranges [¶](https://www.tinybird.co/docs/about:blank#decimal-value-ranges)

- Decimal32(S) - ( -1 * 10^(9 - S), 1 * 10^(9 - S) )
- Decimal64(S) - ( -1 * 10^(18 - S), 1 * 10^(18 - S) )
- Decimal128(S) - ( -1 * 10^(38 - S), 1 * 10^(38 - S) )
- Decimal256(S) - ( -1 * 10^(76 - S), 1 * 10^(76 - S) )

For example, Decimal32(4) can contain numbers from -99999.9999 to 99999.9999 with 0.0001 step.

## Operations and result type [¶](https://www.tinybird.co/docs/about:blank#operations-and-result-type)

Binary operations on Decimal result in wider result type (with any order of arguments).

- `Decimal64(S1)  Decimal32(S2) -> Decimal64(S)`
- `Decimal128(S1)  Decimal32(S2) -> Decimal128(S)`
- `Decimal128(S1)  Decimal64(S2) -> Decimal128(S)`
- `Decimal256(S1)  Decimal<32|64|128>(S2) -> Decimal256(S)`

Rules for scale:

- add, subtract: S = max(S1, S2).
- multiply: S = S1 + S2.
- divide: S = S1.

For similar operations between Decimal and integers, the result is Decimal of the same size as an argument.

Operations between Decimal and Float32/Float64 aren't defined. If you need them, you can explicitly cast one of argument using toDecimal32, toDecimal64, toDecimal128 or toFloat32, toFloat64 builtins. Keep in mind that the result will lose precision and type conversion is a computationally expensive operation.

Some functions on Decimal return result as Float64 (for example, var or stddev). Intermediate calculations might still be performed in Decimal, which might lead to different results between Float64 and Decimal inputs with the same values.

## Overflow checks [¶](https://www.tinybird.co/docs/about:blank#overflow-checks)

During calculations on Decimal, integer overflows might happen. Excessive digits in a fraction are discarded (not rounded). Excessive digits in integer part will lead to an exception.

Overflow check isn't implemented for Decimal128 and Decimal256. In case of overflow incorrect result is returned, no exception is thrown.

SELECT toDecimal32(2, 4) AS x, x / 3 ┌──────x─┬─divide(toDecimal32(2, 4), 3)─┐
│ 2.0000 │                       0.6666 │
└────────┴──────────────────────────────┘ SELECT toDecimal32(4.2, 8) AS x, x * x DB::Exception: Scale is out of bounds. SELECT toDecimal32(4.2, 8) AS x, 6 * x DB::Exception: Decimal math overflow. Overflow checks happen not only on arithmetic operations but also on value comparison:

SELECT toDecimal32(1, 8) < 100 DB::Exception: Can't compare.

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/datetime64
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "DateTime64 · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the DateTime64 data type."
---




# DateTime64 [¶](https://www.tinybird.co/docs/about:blank#datetime64)

Allows to store an instant in time, that can be expressed as a calendar date and a time of a day, with defined sub-second precision

Tick size (precision): 10^(-precision) seconds. Valid range: [ 0 : 9 ].
Typical values: 3 (milliseconds), 6 (microseconds), 9 (nanoseconds).

## Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

DateTime64(precision, [timezone]) Internally, stores data as a number of ‘ticks’ since epoch start (1970-01-01 00:00:00 UTC) as Int64. The tick resolution is determined by the precision parameter. Additionally, the `DateTime64` type can store time zone that is the same for the entire column, that affects how the values of the `DateTime64` type values are displayed in text format and how the values specified as strings are parsed (‘2020-01-01 05:00:01.000’). The time zone isn't stored in the rows of the table (or in resultset), but is stored in the column metadata.

Supported range of values: [1900-01-01 00:00:00, 2299-12-31 23:59:59.99999999]

Note: The precision of the maximum value is 8. If the maximum precision of 9 digits (nanoseconds) is used, the maximum supported value is `2262-04-11 23:47:16` in UTC.

## Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

1. Creating a table with `DateTime64`   -type column and inserting data into it:

##### dt64.datasource

SCHEMA >
    timestamp DateTime64(3, 'Asia/Istanbul') `json:$.ts`
    event_id UInt8 `json:$.event_id`

ENGINE "MergeTree" tb push datasources/dt64.datasource

echo '{"ts":1546300800123,"event_id":1}' > dt64.ndjson
echo '{"ts":1546300800.123,"event_id":2}' >> dt64.ndjson
echo '{"ts":"2019-01-01 00:00:00","event_id":3}' >> dt64.ndjson

tb datasource append dt64 dt64.ndjson

tb sql "SELECT * FROM dt64"
--------------------------------------
| timestamp               | event_id |
--------------------------------------
| 2019-01-01 03:00:00.000 |        3 |
| 2019-01-01 03:00:00.123 |        1 |
| 2019-01-01 03:00:00.123 |        2 |
--------------------------------------
- When inserting datetime as an integer, it's treated as an appropriately scaled Unix Timestamp (UTC). `1546300800000`   (with precision 3) represents `'2019-01-01 00:00:00'`   UTC. However, as `timestamp`   column has `Asia/Istanbul`   (UTC+3) timezone specified, when outputting as a string the value will be shown as `'2019-01-01 03:00:00'`   . Inserting datetime as a decimal will treat it similarly as an integer, except the value before the decimal point is the Unix Timestamp up to and including the seconds, and after the decimal point will be treated as the precision.
- When inserting string value as datetime, it's treated as being in column timezone. `'2019-01-01 00:00:00'`   will be treated as being in `Asia/Istanbul`   timezone and stored as `1546290000000`  .

1. Filtering on `DateTime64`   values

SELECT * FROM dt64 WHERE timestamp = toDateTime64('2019-01-01 03:00:00', 3, 'Asia/Istanbul') --------------------------------------
| timestamp               | event_id |
--------------------------------------
| 2019-01-01 03:00:00.000 |        3 |
-------------------------------------- Unlike `DateTime`, `DateTime64` values aren't converted from `String` automatically.

SELECT * FROM dt64 WHERE timestamp = toDateTime64(1546300800.123, 3) --------------------------------------
| timestamp               | event_id |
--------------------------------------
| 2019-01-01 03:00:00.123 |        1 |
| 2019-01-01 03:00:00.123 |        2 |
-------------------------------------- Contrary to inserting, the `toDateTime64` function will treat all values as the decimal variant, so precision needs to be given after the decimal point.

1. Getting a time zone for a `DateTime64`   -type value:

SELECT toDateTime64(now(), 3, 'Asia/Istanbul') AS column, toTypeName(column) AS x ------------------------------------------------------------
| column                  | x                              |
------------------------------------------------------------
| 2024-12-26 18:07:25.000 | DateTime64(3, 'Asia/Istanbul') |
------------------------------------------------------------
1. Timezone conversion

SELECT
toDateTime64(timestamp, 3, 'Europe/London') as lon_time,
toDateTime64(timestamp, 3, 'Asia/Istanbul') as istanbul_time
FROM dt64 -----------------------------------------------------
| lon_time                | istanbul_time           |
-----------------------------------------------------
| 2019-01-01 00:00:00.000 | 2019-01-01 03:00:00.000 |
| 2019-01-01 00:00:00.123 | 2019-01-01 03:00:00.123 |
| 2019-01-01 00:00:00.123 | 2019-01-01 03:00:00.123 |
-----------------------------------------------------

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/datetime
Last update: 2024-12-05T17:20:25.000Z
Content:
---
title: "DateTime · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the DateTime data type."
---




# DateTime [¶](https://www.tinybird.co/docs/about:blank#datetime)

Allows to store an instant in time, that can be expressed as a calendar date and a time of a day.

Syntax:

DateTime([timezone]) Supported range of values: [1970-01-01 00:00:00, 2106-02-07 06:28:15].

Resolution: 1 second.

## Speed [¶](https://www.tinybird.co/docs/about:blank#speed)

The `Date` datatype is faster than `DateTime` under most conditions.

The `Date` type requires 2 bytes of storage, while `DateTime` requires 4. However, when the database compresses the database, this difference is amplified. This amplification is due to the minutes and seconds in `DateTime` being less compressible. Filtering and aggregating `Date` instead of `DateTime` is also faster.

## Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

### Filter on `DateTime` values [¶](https://www.tinybird.co/docs/about:blank#filter-on-values)

SELECT * FROM dt WHERE timestamp = toDateTime('2019-01-01 00:00:00', 'Asia/Istanbul') ┌───────────timestamp─┬─event_id─┐
│ 2019-01-01 00:00:00 │        1 │
└─────────────────────┴──────────┘ `DateTime` column values can be filtered using a string value in `WHERE` predicate. It will be converted to `DateTime` automatically:

SELECT * FROM dt WHERE timestamp = '2019-01-01 00:00:00' ┌───────────timestamp─┬─event_id─┐
│ 2019-01-01 00:00:00 │        1 │
└──────────────────────┴──────────┘
### Get a time zone for a `DateTime` -type column: [¶](https://www.tinybird.co/docs/about:blank#get-a-time-zone-for-a-type-column)

SELECT toDateTime(now(), 'Asia/Istanbul') AS column, toTypeName(column) AS x ┌──────────────column─┬─x─────────────────────────┐
│ 2019-10-16 04:12:04 │ DateTime('Asia/Istanbul') │
└─────────────────────┴───────────────────────────┘
### Timezone conversion [¶](https://www.tinybird.co/docs/about:blank#timezone-conversion)

SELECT
toDateTime(timestamp, 'Europe/London') as lon_time,
toDateTime(timestamp, 'Asia/Istanbul') as mos_time
FROM dt ┌───────────lon_time──┬────────────mos_time─┐
│ 2019-01-01 00:00:00 │ 2019-01-01 03:00:00 │
│ 2018-12-31 21:00:00 │ 2019-01-01 00:00:00 │
└─────────────────────┴─────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/date32
Last update: 2024-12-05T17:20:25.000Z
Content:
---
title: "Date32 · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Date32 data type."
---




# Date32 [¶](https://www.tinybird.co/docs/about:blank#date32)

A date. Supports the date range same with DateTime64. Stored as a signed 32-bit integer in native byte order with the value representing the days since 1970-01-01 (0 represents 1970-01-01 and negative values represent the days before 1970).



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/date
Last update: 2024-12-05T17:20:25.000Z
Content:
---
title: "Date · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Date data type."
---




# Date [¶](https://www.tinybird.co/docs/about:blank#date)

A date. Stored in two bytes as the number of days since 1970-01-01 (unsigned). Allows storing values from just after the beginning of the Unix Epoch to the upper threshold defined by a constant at the compilation stage (currently, this is until the year 2149, but the final fully-supported year is 2148).

Supported range of values: [1970-01-01, 2149-06-06].

The date value is stored without the time zone.



---

URL: https://www.tinybird.co/docs/sql-reference/data-types/boolean
Last update: 2024-12-26T21:50:29.000Z
Content:
---
title: "Bool · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Bool data type."
---




# Bool [¶](https://www.tinybird.co/docs/about:blank#bool)

Type `bool` is internally stored as UInt8. Possible values are `true` (1), `false` (0).

select true as col, toTypeName(col)
┌─col──┬─toTypeName(true)─┐
│ true │ Bool             │
└──────┴──────────────────┘

select true == 1 as col, toTypeName(col)
┌─col─┬─toTypeName(equals(true, 1))─┐
│   1 │ UInt8                       │
└─────┴─────────────────────────────┘

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/array
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Array · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Array data type."
---




# Array(T) [¶](https://www.tinybird.co/docs/about:blank#arrayt)

An array of `T` -type items, with the starting array index as 1. `T` can be any data type, including an array.

## Creating an array [¶](https://www.tinybird.co/docs/about:blank#creating-an-array)

You can use a function to create an array:

array(T) You can also use square brackets.

[] Example of creating an array:

SELECT array(1, 2) AS x, toTypeName(x) ┌─x─────┬─toTypeName(array(1, 2))─┐
│ [1,2] │ Array(UInt8)            │
└───────┴─────────────────────────┘ SELECT [1, 2] AS x, toTypeName(x) ┌─x─────┬─toTypeName([1, 2])─┐
│ [1,2] │ Array(UInt8)       │
└───────┴────────────────────┘
## Working with data types [¶](https://www.tinybird.co/docs/about:blank#working-with-data-types)

When creating an array on the fly, the system automatically defines the argument type as the narrowest data type that can store all the listed arguments. If there are any Nullable or literal NULL values, the type of an array element also becomes Nullable.

If the system couldn’t determine the data type, it generates an exception. For instance, this happens when trying to create an array with strings and numbers simultaneously ( `SELECT array(1, 'a')` ).

Examples of automatic data type detection:

SELECT array(1, 2, NULL) AS x, toTypeName(x) ┌─x──────────┬─toTypeName(array(1, 2, NULL))─┐
│ [1,2,NULL] │ Array(Nullable(UInt8))        │
└────────────┴───────────────────────────────┘ If you try to create an array of incompatible data types, the system throws an exception:

SELECT array(1, 'a') Received exception from server (version 1.1.54388):
Code: 386. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: There is no supertype for types UInt8, String because some of them are String/FixedString and some of them aren't.

---

URL: https://www.tinybird.co/docs/sql-reference/data-types/aggregatefunction
Last update: 2025-01-23T08:42:52.000Z
Content:
---
title: "AggregateFunction · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the AggregateFunction data type."
---




# AggregateFunction [¶](https://www.tinybird.co/docs/about:blank#aggregatefunction)

This data type isn't supported at ingest. It is only supported at query time and to create Copy Data Sources or Materialized View Data Sources.

Aggregate functions have an implementation-defined intermediate state that can be serialized to an `AggregateFunction(...)` data type and stored in a table, usually, by means of a materialized view. The common way to produce an aggregate function state is by calling the aggregate function with the `-State` suffix. To get the final result of aggregation in the future, you must use the same aggregate function with the `-Merge` suffix.

`AggregateFunction(name, types_of_arguments...)` : parametric data type.

### Parameters [¶](https://www.tinybird.co/docs/about:blank#parameters)

- Name of the aggregate function. If the function is parametric, specify its parameters too.
- Types of the aggregate function arguments.

## Data selection [¶](https://www.tinybird.co/docs/about:blank#data-selection)

When selecting data from `AggregatingMergeTree` table, use `GROUP BY` clause and the same aggregate functions as when inserting data, but using `-Merge` suffix.

An aggregate function with `-Merge` suffix takes a set of states, combines them, and returns the result of complete data aggregation.

For example, the following two queries return the same result:

SELECT uniq(UserID) FROM table

SELECT uniqMerge(state) FROM (SELECT uniqState(UserID) AS state FROM table GROUP BY RegionID)

---

URL: https://www.tinybird.co/docs/publish/sinks/s3-sink
Last update: 2025-02-13T16:23:58.000Z
Content:
---
title: "S3 Sink · Tinybird Docs"
theme-color: "#171612"
description: "Offload data to S3 on a batch-based schedule using Tinybird's fully managed S3 Sink Connector."
---




# S3 Sink [¶](https://www.tinybird.co/docs/about:blank#s3-sink)

Tinybird's S3 Sink allows you to offload data to Amazon S3, either on a pre-defined schedule or on demand. It's good for a variety of different scenarios where Amazon S3 is the common ground, for example:

- You're building a platform on top of Tinybird, and need to send data extracts to your clients on a regular basis.
- You want to export new records to Amazon S3 every day, so you can load them into Snowflake to run ML recommendation jobs.
- You need to share the data you have in Tinybird with other systems in your organization, in bulk.

Tinybird represents Sinks using the icon.

The Tinybird S3 Sink feature is available for Developer and Enterprise plans. See [Plans](https://www.tinybird.co/docs/docs/get-started/plans).

## About Tinybird's S3 Sink [¶](https://www.tinybird.co/docs/about:blank#about-tinybirds-s3-sink)

### How it works [¶](https://www.tinybird.co/docs/about:blank#how-it-works)

Tinybird's S3 Sink is fully managed and requires no additional tooling. You create a new connection to an Amazon S3 bucket, then choose a Pipe whose result gets written to Amazon S3. Tinybird provides you with complete observability and control over the executions, resulting files, their size, data transfer, and more.

### Sink Pipes [¶](https://www.tinybird.co/docs/about:blank#sink-pipes)

The Sink connector is built on Tinybird's Sink Pipes, an extension of the [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) concept, similar to [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) . Sink Pipes allow you to capture the result of a Pipe at a moment in time, and store the output. Currently, Amazon S3 is the only service Tinybird's Sink Pipes support. Sink Pipes can be run on a schedule, or executed on demand.

### Supported regions [¶](https://www.tinybird.co/docs/about:blank#supported-regions)

The Tinybird S3 Sink feature only supports exporting data to the following AWS regions:

- `us-east-*`
- `us-west-*`
- `eu-central-*`
- `eu-west-*`
- `eu-south-*`
- `eu-north-*`

### Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To use the Tinybird S3 Sink feature, you should be familiar with Amazon S3 buckets and have the necessary permissions to set up a new policy and role in AWS.

### Static or dynamic [¶](https://www.tinybird.co/docs/about:blank#static-or-dynamic)

When configuring an S3 sink, you can specify a file template for the exported files. This template can be static or dynamic. If you use a static filename in the file template, it means you're not including any dynamic elements like timestamps or column values.

The behavior when using a static filename depends on the write strategy you choose. There are two options:

1. `new`   (default): If you use this strategy with a static filename, each execution of the sink adds a new file with a suffix to avoid overwriting existing files.
2. `truncate`   : With this strategy, if you use a static filename, each execution replaces the existing file with the same name.

You can set the write strategy through CLI configuration. You can also override these settings for individual executions of the sink pipe. This means you can potentially change the file template or write strategy for a specific run.

While static filenames are possible, use some form of dynamic naming or partitioning to avoid potential conflicts or data loss, especially when dealing with recurring exports.

### Scheduling considerations [¶](https://www.tinybird.co/docs/about:blank#scheduling-considerations)

The schedule applied to a [Sink Pipe](https://www.tinybird.co/docs/about:blank#sink-pipes) doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.

To reduce the chances of a busy queue affecting your Sink Pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.

For Enterprise customers, these settings can be customized. Reach out to your Customer Success team or email us at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co).

### Query parameters [¶](https://www.tinybird.co/docs/about:blank#query-parameters)

You can add [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) to your Sink Pipes, the same way you do in API Endpoints or Copy Pipes.

- For on-demand executions, you can set parameters when you trigger the Sink Pipe to whatever values you wish.
- For scheduled executions, the default values for the parameters will be used when the Sink Pipe runs.

## Set up [¶](https://www.tinybird.co/docs/about:blank#set-up)

The setup process involves configuring both Tinybird and AWS:

1. Create your Pipe and promote it to Sink Pipe
2. Create the AWS S3 connection
3. Choose the scheduling options
4. Configure destination path and file names
5. Preview and trigger your new Sink Pipe

### Using the UI [¶](https://www.tinybird.co/docs/about:blank#using-the-ui)

#### 1. Create a Pipe and promote it to Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#1-create-a-pipe-and-promote-it-to-sink-pipe)

In the Tinybird UI, create a Pipe and write the query that produces the result you want to export. In the top right "Create API Endpoint" menu, select "Create Sink". In the modal, choose the destination (Amazon S3), and enter the bucket name and region.

Follow the step-by-step process on the modal to guide you through the AWS setup steps, or use the docs below.

#### 2. Create the AWS S3 Connection [¶](https://www.tinybird.co/docs/about:blank#2-create-the-aws-s3-connection)

##### 2.1. Create the S3 access policy

First, create an IAM policy that grants the IAM role permissions to write to S3. Open the AWS console and navigate to IAM > Policies, then select “Create Policy”:



<-figure->
![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fs3-sink-step-1-access-policy.png&w=3840&q=75)


On the “Specify Permissions” screen, select “JSON” and paste the policy generated in the UI by clicking on the Copy icon. It'll look like something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3::://*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::"
        }
    ]
} Select “Next”, add a memorable name in the following dialog box (you'll need it later!), and select “Create Policy”.

##### 2.2. Create the IAM role

In the AWS console, navigate to IAM > Roles and select “Create Role”:



<-figure->
![image](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fs3-sink-step-3-roles.png&w=3840&q=75)


On the “Select trusted entity” dialog box, select the "Custom trust policy” option. Copy the generated JSON and paste in into the Tinybird UI modal. Select "Next".

On the “Add permissions” screen, find the policy for S3 access you just created and tick the checkbox to the left of it. Select "Next" and give it a meaningful name and description. Confirm that the trusted entities and permissions granted are the expected ones, and select "Create Role".

You'll need the role's ARN (Amazon Resource Name) in order to create the connection in the next step. To save you having to come back and look for it, go to IAM > Roles and browse the search box for the role you just created. Select it to open more role details, including the role's ARN. Copy it down somewhere you can find it easily again. It'll look like something like `arn:aws:iam::111111111111:role/my-awesome-role`.

Return to Tinybird's UI and enter the role ARN and Connection name in the modal. The Connection to AWS S3 is now created in Tinybird, and can be reused in multiple Sinks.

#### 3. Choose the scheduling options [¶](https://www.tinybird.co/docs/about:blank#3-choose-the-scheduling-options)

You can configure your Sink to run "on demand" 9meaning you'll need to manually trigger it) or using a cron expression, so it runs automatically when needed.

#### 4. Configure destination path and file names [¶](https://www.tinybird.co/docs/about:blank#4-configure-destination-path-and-file-names)

Enter the bucket URI where files are generated and the file name template. When generating multiple files, the Sink creates them using this template. You have multiple ways to configure this, see the [File template](https://www.tinybird.co/docs/about:blank#file-template) section.

#### 5. Preview and create [¶](https://www.tinybird.co/docs/about:blank#5-preview-and-create)

The final step is to check and confirm that the preview matches what you expect.

Congratulations! You've created your first Sink.

Trigger it manually using the "Run Sink now" option in the top right menu, or wait for the next scheduled execution.

When triggering a Sink Pipe you have the option of overriding several of its settings, like format or compression. Refer to the [Sink Pipes API spec](https://www.tinybird.co/docs/docs/api-reference/sink-pipes-api) for the full list of parameters.

Once the Sink Pipe is triggered, it creates a standard Tinybird job that can be followed via the `v0/jobs` API.

### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-cli)

#### 1. Create the AWS S3 Connection [¶](https://www.tinybird.co/docs/about:blank#1-create-the-aws-s3-connection)

To create a connection for an S3 Sink Pipe you need to use a CLI version equal to or higher than 3.5.0.

To start:

1. Run the `tb connection create s3_iamrole`   command.
2. Copy the suggested policy and replace the two bucket placeholders with your bucket name.
3. Log into your AWS Console.
4. Create a new policy in AWS IAM > Policies using the copied text.

In the next step, you'll need the role's ARN (Amazon Resource Name) to create the connection. Go to IAM > Roles and browse the search box for the role you just created. Select it to open more role details, including the role's ARN. Copy it and paste it into the CLI when requested. It'll look like something like `arn:aws:iam::111111111111:role/my-awesome-role`.

Then, you will need to type the region where the bucket is located and choose a name to identify your connection within Tinybird. Once you have completed all these inputs, Tinybird will check access to the bucket and create the connection with the connection name you selected.

#### 2. Create S3 Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#2-create-s3-sink-pipe)

To create a Sink Pipe, create a regular .pipe and filter the data you want to export to your bucket in the SQL section as in any other Pipe. Then, specify the Pipe as a sink type and add the needed configuration. Your Pipe should have the following structure:

NODE node_0

SQL >
    SELECT *
    FROM events
    WHERE time >= toStartOfMinute(now()) - interval 30 minute)

TYPE sink
EXPORT_SERVICE s3_iamrole
EXPORT_CONNECTION_NAME "test_s3"
EXPORT_BUCKET_URI "s3://tinybird-sinks"
EXPORT_FILE_TEMPLATE "daily_prices" # Supports partitioning
EXPORT_SCHEDULE "*/5 * * * *" # Optional
EXPORT_FORMAT "csv"
EXPORT_COMPRESSION "gz" # Optional
EXPORT_WRITE_STRATEGY "truncate" **Sink Pipe parameters**

See the [Sink Pipe parameter docs](https://www.tinybird.co/docs/docs/cli/datafiles/pipe-files#sink-pipe) for more information. For this step, your details will be:

| Key | Type | Description |
| --- | --- | --- |
| EXPORT_CONNECTION_NAME | string | Required. The connection name to the destination service. This the connection created in Step 1. |
| EXPORT_BUCKET_URI | string | Required. The path to the destination bucket. Example: `s3://tinybird-export` |
| EXPORT_FILE_TEMPLATE | string | Required. The target file name. Can use parameters to dynamically name and partition the files. See File partitioning section below. Example: `daily_prices_{customer_id}` |
| EXPORT_FORMAT | string | Optional. The output format of the file. Values: CSV, NDJSON, Parquet. Default value: CSV |
| EXPORT_COMPRESSION | string | Optional. Accepted values: `none`  , `gz`   for gzip, `br`   for brotli, `xz`   for LZMA, `zst`   for zstd. Default: `none` |
| EXPORT_SCHEDULE | string | A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |
| EXPORT_WRITE_STRATEGY | string | Option. The write mode to define if files with the same name will be replaced (truncate) or not (new). Accepted values: `new`  , `truncate`   . Default: `new` |

When ready, push the datafile to your Workspace using `tb push` or `tb deploy` to create the Sink Pipe.

## File template [¶](https://www.tinybird.co/docs/about:blank#file-template)

The export process allows you to partition the result in different files, allowing you to organize your data and get smaller files. The partitioning is defined in the file template and based on the values of columns of the result set.

### Partition by column [¶](https://www.tinybird.co/docs/about:blank#partition-by-column)

Add a template variable like `{COLUMN_NAME}` to the filename. For instance, consider the following query schema and result for an export:

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| GLOBEX | INV-ABC-789 | 35.34 |
| OSCORP | INVOICE2023-06-08 | 57 |
| ACME | INV-XYZ-98765 | 23.16 |
| OSCORP | INV210608-001 | 62.23 |
| GLOBEX | 987INV654 | 36.23 |

With the given file template `invoice_summary_{customer_id}.csv` you'd get 3 files:

`invoice_summary_ACME.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| ACME | INV20230608 | 23.45 |
| ACME | 12345INV | 12.3 |
| ACME | INV-XYZ-98765 | 23.16 |

`invoice_summary_OSCORP.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| OSCORP | INVOICE2023-06-08 | 57 |
| OSCORP | INV210608-001 | 62.23 |

`invoice_summary_GLOBEX.csv`

| customer_id | invoice_id | amount |
| --- | --- | --- |
| GLOBEX | INV-ABC-789 | 35.34 |
| GLOBEX | 987INV654 | 36.23 |

### Values format [¶](https://www.tinybird.co/docs/about:blank#values-format)

In the case of DateTime columns, it can be dangerous to partition just by the column. Why? Because you could end up with as many files as seconds, as they're the different values for a DateTime column. In an hour, that's potentially 3600 files.

To help partition in a sensible way, you can add a format string to the column name using the following placeholders:

| Placeholder | Description | Example |
| --- | --- | --- |
| %Y | Year | 2023 |
| %m | Month as an integer number (01-12) | 06 |
| %d | Day of the month, zero-padded (01-31) | 07 |
| %H | Hour in 24h format (00-23) | 14 |
| %i | Minute (00-59) | 45 |

For instance, for a result like this:

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |

Note that all 7 events have different times in the column timestamp. Using a file template like `invoices_{timestamp}` would create 7 different files.

If you were interested in writing one file per hour, you could use a file template like `invoices_{timestamp, '%Y%m%d-%H'}` . You'd then get only two files for that dataset:

`invoices_20230707-09.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-07 09:07:05 | INV20230608 | 23.45 |
| 2023-07-07 09:07:01 | 12345INV | 12.3 |
| 2023-07-07 09:06:45 | INV-ABC-789 | 35.34 |
| 2023-07-07 09:05:35 | INVOICE2023-06-08 | 57 |

`invoices_20230706-23.csv`

| timestamp | invoice_id | amount |
| --- | --- | --- |
| 2023-07-06 23:14:05 | INV-XYZ-98765 | 23.16 |
| 2023-07-06 23:14:02 | INV210608-001 | 62.23 |
| 2023-07-06 23:10:55 | 987INV654 | 36.23 |

### By number of files [¶](https://www.tinybird.co/docs/about:blank#by-number-of-files)

You also have the option to write the result into X files. Instead of using a column name, use an integer between brackets.

Example: `invoice_summary.{8}.csv`

This is convenient to reduce the file size of the result, especially when the files are meant to be consumed by other services, like Snowflake where uploading big files is discouraged.

The results are written in random order. This means that the final result rows would be written in X files, but you can't count the specific order of the result.

There are a maximum of 16 files.

### Combining different partitions [¶](https://www.tinybird.co/docs/about:blank#combining-different-partitions)

It's possible to add more than one partitioning parameter in the file template. This is useful, for instance, when you do a daily dump of data, but want to export one file per hour.

Setting the file template as `invoices/dt={timestamp, '%Y-%m-%d'}/H{timestamp, '%H}.csv` would create the following file structure in different days and executions:

Invoices
├── dt=2023-07-07
│   └── H23.csv
│   └── H22.csv
│   └── H21.csv
│   └── ...
├── dt=2023-07-06
│   └── H23.csv
│   └── H22.csv You can also mix column names and number of files. For instance, setting the file template as `invoices/{customer_id}/dump_{4}.csv` would create the following file structure in different days and executions:

Invoices
├── ACME
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv
├── OSCORP
│   └── dump_0.csv
│   └── dump_1.csv
│   └── dump_2.csv
│   └── dump_3.csv Be careful with excessive partitioning. Take into consideration that the write process will create as many files as combinations of the values of the partitioning columns for a given result set.

## Iterating a Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#iterating-a-sink-pipe)

Sink Pipes can be iterated using [version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work) just like any other resource in your Data Project. However, you need to understand how connections work in Branches and deployments in order to select the appropriate strategy for your desired changes.

**Branches don't execute on creation by default recurrent jobs** , like the scheduled ones for Sink Pipes (they continue executing in Production as usual).

To iterate a Sink Pipe, create a new one (or recreate the existing one) with the desired configuration. The new Sink Pipe will start executing from the Branch too (without affecting the unchanged production resource). It will use the new configuration and export the new files into a **branch-specific folder** `my_bucket//branch_/` (the `` is optional). This means you can test the changes without mixing the test resource with your production exports.

When you deploy that Branch, the specific folder in the path is automatically ignored and production continues to point to `my_bucket/prefix/` or the new path you changed.

Take into account that, for now, while you can change the Sink Pipe configuration using version control, new connections to S3 must be created directly in the Workspace.

There is an example about how to create a Pipe Sink to S3 with version control [here](https://github.com/tinybirdco/use-case-examples/tree/main/create_pipe_sink).

## Observability [¶](https://www.tinybird.co/docs/about:blank#observability)

Sink Pipes operations are logged in the [tinybird.sinks_ops_log](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-sinks-ops-log) Service Data Source.

Data Transfer incurred by Sink Pipes is tracked in [tinybird.data_transfer](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-data-transfer) Service Data Source.

## Limits & quotas [¶](https://www.tinybird.co/docs/about:blank#limits-quotas)



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing [¶](https://www.tinybird.co/docs/about:blank#billing)

Tinybird uses two metrics for billing Sink Pipes: Processed Data and Data Transfer. A Sink Pipe executes the Pipe's query (Processed Data), and writes the result into a Bucket (Data Transfer). If the resulting files are compressed, Tinybird accounts for the compressed size.

### Processed Data [¶](https://www.tinybird.co/docs/about:blank#processed-data)

Any Processed Data incurred by a Sink Pipe is charged at the standard rate for your account. The Processed Data is already included in your plan, and counts towards your commitment. If you're on an Enterprise plan, view your plan and commitment on the [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) tab in the UI.

### Data Transfer [¶](https://www.tinybird.co/docs/about:blank#data-transfer)

Data Transfer depends on your environment. There are two scenarios:

- The destination bucket is in the**  same**   cloud provider and region as your Tinybird Workspace: $0.01 / GB
- The destination bucket is in a**  different**   cloud provider or region as your Tinybird Workspace: $0.10 / GB

### Enterprise customers [¶](https://www.tinybird.co/docs/about:blank#enterprise-customers)

Tinybird includes 50 GB for free for every Enterprise customer, so you can test the feature and validate your use case. After that, Tinybird can set up a meeting to understand your use case and adjust your contract accordingly, to accommodate the necessary Data Transfer.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Get familiar with the[  Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources)   and see what's going on in your account
- Deep dive on Tinybird's[  Pipes concept](https://www.tinybird.co/docs/docs/work-with-data/query/pipes)



---

URL: https://www.tinybird.co/docs/publish/sinks/kafka-sink
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Kafka Sink · Tinybird Docs"
theme-color: "#171612"
description: "Push events to Kafka on a batch-based schedule using Tinybird's fully managed Kafka Sink Connector."
---




# Kafka Sink [¶](https://www.tinybird.co/docs/about:blank#kafka-sink)

Kafka Sinks are currently in private beta. If you have any feedback or suggestions, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

Tinybird's Kafka Sink allows you to push the results of a query to a Kafka topic. Queries can be executed on a defined schedule or on-demand.

Common uses for the Kafka Sink include:

- Push events to Kafka as part of an event-driven architecture.
- Exporting data to other systems that consume data from Kafka.
- Hydrating a data lake or data warehouse with real-time data.

Tinybird represents Sinks using the icon.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To use the Kafka Sink, you need to have a Kafka cluster that Tinybird can reach via the internet, or via private networking for Enterprise customers.

## Configure using the UI [¶](https://www.tinybird.co/docs/about:blank#configure-using-the-ui)

### 1. Create a Pipe and promote it to Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#1-create-a-pipe-and-promote-it-to-sink-pipe)

In the Tinybird UI, create a Pipe and write the query that produces the result you want to export. In the top right "Create API Endpoint" menu, select "Create Sink". In the modal, choose the destination (Kafka).

### 2. Choose the scheduling options [¶](https://www.tinybird.co/docs/about:blank#2-choose-the-scheduling-options)

You can configure your Sink to run using a cron expression, so it runs automatically when needed.

### 3. Configure destination topic [¶](https://www.tinybird.co/docs/about:blank#3-configure-destination-topic)

Enter the Kafka topic where events are going to be pushed.

### 4. Preview and create [¶](https://www.tinybird.co/docs/about:blank#4-preview-and-create)

The final step is to check and confirm that the preview matches what you expect.

Congratulations! You've created your first Sink.

## Configure using the CLI [¶](https://www.tinybird.co/docs/about:blank#configure-using-the-cli)

### 1. Create the Kafka Connection [¶](https://www.tinybird.co/docs/about:blank#1-create-the-kafka-connection)

Run the `tb connection create kafka` command, and follow the instructions.

### 2. Create Kafka Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#2-create-kafka-sink-pipe)

To create a Sink Pipe, create a regular .pipe and filter the data you want to export to your bucket in the SQL section as in any other Pipe. Then, specify the Pipe as a sink type and add the needed configuration. Your Pipe should have the following structure:

NODE node_0

SQL >
    SELECT *
    FROM events
    WHERE time >= toStartOfMinute(now()) - interval 30 minute)

TYPE sink
EXPORT_SERVICE kafka
EXPORT_CONNECTION_NAME "test_kafka"
EXPORT_KAFKA_TOPIC "test_kafka_topic"
EXPORT_SCHEDULE "*/5 * * * *" **Pipe parameters**

For this step, you will need to configure the following [Pipe parameters](https://www.tinybird.co/docs/docs/cli/datafiles/pipe-files#sink-pipe):

| Key | Type | Description |
| --- | --- | --- |
| EXPORT_CONNECTION_NAME | string | Required. The connection name to the destination service. This the connection created in Step 1. |
| EXPORT_KAFKA_TOPIC | string | Required. The desired topic for the export data. |
| EXPORT_SCHEDULE | string | A crontab expression that sets the frequency of the Sink operation or the @on-demand string. |

Once ready, push the datafile to your Workspace using `tb push` (or `tb deploy` if you are using [version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work) ) to create the Sink Pipe.

## Scheduling [¶](https://www.tinybird.co/docs/about:blank#scheduling)

The schedule applied doesn't guarantee that the underlying job executes immediately at the configured time. The job is placed into a job queue when the configured time elapses. It is possible that, if the queue is busy, the job could be delayed and executed after the scheduled time.

To reduce the chances of a busy queue affecting your Sink Pipe execution schedule, distribute the jobs over a wider period of time rather than grouping them close together.

For Enterprise customers, these settings can be customized. Reach out to your Customer Success team or email us at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co).

## Query parameters [¶](https://www.tinybird.co/docs/about:blank#query-parameters)

You can add [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) to your Sink, the same way you do in API Endpoints or Copy Pipes.

For scheduled executions, the default values for the parameters will be used when the Sink runs.

## Iterating a Kafka Sink (Coming soon) [¶](https://www.tinybird.co/docs/about:blank#iterating-a-kafka-sink-coming-soon)

Iterating features for Kafka Sinks aren't yet supported in the beta. They are documented here for future reference.

Sinks can be iterated using [version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work) , similar to other resources in your project. When you create a Branch, resources are cloned from the main Branch.

However, there are two considerations for Kafka Sinks to understand:

**1. Schedules**

When you create a Branch with an existing Kafka Sink, the resource will be cloned into the new Branch. However, **it will not be scheduled** . This prevents Branches from running exports unintentially and consuming resources, as it's common that development Branches don't need to export to external systems. If you want these queries to run in a Branch, you must recreate the Kafka Sink in the new Branch.

**2. Connections**

Connections aren't cloned when you create a Branch. You need to create a new Kafka connection in the new Branch for the Kafka Sink.

## Observability [¶](https://www.tinybird.co/docs/about:blank#observability)

Kafka Sink operations are logged in the [tinybird.sinks_ops_log](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-sinks-ops-log) Service Data Source.

## Limits & quotas [¶](https://www.tinybird.co/docs/about:blank#limits-quotas)



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

## Billing [¶](https://www.tinybird.co/docs/about:blank#billing)

Any Processed Data incurred by a Kafka Sink is charged at the standard rate for your account. The Processed Data is already included in your plan, and counts towards your commitment. If you're on an Enterprise plan, view your plan and commitment on the [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) tab in the UI.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Get familiar with the[  Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources)   and see what's going on in your account
- Deep dive on Tinybird's[  Pipes concept](https://www.tinybird.co/docs/docs/work-with-data/query/pipes)



---

URL: https://www.tinybird.co/docs/publish/charts/guides
Content:
---
title: "Charts guides · Tinybird Docs"
theme-color: "#171612"
description: "Guides for using Tinybird Charts."
---




# Charts guides [¶](https://www.tinybird.co/docs/about:blank#charts-guides)

Learn how to create and integrate Tinybird Charts into your applications. These guides cover different approaches, from no-code solutions using the Tinybird UI to code-strong implementations with the React library.

Tinybird Charts make it easy to visualize your data from published API Endpoints. You can:

- Create charts directly in the Tinybird UI with no coding required.
- Embed charts in your applications using iframes.
- Build custom chart implementations using the `@tinybirdco/charts`   React library.

The following guides are available:



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/list-of-errors
Last update: 2024-11-05T10:29:52.000Z
Content:
---
title: "List of API Endpoint database errors · Tinybird Docs"
theme-color: "#171612"
description: "The following list contains all internal database errors that an API Endpoint might return, and their numbers."
---




# List of internal database errors [¶](https://www.tinybird.co/docs/about:blank#list-of-internal-database-errors)

API Endpoint responses have an additional HTTP header, `X-DB-Exception-Code` , where you can check the internal database error, reported as a stringified number.

The following list contains all internal database errors and their numbers:

- `UNSUPPORTED_METHOD = "1"`
- `UNSUPPORTED_PARAMETER = "2"`
- `UNEXPECTED_END_OF_FILE = "3"`
- `EXPECTED_END_OF_FILE = "4"`
- `CANNOT_PARSE_TEXT = "6"`
- `INCORRECT_NUMBER_OF_COLUMNS = "7"`
- `THERE_IS_NO_COLUMN = "8"`
- `SIZES_OF_COLUMNS_DOESNT_MATCH = "9"`
- `NOT_FOUND_COLUMN_IN_BLOCK = "10"`
- `POSITION_OUT_OF_BOUND = "11"`
- `PARAMETER_OUT_OF_BOUND = "12"`
- `SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH = "13"`
- `DUPLICATE_COLUMN = "15"`
- `NO_SUCH_COLUMN_IN_TABLE = "16"`
- `DELIMITER_IN_STRING_LITERAL_DOESNT_MATCH = "17"`
- `CANNOT_INSERT_ELEMENT_INTO_CONSTANT_COLUMN = "18"`
- `SIZE_OF_FIXED_STRING_DOESNT_MATCH = "19"`
- `NUMBER_OF_COLUMNS_DOESNT_MATCH = "20"`
- `CANNOT_READ_ALL_DATA_FROM_TAB_SEPARATED_INPUT = "21"`
- `CANNOT_PARSE_ALL_VALUE_FROM_TAB_SEPARATED_INPUT = "22"`
- `CANNOT_READ_FROM_ISTREAM = "23"`
- `CANNOT_WRITE_TO_OSTREAM = "24"`
- `CANNOT_PARSE_ESCAPE_SEQUENCE = "25"`
- `CANNOT_PARSE_QUOTED_STRING = "26"`
- `CANNOT_PARSE_INPUT_ASSERTION_FAILED = "27"`
- `CANNOT_PRINT_FLOAT_OR_DOUBLE_NUMBER = "28"`
- `CANNOT_PRINT_INTEGER = "29"`
- `CANNOT_READ_SIZE_OF_COMPRESSED_CHUNK = "30"`
- `CANNOT_READ_COMPRESSED_CHUNK = "31"`
- `ATTEMPT_TO_READ_AFTER_EOF = "32"`
- `CANNOT_READ_ALL_DATA = "33"`
- `TOO_MANY_ARGUMENTS_FOR_FUNCTION = "34"`
- `TOO_FEW_ARGUMENTS_FOR_FUNCTION = "35"`
- `BAD_ARGUMENTS = "36"`
- `UNKNOWN_ELEMENT_IN_AST = "37"`
- `CANNOT_PARSE_DATE = "38"`
- `TOO_LARGE_SIZE_COMPRESSED = "39"`
- `CHECKSUM_DOESNT_MATCH = "40"`
- `CANNOT_PARSE_DATETIME = "41"`
- `NUMBER_OF_ARGUMENTS_DOESNT_MATCH = "42"`
- `ILLEGAL_TYPE_OF_ARGUMENT = "43"`
- `ILLEGAL_COLUMN = "44"`
- `ILLEGAL_NUMBER_OF_RESULT_COLUMNS = "45"`
- `UNKNOWN_FUNCTION = "46"`
- `UNKNOWN_IDENTIFIER = "47"`
- `NOT_IMPLEMENTED = "48"`
- `LOGICAL_ERROR = "49"`
- `UNKNOWN_TYPE = "50"`
- `EMPTY_LIST_OF_COLUMNS_QUERIED = "51"`
- `COLUMN_QUERIED_MORE_THAN_ONCE = "52"`
- `TYPE_MISMATCH = "53"`
- `STORAGE_DOESNT_ALLOW_PARAMETERS = "54"`
- `STORAGE_REQUIRES_PARAMETER = "55"`
- `UNKNOWN_STORAGE = "56"`
- `TABLE_ALREADY_EXISTS = "57"`
- `TABLE_METADATA_ALREADY_EXISTS = "58"`
- `ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER = "59"`
- `UNKNOWN_TABLE = "60"`
- `ONLY_FILTER_COLUMN_IN_BLOCK = "61"`
- `SYNTAX_ERROR = "62"`
- `UNKNOWN_AGGREGATE_FUNCTION = "63"`
- `CANNOT_READ_AGGREGATE_FUNCTION_FROM_TEXT = "64"`
- `CANNOT_WRITE_AGGREGATE_FUNCTION_AS_TEXT = "65"`
- `NOT_A_COLUMN = "66"`
- `ILLEGAL_KEY_OF_AGGREGATION = "67"`
- `CANNOT_GET_SIZE_OF_FIELD = "68"`
- `ARGUMENT_OUT_OF_BOUND = "69"`
- `CANNOT_CONVERT_TYPE = "70"`
- `CANNOT_WRITE_AFTER_END_OF_BUFFER = "71"`
- `CANNOT_PARSE_NUMBER = "72"`
- `UNKNOWN_FORMAT = "73"`
- `CANNOT_READ_FROM_FILE_DESCRIPTOR = "74"`
- `CANNOT_WRITE_TO_FILE_DESCRIPTOR = "75"`
- `CANNOT_OPEN_FILE = "76"`
- `CANNOT_CLOSE_FILE = "77"`
- `UNKNOWN_TYPE_OF_QUERY = "78"`
- `INCORRECT_FILE_NAME = "79"`
- `INCORRECT_QUERY = "80"`
- `UNKNOWN_DATABASE = "81"`
- `DATABASE_ALREADY_EXISTS = "82"`
- `DIRECTORY_DOESNT_EXIST = "83"`
- `DIRECTORY_ALREADY_EXISTS = "84"`
- `FORMAT_IS_NOT_SUITABLE_FOR_INPUT = "85"`
- `RECEIVED_ERROR_FROM_REMOTE_IO_SERVER = "86"`
- `CANNOT_SEEK_THROUGH_FILE = "87"`
- `CANNOT_TRUNCATE_FILE = "88"`
- `UNKNOWN_COMPRESSION_METHOD = "89"`
- `EMPTY_LIST_OF_COLUMNS_PASSED = "90"`
- `SIZES_OF_MARKS_FILES_ARE_INCONSISTENT = "91"`
- `EMPTY_DATA_PASSED = "92"`
- `UNKNOWN_AGGREGATED_DATA_VARIANT = "93"`
- `CANNOT_MERGE_DIFFERENT_AGGREGATED_DATA_VARIANTS = "94"`
- `CANNOT_READ_FROM_SOCKET = "95"`
- `CANNOT_WRITE_TO_SOCKET = "96"`
- `CANNOT_READ_ALL_DATA_FROM_CHUNKED_INPUT = "97"`
- `CANNOT_WRITE_TO_EMPTY_BLOCK_OUTPUT_STREAM = "98"`
- `UNKNOWN_PACKET_FROM_CLIENT = "99"`
- `UNKNOWN_PACKET_FROM_SERVER = "100"`
- `UNEXPECTED_PACKET_FROM_CLIENT = "101"`
- `UNEXPECTED_PACKET_FROM_SERVER = "102"`
- `RECEIVED_DATA_FOR_WRONG_QUERY_ID = "103"`
- `TOO_SMALL_BUFFER_SIZE = "104"`
- `CANNOT_READ_HISTORY = "105"`
- `CANNOT_APPEND_HISTORY = "106"`
- `FILE_DOESNT_EXIST = "107"`
- `NO_DATA_TO_INSERT = "108"`
- `CANNOT_BLOCK_SIGNAL = "109"`
- `CANNOT_UNBLOCK_SIGNAL = "110"`
- `CANNOT_MANIPULATE_SIGSET = "111"`
- `CANNOT_WAIT_FOR_SIGNAL = "112"`
- `THERE_IS_NO_SESSION = "113"`
- `CANNOT_CLOCK_GETTIME = "114"`
- `UNKNOWN_SETTING = "115"`
- `THERE_IS_NO_DEFAULT_VALUE = "116"`
- `INCORRECT_DATA = "117"`
- `ENGINE_REQUIRED = "119"`
- `CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE = "120"`
- `UNSUPPORTED_JOIN_KEYS = "121"`
- `INCOMPATIBLE_COLUMNS = "122"`
- `UNKNOWN_TYPE_OF_AST_NODE = "123"`
- `INCORRECT_ELEMENT_OF_SET = "124"`
- `INCORRECT_RESULT_OF_SCALAR_SUBQUERY = "125"`
- `CANNOT_GET_RETURN_TYPE = "126"`
- `ILLEGAL_INDEX = "127"`
- `TOO_LARGE_ARRAY_SIZE = "128"`
- `FUNCTION_IS_SPECIAL = "129"`
- `CANNOT_READ_ARRAY_FROM_TEXT = "130"`
- `TOO_LARGE_STRING_SIZE = "131"`
- `AGGREGATE_FUNCTION_DOESNT_ALLOW_PARAMETERS = "133"`
- `PARAMETERS_TO_AGGREGATE_FUNCTIONS_MUST_BE_LITERALS = "134"`
- `ZERO_ARRAY_OR_TUPLE_INDEX = "135"`
- `UNKNOWN_ELEMENT_IN_CONFIG = "137"`
- `EXCESSIVE_ELEMENT_IN_CONFIG = "138"`
- `NO_ELEMENTS_IN_CONFIG = "139"`
- `ALL_REQUESTED_COLUMNS_ARE_MISSING = "140"`
- `SAMPLING_NOT_SUPPORTED = "141"`
- `NOT_FOUND_NODE = "142"`
- `FOUND_MORE_THAN_ONE_NODE = "143"`
- `FIRST_DATE_IS_BIGGER_THAN_LAST_DATE = "144"`
- `UNKNOWN_OVERFLOW_MODE = "145"`
- `QUERY_SECTION_DOESNT_MAKE_SENSE = "146"`
- `NOT_FOUND_FUNCTION_ELEMENT_FOR_AGGREGATE = "147"`
- `NOT_FOUND_RELATION_ELEMENT_FOR_CONDITION = "148"`
- `NOT_FOUND_RHS_ELEMENT_FOR_CONDITION = "149"`
- `EMPTY_LIST_OF_ATTRIBUTES_PASSED = "150"`
- `INDEX_OF_COLUMN_IN_SORT_CLAUSE_IS_OUT_OF_RANGE = "151"`
- `UNKNOWN_DIRECTION_OF_SORTING = "152"`
- `ILLEGAL_DIVISION = "153"`
- `AGGREGATE_FUNCTION_NOT_APPLICABLE = "154"`
- `UNKNOWN_RELATION = "155"`
- `DICTIONARIES_WAS_NOT_LOADED = "156"`
- `ILLEGAL_OVERFLOW_MODE = "157"`
- `TOO_MANY_ROWS = "158"`
- `TIMEOUT_EXCEEDED = "159"`
- `TOO_SLOW = "160"`
- `TOO_MANY_COLUMNS = "161"`
- `TOO_DEEP_SUBQUERIES = "162"`
- `TOO_DEEP_PIPELINE = "163"`
- `READONLY = "164"`
- `TOO_MANY_TEMPORARY_COLUMNS = "165"`
- `TOO_MANY_TEMPORARY_NON_CONST_COLUMNS = "166"`
- `TOO_DEEP_AST = "167"`
- `TOO_BIG_AST = "168"`
- `BAD_TYPE_OF_FIELD = "169"`
- `BAD_GET = "170"`
- `CANNOT_CREATE_DIRECTORY = "172"`
- `CANNOT_ALLOCATE_MEMORY = "173"`
- `CYCLIC_ALIASES = "174"`
- `CHUNK_NOT_FOUND = "176"`
- `DUPLICATE_CHUNK_NAME = "177"`
- `MULTIPLE_ALIASES_FOR_EXPRESSION = "178"`
- `MULTIPLE_EXPRESSIONS_FOR_ALIAS = "179"`
- `THERE_IS_NO_PROFILE = "180"`
- `ILLEGAL_FINAL = "181"`
- `ILLEGAL_PREWHERE = "182"`
- `UNEXPECTED_EXPRESSION = "183"`
- `ILLEGAL_AGGREGATION = "184"`
- `UNSUPPORTED_MYISAM_BLOCK_TYPE = "185"`
- `UNSUPPORTED_COLLATION_LOCALE = "186"`
- `COLLATION_COMPARISON_FAILED = "187"`
- `UNKNOWN_ACTION = "188"`
- `TABLE_MUST_NOT_BE_CREATED_MANUALLY = "189"`
- `SIZES_OF_ARRAYS_DOESNT_MATCH = "190"`
- `SET_SIZE_LIMIT_EXCEEDED = "191"`
- `UNKNOWN_USER = "192"`
- `WRONG_PASSWORD = "193"`
- `REQUIRED_PASSWORD = "194"`
- `IP_ADDRESS_NOT_ALLOWED = "195"`
- `UNKNOWN_ADDRESS_PATTERN_TYPE = "196"`
- `SERVER_REVISION_IS_TOO_OLD = "197"`
- `DNS_ERROR = "198"`
- `UNKNOWN_QUOTA = "199"`
- `QUOTA_DOESNT_ALLOW_KEYS = "200"`
- `QUOTA_EXCEEDED = "201"`
- `TOO_MANY_SIMULTANEOUS_QUERIES = "202"`
- `NO_FREE_CONNECTION = "203"`
- `CANNOT_FSYNC = "204"`
- `NESTED_TYPE_TOO_DEEP = "205"`
- `ALIAS_REQUIRED = "206"`
- `AMBIGUOUS_IDENTIFIER = "207"`
- `EMPTY_NESTED_TABLE = "208"`
- `SOCKET_TIMEOUT = "209"`
- `NETWORK_ERROR = "210"`
- `EMPTY_QUERY = "211"`
- `UNKNOWN_LOAD_BALANCING = "212"`
- `UNKNOWN_TOTALS_MODE = "213"`
- `CANNOT_STATVFS = "214"`
- `NOT_AN_AGGREGATE = "215"`
- `QUERY_WITH_SAME_ID_IS_ALREADY_RUNNING = "216"`
- `CLIENT_HAS_CONNECTED_TO_WRONG_PORT = "217"`
- `TABLE_IS_DROPPED = "218"`
- `DATABASE_NOT_EMPTY = "219"`
- `DUPLICATE_INTERSERVER_IO_ENDPOINT = "220"`
- `NO_SUCH_INTERSERVER_IO_ENDPOINT = "221"`
- `ADDING_REPLICA_TO_NON_EMPTY_TABLE = "222"`
- `UNEXPECTED_AST_STRUCTURE = "223"`
- `REPLICA_IS_ALREADY_ACTIVE = "224"`
- `NO_ZOOKEEPER = "225"`
- `NO_FILE_IN_DATA_PART = "226"`
- `UNEXPECTED_FILE_IN_DATA_PART = "227"`
- `BAD_SIZE_OF_FILE_IN_DATA_PART = "228"`
- `QUERY_IS_TOO_LARGE = "229"`
- `NOT_FOUND_EXPECTED_DATA_PART = "230"`
- `TOO_MANY_UNEXPECTED_DATA_PARTS = "231"`
- `NO_SUCH_DATA_PART = "232"`
- `BAD_DATA_PART_NAME = "233"`
- `NO_REPLICA_HAS_PART = "234"`
- `DUPLICATE_DATA_PART = "235"`
- `ABORTED = "236"`
- `NO_REPLICA_NAME_GIVEN = "237"`
- `FORMAT_VERSION_TOO_OLD = "238"`
- `CANNOT_MUNMAP = "239"`
- `CANNOT_MREMAP = "240"`
- `MEMORY_LIMIT_EXCEEDED = "241"`
- `TABLE_IS_READ_ONLY = "242"`
- `NOT_ENOUGH_SPACE = "243"`
- `UNEXPECTED_ZOOKEEPER_ERROR = "244"`
- `CORRUPTED_DATA = "246"`
- `INCORRECT_MARK = "247"`
- `INVALID_PARTITION_VALUE = "248"`
- `NOT_ENOUGH_BLOCK_NUMBERS = "250"`
- `NO_SUCH_REPLICA = "251"`
- `TOO_MANY_PARTS = "252"`
- `REPLICA_IS_ALREADY_EXIST = "253"`
- `NO_ACTIVE_REPLICAS = "254"`
- `TOO_MANY_RETRIES_TO_FETCH_PARTS = "255"`
- `PARTITION_ALREADY_EXISTS = "256"`
- `PARTITION_DOESNT_EXIST = "257"`
- `UNION_ALL_RESULT_STRUCTURES_MISMATCH = "258"`
- `CLIENT_OUTPUT_FORMAT_SPECIFIED = "260"`
- `UNKNOWN_BLOCK_INFO_FIELD = "261"`
- `BAD_COLLATION = "262"`
- `CANNOT_COMPILE_CODE = "263"`
- `INCOMPATIBLE_TYPE_OF_JOIN = "264"`
- `NO_AVAILABLE_REPLICA = "265"`
- `MISMATCH_REPLICAS_DATA_SOURCES = "266"`
- `STORAGE_DOESNT_SUPPORT_PARALLEL_REPLICAS = "267"`
- `CPUID_ERROR = "268"`
- `INFINITE_LOOP = "269"`
- `CANNOT_COMPRESS = "270"`
- `CANNOT_DECOMPRESS = "271"`
- `CANNOT_IO_SUBMIT = "272"`
- `CANNOT_IO_GETEVENTS = "273"`
- `AIO_READ_ERROR = "274"`
- `AIO_WRITE_ERROR = "275"`
- `INDEX_NOT_USED = "277"`
- `ALL_CONNECTION_TRIES_FAILED = "279"`
- `NO_AVAILABLE_DATA = "280"`
- `DICTIONARY_IS_EMPTY = "281"`
- `INCORRECT_INDEX = "282"`
- `UNKNOWN_DISTRIBUTED_PRODUCT_MODE = "283"`
- `WRONG_GLOBAL_SUBQUERY = "284"`
- `TOO_FEW_LIVE_REPLICAS = "285"`
- `UNSATISFIED_QUORUM_FOR_PREVIOUS_WRITE = "286"`
- `UNKNOWN_FORMAT_VERSION = "287"`
- `DISTRIBUTED_IN_JOIN_SUBQUERY_DENIED = "288"`
- `REPLICA_IS_NOT_IN_QUORUM = "289"`
- `LIMIT_EXCEEDED = "290"`
- `DATABASE_ACCESS_DENIED = "291"`
- `MONGODB_CANNOT_AUTHENTICATE = "293"`
- `INVALID_BLOCK_EXTRA_INFO = "294"`
- `RECEIVED_EMPTY_DATA = "295"`
- `NO_REMOTE_SHARD_FOUND = "296"`
- `SHARD_HAS_NO_CONNECTIONS = "297"`
- `CANNOT_PIPE = "298"`
- `CANNOT_FORK = "299"`
- `CANNOT_DLSYM = "300"`
- `CANNOT_CREATE_CHILD_PROCESS = "301"`
- `CHILD_WAS_NOT_EXITED_NORMALLY = "302"`
- `CANNOT_SELECT = "303"`
- `CANNOT_WAITPID = "304"`
- `TABLE_WAS_NOT_DROPPED = "305"`
- `TOO_DEEP_RECURSION = "306"`
- `TOO_MANY_BYTES = "307"`
- `UNEXPECTED_NODE_IN_ZOOKEEPER = "308"`
- `FUNCTION_CANNOT_HAVE_PARAMETERS = "309"`
- `INVALID_SHARD_WEIGHT = "317"`
- `INVALID_CONFIG_PARAMETER = "318"`
- `UNKNOWN_STATUS_OF_INSERT = "319"`
- `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE = "321"`
- `BARRIER_TIMEOUT = "335"`
- `UNKNOWN_DATABASE_ENGINE = "336"`
- `DDL_GUARD_IS_ACTIVE = "337"`
- `UNFINISHED = "341"`
- `METADATA_MISMATCH = "342"`
- `SUPPORT_IS_DISABLED = "344"`
- `TABLE_DIFFERS_TOO_MUCH = "345"`
- `CANNOT_CONVERT_CHARSET = "346"`
- `CANNOT_LOAD_CONFIG = "347"`
- `CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN = "349"`
- `INCOMPATIBLE_SOURCE_TABLES = "350"`
- `AMBIGUOUS_TABLE_NAME = "351"`
- `AMBIGUOUS_COLUMN_NAME = "352"`
- `INDEX_OF_POSITIONAL_ARGUMENT_IS_OUT_OF_RANGE = "353"`
- `ZLIB_INFLATE_FAILED = "354"`
- `ZLIB_DEFLATE_FAILED = "355"`
- `BAD_LAMBDA = "356"`
- `RESERVED_IDENTIFIER_NAME = "357"`
- `INTO_OUTFILE_NOT_ALLOWED = "358"`
- `TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "359"`
- `CANNOT_CREATE_CHARSET_CONVERTER = "360"`
- `SEEK_POSITION_OUT_OF_BOUND = "361"`
- `CURRENT_WRITE_BUFFER_IS_EXHAUSTED = "362"`
- `CANNOT_CREATE_IO_BUFFER = "363"`
- `RECEIVED_ERROR_TOO_MANY_REQUESTS = "364"`
- `SIZES_OF_NESTED_COLUMNS_ARE_INCONSISTENT = "366"`
- `TOO_MANY_FETCHES = "367"`
- `ALL_REPLICAS_ARE_STALE = "369"`
- `DATA_TYPE_CANNOT_BE_USED_IN_TABLES = "370"`
- `INCONSISTENT_CLUSTER_DEFINITION = "371"`
- `SESSION_NOT_FOUND = "372"`
- `SESSION_IS_LOCKED = "373"`
- `INVALID_SESSION_TIMEOUT = "374"`
- `CANNOT_DLOPEN = "375"`
- `CANNOT_PARSE_UUID = "376"`
- `ILLEGAL_SYNTAX_FOR_DATA_TYPE = "377"`
- `DATA_TYPE_CANNOT_HAVE_ARGUMENTS = "378"`
- `UNKNOWN_STATUS_OF_DISTRIBUTED_DDL_TASK = "379"`
- `CANNOT_KILL = "380"`
- `HTTP_LENGTH_REQUIRED = "381"`
- `CANNOT_LOAD_CATBOOST_MODEL = "382"`
- `CANNOT_APPLY_CATBOOST_MODEL = "383"`
- `PART_IS_TEMPORARILY_LOCKED = "384"`
- `MULTIPLE_STREAMS_REQUIRED = "385"`
- `NO_COMMON_TYPE = "386"`
- `DICTIONARY_ALREADY_EXISTS = "387"`
- `CANNOT_ASSIGN_OPTIMIZE = "388"`
- `INSERT_WAS_DEDUPLICATED = "389"`
- `CANNOT_GET_CREATE_TABLE_QUERY = "390"`
- `EXTERNAL_LIBRARY_ERROR = "391"`
- `QUERY_IS_PROHIBITED = "392"`
- `THERE_IS_NO_QUERY = "393"`
- `QUERY_WAS_CANCELLED = "394"`
- `FUNCTION_THROW_IF_VALUE_IS_NON_ZERO = "395"`
- `TOO_MANY_ROWS_OR_BYTES = "396"`
- `QUERY_IS_NOT_SUPPORTED_IN_MATERIALIZED_VIEW = "397"`
- `UNKNOWN_MUTATION_COMMAND = "398"`
- `FORMAT_IS_NOT_SUITABLE_FOR_OUTPUT = "399"`
- `CANNOT_STAT = "400"`
- `FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME = "401"`
- `CANNOT_IOSETUP = "402"`
- `INVALID_JOIN_ON_EXPRESSION = "403"`
- `BAD_ODBC_CONNECTION_STRING = "404"`
- `PARTITION_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT = "405"`
- `TOP_AND_LIMIT_TOGETHER = "406"`
- `DECIMAL_OVERFLOW = "407"`
- `BAD_REQUEST_PARAMETER = "408"`
- `EXTERNAL_EXECUTABLE_NOT_FOUND = "409"`
- `EXTERNAL_SERVER_IS_NOT_RESPONDING = "410"`
- `PTHREAD_ERROR = "411"`
- `NETLINK_ERROR = "412"`
- `CANNOT_SET_SIGNAL_HANDLER = "413"`
- `ALL_REPLICAS_LOST = "415"`
- `REPLICA_STATUS_CHANGED = "416"`
- `EXPECTED_ALL_OR_ANY = "417"`
- `UNKNOWN_JOIN = "418"`
- `MULTIPLE_ASSIGNMENTS_TO_COLUMN = "419"`
- `CANNOT_UPDATE_COLUMN = "420"`
- `CANNOT_ADD_DIFFERENT_AGGREGATE_STATES = "421"`
- `UNSUPPORTED_URI_SCHEME = "422"`
- `CANNOT_GETTIMEOFDAY = "423"`
- `CANNOT_LINK = "424"`
- `SYSTEM_ERROR = "425"`
- `CANNOT_COMPILE_REGEXP = "427"`
- `UNKNOWN_LOG_LEVEL = "428"`
- `FAILED_TO_GETPWUID = "429"`
- `MISMATCHING_USERS_FOR_PROCESS_AND_DATA = "430"`
- `ILLEGAL_SYNTAX_FOR_CODEC_TYPE = "431"`
- `UNKNOWN_CODEC = "432"`
- `ILLEGAL_CODEC_PARAMETER = "433"`
- `CANNOT_PARSE_PROTOBUF_SCHEMA = "434"`
- `NO_COLUMN_SERIALIZED_TO_REQUIRED_PROTOBUF_FIELD = "435"`
- `PROTOBUF_BAD_CAST = "436"`
- `PROTOBUF_FIELD_NOT_REPEATED = "437"`
- `DATA_TYPE_CANNOT_BE_PROMOTED = "438"`
- `CANNOT_SCHEDULE_TASK = "439"`
- `INVALID_LIMIT_EXPRESSION = "440"`
- `CANNOT_PARSE_DOMAIN_VALUE_FROM_STRING = "441"`
- `BAD_DATABASE_FOR_TEMPORARY_TABLE = "442"`
- `NO_COLUMNS_SERIALIZED_TO_PROTOBUF_FIELDS = "443"`
- `UNKNOWN_PROTOBUF_FORMAT = "444"`
- `CANNOT_MPROTECT = "445"`
- `FUNCTION_NOT_ALLOWED = "446"`
- `HYPERSCAN_CANNOT_SCAN_TEXT = "447"`
- `BROTLI_READ_FAILED = "448"`
- `BROTLI_WRITE_FAILED = "449"`
- `BAD_TTL_EXPRESSION = "450"`
- `BAD_TTL_FILE = "451"`
- `SETTING_CONSTRAINT_VIOLATION = "452"`
- `MYSQL_CLIENT_INSUFFICIENT_CAPABILITIES = "453"`
- `OPENSSL_ERROR = "454"`
- `SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY = "455"`
- `UNKNOWN_QUERY_PARAMETER = "456"`
- `BAD_QUERY_PARAMETER = "457"`
- `CANNOT_UNLINK = "458"`
- `CANNOT_SET_THREAD_PRIORITY = "459"`
- `CANNOT_CREATE_TIMER = "460"`
- `CANNOT_SET_TIMER_PERIOD = "461"`
- `CANNOT_DELETE_TIMER = "462"`
- `CANNOT_FCNTL = "463"`
- `CANNOT_PARSE_ELF = "464"`
- `CANNOT_PARSE_DWARF = "465"`
- `INSECURE_PATH = "466"`
- `CANNOT_PARSE_BOOL = "467"`
- `CANNOT_PTHREAD_ATTR = "468"`
- `VIOLATED_CONSTRAINT = "469"`
- `QUERY_IS_NOT_SUPPORTED_IN_LIVE_VIEW = "470"`
- `INVALID_SETTING_VALUE = "471"`
- `READONLY_SETTING = "472"`
- `DEADLOCK_AVOIDED = "473"`
- `INVALID_TEMPLATE_FORMAT = "474"`
- `INVALID_WITH_FILL_EXPRESSION = "475"`
- `WITH_TIES_WITHOUT_ORDER_BY = "476"`
- `INVALID_USAGE_OF_INPUT = "477"`
- `UNKNOWN_POLICY = "478"`
- `UNKNOWN_DISK = "479"`
- `UNKNOWN_PROTOCOL = "480"`
- `PATH_ACCESS_DENIED = "481"`
- `DICTIONARY_ACCESS_DENIED = "482"`
- `TOO_MANY_REDIRECTS = "483"`
- `INTERNAL_REDIS_ERROR = "484"`
- `SCALAR_ALREADY_EXISTS = "485"`
- `CANNOT_GET_CREATE_DICTIONARY_QUERY = "487"`
- `UNKNOWN_DICTIONARY = "488"`
- `INCORRECT_DICTIONARY_DEFINITION = "489"`
- `CANNOT_FORMAT_DATETIME = "490"`
- `UNACCEPTABLE_URL = "491"`
- `ACCESS_ENTITY_NOT_FOUND = "492"`
- `ACCESS_ENTITY_ALREADY_EXISTS = "493"`
- `ACCESS_ENTITY_FOUND_DUPLICATES = "494"`
- `ACCESS_STORAGE_READONLY = "495"`
- `QUOTA_REQUIRES_CLIENT_KEY = "496"`
- `ACCESS_DENIED = "497"`
- `LIMIT_BY_WITH_TIES_IS_NOT_SUPPORTED = "498"`
- S3_ERROR = "499"
- `AZURE_BLOB_STORAGE_ERROR = "500"`
- `CANNOT_CREATE_DATABASE = "501"`
- `CANNOT_SIGQUEUE = "502"`
- `AGGREGATE_FUNCTION_THROW = "503"`
- `FILE_ALREADY_EXISTS = "504"`
- `CANNOT_DELETE_DIRECTORY = "505"`
- `UNEXPECTED_ERROR_CODE = "506"`
- `UNABLE_TO_SKIP_UNUSED_SHARDS = "507"`
- `UNKNOWN_ACCESS_TYPE = "508"`
- `INVALID_GRANT = "509"`
- `CACHE_DICTIONARY_UPDATE_FAIL = "510"`
- `UNKNOWN_ROLE = "511"`
- `SET_NON_GRANTED_ROLE = "512"`
- `UNKNOWN_PART_TYPE = "513"`
- `ACCESS_STORAGE_FOR_INSERTION_NOT_FOUND = "514"`
- `INCORRECT_ACCESS_ENTITY_DEFINITION = "515"`
- `AUTHENTICATION_FAILED = "516"`
- `CANNOT_ASSIGN_ALTER = "517"`
- `CANNOT_COMMIT_OFFSET = "518"`
- `NO_REMOTE_SHARD_AVAILABLE = "519"`
- `CANNOT_DETACH_DICTIONARY_AS_TABLE = "520"`
- `ATOMIC_RENAME_FAIL = "521"`
- `UNKNOWN_ROW_POLICY = "523"`
- `ALTER_OF_COLUMN_IS_FORBIDDEN = "524"`
- `INCORRECT_DISK_INDEX = "525"`
- `NO_SUITABLE_FUNCTION_IMPLEMENTATION = "527"`
- `CASSANDRA_INTERNAL_ERROR = "528"`
- `NOT_A_LEADER = "529"`
- `CANNOT_CONNECT_RABBITMQ = "530"`
- `CANNOT_FSTAT = "531"`
- `LDAP_ERROR = "532"`
- `INCONSISTENT_RESERVATIONS = "533"`
- `NO_RESERVATIONS_PROVIDED = "534"`
- `UNKNOWN_RAID_TYPE = "535"`
- `CANNOT_RESTORE_FROM_FIELD_DUMP = "536"`
- `ILLEGAL_MYSQL_VARIABLE = "537"`
- `MYSQL_SYNTAX_ERROR = "538"`
- `CANNOT_BIND_RABBITMQ_EXCHANGE = "539"`
- `CANNOT_DECLARE_RABBITMQ_EXCHANGE = "540"`
- `CANNOT_CREATE_RABBITMQ_QUEUE_BINDING = "541"`
- `CANNOT_REMOVE_RABBITMQ_EXCHANGE = "542"`
- `UNKNOWN_MYSQL_DATATYPES_SUPPORT_LEVEL = "543"`
- `ROW_AND_ROWS_TOGETHER = "544"`
- `FIRST_AND_NEXT_TOGETHER = "545"`
- `NO_ROW_DELIMITER = "546"`
- `INVALID_RAID_TYPE = "547"`
- `UNKNOWN_VOLUME = "548"`
- `DATA_TYPE_CANNOT_BE_USED_IN_KEY = "549"`
- `CONDITIONAL_TREE_PARENT_NOT_FOUND = "550"`
- `ILLEGAL_PROJECTION_MANIPULATOR = "551"`
- `UNRECOGNIZED_ARGUMENTS = "552"`
- `LZMA_STREAM_ENCODER_FAILED = "553"`
- `LZMA_STREAM_DECODER_FAILED = "554"`
- `ROCKSDB_ERROR = "555"`
- `SYNC_MYSQL_USER_ACCESS_ERROR = "556"`
- `UNKNOWN_UNION = "557"`
- `EXPECTED_ALL_OR_DISTINCT = "558"`
- `INVALID_GRPC_QUERY_INFO = "559"`
- `ZSTD_ENCODER_FAILED = "560"`
- `ZSTD_DECODER_FAILED = "561"`
- `TLD_LIST_NOT_FOUND = "562"`
- `CANNOT_READ_MAP_FROM_TEXT = "563"`
- `INTERSERVER_SCHEME_DOESNT_MATCH = "564"`
- `TOO_MANY_PARTITIONS = "565"`
- `CANNOT_RMDIR = "566"`
- `DUPLICATED_PART_UUIDS = "567"`
- `RAFT_ERROR = "568"`
- `MULTIPLE_COLUMNS_SERIALIZED_TO_SAME_PROTOBUF_FIELD = "569"`
- `DATA_TYPE_INCOMPATIBLE_WITH_PROTOBUF_FIELD = "570"`
- `DATABASE_REPLICATION_FAILED = "571"`
- `TOO_MANY_QUERY_PLAN_OPTIMIZATIONS = "572"`
- `EPOLL_ERROR = "573"`
- `DISTRIBUTED_TOO_MANY_PENDING_BYTES = "574"`
- `UNKNOWN_SNAPSHOT = "575"`
- `KERBEROS_ERROR = "576"`
- `INVALID_SHARD_ID = "577"`
- `INVALID_FORMAT_INSERT_QUERY_WITH_DATA = "578"`
- `INCORRECT_PART_TYPE = "579"`
- `CANNOT_SET_ROUNDING_MODE = "580"`
- `TOO_LARGE_DISTRIBUTED_DEPTH = "581"`
- `NO_SUCH_PROJECTION_IN_TABLE = "582"`
- `ILLEGAL_PROJECTION = "583"`
- `PROJECTION_NOT_USED = "584"`
- `CANNOT_PARSE_YAML = "585"`
- `CANNOT_CREATE_FILE = "586"`
- `CONCURRENT_ACCESS_NOT_SUPPORTED = "587"`
- `DISTRIBUTED_BROKEN_BATCH_INFO = "588"`
- `DISTRIBUTED_BROKEN_BATCH_FILES = "589"`
- `CANNOT_SYSCONF = "590"`
- `SQLITE_ENGINE_ERROR = "591"`
- `DATA_ENCRYPTION_ERROR = "592"`
- `ZERO_COPY_REPLICATION_ERROR = "593"`
- BZIP2_STREAM_DECODER_FAILED = "594"
- BZIP2_STREAM_ENCODER_FAILED = "595"
- `INTERSECT_OR_EXCEPT_RESULT_STRUCTURES_MISMATCH = "596"`
- `NO_SUCH_ERROR_CODE = "597"`
- `BACKUP_ALREADY_EXISTS = "598"`
- `BACKUP_NOT_FOUND = "599"`
- `BACKUP_VERSION_NOT_SUPPORTED = "600"`
- `BACKUP_DAMAGED = "601"`
- `NO_BASE_BACKUP = "602"`
- `WRONG_BASE_BACKUP = "603"`
- `BACKUP_ENTRY_ALREADY_EXISTS = "604"`
- `BACKUP_ENTRY_NOT_FOUND = "605"`
- `BACKUP_IS_EMPTY = "606"`
- `CANNOT_RESTORE_DATABASE = "607"`
- `CANNOT_RESTORE_TABLE = "608"`
- `FUNCTION_ALREADY_EXISTS = "609"`
- `CANNOT_DROP_FUNCTION = "610"`
- `CANNOT_CREATE_RECURSIVE_FUNCTION = "611"`
- `OBJECT_ALREADY_STORED_ON_DISK = "612"`
- `OBJECT_WAS_NOT_STORED_ON_DISK = "613"`
- `POSTGRESQL_CONNECTION_FAILURE = "614"`
- `CANNOT_ADVISE = "615"`
- `UNKNOWN_READ_METHOD = "616"`
- LZ4_ENCODER_FAILED = "617"
- LZ4_DECODER_FAILED = "618"
- `POSTGRESQL_REPLICATION_INTERNAL_ERROR = "619"`
- `QUERY_NOT_ALLOWED = "620"`
- `CANNOT_NORMALIZE_STRING = "621"`
- `CANNOT_PARSE_CAPN_PROTO_SCHEMA = "622"`
- `CAPN_PROTO_BAD_CAST = "623"`
- `BAD_FILE_TYPE = "624"`
- `IO_SETUP_ERROR = "625"`
- `CANNOT_SKIP_UNKNOWN_FIELD = "626"`
- `BACKUP_ENGINE_NOT_FOUND = "627"`
- `OFFSET_FETCH_WITHOUT_ORDER_BY = "628"`
- `HTTP_RANGE_NOT_SATISFIABLE = "629"`
- `HAVE_DEPENDENT_OBJECTS = "630"`
- `UNKNOWN_FILE_SIZE = "631"`
- `UNEXPECTED_DATA_AFTER_PARSED_VALUE = "632"`
- `QUERY_IS_NOT_SUPPORTED_IN_WINDOW_VIEW = "633"`
- `MONGODB_ERROR = "634"`
- `CANNOT_POLL = "635"`
- `CANNOT_EXTRACT_TABLE_STRUCTURE = "636"`
- `INVALID_TABLE_OVERRIDE = "637"`
- `SNAPPY_UNCOMPRESS_FAILED = "638"`
- `SNAPPY_COMPRESS_FAILED = "639"`
- `NO_HIVEMETASTORE = "640"`
- `CANNOT_APPEND_TO_FILE = "641"`
- `CANNOT_PACK_ARCHIVE = "642"`
- `CANNOT_UNPACK_ARCHIVE = "643"`
- `REMOTE_FS_OBJECT_CACHE_ERROR = "644"`
- `NUMBER_OF_DIMENSIONS_MISMATCHED = "645"`
- `CANNOT_BACKUP_DATABASE = "646"`
- `CANNOT_BACKUP_TABLE = "647"`
- `WRONG_DDL_RENAMING_SETTINGS = "648"`
- `INVALID_TRANSACTION = "649"`
- `SERIALIZATION_ERROR = "650"`
- `CAPN_PROTO_BAD_TYPE = "651"`
- `ONLY_NULLS_WHILE_READING_SCHEMA = "652"`
- `CANNOT_PARSE_BACKUP_SETTINGS = "653"`
- `WRONG_BACKUP_SETTINGS = "654"`
- `FAILED_TO_SYNC_BACKUP_OR_RESTORE = "655"`
- `MEILISEARCH_EXCEPTION = "656"`
- `UNSUPPORTED_MEILISEARCH_TYPE = "657"`
- `MEILISEARCH_MISSING_SOME_COLUMNS = "658"`
- `UNKNOWN_STATUS_OF_TRANSACTION = "659"`
- `HDFS_ERROR = "660"`
- `CANNOT_SEND_SIGNAL = "661"`
- `FS_METADATA_ERROR = "662"`
- `INCONSISTENT_METADATA_FOR_BACKUP = "663"`
- `ACCESS_STORAGE_DOESNT_ALLOW_BACKUP = "664"`
- `CANNOT_CONNECT_NATS = "665"`
- `NOT_INITIALIZED = "667"`
- `INVALID_STATE = "668"`
- `NAMED_COLLECTION_DOESNT_EXIST = "669"`
- `NAMED_COLLECTION_ALREADY_EXISTS = "670"`
- `NAMED_COLLECTION_IS_IMMUTABLE = "671"`
- `INVALID_SCHEDULER_NODE = "672"`
- `RESOURCE_ACCESS_DENIED = "673"`
- `RESOURCE_NOT_FOUND = "674"`
- CANNOT_PARSE_IPV4 = "675"
- CANNOT_PARSE_IPV6 = "676"
- `THREAD_WAS_CANCELED = "677"`
- `IO_URING_INIT_FAILED = "678"`
- `IO_URING_SUBMIT_ERROR = "679"`
- `MIXED_ACCESS_PARAMETER_TYPES = "690"`
- `UNKNOWN_ELEMENT_OF_ENUM = "691"`
- `TOO_MANY_MUTATIONS = "692"`
- `AWS_ERROR = "693"`
- `ASYNC_LOAD_CYCLE = "694"`
- `ASYNC_LOAD_FAILED = "695"`
- `ASYNC_LOAD_CANCELED = "696"`
- `CANNOT_RESTORE_TO_NONENCRYPTED_DISK = "697"`
- `INVALID_REDIS_STORAGE_TYPE = "698"`
- `INVALID_REDIS_TABLE_STRUCTURE = "699"`
- `USER_SESSION_LIMIT_EXCEEDED = "700"`
- `CLUSTER_DOESNT_EXIST = "701"`
- `CLIENT_INFO_DOES_NOT_MATCH = "702"`
- `INVALID_IDENTIFIER = "703"`
- `QUERY_CACHE_USED_WITH_NONDETERMINISTIC_FUNCTIONS = "704"`
- `TABLE_NOT_EMPTY = "705"`
- `LIBSSH_ERROR = "706"`
- `GCP_ERROR = "707"`
- `ILLEGAL_STATISTICS = "708"`
- `CANNOT_GET_REPLICATED_DATABASE_SNAPSHOT = "709"`
- `FAULT_INJECTED = "710"`
- `FILECACHE_ACCESS_DENIED = "711"`
- `TOO_MANY_MATERIALIZED_VIEWS = "712"`
- `BROKEN_PROJECTION = "713"`
- `UNEXPECTED_CLUSTER = "714"`
- `CANNOT_DETECT_FORMAT = "715"`
- `CANNOT_FORGET_PARTITION = "716"`
- `EXPERIMENTAL_FEATURE_ERROR = "717"`
- `TOO_SLOW_PARSING = "718"`
- `QUERY_CACHE_USED_WITH_SYSTEM_TABLE = "719"`
- `USER_EXPIRED = "720"`
- `DEPRECATED_FUNCTION = "721"`
- `ASYNC_LOAD_WAIT_FAILED = "722"`
- `PARQUET_EXCEPTION = "723"`
- `TOO_MANY_TABLES = "724"`
- `TOO_MANY_DATABASES = "725"`
- `DISTRIBUTED_CACHE_ERROR = "900"`
- `CANNOT_USE_DISTRIBUTED_CACHE = "901"`
- `KEEPER_EXCEPTION = "999"`
- `POCO_EXCEPTION = "1000"`
- `STD_EXCEPTION = "1001"`
- `UNKNOWN_EXCEPTION = "1002"`



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides
Content:
---
title: "API Endpoints guides · Tinybird Docs"
theme-color: "#171612"
description: "Guides for using Tinybird API Endpoints."
---




# API Endpoints guides [¶](https://www.tinybird.co/docs/about:blank#api-endpoints-guides)

Tinybird API Endpoints make it easy to use your data in applications. These guides show you how to integrate API Endpoints into different tools and frameworks:

- Build interactive frontends by consuming APIs in Next.js with JWT authentication.
- Explore and visualize data by using API Endpoints in Jupyter notebooks.
- Create monitoring dashboards by integrating with Grafana.
- Export metrics in Prometheus format for observability tools.

Each guide provides step-by-step instructions and code examples to help you get started quickly. The guides cover common integration patterns like authentication, data filtering, and visualization.

The following guides are available:



---

URL: https://www.tinybird.co/docs/get-started/quick-start/vector-search-recommendation
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Build a content recommendation API using vector search · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to compute embeddings in Python and use vector search SQL functions in Tinybird to build a content recommendation API."
---




# Build a content recommendation API using vector search [¶](https://www.tinybird.co/docs/about:blank#build-a-content-recommendation-api-using-vector-search)

Read on to learn how to calculate vector embeddings using HuggingFace models and use Tinybird to perform vector search to find similar content based on vector distances.

[GitHub Repository](https://github.com/tinybirdco/demo_vector_search_recommendation/tree/main)

<-figure->
![Tinybird blog related posts uses vector search recommendation algorithm.](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-vector-search-recommendation-1.png&w=3840&q=75)


In this tutorial, you learn how to:

1. Use Python to fetch content from an RSS feed.
2. Calculate vector embeddings on long form content (blog posts) using SentenceTransformers in Python.
3. Post vector embeddings to a Tinybird Data Source using the Tinybird Events API.
4. Write a dynamic SQL query to calculate the closest content matches to a given blog post based on vector distances.
5. Publish your query as an API and integrate it into a frontend application.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you need the following:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. An empty Tinybird Workspace
3. Python 3.8 or higher

This tutorial doesn't include a frontend. An example snippet is provided to show how you can integrate the published API into a React frontend.

1
## Setup [¶](https://www.tinybird.co/docs/about:blank#setup)

Clone the `demo_vector_search_recommendation` repo.

Authenticate the Tinybird CLI using your user admin token from your Tinybird Workspace:

cd tinybird
tb auth --token $USER_ADMIN_TOKEN 2
## Fetch content and calculate embeddings [¶](https://www.tinybird.co/docs/about:blank#fetch-content-and-calculate-embeddings)

This tutorial uses the [Tinybird Blog RSS feed](https://www.tinybird.co/blog-posts/rss.xml) to fetch blog posts. You can use any `rss.xml` feed to fetch blog posts and calculate embeddings from their content.

You can fetch and parse the RSS feed using the `feedparser` library in Python, get a list of posts, and then fetch each post and parse the content with the `BeautifulSoup` library.

Once you've fetched each post, you can calculate an embedding using the HuggingFace `sentence_transformers` library. This demo uses the `all-MiniLM-L6-v2` model, which maps sentences and paragraphs to a 384 dimensional dense vector space:

from bs4 import BeautifulSoup
from sentence_transformers import SentenceTransformer

import datetime
import feedparser
import requests
import json

timestamp = datetime.datetime.now().isoformat()
url = "https://www.tinybird.co/blog-posts/rss.xml" # Update to your preferred RSS feed
feed = feedparser.parse(url)

model = SentenceTransformer("all-MiniLM-L6-v2")

posts = []
for entry in feed.entries:
    doc = BeautifulSoup(requests.get(entry.link).content, features="html.parser")
    if (content := doc.find(id="content")):
        embedding = model.encode([content.get_text()])
        posts.append(json.dumps({
            "timestamp": timestamp,
            "title": entry.title,
            "url": entry.link,
            "embedding": embedding.mean(axis=0).tolist()
        })) 3
## Post content metadata and embeddings to Tinybird [¶](https://www.tinybird.co/docs/about:blank#post-content-metadata-and-embeddings-to-tinybird)

After calculating the embeddings, you can push them along with the content metadata to Tinybird using the [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api).

First, set up some environment variables for your Tinybird host and token with `DATASOURCES:WRITE` scope:

export TB_HOST=your_tinybird_host
export TB_TOKEN=your_tinybird_token Next, set up a Tinybird Data Source to receive your data. In the `tinybird/datasources` folder of the repository, find a `posts.datasource` file that looks like this:

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `title` String `json:$.title`,
    `url` String `json:$.url`,
    `embedding` Array(Float32) `json:$.embedding[:]`

ENGINE ReplacingMergeTree
ENGINE_PARTITION_KEY ""
ENGINE_SORTING_KEY title, url
ENGINE_VER timestamp This Data Source receives the updated post metadata and calculated embeddings and deduplicates based on the most up to date data retrieval. The `ReplacingMergeTree` is used to deduplicate, relying on the `ENGINE_VER` setting, which in this case is set to the `timestamp` column. This tells the engine that the versioning of each entry is based on the `timestamp` column, and only the entry with the latest timestamp is kept in the Data Source.

The Data Source has the `title` column as its primary sorting key, because you filter by title to retrieve the embedding for the current post. Having `title` as the primary sorting key makes that filter more performant.

Push this Data Source to Tinybird:

cd tinybird
tb push datasources/posts.datasource Then, you can use a Python script to push the post metadata and embeddings to the Data Source using the Events API:

import os
import requests

TB_APPEND_TOKEN=os.getenv("TB_APPEND_TOKEN")
TB_HOST=os.getenv("TB_HOST")

def send_posts(posts):
    params = {
        "name": "posts",
        "token": TB_APPEND_TOKEN
    }
    data = "\n".join(posts) # ndjson
    r = requests.post(f"{TB_HOST}/v0/events", params=params, data=data)
    print(r.status_code)

send_posts(posts) To keep embeddings up to date, you should retrieve new content on a schedule and push it to Tinybird. In the repository, you can find a GitHub Action called [tinybird_recommendations.yml](https://github.com/tinybirdco/demo_vector_search_recommendation/blob/main/.github/workflows/tinybird_recommendations.yml) that fetches new content from the Tinybird blog every 12 hours and pushes it to Tinybird. The Tinybird Data Source in this project uses a ReplacingMergeTree to deduplicate blog post metadata and embeddings as new data arrives.

4
## Calculate distances in SQL using Tinybird Pipes [¶](https://www.tinybird.co/docs/about:blank#calculate-distances-in-sql-using-tinybird-pipes)

If you've completed the previous steps, you should have a `posts` Data Source in your Tinybird Workspace containing the last fetched timestamp, title, URL, and embedding for each blog post fetched from your RSS feed.

You can verify that you have data from the Tinybird CLI with:

tb sql 'SELECT * FROM posts' This tutorial includes a single-node SQL Pipe to calculate the vector distance of each post to specific post supplied as a query parameter. The Pipe config is contained in the `similar_posts.pipe` file in the `tinybird/pipes` folder, and the SQL is copied in the following snippet for reference and explanation.

%
WITH
  (
    SELECT embedding
    FROM
    (
      SELECT 0 AS id, embedding
      FROM posts
      WHERE
          title = {{ String(title) }}
      ORDER BY timestamp DESC
      LIMIT 1
      UNION ALL
      SELECT 999 AS id, arrayWithConstant(384, 0.0) embedding
    )
      ORDER BY id
      LIMIT 1
    ) AS post_embedding
SELECT title, url, L2Distance(embedding, post_embedding) similarity
FROM posts FINAL
WHERE title <> {{ String(title) }}
ORDER BY similarity ASC
LIMIT 10 This query first fetches the embedding of the requested post, and returns an array of 0s in the event an embedding can't be fetched. It then calculates the Euclidean vector distance between each additional post and the specified post using the `L2Distance()` function, sorts them by ascending distance, and limits to the top 10 results.

You can push this Pipe to your Tinybird server with:

cd tinybird
tb push pipes/similar_posts.pipe When you push it, Tinybird automatically publishes it as a scalable, dynamic REST API Endpoint that accepts a `title` query parameter.

You can test your API Endpoint with a cURL. First, create an envvar with a token that has `PIPES:READ` scope for your Pipe. You can get this token from your Workspace UI or in the CLI with `tb token` [commands](https://www.tinybird.co/docs/docs/cli/command-ref#tb-token).

export TB_READ_TOKEN=your_read_token Then request your endpoint:

curl --compressed -H "Authorization: Bearer $TB_READ_TOKEN" https://api.tinybird.co/v0/pipes/similar_posts.json?title='Some blog post title' A JSON object appears containing the 10 most similar posts to the post whose title you supplied in the request.

5
## Integrate into the frontend [¶](https://www.tinybird.co/docs/about:blank#integrate-into-the-frontend)

Integrating your vector search API into the frontend is relatively straightforward. Here's an example implementation:

export async function getRelatedPosts(title: string) {
  const recommendationsUrl = `${host}/v0/pipes/similar_posts.json?token=${token}&title=${title}`;
  const recommendationsResponse = await fetch(recommendationsUrl).then(
    function (response) {
      return response.json();
    }
  );
  if (!recommendationsResponse.data) return;
  return Promise.all(
    recommendationsResponse.data.map(async ({ url }) => {
      const slug = url.split("/").pop();
      return await getPost(slug);
    })
  ).then((data) => data.filter(Boolean));
} 6
## See it in action [¶](https://www.tinybird.co/docs/about:blank#see-it-in-action)

You can see how this looks by checking out any blog post in the [Tinybird Blog](https://www.tinybird.co/blog) . At the bottom of each post, you can find a Related Posts section that's powered by a real Tinybird API

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read more about[  vector search](https://www.tinybird.co/docs/docs/use-cases/vector-search)   and[  content recommendation](https://www.tinybird.co/docs/docs/use-cases/content-recommendation)   use cases.
- Join the[  Tinybird Slack Community](https://www.tinybird.co/community)   for additional support.



---

URL: https://www.tinybird.co/docs/get-started/quick-start/user-facing-web-analytics
Last update: 2025-01-28T15:30:39.000Z
Content:
---
title: "Build a user-facing web analytics dashboard · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to build a user-facing web analytics dashboard using Tinybird for real-time, user-facing analytics."
---




# Build a user-facing web analytics dashboard [¶](https://www.tinybird.co/docs/about:blank#build-a-user-facing-web-analytics-dashboard)

Read on to learnhow to build a user-facing web analytics dashboard. Use Tinybird to capture web clickstream events, process the data in real-time, and expose metrics as APIs. Then, deploy a Next.js app to visualize your metrics.

[GitHub Repository](https://github.com/tinybirdco/demo-user-facing-web-analytics) The guide is divided into the following steps:

1. Stream unstructured events data to Tinybird with the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .
2. Parse those events with a global SQL node that you can reuse in all your subsequent[  Tinybird Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes)  .
3. Build performant queries to calculate user-facing analytics metrics.
4. Optimize query performance with[  Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views)  .
5. Publish your metrics as[  API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints)   and integrate them into a user-facing Next.js app.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you need the following:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. An empty Tinybird Workspace
3. Node.js 20.11 or higher
4. Python 3.8 or higher

This tutorial includes a [Next.js](https://nextjs.org/) app for frontend visualization. For more information about how the Next.js app is designed and deployed, read the [repository README](https://github.com/tinybirdco/demo-user-facing-web-analytics/tree/main/app/README.md).

The steps in this tutorial are completed using the Tinybird Command Line Interface (CLI). If you're not familiar with it, [read the CLI docs](https://www.tinybird.co/docs/docs/cli/install) . You can copy and paste every code snippet and command in this tutorial.

1
## Create a Tinybird Data Source to store your events [¶](https://www.tinybird.co/docs/about:blank#create-a-tinybird-data-source-to-store-your-events)

First, create a Tinybird Data Source to store your web clickstream events. Create a new directory called `tinybird` in your project folder and install the Tinybird CLI:

##### Install the Tinybird CLI

mkdir tinybird
cd tinybird
python -m venv .venv
source .venv/bin/activate
pip install tinybird-cli Copy the user admin Token and authenticate the CLI:

##### Authenticate the Tinybird CLI

tb auth --token  Initialize an empty Tinybird project and navigate to the `/datasources` directory, then create a new file called `analytics_events.datasource`:

##### Create a Data Source

tb init
cd datasources
touch analytics_events.datasource Open the file in your preferred code editor and paste the following contents:

##### analytics_events.datasource

DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE MergeTree
ENGINE_PARTITION_KEY toYYYYMM(timestamp)
ENGINE_SORTING_KEY timestamp
ENGINE_TTL timestamp + toIntervalDay(60) If you pass a non-existent Data Source name to the Events API, Tinybird automatically creates a new Data Source of that name with an [inferred schema](https://www.tinybird.co/docs/docs/get-data-in#get-started) . By creating the Data Source ahead of time in this file, you have more control over the schema definition, including column types and sorting keys. For more information about creating Tinybird Data Sources, see [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources).

In the `/tinybird` directory, save and push the file to Tinybird:

##### Push the Data Source to Tinybird

cd ..
tb push datasources/analytics_events.datasource Confirm that you have a new Data Source:

tb datasource ls You should see `analytics_events` in the result. Congrats, you have a Tinybird Data Source!

2
## Stream mock data to your Data Source [¶](https://www.tinybird.co/docs/about:blank#stream-mock-data-to-your-data-source)

This tutorial uses [Mockingbird](https://mockingbird.tinybird.co/docs) , an open source mock data stream generator, to stream mock web clickstream events to your Data Source. Mockingbird generates a JSON payload based on a predefined schema and posts it to the [Tinybird Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) , which then writes the data to your Data Source. You can explore the [Mockingbird web UI](https://mockingbird.tinybird.co/) , or follow the steps to complete the same actions using the Mockingbird CLI.

In a separate terminal window, install the Mockingbird CLI:

##### Install Mockingbird

npm install -g @tinybirdco/mockingbird-cli Run the following command to stream 50,000 mock web clickstream events to your `analytics_events` Data Source at 50 events per second through the Events API. This command uses the predefined Web Analytics template schema to generate mock web clickstream events.

Copy your User Admin Token to the clipboard with `tb token copy dashboard` , and use it in the following command. Change the `endpoint` argument depending on your Workspace region if required:

##### Stream to Tinybird with a template

mockingbird-cli tinybird --template "Web Analytics template" --eps 50 --limit 50000 --datasource analytics_events --token  --endpoint gcp_europe_west3 Confirm that events are written to the `analytics_events` Data Source by running the following command a few times:

tb sql 'select count() from analytics_events' You should see the count incrementing up by 50 every second or so. Congratulations, you're ready to start processing your events data!

3
## Parse the raw JSON events [¶](https://www.tinybird.co/docs/about:blank#parse-the-raw-json-events)

The `analytics_events` Data Source has a `payload` column which stores a string of JSON data. To begin building your analytics metrics, you need to parse this JSON data using a Tinybird [Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes).

When you're dealing with unstructured data that's likely to change in the future, retain the unstructured data as a JSON string in a single column. This gives you flexibility to change your upstream producers without breaking ingestion. You can then parse and materialize this data downstream.

Navigate to the `/pipes` directory and create a new file called `analytics_hits.pipe`:

##### Create a Pipe

touch analytics_hits.pipe Open the file and paste the following contents:

##### analytics_hits.pipe

DESCRIPTION >
    Parsed `page_hit` events, implementing `browser` and `device` detection logic.

TOKEN "dashboard" READ

NODE parsed_hits
DESCRIPTION >
    Parse raw page_hit events

SQL >
    SELECT
        timestamp,
        action,
        version,
        coalesce(session_id, '0') as session_id,
        JSONExtractString(payload, 'locale') as locale,
        JSONExtractString(payload, 'location') as location,
        JSONExtractString(payload, 'referrer') as referrer,
        JSONExtractString(payload, 'pathname') as pathname,
        JSONExtractString(payload, 'href') as href,
        lower(JSONExtractString(payload, 'user-agent')) as user_agent
    FROM analytics_events
    where action = 'page_hit'

NODE endpoint
SQL >
    SELECT
        timestamp,
        action,
        version,
        session_id,
        location,
        referrer,
        pathname,
        href,
        case
            when match(user_agent, 'wget|ahrefsbot|curl|urllib|bitdiscovery|\+https://|googlebot')
            then 'bot'
            when match(user_agent, 'android')
            then 'mobile-android'
            when match(user_agent, 'ipad|iphone|ipod')
            then 'mobile-ios'
            else 'desktop'
        END as device,
        case
            when match(user_agent, 'firefox')
            then 'firefox'
            when match(user_agent, 'chrome|crios')
            then 'chrome'
            when match(user_agent, 'opera')
            then 'opera'
            when match(user_agent, 'msie|trident')
            then 'ie'
            when match(user_agent, 'iphone|ipad|safari')
            then 'safari'
            else 'Unknown'
        END as browser
    FROM parsed_hits This Pipe contains two [nodes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#nodes) . The first node, called `parsed_hits` , extracts relevant information from the JSON `payload` using the `JSONExtractString()` function and filters to only include `page_hit` actions.

The second node, called `endpoint` , selects from the `parsed_hits` node and further parses the `user_agent` to get the `device` and `browser` for each event.

Additionally, this code gives the Pipe a description, and creates a [Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) called `dashboard` with `READ` scope for this Pipe.

Navigate back up to the `/tinybird` directory and push the Pipe to Tinybird:

##### Push the Pipe to Tinybird

tb push pipes/analytics_hits.pipe When you push a Pipe file, Tinybird automatically publishes the last node as an API Endpoint unless you specify the Pipe as something else, so it's best practice to call your final node "endpoint". You can unpublish an API Endpoint at any time using `tb pipe unpublish `.

You now have a public REST API that returns the results of the `analytics_hits` Pipe. Get your Admin Token again with `tb token copy dashboard` and test your API with the command:

curl "https://api.tinybird.co/v0/pipes/analytics_hits.json?token=" You should see a JSON response that looks something like this:

##### Example API response

{
    "meta":
    [
        {
            "name": "timestamp",
            "type": "DateTime"
        },
        {
            "name": "action",
            "type": "LowCardinality(String)"
        },
        {
            "name": "version",
            "type": "LowCardinality(String)"
        },
        {
            "name": "session_id",
            "type": "String"
        },
        {
            "name": "location",
            "type": "String"
        },
        {
            "name": "referrer",
            "type": "String"
        },
        {
            "name": "pathname",
            "type": "String"
        },
        {
            "name": "href",
            "type": "String"
        },
        {
            "name": "device",
            "type": "String"
        },
        {
            "name": "browser",
            "type": "String"
        }
    ],

    "data":
    [
        {
            "timestamp": "2024-04-24 18:24:21",
            "action": "page_hit",
            "version": "1",
            "session_id": "713355c6-6b98-4c7a-82a9-e19a7ace81fe",
            "location": "",
            "referrer": "https:\/\/www.kike.io",
            "pathname": "\/blog-posts\/data-market-whitebox-replaces-4-data-stack-tools-with-tinybird",
            "href": "https:\/\/www.tinybird.co\/blog-posts\/data-market-whitebox-replaces-4-data-stack-tools-with-tinybird",
            "device": "bot",
            "browser": "chrome"
        },
        ...
    ]

    "rows": 150,

    "statistics":
    {
        "elapsed": 0.006203411,
        "rows_read": 150,
        "bytes_read": 53609
    }
} 4
## Calculate aggregates for pageviews, sessions, and sources [¶](https://www.tinybird.co/docs/about:blank#calculate-aggregates-for-pageviews-sessions-and-sources)

Next, create three [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) to store aggregates for the following:

1. pageviews
2. sessions
3. sources

Later on, you query from the Materialized Views that you're creating here.

From the `/datasources` directory in the Tinybird project, create three new Data Source files:

touch analytics_pages_mv.datasource analytics_sessions_mv.datasource analytics_sources_mv.datasource Open the `analytics_pages_mv.datasource` file and paste in the following contents:

##### analytics_pages_mv.datasource

SCHEMA >
    `date` Date,
    `device` String,
    `browser` String,
    `location` String,
    `pathname` String,
    `visits` AggregateFunction(uniq, String),
    `hits` AggregateFunction(count)

ENGINE AggregatingMergeTree
ENGINE_PARTITION_KEY toYYYYMM(date)
ENGINE_SORTING_KEY date, device, browser, location, pathname Do the same for `analytics_sessions_mv.datasource` and `analytics_sources_mv.datasource` , copying the code from the [GitHub repository](https://github.com/tinybirdco/demo-user-facing-web-analytics/tree/main/tinybird/datasources) for this tutorial.

Next, create three Pipes that calculate the aggregates and store the data in the Materialized View Data Sources you've created. From the `/pipes` directory, create three new Pipe files:

touch analytics_pages.pipe analytics_sessions.pipe analytics_sources.pipe Open `analytics_pages.pipe` and paste the following:

##### analytics_pages.pipe

NODE analytics_pages_1
DESCRIPTION >
    Aggregate by pathname and calculate session and hits

SQL >
    SELECT
        toDate(timestamp) AS date,
        device,
        browser,
        location,
        pathname,
        uniqState(session_id) AS visits,
        countState() AS hits
    FROM analytics_hits
    GROUP BY date, device, browser, location, pathname

TYPE MATERIALIZED
DATASOURCE analytics_pages_mv This code calculates aggregates for page views, and designates the Pipe as a Materialized View with `analytics_pages_mv` as the target Data Source. Do this for the remaining two Pipes, copying the code from the [GitHub repository](https://github.com/tinybirdco/demo-user-facing-web-analytics/tree/main/tinybird/pipes).

Back in the `/tinybird` directory, push these new Pipes and Data Sources to Tinybird. This populates the Materialized Views with your Mockingbird data:

##### Push to Tinybird

tb push pipes --push-deps --populate Now, as new events arrive in the `analytics_events` Data Source, these Pipes process the data and update the aggregate states in your Materialized Views as new data arrives.

5
## Generate session count trend for the last 30 minutes [¶](https://www.tinybird.co/docs/about:blank#generate-session-count-trend-for-the-last-30-minutes)

The first Pipe you create, called `trend` , calculates the number of sessions over the last 30 minutes, grouped by 1 minute intervals.

From the `/pipes` directory, create a file called `trend.pipe`:

##### Create trend.pipe

touch trend.pipe Open this file and paste the following:

##### trend.pipe

DESCRIPTION >
    Visits trend over time for the last 30 minutes, filling in the blanks.

TOKEN "dashboard" READ

NODE timeseries
DESCRIPTION >
    Generate a timeseries for the last 30 minutes, so we call fill empty data points

SQL >
    with (now() - interval 30 minute) as start
    select addMinutes(toStartOfMinute(start), number) as t
    from (select arrayJoin(range(1, 31)) as number)

NODE hits
DESCRIPTION >
    Get last 30 minutes metrics grouped by minute

SQL >
    select toStartOfMinute(timestamp) as t, uniq(session_id) as visits
    from analytics_hits
    where timestamp >= (now() - interval 30 minute)
    group by toStartOfMinute(timestamp)
    order by toStartOfMinute(timestamp)

NODE endpoint
DESCRIPTION >
    Join and generate timeseries with metrics for the last 30 minutes

SQL >
    select a.t, b.visits from timeseries a left join hits b on a.t = b.t order by a.t This Pipe contains three nodes:

1. The first node, called `timeseries`   , generates a simple result set with 1-minute intervals for the last 30 minutes.
2. The second node, called `hits`   , calculates total sessions over the last 30 minutes, grouped by 1-minute intervals.
3. The third node, called `endpoint`   , performs a left join between the first two nodes, retaining all of the 1-minute intervals from the `timeseries`   node.

6
## Calculate the top pages visited [¶](https://www.tinybird.co/docs/about:blank#calculate-the-top-pages-visited)

Next, create a Pipe called `top_pages` to calculate a sorted list of the top pages visited over a specified time range. This Pipe queries the `analytics_pages_mv` Data Source you created in the prior steps, and it uses Tinybird's templating language to define [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) that you can use to dynamically select a time range and implement pagination in the response.

From the `/pipes` directory, create the `top_pages.pipe` file:

##### Create top_pages.pipe

touch top_pages.pipe Open the file and paste the following:

DESCRIPTION >
    Most visited pages for a given period.
    Accepts `date_from` and `date_to` date filter. Defaults to last 7 days.
    Also `skip` and `limit` parameters for pagination.

TOKEN "dashboard" READ

NODE endpoint
DESCRIPTION >
    Group by pagepath and calculate hits and visits

SQL >
    %
    select pathname, uniqMerge(visits) as visits, countMerge(hits) as hits
    from analytics_pages_mv
    where
        {% if defined(date_from) %}
            date
            >=
            {{ Date(date_from, description="Starting day for filtering a date range", required=False) }}
        {% else %} date >= timestampAdd(today(), interval -7 day)
        {% end %}
        {% if defined(date_to) %}
            and date
            <=
            {{ Date(date_to, description="Finishing day for filtering a date range", required=False) }}
        {% else %} and date <= today()
        {% end %}
    group by pathname
    order by visits desc
    limit {{ Int32(skip, 0) }},{{ Int32(limit, 50) }} Note the use of the `-Merge` modifiers on the end of the aggregate function. This modifier performs a final merge on the aggregate states in the Materialized View. [Read this Guide](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views/best-practices) for more details.

7
## Create the remaining API Endpoints [¶](https://www.tinybird.co/docs/about:blank#create-the-remaining-api-endpoints)

In the [GitHub repository](https://github.com/tinybirdco/demo-user-facing-web-analytics/tree/main/tinybird/pipes) , you can find five additional Pipe files that calculate other various user-facing metrics:

-  `kpis.pipe`
-  `top_browsers.pipe`
-  `top_devices.pipe`
-  `top_locations.pipe`
-  `top_sources.pipe`

Create those into your `/pipes` directory:

touch kpis.pipe top_browsers.pipe top_devices.pipe top_locations.pipe top_sources.pipe And copy the file contents from the GitHub examples into your files. Finally, in the `/tinybird` directory, push all these new Pipes to Tinybird:

tb push pipes You now have seven API Endpoints that you can integrate into your Next.js app to provide data to your dashboard components.

8
## Deploy the Next.js app [¶](https://www.tinybird.co/docs/about:blank#deploy-the-next-js-app)

You can deploy the accompanying Next.js app to Vercel by clicking this button:

[Deploy with Vercel](https://vercel.com/new/clone?repository-url=https%253A%252F%252Fgithub.com%252Ftinybirdco%252Fdemo-user-facing-web-analytics%252Ftree%252Fmain%252Fapp&env=NEXT_PUBLIC_TINYBIRD_AUTH_TOKEN,NEXT_PUBLIC_TINYBIRD_HOST,NEXT_PUBLIC_BASE_URL&envDescription=Tinybird%2520configuration&project-name=user-facing-web-analytics&repository-name=user-facing-web-analytics) First, select the Git provider where you can clone the Git repository:



<-figure->
![Select Git provider](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-user-facing-web-analytics-deploy-1.png&w=3840&q=75)


Next, set the following environment variables:

- `NEXT_PUBLIC_TINYBIRD_AUTH_TOKEN`   : your[  Tinybird Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens)
- `NEXT_PUBLIC_TINYBIRD_HOST`   : your Tinybird Region (e.g. `https://api.tinybird.co`   )
- `NEXT_PUBLIC_BASE_URL`   : The URL where you will publish your app (e.g. `https://my-analytics.com`   )



<-figure->
![Set env variables](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-user-facing-web-analytics-deploy-2.png&w=3840&q=75)


Select **Deploy** and you're done. Explore your dashboard and have a think about how you'd like to adapt or extend it in the future.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Understand today's real-time analytics landscape with[  Tinybird's definitive guide](https://www.tinybird.co/blog-posts/real-time-analytics-a-definitive-guide)  .
- Learn how to implement[  multi-tenant security](https://www.tinybird.co/blog-posts/multi-tenant-saas-options)   in your user-facing analytics.



---

URL: https://www.tinybird.co/docs/get-started/quick-start/tinybird-101-tutorial
Last update: 2025-01-21T08:14:33.000Z
Content:
---
title: "Tinybird 101 Tutorial · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird provides you with an easy way to ingest and query large amounts of data with low-latency, and instantly create API Endpoints to consume those queries. This makes it extremely easy to build fast and scalable applications that query your data; no backend needed!"
---




# Tinybird 101 [¶](https://www.tinybird.co/docs/about:blank#tinybird-101)

Tinybird provides you with a simple way to ingest and query large amounts of data with low latency, and instantly create API Endpoints to consume those queries. This means you can easily build fast and scalable applications that query your data.

## Example use case: ecommerce [¶](https://www.tinybird.co/docs/about:blank#example-use-case-ecommerce)

This walkthrough demonstrates how to build an API Endpoint that returns the top 10 most searched products in an ecommerce website. It follows the process of "ingest > query > publish".

1. First, you ingest a set of ecommerce events based on user actions, such as viewing an item, adding items to their cart, or going through the checkout. This data is available as a CSV file with 50 million rows.
2. Next, you write queries to filter, aggregate, and transform the data into the top 10 list.
3. Finally, you publish that top 10 result as an HTTP Tinybird API Endpoint.

## Your first Workspace [¶](https://www.tinybird.co/docs/about:blank#your-first-workspace)

After [creating your account](https://www.tinybird.co/signup) , select a region, and name your Workspace. You can call the Workspace whatever you want. Leave the template menu blank.

1
## Create a Data Source [¶](https://www.tinybird.co/docs/about:blank#create-a-data-source)

Tinybird can import data from many different sources. Start with a CSV file that Tinybird has posted online for you.

In your Workspace, find the **Data Sources** section and select the **+** icon to add a new Data Source.

In the dialog that opens, select the **Remote URL** connector. Make sure that `csv` is selected, then paste the following URL into the text box:

https://storage.googleapis.com/tinybird-assets/datasets/guides/events_50M_1.csv Select **Add** and give the Data Source a name and description. Tinybird also shows you a preview of the schema and data.

Change the name to something more descriptive, for example `shopping_data`.

### Start the data import [¶](https://www.tinybird.co/docs/about:blank#start-the-data-import)

After setting the name of your first Data Source, select **Create Data Source** to start importing the data.

You've ingested your first data. Now you can move on to creating your first Pipe.

2
## Create a Pipe [¶](https://www.tinybird.co/docs/about:blank#create-a-pipe)

In Tinybird, SQL queries are written inside Pipes. One Pipe can be made up of many individual SQL queries called nodes. Each node is a single SQL SELECT statement. A node can query the output of another Nnde in the same Pipe. This means that you can break large queries down into a multiple smaller, more modular, queries and chain them together.

Add a new Pipe by selecting the **+** icon next to the Pipes category. This adds a new Pipe with an auto-generated default name. Select the name and description to change it. Call this Pipe `top_10_searched_products`.

### Filter the data [¶](https://www.tinybird.co/docs/about:blank#filter-the-data)

At the top of your new Pipe is the first node, which is prepopulated with a simple SELECT over the data in your Data Source. Before you start modifying the query in the node, select **Run** . Hitting **Run** executes the query in the Node, and shows a preview of the query result. You can execute any node in your Pipe to see the result.

In this Pipe, you want to create a list of the top 10 most searched products. If you take a look at the data, you might notice an `event` column, which describes what kind of event happened. This column has various values, including `view`, `search` , and `buy` . You are only interested in rows where the `event` is `search` , so modify the query to filter the rows.

Replace the node SQL with the following query:

SELECT * 
FROM shopping_data
WHERE event == 'search' Select **Run** again. The node is now applying a filter to the data, so you only see the rows of interest. Call this node `search_events`.

### Aggregate the data [¶](https://www.tinybird.co/docs/about:blank#aggregate-the-data)

Next, you want to work out how many times each individual product has been searched for. To do this, you need to count and aggregate by the product id. To keep your queries simpler, create a second node to do this aggregation.

Use the following query for the next node:

SELECT product_id, count() as total 
FROM search_events
GROUP BY product_id
ORDER BY total DESC Select **Run** again to see the results of the query. Call this node `aggregate_by_product_id`.

3
## Transform the result [¶](https://www.tinybird.co/docs/about:blank#transform-the-result)

Finally, create the last node that you are going to use to publish as a Tinybird API Endpoint and limit the results to the top 10 products.

Create a third node and use the following query:

SELECT product_id, total 
FROM aggregate_by_product_id
LIMIT 10 Follow the common convention to name this node `endpoint` . Select **Run** to preview the results.

4
## Publish and use your API Endpoint [¶](https://www.tinybird.co/docs/about:blank#publish-and-use-your-api-endpoint)

Tinybird API Endpoints are published directly from selected nodes. API Endpoints come with an extensive feature set, including support for dynamic query parameters and auto-generated docs complete with code samples.

To publish a node as an API Endpoint, select **Create API Endpoint** , then select **Create API Endpoint**.

### Test the API Endpoint [¶](https://www.tinybird.co/docs/about:blank#test-the-api-endpoint)

On your API overview page, scroll down to the **Sample** usage section, and copy the HTTP URL from the snippet box. Open this URL in a new tab in your browser.

Hitting the API Endpoint triggers your Pipe to execute, and you get a JSON formatted response with the results.

5
## Build Charts showing your data [¶](https://www.tinybird.co/docs/about:blank#build-charts-showing-your-data)

On your API overview page, select **Create Chart**.

You can also build Charts to embed in your own application. See the [Charts documentation](https://www.tinybird.co/docs/docs/publish/charts) for more.

6
## Celebrate [¶](https://www.tinybird.co/docs/about:blank#celebrate)

Congrats! You have finished creating your first API Endpoint in Tinybird!

You have imported 50 million events, built a variety of queries with latencies measured in milliseconds, and stood up an API Endpoint that can serve thousands of concurrent requests.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

Tinybird provides built-in connectors to easily ingest data from Kafka, Confluent Cloud, Big Query, and Amazon S3. If you want to stream data over HTTP, you can send data directly to Tinybird's [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) with no additional infrastructure.



---

URL: https://www.tinybird.co/docs/get-started/quick-start/leaderboard
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Build a real-time game leaderboard · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to build a real-time leaderboard using Tinybird."
---




# Build a real-time game leaderboard [¶](https://www.tinybird.co/docs/about:blank#build-a-real-time-game-leaderboard)

Read on to learn how to build a real-time leaderboard using Tinybird.

Leaderboards are a visual representation that ranks things by one or more attributes. For gaming use cases, commonly-displayed attributes include total points scored, high game scores, and number of games played.

Leaderboards are used for far more than games. For example, app developers use leaderboards to display miles biked, donations raised, documentation pages visited most often, and countless other examples - basically, anywhere there is some user attribute that can be ranked to compare results. This tutorial is a great starting point for building your own leaderboard.

[GitHub Repository](https://github.com/tinybirdco/demo-user-facing-leaderboard)

<-figure->
![A fast, fun leaderboard built on Tinybird](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fleaderboard-tutorial-2.png&w=3840&q=75)


The tutorial consists of the following steps:

1. Generate a mock game event stream that mimics a high-intensity Flappybird global tournament.
2. Post these mock events to your Tinybird Workspace using the Events API.
3. Transform (rank) this data using Tinybird Pipes and SQL.
4. Optimize your data handling with a Materialized View.
5. Publish the results as a Tinybird API Endpoint.
6. Generate a leaderboard that makes calls to your API Endpoint securely and directly from the browser.

Each time a leaderboard request is made, up-to-the-second results are returned for the leaderboard app to render. When embedded in the game, a `leaderboard` API Endpoint is requested when a game ends. The app makes requests on a specified interval and has a button for ad-hoc requests.

Game events consist of three values:

- `core`   - Generated when a point is scored.
- `game_over`   - Generated when a game ends.
- `purchase`   - Generated when a 'make-the-game-easier' coupon is redeemed.

Each event object has the following JSON structure:

##### Example JSON event object

{
    "session_id": "1f2c8bcf-8a5b-4eb1-90bf-8726e63d81b7",
    "name": "Marley",
    "timestamp": "2024-06-20T19:06:15.373Z",
    "type": "game_over",
    "event": "Mockingbird"
} Here's how it all fits together:



<-figure->
![Events and process of the leaderboard](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fleaderboard-tutorial-1.png&w=3840&q=75)


## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you need the following:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. An empty Tinybird Workspace
3. Node.js 20.11 or higher
4. Python 3.8 or higher

1
## Create a Tinybird Workspace [¶](https://www.tinybird.co/docs/about:blank#create-a-tinybird-workspace)

Go to ( [app.tinybird.co](https://app.tinybird.co/) ) and create an empty Tinybird Workspace called `tiny_leaderboard` in your preferred region.

2
## Create a Data Source for events [¶](https://www.tinybird.co/docs/about:blank#create-a-data-source-for-events)

You can create a Data Source based on a schema that you define or rely on the [Mockingbird](https://mockingbird.tinybird.co/docs) tool used to stream mock data to create the Data Source for you. While the Mockingbird method is faster, building your own Data Source gives you more control and introduces some fundamental concepts along the way.

In the Tinybird UI, add a new Data Source and use the `Write schema` option. In the schema editor, use the [following schema](https://github.com/tinybirdco/demo-user-facing-leaderboard/blob/main/tinybird/datasources/game_events.datasource):

##### Data Source schema

SCHEMA >
    `name` String `json:$.name`,
    `session_id` String `json:$.session_id`,
    `timestamp` DateTime64(3) `json:$.timestamp`,
    `type` LowCardinality(String) `json:$.type`,
    `event` String `json:$.event`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(timestamp)"
ENGINE_SORTING_KEY "event, name, timestamp" Name the Data Source `game_events` and select **Create Data Source**.

This schema definition shows how the incoming JSON events are parsed and assigned to each of schema fields. The definition also defines [database table ‘engine’ details](https://www.tinybird.co/docs/docs/get-data-in/data-sources#supported-engines-settings) . Tinybird projects are made of Data Source and Pipe definition files like this example, and they can be managed like any other code project using Git.

3
## Create a mock data stream [¶](https://www.tinybird.co/docs/about:blank#create-a-mock-data-stream)

In a real-life scenario, you'd stream your game events into the `game_events` Data Source. For this tutorial, you use [Mockingbird](https://mockingbird.tinybird.co/docs) , an open source mock data stream generator, to stream mock events instead. Mockingbird generates a JSON payload based on a predefined schema and posts it to the [Tinybird Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) , which then writes the data to your Data Source.

Use [this Mockingbird link](https://mockingbird.tinybird.co/?host=eu_gcp&datasource=game_events&eps=10&withLimit=on&generator=Tinybird&endpoint=eu_gcp&limit=-1&generatorName=Tinybird&template=Flappybird&schema=Preset) to generate fake data for the `game_events` Data Source. Using the link provides a preconfigured schema.

Enter your Workspace admin Token and select the Host region that matches your Workspace region. Select **Save** , then scroll down and select **Start Generating**.



Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

In the Tinybird UI, confirm that the `game_events` Data Source is successfully receiving data.

Leaderboards typically leverage a concise data schema with just a user/item name, the ranked attribute, and a timestamp. This tutorial is based on this schema:

- `name`   String
- `session_id`   String
- `timestamp`   DateTime64(3)
- `type`   LowCardinality(String)
- `event`   String

Ranking algorithms can be based on a single score, time-based metrics, or weighted combinations of factors.

4
## Transform and publish your data [¶](https://www.tinybird.co/docs/about:blank#transform-and-publish-your-data)

Your Data Source is collecting events, so now it's time to create some [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) . Pipes are made up of chained, reusable SQL [nodes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#nodes) and form the logic that ranks the results.

Start by creating a `leaderboard` Pipe with two nodes. The first node returns all 'score' events. The second node takes those results and counts these events by player and session (which defines a single game), and returns the top 10 results.

In the Tinybird UI, create a new Pipe called `leaderboard` . Paste in the following SQL and rename the first node `get_all_scores`:

##### get_all_scores Node

%
SELECT name AS player_id, timestamp, session_id, event
FROM game_events
WHERE
    type = 'score'
    AND event == {{ String(event_param, 'Mockingbird', description="Event to filter on") }} This query returns all events where the type is `score`.

This node creates a query parameter named `event_param` using the [Tinybird templating syntax](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) .  This instance of Flappybird supports an ‘event’ attribute that supports organizing players, games, and events into separate groups. As shown previously, incoming Mockingbird game events have a `"event": "Mockingbird"` attribute.

Select **Run** and add a new node underneath, called `endpoint` . Paste in:

##### endpoint Node

SELECT player_id, 
  session_id, 
  event,
  count() AS score
FROM get_all_scores
GROUP BY player_id, session_id, event
ORDER BY score DESC
LIMIT 10 Select **Run** , then select **Create API Endpoint** . Your data is now ranked, published, and available for consuming.

5
## Optimize with Materialized Views [¶](https://www.tinybird.co/docs/about:blank#optimize-with-materialized-views)

Before you run the frontend for your leaderboard, there are a few optimizations to make. Even with small datasets, it's a great habit to get into.

[Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) are updated as data is ingested, and create intermediate states that are merged with already-processed data. The Materialized View (MV) continuously re-evaluates queries as new events are inserted, reducing both latency and processed-data-per-query. In this case, the MV you create precalculates the top scores, and merges those with recently-received events. This significantly improves query performance by reducing the amount of data that needs to be processed for each leaderboard request.

To create a new Materialized View, begin by adding a new Pipe and call it `user_stats_mv` . Then paste the following SQL into the first Node:

SELECT
    event,
    name AS player_id,
    session_id,
    countIfState(type = 'score') AS scores,
    countIfState(type = 'game_over') AS games,
    countIfState(type = 'purchase') AS purchases,
    minState(timestamp) AS start_ts,
    maxState(timestamp) AS end_ts
FROM games_events
GROUP BY
    event,
    player_id,
    session_id This query relies on the `countIfState` function, which includes the `-State` operator to maintain immediate states containing recent data. When triggered by a `-Merge` operator, these intermediate states are combined with the precalculated data. The `countIfState` function is used to maintain counts of each type of game event.

Name this node `populate_mv` , then [publish it as a Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) . Name your Materialized View `user_stats`.

You now have a new Data Source called `user_stats` , which is a Materialized View that is continuously updated with the latest game events. The `-State` modifier that maintains intermediate states as new data arrives is paired with a `-Merge` modifier in Pipes that pull from the `user_stats` Data Source.

6
## Update leaderboard Pipe [¶](https://www.tinybird.co/docs/about:blank#update-leaderboard-pipe)

Now that `user_stats` is available, rebuild the `leaderboard` Pipe to take advantage of this more efficient Data Source. This step helps prepare your leaderboard feature to handle massive amounts of game events while serving requests to thousands of users.

The updated leaderboard Pipe consists of three nodes:

- `rank_games`   - Applies the countMerge(scores) function to get the current total from the user_stats Data Source.
- `last_game`   - Retrieves the score from the player's most recent game and determines the player's rank.
- `endpoint`   - Combines the results of these two nodes and ranks by score.

The `last_game` node introduces the user-facing aspect of the leaderboard. This node retrieves a specific user's data and blends it into the leaderboard results.

To get started, update the `leaderboard` Pipe to use the `user_stats` Materialized View.

Return to the `leaderboard` Pipe and un-publish it.

Now, change the name of the first node to `rank_games` and update the SQL to:

##### rank_games Node

%
SELECT
    ROW_NUMBER() OVER (ORDER BY total_score DESC, t) AS rank,
    player_id,
    session_id,
    countMerge(scores) AS total_score,
    maxMerge(end_ts) AS t
FROM user_stats
GROUP BY player_id, session_id
ORDER BY rank A few things to notice:

1. The `rank_games`   node now uses the `user_stats`   Materialized View instead of the `game_events`   Data Source.
2. The use of the `countMerge(scores)`   function. The `-Merge`   operator triggers the MV-based `user_stats`   Data Source to combine any intermediate states with the pre-calculated data and return the results.
3. The use of the `ROW_NUMBER()`   window function that returns a ranking of top scores. These rankings are based on the merged scores (aliased as `total_scores`   ) retrieved from the `user_stats`   Data Source.

Next, change the name of the second node to `last_game` and update the SQL to:

##### last_game Node

%
SELECT
    argMax(rank, t) AS rank,
    player_id,
    argMax(session_id, t) AS session_id,
    argMax(total_score, t) AS total_score
FROM rank_games
WHERE
    player_id = {{ String(player_id, 'Jim', description="Player to filter on", required=True) }}
GROUP BY player_id This query returns the highest rank of a specified player and introduces a `player_id` query parameter.

To combine these results, add a new node called `endpoint` and paste the following SQL:

##### endpoint Node

SELECT *
FROM
    (
        SELECT rank, player_id, session_id, total_score
        FROM rank_games
        WHERE (player_id, session_id) NOT IN (SELECT player_id, session_id FROM last_game)
        LIMIT 10
        UNION ALL
        SELECT rank, player_id, session_id, total_score
        FROM last_game
    )
ORDER BY rank ASC This query applies the `UNION ALL` statement to combine the two result sets. The selected attribute data types must match to be combined.

This completes the `leaderboard` Pipe. Publish it as an API Endpoint.

Now that the final version of the 'leaderboard' Endpoint has been published, create one last Pipe in the UI. This one gets the overall stats for the leaderboard, the number of players and completed games. Name the Pipe `get_stats` and create a single node named `endpoint`:

##### endpoint node in the get_stats Pipe

WITH player_count AS (
  SELECT COUNT(DISTINCT player_id) AS players
  FROM user_stats
),
game_count AS (
  SELECT COUNT(*) AS games
  FROM game_events
  WHERE type == 'game_over'
)
SELECT players, games
FROM player_count, game_count Publish this node as an API Endpoint. You're ready to get it all running!

7
## Run your app [¶](https://www.tinybird.co/docs/about:blank#run-your-app)

Clone the `demo-user-facing-leaderboard` repo locally.

Install the app dependencies by running this command from the `app` dir of the cloned repo:

npm install
### Add your Tinybird settings as environment variables [¶](https://www.tinybird.co/docs/about:blank#add-your-tinybird-settings-as-environment-variables)

Create a new `.env.local` file:

touch .env.local Copy your Tinybird Admin Token, Workspace UUID, obtained from **Workspace** > **Settings** > **Advanced settings** >** `...`** , and API host URL from your Tinybird Workspace into the new `.env.local`:

TINYBIRD_SIGNING_TOKEN="YOUR SIGNING TOKEN" # Use your Admin Token as the signing token
TINYBIRD_WORKSPACE="YOUR WORKSPACE ID" # The UUID of your Workspace
NEXT_PUBLIC_TINYBIRD_HOST="YOUR TINYBIRD API HOST e.g. https://api.tinybird.co" # Your regional API host
### Run your app [¶](https://www.tinybird.co/docs/about:blank#run-your-app)

Run your app locally and navigate to `http://localhost:3000`:

npm run dev You now have an optimized gaming leaderboard ingesting real-time data! Have a think about how you'd adapt or extend it for your own use case.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read the in-depth blog post on[  building a real-time leaderboard](https://www.tinybird.co/blog-posts/building-real-time-leaderboards-with-tinybird)  .
- Understand today's real-time analytics landscape with[  Tinybird's definitive guide](https://www.tinybird.co/blog-posts/real-time-analytics-a-definitive-guide)  .
- Learn how to implement[  multi-tenant security](https://www.tinybird.co/blog-posts/multi-tenant-saas-options)   in your user-facing analytics.



---

URL: https://www.tinybird.co/docs/get-started/quick-start/core-concepts
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Core concepts · Tinybird Docs"
theme-color: "#171612"
description: "Find Tinybird-related core terms and their definitions."
---




# Core concepts [¶](https://www.tinybird.co/docs/about:blank#core-concepts)

Familiarize yourself with Tinybird's core concepts and terminology to get a better understanding of how Tinybird works and how you can make the most of its features.

## Workspaces [¶](https://www.tinybird.co/docs/about:blank#workspaces)

Workspaces help you organize and collaborate on your Tinybird projects. You can have more than one Workspace.

A Workspace contains the project resources, data, and state. You can share resources, such as Pipes or Data Sources, between Workspaces. You can also invite users to your Workspaces and define their role and permissions.

A typical usage of Workspaces is to provide a team or project with a space to work in.

See [Workspaces](https://www.tinybird.co/docs/docs/get-started/administration/workspaces) for more information.

## Data Sources [¶](https://www.tinybird.co/docs/about:blank#data-sources)

Data Sources are how you ingest and store data in Tinybird.

All your data lives inside a Data Source, and you write SQL queries against Data Sources. You can bulk upload or stream data into a Data Source, and they support several different incoming data formats, such as CSV, JSON, and Parquet.

See [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources) for more information.

## Pipes [¶](https://www.tinybird.co/docs/about:blank#pipes)

Pipes are how you write SQL logic in Tinybird.

Pipes are a collection of one or more SQL queries chained together and compiled into a single query. Pipes let you break larger queries down into smaller queries that are easier to read. You can publish Pipes as API Endpoints, copy them, and create Materialized Views.

See [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) for more information.

## Nodes [¶](https://www.tinybird.co/docs/about:blank#nodes)

A node is a single SQL `SELECT` statement that selects data from a Data Source or another node or API Endpoint. Nodes live within [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes).

## API Endpoints [¶](https://www.tinybird.co/docs/about:blank#api-endpoints)

You can build your SQL logic inside a Pipe and then publish the result of your query as an HTTP API Endpoint.

See [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints) for more information.

## Charts [¶](https://www.tinybird.co/docs/about:blank#charts)

Charts visualize your data. You can create and publish Charts in Tinybird from your published API Endpoints.

See [Charts](https://www.tinybird.co/docs/docs/publish/charts) for more information.

## Tokens [¶](https://www.tinybird.co/docs/about:blank#tokens)

Tokens authorize requests. Tokens can be static for back-end integrations, or custom JWTs for front-end applications.

See [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) for more information.

## Branches [¶](https://www.tinybird.co/docs/about:blank#branches)

Branches let you create a copy of your Workspace where you can make changes, run tests, and develop new features. You can then merge the changes back into the original Workspace.

See [Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches) for more information.

## CLI [¶](https://www.tinybird.co/docs/about:blank#cli)

Use the Tinybird command line interface (CLI) to interact with Tinybird from the terminal. You can install it on your local machine or embed it into your CI/CD pipelines.

See [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/quick-start) for more information.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Understand Tinybird's[  underlying architecture](https://www.tinybird.co/docs/docs/get-started/architecture)  .
- Check out the[  Tinybird Quick Start](https://www.tinybird.co/docs/docs/get-started/quick-start)  .



---

URL: https://www.tinybird.co/docs/get-started/plans/support
Last update: 2025-01-29T13:58:37.000Z
Content:
---
title: "Support · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird is here to help. Learn about our support options."
---




# Support [¶](https://www.tinybird.co/docs/about:blank#support)

Tinybird provides support through different channels depending on your plan. See [Plans](https://www.tinybird.co/docs/docs/get-started/plans).

Read on to learn more about support options and common solutions.

## Channels [¶](https://www.tinybird.co/docs/about:blank#channels)

Tinybird provides support through the following channels depending on your plan:

| Plan | Channels |
| --- | --- |
| Free | Support for the Free plan is available through the[  Community Slack](https://www.tinybird.co/docs/docs/community)   , which is monitored by the Tinybird team. |
| Developer | In addition to the Community Slack, priority support is provided through[  support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co)  . |
| Enterprise | In addition to priority email support, Enterprise customers can request a dedicated Slack channel for direct support. |

## Integrated troubleshooting [¶](https://www.tinybird.co/docs/about:blank#integrated-troubleshooting)

Tinybird tries to give you direct feedback and notifications if it spots anything going wrong. Use Tinybird's [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) to get more details on what's going on in your data and queries.

## Recover deleted items [¶](https://www.tinybird.co/docs/about:blank#recover-deleted-items)

Tinybird creates and backs up daily snapshots and retains them for 7 days.

## Copy or move data [¶](https://www.tinybird.co/docs/about:blank#copy-or-move-data)

To copy data between Data Sources, use [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes).

## Recover data from quarantine [¶](https://www.tinybird.co/docs/about:blank#recover-data-from-quarantine)

The quickest way to recover rows from quarantine is to fix the cause of the errors and then reingest the data. See [Recover data from quarantine](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine#recovering-rows-from-quarantine).

You can also use the [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources) , like `datasources_ops_log`.

## Get help [¶](https://www.tinybird.co/docs/about:blank#get-help)

If you haven't been able to solve the issue, or it looks like there is a problem on Tinybird's side, get in touch. You can always contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

If you have an Enterprise account with Tinybird, contact us using your shared Slack channel.



---

URL: https://www.tinybird.co/docs/get-started/plans/limits
Last update: 2025-02-18T15:52:20.000Z
Content:
---
title: "Limits · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird has limits on certain operations and processes to ensure the highest performance."
---




# Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

Tinybird has limits on certain operations and processes to ensure the highest performance.

## Free plan limits [¶](https://www.tinybird.co/docs/about:blank#free-limits)

Organizations on the Free plan have the following limits:

- 1k queries per day in total for all Workspaces in the Organization. Applies to API Endpoints and Query API calls.
- 10 queries per second (QPS) limit. Applies to API Endpoints and Query API calls. Your operations can take x2 QPS per second allowed in you plan for API endpoint requests or SQL queries. See[  Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode)  .
- 10 GB storage, including all Data Sources from all Workspaces.
- 0.5 vCPU limit, limiting the concurrency but not the total active minutes. See[  Active minutes](https://www.tinybird.co/docs/docs/get-started/plans/billing#active-minutes)  .
- Sinks are not available on the Free plan.

If you reach the limits, you will receive an email notification. See [Email notifications](https://www.tinybird.co/docs/docs/get-started/plans/limits#email-notifications).

See [Free plan](https://www.tinybird.co/docs/docs/get-started/plans#free) for more information.

## Developer plan limits [¶](https://www.tinybird.co/docs/about:blank#developer-limits)

Depending on the size of your Developer plan, the following limits apply:

- You can't exceed the vCPU time limit. The limit relates to vCPU time used during a minute. Tinybird allows usage bursts of 2x the vCPU time per minute. If you reach the vCPU limit you're allowed 75% of the QPS limit of your plan size during the next 5 minutes. See[  Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode)  .
- You can't exceed the queries per second limit (QPS). This limit relates to API Endpoints and queries. Your operations can take x2 QPS per second allowed in you plan for API endpoint requests or SQL queries. See[  Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode)  .
- If you exceed the active minutes in your plan, additional minutes are billed at a fixed rate per minute. See the[  Pricing page](https://www.tinybird.co/pricing)   for more information.

Limits are applied at the organization level, so all Workspaces in an Organization share the same limits. If you reach the limits, you will receive an email notification. See [Email notifications](https://www.tinybird.co/docs/docs/get-started/plans/limits#email-notifications).

See [Developer plan](https://www.tinybird.co/docs/docs/get-started/plans#developer) for more information.

## Enterprise plan limits [¶](https://www.tinybird.co/docs/about:blank#enterprise-limits)

Enterprise plans limits depend on the infrastructure type: shared or dedicated.

### Shared infrastructure [¶](https://www.tinybird.co/docs/about:blank#shared-infrastructure)

Enterprise plans starting at 4 vCPUs on shared infrastructure have the same limits as the Developer plans.

- You can't exceed the vCPU time limit. The limit relates to vCPU time used during a minute. Tinybird allows usage bursts of 2x the vCPU time per minute. If you reach the vCPU limit you're allowed 75% of the QPS limit of your plan size during the next 5 minutes. See[  Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode)  .
- You can't exceed the queries per second limit (QPS). This limit relates to API Endpoints and queries. Your operations can take x2 QPS per second allowed in you plan for API endpoint requests or SQL queries. See[  Burst mode](https://www.tinybird.co/docs/docs/get-started/plans/concepts#burst-mode)  .
- If you exceed the active minutes in your plan, additional minutes are billed at a fixed rate per minute. See the[  Pricing page](https://www.tinybird.co/pricing)   for more information.

Limits are applied at the organization level, so all Workspaces in an Organization share the same limits. If you reach the limits, you will receive an email notification. See [Email notifications](https://www.tinybird.co/docs/docs/get-started/plans/limits#email-notifications).

### Dedicated infrastructure [¶](https://www.tinybird.co/docs/about:blank#dedicated-infrastructure)

Enterprise plans with dedicated infrastructure have no predefined limits. The only limits are the ones related to the underlying infrastructure and available capacity.

See [Rate limiter](https://www.tinybird.co/docs/docs/get-started/plans/billing#rate-limiter) for more information on rate limiting in dedicated infrastructure plans.

## Email notifications [¶](https://www.tinybird.co/docs/about:blank#email-notifications)

When reaching or temporarily exceeding your plan's limits, you will receive one of the following emails.

### Free plans [¶](https://www.tinybird.co/docs/about:blank#free-plans)

The following table shows the limits' notifications you might receive for the Free plan.

| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage (1/2 vCPU) | Triggered after 3 minutes at 150-200% usage in last 30 minutes. | Triggered after 1 minute at >200% usage in last 30 minutes. |
| [  Queries Per Second](https://www.tinybird.co/docs/docs/get-started/plans/concepts#queries-per-second)   (10 QPS) | Triggered after 30 seconds at >75% of limit in last 30 minutes. | Triggered after 30 seconds at >100% of limit in last 30 minutes. |
| [  Active minutes](https://www.tinybird.co/docs/docs/get-started/plans/concepts#active-vcpu-minutes-hours) | Used more than 75% of allocated minutes. | Used more than 100% of allocated minutes. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |
| Requests per day (1,000 queries) | Used more than 75% of daily query limit. | Used more than 100% of daily query limit. |

### Developer and shared plans [¶](https://www.tinybird.co/docs/about:blank#developer-and-shared-plans)

The following table shows the limits' notifications you might receive for plans on shared infrastructure.

| Limit | Warning email | Alert email |
| --- | --- | --- |
| vCPU usage | Triggered after 3 minutes at 150-200% usage in last 30 minutes. | Triggered after 1 minute at >200% usage in last 30 minutes. |
| [  Queries Per Second](https://www.tinybird.co/docs/docs/get-started/plans/concepts#queries-per-second) | Triggered after 30 seconds at >75% of limit in last 30 minutes. | Triggered after 30 seconds at >100% of limit in last 30 minutes. |
| [  Active minutes](https://www.tinybird.co/docs/docs/get-started/plans/concepts#active-vcpu-minutes-hours) | Used more than 75% of allocated minutes. | Used more than 100% of allocated minutes. |
| Storage Usage | Used more than 75% of allocated storage. | Used more than 100% of allocated storage. |

## Workspace limits [¶](https://www.tinybird.co/docs/about:blank#workspace-limits)

The following limits apply to all workspaces in an organization, regardless of the plan.

| Description | Limit |
| --- | --- |
| Number of Workspaces | 90 Workspaces. Soft limit, contact support to increase it. |
| Number of seats | 90 seats. Soft limit, contact support to increase it. |
| Number of branches, including `main` | Default 4. Soft limit, contact support to increase it. |
| Number of Data Sources | 100 Data Sources. Soft limit, contact support to increase it. |
| Number of Copy Pipes | 10 Pipes. Soft limit, contact support to increase it. |
| 10 Pipes. Soft limit, contact support to increase it. |  |
| Number of Tokens | 100,000 tokens. If you need more you should take a look at[  JWT tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#json-web-tokens-jwts)   ) |
| Number of secrets | 100 secrets. |

See [Rate limits for JWTs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#json-web-tokens-jwts) for more detail specifically on JWT limits.

## Ingestion limits [¶](https://www.tinybird.co/docs/about:blank#ingestion-limits)

The following ingestion limits apply to all workspaces in an organization, regardless of the plan.

| Description | Limit |
| --- | --- |
| Data Source max columns | 500 |
| Full body upload | 8 MB |
| Multipart upload - CSV and NDJSON | 500 MB |
| Multipart upload - Parquet | 50 MB |
| Max file size - Parquet - Free plan | 1 GB |
| Max file size - Parquet - Developer and Enterprise plan | 5 GB |
| Max file size (uncompressed) - Free plan | 10 GB |
| Max file size (uncompressed) - Developer and Enterprise plan | 32 GB |
| Kafka topics | Default is 5. Soft limit, contact support to increase it. |
| Max parts created at once - NDJSON/Parquet jobs and Events API | 12 parts |

### Ingestion limits (API) [¶](https://www.tinybird.co/docs/about:blank#ingestion-limits-api)

Tinybird throttles requests based on the capacity. So if your queries are using 100% resources you might not be able to run more queries until the running ones finish.

| Description | Limit and time window |
| --- | --- |
| Request size - Events API | 10 MB |
| Response size | 100 MB |
| Create Data Source from schema | 25 times per minute |
| Create Data Source from file or URL* | 5 times per minute |
| Append data to Data Source* | 5 times per minute |
| Append data to Data Source using v0/events | 1,000 times per second |
| Replace data in a Data Source* | 5 times per minute |

- The quota is shared at Workspaces level when creating, appending data, or replacing data. For example, you can't do 5 requests of each type per minute, for a total of 15 requests. You can do at most a grand total of 5 requests of those types combined.

The number of rows in append requests doesn't impact the ingestion limit; each request counts as a single ingestion.

If you exceed your rate limit, your request is throttled and you receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

### BigQuery Connector limits [¶](https://www.tinybird.co/docs/about:blank#bigquery-connector-limits)

The import jobs run in a pool, with capacity for up to 2 concurrent jobs. If more scheduled jobs overlap, they're queued.

| Description | Limit and time window |
| --- | --- |
| Maximum frequency for the scheduled jobs | 5 minutes |
| Maximum rows per append or replace | 50 million rows. Exports that exceed this number of rows are truncated to this amount |

You can't pause a Data Source with an ongoing import. You must wait for the import to finish before pausing the Data Source.

### DynamoDB Connector limits [¶](https://www.tinybird.co/docs/about:blank#dynamodb-connector-limits)

| Description | Limit and time window |
| --- | --- |
| Storage | 500 GB |
| Throughput | 250 Write Capacity Units (WCU), equivalent to 250 writes of at most 1 KB per second |

## Query limits [¶](https://www.tinybird.co/docs/about:blank#query-limits)

The following query limits apply to all workspaces in an organization, regardless of the plan.

| Description | Limit |
| --- | --- |
| SQL length | 8KB |
| Result length | 100 MB |
| Query execution time | 10 seconds |

If you exceed your rate limit, your request will be throttled and you will receive *HTTP 429 Too Many Requests* response codes from the API. Each response contains a set of HTTP headers with your current rate limit status.

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

### Query timeouts [¶](https://www.tinybird.co/docs/about:blank#query-timeouts)

If query execution time exceeds the default limit of 10 seconds, an error message appears. Long execution times hint at issues that need to be fixed in the query or the Data Source schema.

To avoid query timeouts, optimize your queries to remove inefficiencies and common mistakes. See [Optimizations](https://www.tinybird.co/docs/docs/work-with-data/optimization) for advice on how to detect and solve issues in your queries that might cause timeouts.

If you still need to increase the timeout limit, contact support. See [Get help](https://www.tinybird.co/docs/docs/get-started/plans/support#get-help).

Only paid accounts can raise the timeout limit.

## Publishing limits [¶](https://www.tinybird.co/docs/about:blank#publishing-limits)

The following publishing limits apply to all workspaces in an organization, regardless of the plan.

### Materialized Views limits [¶](https://www.tinybird.co/docs/about:blank#materialized-views-limits)

No numerical limits, certain operations are [inadvisable when using Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views#limitations).

### Sink limits [¶](https://www.tinybird.co/docs/about:blank#sink-limits)

Sink Pipes have the following limits:

| Description | Limit |
| --- | --- |
| Execution time | 300s |
| Frequency | Up to every minute |
| Memory usage per query | 10 GB |
| Active jobs (running or queued) | 6 |

### Copy Pipe limits [¶](https://www.tinybird.co/docs/about:blank#copy-pipe-limits)

Copy Pipes have the following limits:

| Description | Limit |
| --- | --- |
| Execution time | 50% of the scheduling period,  max |
| Frequency | Up to every minute |
| Active jobs (running or queued) | 6 |

## Delete limits [¶](https://www.tinybird.co/docs/about:blank#delete-limits)

Delete jobs have the following limits:

| Description | Limit |
| --- | --- |
| Active delete jobs per Workspace | 6 |

## Legacy billing [¶](https://www.tinybird.co/docs/about:blank#legacy-billing)

The following tables provide details on how each operation is limited in Legacy plans.

### Sink limits (Legacy) [¶](https://www.tinybird.co/docs/about:blank#sink-limits-legacy)

Sink Pipes have the following limits, depending on your billing plan:

| Plan | Sink Pipes per Workspace | Execution time | Frequency | Memory usage per query | Active jobs (running or queued) |
| --- | --- | --- | --- | --- | --- |
| Pro | 3 | 30s | Up to every 10 min | 10 GB | 3 |
| Enterprise | 10 | 300s | Up to every minute | 10 GB | 6 |

### Copy Pipe limits (Legacy) [¶](https://www.tinybird.co/docs/about:blank#copy-pipe-limits-legacy)

Copy Pipes have the following limits, depending on your billing plan:

| Plan | Copy Pipes per Workspace | Execution time | Frequency | Active jobs (running or queued) |
| --- | --- | --- | --- | --- |
| Build | 1 | 20s | Once an hour | 1 |
| Pro | 3 | 30s | Up to every 10 minutes | 3 |
| Enterprise | 10 | 50% of the scheduling period, 30 minutes max | Up to every minute | 6 |

## Delete limits (Legacy) [¶](https://www.tinybird.co/docs/about:blank#delete-limits-legacy)

Delete jobs have the following limits, depending on your billing plan:

| Plan | Active delete jobs per Workspace |
| --- | --- |
| Build | 1 |
| Pro | 3 |
| Enterprise | 6 |

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Understand how Tinybird[  plans and billing work](https://www.tinybird.co/docs/docs/get-started/plans/billing)  .
- Explore popular use cases for user-facing analytics (like dashboards) in Tinybird's[  Use Case Hub](https://www.tinybird.co/docs/docs/use-cases)  .



---

URL: https://www.tinybird.co/docs/get-started/plans/concepts
Last update: 2025-02-11T11:13:47.000Z
Content:
---
title: "Key concepts · Tinybird Docs"
theme-color: "#171612"
description: "Key concepts for understanding your bill and plan limits."
---




# Billing and limits concepts [¶](https://www.tinybird.co/docs/about:blank#billing-and-limits-concepts)

Tinybird [plans](https://www.tinybird.co/docs/docs/get-started/plans) are sized and billed according to available resources and usage, with limits that you can exceed temporarily.

Read on to understand the key concepts behind your bill and plan limits.

## Active vCPU minutes / hours [¶](https://www.tinybird.co/docs/about:blank#active-vcpu-minutes-hours)

Developer plans bill vCPU usage using active minutes. An active minute is when any operation used a vCPU for a minute. Multiple operations executed within the same minute still count as a single active minute. When using fractioned vCPUs, an active minute is proportional to the fraction, for example 30 seconds of 0.5 vCPU.

Plan sizes come with a number of active hours, which is the number of active minutes you can use divided by 60. If you consume all your active minutes, the overage is billed at a fixed rate per minute. Usage bursts allows you to temporarily exceed the vCPU usage limit. See [Burst mode](https://www.tinybird.co/docs/about:blank#burst-mode).

## Queries per second [¶](https://www.tinybird.co/docs/about:blank#queries-per-second)

Queries per second (QPS) is the maximum number of queries per second that your plan allows. Calls to API endpoints and queries sent to the Query API count towards your QPS limit. Queries made from the UI are excluded from the limit.

Plan sizes come with a number of QPS, which is the maximum number of queries per second that your plan allows. If exceed the QPS limit, your QPS are temporarily reduced for a fixed amount of time. Usage bursts allows you to temporarily exceed the QPS limit. See [Burst mode](https://www.tinybird.co/docs/about:blank#burst-mode).

## Burst mode [¶](https://www.tinybird.co/docs/about:blank#burst-mode)

To allow for unexpected spikes in traffic, each plan includes a burst mode. Burst mode allows you to temporarily exceed your vCPU and QPS limits. If you temporarily exceed your limits, you won't be billed and you'll receive an email alerting you of the situation and suggesting to increase your plan size.

### vCPU burst mode [¶](https://www.tinybird.co/docs/about:blank#vcpu-burst-mode)

Your operations can take x2 vCPU time per minute allowed in you plan for real-time operations. For batch operations, like populates or copies, we allow the whole operation to run until it reaches a platform limit, like maximum available memory.

For example, for a populate that needs 180 seconds of CPU time in a minute, if you are on a Developer Plan S-1 where we allow operations to run up to 120 seconds per minute, the operation will finish and the limit won't be triggered.

### QPS burst mode [¶](https://www.tinybird.co/docs/about:blank#qps-burst-mode)

Your operations can take x2 QPS per second allowed in you plan for API endpoint requests or SQL queries.

To better understand how burst mode works, consider the following example:

- You’re on a plan with a standard rate of 10 QPS and a burst capacity of 20 QPS.
- Your plan allows 10 queries per second as the normal rate (leak rate).
- You have a burst capacity of 20 QPS, meaning you can temporarily handle up to 20 queries per second for short bursts.
- Each second, your bucket can "leak" 10 queries and can temporarily hold more if needed.

Here's how burst mode works in practice:

- If you send 15 queries in one second, 10 are processed at the normal rate and 5 use burst capacity.
- If you send 25 queries in one second, 10 are processed at normal rate, 10 use burst capacity, and 5 are rejected (over the 20 QPS burst limit).
- The burst capacity "leaks" back to normal levels at a rate of 10 QPS, meaning after a burst, your capacity gradually returns to the standard rate.

The leaking mechanism ensures you can handle occasional traffic spikes while maintaining overall performance and preventing system overload.

For example, if you use your full burst capacity of 20 QPS, it takes about 1 second of processing at the normal 10 QPS rate before you can burst again. This helps balance flexibility for traffic spikes with consistent system performance.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn how to[  estimate your plan size](https://www.tinybird.co/docs/docs/get-started/plans#estimate-your-plan-size)  .
- Read the[  billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing)   to understand which data operations count towards your bill, and how to optimize your usage.
- Learn about[  limits](https://www.tinybird.co/docs/docs/get-started/plans/limits)   and how to adjust them.



---

URL: https://www.tinybird.co/docs/get-started/plans/billing
Last update: 2025-02-12T12:56:56.000Z
Content:
---
title: "Understand your bill · Tinybird Docs"
theme-color: "#171612"
description: "Information about billing, what it's based on, as well as Tinybird pricing plans."
---




# Understand your bill [¶](https://www.tinybird.co/docs/about:blank#understand-your-bill)

If you are on a paid plan, read on to learn how billing works for each plan. See [Tinybird plans](https://www.tinybird.co/docs/docs/get-started/plans) for information on each plan's features.

To learn about the prices of each plan, see [Pricing](https://www.tinybird.co/pricing).

## Developer plans [¶](https://www.tinybird.co/docs/about:blank#developer-plans)

Developer plans come in many sizes, depending on the compute resources you need. Each size includes a base capacity for your organization, which is a fixed amount you pay for. Other items are billed according to usage beyond a threshold.

Developer plans billing is calculated from the following resources:

| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available. More than 3 vCPUs require an Enterprise plan. Each vCPU size includes a number of active minutes. | Fixed. |
| Active minutes | An active minute is when any operation used a vCPU for a minute. See[  Active minutes](https://www.tinybird.co/docs/docs/get-started/plans/concepts#active-minutes)  . | Usage based. |
| Queries per second (QPS) | Maximum number of queries per second (QPS). Each vCPU size includes a QPS limit. See[  Queries per second](https://www.tinybird.co/docs/docs/get-started/plans/concepts#queries-per-second)  . | Fixed. |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the 25 GB included in the plan. |
| Data transfer | When using[  Sinks](https://www.tinybird.co/docs/docs/api-reference/sink-pipes-api)   , usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |

### Self service [¶](https://www.tinybird.co/docs/about:blank#self-service)

After you enter your credit card details, you can pick whatever size you want for your Developer plan, with up to 3 vCPUs supported. You can resize your plan every 24 hours. See [Resize and upgrade](https://www.tinybird.co/docs/docs/get-started/plans#developer-resize-and-upgrade).

To use more than 3 vCPUs, you need to sign up to an Enterprise plan. See [Enterprise plans](https://www.tinybird.co/docs/about:blank#enterprise-plans).

### Periodicity [¶](https://www.tinybird.co/docs/about:blank#periodicity)

Developer plans are billed monthly. Each monthly invoice contain fixed costs and usage-based costs calculated from the previous period.

The billing period starts on the subscription date.

## Enterprise plans [¶](https://www.tinybird.co/docs/about:blank#enterprise-plans)

Enterprise plans provide plans greater than 3 vCPUs. The size of each plan determines the fixed amount you pay for. Other items are billed according to usage beyond a threshold.

Enterprise plans start at 4 vCPUs of capacity and have a minimum storage of 1 terabyte. Resizing requires contacting sales.

Plans with dedicated compute resources are also available. Contact Tinybird support at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) to learn more.

### What are credits? [¶](https://www.tinybird.co/docs/about:blank#what-are-credits)

Enterprise billing is based on [credits](https://www.tinybird.co/docs/about:blank#what-are-credits) . A credit is a unit of resource usage. You can acquire and spend credits across the entire Tinybird offering, regardless of the region or feature.

Credits provide a predictable, consistent, and easy to track way of paying for Tinybird services. Instead of committing to specific features or an amount of resources, you commit to spending a number of credits over at least a 12 months period. You can spend credits on storage, compute, support, and so on.

### Credits usage [¶](https://www.tinybird.co/docs/about:blank#credits-usage)

The following table shows how Tinybird calculates credits usage for each resource:

| Resource | Billing | Type |
| --- | --- | --- |
| vCPUs | Number of vCPUs available. Each vCPU size includes a number of active minutes. | Fixed. |
| Active minutes | An active minute is when any operation used a vCPU for a minute. See[  Active minutes](https://www.tinybird.co/docs/docs/get-started/plans/concepts#active-minutes)  . | Usage based. |
| Queries per second (QPS) | Maximum number of queries per second (QPS). Each vCPU size includes a QPS limit. See[  Queries per second](https://www.tinybird.co/docs/docs/get-started/plans/concepts#queries-per-second)  . | Fixed. |
| Storage | Average of daily maximum usage of the compressed disk storage of all your data, in gigabytes. | Usage based when usage exceeds the storage included in the plan. |
| Data transfer | When using[  Sinks](https://www.tinybird.co/docs/docs/api-reference/sink-pipes-api)   , usage is billed depending on the destination, which can be the same cloud provider and region as your Tinybird cluster, or a different one. | Usage based. |
| Support | Premier or Enterprise monthly support fee. | Fixed. |
| Private Link | Private connection. Optional. Billed monthly. | Usage based. |

### Periodicity [¶](https://www.tinybird.co/docs/about:blank#periodicity)

Enterprise plans are billed every month according to the amount of [credits](https://www.tinybird.co/docs/about:blank#what-are-credits) you've used. The billing cycle starts on the first day of the month and ends on the last day of the month.

### Track invoices [¶](https://www.tinybird.co/docs/about:blank#track-invoices)

In Enterprise plans, invoices are issued upon [credits](https://www.tinybird.co/docs/about:blank#what-are-credits) purchase, which can happen when signing the contract or when purchasing additional credits. You can check your invoices from the customer portal.

### Monitor usage [¶](https://www.tinybird.co/docs/about:blank#monitor-usage)

You can monitor [credits](https://www.tinybird.co/docs/about:blank#what-are-credits) usage, including remaining credits, cluster usage, and current commitment through your organization's dashboard. See [Enterprise infrastructure monitoring](https://www.tinybird.co/docs/docs/get-started/administration/organizations#dedicated-infrastructure-monitoring) . You can also check usage using the monthly usage receipts.

### Rate limiter [¶](https://www.tinybird.co/docs/about:blank#rate-limiter)

The rate limiter monitors the status of the cluster and limits the number of concurrent requests to prevent the cluster from crashing due to insufficient memory. This allows the cluster to continue working, albeit with a rate limit.

The rate limiter activates when the following situation occurs:

- When total memory usage in a host in the cluster is over 70% in clusters with less than 64GB of memory per host and 80% in the rest.
- Percentage of 408 Timeout Exceeded and 500 Internal Server Error due to memory limits for a Pipe endpoint exceeds 10% of the total requests.

If both conditions are met, the maximum number of concurrent requests to the Pipe endpoint is limited proportionally to the percentage of errors. Workspace administrators in dedicated infrastructure receive an email indicating the affected Pipe endpoints and the concurrency limit.

The rate limiter rolls back after 5 minutes and it's activated again if the previously described conditions repeat.

For example, if a Pipe endpoint is receiving 10 requests per second and 5 failed during a high memory usage scenario due to a timeout or memory error, the number of concurrent queries is limited to a half, that is, 5 concurrent requests for that specific Pipe endpoint.

While the Rate limiter is active, endpoints return a 429 HTTP status code. You can retry those requests using a backoff mechanism. For example, you can space requests 1 second between each other.

## Legacy billing [¶](https://www.tinybird.co/docs/about:blank#legacy-billing)

The following tables provide details on how each operation is billed in Legacy plans.

### Data ingestion (Legacy) [¶](https://www.tinybird.co/docs/about:blank#data-ingestion-legacy)

The following information applies to legacy plans. For information on how to migrate to Shared or Enterprise plans, see [Plans](https://www.tinybird.co/docs/docs/get-started/plans).

| Service | Operation | Processing fee | Description |
| --- | --- | --- | --- |
| Data Sources API | Write | US$0.07 per GB | Low frequency: Append data to an existing Data Source (imports, backfilling, and so on). |
| Events API | Write | US$0.07 per GB | High frequency: Insert events in real-time (individual or batched). |
| Connectors | Write | US$0.07 per GB | Any connector that ingests data into Tinybird (Kafka, S3, GCS, BigQuery, and so on). |

### Data manipulation (Legacy) [¶](https://www.tinybird.co/docs/about:blank#data-manipulation-legacy)

The following information applies to legacy Pro plans. For information on how to migrate to Shared or Enterprise plans, see [Migrate from legacy plans](https://www.tinybird.co/docs/docs/#migrate-from-legacy-plans).

| Service | Operation | Processing fee | Description |
| --- | --- | --- | --- |
| Pipes API | Read | US$0.07 per GB | Interactions with Pipes to retrieve data from Tinybird generate read operations. |
| Query API | Read | US$0.07 per GB | Interactions with the Query API to retrieve data from Tinybird. |
| Materialized Views (Populate) | Read/Write | Free | Executed as soon as you create the MV to populate it. Tinybird doesn't charge any processing fee. Data is written into a new or existing Data Source. |
| Materialized Views (Append) | Read/Write | US$0.07 per GB | New data is read from an origin Data Source, filtered, and written to a destination Data Source. |
| Copy Pipes | Read/Write | US$0.07 per GB | On-demand or scheduled operations. Data is read from the Data Source, filtered, and written to a destination Data Source. |
| Replace | Read/Write | US$0.07 per GB | Replacing data entirely or selectively. |
| Delete data | Read/Write | US$0.07 per GB | Selective data delete from a Data Source. |
| Delete an entire Data Source | Read/Write | US$0.07 per GB | Delete all the data inside a Data Source. |
| Truncate | Write | US$0.07 per GB | Delete all the data from a Data Source. |
| Time-to-live (TTL) operations | Write | Free | Anytime Tinybird deletes data as a result of a TTL. |
| BI Connector | Read | US$0.07 per GB | Data read from Tinybird using the BI connector. |

### Data transfer (Legacy) [¶](https://www.tinybird.co/docs/about:blank#data-transfer-legacy)

The following information applies to legacy Pro plans. For information on how to migrate to Shared or Enterprise plans, see [Plans](https://www.tinybird.co/docs/docs/get-started/plans).

| Service | Operation | Processing fee | Data transfer fee | Description |
| --- | --- | --- | --- | --- |
| S3 Sink | Read/Transfer (no write fees) | US$0.07 per GB | Same region: US$0.01 per GB. Different region: US$0.10 per GB | Data is read, filtered, and then transferred to the destination bucket. This is an on-demand or scheduled operation. Data transfer fees apply. |

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- Explore different[  Tinybird plans](https://www.tinybird.co/docs/docs/get-started/plans)  .
- [  Sign up for a free Tinybird account and follow the quick start](https://www.tinybird.co/docs/docs/get-started/quick-start)  .
- Learn more about[  using the Playground and Time Series](https://www.tinybird.co/docs/docs/work-with-data/query#use-the-playground)  .



---

URL: https://www.tinybird.co/docs/get-started/administration/workspaces
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Workspaces · Tinybird Docs"
theme-color: "#171612"
description: "Workspaces are containers for all of your Tinybird resources. Learn all about Workspaces here!"
---




# Workspaces [¶](https://www.tinybird.co/docs/about:blank#workspaces)

A Workspace is a set of Tinybird resources, like Data Sources, Pipes, nodes, API Endpoints, and Tokens. Workspaces are always created inside organizations. See [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations).

You can use Workspaces to manage separate projects, use cases, and dev, staging, or production environments in Tinybird. Each Workspace has administrators and members who can view and edit resources.

Tinybird represents Workspaces using the icon.

## Create a Workspace [¶](https://www.tinybird.co/docs/about:blank#create-a-workspace)

To create a new Workspace, select the name of an existing Workspace. In the menu, select **Create Workspace (+)**.

Complete the dialog with the details of your new Workspace, and select "Create Workspace".

Workspaces must have unique names within a region.

### Create a Workspace using the CLI [¶](https://www.tinybird.co/docs/about:blank#create-a-workspace-using-the-cli)

To create a new Workspace using the CLI, use the following command:

tb workspace create You can use this command interactively or by providing the required inputs with flags. To use it interactively, run the command without any flags. For example, `tb workspace create`.

Supply [your user Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#your-user-token) by pasting it into the prompt. Next, supply a name for your Workspace. Workspaces must have unique names within a region.

Workspace name [new_workspace_9479]: internal_creating_new_workspaces_example When Tinybird creates the Workspace, a message similar to the following appears:

** Workspace 'internal_creating_new_workspaces_example' has been created If you are using the CLI in an automated system, you can instead pass each value using flags. For example:

tb workspace create --user_token  --starter-kit 1 internal_creating_new_workspaces_example
## Workspace ID [¶](https://www.tinybird.co/docs/about:blank#workspace-id)

The Workspace ID is a unique identifier for each Workspace. You can copy the Workspace ID from the list of Workspaces in the UI, or by selecting the More actions (⋯) icon and selecting **Copy ID** in the Workspace settings.

To find the Workspace ID using the Tinybird CLI, run `tb workspace current` from the CLI:

tb workspace current
** Current workspace:
--------------------------------------------------------------------------------------------
| name                   | id                                   | role  | plan   | current |
--------------------------------------------------------------------------------------------
| tinybird_web_analytics | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | guest | Custom | True    |
--------------------------------------------------------------------------------------------
## Delete a Workspace [¶](https://www.tinybird.co/docs/about:blank#delete-a-workspace)

Deleting a Workspace deletes all resources within the Workspace, including Data Sources, ingested data, Pipes, and published API Endpoints.

Deleted Workspaces can't be recovered.

To delete a Workspace, select the **Settings** icon. In the dialog that appears, select **Advanced Settings** and then select **Delete Workspace** . Confirm the deletion.

### Delete a Workspace using the CLI [¶](https://www.tinybird.co/docs/about:blank#delete-a-workspace-using-the-cli)

To delete a Workspace using the CLI, use the following command:

tb workspace delete Provide the name of the Workspace and [your user Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#your-user-token) . For example:

tb workspace delete my_workspace --user_token 
## Manage Workspace members [¶](https://www.tinybird.co/docs/about:blank#manage-workspace-members)

You can invite as many members to a Workspace as you want. A member can belong to multiple Workspaces.

Members always have a role assigned. You can modify the role of a user at any time.

Tinybird has the following member roles for Workspaces:

| Role | Manage resources | Manage users | Access to billing information | Create a branch |
| --- | --- | --- | --- | --- |
| `Admin` | Yes | Yes | Yes | Yes |
| `Guest` | Yes | No | No | Yes |
| `Viewer` | No | No | No | Yes |

### Manage Workspace members in the UI [¶](https://www.tinybird.co/docs/about:blank#manage-workspace-members-in-the-ui)

In the top right corner of the Tinybird UI, select the Cog icon. In the modal, navigate to "Members" to review any members already part of your Workspace.

Add a new member by entering their email address and confirming their role from the dropdown options. You can invite multiple users at the same time by adding multiple email addresses separated by a comma.

The users you invite will get an email notifying them that they have been invited. If they don't already have a Tinybird account, they will be prompted to create one to accept your invite.

Invited users appear in the user management modal and by default have the **Guest** role. If the user loses their invite link, you can resend it here too, or copy the link to your clipboard. You can also remove members from here using the "..." menu and selecting "Remove".

### Adding Workspace users in the CLI [¶](https://www.tinybird.co/docs/about:blank#adding-workspace-users-in-the-cli)

To add new users, use the following command:

tb workspace members add Supply your [user Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#your-user-token) using the `--user_token` flag. Add the email address of the user you want to invite as the final argument to the command.

tb workspace members add --user_token  my_new_team_member@example.com A successful invite returns the following output:

** User my_new_team_member@example.com added to workspace 'internal_getting_started_guide'
## Share Data Sources between Workspaces [¶](https://www.tinybird.co/docs/about:blank#share-data-sources-between-workspaces)

Sometimes you might want to share resources between different teams or projects. For example, you might have a Data Source of events data that multiple teams want to use to build features with. Rather than duplicating this data, you can share it between multiple Workspaces.

To share a Data Source, follow these steps:

1. Open the Data Source that you want to share.
2. When you hover over the Data Source, select the**  More actions (⋯)**   icon.
3. In the menu, select**  Share**  .
4. In the**  Search**   box, type the name of the Workspace that you want to share the Data Source with. You can only share Data Sources with Workspaces that you are a member of.
5. Select the Workspace. The Workspace appears in the**  Shared**   section.
6. Select**  Done**   to close the dialog.

You can't share Data Sources between Workspaces in different regions.

### Share Data Sources using the CLI [¶](https://www.tinybird.co/docs/about:blank#share-data-sources-using-the-cli)

To share a Data Source, use the following command:

tb datasource share Supply [your user Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#your-user-token) using the `--user_token` flag. Then, supply the Data Source name and target Workspace name as the following arguments:

tb datasource share --user_token  shopping_data my_second_workspace The Data Source that you want to share must exist in the Workspace that your Tinybird CLI is authenticated against. To check which Workspace your CLI is currently authenticated with, use:

tb auth info You can also run `tb push --user_token ` to push a Data Source with a `SHARED_WITH` parameter to share it with another Workspace.

## Regions [¶](https://www.tinybird.co/docs/about:blank#regions)

A Workspace belongs to one region. The following table lists the available regions and their corresponding API base URLs:

| Region | Provider | Provider region | API base URL |
| --- | --- | --- | --- |
| Europe | GCP | europe-west3 | [  https://api.tinybird.co](https://api.tinybird.co/) |
| US East | GCP | us-east4 | [  https://api.us-east.tinybird.co](https://api.us-east.tinybird.co/) |
| Europe | AWS | eu-central-1 | [  https://api.eu-central-1.aws.tinybird.co](https://api.eu-central-1.aws.tinybird.co/) |
| US East | AWS | us-east-1 | [  https://api.us-east.aws.tinybird.co](https://api.us-east.aws.tinybird.co/) |
| US West | AWS | us-west-2 | [  https://api.us-west-2.aws.tinybird.co](https://api.us-west-2.aws.tinybird.co/) |

Additional regions for GCP, AWS, and Azure are available for Enterprise customers.

Tinybird documentation uses `https://api.tinybird.co` as the default example API base URL. If you aren't using the Europe GCP region, replace the URL with the API base URL for your region.

## Single Sign-On (SSO) [¶](https://www.tinybird.co/docs/about:blank#single-sign-on-sso)

Tinybird provides email and OAuth providers for logging in into the platform.

If you have a requirement for SSO integration, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

## Secure cloud connections [¶](https://www.tinybird.co/docs/about:blank#secure-cloud-connections)

Tinybird supports TLS across all ingest connectors, providing encryption on the wire for incoming data.

If you have a requirement for secure cloud connections, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).



---

URL: https://www.tinybird.co/docs/get-started/administration/team-integration-governance
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Team integration and data governance · Tinybird Docs"
theme-color: "#171612"
description: "Learn how different teams work with Tinybird, and how Tinybird supports your team to manage data."
---




# Team integration and data governance [¶](https://www.tinybird.co/docs/about:blank#team-integration-and-data-governance)

Tinybird supports a wide range of industries and products. Customers organize themselves and their businesses in different ways, but there are overarching principles you can adopt and adapt. Knowing how to integrate your team with Tinybird is important to get the most value out of the platform.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

You need to be familiar with the following [core Tinybird concepts](https://www.tinybird.co/docs/docs/get-started/quick-start/core-concepts):




- Data Source: Where data is ingested and stored.
- Pipe: How data is transformed.
- Workspace: How data projects are organized, containing Data Sources and Pipes.
- Shared Data Source: A Data Source shared between Workspaces.
- Roles: Each Workspace has Admin, Guest, and Viewer roles.
- Organizations: Contain all Workspaces and members.

## Roles and responsibilities [¶](https://www.tinybird.co/docs/about:blank#roles-and-responsibilities)

Tinybird allows you to share Data Sources across Workspaces. This means you can create Workspaces that map your organization, and not have to duplicate the Data Sources. In general, most Tinybird users have an ingestion Workspace where the owners of the data ingest the data, clean the data, and prepare it for onward consumption. They then share these Data Sources with other internal teams using the [Sharing Data Sources](https://www.tinybird.co/docs/docs/get-started/administration/workspaces#sharing-data-sources-between-workspaces) feature.

You can have as many ingestion Workspaces as you need: bigger organizations group their Workspaces by domain, some organizations group them by team. For instance, in a bank, you might find different teams managing their own data and therefore several ingestion Workspaces where data is curated and exposed to other teams. In this case, each team maps to a domain:



<-figure->
![Diagram showing each ingestion team mapping to a specific domain](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguide-team-integration-governance-team-1.png&w=3840&q=75)


However, in other companies where data is centralized and managed by the data platform, you might find a single ingestion Workspace where all the data is ingested and shared with other onward Workspaces where specific domain users build their own use cases:



<-figure->
![Diagram showing each ingestion team mapping to a specific domain](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguide-team-integration-governance-team-2.png&w=3840&q=75)


Some organizations rely on a hybrid solution, where data is provided by the data platform but each domain group also ingest their own data:



<-figure->
![Diagram showing each ingestion team mapping to a specific domain](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguide-team-integration-governance-team-3.png&w=3840&q=75)


Whatever your approach, it's an established pattern to have an ingestion or data-platform Workspace or team who own ingestion and data preparation, and share the desired Data Sources with other teams. These downstream, domain-specific teams then create the Pipe logic specific to their own area, usually in a Workspace specifically for that domain. That way, the responsibilities reflect a manageable, clear separation of concerns.

## Enforcing data governance [¶](https://www.tinybird.co/docs/about:blank#enforcing-data-governance)

Tinybird supports your data governance efforts in the following ways:

- **  Availability**   is assured by the platform's uptime[  SLA](https://www.tinybird.co/terms-and-conditions)   . Tinybird wants all your teams to be able to access all the data they need, any time they need. You can monitor availability using[  monitoring tools](https://www.tinybird.co/docs/about:blank#monitoring)   , which includes monitoring ingestion, API Endpoints, and quarantined rows. Tinybird offers a straightforward way to reingest quarantined rows and maintain Materialized Views to automatically reingest data.
- **  Control**   over data access is managed through a single Organization page in Tinybird. You can enforce the principle of least privilege by assigning different roles to Workspace members, and easily check data quality and consumption using Tinybird's Service Data Sources. Tinybird also supports schema evolution and you can[  keep multiple schema versions running](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source)   at the same time so consumers can adjust at their own pace.
- **  Usability**   is maximized by having ingestion Workspaces. They allow you to share cleaned, curated data, with specific and adjusted schemas giving consumers precisely what they need. Workspace members have the flexibility to create as many Workspaces as they need, and[  use the Playground feature](https://www.tinybird.co/docs/docs/work-with-data/query#use-the-playground)   to sandbox new ideas.
- **  Consistency**   : Data owners have responsibility over what they want to share with others. You can monitor which Workspace is ingesting data.
- **  Data integrity and quality**   , especially at scale and at speed, is essential. Just like availability, it's a use case for leveraging Tinybird's monitoring capabilities. See[  Additional ecosystem tools](https://www.tinybird.co/docs/about:blank#additional-ecosystem-tools)   . Ingestion teams can build Pipes to monitor everything about their inbound data and create alerts. Alerts can be technical or business-related.
- **  Data security**   : This information is available at the top-level Organizations page and also in individual Workspaces.

## Additional ecosystem tools [¶](https://www.tinybird.co/docs/about:blank#additional-ecosystem-tools)



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

Tinybird is built around the idea of data that changes or grows continuously, and provides the following tools as part of Tinybird. These tools help you get insights on and monitor your Workspaces, data, and resources.

### Operations log [¶](https://www.tinybird.co/docs/about:blank#operations-log)

The [Operations log](https://www.tinybird.co/docs/docs/monitoring/health-checks#operations-log) shows information on each individual Data Source, including its size, the number of rows, the number of rows in the quarantine Data Source (if any), and when it was last updated.

The Operations log contains details of the events for the Data Source, which are displayed as the results of the query. Use it to see every single call made to the system (API call, ingestion, jobs). This is helpful if you're concerned about one specific Data Source and need to investigate.

### Monitoring [¶](https://www.tinybird.co/docs/about:blank#monitoring)

You can use the [Organizations](https://www.tinybird.co/docs/docs/get-started/administration/organizations) feature for managing Workspaces and Members, and monitoring their entire consolidated Tinybird consumption in one place.

For example, you can track costs and usage for each individual Workspace.

### Testing [¶](https://www.tinybird.co/docs/about:blank#testing)

To ensure that all production load is efficient and accurate, all Tinybird API Endpoints that you create in your Workspaces can be tested before going to production. You can do this by [using version control](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work).

### Alerts and health checks [¶](https://www.tinybird.co/docs/about:blank#alerts-and-health-checks)

To ensure everything is working as expected once you're in production, any team can create alerts and health checks on top of Tinybird's [Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources).

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Watch this video to understand how Factorial sets domain boundaries and[  organizes their teams](https://youtu.be/8rctUKRXcdw?t=574)  .
- Build something fast and fun - follow the[  quick start tutorial](https://www.tinybird.co/docs/docs/get-started/quick-start)  .



---

URL: https://www.tinybird.co/docs/get-started/administration/organizations
Last update: 2025-02-17T16:47:35.000Z
Content:
---
title: "Organizations · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird Organizations provide enterprise customers with a single pane of glass to monitor usage across multiple Workspaces."
---




# Organizations [¶](https://www.tinybird.co/docs/about:blank#organizations)

Organizations provide a centralized way of managing Workspaces and members in a region. From the **Organization settings** section you can monitor resource usage and check your current plan's usage and billing if you're on a paid plan. See [Plans](https://www.tinybird.co/docs/docs/get-started/plans).

The **Organization settings** screen consists of the following areas:

- Billing (Org admins only)
- Usage
- Workspaces
- Members
- Monitoring

All Workspaces must belong to an organization in Tinybird. See [Workspaces](https://www.tinybird.co/docs/docs/get-started/administration/workspaces).

Tinybird represents organizations using the icon.

## Users and organizations [¶](https://www.tinybird.co/docs/about:blank#users-and-organizations)

Users can be members of one or more organizations. A user can only be the admin of one organization.

## Access the organization settings [¶](https://www.tinybird.co/docs/about:blank#access-the-organization-settings)

To access the **Organization settings** screen, log in and select your organization name from a Workspace.



<-figure->
![Organization settings](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fyour-organization.png&w=3840&q=75)


You can also access the screen by selecting **Organization settings** in the settings screen of a Workspace.

## Billing [¶](https://www.tinybird.co/docs/about:blank#billing)

The **Billing** page contains a summary of the credits balance for your plan, with links to billing details and a summary of your plan. See [Billing](https://www.tinybird.co/docs/docs/get-started/plans/billing).

### Free and Developer plans [¶](https://www.tinybird.co/docs/about:blank#free-and-developer-plans)

From the **Billing** page you can upgrade or resize your plan, or cancel it if you wish to downgrade to Free.

If you're on a Developer plan, the usage diagram shows the total expenditure by monthly invoice.

Only organization administrators can access the **Billing** section.

### Enterprise plans [¶](https://www.tinybird.co/docs/about:blank#enterprise-plans)

The **Credits usage** section shows a chart with your credits usage over time and a forecast.



<-figure->
![Credits usage](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcredits-usage-new.png&w=3840&q=75)


## Usage overview [¶](https://www.tinybird.co/docs/about:blank#usage-overview)

The **Usage** page shows details about your resource usage, followed by a detailed breakdown of your consumption.

### Usage charts [¶](https://www.tinybird.co/docs/about:blank#usage-charts)

The following charts are available depending on your plan:

| Plan | Charts |
| --- | --- |
| Free | vCPU time, Memory, Queries per second, Errors |
| Developer | vCPU time, Memory, Queries per second, Errors |
| Enterprise (Shared infrastructure) | vCPU time, Memory, Queries per second, Errors |
| Enterprise (Dedicated infrastructure) | Load, Memory, Requests per Second, Errors |

You can select between average and maximum values for all usage charts, as well as the period, from **Last hour** to **Last 7 days**.

### Usage table [¶](https://www.tinybird.co/docs/about:blank#usage-table)

The **Usage** table shows resource usage by Workspace or resource. You can filter by resource name by typing in the text box.

## Workspaces [¶](https://www.tinybird.co/docs/about:blank#workspaces)

The **Workspaces** page shows details of all your Workspaces, including their name, members, and resources used by each Workspace.

New Workspaces created by a user with an email domain linked to, or matching an organization are automatically added to that organization.

## Members [¶](https://www.tinybird.co/docs/about:blank#members)

The **Members** page shows details of your organization members, the Workspaces they belong to, and their roles. From this page you can manage existing members, their permissions, or invite new members.

The table shows the following information:

- Email
- Workspaces and roles

To view the detail of a member’s Workspaces and roles, select the arrow next to the Workspace count. A menu shows all the Workspaces that user is part of, plus their role in each Workspace.

To change a user’s role or remove them from a Workspace, hover over the Workspace name and follow the arrow. Select a new role from **Admin**, **Guest** , or **Viewer** , or remove them from the Workspace. You don't need to be a user in that Workspace to make changes to its users.

### Add or remove members [¶](https://www.tinybird.co/docs/about:blank#add-or-remove-members)

To add a user, select **Add member**.

To remove a user from the organization, select **Remove member** in the menu.

Only organization administrators can manage users in the **Members** page.

### Add an organization admin [¶](https://www.tinybird.co/docs/about:blank#add-an-organization-admin)

To add another user as an organization administrator, follow these steps:

1. Navigate to the**  Organization settings**   page.
2. Go to the**  Members**   section.
3. Locate the user you want to make an administrator.
4. Select the**  More actions (⋯)**   icon.
5. Select**  Organization admin**   in the menu.

This grants organization administrator access to the selected users.

## Monitoring [¶](https://www.tinybird.co/docs/about:blank#monitoring)

The **Monitoring** page provides customized code snippets to monitor usage using the Tinybird API and the Service Data Sources. See [Organization Service Data Sources](https://www.tinybird.co/docs/docs/monitoring/service-datasources#organization-service-data-sources).

### Prometheus metrics endpoint [¶](https://www.tinybird.co/docs/about:blank#prometheus-metrics-endpoint)

The Prometheus endpoint is available at `/v0/metrics` . Use your organization's observability Token, which you can find in the **Prometheus** tab under **Monitoring**, **Organization settings**.

Tinybird reports all the metrics available in the service Data Source, except for `InstanceType` , which is reported as a metric label.

### Refreshing your organization's Observability Token [¶](https://www.tinybird.co/docs/about:blank#refreshing-your-organizations-observability-token)

If your organization's observability Token gets compromised or is lost, refresh it using the following endpoint:

`/v0/organizations//tokens/Observability%20%28builtin%29/refresh?token=`

You must use your `user token` for this call, which you can copy from any of your workspaces.



---

URL: https://www.tinybird.co/docs/get-started/administration/auth-tokens
Last update: 2025-01-22T09:19:22.000Z
Content:
---
title: "Tokens · Tinybird Docs"
theme-color: "#171612"
description: "Tokens allow you to secure your Tinybird resources, providing fine-grained access control to your users. Learn all about Tokens here!"
---




# Tokens [¶](https://www.tinybird.co/docs/about:blank#tokens)

Tokens protect access to your Tinybird resources. Any operations to manage your Tinybird resources using the CLI or REST API require a valid Token with the necessary permissions. Access to the APIs you publish in Tinybird are also protected with Tokens.

Tokens can have different scopes. This means you can limit which operations a specific Token can do. You can create Tokens that are, for example, only able to do admin operations on Tinybird resources, or only have `READ` permission for a specific Data Source.

Tinybird represents tokens using the icon.

## Token types [¶](https://www.tinybird.co/docs/about:blank#token-types)

There are two types of Tokens:

- Static[  Tokens](https://www.tinybird.co/docs/about:blank#tokens)   : Used when performing operations on your account, like importing data, creating Data Sources, or publishing APIs using the CLI or REST API.
- [  JWT Tokens](https://www.tinybird.co/docs/about:blank#json-web-tokens-jwts)   : Used when publishing an API that exposes your data to an application.

## Tokens [¶](https://www.tinybird.co/docs/about:blank#tokens)

Tinybird Tokens, also known as static Tokens, are permanent and long-term. They're stored inside Tinybird and don't have an expiration date or time. They're useful for backend-to-backend integrations, where you call Tinybird as another service.

### Token scopes [¶](https://www.tinybird.co/docs/about:blank#token-scopes)

When a Token is created, you can give it a set of zero or more scopes that define which tables can be accessed by that Token, and which methods can be used to access them.

A `READ` Token can be augmented with an SQL filter. A filter allows you to further restrict what data a Token grants access to. Using a filter, you can also [implement row-level security](https://www.tinybird.co/blog-posts/row-level-security-in-tinybird) on a `READ` -scoped Token.

The following scopes are available:

| Value | Description |
| --- | --- |
| `DATASOURCES:CREATE` | Enables your Token to create and append data to Data Sources. |
| `DATASOURCES:APPEND:datasource_name` | Allows your Token to append data to the defined Data Sources. |
| `DATASOURCES:DROP:datasource_name` | Allows your Token to delete the specified Data Sources. |
| `DATASOURCES:READ:datasource_name` | Gives your Token read permissions for the specified Data Sources. Also gives read for the quarantine Data Source. |
| `DATASOURCES:READ:datasource_name:sql_filter` | Gives your Token read permissions for the specified table with the `sql_filter`   applied. |
| `PIPES:CREATE` | Allows your Token to create new Pipes and manipulate existing ones. |
| `PIPES:DROP:pipe_name` | Allows your Token to delete the specified Pipe. |
| `PIPES:READ:pipe_name` | Gives your Token read permissions for the specified Pipe. |
| `PIPES:READ:pipe_name:sql_filter` | Gives your Token read permissions for the specified Pipe with the `sql_filter`   applied. |
| `TOKENS` | Gives your Token the capacity of managing Tokens. |
| `ADMIN` | All permissions are granted. Do not use this Token except in really specific cases. |
| `ORG_DATASOURCES:READ` | Gives your Token the capacity of reading[  organization service datasources](https://www.tinybird.co/docs/docs/monitoring/service-datasources#organization-service-data-sources)   , without using an org admin-level token. |

When adding the `DATASOURCES:READ` scope to a Token, it automatically gives read permissions to the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources#the-quarantine-data-source) associated with it.

Applying Tokens with filters to queries that use the FINAL clause isn't supported. If you need to apply auth filters to deduplications, use an alternative strategy. See [deduplication strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies#different-alternatives-based-on-your-requirements).

### Default Workspace Tokens [¶](https://www.tinybird.co/docs/about:blank#default-workspace-tokens)

All Workspaces are created with a set of basic Tokens that you can add to by creating additional Tokens:

- `admin token`   for that Workspace, used for signing JWTs.
- `admin  token`   for that Workspace that belongs specifically to your user account for CLI usage.
- `create datasource token`   for creating Data Sources in that Workspace.
- `user token`   for creating new Workspaces or deleting ones where are an admin.

Some Tokens are created automatically by Tinybird during certain operations like scheduled copies, and [can be updated](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes#change-copy-pipe-token-reference).

### User Token [¶](https://www.tinybird.co/docs/about:blank#user-token)

Your User Token is specific to your user account. It's a permanent Token that allows you to perform operations that aren't limited to a single Workspace, such as creating new Workspaces.

You can only obtain your User Token from your Workspace by going to **Tokens** and retrieving the `user token`.

### Create a Token [¶](https://www.tinybird.co/docs/about:blank#create-a-token)

In the Tinybird UI, navigate to **Tokens** > **Plus (+) icon** . Rename the new Token and update its scopes using the previous table as a guide.

## JSON Web Tokens (JWTs) BETA

[¶](https://www.tinybird.co/docs/about:blank#json-web-tokens-jwts)

JWTs are currently in public beta. They aren't feature-complete and may change in the future. If you have any feedback or suggestions, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

JWTs are signed tokens that allow you to securely authorize and share data between your application and Tinybird.

Unlike static Tokens, JWTs are not stored in Tinybird. They're created by you, inside your application, and signed with a shared secret between your application and Tinybird. Tinybird validates the signature of the JWT, using the shared secret, to ensure it's authentic.

### When to use JWTs [¶](https://www.tinybird.co/docs/about:blank#when-to-use-jwts)

The primary purpose for JWTs is to allow your app to call Tinybird API Endpoints from the frontend without proxying through your backend.

If you are building an application where a frontend component needs data from a Tinybird API Endpoint, you can use JWTs to authorize the request directly from the frontend.

The typical pattern looks like this:

1. A user starts a session in your application.
2. The frontend requests a JWT from your backend.
3. Your backend generates a new JWT, signed with the Tinybird shared secret, and returns to the frontend.
4. The frontend uses the JWT to call the Tinybird API Endpoints directly.

### JWT payload [¶](https://www.tinybird.co/docs/about:blank#jwt-payload)

The payload of a JWT is a JSON object that contains the following fields:

| Key | Example Value | Required | Description |
| --- | --- | --- | --- |
| workspace_id | workspaces_id | Yes | The UUID of your Tinybird Workspace, found in the Workspace list. See[  Workspace ID](https://www.tinybird.co/docs/docs/get-started/administration/workspaces#workspace-id)  . |
| name | frontend_jwt | Yes | Used to identify the token in the `tinybird.pipe_stats_rt`   table, useful for analytics. Doesn't need to be unique. |
| exp | 123123123123 | Yes | The Unix timestamp (UTC) showing the expiry date & time. After a token has expired, Tinybird returns a 403 HTTP status code. |
| scopes | [{"type": "PIPES:READ", "resource": "requests_per_day", "fixed_params": {"org_id": "testing"}}] | Yes | Used to pass data to Tinybird, including the Tinybird scope, resources and fixed parameters. |
| scopes.type | PIPES:READ | Yes | The type of scope, for example `READ`   . See[  JWT scopes](https://www.tinybird.co/docs/about:blank#jwt-scopes)   for supported scopes. |
| scopes.resource | t_b9427fe2bcd543d1a8923d18c094e8c1 or top_airlines | Yes | The ID or name of the Pipe that the scope applies to, like which API Endpoint the token can access. |
| scopes.fixed_params | {"org_id": "testing"} | No | Pass arbitrary fixed values to the API Endpoint. These values can be accessed by Pipe templates to supply dynamic values at query time. |
| limits | {"rps": 10} | No | You can limit the number of requests per second the JWT can perform. See[  JWT rate limit](https://www.tinybird.co/docs/about:blank#rate-limits-for-jwt-tokens)  . |

Check out the [JWT example](https://www.tinybird.co/docs/about:blank#jwt-example) to see what a complete payload looks like.

### JWT algorithm [¶](https://www.tinybird.co/docs/about:blank#jwt-algorithm)

Tinybird always uses HS256 as the algorithm for JWTs and doesn't read the `alg` field in the JWT header. You can skip the `alg` field in the header.

### JWT scopes [¶](https://www.tinybird.co/docs/about:blank#jwt-scopes)

| Value | Description |
| --- | --- |
| `PIPES:READ:pipe_name` | Gives your Token read permissions for the specified Pipe |

### JWT expiration [¶](https://www.tinybird.co/docs/about:blank#jwt-expiration)

JWTs can have an expiration time that gives each Token a finite lifespan.

Setting the `exp` field in the JWT payload is mandatory, and not setting it results in a 403 HTTP status code from Tinybird when requesting the API Endpoint.

Tinybird validates that a JWT hasn't expired before allowing access to the API Endpoint.

If a Token has expired, Tinybird returns a 403 HTTP status code.

### JWT fixed parameters [¶](https://www.tinybird.co/docs/about:blank#jwt-fixed-parameters)

Fixed parameters allow you to pass arbitrary values to the API Endpoint. These values can be accessed by Pipe templates to supply dynamic values at query time.

For example, consider the following fixed parameter:

##### Example fixed parameters

{
  "fixed_params": {
    "org_id": "testing"
  }
} This passes a parameter called `org_id` with the value `testing` to the API Endpoint. You can then use this value in your SQL queries:

##### Example SQL query

SELECT fieldA, fieldB FROM my_pipe WHERE org_id = '{{ String(org_id) }}' This is particularly useful when you want to pass dynamic values to an API Endpoint that are set by your backend and must be safe from user tampering. A good example is multi-tenant applications that require row-level security, where you need to filter data based on a user or tenant ID.

The value `org_id` is always the one specified in the `fixed_params` . Even if you specify a new value in the URL when requesting the endpoint, Tinybird always uses the one specified in the JWT.

You can use JWT fixed parameters in combination with Pipe [dynamic parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters).

### JWT example [¶](https://www.tinybird.co/docs/about:blank#jwt-example)

Consider the following payload with all [required and optional fields](https://www.tinybird.co/docs/about:blank#jwt-payload):

##### Example payload

{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        }
    ],
    "limits": {
      "rps": 10
    }
} Use the Admin Token from your Workspace to sign the payload, for example:

##### Example Workspace Admin Token

p.eyJ1IjogIjA1ZDhiYmI0LTdlYjctND... With the payload and Admin Token, the signed JWT payload would look like this:

##### Example JWT

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ3b3Jrc3BhY2V...
### JWT limitations [¶](https://www.tinybird.co/docs/about:blank#jwt-limitations)

The following limitations apply to JWTs:

- You can't refresh JWTs individually from inside Tinybird as they aren't stored in Tinybird. You must do this from your application, or you can globally invalidate all JWTs by refreshing your Admin Token.
- If you refresh your Admin Token, all the tokens are invalidated.
- If your token expires or is invalidated, you get a 403 HTTP status code from Tinybird when requesting the API Endpoint.

### Create a JWT in production [¶](https://www.tinybird.co/docs/about:blank#create-a-jwt-in-production)

There is wide support for creating JWTs in many programming languages and frameworks. Any library that supports JWTs should work with Tinybird.

A common library to use with Python is [PyJWT](https://github.com/jpadilla/pyjwt/tree/master) . Common libraries for JavaScript are [jsonwebtoken](https://github.com/auth0/node-jsonwebtoken#readme) and [jose](https://github.com/panva/jose).

- JavaScript (Next.js)
- Python

##### Create a JWT in Python using pyjwt

import jwt
import datetime
import os

TINYBIRD_SIGNING_KEY = os.getenv('TINYBIRD_SIGNING_KEY')

def generate_jwt():
  expiration_time = datetime.datetime.utcnow() + datetime.timedelta(hours=3)
  payload = {
      "workspace_id": "workspaces_id",
      "name": "frontend_jwt",
      "exp": expiration_time,
      "scopes": [
          {
              "type": "PIPES:READ",
              "resource": "requests_per_day",
              "fixed_params": {
                  "org_id": "testing"
              }
          }
      ]
  }

  return jwt.encode(payload, TINYBIRD_SIGNING_KEY, algorithm='HS256')
### Create a JWT using the CLI or the API [¶](https://www.tinybird.co/docs/about:blank#create-a-jwt-using-the-cli-or-the-api)

If for any reason you don't want to generate a JWT on your own, Tinybird provides an API and a CLI utility to create JWTs.

- API
- CLI

##### Create a JWT with the Tinybird CLI

tb token create jwt my_jwt --ttl 1h --scope PIPES:READ --resource my_pipe --filters "column_name=value"
### Error handling [¶](https://www.tinybird.co/docs/about:blank#error-handling)

There are many reasons why a request might return a `403` status code. When a `403` is received, check the following:

1. Confirm the JWT is valid and hasn't expired. The expiration time is in the `exp`   field in the JWT's payload.
2. The generated JWTs can only read Tinybird API Endpoints. Confirm you're not trying to use the JWT to access other APIs.
3. Confirm the JWT has a scope to read the endpoint you are trying to read.
4. If you generated the JWT outside of Tinybird, without using the API or the CLI, make sure you are using the**  Workspace** `admin token`   , not your personal one.

### Rate limits for JWTs [¶](https://www.tinybird.co/docs/about:blank#rate-limits-for-jwts)



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

When you specify a `limits.rps` field in the payload of the JWT, Tinybird uses the name specified in the payload of the JWT to track the number of requests being done. If the number of requests goes beyond the limit, Tinybird starts rejecting new requests and returns an "HTTP 429 Too Many Requests" error. See [limits docs](https://www.tinybird.co/docs/docs/get-started/plans/limits) for more information.

The following example shows the tracking of all requests done by `frontend_jwt` . Once you reach 10 requests per second, Tinybird would start rejecting requests:

##### Example payload with global rate limit

{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        }
    ],
    "limits": {
      "rps": 10
    }
} If `rps <= 0` , Tinybird ignores the limit and assumes there is no limit.

As the `name` field doesn't have to be unique, all the tokens generated using the `name=frontend_jwt` would be under the same umbrella. This can be useful if you want to have a global limit in one of your apps or components.

If you want to limit for each specific user, you can generate a JWT using the following payload. In this case, you would specify a unique name so the limits only apply to each user:

##### Example of a payload with isolated rate limit

{
    "workspace_id": "workspaces_id",
    "name": "frontend_jwt_user_",
    "exp": 123123123123,
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "requests_per_day",
            "fixed_params": {
                "org_id": "testing"
            }
        }
    ],
    "limits": {
      "rps": 10
    }
}
## Monitor Token usage [¶](https://www.tinybird.co/docs/about:blank#monitor-token-usage)

You can monitor Token usage using Tinybird's Service Data Sources. See ["Monitor API Performance"](https://www.tinybird.co/docs/docs/monitoring/analyze-endpoints-performance#example-4-monitor-usage-of-tokens).

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Follow a walkthrough guide:[  How to consume APIs in a Next.js frontend with JWTs](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-apis-nextjs)  .
- Read about the Tinybird[  Tokens API](https://www.tinybird.co/docs/docs/api-reference/token-api)  .
- Understand[  Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches)   in Tinybird.



---

URL: https://www.tinybird.co/docs/get-data-in/table-functions/url
Last update: 2025-01-22T16:59:23.000Z
Content:
---
title: "URL table function · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird URL table function"
---




# URL table function BETA

[¶](https://www.tinybird.co/docs/about:blank#url-table-function)

The Tinybird `url()` table function is currently in private beta. If you're interested in early access, reach out to support.

The Tinybird `url()` table function allows you to read data from an existing URL into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `url` function keyword, then publish the node as a Copy Pipe that does a sync on every run. See [Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions) for general information and tips.

## Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

Create a new Pipe Node. Call the `url` table function and pass the URL. Optionally, pass the format and the structure:

##### Example query logic

SELECT
    JSONExtractString(data, 'article') AS article,
    JSONExtractInt(data, 'views') AS views,
    JSONExtractInt(data, 'rank') AS rank
FROM
    (
        SELECT toJSONString(arrayJoin(items.articles)) AS data
        FROM
            url(
                'https://wikimedia.org/api/rest_v1/metrics/pageviews/top/en.wikipedia.org/all-access/2024/03/all-days',
                'JSONColumns',
                'items Tuple(access Nullable(String), articles Array(Tuple(article Nullable(String), rank Nullable(Int64), views Nullable(Int64))), day Nullable(String), month Nullable(String), project Nullable(String), year Nullable(String))'
            )
    ) Publish this node as a Copy Pipe. You can choose to append only new data or replace all data.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions)
- [  Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes)



---

URL: https://www.tinybird.co/docs/get-data-in/table-functions/postgresql
Last update: 2025-01-22T20:58:43.000Z
Content:
---
title: "PostgreSQL table function · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird PostgreSQL table function"
---




# PostgreSQL table function BETA

[¶](https://www.tinybird.co/docs/about:blank#postgresql-table-function)

The Tinybird `postgresql()` table function is currently in public beta.

The Tinybird `postgresql()` table function allows you to read data from your existing PostgreSQL database into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `postgresql` function keyword, then publish the node as a Copy Pipe that does a sync on every run. See [Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions) for general information and tips.

## Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

Create a new Pipe Node. Call the `postgresql` table function and pass the hostname and port, database, table, user, and password:

##### Example query logic

SELECT *
FROM postgresql(
  'aws-0-eu-central-1.TODO.com:3866',
  'postgres',
  'orders',
  {{tb_secret('pg_username')}},
  {{tb_secret('pg_password')}},
) Publish this node as a Copy Pipe. You can choose to append only new data or replace all data.

## Type support and inference [¶](https://www.tinybird.co/docs/about:blank#type-support-and-inference)

Here's a detailed conversion table:

| PostgreSQL data type | Tinybird data type |
| --- | --- |
| BOOLEAN | UInt8 or Bool |
| SMALLINT | Int16 |
| INTEGER | Int32 |
| BIGINT | Int64 |
| REAL | Float32 |
| DOUBLE PRECISION | Float64 |
| NUMERIC or DECIMAL | Decimal(p, s) |
| CHAR(n) | FixedString(n) |
| VARCHAR (n) | String |
| TEXT | String |
| BYTEA | String |
| TIMESTAMP | DateTime |
| TIMESTAMP WITH TIME ZONE | DateTime (with appropriate timezone handling) |
| DATE | Date |
| TIME | String (since there is no direct TIME type) |
| TIME WITH TIME ZONE | String |
| INTERVAL | String |
| UUID | UUID |
| ARRAY | Array(T) where T is the array element type |
| JSON | String or JSON |
| JSONB | String |
| INET | String |
| CIDR | String |
| MACADDR | String |
| ENUM | Enum8 or Enum16 |
| GEOMETRY | String |

## Considerations [¶](https://www.tinybird.co/docs/about:blank#considerations)

The following considerations apply to the `postgresql()` table function:

- Tinybird doesn't support all PostgreSQL types directly, so some types are mapped to String, which is the most flexible type for arbitrary data.
- For the `NUMERIC`   and `DECIMAL`   types, `Decimal(p, s)`   in Tinybird requires specifying precision (p) and scale (s).
- Time zone support in Tinybird's `DateTime`   can be managed via additional functions or by ensuring consistent storage and retrieval time zones.
- Some types like `INTERVAL`   don't have a direct equivalent in Tinybird and are usually stored as String or decomposed into separate fields.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions)
- [  Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes)
- [  Migrate from Postgres to Tinybird](https://www.tinybird.co/docs/docs/get-data-in/migrate/migrate-from-postgres)



---

URL: https://www.tinybird.co/docs/get-data-in/table-functions/mysql
Last update: 2025-01-22T20:58:43.000Z
Content:
---
title: "MySQL table function · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird MySQL table function."
---




# MySQL table function BETA

[¶](https://www.tinybird.co/docs/about:blank#mysql-table-function)

The Tinybird `mysql()` table function is currently in private beta. If you're interested in early access, reach out to support.

The Tinybird `mysql()` table function allows you to read data from your existing MySQL database into Tinybird, then schedule a regular Copy Pipe to orchestrate synchronization. You can load full tables, and every run performs a full replace on the Data Source.

To use it, define a node using standard SQL and the `mysql` function keyword, then publish the node as a Copy Pipe that does a sync on every run. See [Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions) for general information and tips.

## Syntax [¶](https://www.tinybird.co/docs/about:blank#syntax)

Create a new Pipe Node. Call the `mysql` table function and pass the hostname and port, database, table, user, and password:

##### Example query logic

SELECT *
FROM mysql(
  'aws-0-eu-central-1.TODO.com:3866',
  'mysql',
  'orders',
  {{tb_secret('my_username')}},
  {{tb_secret('my_password')}},
) Publish this node as a Copy Pipe. You can choose to append only new data or replace all data.

## Type support and inference [¶](https://www.tinybird.co/docs/about:blank#type-support-and-inference)

Here's a detailed conversion table:

| MySQL data type | Tinybird data type |
| --- | --- |
| UNSIGNED TINYINT | UInt8 |
| TINYINT | Int8 |
| UNSIGNED SMALLINT | UInt16 |
| SMALLINT | Int16 |
| UNSIGNED INT, UNSIGNED MEDIUMINT | UInt32 |
| INT, MEDIUMINT | Int32 |
| UNSIGNED BIGINT | UInt64 |
| BIGINT | Int64 |
| FLOAT | Float32 |
| DOUBLE | Float64 |
| DATE | Date |
| DATETIME, TIMESTAMP | DateTime |
| BINARY | FixedString |

## Considerations [¶](https://www.tinybird.co/docs/about:blank#considerations)

The following considerations apply to the `mysql()` table function:

- Tinybird doesn't support all MySQL types directly, so some types are mapped to String, which is the most flexible type for arbitrary data.
- Time zone support in Tinybird's `DateTime`   can be managed via additional functions or by ensuring consistent storage and retrieval time zones.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Table functions](https://www.tinybird.co/docs/docs/get-data-in/table-functions)
- [  Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes)



---

URL: https://www.tinybird.co/docs/get-data-in/migrate/migrate-from-rockset
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Migrate from Rockset · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to migrate from Rockset to Tinybird, and the overview of how to quickly & safely recreate your setup."
---




# Migrate from Rockset [¶](https://www.tinybird.co/docs/about:blank#migrate-from-rockset)

In this guide, you'll learn how to migrate from Rockset to Tinybird, and the overview of how to quickly & safely recreate your setup.

Rockset will [no longer be active](https://docs.rockset.com/documentation/docs/faq) after September 30th, 2024. This guide explains the parallels between Rockset and Tinybird features, and how to migrate to using Tinybird.

Wondering how to create an account? It's free! [Start here](https://www.tinybird.co/signup).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You don't need an active Tinybird Workspace to read through this guide, but it's good idea to understand the foundational concepts and how Tinybird integrates with your team. If you're new to Tinybird, read the [team integration guide](https://www.tinybird.co/docs/docs/get-started/administration/team-integration-governance).

## At a high level [¶](https://www.tinybird.co/docs/about:blank#at-a-high-level)

Tinybird is a great alternative to Rockset's analytical capabilities.

Tinybird is a data platform for data and engineering teams to solve complex real-time, operational, and user-facing analytics use cases at any scale, with end-to-end latency in milliseconds for streaming ingest and high QPS workloads.

It's a SQL-first analytics engine, purpose-built for the cloud, with real-time data ingest and full JOIN support. Native, managed ingest connectors make it easy to ingest data from a variety of sources. SQL queries can be published as production-grade, scalable REST APIs for public use or secured with JWTs.

Tinybird is a managed platform that scales transparently, requiring no cluster operations, shard management or worrying about replicas.

See how Tinybird is used by industry-leading companies today in the [Customer Stories](https://www.tinybird.co/customer-stories) hub.

## Concepts [¶](https://www.tinybird.co/docs/about:blank#concepts)

A lot of concepts are the same between Rockset and Tinybird, and there are a handful of others that have a 1:1 mapping. In Tinybird:




- Data Source: Where data is ingested and stored.
- Pipe: How data is transformed.
- Workspace: How data projects are organized, containing Data Sources and Pipes.
- Shared Data Source: A Data Source shared between Workspaces.
- Roles: Each Workspace has Admin, Guest, and Viewer roles.
- Organizations: Contain all Workspaces and members.

### Key concept comparison [¶](https://www.tinybird.co/docs/about:blank#key-concept-comparison)

#### Data Sources [¶](https://www.tinybird.co/docs/about:blank#data-sources)

Super similar. Rockset and Tinybird both support ingesting data from many types of data sources. You ingest into Tinybird and create a Tinybird **Data Source** that you then have control over - you can iterate the schema, monitor your ingestion, and more. See the [Data Sources docs](https://www.tinybird.co/docs/docs/get-data-in/data-sources).

#### Workspaces [¶](https://www.tinybird.co/docs/about:blank#workspaces)

Again, very similar. In Rockset, Workspaces contain resources like Collections, Aliases, Views, and Query Lambdas. In Tinybird, **Workspaces** serve the same purpose (holding resources), and you can also share Data Sources between *multiple* Workspaces. Enterprise users monitor and manage Workspaces using the [Organizations feature](https://www.tinybird.co/docs/docs/get-started/administration/organizations) . See the [Workspace docs](https://www.tinybird.co/docs/docs/get-started/administration/workspaces#what-is-a-workspace).

#### Ingest Transformations [¶](https://www.tinybird.co/docs/about:blank#ingest-transformations)

These are analogous to Tinybird's **Pipes** . It's where you transform your data. The difference is that Rockset does this on initial load (on raw data), whereas Tinybird lets you create and manage a Data Source first, then transform it however you need. See the [Pipes docs](https://www.tinybird.co/docs/docs/work-with-data/query/pipes).

#### Views [¶](https://www.tinybird.co/docs/about:blank#views)

Similar to Tinybird's **nodes** - the modular, chainable "bricks" of SQL queries that compose a Pipe. Like Views, nodes can reference resources like other nodes, Pipes, Data Sources, and more. See the [Pipes > Nodes docs](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#nodes).

#### Rollups [¶](https://www.tinybird.co/docs/about:blank#rollups)

The Tinybird equivalent of rollups is **Materialized Views** . Materialized Views give you a way to pre-aggregate and pre-filter large Data Sources incrementally, adding simple logic using SQL to produce a more relevant Data Source with significantly fewer rows. Put simply, Materialized Views shift computational load from query time to ingestion time, so your API Endpoints stay fast. See the [Materialized Views docs](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views).

#### Query Lambdas [¶](https://www.tinybird.co/docs/about:blank#query-lambdas)

The Tinybird equivalent of Query Lambdas is **API Endpoints** . You can publish the result of any SQL query in your Tinybird Workspace as an HTTP API Endpoint. See the [API Endpoint docs](https://www.tinybird.co/docs/docs/publish/api-endpoints).

### Schemaless ingestion [¶](https://www.tinybird.co/docs/about:blank#schemaless-ingestion)

You can do schemaless/variable schema event ingestion on Tinybird by storing the whole JSON in a column. Use the following schema in your Data Source definition and use [JSONExtract functions](https://www.tinybird.co/docs/docs/sql-reference/functions/json-functions#jsonextract-functions) to parse the result afterwards.

##### schemaless.datasource

SCHEMA >
    `root` String `json:$`

ENGINE "MergeTree" If your data has some common fields, be sure to extract them and add them to the sorting key.

It's definitely possible to do schemaless, but having a defined schema is a great idea. Tinybird provides you with an easy way to manage your schema [using .datasource schema files](https://www.tinybird.co/docs/docs/get-data-in#create-your-schema).

Read the docs on using the [JSONPath syntax in Tinybird](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths-and-the-root-object) for more information.

## Ingest data and build a POC [¶](https://www.tinybird.co/docs/about:blank#ingest-data-and-build-a-poc)

Tinybird allows you to ingest your data from a variety of sources, then create Tinybird Data Sources in your Workspace that can be queried, published, materialized, and more.

Just like Rockset, Tinybird supports ingestion from:

- Data streams (Kafka, Kinesis).
- OLTP databases (DynamoDB, MongoDB, MySQL, PostgreSQL).
- Data lakes (S3, GCS).

A popular option is connecting DynamoDB to Tinybird. Follow [the guide here](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb) or pick another source from the side nav under "Ingest".

Materialized Views give you a way to pre-aggregate and pre-filter large Data Sources incrementally, adding simple logic using SQL to produce a more relevant Data Source with significantly fewer rows.

Put simply, Materialized Views shift computational load from query time to ingestion time, so your API Endpoints stay fast.

## Useful resources [¶](https://www.tinybird.co/docs/about:blank#useful-resources)

Migrating to a new tool, especially at speed, can be challenging. Here are some helpful resources to get started on Tinybird:

- Set up a[  DynamoDB Data Source](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb)   to start streaming data today.
- Read the blog post[  "Migrating from Rockset? See how Tinybird features compare"](https://www.tinybird.co/blog-posts/migrating-from-rockset-feature-comparison)  .
- Read the blog post[  "A practical guide to real-time CDC with MongoDB"](https://www.tinybird.co/blog-posts/mongodb-cdc)  .

## Billing and limits [¶](https://www.tinybird.co/docs/about:blank#billing-and-limits)

Read the [billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing) to understand how Tinybird charges for different data operations. Remember, [UI usage is free](https://www.tinybird.co/docs/docs/get-started/plans/billing#exceptions) (Pipes, Playgrounds, Time Series - anywhere you can hit a "Run" button) as is anything on a [Free plan](https://www.tinybird.co/docs/docs/get-started/plans) so get started today for free and iterate ***fast***.



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

If you'd like assistance with your migration, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

- Set up a free Tinybird account and build a working prototype:[  Sign up here](https://www.tinybird.co/signup)  .
- Run through a quick example with your free account: Tinybird[  quick start](https://www.tinybird.co/docs/docs/get-started/quick-start)  .
- Read the[  billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing)   to understand plans and pricing on Tinybird.



---

URL: https://www.tinybird.co/docs/get-data-in/migrate/migrate-from-postgres
Last update: 2025-01-24T09:36:45.000Z
Content:
---
title: "Migrate from Postgres · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to migrate events from Postgres to Tinybird so that you can begin building performant, real-time analytics over your event data."
---




# Migrate from Postgres [¶](https://www.tinybird.co/docs/about:blank#migrate-from-postgres)

In this guide, you'll learn how to migrate events from Postgres to Tinybird so that you can begin building performant, real-time analytics over your event data.

Need to create a Tinybird account? It's free! [Start here](https://www.tinybird.co/signup).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You'll need a [free Tinybird account](https://www.tinybird.co/signup) and a Workspace.

## At a high level [¶](https://www.tinybird.co/docs/about:blank#at-a-high-level)

Postgres is an incredible general purpose database, and it can even be extended to support columnar functionality for analytics.

Tinybird is a data platform for data and engineering teams to solve complex real-time, operational, and user-facing analytics use cases at any scale, with end-to-end latency in milliseconds for streaming ingest and high QPS workloads.

It's a SQL-first analytics engine, purpose-built for the cloud, with real-time data ingest and full JOIN support. Native, managed ingest connectors make it easy to ingest data from a variety of sources. SQL queries can be published as production-grade, scalable REST APIs for public use or secured with JWTs.

Tinybird is a managed platform that scales transparently, requiring no cluster operations, shard management, or worrying about replicas.

See how Tinybird is used by industry-leading companies today in the [Customer Stories](https://www.tinybird.co/customer-stories) hub.

## Follow these steps to migrate from Postgres to Tinybird [¶](https://www.tinybird.co/docs/about:blank#follow-these-steps-to-migrate-from-postgres-to-tinybird)

Below you'll find an example walkthrough migrating 100M rows of events data from Postgres to Tinybird. You can apply the same workflow to your existing Postgres instance. If at any point you get stuck and would like assistance with your migration, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Slack Community](https://www.tinybird.co/docs/docs/community).

### The Postgres table [¶](https://www.tinybird.co/docs/about:blank#the-postgres-table)

Suppose you have a table in Postgres that looks like this:

postgres=# CREATE TABLE events (
    id SERIAL PRIMARY KEY,
    timestamp TIMESTAMPTZ NOT NULL,
    user_id TEXT NOT NULL,
    session_id TEXT NOT NULL,
    action TEXT NOT NULL,
    version TEXT NOT NULL,
    payload TEXT NOT NULL
); The table contains 100 million rows totalling about 15GB of data:

postgres=# SELECT pg_size_pretty(pg_relation_size('events')) AS size;
 size
-------
 15 GB
(1 row) The table stores website click events, including an unstructured JSON `payload` column.

### Setup [¶](https://www.tinybird.co/docs/about:blank#setup)

Within your Postgres, create a user with read only permissions over the table (or tables) you need to export:

postgres=# CREATE USER tb_read_user WITH PASSWORD '';
postgres=# GRANT CONNECT ON DATABASE test_db TO tb_read_user;
postgres=# GRANT USAGE ON SCHEMA public TO tb_read_user;
postgres=# GRANT SELECT ON TABLE events TO tb_read_user;
#### Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

To perform this migration, we'll be running a series of [Copy Jobs](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) to incrementally migrate the events from Postgres to Tinybird. We break it up into chunks so as to remain under the limits of both Tinybird and Postgres.

There are two limits to take into account:

1. [  Copy Pipe limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#copy-pipe-limits)   : Copy Pipes have a default max execution time of 20s for Build plans, 30s for Pro plans, 30m for Enterprise plans. If you're on a Build or Pro plan and need to temporarily extend your limits to perform the migration, please reach out to us at[  support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co)  .
2. The max execution time of queries in Postgres. This is controlled by the `statement_timeout`   setting. We recommendation that you set the value in Postgres equal or similar to the max execution time of the Copy Pipe in Tinybird. For this example, we'll use three minutes:

postgres=# ALTER ROLE tb_read_user SET statement_timeout = '180000';  -- 3 minutes
#### Create a local Tinybird project [¶](https://www.tinybird.co/docs/about:blank#create-a-local-tinybird-project)

Install [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/install) , then create a new Data Project:

export TB_ADMIN_TOKEN=
export TB_HOST=https://api.us-east.aws.tinybird.co #replace with your host

tb auth --host $TB_HOST --token $TB_ADMIN_TOKEN
tb init Create the target Data Source in Tinybird:

touch datasources/events.datasource Define a schema that matches your Postgres schema, keeping in mind that Tinybird may use different data types. For our example:

# datasources/events.datasource
SCHEMA >
    `id` Int32,
    `timestamp` DateTime64(6),
    `user_id` String,
    `session_id` String,
    `action` String,
    `version` String,
    `payload` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(timestamp)"
ENGINE_SORTING_KEY "timestamp, session_id, user_id" Push the Data Source to the Tinybird server:

tb push datasources/events.datasource
### Backfilling your existing Postgres data [¶](https://www.tinybird.co/docs/about:blank#backfilling-your-existing-postgres-data)

We're going to create a parameterized Copy Pipe to perform the initial backfill in chunks. We'll use a script to run the Copy Job on demand.

#### Storing secrets in Tinybird [¶](https://www.tinybird.co/docs/about:blank#storing-secrets-in-tinybird)

Start by adding two secrets to Tinybird using the [Environment Variables API](https://www.tinybird.co/docs/docs/api-reference/environment-variables-api) . This will prevent hard-coded credentials in your Copy Pipe.

Create one for your Postgres username:

curl \
  -X POST "${TB_HOST}/v0/variables" \
  -H "Authorization: Bearer ${TB_ADMIN_TOKEN}" \
  -d "type=secret" \
  -d "name=tb_read_user" \
  -d "value=tb_read_user" And one for the password:

curl \
  -X POST "${TB_HOST}/v0/variables" \
  -H "Authorization: Bearer ${TB_ADMIN_TOKEN}" \
  -d "type=secret" \
  -d "name=tb_read_password" \
  -d "value="
#### Define the Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#define-the-copy-pipe)

Create a new Pipe:

touch pipes/backfill_postgres.pipe And paste the following code, changing the url/port, name, and table name of your Postgres based on your specific setup:

NODE migrate
SQL >
    %
    SELECT *
    FROM postgresql(
        'https://your.postgres.url::port',
        'your_postgres_instance_name',
        'your_postgres_table name',
        {{tb_secret('tb_read_user')}},
        {{tb_secret('tb_read_password')}},
        'public'
    )
    WHERE
        timestamp > {{DateTime(from_date, '2020-01-01 00:00:00')}} --adjust based on your data
        AND timestamp <= {{DateTime(to_date, '2020-01-01 00:00:01')}} --use a small default range

TYPE COPY
TARGET_DATASOURCE events This uses the [PostgreSQL Table Function](https://www.tinybird.co/docs/docs/get-data-in/table-functions/postgresql) to select data from the remote Postgres table. It pushes the timestamp filters down to Postgres, incrementally querying your Postgres table and copying them into your `events` Data Source in Tinybird.

Push this Pipe to the server:

tb push pipes/backfill_postgres.pipe
### Backfill in one go [¶](https://www.tinybird.co/docs/about:blank#backfill-in-one-go)

Depending on the size of your Postgres table, you may be able to perform the migration in a single Copy Job.

For example, get the minimum timestamp from Postgres (and the current datetime):

postgres=# SELECT min(timestamp) FROM events;
          min
------------------------
 2023-01-01 00:00:00+00
(1 row)

❯ date -u +"%Y-%m-%d %H:%M:%S"
2024-08-29 10:20:57 And run the Copy Job with those parameters:

tb pipe copy run migrate_pg_to_events --param from_date="2023-01-01 00:00:00" --param to_date="2024-08-29 10:20:57" --wait --yes If it succeeds, you'll see something like this:

** Running migrate_pg_to_events
** Copy to 'events' job created: https://api.us-east.aws.tinybird.co/v0/jobs/4dd482f9-168b-44f7-a4c9-d1b64fc9665d
** Copying data   [████████████████████████████████████]  100%
** Data copied to 'events' And you'll be able to query the resulting Data Source:

tb sql "select count() from events"
-------------
|   count() |
-------------
| 100000000 |
-------------

tb sql "select count() as c, action from events group by action order by c asc" --stats
** Query took 0.228730096 seconds
** Rows read: 100,000,000
** Bytes read: 1.48 GB
-----------------------
|      c   | action   |
-----------------------
| 19996881 | logout   |
| 19997421 | signup   |
| 20000982 | purchase |
| 20001649 | view     |
| 20003067 | click    |
----------------------- Note that Copy operations in Tinybird are atomic, so a bulk backfill will either succeed or fail completely with some error.

For instance, if the `statement_timeout` in Postgres isn't large enough to export the table with a single query, you'll get an error like this:

** Copy to 'copy_migrate_events_from_pg' job created: https://api.us-east.aws.tinybird.co/v0/jobs/ec58749a-f4c3-4302-9236-f8036f0cb67b
** Copying data
Error:
** Failed creating copy job:
** Error while running job: There was a problem while copying data: [Error] Query cancelled due to statement timeout in postgres. Make sure you use a user with a proper statement timeout to run this type of query. In this case you can try to increaste the `statement_timeout` or try the backfilling in chunks. As a reference, copying 100M rows from Postgres to Tinybird takes about 150s if Postgres and Tinybird are in the same cloud and region.

The Tinybird PostgreSQL Table Function uses internally a PostgreSQL `COPY TO` statement. You can tweak some other settings in Postgres if necessary, but usually it's not needed, so refer to your Postgres provider or admin.

### Backfilling in chunks [¶](https://www.tinybird.co/docs/about:blank#backfilling-in-chunks)

If you find that you're hitting the limits of either your Postgres or Tinybird's Copy Pipes, you can backfill in chunks.

First of all, make sure your Postgres table is indexed by the column you are filtering on, in this case `timestamp`:

postgres=# CREATE INDEX idx_events_timestamp ON events (timestamp);
postgres=# VACUUM ANALYZE events; And make sure a query like the one sent from Tinybird will use the indexes (see the Index Scan below):

postgres=# explain select * from events where timestamp > '2024-01-01 00:00:00' and timestamp <= '2024-01-02 00:00.00';
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_events_timestamp on events  (cost=0.57..607150.89 rows=151690 width=115)
   Index Cond: (("timestamp" > '2024-01-01 00:00:00+00'::timestamp with time zone) AND ("timestamp" <= '2024-01-02 00:00:00+00'::timestamp with time zone))
 JIT:
   Functions: 2
   Options: Inlining true, Optimization true, Expressions true, Deforming true
(5 rows) Then run multiple Copy jobs, adjusting the amount of data copied to stay within your Postgres statement timeout and Tinybird max execution time. This is a trial and error process depending on the granularity of data.

For example, here's a migration script that first tries a full backfill, and if it fails uses daily chunks:

#!/bin/bash


HOST="YOUR_TB_HOST"
TOKEN="YOUR_TB_TOKEN"
PIPE_NAME="backfill_postgres"
FROM_DATE="2023-01-01 00:00:00"
TO_DATE="2024-08-31 00:00:00"
LOG_FILE="pipe_copy.log"

run_command() {
    local from_date="$1"
    local to_date="$2"
    echo "Copying from $from_date to $to_date" | tee -a $LOG_FILE

    if output=$(tb --host $HOST --token $TOKEN pipe copy run $PIPE_NAME --param from_date="$from_date" --param to_date="$to_date" --wait --yes 2>&1); then
        echo "Success $from_date - $to_date" | tee -a $LOG_FILE
        return 0
    else
        echo "Error $from_date - $to_date" | tee -a $LOG_FILE
        echo "Error detail: $output" | tee -a $LOG_FILE
        return 1
    fi
}

iterate_chunks() {
    local from_date="$1"
    local to_date="$2"

    local current_from="$from_date"
    local next_to=""

    while [[ "$(date -d "$current_from" +"%s")" -lt "$(date -d "$to_date" +"%s")" ]]; do
        # End of current day (23:59:59)
        next_to=$(date -d "$current_from +1 day -1 second" +"%Y-%m-%d")" 23:59:59"
        # Adjust next_to if it's bigger than to_date
        if [[ "$(date -d "$next_to" +"%s")" -ge "$(date -d "$to_date" +"%s")" ]]; then
            next_to="$to_date"
        fi

        # Create copy job for one single day
        if ! run_command "$current_from" "$next_to"; then
            echo "Error processing $current_from to $next_to"
            return 1
        fi

        # Go to next day (starting at 00:00:00)
        current_from=$(date -d "$(date -d "$current_from" +'%Y-%m-%d') +1 day $(date -d "$current_from" +'%H:%M:%S')" +'%Y-%m-%d %H:%M:%S')
    done
}

# Step 1: Try full backfill
echo "Running full backfill..." | tee -a $LOG_FILE

if ! run_command "$FROM_DATE" "$TO_DATE"; then
    echo "Full backfill failed, iterating in daily chunks..." | tee -a $LOG_FILE
    iterate_chunks "$FROM_DATE" "$TO_DATE"
fi

echo "Process completed." | tee -a $LOG_FILE Using either a full backfill or backfilling in chunks, you can successfully migrate your data from Postgres to Tinybird.

### Syncing new events from Postgres to Tinybird [¶](https://www.tinybird.co/docs/about:blank#syncing-new-events-from-postgres-to-tinybird)

The next step is keeping your Tinybird Data Source in sync with events in your Postgres as new events arrive.

The steps below will show you how to use Tinybird's PostgreSQL Table Function and scheduled Copy Jobs to continually sync data from Postgres to Tinybird, however, you should consider sending future events Tinybird directly using either the [Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api) or another streaming Data Source connector, as this will be more resource efficient (and more real-time).

#### Create the incremental Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#create-the-incremental-copy-pipe)

Create another Copy Pipe to perform the incremental syncs:

touch pipes/sync_events_from_pg.pipe Paste in this code, again updating your Postgres details as well as the desired schedule to sync. Note the Copy limits apply here.

NODE sync_from_pg
SQL >
    %
    SELECT *
    FROM postgresql(
        'https://your.postgres.url::port',
        'your_postgres_instance_name',
        'your_postgres_table name',
        {{tb_secret('tb_read_user')}},
        {{tb_secret('tb_read_password')}},
        'public'
    )
    WHERE timestamp > (SELECT max(timestamp) FROM events)

TYPE COPY
TARGET_DATASOURCE events
COPY_SCHEDULE */5 * * * * Push this to the Tinybird server:

tb push pipes/sync_events_from_pg.pipe It's important to first complete the backfill operation before pushing the sync Pipe. The sync Pipe uses the latest timestamp in the Tinybird copy to perform a filtered select from Postgres. Failure to backfill will result in a full scan of your Postgres table on your configured schedule.

Once you've pushed this Pipe, Tinybird will sync with your Postgres updates based on the schedule you set.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

If you'd like assistance with your migration, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

- Set up a free Tinybird account and build a working prototype:[  Sign up here](https://www.tinybird.co/signup)  .
- Run through a quick example with your free account: Tinybird[  quick start](https://www.tinybird.co/docs/docs/get-started/quick-start)  .
- Read the[  billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing)   to understand plans and pricing on Tinybird.



---

URL: https://www.tinybird.co/docs/get-data-in/migrate/migrate-from-doublecloud
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Migrate from DoubleCloud · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to migrate from DoubleCloud to Tinybird, and the overview of how to quickly & safely recreate your setup."
---




# Migrate from DoubleCloud [¶](https://www.tinybird.co/docs/about:blank#migrate-from-doublecloud)

In this guide, you'll learn how to migrate from DoubleCloud to Tinybird, and the overview of how to quickly & safely recreate your setup.

DoubleCloud, a managed data services platform that offers ClickHouse® as a service, is [shutting down operations](https://double.cloud/blog/posts/2024/10/doublecloud-final-update/) . As of October 1, 2024 you can't create new DoubleCloud accounts, and all existing DoubleCloud services must be migrated by March 1, 2025.

Tinybird offers a solution that can be a suitable alternative for existing users of DoubleCloud's ClickHouse service.

Follow this guide to learn two approaches for migrating data from your DoubleCloud instance to Tinybird:

1. Option 1: Use the S3 table function to export data from DoubleCloud Managed ClickHouse to Amazon S3, then use the Tinybird S3 Connector to import data from S3.
2. Option 2: Export your ClickHouse tables locally, then import files into Tinybird using the Datasources API.

Wondering how to create a Tinybird account? It's free! [Start here](https://www.tinybird.co/signup) . Need DoubleCloud migration assistance? Please [contact us](https://www.tinybird.co/doublecloud).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You don't need an active Tinybird Workspace to read through this guide, but it's good idea to understand the foundational concepts and how Tinybird integrates with your team. If you're new to Tinybird, read the [team integration guide](https://www.tinybird.co/docs/docs/get-started/administration/team-integration-governance).

## At a high level [¶](https://www.tinybird.co/docs/about:blank#at-a-high-level)

Tinybird is a great alternative to DoubleCloud's managed ClickHouse implementation.

Tinybird is a data platform built for data and engineering teams to solve complex real-time, operational, and user-facing analytics use cases at any scale, with end-to-end latency in milliseconds for streaming ingest and high QPS workloads.

It offers the same or comparable performance as DoubleCloud, with additional features such as native, managed ingest connectors, multi-node SQL notebooks, and scalable REST APIs for public use or secured with JWTs.

Tinybird is a managed platform that scales transparently, requiring no cluster operations, shard management or worrying about replicas.

See how Tinybird is used by industry-leading companies today in the [Customer Stories](https://www.tinybird.co/customer-stories).

## Migrate from DoubleCloud to Tinybird using Amazon S3 [¶](https://www.tinybird.co/docs/about:blank#migrate-from-doublecloud-to-tinybird-using-amazon-s3)

In this approach, you'll use the `s3` table function to export tables to an Amazon S3 bucket, and then import them into Tinybird with the S3 Connector.

This guide assumes that you already have the necessary IAM Roles with the necessary permissions to write to (from DoubleCloud) and read from (to Tinybird) the S3 bucket.

### Export your table to Amazon S3 [¶](https://www.tinybird.co/docs/about:blank#export-your-table-to-amazon-s3)

In this guide, we're using a table on our DoubleCloud ClickHouse Cluster called `timeseriesdata` . The data has 3 columns and 1M rows.

Export your table to Amazon S3 In this guide, we're using a table on our DoubleCloud ClickHouse Cluster called `timeseriesdata` . The data has 3 columns and 1M rows.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-1.png&w=3840&q=75)

<-figcaption->
Example timeseries data table in DoubleCloud





You can export data in your DoubleCloud ClickHouse tables to Amazon S3 with the `s3` table function.

Note: If you don't want to expose your AWS credentials in the query, use a named collection.

INSERT INTO FUNCTION s3(
    'https://tmp-doublecloud-migration.s3.us-east-1.amazonaws.com/exports/timeseriesdata.csv',
    'AWS_ACCESS_KEY_ID',
    'AWS_SECRET_ACCESS_KEY',
    'CSV'
)
SELECT *
FROM timeseriesdata
SETTINGS s3_create_new_file_on_insert = 1
### Import to Tinybird with the S3 Connector [¶](https://www.tinybird.co/docs/about:blank#import-to-tinybird-with-the-s3-connector)

Once your table is exported to Amazon S3, import it to Tinybird using the [Amazon S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3).



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-2.png&w=3840&q=75)

<-figcaption->
Select the S3 Connector in the Tinybird UI.





The basic steps for using the S3 Connector are:

1. Define an S3 Connection with IAM Policy and Role that allow Tinybird to read from S3. Tinybird will automatically generated the JSON for these policies.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-3.png&w=3840&q=75)

<-figcaption->
Create an S3 Connection with automatically generated IAM policies





1. Supply the file URI (with wildcards as necessary) to define the file(s) containing the contents of your ClickHouse table(s).



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-4.png&w=3840&q=75)

<-figcaption->
Specify the file URI for the files containing your ClickHouse tables





1. Create an On Demand (one-time) sync.
2. Define the schema of the resulting table in Tinybird. You can do this within the S3 Connector UI…



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-5.png&w=3840&q=75)

<-figcaption->
Define the schema of your tables within the Tinybird UI





...or by creating a .datasource file and pushing it to Tinybird. An example .datasource file for `timeseriesdata` table to match the DoubleCloud schema and create the import job from the existing S3 Connection would look like this:

SCHEMA >
    `tank_id` String,
    `volume` Float32,
    `usage` Float32

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tank_id"

IMPORT_SERVICE 's3_iamrole'
IMPORT_CONNECTION_NAME 'DoubleCloudS3'
IMPORT_BUCKET_URI 's3://tmp-doublecloud-migration/timeseriesdata.csv'
IMPORT_STRATEGY 'append'
IMPORT_SCHEDULE '@on-demand'
1. Tinybird will then create and run a batch import job to ingest the data from Amazon S3 and create a new table that matches your table in DoubleCloud. You can monitor the job from the `datasource_ops_log`   Service Data Source.

## Migrate from DoubleCloud to Tinybird using local exports [¶](https://www.tinybird.co/docs/about:blank#migrate-from-doublecloud-to-tinybird-using-local-exports)

Depending on the size of your tables, you might be able to simply export your tables to a local file using `clickhouse-client` and ingest them to Tinybird directly.

### Export your tables from DoubleCloud using clickhouse-client [¶](https://www.tinybird.co/docs/about:blank#export-your-tables-from-doublecloud-using-clickhouse-client)

First, use `clickhouse-client` to export your tables into local files. Depending on the size of your data, you can choose to compress as necessary. Tinybird can ingest CSV (including Gzipped CSV), NDJSON, and Parquet files.

./clickhouse client --host your_doublecloud_host --port 9440 --secure --user your_doublecloud_user --password your_doublecloud_password --query "SELECT * FROM timeseriesdata" --format CSV > timeseriesdata.csv
### Import your files to Tinybird [¶](https://www.tinybird.co/docs/about:blank#import-your-files-to-tinybird)

You can drag and drop files into the Tinybird UI…



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmigrate-from-doublecloud-6.png&w=3840&q=75)

<-figcaption->
Drag and drop a file into the Tinybird UI to create a file-based Data Source





or upload them using the Tinybird CLI:

tb datasource generate timeseriesdata.csv
tb push datasources/timeseriesdata.datasource
tb datasource append timeseriesdata timeseriesdata.csv Note that Tinybird will automatically infer the appropriate schema from the supplied file, but you may need to change the column names, data types, table engine, and sorting key to match your table in DoubleCloud.

## Migration support [¶](https://www.tinybird.co/docs/about:blank#migration-support)

If your migration is more complex, involving many or very large tables, materialized views + populates, or other complex logic, please [contact us](https://www.tinybird.co/doublecloud) and we will assist with your migration.

## Tinybird Pricing vs DoubleCloud [¶](https://www.tinybird.co/docs/about:blank#tinybird-pricing-vs-doublecloud)

Tinybird's Free plan is free, with no time limit or credit card required. The Free plan includes 10 GB of data storage (compressed) and 1,000 published API requests per day.

Tinybird's paid plans are available with both infrastructure-based pricing and usage-based pricing. DoubleCloud customers will likely be more familiar with infrastructure-based pricing.

For more information about infrastructure-based pricing and to get a quote based on your existing DoubleCloud cluster, [contact us](https://www.tinybird.co/doublecloud).

If you are interested in usage-based pricing, you can learn more about [usage-based billing here](https://www.tinybird.co/docs/docs/get-started/plans/billing).

### ClickHouse Limits [¶](https://www.tinybird.co/docs/about:blank#clickhouse-limits)

Note that Tinybird takes a different approach to ClickHouse deployment than DoubleCloud. Rather than provide a full interface to a hosted ClickHouse cluster, Tinybird provides a serverless ClickHouse implementation and abstracts the database interface via our [APIs](https://www.tinybird.co/docs/docs/api-reference) , UI, and CLI, only exposing the SQL Editor within our [Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) interface.

Additionally, not all ClickHouse SQL functions, data types, and table engines are supported out of the box. You can find a full list of [supported engines and settings here](https://www.tinybird.co/docs/docs/get-data-in/data-sources#supported-engines-settings) . If your use case requires engines or settings that aren't listed, please [contact us](https://www.tinybird.co/doublecloud).

## Useful resources [¶](https://www.tinybird.co/docs/about:blank#useful-resources)

Migrating to a new tool, especially at speed, can be challenging. Here are some helpful resources to get started on Tinybird:

- [  Read how Tinybird compares to ClickHouse (especially ClickHouse Cloud)](https://www.tinybird.co/blog-posts/tinybird-vs-clickhouse)  .
- [  Read how Tinybird compares to other Managed ClickHouse offerings](https://www.tinybird.co/blog-posts/managed-clickhouse-options)  .
- Join our[  Slack Community](https://www.tinybird.co/community)   for help understanding Tinybird concepts.
- [  Contact us](https://www.tinybird.co/doublecloud)   for migration assistance.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

If you'd like assistance with your migration, [contact us](https://www.tinybird.co/doublecloud).

- Set up a free Tinybird account and build a working prototype:[  Sign up here](https://www.tinybird.co/signup)  .
- Run through a quick example with your free account: Tinybird[  quick start](https://www.tinybird.co/docs/docs/get-started/quick-start)  .
- Read the[  billing docs](https://www.tinybird.co/docs/docs/get-started/plans/billing)   to understand plans and pricing on Tinybird.

Tinybird is not affiliated with, associated with, or sponsored by ClickHouse, Inc. ClickHouse® is a registered trademark of ClickHouse, Inc.

---

URL: https://www.tinybird.co/docs/get-data-in/ingest-apis/events-api
Last update: 2025-01-28T11:47:33.000Z
Content:
---
title: "Events API · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird Events API"
---




# Events API [¶](https://www.tinybird.co/docs/about:blank#events-api)

The Events API enables high-throughput streaming ingestion into Tinybird from an easy-to-use HTTP API.

This page gives examples of how to use the Events API to perform various tasks. For more information, read the [Events API Reference](https://www.tinybird.co/docs/docs/api-reference/events-api) docs.

## Send individual JSON events [¶](https://www.tinybird.co/docs/about:blank#send-individual-json-events)

You can send individual JSON events to the Events API by including the JSON event in the Request body.

Supported event formats are JSON and NDJSON (newline delimited JSON).

For example, to send an individual NDJSON event using cURL:

##### Sending individual NDJSON events

curl \
-H "Authorization: Bearer " \
-d '{"date": "2020-04-05 00:05:38", "city": "Chicago"}' \
'https://api.tinybird.co/v0/events?name=events_test' The `name` parameter defines the name of the Data Source in which to insert events. If the Data Source doesn't exist, Tinybird creates the Data Source by inferring the schema of the JSON.

The Token used to send data to the Events API needs the appropriate scopes. To append data to an existing Data Source, the `DATASOURCE:APPEND` scope is required. If the Data Source doesn't already exist, the `DATASOURCE:CREATE` scope is required to create the new Data Source.

### Define the schema [¶](https://www.tinybird.co/docs/about:blank#define-the-schema)

Defining your schema allows you to set data types, sorting key, TTL and more. Read the [schema definition docs here](https://www.tinybird.co/docs/docs/get-data-in#define-the-schema-yourself).

## Send batches of JSON events [¶](https://www.tinybird.co/docs/about:blank#send-batches-of-json-events)

Sending batches of events enables you to achieve much higher total throughput than sending individual events.

You can send batches of JSON events to the Events API by formatting the events as NDJSON (newline delimited JSON). Each individual JSON event should be separated by a newline ( `\n` ) character.

##### Sending batches of JSON events

curl \
-H "Authorization: Bearer " \
-d $'{"date": "2020-04-05 00:05:38", "city": "Chicago"}\n{"date": "2020-04-05 00:07:22", "city": "Madrid"}\n' \
'https://api.tinybird.co/v0/events?name=events_test' The `name` parameter defines the name of the Data Source in which to insert events. If the Data Source doesn't exist, Tinybird creates the Data Source by inferring the schema of the JSON.

The Token used to send data to the Events API must have the appropriate scopes. To append data to an existing Data Source, the `DATASOURCE:APPEND` scope is required. If the Data Source doesn't already exist, the `DATASOURCE:CREATE` scope is required to create the new Data Source.

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

The Events API delivers a default capacity of:

- Up to 1000 requests/second per Data Source
- Up to 20MB/s per Data Source
- Up to 10MB per request per Data Source. If you reach this limit you'll receive a `HTTP 413 - Request Entity Too Large`   error. For larger requests, we recommend splitting them into multiple smaller ones to ensure smooth data ingestion.

Throughput beyond these limits is offered as best-effort.

The Events API is able to scale beyond these limits. If you are reaching these limits, contact [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co).

**Rate limit headers**

| Header Name | Description |
| --- | --- |
| `X-RateLimit-Limit` | The maximum number of requests you're permitted to make in the current limit window. |
| `X-RateLimit-Remaining` | The number of requests remaining in the current rate limit window. |
| `X-RateLimit-Reset` | The time in seconds after the current rate limit window resets. |
| `Retry-After` | The time to wait before making a another request. Only present on 429 responses. |

Events API is a high-throughput streaming ingestion and as a distributed system, the values in these headers are offered as best-effort.

### Check the payload size [¶](https://www.tinybird.co/docs/about:blank#check-the-payload-size)

To avoid hitting the request limit size, you can check your payload size before sending. For example:

- shell
- python
- js

echo '{"date": "2020-04-05", "city": "Chicago"}' | wc -c | awk '{print $1/1024/1024 " MB"}'
## Compression [¶](https://www.tinybird.co/docs/about:blank#compression)

NDJSON events sent to the Events API can be compressed with Gzip. However, it's only recommended to do this when necessary, such as when you have big events that are grouped into large batches. Compressing events adds overhead to the ingestion process, which can introduce latency, although it's typically minimal.

Here is an example of sending a JSON event compressed with Gzip from the command line:

echo '{"timestamp":"2022-10-27T11:43:02.099Z","transaction_id":"8d1e1533-6071-4b10-9cda-b8429c1c7a67","name":"Bobby Drake","email":"bobby.drake@pressure.io","age":42,"passport_number":3847665,"flight_from":"Barcelona","flight_to":"London","extra_bags":1,"flight_class":"economy","priority_boarding":false,"meal_choice":"vegetarian","seat_number":"15D","airline":"Red Balloon"}' | gzip > body.gz 

curl \
    -X POST 'https://api.tinybird.co/v0/events?name=gzip_events_example' \
    -H "Authorization: Bearer ." \
    -H "Content-Encoding: gzip" \
    --data-binary @body.gz
## Write acknowledgements [¶](https://www.tinybird.co/docs/about:blank#write-acknowledgements)

When you send data to the Events API, you usually receive a `HTTP202` response, which indicates that the request was successful - however it doesn't confirm that the data has been committed into the underlying database. This is useful when guarantees on writes aren't strictly necessary. Typically, it should take under 2 seconds to receive a response from the Events API in this case.

curl \
    -X POST 'https://api.tinybird.co/v0/events?name=events_example' \
    -H "Authorization: Bearer " \
    -d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'

< HTTP/2 202 
< content-type: application/json
< content-length: 42
< 
{"successful_rows":2,"quarantined_rows":0} However, if your use case requires absolute guarantees that data is committed, use the `wait` parameter.

The `wait` parameter is a boolean that accepts a value of `true` or `false` . A value of `false` is the default behavior, equivalent to omitting the parameter entirely.

Using `wait=true` with your request will ask the Events API to wait for acknowledgement that the data you sent has been committed into the underlying database. You will receive a `HTTP200` response that confirms data has been committed.

Note that adding `wait=true` to your request can result in a slower response time. Use a time-out of at least 10 seconds when waiting for the response.

For example:

curl \
    -X POST 'https://api.tinybird.co/v0/events?name=events_example&wait=true' \
    -H "Authorization: Bearer " \
    -d $'{"timestamp":"2022-10-27T11:43:02.099Z"}'

< HTTP/2 200 
< content-type: application/json
< content-length: 42
< 
{"successful_rows":2,"quarantined_rows":0} It is good practice to log your requests to, and responses from, the Events API. This will help to give you visibility into any failures for reporting or recovery.



---

URL: https://www.tinybird.co/docs/get-data-in/ingest-apis/datasource-api
Last update: 2024-12-18T09:46:02.000Z
Content:
---
title: "Data Sources API · Tinybird Docs"
theme-color: "#171612"
description: "Use the Data Sources API to create and manage your Data Sources as well as importing data into them."
---




# Data Sources API [¶](https://www.tinybird.co/docs/about:blank#data-sources-api)

Use Tinybird's Data Sources API to import files into your Tinybird Data Sources. With the Data Sources API you can use files to create new Data Sources, and append data to, or replace data from, an existing Data Source. See [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources).

The following examples show how to use the Data Sources API to perform various tasks. See the [Data Sources API Reference](https://www.tinybird.co/docs/docs/api-reference/datasource-api) for more information.

## Import a file into a new Data Source [¶](https://www.tinybird.co/docs/about:blank#import-a-file-into-a-new-data-source)

Tinybird can create a Data Source from a file. This operation supports CSV, NDJSON, and Parquet files. You can create a Data Source from local or remote files.

Automatic schema inference is supported for CSV files, but isn't supported for NDJSON or Parquet files.

### CSV files [¶](https://www.tinybird.co/docs/about:blank#csv-files)

CSV files must follow these requirements:

- One line per row
- Comma-separated

Tinybird supports Gzip compressed CSV files with .csv.gz extension.

The Data Sources API automatically detects and optimizes your column types, so you don't need to manually define a schema. You can use the `type_guessing=false` parameter to force Tinybird to use `String` for every column.

CSV headers are optional. When creating a Data Source from a CSV file, if your file contains a header row, Tinybird uses the header to name your columns. If no header is present, your columns receive default names with an incrementing number.

When appending a CSV file to an existing Data Source, if your file has a header, Tinybird uses the headers to identify the columns. If no header is present, Tinybird uses the order of columns. If the order of columns in the CSV file is always the same, you can omit the header line.

For example, to create a new Data Source from a local file using cURL:

##### Creating a Data Source from a local CSV file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=my_datasource_name" \
-F csv=@local_file.csv From a remote file:

##### Creating a Data Source from a remote CSV file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=my_datasource_name" \
-d url='https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2018-12.csv' When importing a remote file from a URL, the response contains the details of an import Job. To see the status of the import, use the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api).

### NDJSON and Parquet files [¶](https://www.tinybird.co/docs/about:blank#ndjson-and-parquet-files)

The Data Sources API doesn't support automatic schema inference for NDJSON and Parquet files. You must specify the `schema` parameter with a valid schema to parse the files.

The schema for both NDJSON and Parquet files uses [JSONPaths](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths) to identify columns in the data. You can add default values to the schema.

Tinybird supports compressed NDJSON and Parquet files with .ndjson.gz and .parquet.gz extensions.

You can use the [Analyze API](https://www.tinybird.co/docs/about:blank#generate-schemas-with-the-analyze-api) to automatically generate a schema definition from a file.

For example, assume your NDJSON or Parquet data looks like this:

##### Simple NDJSON data example

{ "id": 123, "name": "Al Brown"} Your schema definition must provide the JSONPath expressions to identify the columns `id` and `name`:

##### Simple NDJSON schema definition

id Int32 `json:$.id`,
name String `json:$.name` To create a new Data Source from a local file using cURL, you must URL encode the Schema as a query parameter. The following examples use NDJSON. To use Parquet, adjust the `format` parameter to `format=parquet`:

##### Creating a Data Source from a local NDJSON file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=events&mode=create&format=ndjson&schema=id%20Int32%20%60json%3A%24.id%60%2C%20name%20String%20%60json%3A%24.name%60" \
-F ndjson=@local_file.ndjson From a remote file:

##### Creating a Data Source from a remote NDJSON file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=events&mode=create&format=ndjson" \
--data-urlencode "schema=id Int32 \`json:$.id\`, name String \`json:$.name\`" \
-d url='http://example.com/file.json' Note the escape characters in this example are only required due to backticks in cURL.

When importing a remote file from a URL, the response contains the details of an import Job. To see the status of the import, you must the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api).

To add default values to the schema, use the `DEFAULT` parameter after the JSONPath expressions. For example:

##### Simple NDJSON schema definition with default values

id Int32 `json:$.id` DEFAULT 1,
name String `json:$.name` DEFAULT 'Unknown'
## Create an NDJSON Data Source from a schema using JSON type [¶](https://www.tinybird.co/docs/about:blank#create-an-ndjson-data-source-from-a-schema-using-json-type)

You can create an NDJSON Data Source using the JSON column type through the `schema` parameter. For example:

curl \\
            -H "Authorization: Bearer " \\
            -X POST "https://api.tinybird.co/v0/datasources" \\
            -d "name=example" \\
            -d "format=ndjson" \\
            -d "schema=data JSON `json:$`"
## Append a file into an existing Data Source [¶](https://www.tinybird.co/docs/about:blank#append-a-file-into-an-existing-data-source)

If you already have a Data Source, you can append the contents of a file to the existing data.

This operation supports CSV, NDJSON, and Parquet files.

You can append data from local or remote files.

When appending CSV files, you can improve performance by excluding the CSV Header line. However, in this case, you must ensure the CSV columns are ordered. If you can't guarantee the order of column in your CSV, include the CSV Header.

For example, to append data into an existing Data Source from a local file using cURL:

##### Appending data to a Data Source from a local CSV file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=append&name=my_datasource_name" \
-F csv=@local_file.csv From a remote file:

##### Appending data to a Data Source from a remote CSV file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=append&name=my_datasource_name" \
-d url='https://s3.amazonaws.com/nyc-tlc/trip+data/fhv_tripdata_2018-12.csv' If the Data Source has dependent Materialized Views, data is appended in cascade.

## Replace data in an existing Data Source with a file [¶](https://www.tinybird.co/docs/about:blank#replace-data-in-an-existing-data-source-with-a-file)

If you already have a Data Source, you can replace existing data with the contents of a file. You can replace all data or a selection of data.

This operation supports CSV, NDJSON, and Parquet files.

You can replace with data from local or remote files.

For example, to replace all the data in a Data Source with data from a local file using cURL:

##### Replacing a Data Source from a URL

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=replace&name=data_source_name&format=csv" \
-F csv=@local_file.csv From a remote file:

##### Replacing a Data Source from a URL

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=replace&name=data_source_name&format=csv" \
--data-urlencode "url=http://example.com/file.csv" Rather than replacing all data, you can also replace specific partitions of data. This operation is atomic.

To do this, use the `replace_condition` parameter. This parameter defines the filter that's applied, where all matching rows are deleted before finally ingesting the new file. Only the rows matching the condition are ingested.

If the source file contains rows that don't match the filter, the rows are ignored.

Replacements are made by partition, so make sure that the `replace_condition` filters on the partition key of the Data Source.

To replace filtered data in a Data Source with data from a local file using cURL, you must URL encode the `replace_condition` as a query parameter. For example:

##### Replace filtered data in a Data Source with data from a local file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=replace&name=data_source_name&format=csv&replace_condition=my_partition_key%20%3E%20123" \
-F csv=@local_file.csv From a remote file:

##### Replace filtered data in a Data Source with data from a remote file

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?mode=replace&name=data_source_name&format=csv" \
-d replace_condition='my_partition_key > 123' \
--data-urlencode "url=http://example.com/file.csv" All the dependencies of the Data Source, for example Materialized Views, are recalculated so that your data is consistent after the replacement. If you have n-level dependencies, they're also updated by this operation.

Taking the example `A --> B --> C` , if you replace data in A, Data Sources B and C are automatically updated. The Partition Key of Data Source C must also be compatible with Data Source A.

You can find more examples in the [Replace and delete data](https://www.tinybird.co/docs/docs/get-data-in/data-operations/replace-and-delete-data#replace-data-selectively) guide.

Although replacements are atomic, Tinybird can't assure data consistency if you continue appending data to any related Data Source at the same time the replacement takes place. The new incoming data is discarded.

## Creating an empty Data Source from a schema [¶](https://www.tinybird.co/docs/about:blank#creating-an-empty-data-source-from-a-schema)

When you want to have more granular control about the Data Source schema, you can manually create the Data Source with a specified schema.

For example, to create an empty Data Source with a set schema using cURL:

##### Create an empty Data Source with a set schema

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=stocks" \
-d "schema=symbol String, date Date, close Float32" To create an empty Data Source, you must pass a `schema` with your desired column names and types and leave the `url` parameter empty.

## Generate schemas with the Analyze API [¶](https://www.tinybird.co/docs/about:blank#generate-schemas-with-the-analyze-api)

The Analyze API can analyze a given NDJSON or Parquet file to produce a valid schema. The column names, types, and JSONPaths are inferred from the file.

For example, to analyze a local NDJSON file using cURL:

##### analyze an NDJSON file to get a valid schema

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/analyze" \
-F "ndjson=@local_file_path" The response contains a `schema` field that can be used to create your Data Source. For example:

##### Successful analyze response

{
    "analysis": {
        "columns": [{
                "path": "$.a_nested_array.nested_array[:]",
                "recommended_type": "Array(Int16)",
                "present_pct": 3,
                "name": "a_nested_array_nested_array"
            },
            {
                "path": "$.an_array[:]",
                "recommended_type": "Array(Int16)",
                "present_pct": 3,
                "name": "an_array"
            },
            {
                "path": "$.field",
                "recommended_type": "String",
                "present_pct": 1,
                "name": "field"
            },
            {
                "path": "$.nested.nested_field",
                "recommended_type": "String",
                "present_pct": 1,
                "name": "nested_nested_field"
            }
        ],
        "schema": "a_nested_array_nested_array Array(Int16) `json:$.a_nested_array.nested_array[:]`, an_array Array(Int16) `json:$.an_array[:]`, field String `json:$.field`, nested_nested_field String `json:$.nested.nested_field`"
    },
    "preview": {
        "meta": [{
                "name": "a_nested_array_nested_array",
                "type": "Array(Int16)"
            },
            {
                "name": "an_array",
                "type": "Array(Int16)"
            },
            {
                "name": "field",
                "type": "String"
            },
            {
                "name": "nested_nested_field",
                "type": "String"
            }
        ],
        "data": [{
            "a_nested_array_nested_array": [
                1,
                2,
                3
            ],
            "an_array": [
                1,
                2,
                3
            ],
            "field": "test",
            "nested_nested_field": "bla"
        }],
        "rows": 1,
        "statistics": {
            "elapsed": 0.00032175,
            "rows_read": 2,
            "bytes_read": 142
        }
    }
}
## Error handling [¶](https://www.tinybird.co/docs/about:blank#error-handling)

Most errors return an HTTP Error code, for example `HTTP 4xx` or `HTTP 5xx`.

However, if the imported file is valid, but some rows failed to ingest due to an incompatible schema, you might still receive an `HTTP 200` . In this case, the Response body contains two keys, `invalid_lines` and `quarantine_rows` , which tell you how many rows failed to ingest.

Additionally, an `error` key is present with an error message.

##### Successful ingestion with errors

{
"import_id": "e9ae235f-f139-43a6-7ad5-a1e17c0071c2",
"datasource": {
"id": "t_0ab7a11969fa4f67985cec481f71a5c2",
"name": "your_datasource_name",
"cluster": null,
"tags": {},
"created_at": "2019-03-12 17:45:04",
"updated_at": "2019-03-12 17:45:04",
"statistics": {
"bytes": 1397,
"row_count": 4
},
"replicated": false,
"version": 0,
"project": null,
"used_by": []
},
"error": "There was an error with file contents: 2 rows in quarantine and 2 invalid lines",
"quarantine_rows": 2,
"invalid_lines": 2
}

---

URL: https://www.tinybird.co/docs/get-data-in/guides/python-sdk
Last update: 2024-12-18T09:46:02.000Z
Content:
---
title: "Send Python logs to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Send your Python logs to Tinybird using the standard logging library and Tinybird Python SDK."
---




# Send Python logs to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-python-logs-to-tinybird)

You can send logs from a Python application or service to Tinybird using the standard Python logging library and the [tinybird-python-sdk](https://pypi.org/project/tinybird-python-sdk/).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To use the Tinybird Python SDK you need Python 3.11 or higher.

## Configure the logging handler [¶](https://www.tinybird.co/docs/about:blank#configure-the-logging-handler)

First, configure a Tinybird logging handler in your application. For example:

import logging
from multiprocessing import Queue
from tb.logger import TinybirdLoggingQueueHandler

logger = logging.getLogger('your-logger-name')
handler = TinybirdLoggingHandler(, , 'your-app-name')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler) Each time you call the logger, the SDK sends an event to the `tb_logs` Data Source in your Workspace.

To configure the Data Source name, initialize the `TinybirdLoggingHandler` like this:

handler = TinybirdLoggingHandler(, , 'your-app-name', ds_name="your_tb_ds_name")
## Non-blocking logging [¶](https://www.tinybird.co/docs/about:blank#non-blocking-logging)

If you want to avoid blocking the main thread, use a queue to send the logs to a different thread. For example:

import logging
from multiprocessing import Queue
from tb.logger import TinybirdLoggingQueueHandler
from dotenv import load_dotenv

load_dotenv()
TB_API_URL = os.getenv("")
TB_WRITE_TOKEN = os.getenv("")

logger = logging.getLogger('your-logger-name')
handler = TinybirdLoggingQueueHandler(Queue(-1), TB_API_URL, TB_WRITE_TOKEN, 'your-app-name', ds_name="your_tb_ds_name")
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

---

URL: https://www.tinybird.co/docs/get-data-in/guides/postgres-cdc-with-redpanda-connect
Last update: 2025-01-22T11:12:31.000Z
Content:
---
title: "Postgres CDC with Redpanda Connect · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to ingest data from a Postgres database using Redpanda Connect and the Postgres CDC input."
---




# PostgreSQL CDC with Redpanda Connect [¶](https://www.tinybird.co/docs/about:blank#postgresql-cdc-with-redpanda-connect)

[Redpanda Connect](https://www.redpanda.com/blog/redpanda-connect) is an ecosystem of high-performance streaming connectors that serves as a simplified and powerful alternative to Kafka Connect.

Tinybird is the ideal complement to Postgres for handling OLAP workloads. The following guide shows you how to use Redpanda Connect to ingest data from a Postgres database into Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Postgres to Redpanda, ensure:

- You have a Redpanda cluster and Redpanda Connect installed with version 4.43.0 or higher. The following instructions use Redpanda Serverless, but you can use Redpanda Cloud Dedicated or self-hosted.
- You have a PostgreSQL database with logical replication enabled.

## Connect Postgres to Redpanda [¶](https://www.tinybird.co/docs/about:blank#connect-postgres-to-redpanda)

1. In the Redpanda Cloud console, select**  Connect**   from the navigation menu, then select**  Create Pipeline**  .
2. Add the pipeline configuration. You need the following information:

- Postgres connection string ( `dsn`   )
- Redpanda brokers ( `seed_brokers`   )
- SASL mechanism ( `mechanism`   )
- Username ( `username`   )
- Password ( `password`   )

Use the following YAML template:

input:
  label: "postgres_cdc"
  postgres_cdc:
    dsn: <> 
    include_transaction_markers: false
    slot_name: test_slot_native_decoder
    snapshot_batch_size: 100000
    stream_snapshot: true
    temporary_slot: true
    schema: public
    tables:
      - <
>

output:
 redpanda:
   seed_brokers:
     - ${REDPANDA_BROKERS}
   topic: <>
   tls:
     enabled: false
   sasl:
     - mechanism: SCRAM-SHA-512
       password: <>
       username: <> See the Redpanda Connect docs for more information on the `redpanda` output and `postgres_cdc` input.

1. Start the Redpanda Connect pipeline

Select **Create** to save and create the pipeline. This takes you back to the pipeline screen, where you can find your new pipeline. Open the new pipeline to view the logs and confirm that the pipeline is running.

Select the Topics page from the navigation menu and confirm that the topic exists and that messages are being produced.

1. Connect Redpanda to Tinybird

In Tinybird, add a new Data Source and select the **Redpanda** connector. Enter your Redpanda connection details, then select the topic used in the Redpanda Connect pipeline. Confirm your schema and select **Create Data Source**.

Redpanda Connect continuosly consumes changes from Postgres and pushes them to your Redpanda topic. Tinybird consumes the changes from Redpanda in real time, making them available to query with minimal latency.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Tinybird Redpanda Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/redpanda)
-  `redpanda`   output
-  `postgres_cdc`   input



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-with-estuary
Last update: 2025-01-17T08:20:45.000Z
Content:
---
title: "Ingest with Estuary · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to use Estuary to push data streams to Tinybird."
---




# Ingest with Estuary [¶](https://www.tinybird.co/docs/about:blank#ingest-with-estuary)

In this guide, you'll learn how to use Estuary to push data streams to Tinybird.

[Estuary](https://estuary.dev/) is a real-time ETL tool that allows you capture data from a range of source, and push it to a range of destinations. Using Estuary's Dekaf, you can connect Tinybird to Estuary as if it was a Kafka broker - meaning you can use Tinybird's native Kafka Connector to consume data from Estuary.

[Read more about Estuary Dekaf.](https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#connection-details)

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

- An Estuary account & collection
- A Tinybird account & Workspace

## Connecting to Estuary [¶](https://www.tinybird.co/docs/about:blank#connecting-to-estuary)

In Estuary, create a new token to use for the Tinybird connection. You can do this from the Estuary Admin Dashboard.

In your Tinybird Workspace, create a new Data Source and use the [Kafka Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/kafka).

To configure the connection details, use the following settings (these can also be found in the [Estuary Dekaf docs](https://docs.estuary.dev/guides/dekaf_reading_collections_from_kafka/#connection-details) ).

- Bootstrap servers: `dekaf.estuary-data.com`
- SASL Mechanism: `PLAIN`
- SASL Username: `{}`
- SASL Password: Estuary Refresh Token (Generate your token in the Estuary Admin Dashboard)

Tick the `Decode Avro messages with Schema Register` box, and use the following settings:

- URL: `https://dekaf.estuary-data.com`
- Username: `{}`
- Password: The same Estuary Refresh Token as above

Click **Next** and you will see a list of topics. These topics are the collections you have in Estuary. Select the collection you want to ingest into Tinybird, and click **Next**.

Configure your consumer group as needed.

Finally, you will see a preview of the Data Source schema. Feel free to make any modifications as required, then click **Create Data Source**.

This will complete the connection with Estuary, and new data from the Estuary collection will arrive in your Tinybird Data Source in real-time.



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-vercel-logdrains
Last update: 2024-12-24T11:01:26.000Z
Content:
---
title: "Send Vercel log drains to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send Vercel events to Tinybird using webhooks and the Events API."
---




# Send Vercel log drains to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-vercel-log-drains-to-tinybird)

[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.

Some common use cases for sending Vercel Log Drains to Tinybird include:

1. Analyze logs from your applications.
2. Monitor logs from your applications.
3. Create custom analytical dashboards.
4. Build an alert system based on logging patterns.

Read on to learn how to send logs from Vercel to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Vercel Log Drains to Tinybird, ensure:

- You have a Vercel account.
- You have a Tinybird Workspace.

## Connect Vercel to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-vercel-to-tinybird)

1. Choose your team scope on the dashboard, and go to**  Team Settings**   >**  Log Drains**  .
2. Select the**  Projects**   to send logs to Tinybird.
3. Select**  Sources**   you want to send logs to Tinybird.
4. Select**  NDJSON**   as Delivery Format.
5. Select**  Environments**   and**  Sampling Rate**  .
6. In Tinybird, create a Data Source, called `vercel_logs`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel_logs.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" The proxy column is a JSON string. Use the [JSONExtract](https://www.tinybird.co/docs/docs/sql-reference/functions/json-functions) functions to extract the data you need in your Pipes.

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Log Drains Endpoint. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

Log Drains webhook needs to be verified by Vercel. You can do this by adding the `x-vercel-verify` parameter to the request.

https://api.tinybird.co/v0/events?name=vercel_logs&x-vercel-verify=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Custom Headers**   , add `Authorization`   with the value `Bearer `   and select**  Add**  .
2. Select**  Verify**   and optionally use**  Test Log Drain**   from Vercel to check data gets to the `vercel_logs`   Data Source in Tinybird.
3. You're done. Any of the Vercel Log Drains you selected is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

Check the status of the from the **Log** tab in the Tinybird `vercel_logs` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Vercel Log Drains](https://vercel.com/docs/observability/log-drains/log-drains-reference)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-ndjson-data
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Ingest NDJSON data · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to ingest unstructured data, like NDJSON to Tinybird."
---




# Ingest NDJSON data [¶](https://www.tinybird.co/docs/about:blank#ingest-ndjson-data)

In this guide you'll learn how to ingest unstructured NDJSON data into Tinybird.

## Overview [¶](https://www.tinybird.co/docs/about:blank#overview)

A common scenario is having a document-based database, using nested records on your data warehouse or generated events in JSON format from a web application.

For cases like this, the process used to be: Export the `JSON` objects as if they were a `String` in a CSV file, ingest them to Tinybird, and then use the built-in `JSON` functions to prepare the data for real-time analytics as it was being ingested.

But this isn't needed anymore, as Tinybird now accepts JSON imports by default!

Although Tinybird allows you to ingest `.json` and `.ndjson` files, it only accepts the [Newline Delimited JSON](https://github.com/ndjson/ndjson-spec) as content. Each line must be a valid JSON object and every line has to end with `\n` . The API will return an error if each line isn't a valid JSON value.

## Ingest to Tinybird [¶](https://www.tinybird.co/docs/about:blank#ingest-to-tinybird)

This guide will use an example scenario including this [100k rows NDJSON file](https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson) , which contains events from an ecommerce website with different properties.

### With the API [¶](https://www.tinybird.co/docs/about:blank#with-the-api)

Ingesting NDJSON files using the API is similar to the CSV process. There are only two differences to be managed in the query parameters:

- **  format**   : It has to be "ndjson"
- **  schema**   : Usually, the name and the type are provided for every column but in this case it needs an additional property, called the `jsonpath` (see the[  JSONPath syntax](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths)   ). Example:*  "schema=event_name String `json:$.event.name`"*

You can guess the `schema` by first calling the [Analyze API](https://www.tinybird.co/docs/docs/api-reference/analyze-api) . It's a very handy way to not have to remember the `schema` and `jsonpath` syntax: Just send a sample of your file and the Analyze API will describe what's inside (columns, types, schema, a preview, etc.).

##### Analyze API request

curl \
   -H "Authorization: Bearer $TOKEN" \
   -G -X POST "https://api.tinybird.co/v0/analyze" \
   --data-urlencode "url=https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson" Take the `schema` attribute in the response and either use it right away in the next API request to create the Data Source, or modify as you wish: Column names, types, remove any columns, etc.

##### Analyze API response excerpt

{
    "analysis": {
         "columns": [
             {
                "path": "$.date",
                "recommended_type": "DateTime",
                "present_pct": 1,
                "name": "date"
             },
    ...
    ...
    ...
         "schema": "date DateTime `json:$.date`, event LowCardinality(String) `json:$.event`, extra_data_city LowCardinality(String) `json:$.extra_data.city`, product_id String `json:$.product_id`, user_id Int32 `json:$.user_id`, extra_data_term Nullable(String) `json:$.extra_data.term`, extra_data_price Nullable(Float64) `json:$.extra_data.price`"
    },
    "preview": {
         "meta": [
             {
                "name": "date",
                "type": "DateTime"
             },
    ...
    ...
    ...
} Now you've analyzed the file, create the Data Source. In the example below, you will ingest the 100k rows NDJSON file only taking 3 columns from it: date, event, and product_id. The `jsonpath` allows Tinybird to match the Data Source column with the JSON property path:

##### Ingest NDJSON to Tinybird

TOKEN=

curl \
  -H "Authorization: Bearer $TOKEN" \
  -X POST "https://api.tinybird.co/v0/datasources" \
  -G --data-urlencode "name=events_example" \
  -G --data-urlencode "mode=create" \
  -G --data-urlencode "format=ndjson" \
  -G --data-urlencode "schema=date DateTime \`json:$.date\`, event String \`json:$.event\`, product_id String \`json:$.product_id\`" \
  -G --data-urlencode "url=https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson"
### With the Command Line Interface [¶](https://www.tinybird.co/docs/about:blank#with-the-command-line-interface)

There are no changes in the CLI in order to ingest an NDJSON file. Just run the command you are used to with CSV:

##### Generate Data Source schema from NDJSON

tb datasource generate https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson Once it's finished, it automatically generates a .datasource file with all the columns, with their proper types, and `jsonpaths` . For example:

##### Generated Data Source schema

DESCRIPTION generated from https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson

SCHEMA >
    date DateTime `json:$.date`,
    event String `json:$.event`,
    extra_data_city String `json:$.extra_data.city`,
    product_id String `json:$.product_id`,
    user_id Int32 `json:$.user_id`,
    extra_data_price Nullable(Float64) `json:$.extra_data.price`,
    extra_data_city Nullable(String) `json:$.extra_data.city` You can then push that .datasource file to Tinybird and start using it in your Pipes or append new data to it:

##### Push Data Source to Tinybird and append new data

tb push events_100k.datasource
tb datasource append events_100k https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_100k.ndjson
### With the UI [¶](https://www.tinybird.co/docs/about:blank#with-the-ui)

To create a new Data Source from an NDJSON file, navigate to your Workspace and select the **Add Data Source** button.

In the modal, select "File upload" and upload the NDJSON/JSON file or drag and drop onto the modal. You can use provide a URL such as the one provided in this guide. Confirm you're happy with the schema and data, and select "Create Data Source".

Once your data is imported, you will have a Data Source with your JSON data structured in columns, which are easy to transform and consume in any Pipe.

Ingest just the columns you need. After exploration of your data, always remember to create a Data Source that only has the columns needed for your analyses. That will help to make your ingestion, materialization, and your real time data project faster.

## When new JSON fields are added [¶](https://www.tinybird.co/docs/about:blank#when-new-json-fields-are-added)

Tinybird can automatically detect if a new JSON property is being added when new data is being ingested.

Using the Data Source import example from the previous paragraph, you can include a new property to know the origin country of the event, complementing the city. Append new JSON data with the extra property ( [using this example file](https://storage.googleapis.com/tinybird-assets/datasets/guides/how-to-ingest-ndjson-data/events_with_country.ndjson) ).

After finishing the import, open the Data Source modal and confirm that a new blue banner appears, warning you about the new properties detected in the last ingestion:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhow-to-ingest-ndjson-data-4.png&w=3840&q=75)

<-figcaption->
Automatically suggesting new columns





Once you accept viewing those new columns, the application will allow you to add them, change the column types and the column names, as it did in the preview step during the import:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhow-to-ingest-ndjson-data-5.png&w=3840&q=75)

<-figcaption->
Accepting new columns





From now on, whenever you append new data where the new column is defined and has a value, it will appear in the Data Source and will be available to be consumed from your Pipes:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhow-to-ingest-ndjson-data-6.png&w=3840&q=75)

<-figcaption->
New column receiving data





Tinybird automatically detects if there are new columns available. If you ingest data periodically into your NDJSON Data Source (from a file or a Kafka connection) and new columns are coming in, you will see a blue dot in the Data Source icon that appears in the sidebar (see Mark 1 below). Click on the Data Source to view the new columns and add them to the schema, following the steps above.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhow-to-ingest-ndjson-data-7.png&w=3840&q=75)

<-figcaption->
New columns detected, notified by a blue dot





## JSONPaths [¶](https://www.tinybird.co/docs/about:blank#jsonpaths)

This section applies to both NDJSON **and** Parquet data.

When creating a Data Source using NDJSON/Parquet data, for each column in the `schema` you have to provide a JSONPath using the [JSONPath syntax](https://goessner.net/articles/JsonPath).

This is easy for simple schemas, but it can get complex if you have nested fields and arrays.

For example, given this NDJSON object:

{
  "field": "test",
  "nested": { "nested_field": "bla" },
  "an_array": [1, 2, 3],
  "a_nested_array": { "nested_array": [1, 2, 3] }
} The schema would be something like this:

##### schema with jsonpath

a_nested_array_nested_array Array(Int16) `json:$.a_nested_array.nested_array[:]`,
an_array Array(Int16) `json:$.an_array[:]`,
field String `json:$.field`,
nested_nested_field String `json:$.nested.nested_field` Tinybird's JSONPath syntax support has some limitations: It support nested objects at multiple levels, but it supports nested arrays only at the first level, as in the example above. To ingest and transform more complex JSON objects, use the root object JSONPath syntax as described in the next section.

### JSONPaths and the root object [¶](https://www.tinybird.co/docs/about:blank#jsonpaths-and-the-root-object)

Defining a column as "column_name String `json:$` " in the Data Source schema will ingest each line in the NDJSON file as a String in the `column_name` column.

This is very useful in some scenarios.

When you have nested arrays, such as polygons:

##### Nested arrays

{
   "id": 49518,
   "polygon": [
      [
         [30.471785843000134, -1.066836591999916],
         [30.463855835000118, -1.075127054999925],
         [30.456156047000093, -1.086082457999908],
         [30.453003785000135, -1.097347919999962],
         [30.456311076000134, -1.108096617999891],
         [30.471785843000134, -1.066836591999916]
      ]
   ]
} You can parse the `id` and then add the whole JSON string to the root column to extract the polygon with JSON functions.

##### schema definition

id String `json:$.id`,
root String `json:$` When you have complex objects:

##### Complex JSON objects

{
   "elem": {
      "payments": [
            {
               "users": [
                  {
                        "user_id": "Admin_XXXXXXXXX",
                        "value": 4
                  }
               ]
            }
      ]
   }
} Or if you have variable schema ("schemaless") events:

##### Schemaless events

{
   "user_id": "1",
   "data": {
      "whatever": "bla",
      "whatever2": "bla"
   }
}

{
   "user_id": "1",
   "data": [1, 2, 3]
} You can simply put the whole event in the root column and parse as needed:

##### schema definition

root String `json:$`
## JSON data type BETA

[¶](https://www.tinybird.co/docs/about:blank#json-data-type)

You can use the `JSON` data type as an alternative to ingesting NDJSON data.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

### Schemaless JSON type [¶](https://www.tinybird.co/docs/about:blank#schemaless-json-type)

If you expect a completely variable schema, you can store the payload as JSON as follows:

##### events_100k.datasource

SCHEMA >
    `payload` JSON `json:$`

ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()" Fields are stored as `Dynamic` type columns, so you might want to cast them for operations where type matters.

### JSON type queries [¶](https://www.tinybird.co/docs/about:blank#json-type-queries)

You can use dot notation with `json.field` or `getSubcolumn(json, 'field')` . To query nested JSONs, like `extra_data` in the example, use the `^` syntax.

SELECT
    payload,
    payload.date as date,
    payload.event as event,
    payload.^extra_data as extra_data,
    getSubcolumn(payload, 'product_id') as product_id,
    getSubcolumn(payload, 'user_id') as user_id
FROM events_100k The previous example uses `getSubcolumn(payload, 'date')` and `payload.date` indistinctly.

If you ever find any issue with dot notation, try `getSubcolumn(json,'path')` syntax.

### Explicit JSON typing [¶](https://www.tinybird.co/docs/about:blank#explicit-json-typing)

Storing everything as `Dynamic` has performance implications. If you know the schema you can use JSONPaths as described in previous sections, use `JSON(field_name field_type)` , or a mix of both approaches. For example:

##### events_100k_typed_json.datasource

SCHEMA >
    `payload` JSON(date DateTime, event LowCardinality(String), product_id String, user_id Int32) `json:$`,
    `extra_data` JSON(city String, price Float32) `json:$.extra_data`
    
ENGINE "MergeTree"
ENGINE_SORTING_KEY "tuple()"
### JSON dynamic paths and types [¶](https://www.tinybird.co/docs/about:blank#json-dynamic-paths-and-types)

In addition to explicit typing, the `JSON` type has two optional arguments to control how data is stored: `max_dynamic_paths` and `max_dynamic_types` . See [JSOn data type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) for details.

If you don't specify these arguments, the default values are the following:

- `max_dynamic_paths`   : 16
- `max_dynamic_types`   : 2

This is to avoid potential problems with JSON objects containing many properties: up to `max_dynamic_paths` could be stored into specific sets of files for each property. A number too big might cause performance degradation in your data ingest and endpoints.

If your JSON objects contain many properties, consider using the `SKIP` argument to omit those not needed. Using complex objects can degrade the performance of ingestion and reads.

Use the default values. Check with Tinybird support before using custom values.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about the[  Data Sources API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/datasource-api)  .
- Want to schedule your data ingestion? Read the docs on[  cron and GitHub Actions](https://www.tinybird.co/docs/docs/get-data-in/data-operations/scheduling-with-github-actions-and-cron)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-vercel
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Vercel Webhooks to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send Vercel events to Tinybird using webhooks and the Events API."
---




# Send Vercel events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-vercel-events-to-tinybird)

[Vercel](https://vercel.com/) is a platform for building and deploying web applications. By integrating Vercel with Tinybird, you can analyze your Vercel events in real time and enrich it with other data sources.

Some common use cases for sending Vercel events to Tinybird include:

1. Tracking deployments, projects, integrations and domains status and errors.
2. Creating custom analytical dashboards.
3. Monitoring attacks.

Read on to learn how to send data from Vercel to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Vercel webhooks to Tinybird, ensure:

- You have a Vercel account.
- You have a Tinybird Workspace.

## Connect Vercel to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-vercel-to-tinybird)

1. Choose your team scope on the dashboard, and go to**  Settings**   >**  Webhooks**  .
2. Select the Webhooks and Projects you want to send to Tinybird.
3. In Tinybird, create a Data Source, called `vercel`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/vercel.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Vercel webhooks in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Vercel, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=vercel&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. You're done. Any of the Vercel events you selected is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

You can check the status of the integration from the **Log** tab in the Tinybird `vercel` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Vercel webhooks](https://vercel.com/docs/observability/webhooks-overview)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-vector
Last update: 2025-02-07T10:57:18.000Z
Content:
---
title: "Ingest data using Vector.dev · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to use the Events API as a Vector sink."
---




# Ingest data using Vector.dev [¶](https://www.tinybird.co/docs/about:blank#ingest-data-using-vector-dev)

[Vector.dev](https://vector.dev/) is an open-source tool created by DataDog for collecting, transforming, and shipping logs, metrics, and traces.

Some common use cases for using Vector.dev as a Tinybird sink include:

1. Ingesting data from a number of[  Vector.dev sources](https://vector.dev/components/)   to Tinybird.
2. Enriching other data sources with real-time Vector metrics.
3. Aggregate logs and metrics from Vector.dev to Tinybird.
4. Transform and redact sensitive data before ingesting it into Tinybird.

Read on to learn how to use the Events API as a Vector.dev sink.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Vector.dev to Tinybird, ensure:

- You have installed Vector.dev.
- You have a Tinybird Workspace.

## Use the Events API as a Vector.dev sink [¶](https://www.tinybird.co/docs/about:blank#use-the-events-api-as-a-vector-dev-sink)

To push events to Tinybird from Vector.dev, you need to configure Vector to use the Events API as a sink.

You can use the following example Vector configuration to push events, in this case Docker logs, to Tinybird:

sources:
  docker_logs:
    type: "docker_logs"

transforms:
  remap_docker_logs:
    inputs:
      - "docker_logs"
    type: "remap"
    source: |
      . = parse_json!(.log)

sinks:
  push_docker_logs_to_tinybird:
    inputs:
      - "remap_docker_logs"
    type: "http"
    uri: "$TINYBIRD_HOST/v0/events?name=docker"
    auth:
      strategy: "bearer"
      token: "$TINYBIRD_TOKEN"
    encoding:
      codec: "json"
    framing:
        method: "newline_delimited" The previous snippet uses the `docker_logs` source to collect Docker logs, and the `remap_docker_logs` transform to parse the logs as JSON.

The `push_docker_logs_to_tinybird` sink uses the Events API to push the transformed logs to Tinybird in NDJSON ( `newline_delimited` ) format.

You can customize the `$TINYBIRD_HOST` and `$TINYBIRD_TOKEN` environment variables to use your Tinybird workspace.

Learn more about other sources you can use to ingest data into Tinybird in the [Vector.dev documentation](https://vector.dev/docs/reference/configuration/sources/).

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Vector.dev](https://vector.dev/)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-the-events-api
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Stream with HTTP Requests · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to use the Tinybird Events API to ingest thousands of JSON messages per second."
---




# Stream with HTTP Requests [¶](https://www.tinybird.co/docs/about:blank#stream-with-http-requests)

In this guide, you'll learn how to use the Tinybird Events API to ingest thousands of JSON messages per second with HTTP Requests.

For more details about the Events API endpoint, read the [Events API](https://www.tinybird.co/docs/docs/api-reference/events-api) docs.

## Setup: Create the target Data Source [¶](https://www.tinybird.co/docs/about:blank#setup-create-the-target-data-source)

Firstly, you need to create an NDJSON Data Source. You can use the [API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) , or simply drag & drop a file on the UI.

Even though you can add new columns later on, you have to upload an initial file. The Data Source will be created and ordered based upon those initial values.

As an example, upload this NDJSON file:

{"date": "2020-04-05 00:05:38", "city": "New York"}
## Ingest from the browser: JavaScript [¶](https://www.tinybird.co/docs/about:blank#ingest-from-the-browser-javascript)

Ingesting from the browser requires making a standard POST request; see below for an example. Input your own Token and change the name of the target Data Source to the one you created. Check your URL ( `const url` ) is the corresponding [URL for your region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints).

##### Browser High-Frequency Ingest

async function sendEvents(events){
    const date = new Date();
    events.forEach(ev => {
        ev.date = date.toISOString()
    });
    const headers = {
        'Authorization': 'Bearer TOKEN_HERE',
    };
    const url = 'https://api.tinybird.co/' // you may be on a different host
    const rawResponse = await fetch(`${url}v0/events?name=hfi_multiple_events_js`, {
        method: 'POST',
        body: events.map(JSON.stringify).join('\n'),
        headers: headers,
    });
    const content = await rawResponse.json();
    console.log(content);
}

sendEvents([
    { 'city': 'Jamaica', 'action': 'view'},
    { 'city': 'Jamaica', 'action': 'click'},
]); Remember: Publishing your Admin Token on a public website is a security vulnerability. It is **highly recommend** that you [create a new Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#create-a-token) that restricts access granularity.

## Ingest from the backend: Python [¶](https://www.tinybird.co/docs/about:blank#ingest-from-the-backend-python)

Ingesting from the backend is a similar process to ingesting from the browser. Use the following Python snippet and replace the Auth Token and Data Source name, as in the example above.

##### Python High-Frequency Ingest

import requests
import json
import datetime

def send_events(events):
    params = {
        'name': 'hfi_multiple_events_py',
        'token': 'TOKEN_HERE',
    }
    for ev in events:
        ev['date'] = datetime.datetime.now().isoformat()
    data = '\n'.join([json.dumps(ev) for ev in events])
    r = requests.post('https://api.tinybird.co/v0/events', params=params, data=data)
    print(r.status_code)
    print(r.text)

send_events([
    {'city': 'Pretoria', 'action': 'view'},
    {'city': 'Pretoria', 'action': 'click'},
])
## Ingest from the command line: curl [¶](https://www.tinybird.co/docs/about:blank#ingest-from-the-command-line-curl)

The following curl snippet sends two events in the same request:

##### curl High-Frequency Ingest

curl -i -d $'{"date": "2020-04-05 00:05:38", "city": "Chicago"}\n{"date": "2020-04-05 00:07:22", "city": "Madrid"}\n' -H "Authorization: Bearer $TOKEN" 'https://api.tinybird.co/v0/events?name=hfi_test'
## Add new columns from the UI [¶](https://www.tinybird.co/docs/about:blank#add-new-columns-from-the-ui)

As you add extra information in the form of new JSON fields, the UI will prompt you to include those new columns on the Data Source.

For instance, if you send a new event with an extra field:

##### curl High-Frequency Ingest

curl -i -d '{"date": "2020-04-05 00:05:38", "city": "Chicago", "country": "US"}' -H "Authorization: Bearer $TOKEN" 'https://api.tinybird.co/v0/events?name=hfi_test' And navigate to the UI's Data Source screen, you'll be asked if you want to add the new column:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhigh-frequency-ingestion-1.png&w=3840&q=75)


Here, you'll be able to select the desired columns and adjust the types:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhigh-frequency-ingestion-2.png&w=3840&q=75)


After you confirm the addition of the column, it will be populated by new events:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fhigh-frequency-ingestion-3.png&w=3840&q=75)


## Error handling and retries [¶](https://www.tinybird.co/docs/about:blank#error-handling-and-retries)

Read more about the possible [responses returned by the Events API](https://www.tinybird.co/docs/docs/api-reference/events-api).

When using the Events API to send data to Tinybird, you can choose to 'fire and forget' by sending a POST request and ignoring the response. This is a common choice for non-critical data, such as tracking page hits if you're building Web Analytics, where some level of loss is acceptable.

However, if you're sending data where you can't tolerate events being missed, you must implement some error handling & retry logic in your application.

### Wait for acknowledgement [¶](https://www.tinybird.co/docs/about:blank#wait-for-acknowledgement)

When you send data to the Events API, you'll usually receive a `HTTP202` response, which indicates that the request was successful. However, it's important to note that this response only indicates that the Events API successfully **accepted** your HTTP request. It doesn't confirm that the data has been **committed** into the underlying database.

Using the `wait` parameter with your request will ask the Events API to wait for acknowledgement that the data you sent has been committed into the underlying database.

If you use the `wait` parameter, you will receive a `HTTP200` response that confirms data has been committed.

To use this, your Events API request should include `wait` as a query parameter, with a value of `true`.

For example:

https://api.tinybird.co/v0/events?wait=true It is good practice to log your requests to, and responses from, the Events API. This will help give you visibility into any failures for reporting or recovery.

### When to retry [¶](https://www.tinybird.co/docs/about:blank#when-to-retry)

Failures are indicated by a `HTTP4xx` or `HTTP5xx` response.

It's recommended to only implement automatic retries for `HTTP5xx` responses, which indicate that a retry might be successful.

`HTTP4xx` responses should be logged and investigated, as they often indicate issues that can't be resolved by simply retrying with the same request.

For HTTP2 clients, you may receive the `0x07 GOAWAY` error. This indicates that there are too many alive connections. It is safe to recreate the connection and retry these errors.

### How to retry [¶](https://www.tinybird.co/docs/about:blank#how-to-retry)

You should aim to retry any requests that fail with a `HTTP5xx` response.

In general, you should retry these requests 3-5 times. If the failure persists beyond these retries, log the failure, and attempt to store the data in a buffer to resend later (for example, in Kafka, or a file in S3).

It's recommended to use an exponential backoff between retries. This means that, after a retry fails, you should increase the amount of time you wait before sending the next retry. If the issue causing the failure is transient, this gives you a better chance of a successful retry.

Be careful when calculating backoff timings, so that you don't run into memory limits on your application.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about[  the schema](https://www.tinybird.co/docs/docs/get-data-in#create-your-schema)   and why it's important.
- Ingested your data and ready to go? Start[  querying your Data Sources](https://www.tinybird.co/docs/docs/work-with-data/query)   and build some Pipes!



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-stripe
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Stripe Events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send Stripe events to Tinybird using webhooks and the Events API."
---




# Send Stripe events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-stripe-events-to-tinybird)

[Stripe](https://stripe.com/) is a platform for payments and financial services, and it provides a way to send events to Tinybird using webhooks.

Some common use cases for sending Stripe events to Tinybird include:

1. Monitor Stripe events.
2. Run analytical workflows based on Stripe events.
3. Create custom dashboards based on Stripe events.
4. Create alerts and notifications based on Stripe events.
5. Join Stripe events with other Data Sources to enrich your user data.

Read on to learn how to send events from Stripe to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Stripe to Tinybird, ensure:

- You have a Stripe account.
- You have a Tinybird Workspace.

## Connect Stripe to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-stripe-to-tinybird)

Stripe provides a variety of [webhook event types](https://docs.stripe.com/api/events/object) that you can use to send events to Tinybird.

This guide covers the base case for sending Stripe events to Tinybird.

1. In Stripe, go to[  Webhooks](https://dashboard.stripe.com/webhooks)
2. Select**  Add endpoint**  .
3. In Tinybird, create a Data Source, called `stripe`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/stripe.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Stripe in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Stripe, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=stripe&format=json&token= Make sure to use the `format=json` query parameter.



Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Select events**   and choose the events you want to send to Tinybird.
2. Save and You're done.

Check the status of the integration by selecting the webhook in Stripe or from the **Log** tab in the Tinybird `stripe` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Stripe Webhooks](https://docs.stripe.com/webhooks)
- [  Stripe Events](https://docs.stripe.com/api/events/object)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-snowflake-using-incremental-updates
Last update: 2025-01-24T09:36:45.000Z
Content:
---
title: "Ingest from Snowflake using incremental updates · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to ingest data from Snowflake doing incremental appends, so you can keep last transactional data sources in sync with Tinybird."
---




# Ingest from Snowflake using incremental updates [¶](https://www.tinybird.co/docs/about:blank#ingest-from-snowflake-using-incremental-updates)

Read on to learn how to incrementally load data from a Snowflake table into Tinybird, using Amazon S3 as an intermediary staging area.

An incremental loading strategy ensures that only new or updated rows are transferred.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you begin, ensure the following:

- Snowflake Account: A Snowflake instance with the data you want to load.
- Amazon S3 Bucket: Access to an S3 bucket for staging data, along with appropriate permissions (write from Snowflake and read from Tinybird).
- Tinybird Account: A Tinybird workspace with an appropriate Data Source set up.
- Snowflake Permissions: Ensure the Snowflake user has privileges to:
  - Access the target table.
  - Create and manage stages.
  - Unload data into S3.
- AWS Credentials: Ensure Snowflake can use AWS credentials (IAM Role or Access Key/Secret Key pair) to write to the S3 bucket.

1
## Create the unload task in Snowflake [¶](https://www.tinybird.co/docs/about:blank#create-the-unload-task-in-snowflake)

Follow these steps to create the unload task in Snowflake:

1. Grant the required permissions in AWS IAM Console

Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
            "Resource": "arn:aws:s3:::your-bucket-name/path/*"
        }
    ]
} Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to.

1. Create the storage integration

Run the following SQL statement to create the storage integration:

/* Create the S3 integration.
 */
CREATE or replace STORAGE INTEGRATION tinybird_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = ''
  STORAGE_ALLOWED_LOCATIONS = ('*');

-- describe integration tinybird_integration; Replace `` with the ARN of the role created in the previous step.

1. Create the file format

Run the following SQL statement to create the file format:

/* Create the file format for the output files generated.
 */
CREATE OR REPLACE FILE FORMAT csv_unload_format
  TYPE = 'CSV';
1. Create the stage

Run the following SQL statement to create the stage:

/* And finally the stage we'll use to unload the data to.
 */
CREATE or replace STAGE tinybird_stage
  STORAGE_INTEGRATION = tinybird_integration
  URL = 's3://your-bucket-name/path/'
  FILE_FORMAT = csv_unload_format; Replace `your-bucket-name` and `path` with your S3 bucket details.

2
## Create the unload task [¶](https://www.tinybird.co/docs/about:blank#create-the-unload-task)

Run the following SQL statement to create the scheduled task that unloads the new records since the last successful execution to the S3 bucket:

/* Create the scheduled task that unloads the new records since
 * last successful execution to the S3 bucket.
 * 
 * Note how it reads the timestamp of the last successful execution,
 * and leaves a one hour margin.
 *
 * Orders need to be deduplicated later in Tinybird.
 */
CREATE or replace TASK export_order_deltas
    WAREHOUSE = compute_wh
    SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
BEGIN
   LET sql := 'COPY INTO @tinybird_stage/orders/orders_ from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from tinybird.samples.orders_incremental
    where o_orderdate >= (
        SELECT coalesce(timestampadd(hour,-1,max(QUERY_START_TIME)),\'1970-01-01\')
        FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(TASK_NAME=>\'export_order_deltas\'))
        where state = \'SUCCEEDED\'
        ORDER BY SCHEDULED_TIME
    )) max_file_size=1000000000';

   sql := REPLACE(sql, '', TO_VARCHAR(CONVERT_TIMEZONE('UTC',current_timestamp()), 'YYYY_MM_DD__hh24_mi_ss'));
   
   EXECUTE IMMEDIATE (sql);

   RETURN sql;
END; 3
## Configure the ingestion in Tinybird [¶](https://www.tinybird.co/docs/about:blank#configure-the-ingestion-in-tinybird)

Create the S3 connection. For example, using the CLI:

tb connection create s3_iamrole Follow the instructions to grant Tinybird read access to the same S3 bucket you used to unload data from Snowflake.

Then, create the data source using the S3 Connector, using a schema similar to the following:

SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"

IMPORT_SERVICE 's3_iamrole'
IMPORT_CONNECTION_NAME 'tinybird-tb-s3'
IMPORT_BUCKET_URI 's3://tinybird-tb/snowflake/csv/orders/*.csv.gz'
IMPORT_STRATEGY 'append'
IMPORT_SCHEDULE '@auto' Push your project to Tinybird using:

tb push The new files Snowflake writes to the bucket are automatically ingested by Tinybird in a few seconds.

4
## Handle duplicates in Tinybird [¶](https://www.tinybird.co/docs/about:blank#handle-duplicates-in-tinybird)

Use a materialized view to handle duplicates in Tinybird. For example:

NODE sf_orders_0
SQL >

    SELECT *
    FROM sf_orders_incremental_landing

TYPE materialized
DATASOURCE sf_orders Note the `ReplacingMergeTree` and the `ENGINE_VER` option to handle deduplicates using the primary key:

SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String

ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERKEY"
ENGINE_VER "O_ORDERDATE" Remember to use `final` when querying.

5
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

See the following resources:

- [  Ingest from Snowflake using Azure Blob Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-azure-blob-storage)
- [  Ingest from Snowflake using AWS S3](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-snowflake-using-azure-blob-storage
Last update: 2025-01-24T09:36:45.000Z
Content:
---
title: "Ingest from Snowflake using Azure Blob Storage · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using Azure Blob Storage."
---




# Ingest from Snowflake using Azure Blob Storage [¶](https://www.tinybird.co/docs/about:blank#ingest-from-snowflake-using-azure-blob-storage)

Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.

This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview) , or bulk exporting, data as gzipped CSVs and then ingesting them using the [Data Sources API](https://www.tinybird.co/docs/docs/ingest/datasource-api).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To follow these steps you need a Tinybird account and access to Snowflake and permissions to create SAS Tokens for Azure Blob Storage.

1
## Unload the Snowflake table [¶](https://www.tinybird.co/docs/about:blank#unload-the-snowflake-table)

Snowflake lets you [unload](https://docs.snowflake.com/en/user-guide/data-unload-overview) query results to flat files to and external storage service. For example:

COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
  FROM mytable
  CREDENTIALS = ( AZURE_SAS_TOKEN='****' )
  FILE_FORMAT = ( TYPE = CSV  COMPRESSION = GZIP )
  HEADER = FALSE; The most basic implementation is [unloading directly](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location#unloading-data-from-a-table-directly-to-files-in-an-external-location) , but for production use cases consider adding a [named stage](https://docs.snowflake.com/en/user-guide/data-unload-azure#unloading-data-into-an-external-stage) as suggested in the Snowflake docs. Stages give you fine-grained control to access rights.

2
## Create a SAS token for the file [¶](https://www.tinybird.co/docs/about:blank#create-a-sas-token-for-the-file)

Using [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli) , generate a [shared access signature (SAS) token](https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=blobs) so Tinybird can read the file:

az storage blob generate-sas \
    --account-name myaccount \
    --account-key '****' \
    --container-name unload \
    --name data.csv.gz \
    --permissions r \
    --expiry  \
    --https-only \
    --output tsv \
    --full-uri

> 'https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D' You can use the same behavior in S3 and GCS to generate presigned URLs.

3
## Ingest into Tinybird [¶](https://www.tinybird.co/docs/about:blank#ingest-into-tinybird)

Take the generated URL and make a call to Tinybird. You need a [Token](https://www.tinybird.co/docs/concepts/auth-tokens#tokens) with `DATASOURCES:CREATE` permissions:

curl \
-H "Authorization: Bearer " \
-X POST "https://api.tinybird.co/v0/datasources?name=my_datasource_name" \
-d url='https://myaccount.blob.core.windows.net/unload/data.csv.gz?se=2024-05-31T10%3A57%3A41Z&sp=r&spr=https&sv=2022-11-02&sr=b&sig=PMC%2E9ZvOFtKATczsBQgFSsH1%2BNkuJvO9dDPkTpxXH0g%5D' You now have your Snowflake Table in Tinybird.

## Automation [¶](https://www.tinybird.co/docs/about:blank#automation)

To adapt to production scenarios, like having to append data on a timely basis or replacing data that has been updated in Snowflake, you might need to define scheduled actions to move the data. See examples in the [Ingest from Google Cloud Storage guide](https://www.tinybird.co/docs/docs/guides/ingesting-data/ingest-from-google-gcs#automatically-sync-files-with-google-cloud-functions) and in [Schedule data ingestion with cron and GitHub Actions guide](https://www.tinybird.co/docs/docs/guides/ingesting-data/scheduling-with-github-actions-and-cron).

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

Because you're using the Data Sources API, its [limits](https://www.tinybird.co/docs/docs/api-reference/overview#limits) apply.

You might need to adjust your [COPY INTO ](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000` . For example:

COPY INTO 'azure://myaccount.blob.core.windows.net/unload/'
  FROM mytable 
  CREDENTIALS=( AZURE_SAS_TOKEN='****')
  FILE_FORMAT = ( TYPE = CSV  COMPRESSION = GZIP )
  HEADER = FALSE
  MAX_FILE_SIZE = 5000000000;
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

See the following resources:

- [  Ingest from Snowflake using AWS S3](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3)
- [  Ingest from Snowflake using incremental updates](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-incremental-updates)
- [  Ingest from Google Cloud Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-google-gcs)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3
Last update: 2025-01-24T09:36:45.000Z
Content:
---
title: "Ingest from Snowflake using AWS S3 · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send data from Snowflake to Tinybird using AWS S3."
---




# Ingest data from Snowflake using AWS S3 [¶](https://www.tinybird.co/docs/about:blank#ingest-data-from-snowflake-using-aws-s3)

Read on to learn how to send data from Snowflake to Tinybird, for example when you need to periodically run full replaces of a table or do a one-off ingest.

This process relies on [unloading](https://docs.snowflake.com/en/user-guide/data-unload-overview) , or bulk exporting, data as gzipped CSVs and then ingesting them using the [Data Sources API](https://www.tinybird.co/docs/docs/ingest/datasource-api) . Data is then ingested using the [S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To follow these steps you need a Tinybird account and access to Snowflake and AWS S3.

1
## Unload the Snowflake table [¶](https://www.tinybird.co/docs/about:blank#unload-the-snowflake-table)

The first step consists in unloading the Snowflake table to a gzipped CSV file.

1. Grant the required permissions in AWS IAM Console

Make sure the S3 bucket allows Snowflake to write files by setting up an appropriate IAM role or policy. You can use this template to create the policy and attach it to the AWS role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
            "Resource": "arn:aws:s3:::your-bucket-name/path/*"
        }
    ]
} Replace `your-bucket-name/path/*` with your bucket name, and optionally the path you want to grant access to. Attach the policy to the role you want use to unload the data from Snowflake.

1. Create the storage integration

Run the following SQL statement to create the storage integration:

/* Create the S3 integration.
 */
CREATE or replace STORAGE INTEGRATION tinybird_integration
  TYPE = EXTERNAL_STAGE
  STORAGE_PROVIDER = 'S3'
  ENABLED = TRUE
  STORAGE_AWS_ROLE_ARN = ''
  STORAGE_ALLOWED_LOCATIONS = ('*');

-- describe integration tinybird_integration; Replace `` with the ARN of the role created in the previous step.

1. Create the file format

Run the following SQL statement to create the file format:

/* Create the file format for the output files generated.
 */
CREATE OR REPLACE FILE FORMAT csv_unload_format
  TYPE = 'CSV';
1. Create the stage

Run the following SQL statement to create the stage:

/* And finally the stage we'll use to unload the data to.
 */
CREATE or replace STAGE tinybird_stage
  STORAGE_INTEGRATION = tinybird_integration
  URL = 's3://your-bucket-name/path/'
  FILE_FORMAT = csv_unload_format; Replace `your-bucket-name` and `path` with your S3 bucket details.

1. Unload the data

Run the following SQL statement to unload the data:

COPY INTO @tinybird_stage/orders/ from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from my_database.my_schema.orders
) To automate the unloading, you can create a Snowflake task that runs the `COPY INTO` on a schedule. For example:

CREATE or replace TASK export_order_deltas
    WAREHOUSE = compute_wh
    SCHEDULE = 'USING CRON 05 * * * * UTC'
AS
COPY INTO @tinybird_stage/orders from (
    select
        O_ORDERKEY, O_CUSTKEY, O_ORDERSTATUS, O_TOTALPRICE, O_ORDERDATE,
        O_ORDERPRIORITY, O_CLERK
    from my_database.my_schema.orders
) max_file_size=1000000000 2
## Ingest data into Tinybird [¶](https://www.tinybird.co/docs/about:blank#ingest-data-into-tinybird)

Before ingesting your Snowflake data from the S3 bucket, you need to create the S3 connection. See [S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3).

For example, using the CLI, enter the following command to create the connection:

tb connection create s3_iamrole Follow the instructions to grant Tinybird read access to the same S3 bucket you used to unload data from Snowflake.

Then, create the data source using the S3 Connector, using a schema similar to the following:

SCHEMA >
    `O_ORDERKEY` Int64,
    `O_CUSTKEY` Int64,
    `O_ORDERSTATUS` String,
    `O_TOTALPRICE` Float32,
    `O_ORDERDATE` DateTime64(3),
    `O_ORDERPRIORITY` String,
    `O_CLERK` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(O_ORDERDATE)"
ENGINE_SORTING_KEY "O_ORDERDATE, O_ORDERPRIORITY, O_CLERK"

IMPORT_SERVICE 's3_iamrole'
IMPORT_CONNECTION_NAME 'tb-s3'
IMPORT_BUCKET_URI 's3://tb/snowflake/csv/orders/*.csv.gz'
IMPORT_STRATEGY 'append'
IMPORT_SCHEDULE '@auto' With your connection created and Data Source defined, you can now push your project to Tinybird using:

tb push The new files Snowflake writes to the bucket are automatically ingested by Tinybird.

Each file is appended to the Tinybird data source. As records might be duplicated, consider using a [Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) to consolidate a stateful set of your Snowflake table.

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

Because you're using the Data Sources API, its [limits](https://www.tinybird.co/docs/docs/api-reference/overview#limits) apply.

You might need to adjust your [COPY INTO ](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location) expression adding `PARTITION` or `MAX_FILE_SIZE = 5000000000`.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

See the following resources:

- [  Ingest from Snowflake using Azure Blob Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-azure-blob-storage)
- [  Ingest from Snowflake using incremental updates](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-incremental-updates)
- [  Ingest from Google Cloud Storage](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-google-gcs)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-sentry
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Sentry Webhooks to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send Sentry events to Tinybird using webhooks and the Events API."
---




# Send Sentry events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-sentry-events-to-tinybird)

[Sentry](https://sentry.io/) is a platform for monitoring and alerting on errors in your applications. By integrating Sentry with Tinybird, you can analyze your Sentry events in real time and enrich it with other data sources.

Some common use cases for sending Sentry events to Tinybird include:

1. Analyze errors from your applications.
2. Detect patterns in your error data.
3. Build an alert system based on error patterns.
4. Build custom analytical dashboards.

Read on to learn how to send logs from Sentry to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Sentry to Tinybird, ensure:

- You have a Sentry account.
- You have a Tinybird Workspace.

## Connect Sentry to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-sentry-to-tinybird)

1. In Sentry, go to**  Settings**   >**  Developer Settings**   >**  Custom Integrations**  .
2. Select**  Create New Integration**  .
3. In Tinybird, create a Data Source, called `sentry`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/sentry.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.action` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Sentry in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Sentry, paste the Events API URL in your Custom Integration. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=sentry&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Alert Rule Action**  .
2. In the**  Permissions**   box**  Issue and Event**   >**  Read**  .
3. Check all webhooks and**  Save Changes**  .
4. If you also want to send alerts to Tinybird, select**  Alerts**   from the left menu, click on an alert and select**  Edit Rule**   . You can select**  Send Notifications via**   your previously created Custom Integration.
5. You can then select**  Send Test Notification**   to check the connection.
6. You're done. Any of the Sentry events you selected are automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

Check the status of the integration from the **Log** tab in the Tinybird `sentry` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Sentry Webhooks](https://docs.sentry.io/organization/integrations/integration-platform/webhooks/)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-rudderstack
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Stream from RudderStack · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn two different methods to send events from RudderStack to Tinybird."
---




# Stream from RudderStack [¶](https://www.tinybird.co/docs/about:blank#stream-from-rudderstack)

In this guide, you'll learn two different methods to send events from RudderStack to Tinybird.

To better understand the behavior of their customers, companies need to unify timestamped data coming from a wide variety of products and platforms. Typical events to track would be 'sign up', 'login', 'page view' or 'item purchased'. A customer data platform can be used to capture complete customer data like this from wherever your customers interact with your brand. It defines events, collects them from different platforms and products, and routes them to where they need to be consumed.

[RudderStack](https://www.rudderstack.com/) is an open-source customer data pipeline tool. It collects, processes and routes data from your websites, apps, cloud tools, and data warehouse. By using Tinybird's event ingestion endpoint for [high-frequency ingestion](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api) as a Webhook in RudderStack, you can stream customer data in real time to Data Sources.

## Option 1: A separate Data Source for each event type [¶](https://www.tinybird.co/docs/about:blank#option-1-a-separate-data-source-for-each-event-type)

This is the preferred approach. It sends each type of event to a corresponding Data Source.

This [2-minute video](https://www.youtube.com/watch?v=z3TkPvo5CRQ) shows you how to set up high-frequency ingestion through RudderStack using these steps.

The advantages of this method are:

- Your data is well organized from the start.
- Different event types can have different attributes (columns in their Data Source).
- Whenever new attributes are added to an event type you will be prompted to add new columns.
- New event types will get a new Data Source.

Start by generating a Token in the UI to allow RudderStack to write to Tinybird.

### Create a Tinybird Token [¶](https://www.tinybird.co/docs/about:blank#create-a-tinybird-token)

Go to the Workspace in Tinybird where you want to receive data and select "Tokens" in the side panel. Create a new Token by selecting "Create Token" (top right).

Give your Token a descriptive name. In the section "DATA SOURCES SCOPES" mark the "Data Sources management" checkbox (Enabled) to give your Token permission to create Data Sources. Select "Save changes".

### Create a RudderStack Destination [¶](https://www.tinybird.co/docs/about:blank#create-a-rudderstack-destination)

In RudderStack, Select "Destinations" in the side panel and then "New destination" (top right).

Select Webhook:

1. Give the destination a descriptive name.
2. Connect your source(s), you can test with the Rudderstack Sample HTTP Source.
3. Input the following Connection Settings:

- Webhook URL:*  <   https://api.tinybird.co   /v0/events>*
- URL Method:*  POST*
- Headers Key:*  Authorization*
- Headers Value:*  Bearer TINYBIRD_AUTH_TOKEN*



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fstreaming-via-rudderstack-1.png&w=3840&q=75)

<-figcaption->
Webhook connection settings for high-frequency ingestion





On the next page, select "Create new transformation".

You can code a function in the box to apply to events when this transformation is active using the example snippet below (feel free to update it to suit your needs). In this function, you can dynamically append the target Data Source to the target URL of the Webhook. Give your transformation a descriptive name and a helpful description.

##### Transformation code

export function transformEvent(event, metadata){
    event.appendPath="?name=rudderstack_"+event.event.toLowerCase().replace(/[\s\.]/g, '_')
    return event;
} This example snippet uses the prefix `*rudderstack\_*` followed by the name of the event in lower case, with its words separated by an underscore (for instance, a "Product purchased" event would go to a Data Source named `rudderstack_product_purchased` ).

Save the transformation. Your destination has been created successfully!

### Test Ingestion [¶](https://www.tinybird.co/docs/about:blank#test-ingestion)

In Rudderstack, select Sources --> Rudderstack Sample HTTP --> Live events (top right) --> "Send test event" and paste the provided curl command into your terminal. The event will appear on the screen and be sent to Tinybird.

If, after sending some events through RudderStack, you see that your Data Source in Tinybird exists but is empty (0 rows after sending a few events), you will need to authorize the Token that you created to **append** data to the Data Source. In the UI, navigate to "Tokens", select the Token you created, select "Data Sources management" --> "Add Data Source scope", and choose the name of the Data Source that you want to write to. Mark the "Append" checkbox and save the changes.

## Option 2: All events in the same Data Source [¶](https://www.tinybird.co/docs/about:blank#option-2-all-events-in-the-same-data-source)

This alternative approach consists of sending all events into a single Data Source and then splitting them using Tinybird. By pre-configuring the Data Source, any events that RudderStack sends will be ingested with the JSON object in full as a String in a single column. This is very useful when you have complex JSON objects as explained in the [ingesting NDJSON docs](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths) but be aware that using JSONExtract to parse data from the JSON object after ingestion has an impact on performance.

New columns from parsing the data will be detected and you will be asked if you want to save them. You can adjust the inferred data types before saving any new columns. Pipes can be used to filter the Data Source by different events.

The following example assumes you have already installed the Tinybird CLI. If you're not familiar with how to use or install it, [read the CLI docs](https://www.tinybird.co/docs/docs/cli/install).

### Pre-configure a Data Source [¶](https://www.tinybird.co/docs/about:blank#pre-configure-a-data-source)

Authenticate to your Workspace by typing **tb auth** and entering your Token for the Workspace into which you want to ingest data from RudderStack.

Create a new file in your local Workspace, named `rudderstack_events.datasource` , for example, to configure the empty Data Source.

##### Data Source schema

SCHEMA >
'value' String 'json:$'

ENGINE "MergeTree"
ENGINE_SORTING_KEY "value" Push the file to your Workspace using `tb push rudderstack_events.datasource`.

Note that this pre-configured Data Source is only required if you need a column containing the JSON object in full as a String. Otherwise, just skip this step and let Tinybird infer the columns and data types when you send the first event. You will then be able to select which columns you wish to save and adjust their data types. Create the Token as in method 1.

### Create a Tinybird Token [¶](https://www.tinybird.co/docs/about:blank#create-a-tinybird-token)

Go to the Workspace in Tinybird where you want to receive data and select "Tokens" in the side panel. Create a new Token by selecting "Create Token" (top right).

Give your Token a descriptive name. In the section "DATA SOURCES SCOPES", select "Add Data Source scope", choose the name of the Data Source that you just created, and mark the "Append" checkbox. Select "Save changes".

### Create a RudderStack Destination [¶](https://www.tinybird.co/docs/about:blank#create-a-rudderstack-destination)

In RudderStack, Select "Destinations" in the side panel and then "New destination" (top right).

Select Webhook:

1. Give the destination a descriptive name.
2. Connect your source(s), you can test with the Rudderstack Sample HTTP Source.
3. Input the following Connection Settings:

- Webhook URL:*  <   https://api.tinybird.co   /v0/events?name=rudderstack_events>*
- URL Method:*  POST*
- Headers Key:*  Authorization*
- Headers Value:*  Bearer TINYBIRD_AUTH_TOKEN*



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fstreaming-via-rudderstack-3.png&w=3840&q=75)

<-figcaption->
Webhook connection settings with Data Source name





Select 'No transformation needed' and save. Your destination has been created successfully!

### Test Ingestion [¶](https://www.tinybird.co/docs/about:blank#test-ingestion)

Select Sources --> Rudderstack Sample HTTP --> "Live events" (top right) --> "Send test event" and paste the provided curl command into your terminal. The event will appear on the screen and be sent to Tinybird.

The `value` column contains the full JSON object. You will also have the option of having the data parsed into columns. When viewing the new columns you can select which ones to save and adjust their data types.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fstreaming-via-rudderstack-4.png&w=3840&q=75)

<-figcaption->
New columns detected not in schema





Whenever new columns are detected in the stream of events you will be asked if you want to save them.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Need to[  iterate a Data Source, including the schema](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source)   ? Read how here.
- Want to schedule your data ingestion? Read the docs on[  cron and GitHub Actions](https://www.tinybird.co/docs/docs/get-data-in/data-operations/scheduling-with-github-actions-and-cron)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-resend
Last update: 2025-01-13T13:37:38.000Z
Content:
---
title: "Send Resend webhooks to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Resend to Tinybird."
---




# Send Resend webhooks to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-resend-webhooks-to-tinybird)

With [Resend](https://resend.com/) you can send and receive emails programmatically. By integrating Resend with Tinybird, you can analyze your email data in real time.

Some common use cases for sending Resend webhooks to Tinybird include:

1. Tracking email opens and clicks.
2. Monitoring delivery rates and bounces.
3. Analyzing user engagement patterns.
4. Creating custom dashboards for email performance.
5. Enriching other data sources with real-time email metrics.

Read on to learn how to send data from Resend to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Resend to Tinybird, ensure:

- You have a Resend account.
- You have a Tinybird Workspace.

## Connect Resend to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-resend-to-tinybird)

1. Open the Resend UI and go to the Webhooks page.
2. Select**  Add Webhook**  .
3. In Tinybird, create a Data Source, called `resend`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/resend.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Resend in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Resend, paste the Events API URL in your Webhook URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=resend&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select the checkboxes for the Resend events you want to send to Tinybird, and select**  Add**  .
2. You're done. Sending emails to Resend will now push events to Tinybird via the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .
- [  Resend event types](https://resend.com/docs/dashboard/webhooks/event-types)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-pagerduty
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send PagerDuty events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send PagerDuty events to Tinybird using webhooks and the Events API."
---




# Send PagerDuty events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-pagerduty-events-to-tinybird)

[PagerDuty](https://www.pagerduty.com/) is a platform for incident management and alerting. By integrating PagerDuty with Tinybird, you can analyze your incident data in real time and enrich it with other data sources.

Some common use cases for sending PagerDuty events to Tinybird include:

1. Monitoring and alerting on incidents.
2. Creating custom dashboards for incident analysis.
3. Incident logs.

Read on to learn how to send events from PagerDuty to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect PagerDuty to Tinybird, ensure:

- You have an PagerDuty account.
- You have a Tinybird Workspace.

## Connect PagerDuty to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-pagerduty-to-tinybird)

1. From the PagerDuty dashboard, select**  Integrations**   >**  Developer Tools**   >**  Webhooks**  .
2. Select**  New Webhook**  .
3. In Tinybird, create a Data Source, called `pagerduty`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/pagerduty.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event.event_type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from PagerDuty in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in PagerDuty, paste the Events API URL in your Webhook URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=pagerduty

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Add custom header**   and add 'Authorization' as**  Name**   and paste the token you created in Tinybird as**  Value**  .

Bearer 
1. Select all event subcriptions and**  Add webhook**
2. You're done. Any of the PagerDuty events is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

You can check the status of the integration by testing the Webhook integration in PagerDuty or from the **Log** tab in the Tinybird `pagerduty` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  PagerDuty webhooks](https://support.pagerduty.com/main/docs/webhooks)
- [  PagerDuty webhook payload](https://developer.pagerduty.com/docs/webhooks-overview#webhook-payload)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-orb
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Orb events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send Orb events to Tinybird using webhooks and the Events API."
---




# Send Orb events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-orb-events-to-tinybird)

[Orb](https://withorb.com/) is a developer-focused platform to manage your subscription billing and revenue operations. By integrating Orb with Tinybird, you can analyze your subscription billing data in real time and enrich it with other data sources.

Some common use cases for sending Orb events to Tinybird include:

1. Tracking and monitoring subscriptions.
2. Monitoring user churn.
3. Creating custom dashboards for subscription analysis.
4. Subscriptions logs.

Read on to learn how to send events from Orb to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Orb to Tinybird, ensure:

- You have an Orb account.
- You have a Tinybird Workspace.

## Connect Orb to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-orb-to-tinybird)

1. From the Orb dashboard, select**  Developers**   >**  Webhooks**  .
2. Select**  Add Endpoint**  .
3. In Tinybird, create a Data Source, called `orb`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/orb.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Orb in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Orb, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=orb&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Send test request**   to test the connection and check the data gets to the `orb`   Data Source in Tinybird.
2. You're done. Any of the Orb events is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

You can check the status of the integration by clicking on the Webhook endpoint in Orb or from the **Log** tab in the Tinybird `orb` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Orb webhooks](https://docs.withorb.com/guides/integrations-and-exports/webhooks)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-mongodb
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Ingest data from MongoDB · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to ingest data into Tinybird from MongoDB."
---




# Connect MongoDB to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-mongodb-to-tinybird)

In this guide, you'll learn how to ingest data into Tinybird from MongoDB.

You'll use:

- MongoDB Atlas as the source MongoDB database.
- Confluent Cloud's MongoDB Atlas Source connector to capture change events from MongoDB Atlas and push to Kafka
- Tinybird Confluent Cloud connector to ingest the data from Kafka

This guide uses Confluent Cloud as a managed Kafka service, and MongoDB Atlas as a managed MongoDB service. You can use any Kafka service and MongoDB instance, but the setup steps may vary.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes you have:

- An existing Tinybird account & Workspace
- An existing Confluent Cloud account
- An existing MongoDB Atlas account & collection

## 1. Create Confluent Cloud MongoDB Atlas Source [¶](https://www.tinybird.co/docs/about:blank#1-create-confluent-cloud-mongodb-atlas-source)

[Create a new MongoDB Atlas Source in Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-source.html#get-started-with-the-mongodb-atlas-source-connector-for-ccloud) . Use the following template to configure the Source:

{
  "name": "",
  "config": {
    "name": "",

    "connection.host": "",
    "connection.user": "",
    "connection.password": "",
    "database": "",
    "collection": "",
    
    "cloud.provider": "",
    "cloud.environment": "",
    "kafka.region": "",
    "kafka.auth.mode": "KAFKA_API_KEY",
    "kafka.api.key": "",
    "kafka.api.secret": "",
    "kafka.endpoint": "",
    
    "topic.prefix": "",
    "errors.deadletterqueue.topic.name": "",
    
    "startup.mode": "copy_existing",
    "copy.existing": "true",
    "copy.existing.max.threads": "1",
    "copy.existing.queue.size": "16000",

    "poll.await.time.ms": "5000",
    "poll.max.batch.size": "1000",
    "heartbeat.interval.ms": "10000",
    "errors.tolerance": "all",
    "max.batch.size": "100",

    "connector.class": "MongoDbAtlasSource",
    "output.data.format": "JSON",
    "output.json.format": "SimplifiedJson",
    "json.output.decimal.format": "NUMERIC",
    "change.stream.full.document": "updateLookup",
    "change.stream.full.document.before.change": "whenAvailable",
    "tasks.max": "1"
  }
} When the Source is created, you should see a new Kafka topic in your Confluent Cloud account. This topic will contain the change events from your MongoDB collection.

## 2. Create Tinybird Data Source (CLI) [¶](https://www.tinybird.co/docs/about:blank#2-create-tinybird-data-source-cli)

Using the Tinybird CLI, create a new Kafka connection

`tb connection create kafka`

The CLI will prompt you to enter the connection details to your Kafka service. You'll also provide a name for the connection, which is used by Tinybird to reference the connection, and you'll need it below.

Next, create a new file called `kafka_ds.datasource` (you can use any name you want, just use the .datasource extension). Add the following content to the file:

SCHEMA >
    `_id` String `json:$.documentKey._id` DEFAULT JSONExtractString(__value, '_id._id'),
    `operation_type` LowCardinality(String) `json:$.operationType`,
    `database` LowCardinality(String) `json:$.ns.db`,
    `collection` LowCardinality(String) `json:$.ns.coll`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(__timestamp)"
ENGINE_SORTING_KEY "__timestamp, _id"

KAFKA_CONNECTION_NAME ''
KAFKA_TOPIC ''
KAFKA_GROUP_ID ''
KAFKA_AUTO_OFFSET_RESET 'earliest'
KAFKA_STORE_RAW_VALUE 'True'
KAFKA_STORE_HEADERS 'False'
KAFKA_STORE_BINARY_HEADERS 'True'
KAFKA_TARGET_PARTITIONS 'auto'
KAFKA_KEY_AVRO_DESERIALIZATION '' Now push the Data Source to Tinybird using:

tb push kafka_ds.datasource
## 3. Validate the Data Source [¶](https://www.tinybird.co/docs/about:blank#3-validate-the-data-source)

Go to the Tinybird UI and validate that a Data Source has been created.

As changes occur in MongoDB, you should see the data being ingested into Tinybird. Note that this is an append log of all changes, so you will see multiple records for the same document as it's updated.

## 4. Deduplicate with ReplacingMergeTree [¶](https://www.tinybird.co/docs/about:blank#4-deduplicate-with-replacingmergetree)

To deduplicate the data, you can use a `ReplacingMergeTree` engine on a Materialized View. This is explained in more detail in the [deduplication guide](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies#use-the-replacingmergetree-engine).

Tinybird creates a new Data Source using the ReplacingMergeTree engine to store the deduplicated data, and a Pipe to process the data from the original Data Source and write to the new Data Source.

First, create a new Data Source to store the deduplicated data.

Create a new file called `deduped_ds.datasource` and add the following content:

SCHEMA >
    `fullDocument` String,
    `_id` String,
    `database` LowCardinality(String),
    `collection` LowCardinality(String),
    `k_timestamp` DateTime,
    `is_deleted` UInt8

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "_id"
ENGINE_VER "k_timestamp"
ENGINE_IS_DELETED "is_deleted" Now push the Data Source to Tinybird using:

tb push deduped_ds.datasource Then, create a new file called `dedupe_mongo.pipe` and add the following content:

NODE mv
SQL >

    SELECT
        JSONExtractRaw(__value, 'fullDocument') as fullDocument,
        _id,
        database,
        collection,
        __timestamp as k_timestamp,
        if(operation_type = 'delete', 1, 0) as is_deleted
    FROM 

TYPE materialized
DATASOURCE  Now push the Pipe to Tinybird using:

tb push dedupe_mongo.pipe As new data arrives via Kafka, it will be processed automatically through the Materialized View, writing it into the `ReplacingMergeTree` Data Source.

Query this new Data Source to access the deduplicated data:

SELECT * FROM deduped_ds FINAL

---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-mailgun
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Mailgun Events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send Mailgun events to Tinybird using webhooks and the Events API."
---




# Send Mailgun events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-mailgun-events-to-tinybird)

[Mailgun](https://www.mailgun.com/) is a platform for sending email, and it provides a way to send events to Tinybird using webhooks.

Read on to learn how to send events from Mailgun to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Mailgun to Tinybird, ensure:

- You have a Mailgun account.
- You have a Tinybird Workspace.

## Connect Mailgun to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-mailgun-to-tinybird)

Mailgun provides a variety of [webhook event types](https://mailgun-docs.redoc.ly/docs/mailgun/user-manual/events/#event-structure) that you can use to send events to Tinybird.

This guide covers the base case for sending Mailgun events to Tinybird.

1. In Mailgun, go to**  Send**   >**  Sending**   >**  Webhooks**  .
2. Select**  Domain**   and**  Add webhook**  .
3. In Tinybird, create a Data Source, called `mailgun`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/mailgun.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event-data.event` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Mailgun in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Mailgun, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=mailgun&format=json&token= Make sure to use the `format=json` query parameter.



Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Event type**   and choose the event you want to send to Tinybird. You can use the same Tinybird Data Source for multiple events.
2. Select**  Create webhook**   abd you're done.

Check the status of the integration from the **Log** tab in the Tinybird `mailgun` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Mailgun Events](https://mailgun-docs.redoc.ly/docs/mailgun/user-manual/events/#event-structure)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-knock
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Knock Events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send Knock events to Tinybird using webhooks and the Events API."
---




# Send Knock events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-knock-events-to-tinybird)

[Knock](https://knock.app/) is a platform for notifications and alerts, and it provides a way to send events to Tinybird using webhooks.

Some common use cases for sending Knock events to Tinybird include:

1. Monitor Knock message events.
2. Run analytical workflows based on Knock events.
3. Create custom dashboards based on Knock events.
4. Create alerts and notifications based on Knock events.
5. Join Knock message events with other Data Sources to enrich your user data.

Read on to learn how to send events from Knock to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Knock to Tinybird, ensure:

- You have a Knock account.
- You have a Tinybird Workspace.

## Connect Knock to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-knock-to-tinybird)

Knock provides a variety of [webhook event types](https://docs.knock.app/developer-tools/outbound-webhooks/event-types#message-events) that you can use to send events to Tinybird.

This guide covers the base case for sending Knock Message events to Tinybird.

1. In Knock, go to your repository**  Developers**   >**  Webhooks**  .
2. Select**  Create webhook**  .
3. Webhooks payloads vary depending on the event type. You can check here the list of[  Knock events](https://docs.knock.app/developer-tools/outbound-webhooks/event-types#message-events)  .

For this guide, select events related to `message`.

1. In Tinybird, create a Data Source, called `knock`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/knock.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Knock in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Knock, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=knock&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Save webhook**  .
2. You're done.

Check the status of the integration from the `Logs` tab in the Knock webhook or from the **Log** tab in the Tinybird `knock` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Knock Webhooks](https://docs.knock.app/developer-tools/outbound-webhooks/overview)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-google-pubsub
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Ingest from Google Pub/Sub · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Google Pub/Sub to Tinybird."
---




# Stream from Google Pub/Sub [¶](https://www.tinybird.co/docs/about:blank#stream-from-google-pubsub)

In this guide you'll learn how to send data from Google Pub/Sub to Tinybird.

## Overview [¶](https://www.tinybird.co/docs/about:blank#overview)

Tinybird is a Google Cloud partner & supports integrating with Google Cloud services.

[Google Pub/Sub](https://cloud.google.com/pubsub) is often used as a messaging middleware that decouples event stream sources from the end destination. Pub/Sub streams are usually consumed by Google's DataFlow which can send events on to destinations such as BigQuery, BigTable, or Google Cloud Storage.

This DataFlow pattern works with Tinybird too, however, Pub/Sub also has a feature called [Push subscriptions](https://cloud.google.com/pubsub/docs/push) which can forward messages directly from Pub/Sub to Tinybird. The following guide steps use the subscription approach.

## Push messages from Pub/Sub to Tinybird [¶](https://www.tinybird.co/docs/about:blank#push-messages-from-pubsub-to-tinybird)

### 1. Create a Pub/Sub topic [¶](https://www.tinybird.co/docs/about:blank#1-create-a-pubsub-topic)

Start by creating a topic in Google Pub/Sub following the [Google Pub/Sub documentation](https://cloud.google.com/pubsub/docs/admin#create_a_topic).

### 2. Create a push subscription [¶](https://www.tinybird.co/docs/about:blank#2-create-a-push-subscription)

Next, [create a Push subscription in Pub/Sub](https://cloud.google.com/pubsub/docs/create-subscription#push_subscription).

Set the **Delivery Type** to **Push**.

In the **Endpoint URL** field, ue the following snippet (which uses the [Tinybird Events API](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api) ) and pass your own Token, which you can find in your Workspace > Tokens:

##### Endpoint URL

https://api.tinybird.co/v0/events?wait=true&name=&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

If you are sending single-line JSON payload through Pubsub, tick the **Enable payload unwrapping** option to enable unwrapping. This means that data isn't base64 encoded before sending it to Tinybird. If you are sending any other format via Pubsub, leave this unchecked (you'll need to follow the decoding steps at the bottom of this guide).

Set **Retry policy** to **Retry after exponential backoff delay** . Set the **Minimum backoff** to **1** and **Maximum backoff** to **60**.

You don't need to create the Data Source in advance, it will automatically be created for you. This snippet also includes the `wait=true` parameter, which is explained in the [Events API docs](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api#wait-for-acknowledgement).

### 3. Send sample messages [¶](https://www.tinybird.co/docs/about:blank#3-send-sample-messages)

Generate and send some sample messages to test your connection. If you don't have your own messages to test, use [this script](https://gist.github.com/alejandromav/dec8e092ef62d879e6821da06f6459c2).

### 4. Check the Data Source [¶](https://www.tinybird.co/docs/about:blank#4-check-the-data-source)

Pub/sub will start to push data to Tinybird. Check the Tinybird UI to see that the Data Source has been created and events are arriving.

### (Optional) Decode the payload [¶](https://www.tinybird.co/docs/about:blank#optional-decode-the-payload)

If you enabled the **Enable payload unwrapping** option, there is nothing else to do.

However, if you aren't sending single-line JSON payloads (NDJSON, JOSNL) through Pubsub, you'll need to continue to base64 encode data before sending it to Tinybird. When the data arrived in Tinybird, you can decode it using the `base64Decode` function, like this:

SELECT
    message_message_id as message_id,
    message_publish_time,
    base64Decode(message_data) as message_data
  FROM events_demo
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Explore other Google <> Tinybird integrations like[  how to query Google Sheets with SQL](https://www.tinybird.co/blog-posts/query-google-sheets-with-sql-in-real-time)  .
- Ready to start querying your data? Make sure you're familiar with[  how to work with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-google-gcs
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Ingest from Google Cloud Storage · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to automatically synchronize all the CSV files in a Google GCS bucket to a Tinybird Data Source."
---




# Ingest from Google Cloud Storage [¶](https://www.tinybird.co/docs/about:blank#ingest-from-google-cloud-storage)

In this guide, you'll learn how to automatically synchronize all the CSV files in a Google GCS bucket to a Tinybird Data Source.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes you have familiarity with [Google GCS buckets](https://cloud.google.com/storage/docs/buckets) and the basics of [ingesting data into Tinybird](https://www.tinybird.co/docs/docs/get-data-in).

## Perform a one-off load [¶](https://www.tinybird.co/docs/about:blank#perform-a-one-off-load)

When building on Tinybird, people often want to load historical data that comes from another system (called 'seeding' or 'backfilling'). A very common pattern is exporting historical data by creating a dump of CSV files into a Google GCS bucket, then ingesting these CSV files into Tinybird.

You can append these files to a Data Source in Tinybird using the Data Sources API.

Let's assume you have a set of CSV files in your GCS bucket:

##### List of events files

tinybird-assets/datasets/guides/events/events_0.csv
tinybird-assets/datasets/guides/events/events_1.csv
tinybird-assets/datasets/guides/events/events_10.csv
tinybird-assets/datasets/guides/events/events_11.csv
tinybird-assets/datasets/guides/events/events_12.csv
tinybird-assets/datasets/guides/events/events_13.csv
tinybird-assets/datasets/guides/events/events_14.csv
tinybird-assets/datasets/guides/events/events_15.csv
tinybird-assets/datasets/guides/events/events_16.csv
tinybird-assets/datasets/guides/events/events_17.csv
tinybird-assets/datasets/guides/events/events_18.csv
tinybird-assets/datasets/guides/events/events_19.csv
tinybird-assets/datasets/guides/events/events_2.csv
tinybird-assets/datasets/guides/events/events_20.csv
tinybird-assets/datasets/guides/events/events_21.csv
tinybird-assets/datasets/guides/events/events_22.csv
tinybird-assets/datasets/guides/events/events_23.csv
tinybird-assets/datasets/guides/events/events_24.csv
tinybird-assets/datasets/guides/events/events_25.csv
tinybird-assets/datasets/guides/events/events_26.csv
tinybird-assets/datasets/guides/events/events_27.csv
tinybird-assets/datasets/guides/events/events_28.csv
tinybird-assets/datasets/guides/events/events_29.csv
tinybird-assets/datasets/guides/events/events_3.csv
tinybird-assets/datasets/guides/events/events_30.csv
tinybird-assets/datasets/guides/events/events_31.csv
tinybird-assets/datasets/guides/events/events_32.csv
tinybird-assets/datasets/guides/events/events_33.csv
tinybird-assets/datasets/guides/events/events_34.csv
tinybird-assets/datasets/guides/events/events_35.csv
tinybird-assets/datasets/guides/events/events_36.csv
tinybird-assets/datasets/guides/events/events_37.csv
tinybird-assets/datasets/guides/events/events_38.csv
tinybird-assets/datasets/guides/events/events_39.csv
tinybird-assets/datasets/guides/events/events_4.csv
tinybird-assets/datasets/guides/events/events_40.csv
tinybird-assets/datasets/guides/events/events_41.csv
tinybird-assets/datasets/guides/events/events_42.csv
tinybird-assets/datasets/guides/events/events_43.csv
tinybird-assets/datasets/guides/events/events_44.csv
tinybird-assets/datasets/guides/events/events_45.csv
tinybird-assets/datasets/guides/events/events_46.csv
tinybird-assets/datasets/guides/events/events_47.csv
tinybird-assets/datasets/guides/events/events_48.csv
tinybird-assets/datasets/guides/events/events_49.csv
tinybird-assets/datasets/guides/events/events_5.csv
tinybird-assets/datasets/guides/events/events_6.csv
tinybird-assets/datasets/guides/events/events_7.csv
tinybird-assets/datasets/guides/events/events_8.csv
tinybird-assets/datasets/guides/events/events_9.csv
### Ingest a single file [¶](https://www.tinybird.co/docs/about:blank#ingest-a-single-file)

To ingest a single file, [generate a signed URL in GCP](https://cloud.google.com/storage/docs/access-control/signed-urls) , and send the URL to the Data Sources API using the `append` mode flag:

##### Example POST request with append mode flag

curl -H "Authorization: Bearer " \
   -X POST "https://api.tinybird.co/v0/datasources?name=&mode=append" \
   --data-urlencode "url="
### Ingest multiple files [¶](https://www.tinybird.co/docs/about:blank#ingest-multiple-files)

If you want to ingest multiple files, you probably don't want to manually write each cURL. Instead, create a script to iterate over the files in the bucket and generate the cURL commands automatically.

The following script example requires the [gsutil tool](https://cloud.google.com/storage/docs/gsutil) and assumes you have already created your Tinybird Data Source.

You can use the `gsutil` tool to list the files in the bucket, extract the name of the CSV file, and create a signed URL. Then, generate a cURL to send the signed URL to Tinybird.

To avoid hitting [API rate limits](https://www.tinybird.co/docs/docs/api-reference#limits) you should delay 15 seconds between each request.

Here's an example script in bash:

##### Ingest CSV files from a Google Cloud Storage Bucket to Tinybird

TB_HOST=
TB_TOKEN=
BUCKET=gs://
DESTINATION_DATA_SOURCE=
GOOGLE_APPLICATION_CREDENTIALS=
REGION=

for url in $(gsutil ls $BUCKET | grep csv)
do
  echo $url
  SIGNED=`gsutil signurl -r $REGION $GOOGLE_APPLICATION_CREDENTIALS $url | tail -n 1 | python3 -c "import sys; print(sys.stdin.read().split('\t')[-1])"`
  curl -H "Authorization: Bearer $TB_TOKEN" \
    -X POST "$TB_HOST/v0/datasources?name=$DESTINATION_DATA_SOURCE&mode=append" \
    --data-urlencode "url=$SIGNED"
  echo
  sleep 15
done The script uses the following variables:

- `TB_HOST`   as the corresponding URL for[  your region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints)  .
- `TB_TOKEN`   as a Tinybird[  Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens)   with `DATASOURCE:CREATE`   or `DATASOURCE:APPEND`   scope. See the[  Tokens API](https://www.tinybird.co/docs/docs/api-reference/token-api)   for more information.
- `BUCKET`   as the GCS URI of the bucket containing the events CSV files.
- `DESTINATION_DATA_SOURCE`   as the name of the Data Source in Tinybird, in this case `events`  .
- `GOOGLE_APPLICATION_CREDENTIALS`   as the local path of a Google Cloud service account JSON file.
- `REGION`   as the Google Cloud region name.

## Automatically sync files with Google Cloud Functions [¶](https://www.tinybird.co/docs/about:blank#automatically-sync-files-with-google-cloud-functions)

The previous scenario covered a one-off dump of CSV files in a bucket to Tinybird. A slightly more complex scenario is appending to a Tinybird Data Source each time a new CSV file is dropped into a GCS bucket, which can be done using Google Cloud Functions.

That way you can have your ETL process exporting data from your Data Warehouse (such as Snowflake or BigQuery) or any other origin and you don't have to think about manually synchronizing those files to Tinybird.

Imagine you have a GCS bucket named `gs://automatic-ingestion-poc/` and each time you put a CSV there you want to sync it automatically to an `events` Data Source previously created in Tinybird:

1. Clone  this GitHub repository ( `gcs-cloud-function`   )  .
2. Install and configure the `gcloud`   command line tool.
3. Run `cp .env.yaml.sample .env.yaml`   and set the `TB_HOST`   , and `TB_TOKEN`   variable
4. Run:

##### Syncing from GCS to Tinybird with Google Cloud Functions

# set some environment variables before deploying
PROJECT_NAME=
SERVICE_ACCOUNT_NAME=
BUCKET_NAME=
REGION=
TB_FUNCTION_NAME=

# grant permissions to deploy the cloud function and read from storage to the service account
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/storage.admin
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/iam.serviceAccountTokenCreator
gcloud projects add-iam-policy-binding $PROJECT_NAME --member serviceAccount:$SERVICE_ACCOUNT_NAME --role roles/editor

# deploy the cloud function
gcloud functions deploy $TB_FUNCTION_NAME \
  --runtime python38 \
  --trigger-resource $BUCKET_NAME \
  --trigger-event google.storage.object.finalize \
  --region $REGION \
  --env-vars-file .env.yaml \
  --service-account $SERVICE_ACCOUNT_NAME It deploys a Google Cloud Function with name `TB_FUNCTION_NAME` to your Google Cloud account, which listens for new files in the `BUCKET_NAME` provided (in this case `automatic-ingestion-poc` ), and automatically appends them to the Tinybird Data Source described by the `FILE_REGEXP` environment variable.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fsyncing-data-from-s3-or-gcs-buckets-3.png&w=3840&q=75)

<-figcaption->
Cloud function to sync a GCS bucket to Tinybird





Now you can drop CSV files into the configured bucket:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fsyncing-data-from-s3-or-gcs-buckets-4.gif&w=3840&q=75)

<-figcaption->
Drop files to a GCS bucket and check the datasources_ops_log





A recommended pattern is naming the CSV files in the format `datasourcename_YYYYMMDDHHMMSS.csv` so they are automatically appended to `datasourcename` in Tinybird. For instance, `events_20210125000000.csv` will be appended to the `events` Data Source.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Got your schema sorted and ready to make some queries? Understand[  how to work with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time)  .
- Learn how to[  monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-gitlab
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send GitLab Events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send GitLab events to Tinybird using webhooks and the Events API."
---




# Send GitLab events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-gitlab-events-to-tinybird)

[GitLab](https://gitlab.com/) is a platform for building and deploying web applications. By integrating GitLab with Tinybird, you can analyze your GitLab events in real time and enrich it with other data sources.

Some common use cases for sending GitLab events to Tinybird include:

1. Analyze GitLab issues and merge requests.
2. Analyze GitLab push events.
3. Analyze and monitor GitLab pipeline.
4. Analyze custom DORA metrics.

All this allows you to build a more complete picture of your GitLab events and improve your DevOps processes.

Read on to learn how to send events from GitLab to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect GitLab to Tinybird, ensure:

- You have a GitLab account.
- You have a Tinybird Workspace.

## Connect GitLab to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-gitlab-to-tinybird)

1. In GitLab, go to**  Settings**   >**  Webhooks**  .
2. Select**  Add new webhook**  .
3. Webhooks payloads vary depending on the event type. You can check here the list of[  GitLab events](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html)  .

Select **Issues Events**.

1. In Tinybird, create a Data Source, called `gitlab`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/gitlab.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.object_kind` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from GitLab in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in GitLab, paste the Events API URL in your Webhook URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=gitlab

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  Add custom header**   and add 'Authorization' as**  Header name**   and paste the token you created in Tinybird as**  Header value**  .

Bearer 
1. You're done. You can select**  Test**   to check if the webhook is working.

Check the status of the integration from the **Log** tab in the Tinybird `gitlab` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  GitLab Webhooks](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html)
- [  Tinybird Data Sources](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-github
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send GitHub Events to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to send GitHub events to Tinybird using webhooks and the Events API."
---




# Send GitHub events to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-github-events-to-tinybird)

[GitHub](https://github.com/) is a platform for building and deploying web applications. By integrating GitHub with Tinybird, you can analyze your GitHub events in real time and enrich it with other data sources.

Some common use cases for sending GitHub events to Tinybird include:

1. Analyze GitHub issues and pull requests.
2. Analyze GitHub push events.
3. Analyze and monitor GitHub pipeline.
4. Analyze custom DORA metrics.

All this allows you to build a more complete picture of your GitHub events and improve your DevOps processes.

Read on to learn how to send events from GitHub to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect GitHub to Tinybird, ensure:

- You have a GitHub account.
- You have a Tinybird Workspace.

## Connect GitHub to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-github-to-tinybird)

GitHub provides a variety of webhooks (+70) that you can use to send events to Tinybird at organization, repository or application level.

This guide covers the base case for sending GitHub events from a repository to Tinybird.

1. In GitHub, go to your repository**  Settings**   >**  Webhooks**  .
2. Select**  Add webhook**  .
3. Webhooks payloads vary depending on the event type. You can check here the list of[  GitHub events](https://docs.github.com/en/webhooks/webhook-events-and-payloads)  .

Select **Send me everything**.

1. In Tinybird, create a Data Source, called `github`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/github.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.type` DEFAULT 'unknown',
  `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from GitHub in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in GitHub, paste the Events API URL in your Webhook URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird.

https://api.tinybird.co/v0/events?name=github&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select**  application/json**   as the content type.
2. You're done.

Check the status of the integration from the `Recent deliveries` in the GitHub webhooks panel or from the **Log** tab in the Tinybird `github` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  GitHub Webhook events and payloads](https://docs.github.com/en/webhooks/webhook-events-and-payloads)
- [  GitHub Webhooks](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-dynamodb-single-table-design
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Working with DynamoDB Single-Table Design · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to work with data that follows DynamoDB Single-Table Design."
---




# Working with DynamoDB Single-Table Design [¶](https://www.tinybird.co/docs/about:blank#working-with-dynamodb-single-table-design)

Single-Table Design is a common pattern [recommended by AWS](https://aws.amazon.com/blogs/compute/creating-a-single-table-design-with-amazon-dynamodb/) in which different table schemas are stored in the same table. Single-table design makes it easier to support many-to-many relationships and avoid the need for JOINs, which DynamoDB doesn't support.

Single-Table Design is a good pattern for DynamoDB, but it's not optimal for analytics. To achieve higher performance in Tinybird, normalize data from DynamoDB into multiple tables that support the access patterns of your analytical queries.

The normalization process is achieved entirely within Tinybird by ingesting the raw DynamoDB data into a landing Data Source and then creating Materialized Views to extract items into separate tables.

This guide assumes you're familiar with DynamoDB, Tinybird, creating DynamoDB Data Sources in Tinybird, and Materialized Views.

## Example DynamoDB Table [¶](https://www.tinybird.co/docs/about:blank#example-dynamodb-table)

For example, if Tinybird metadata were stored in DynamoDB using Single-Table Design, the table might look like this:

- **  Partition Key**  : `Org#Org_name`   , example values:**  Org#AWS**   or**  Org#Tinybird**  .
- **  Sort Key**  : `Item_type#Id`   , example values:**  USER#1**   or**  WS#2**  .
- **  Attributes**   : the information stored for each kind of item, like user email or Workspace cores.

## Create the DynamoDB Data Source [¶](https://www.tinybird.co/docs/about:blank#create-the-dynamodb-data-source)

Use the [DynamoDB Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/dynamodb) to ingest your DynamoDB table into a Data Source.

Rather than defining all columns in this landing Data Source, set only the Partition Key (PK) and Sort Key (SK) columns. The rest of the attributes are stored in the `_record` column as JSON. You don't need to define the `_record` column in the schema, as it's created automatically.

SCHEMA >
   `PK` String `json:$.Org#Org_name`,
   `SK` String `json:$.Item_type#Id`

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "PK, SK"
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted"

IMPORT_SERVICE 'dynamodb'
IMPORT_CONNECTION_NAME 
IMPORT_TABLE_ARN 
IMPORT_EXPORT_BUCKET  The following image shows how data looks. The DynamoDB Connector creates some additional rows, such as `_timestamp` , that aren't in the .datasource file:



<-figure->
![DynamoDB Table storing users and worskpaces information](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-ddb-std-2.png&w=3840&q=75)

<-figcaption->
DynamoDB Table storing users and worskpaces information





## Use a Pipe to filter and extract items [¶](https://www.tinybird.co/docs/about:blank#use-a-pipe-to-filter-and-extract-items)

Data is now be available in your landing Data Source. However, you need to use the `JSONExtract` function to access attributes from the `_record` column. To optimize performance, use [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) to extract and store item types in separate Data Sources with their own schemas.

Create a Pipe, use the PK and SK columns as needed to filter for a particular item type, and parse the attributes from the JSON in `_record` column.

The example table has User and Workspace items, requiring a total of two Materialized Views, one for each item type.



<-figure->
![Workspace Data Flow showing std connection, landing DS and users and workspaces Materialized Views](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-ddb-std-4.png&w=3840&q=75)

<-figcaption->
Two Materialized Views from landing DS





To extract the Workspace items, the Pipe uses the SK to filter for Workspace items, and parses the attributes from the JSON in `_record` column. For example:

SELECT
  toLowCardinality(splitByChar('#', PK)[2]) org,
  toUInt32(splitByChar('#', SK)[2]) workspace_id,
  JSONExtractString(_record,'ws_name') ws_name,
  toUInt16(JSONExtractUInt(_record,'cores')) cores,
  JSONExtractUInt(_record,'storage_tb') storage_tb,
  _record,
  _old_record,
  _timestamp,
  _is_deleted
FROM dynamodb_ds_std
WHERE splitByChar('#', SK)[1] = 'WS'
## Create the Materialized Views [¶](https://www.tinybird.co/docs/about:blank#create-the-materialized-views)

Create a Materialized View from the Pipe to store the extracted data in a new Data Source.

The Materialized View must use the ReplacingMergeTree engine to handle the deduplication of rows, supporting updates and deletes from DynamoDB. Use the following engine settings and configure them as needed for your table:

- `ENGINE "ReplacingMergeTree"`   : the ReplacingMergeTree engine is used to deduplicate rows.
- `ENGINE_SORTING_KEY "key1, key2"`   : the columns used to identify unique items, can be one or more columns, typically the part of the PK and SK that isn't idetifying Item type.
- `ENGINE_VER "_timestamp"`   : the column used to identify the most recent row for each key.
- `ENGINE_IS_DELETED "_is_deleted"`   : the column used to identify if a row has been deleted.

For example, the Materialized View for the Workspace items uses the following schema and engine settings:

SCHEMA >
    `org` LowCardinality(String),
    `workspace_id` UInt32,
    `ws_name` String,
    `cores` UInt16,
    `storage_tb` UInt64,
    `_record` String,
    `_old_record` Nullable(String),
    `_timestamp` DateTime64(3),
    `_is_deleted` UInt8

ENGINE "ReplacingMergeTree"
ENGINE_SORTING_KEY "org, workspace_id"
ENGINE_VER "_timestamp"
ENGINE_IS_DELETED "_is_deleted" Repeat the same process for each item type.



<-figure->
![Materialized View for extracting Users attributes](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-ddb-std-3.png&w=3840&q=75)

<-figcaption->
Materialized View for extracting Users attributes





You have now your Data Sources with the extracted columns ready to be queried.

## Review performance gains [¶](https://www.tinybird.co/docs/about:blank#review-performance-gains)

This process offers significant performance gains over querying the landing Data Source. To demonstrate this, you can use a Playground to compare the performance of querying the raw data vs the extracted data.

For the example table, the following queries aggregate the total number of users, workspaces, cores, and storage per organization using the unoptimized raw data and the optimized extracted data. The query over raw data took 335 ms, while the query over the extracted data took 144 ms, for a 2.3x improvement.

NODE users_stats
SQL >
    SELECT org, count() total_users 
    FROM ddb_users_mv FINAL
    GROUP BY org


NODE ws_stats
SQL >
    SELECT org, count() total_workspaces, sum(cores) total_cores, sum(storage_tb) total_storage_tb
    FROM ddb_workspaces_mv FINAL
    GROUP BY org


NODE users_stats_raw
SQL >
    SELECT
      toLowCardinality(splitByChar('#', PK)[2]) org,
      count() total_users
    FROM dynamodb_ds_std FINAL
    WHERE splitByChar('#', SK)[1] = 'USER'
    GROUP BY org


NODE ws_stats_raw
SQL >
    SELECT
      toLowCardinality(splitByChar('#', PK)[2]) org,
      count() total_ws,
      sum(toUInt16(JSONExtractUInt(_record,'cores'))) total_cores,
      sum(JSONExtractUInt(_record,'storage_tb')) total_storage_tb
    FROM dynamodb_ds_std FINAL
    WHERE splitByChar('#', SK)[1] = 'WS'
    GROUP BY org


NODE org_stats
SQL >
    SELECT * FROM users_stats JOIN ws_stats using org


NODE org_stats_raw
SQL >
    SELECT * FROM users_stats_raw JOIN ws_stats_raw using org This is how the outcome looks in Tinybird:



<-figure->
![Comparison of same query](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-ddb-std-5.png&w=3840&q=75)

<-figcaption->
Same info, faster and more efficient from Materialized Views







---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-dub
Last update: 2025-01-14T18:33:52.000Z
Content:
---
title: "Send Dub webhooks to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Dub to Tinybird."
---




# Send Dub webhooks to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-dub-webhooks-to-tinybird)

With [Dub](https://dub.co/) , you can shorten any link and get powerful [conversion analytics](https://dub.co/analytics) . By integrating Dub with Tinybird, you can analyze your events and usage data in real time.

Some common use cases for sending Dub webhooks to Tinybird include:

1. Tracking link clicks.
2. Monitoring link performance.
3. Analyzing user engagement patterns.
4. Creating custom dashboards for link performance.
5. Enriching other data sources with real-time link metrics.

Read on to learn how to send data from Dub to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Dub to Tinybird, ensure:

- You have a Dub account.
- You have a Tinybird Workspace.

## Connect Dub to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-dub-to-tinybird)

1. Open the Dub UI and go to the**  Settings**   >**  Webhooks**   page.
2. Select**  Create Webhook**  .
3. In Tinybird, create a Data Source, called `dub`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/dub.datasource)  :

SCHEMA >
  `event_time` DateTime `json:$.tinybirdIngestTime` DEFAULT now(),
  `event_type` String `json:$.event` DEFAULT 'unknown',
  `event` JSON(max_dynamic_types=2, max_dynamic_paths=16) `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Dub in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Dub, paste the Events API URL as your webhook URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=dub&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Select the checkboxes for the Dub events you want to send to Tinybird, and select**  Create webhook**  .
2. You're done. Dub will now push events to Tinybird via the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .
- [  Dub webhooks docs](https://dub.co/docs/integrations/webhooks)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-csv-files
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Ingest CSV files · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to ingest data into Tinybird using CSV (comma-separated values) files."
---




# Ingest CSV files [¶](https://www.tinybird.co/docs/about:blank#ingest-csv-files)

CSV (comma-separated values) is one of the most widely used formats out there. However, it's used in different ways; some people don't use commas, and other people use escape values differently, or are unsure about using headers.

The Tinybird platform is smart enough to handle many scenarios. If your data doesn't comply with format and syntax best practices, Tinybird will still aim to understand your file and ingest it, but following certain best practices can speed your CSV processing speed by up to 10x.

## Syntax best practices [¶](https://www.tinybird.co/docs/about:blank#syntax-best-practices)

By default, Tinybird processes your CSV file assuming the file follows the most common standard ( [RFC4180](https://datatracker.ietf.org/doc/html/rfc4180#section-2) ). Key points:

- Separate values with commas.
- Each record is a line (with CRLF as the line break). The last line may or may not have a line break.
- First line as a header is optional (though not using one is faster in Tinybird.)
- Double quotes are optional but using them means you can escape values (for example, if your content has commas or line breaks).

Example: Instead of using the backslash `\` as an escape character, like this:

1234567890,0,0,0,0,2021-01-01 10:00:00,"{\"authorId\":\"123456\",\"handle\":\"aaa\"}" Use two double quotes:

##### More performant

1234567890,0,0,0,0,2021-01-01 10:00:00,"{""authorId"":""123456"",""handle"":""aaa""}"
- Fields containing line breaks, double quotes, and commas should be enclosed in double quotes.
- Double quotes can also be escaped by using another double quote (""aaa"",""b""""bb"",""ccc"")

In addition to the previous points, it's also recommended to:

1. Format `DateTime`   columns as `YYYY-MM-DD HH:MM:SS`   and `Date`   columns as `YYYY-MM-DD`  .
2. Send the encoding in the `charset`   part of the `content-type`   header, if it's different to UTF-8. The expectation is UTF-8, so it should look like this `Content-Type: text/html; charset=utf-8`  .
3. You can set values as `null`   in different ways, for example,*  ""[]""*  ,*  """"*   (empty space),*  N*   and*  "N"*  .
4. If you use a delimiter other than a comma, explicitly define it with the API parameter*  ``dialect_delimiter``.*
5. If you use an escape character other than a ", explicitly define it with the API parameter*  ``dialect_escapechar``.*
6. If you have no option but to use a different line break character, explicitly define it with the API parameter `dialect_new_line`  .

For more information, check the [Data Sources API docs](https://www.tinybird.co/docs/docs/api-reference/datasource-api).

## Append data [¶](https://www.tinybird.co/docs/about:blank#append-data)

Once the Data Source schema has been created, you can optimize your performance by not including the header. Just keep the data in the same order.

However, if the header is included and it contains all the names present in the Data Source schema the ingestion will still work (even if the columns follow a different order to the initial creation).

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Got your schema sorted and ready to make some queries? Understand[  how to work with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time)  .
- Learn how to[  monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-clerk
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Clerk webhooks to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send data from Clerk to Tinybird."
---




# Send Clerk webhooks to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-clerk-webhooks-to-tinybird)

[Clerk](https://clerk.com/) is a developer-focused user management platform to handle user authentication with many prebuilt UI components. By integrating Clerk with Tinybird, you can analyze your user authentication data in real time and enrich it with other data sources.

Some common use cases for sending Clerk webhooks to Tinybird include:

1. Tracking net user and organization growth.
2. Monitoring user churn.
3. Identifying common auth errors.
4. Creating custom dashboards for auth analysis.
5. Enriching other data sources with real-time auth metrics.

Read on to learn how to send data from Clerk to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Clerk to Tinybird, ensure:

- You have a Clerk account.
- You have a Tinybird Workspace.

## Connect Clerk to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-clerk-to-tinybird)

1. From the Clerk UI, select**  Configure**   >**  Webhooks**  .
2. Select**  Add Endpoint**  .
3. In Tinybird, create a Data Source, called `clerk`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/clerk.datasource)  :

SCHEMA >
    `event_time` DateTime64(3) `json:$.tinybirdIngestTime` DEFAULT now(),
    `event_type` String `json:$.type` DEFAULT 'unknown',
    `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Clerk in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. Back in Clerk, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird, for example:

https://https://api.tinybird.co.tinybird.co/v0/events?name=clerk

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

1. Return to Tinybird, and copy a token with privileges to write to the Data Source you created. You can use the admin token or create one with the required scope.
2. Return to the Clerk Webhooks page, and update the URL to add a new search parameter `token`   with the token you copied. The final URL looks like the following:

https://europe-west3.tinybird.co/v0/events?name=clerk&token=p.eyXXXXX
1. Select the checkboxes for the Clerk events you want to send to Tinybird, and select**  Create**  .
2. You're done. Any of the Clerk events you selected is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)   . You can test the integration from the**  Testing**   tab in the Clerk Webhooks UI.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-from-aws-kinesis
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Stream from AWS Kinesis · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to send data from AWS Kinesis to Tinybird."
---




# Stream from AWS Kinesis [¶](https://www.tinybird.co/docs/about:blank#stream-from-aws-kinesis)

In this guide, you'll learn how to send data from AWS Kinesis to Tinybird.

If you have a [Kinesis Data Stream](https://aws.amazon.com/kinesis/data-streams/) that you want to send to Tinybird, it should be pretty quick thanks to [Kinesis Firehose](https://aws.amazon.com/kinesis/data-firehose/) . This page explains how to integrate Kinesis with Tinybird using Firehose.

## 1. Push messages From Kinesis To Tinybird [¶](https://www.tinybird.co/docs/about:blank#1-push-messages-from-kinesis-to-tinybird)

### Create a Token with the right scope [¶](https://www.tinybird.co/docs/about:blank#create-a-token-with-the-right-scope)

In your Workspace, create a Token with the `Create new Data Sources or append data to existing ones` scope:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fingest-from-aws-kinesis-1.png&w=3840&q=75)

<-figcaption->
Create a Token with the right scope





### Create a new Data Stream [¶](https://www.tinybird.co/docs/about:blank#create-a-new-data-stream)

Start by creating a new Data Stream in AWS Kinesis (see the [AWS documentation](https://docs.aws.amazon.com/streams/latest/dev/working-with-streams.html) for more information).



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fingest-from-aws-kinesis-2.png&w=3840&q=75)

<-figcaption->
Create a Kinesis Data Stream





### Create a Firehose Delivery Stream [¶](https://www.tinybird.co/docs/about:blank#create-a-firehose-delivery-stream)

Next, [create a Kinesis Data Firehose Delivery Stream](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).

Set the **Source** to **Amazon Kinesis Data Streams** and the **Destination** to **HTTP Endpoint**.

In the **Destination Settings** , set **HTTP Endpoint URL** to point to the [Tinybird Events API](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api).

https://api.tinybird.co/v0/events?name=&wait=true&token= This example is for Workspaces in the `GCP` --> `europe-west3` region. If necessary, replace with the [correct region for your Workspace](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) . Additionally, note the `wait=true` parameter. Learn more about it [in the Events API docs](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api#wait-for-acknowledgement).

You don't need to create the Data Source in advance; it will automatically be created for you.

### Send sample messages and check that they arrive to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-sample-messages-and-check-that-they-arrive-to-tinybird)

If you don't have an active data stream, follow [this python script](https://gist.github.com/GnzJgo/f1a80186a301cd8770a946d02343bafd) to generate dummy data.

Back in Tinybird, you should see 3 columns filled with data in your Data Source. `timestamp` and `requestId` are self explanatory, and your messages are in `records\_\data`:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fingest-from-aws-kinesis-3.png&w=3840&q=75)

<-figcaption->
Firehose Data Source





## 2. Decode message data [¶](https://www.tinybird.co/docs/about:blank#2-decode-message-data)

### Decode message data [¶](https://www.tinybird.co/docs/about:blank#decode-message-data)

The `records\_\data` column contains an array of encoded messages.

In order to get one row per each element of the array, use the ARRAY JOIN Clause. You'll also need to decode the messages with the base64Decode() function.

Now that the raw JSON is in a column, you can use [JSONExtract functions](https://www.tinybird.co/docs/docs/sql-reference/functions/json-functions) to extract the desired fields:

##### Decoding messages

NODE decode_messages
SQL >
   SELECT
       base64Decode(encoded_m) message,
       fromUnixTimestamp64Milli(timestamp) kinesis_ts
   FROM firehose
   ARRAY JOIN records__data as encoded_m
 
NODE extract_message_fields
SQL >
   SELECT
       kinesis_ts,
       toDateTime64(JSONExtractString(message, 'datetime'), 3) datetime,
       JSONExtractString(message, 'event') event,
       JSONExtractString(message, 'product') product
   FROM decode_messages

<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fingest-from-aws-kinesis-4.png&w=3840&q=75)

<-figcaption->
Decoding messages





## Recommended settings [¶](https://www.tinybird.co/docs/about:blank#recommended-settings)

When configuring AWS Kinesis as a Data Source, use the following settings:

- Set `wait=true`   when calling the Events API. See[  the Events API docs](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api#wait-for-acknowledgement)   for more information.
- Set the buffer size lower than 10 Mb in Kinesis.
- Set 128 shards as the maximum in Kinesis.

## Performance optimizations [¶](https://www.tinybird.co/docs/about:blank#performance-optimizations)

It is highly recommended to persist the decoded and unrolled result in a different Data Source. You can do it with a [Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) : A combination of a Pipe and a Data Source that leaves the transformed data into the destination Data Source as soon as new data arrives to the Firehose Data Source.

Don't store what you won't need. In this example, some of the extra columns could be skipped. [Add a TTL](https://www.tinybird.co/docs/docs/get-data-in/data-sources#setting-data-source-ttl) to the Firehose Data Source to prevent keeping more data than you need.

Another alternative is to create the Firehose Data Source with a Null Engine. This way, data ingested there can be transformed and fill the destination Data Source without being persisted in the Data Source with the Null Engine.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Ingest from other sources - see the[  Overview page](https://www.tinybird.co/docs/docs/get-data-in)   and explore.
- Build your first[  Tinybird Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes)  .



---

URL: https://www.tinybird.co/docs/get-data-in/guides/ingest-auth0-logs
Last update: 2025-01-08T09:32:25.000Z
Content:
---
title: "Send Auth0 Log Streams to Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to send Auth0 Log Streams to Tinybird using webhooks and the Events API."
---




# Send Auth0 Logs Streams to Tinybird [¶](https://www.tinybird.co/docs/about:blank#send-auth0-logs-streams-to-tinybird)

[Auth0](https://auth0.com/) is a developer-focused user management platform to handle user authentication with many prebuilt UI components. By integrating Auth0 with Tinybird, you can analyze your user authentication data in real time and enrich it with other data sources.

Some common use cases for sending Auth0 logs to Tinybird include:

1. Tracking net user and organization growth.
2. Monitoring user churn.
3. Identifying common auth errors.
4. Creating custom dashboards for auth analysis.
5. User authentication audit logs.

Read on to learn how to send data from Auth0 Logs Streams to Tinybird.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you connect Auth0 Logs Streams to Tinybird, ensure:

- You have an Auth0 account.
- You have a Tinybird Workspace.

## Connect Auth0 to Tinybird [¶](https://www.tinybird.co/docs/about:blank#connect-auth0-to-tinybird)

1. From the Auth0 dashboard, select**  Monitoring**   >**  Streams**  .
2. Select**  Create Stream**  .
3. In Tinybird, create a Data Source, called `auth0`   . You can follow this[  schema](https://github.com/tinybirdco/tinynest/blob/main/tinybird/datasources/auth0.datasource)  :

SCHEMA >
    `event_time` DateTime64(3) `json:$.tinybirdIngestTime` DEFAULT now(),
    `event_type` String `json:$.data.type` DEFAULT 'unknown',
    `event` JSON `json:$` DEFAULT '{}'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(event_time)"
ENGINE_SORTING_KEY "event_time" Using the [JSON Data Type](https://www.tinybird.co/docs/docs/sql-reference/data-types/json) you can store the semi-structured data you receive from Auth0 Logs Streams in a single column. You can later retrieve various events and their metadata as needed in your Pipes.

The `JSON` data type is in private beta. If you are interested in using this type, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

1. In Tinybird, copy a token with privileges to append to the Data Source you created. You can use the admin token or create one with the required scope.
2. Back in Auth0, paste the Events API URL in your Webhook Endpoint URL. Use the query parameter `name`   to match the name of the Data Source you created in Tinybird. For example:

https://api.tinybird.co/v0/events?name=auth0&token=

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

Content Type is `application/json` and Content Format is `JSON Lines`.

1. Select the any event category to filter, like `All`   , and a date in case you want to perform some backfilling. Then select**  Save**  .
2. You're done. Any of the Auth0 Log Streams events you selected is automatically sent to Tinybird through the[  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)  .

You can check the status of the integration from the **Health** tab in the created webhook or from the **Log** tab in the Tinybird `auth0` Data Source.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Events API](https://www.tinybird.co/docs/docs/get-data-in/ingest-apis/events-api)
- [  Auth0 Logs Streams](https://auth0.com/docs/customize/log-streams/custom-log-streams)



---

URL: https://www.tinybird.co/docs/get-data-in/data-operations/scheduling-with-github-actions-and-cron
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Schedule data ingestion · Tinybird Docs"
theme-color: "#171612"
description: "Cronjobs are the universal way of scheduling tasks. In this guide, you'll learn how to keep your data in sync with cron jobs or GitHub Actions and the Tinybird API."
---




# Schedule data ingestion with cron and GitHub Actions [¶](https://www.tinybird.co/docs/about:blank#schedule-data-ingestion-with-cron-and-github-actions)

Cronjobs are the universal way of scheduling tasks. In this guide, you'll learn how to keep your data in sync with cronjobs or GitHub actions and the Tinybird REST API.

## Overview [¶](https://www.tinybird.co/docs/about:blank#overview)

For this example, let's assume you've already imported a Data Source to your Tinybird account and that you have properly defined its schema and partition key. Once everything is set, you can easily perform some operations using the [Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api) to **periodically append to or replace data** in your Data Sources. This guide shows you some examples.

## About crontab [¶](https://www.tinybird.co/docs/about:blank#about-crontab)

Crontab is a native Unix tool that schedules command execution at a specified time or time interval. It works by defining the schedule, and the command to execute, in a text file. This can be achieved using `sudo crontab -e` . You can learn more about using crontab using many online resources like [crontab.guru](https://crontab.guru/crontab.5.html) and [the man page for crontab](https://man7.org/linux/man-pages/man5/crontab.5.html).

### The cron table format [¶](https://www.tinybird.co/docs/about:blank#the-cron-table-format)

Cron follows a table format like the following (note that you can also use [external tools like crontab.guru](https://crontab.guru/) to help you define the cron job schedule):

##### Cron syntax explanation

*   *   *   *   *  Command_to_execute
|   |   |   |   |       
|   |   |   |   Day of the Week ( 0 - 6 ) ( Sunday = 0 )
|   |   |   |
|   |   |   Month ( 1 - 12 )
|   |   |
|   |   Day of Month ( 1 - 31 )
|   |
|   Hour ( 0 - 23 )
|
Min ( 0 - 59 ) Using this format, the following would be typical cron schedules to execute commands at different times:

- Every five minutes: `0/5 \* \* \* \*`
- Every day at midnight: `0 0 \* \* \*`
- Every first day of month: `\* \* 1 \* \*`
- Every Sunday at midnight: `0 0 \* \* 0`

Be sure you save your scripts in the right location. Save your shell scripts in the `/opt/cronjobs/` folder.

## Append data periodically [¶](https://www.tinybird.co/docs/about:blank#append-data-periodically)

It's very common to have a Data Source that grows over time. There is often is also an ETL process extracting this data from the transactional database and generating CSV files with the last X hours or days of data, therefore you might want to append those recently-generated rows to your Tinybird Data Source.

For this example, imagine you generate new CSV files every day at 00:00 that you want to append to Tinybird everyday at 00:10.

### Option 1: With a shell script [¶](https://www.tinybird.co/docs/about:blank#option-1-with-a-shell-script)

First, you need to create a shell script file containing the Tinybird API request operation:

##### Contents of append.sh

#!/bin/bash
TOKEN=your_token
CSV_URL="http://your_url.com"

curl \
  -H "Authorization: Bearer $TOKEN" \
  -X POST \
  -d url=$CSV_URL \
  -d mode='append' \
  -d name='events' \
  https://api.tinybird.co/v0/datasources Then, add a new line to your crontab file (using `sudo crontab -e` ):

10 0 * * * sh -c /opt/cronjobs/append.sh
### Option 2: Using GitHub Actions [¶](https://www.tinybird.co/docs/about:blank#option-2-using-github-actions)

If your project is hosted on GitHub, you can also use GitHub Actions to schedule periodic jobs. Create a new file called `.github/workflows/append.yml` with the following code to append data from a CSV given its URL every day at 00:10.

##### Contents of .github/workflows/append.yml

name: Append data data every day at 00:10

on:
  push:
  workflow_dispatch:
  schedule:
    - cron:  '10 0 * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
      uses: actions/checkout@v2
    - name: Append new data
      run: |-
        curl \
          -H "Authorization: Bearer $TOKEN" \
          -X POST \
            -d url=$CSV_URL \
            -d mode='append' \
            -d name='events' \
            https://api.tinybird.co/v0/datasources
## Replace data periodically [¶](https://www.tinybird.co/docs/about:blank#replace-data-periodically)

Let's use another example. With this new fictional Data Source, imagine a scenario where you want to replace the whole Data Source with a CSV file sitting in a publicly-accessible URL every first day of the month.

### Option 1: With a shell script [¶](https://www.tinybird.co/docs/about:blank#option-1-with-a-shell-script)

##### Contents of replace.sh

#!/bin/bash
TOKEN=your_token
CSV_URL="http://your_url.com"

curl \
  -H "Authorization: Bearer $TOKEN" \
  -X POST \
  -d url=$CSV_URL \
  -d mode='replace' \
  -d name='events' \
  https://api.tinybird.co/v0/datasources Then edit the crontab file which takes care of periodically executing your script. Run `sudo crontab -e`:

##### Setting up a crontab to run a script periodically

* * 1 * * sh -c /opt/cronjobs/replace.sh
### Option 2: With GitHub Actions [¶](https://www.tinybird.co/docs/about:blank#option-2-with-github-actions)

Create a new file called `.github/workflows/replace.yml` with the following code to replace all your data with given the URL of the CSV with the new data every day at 00:10.

##### Contents of .github/workflows/replace.yml

name: Replace all data every day at 00:10

on:
  push:
  workflow_dispatch:
  schedule:
    - cron:  '10 0 * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
      uses: actions/checkout@v2
    - name: Replace all data
      run: |-
        curl \
          -H "Authorization: Bearer $TOKEN" \
          -X POST \
            -d url=$CSV_URL \  
            -d mode='replace' \
            -d name='events' \
            https://api.tinybird.co/v0/datasources
## Replace just one month of data [¶](https://www.tinybird.co/docs/about:blank#replace-just-one-month-of-data)

Having your API call inside a shell script allows you to script more complex ingestion processes. For example, imagine you want to replace the last month of events data, every day. Then each day, you would export a CSV file to a publicly accessible URL and name it something like `events_YYYY-MM-DD.csv`.

### Option 1: With a shell script [¶](https://www.tinybird.co/docs/about:blank#option-1-with-a-shell-script)

You could script a process that would do a conditional data replacement as follows:

##### Script to replace data selectively on Tinybird

#!/bin/bash

TODAY=`date +"%Y-%m-%d"`
ONE_MONTH_AGO=`date -v -1m +%Y-%m-%d`
TOKEN=your_token
DATASOURCE=events
CSV_URL="http://your_url.com"

curl \
  -H "Authorization: Bearer $TOKEN" \
  -X POST \
  -d url=$CSV_URL \  
  -d mode='replace' \
  -d replace_condition=(created_at+BETWEEN+'${ONE_MONTH_AGO}'+AND+'${TODAY}')" \
  -d name=$DATASOURCE \
  https://api.tinybird.co/v0/datasources Then, after saving that file to `/opt/cronjobs/daily_replace.sh` , add the following line to `crontab` to run it every day at midnight:

##### Setting up a crontab to run a script periodically

0 0 * * * sh -c /opt/cronjobs/daily_replace.sh
### Option 2: With GitHub Actions [¶](https://www.tinybird.co/docs/about:blank#option-2-with-github-actions)

Create a new file called `.github/workflows/replace_last_month.yml` with the following code to replace all the data for the last month every day at 00:10.

##### Contents of .github/workflows/replace.yml

name: Replace all data every day at 00:10

on:
  push:
  workflow_dispatch:
  schedule:
    - cron:  '10 0 * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
      uses: actions/checkout@v2
    - name: Replace all data
      run: |-
        TODAY=`date +"%Y-%m-%d"`                                 
        ONE_MONTH_AGO=`date -v -1m +%Y-%m-%d`
        DATASOURCE=events # could also be set via github secrets
        CSV_URL="http://your_url.com" # could also be set via github secrets
        curl \
          -H "Authorization: Bearer $TOKEN" \
          -X POST \
          -d url=$CSV_URL \  
          -d mode='replace' \
          -d replace_condition=(created_at+BETWEEN+'${ONE_MONTH_AGO}'+AND+'${TODAY}')" \
          -d name=$DATASOURCE \
          https://api.tinybird.co/v0/datasources Use GitHub secrets: Store `TOKEN` as an [encrypted secret](https://docs.github.com/en/actions/reference/encrypted-secrets) to avoid hardcoding secret keys in your repositories, and replace `DATASOURCE` and `CSV_URL` by their values or save them as secrets as well.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about[  GitHub Actions and CI/CD processes on Tinybird](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration)  .
- Understand how to[  work with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time)  .



---

URL: https://www.tinybird.co/docs/get-data-in/data-operations/replace-and-delete-data
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Replace and delete data · Tinybird Docs"
theme-color: "#171612"
description: "Update & delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird."
---




# Replace and delete data in your Tinybird Data Sources [¶](https://www.tinybird.co/docs/about:blank#replace-and-delete-data-in-your-tinybird-data-sources)

Update and delete operations are common in transactional databases over operational data, but sometimes you also need to make these changes on your analytical data in Tinybird.

Sometimes, you need to delete or replace some of your data in Tinybird. Perhaps there was a bug in your app, a transient error in your operational database, or simply an evolution of requirements due to product or regulatory changes.

It's **not safe** to replace data in the partitions where you are actively ingesting data. You may lose the data inserted during the process.

Tinybird works well with append-only workloads but also fully supports replacing and deleting data. It abstracts away the tricky complexities of data replication, partition management and mutations rewriting, allowing you to focus on your data engineering flows and not the internals of real-time analytical databases.

This guide shows you using different examples, how to selectively delete or update data in Tinybird using the REST API. You can then adapt these processes for your own needs.

All operations on this page require a Token with the correct scope. In the code snippets, replace `` by a Token whose [scope](https://www.tinybird.co/docs/docs/api-reference/token-api) is `DATASOURCES:CREATE` or `ADMIN`.

## Delete data selectively [¶](https://www.tinybird.co/docs/about:blank#delete-data-selectively)

To delete data that's within a condition, send a POST request to the [Data Sources /delete API](https://www.tinybird.co/docs/docs/api-reference/datasource-api#post--v0-datasources-(.+)-delete) , providing the name of one of your Data Sources in Tinybird and a `delete_condition` parameter, which is an SQL expression filter.

Delete operations don't automatically cascade to downstream Materialized Views. You may need to perform separate delete operations on Materialized Views.

Imagine you have a Data Source called `events` and you want to remove all the transactions for November 2019. You'd send a POST request like this:

- CLI
- API

##### Delete data selectively

tb datasource delete events --sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" Once you make the request, you can see that the `POST` request to the delete API Endpoint is asynchronous. It returns a [job response](https://www.tinybird.co/docs/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs) , indicating an ID for the job, the status of the job, the `delete_condition` , and some other metadata. Although the delete operation runs asynchronously, the operation waits synchronously for all the mutations to be rewritten and delete the data replicas. Queries reading data either see the state before the operation or after it's complete.

{
  "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job_url": "https://api.tinybird.co/v0/jobs/64e5f541-xxxx-xxxx-xxxx-00524051861b",
  "job": {
    "kind": "delete_data",
    "id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
    "job_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b",
    "status": "waiting",
    "created_at": "2023-04-11 13:52:32.423207",
    "updated_at": "2023-04-11 13:52:32.423213",
    "started_at": null,
    "is_cancellable": true,
    "datasource": {
      "id": "t_c45d5ae6781b41278fcee365f5bxxxxx",
      "name": "shopping_data"
    },
    "delete_condition": "event = 'search'"
  },
  "status": "waiting",
  "delete_id": "64e5f541-xxxx-xxxx-xxxx-00524051861b"
} You can periodically poll the `job_url` with the given ID to check the status of the deletion process. When the status is `done` your job deleted the data matching the SQL expression filter and all your Pipes and API Endpoints continue running with the remaining data in the Data Source.

### Truncate a Data Source [¶](https://www.tinybird.co/docs/about:blank#truncate-a-data-source)

Sometimes you want to delete all data contained in a Data Source. You can perform this action from the UI and API.

Using the API, the [truncate](https://www.tinybird.co/docs/docs/api-reference/datasource-api#post--v0-datasources-(.+)-truncate) endpoint deletes all rows in a Data Source as shown in this example:

- CLI
- API

##### Truncate a Data Source

tb datasource truncate  You can also truncate a Data Source directly from the UI:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Freplacing-and-deleting-data-1.png&w=3840&q=75)

<-figcaption->
Deleting selectively is only available via API, but truncating it to delete all the data is availabl via the UI.





## Replace data selectively [¶](https://www.tinybird.co/docs/about:blank#replace-data-selectively)

The ability to update data is often not the top priority when designing analytical databases, but there are always scenarios where you need to update or replace your analytical data. For example, you might have reconciliation processes over your transactions that affect your original data. Or maybe your ingestion process was simply faulty, and you ingested inaccurate data for a period of time.

In Tinybird, you can specify a condition to replace only part of the data during the ingestion process. For instance, if you want to reingest a CSV with the data for November 2019 and update your Data Source accordingly. To update the data, you pass the `replace_condition` parameter with the `toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'` condition.

- CLI
- API

##### Replace data selectively

tb datasource replace events \
https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv \
--sql-condition "toDate(date) >= '2019-11-01' and toDate(date) <= '2019-11-30'" The response to the previous API call looks like this:

##### Response after replacing data

{
    "id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job_url": "https://api.tinybird.co/v0/jobs/a83fcb35-8d01-47b9-842c-a288d87679d0",
    "job": {
        "kind": "import",
        "id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "job_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0",
        "status": "waiting",
        "statistics": null,
        "datasource": { ... },
        "quarantine_rows": 0,
        "invalid_lines": 0
    },
    "status": "waiting",
    "import_id": "a83fcb35-8d01-47b9-842c-a288d87679d0"
} As in the case of the selective deletion, selective replacement also runs as an asynchronous request, so [check the status of the job](https://www.tinybird.co/docs/docs/api-reference/jobs-api#jobs-api-getting-information-about-jobs) periodically. You can see the status of the job by using the `job_url` returned in the previous response.

### About the replace condition [¶](https://www.tinybird.co/docs/about:blank#about-the-replace-condition)

Conditional replaces apply over partitions and the match condition selects partitions needed for the operation. The records remaining after the match condition determine the partitions involved.

Always include the parition key in the replace condition to maintain consistency.

The replace condition filters the new data that's appended, meaning it excludes rows not matching the condition. The condition is also applied for the selected partitions in the Data Source, removing rows that don't match the condition in these partitions. Rows that don't match the condition and may be present in other partitions remain.

See the [example](https://www.tinybird.co/docs/docs/get-data-in/data-operations/replace-and-delete-data#example) that follows for a better understanding of selectively replacing data in a datasource.

### Linked Materialized Views [¶](https://www.tinybird.co/docs/about:blank#linked-materialized-views)

If you have several connected Materialized Views, then selective replaces proceed in a cascading fashion. For example, if datasource A materializes data to datasource B and from there to datasource C, then when you replace data in datasource A, datasources B and C automatically update accordingly. All three Data Sources need to have compatible partition keys since replaces processed by partition.

The command `tb dependencies --datasource the_data_source --check-for-partial-replace` returns the dependencies for both for datasouces and materialized views and raises an error if any of the dependencies have incompatible partition keys.

Remember: The provided Token must have the `DATASOURCES:CREATE` [scope](https://www.tinybird.co/docs/docs/api-reference/token-api).

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

For this example, consider this Data Source:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Freplacing-example-1.jpeg&w=3840&q=75)


Its partition key is `ENGINE_PARTITION_KEY "profession"` . If you wanted to replace the last two rows with new data, you can send this request with the replace condition `replace_condition=(profession='Jedi')`:

- CLI
- API

##### Replace with partition in condition

echo "50,Mace Windu,Jedi" > jedi.csv
tb datasource replace characters jedi.csv --sql-condition "profession='Jedi'" Since the replace condition column matches the partition key, the result is:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Freplacing-example-2.jpeg&w=3840&q=75)


However, consider what happens if you create the Data Source with `ENGINE_PARTITION_KEY "name"`:

##### characters.datasource

SCHEMA >
    `age` Int16,
    `name` String,
    `profession` String

ENGINE "MergeTree"
ENGINE_SORTING_KEY "age, name, profession"
ENGINE_PARTITION_KEY "name" If you were to run the same replace request, the result probably doesn't make sense:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Freplacing-example-3.jpeg&w=3840&q=75)


Why were the existed rows not removed? Because the `replace` process uses the payload rows to identify which partitions to work on. The Data Source is now partitioned by name and not profession, so the process didn't delete the other "Jedi" rows. They're in different partitions because they have different names.

The rule of thumb is this: **Always make sure the replace condition uses the partition key as the filter field**.

## Replace a Data Source completely [¶](https://www.tinybird.co/docs/about:blank#replace-a-data-source-completely)

To replace a complete Data Source, make an API call similar to the previous example, without providing a `replace_condition`:

- CLI
- API

##### Replace Data Source completely

tb datasource replace events https://storage.googleapis.com/tinybird-assets/datasets/guides/events_1M_november2019_1.csv The example request is replacing a Data Source with the data found in a given URL pointing to a CSV file. You can also replace a Data Source in the Tinybird UI:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Freplacing-and-deleting-data-2.png&w=3840&q=75)

<-figcaption->
Replacing a Data Source completely through the User Interface





Schemas must be identical. When replacing data either selectively or entirely, the schema of the new inbound data must match that of the original Data Source. Rows not containing the same schema go to quarantine.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn how[  to get rows out of quarantine](https://www.tinybird.co/docs/docs/get-data-in/data-sources#the-quarantine-data-source)  .
- Need to[  iterate a Data Source, including the schema](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source)   ? Read how here.



---

URL: https://www.tinybird.co/docs/get-data-in/data-operations/recover-from-quarantine
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Recover data in quarantine · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to recover data from quarantine, and how to fix common errors that cause data to be sent to quarantine."
---




# Recover data from quarantine [¶](https://www.tinybird.co/docs/about:blank#recover-data-from-quarantine)

In this guide you'll learn about the quarantine Data Source, and how to use it to detect and fix errors on your Data Sources.

The quarantine Data Source is named `{datasource_name}_quarantine` and can be queried using Pipes like a regular Data Source.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes you're familiar with the concept of the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources#the-quarantine-data-source).

## Example scenario [¶](https://www.tinybird.co/docs/about:blank#example-scenario)

This guide uses the Tinybird CLI, but all steps can be performed in the UI as well.

### Setup [¶](https://www.tinybird.co/docs/about:blank#setup)

This example uses an NDJSON Data Source that looks like this:

{
  "store_id": 1,
  "purchase": { "product_name": "shoes", "datetime": "2022-01-05 12:13:14" }
} But you could use any ingestion method.

Let's say you generate a Data Source file from this JSON snippet, push the Data Source to Tinybird, and ingest the JSON as a single row:

##### Push the NDJSON_DS Data Source

echo '{"store_id":1,"purchase":{"product_name":"shoes","datetime":"2022-01-05 12:13:14"}}' > ndjson_ds.ndjson
tb datasource generate ndjson_ds.ndjson
tb push --fixtures datasources/ndjson_ds.datasource
tb sql "select * from ndjson_ds" The schema generated from the JSON will look like this:

##### NDJSON_DS.DATASOURCE

DESCRIPTION >
    Generated from ndjson_ds.ndjson

SCHEMA >
    purchase_datetime DateTime `json:$.purchase.datetime`,
    purchase_product_name String `json:$.purchase.product_name`,
    store_id Int16 `json:$.store_id` At this point, you can in the UI and confirm your Data Source had been created and the row ingested. Hooray!



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-1.png&w=3840&q=75)

<-figcaption->
Data Source details can be accessed from your Sidebar





### Add data that doesn't match the schema [¶](https://www.tinybird.co/docs/about:blank#add-data-that-doesnt-match-the-schema)

Now, if you append some rows that don't match the Data Source schema, these rows will end up in the quarantine Data Source.

##### Append rows with wrong schema

echo '{"store_id":2,"purchase":{"datetime":"2022-01-05 12:13:14"}}\n{"store_id":"3","purchase":{"product_name":"shirt","datetime":"2022-01-05 12:13:14"}}' > ndjson_quarantine.ndjson
tb datasource append ndjson_ds ndjson_quarantine.ndjson
tb sql "select * from ndjson_ds_quarantine" This time, if you check in the UI, you'll see a notification warning you about quarantined rows:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-2.png&w=3840&q=75)

<-figcaption->
The quarantine Data Source is always accessible (if it contains any rows) from the Data Source modal window





In the Data Source view you'll find the Log tab, which shows you details about all operations performed on a Data Source. If you're following the steps of this guide, you should see a row with `event_type` as **append** and `written_rows_quarantine` as **2**.

From the quarantine warning notification, navigate to the quarantine Data Source page, and review the problematic rows:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-3.png&w=3840&q=75)

<-figcaption->
Within the quarantine view you can see both, a summary of errors and the rows that have failed





The **Errors** view shows you a summary of all the errors and the number of occurrences for each of those, so you can prioritize fixing the most common ones. The **Rows** view shows you all the rows that have failed, so you can further investigate why.

## Fix quarantine errors [¶](https://www.tinybird.co/docs/about:blank#fix-quarantine-errors)

There are generally three ways of fixing quarantine errors:

### 1. Modify your data producer [¶](https://www.tinybird.co/docs/about:blank#1-modify-your-data-producer)

Usually, the best solution is to fix the problem at the source. This means updating the applications or systems that are producing the data, before they send it to Tinybird.

The benefit of this is that you don't need to do additional processing to normalize the data after it has been ingested, which helps to save cost and reduce overall latency. However, it can come at the cost of having to push changes into a production application, which can be complex or have side effects on other systems.

### 2. Modify the Data Source schema [¶](https://www.tinybird.co/docs/about:blank#2-modify-the-data-source-schema)

Often, the issue that causes a row to end up in quarantine is a mismatch of data types.

A simple solution is to [modify the Data Source schema](https://www.tinybird.co/docs/docs/get-data-in/data-operations/iterate-a-data-source) to accept the new type.

For example, if an application is starting to send integers that are too large for `Int8` , you might update the schema to use `Int16`.

Avoid Nullable columns, as they can have significantly worse performance. Instead, send alternative values like `0` for any `Int` type, or an empty string for a `String` type.

### 3. Transform data with Pipes and Materialized Views [¶](https://www.tinybird.co/docs/about:blank#3-transform-data-with-pipes-and-materialized-views)

This is one of the most powerful capabilities of Tinybird.

If you aren't able to modify the data producer, you can apply a transformation to the erroring columns at ingestion time and materialize the result into another Data Source.

You can read more about this in the [Materialized Views docs](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views).

## Recover rows from quarantine [¶](https://www.tinybird.co/docs/about:blank#recover-rows-from-quarantine)

The quickest way to recover rows from quarantine is to fix the cause of the errors and then re-ingest the data. However, that isn't always possible.

You can recover rows from the quarantine using a recovery Pipe and the Tinybird API:

### Create a recovery Pipe [¶](https://www.tinybird.co/docs/about:blank#create-a-recovery-pipe)

You can create a Pipe to select the rows from the quarantine Data Source and transform them into the appropriate schema.

The previous example showed rows where the `purchase_product_name` contained `null` or the `store_id` contained a `String` rather than an `Int16`:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-5.png&w=3840&q=75)

<-figcaption->
Remember that quarantined columns are Nullable(String)





All columns in a quarantine Data Source are `Nullable()` , which means that you must use the [coalesce()](https://www.tinybird.co/docs/docs/sql-reference/functions/functions-for-nulls#coalesce) function if you want to transform them into a non-nullable type. This example uses coalesce to set a default value of `DateTime(0)`, `''` , or `0` for `DateTime`, `String` and `Int16` types respectively.

Additionally, all columns in a quarantine Data Source are stored as `String` . This means that you must specifically transform any non-String column into its desired type as part of the recovery Pipe. This example transforms the `purchase_datetime` and `store_id` columns to `DateTime` and `Int16` types respectively.

The quarantine Data Source contains additional meta-columns `c__error_column`, `c__error`, `c__import_id` , and `insertion_date` with information about the errors and the rows, so you should not use `SELECT *` to recover rows from quarantine.

The following SQL transforms the quarantined rows from this example into the original Data Source schema:

SELECT
   coalesce(
      parseDateTimeBestEffortOrNull(
            purchase_datetime
      ),
      toDateTime(0)
   ) as purchase_datetime,
   coalesce(
      purchase_product_name,
      ''
   ) as purchase_product_name,
   coalesce(
      coalesce(
            toInt16(store_id),
            toInt16(store_id)
      ),
      0
   ) as store_id
FROM ndjson_ds_quarantine

<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-6.png&w=3840&q=75)

<-figcaption->
Recover endpoint





Just as with any other Pipe, you can publish the results of this recovery Pipe as an API Endpoint.

### Ingest the fixed rows and truncate quarantine [¶](https://www.tinybird.co/docs/about:blank#ingest-the-fixed-rows-and-truncate-quarantine)

You can then use the Tinybird CLI to append the fixed data back into the original Data Source, by hitting the API Endpoint published from the recovery Pipe:

tb datasource append   To avoid dealing with JSONPaths, you can hit the recovery Pipe's CSV endpoint:

tb datasource append ndjson_ds https://api.tinybird.co/v0/pipes/quarantine_recover.csv?token= Check that your Data Source now has the fixed rows, either in the UI, or from the CLI using:

tb sql "select * from ndjson_ds" Finally, truncate the quarantine Data Source to clear out the recovered rows, either in the UI, or from the CLI using:

tb datasource truncate ndjson_ds_quarantine --yes You should see that your Data Source now has all of the rows, and the quarantine notification has disappeared.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-quarantine-7.png&w=3840&q=75)

<-figcaption->
Data Source with the recovered rows and truncated quarantine





If your quarantine has too many rows, you may need to add pagination based on the `insertion_date` and/or `c__import_id` columns.

If you're using a Kafka Data Source, remember to add the Kafka metadata columns.

## Recover rows from quarantine with CI/CD [¶](https://www.tinybird.co/docs/about:blank#recover-rows-from-quarantine-with-cicd)

When you connect your Workspace to Git and it becomes read-only you want all your workflows to go through CI/CD. This is how you recover rows from quarantine in your data project using Git and automating the workflow.

### Prototype the process in a Branch [¶](https://www.tinybird.co/docs/about:blank#prototype-the-process-in-a-branch)

This step is optional, but it's good practice. When you need to perform a change to your data project and it's read-only, you can create a new Branch and prototype the changes there, then later bring them to Git.

To test this process:

1. Create a Branch
2. Ingest a file that creates rows in quarantine
3. Prototype a Copy Pipe
4. Run it
5. Validate data is recovered

### A practical example with Git [¶](https://www.tinybird.co/docs/about:blank#a-practical-example-with-git)

There is an additional guide showing how to [recover quarantine rows from Git using CI/CD](https://github.com/tinybirdco/use-case-examples/tree/main/recover_data_from_quarantine) , where the data project is the [Web Analytics template](https://www.tinybird.co/templates).

When your rows end up in quarantine, you receive an e-mail like this:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgit-quarantine.jpg&w=3840&q=75)


In this additional example, the issue is the `timestamp` column - instead of being a DateTime, it's String Unix time, so the rows can't be properly ingested.

{"timestamp":"1697393030","session_id":"b7b1965c-620a-402a-afe5-2d0eea0f9a34","action":"page_hit","version":"1","payload":"{ \"user-agent\":\"Mozilla\/5.0 (Linux; Android 13; SM-A102U) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/106.0.5249.118 Mobile Safari\/537.36\", \"locale\":\"en-US\", \"location\":\"FR\", \"referrer\":\"https:\/\/www.github.com\", \"pathname\":\"\/pricing\", \"href\":\"https:\/\/www.tinybird.co\/pricing\"}"} To convert the `timestamp` values in quarantine to a `DateTime` , you'd build a Copy Pipe like this:

NODE copy_quarantine
SQL >
   SELECT
      toDateTime(fromUnixTimestamp64Milli(toUInt64(assumeNotNull(timestamp)) * 1000)) timestamp,
      assumeNotNull(session_id) session_id,
      assumeNotNull(action) action,
      assumeNotNull(version) version,
      assumeNotNull(payload) payload
   FROM analytics_events_quarantine

TYPE COPY
TARGET_DATASOURCE analytics_events To test the changes, you'd need to do a custom deployment:

#!/bin/bash
# use set -e to raise errors for any of the commands below and make the CI pipeline to fail
set -e

tb datasource append analytics_events datasources/fixtures/analytics_events_errors.ndjson
tb deploy
tb pipe copy run analytics_events_quarantine_to_final --wait --yes
sleep 10 First append a sample of the quarantined rows, then deploy the Copy Pipe, and finally run the copy operation.

Once changes have been deployed in a test Branch, you can write data quality tests to validate the rows are effectively being copied:

- analytics_events_quarantine:
    max_bytes_read: null
    max_time: null
    sql: |
      SELECT
            count() as c
      FROM analytics_events_quarantine
      HAVING
            c <= 0

- copy_is_executed:
    max_bytes_read: null
    max_time: null
    sql: |
      SELECT count() c, sum(rows) rows
      FROM tinybird.datasources_ops_log
      WHERE datasource_name = 'analytics_events'
      AND event_type = 'copy'
      HAVING rows != 74 and c = 1 `analytics_events_quarantine` checks that effectively some of the rows are in quarantine while `copy_is_executed` tests that the rows in quarantine have been copied to the `analytics_events` Data Source.

Lastly, you need to deploy the Branch:

# use set -e to raise errors for any of the commands below and make the CI pipeline to fail
set -e

tb deploy
tb pipe copy run analytics_events_quarantine_to_final --wait You can now merge the Pull Request, the Copy Pipe will be deployed to the Workspace and the copy operation will be executed ingesting all rows in quarantine.

After that you can optionally truncate the quarantine Data Source using `tb datasource truncate analytics_events_quarantine`.

This is a [working Pull Request](https://github.com/tinybirdco/use-case-examples/pull/6) with all the steps mentioned above.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Make sure you're familiar with the[  challenges of backfilling real-time data](https://www.tinybird.co/docs/docs/work-with-data/strategies/backfill-strategies#the-challenge-of-backfilling-real-time-data)
- Learn how to[  monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion)  .



---

URL: https://www.tinybird.co/docs/get-data-in/data-operations/iterate-a-data-source
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Iterate a Data Source · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to change the schema of a Data Source without using version control."
---




# Iterating a Data Source (change or update schema) [¶](https://www.tinybird.co/docs/about:blank#iterating-a-data-source-change-or-update-schema)

Creating a Data Source for the first time is really straightforward. However, when iterating data projects, sometimes you need to edit the Data Source schema. This can be challenging when the data is already in production, and there are a few different scenarios. With Tinybird you can easily add more columns, but other operations (such as changing the sorting key or changing a column type) require you to fully recreate the Data Source.

This guide is for Workspaces that **aren't** using version control. If your Workspace is linked using the Git<>Tinybird integration, see the repo of [common use cases for iterating when using version control](https://github.com/tinybirdco/use-case-examples).

## Overview [¶](https://www.tinybird.co/docs/about:blank#overview)

This guide walks through the iteration process for 4 different scenarios. Pick the one that's most relevant for you:

- **  Scenario 1: I'm not in production**
- **  Scenario 2: I can stop/pause data ingestion**
- **  Scenario 3: I need to change a Materialized View & I can't stop data ingest**
- **  Scenario 4: It's too complex and I can't figure it out**

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You'll need to be familiar with the Tinybird CLI to follow along with this guide. Never used it before? [Read the docs here](https://www.tinybird.co/docs/docs/cli/quick-start).

All of the guide examples have the same setup - a Data Source with a `nullable(Int64)` column that the user wants to change to a `Int64` for performance reasons. This requires editing the schema and, to keep the existing data, replacing any occurrences of `NULL` with a number, like `0`.

## Scenario 1: I'm not in production [¶](https://www.tinybird.co/docs/about:blank#scenario-1-im-not-in-production)

This scenario assumes that you aren't in production and can accept losing any data you have already ingested.

If you aren't in production, and you can accept losing data, use the Tinybird CLI to pull your Data Source down to a file, modify it, and push it back into Tinybird.

Begin with `tb pull` to pull your Tinybird resources down to files. Then, modify the .datasource file for the Data Source you want to change. When you're finished modifying the Data Source, delete the existing Data Source from Tinybird, either in the CLI with `tb datasource rm` or through the UI. Finally, push the new Data Source to Tinybird with `tb push`.

See a screencast example: [https://www.youtube.com/watch?v=gzpuQfk3Byg](https://www.youtube.com/watch?v=gzpuQfk3Byg).

## Scenario 2: I can stop data ingestion [¶](https://www.tinybird.co/docs/about:blank#scenario-2-i-can-stop-data-ingestion)

This scenario assumes that you have stopped all ingestion into the affected Data Sources.

### 1. Use the CLI to pull your Tinybird resources down into files [¶](https://www.tinybird.co/docs/about:blank#1-use-the-cli-to-pull-your-tinybird-resources-down-into-files)

Use `tb pull --auto` to pull your Tinybird resources down into files. The `--auto` flag will organize the resources into directories, with your Data Sources being places into a `datasources` directory.

### 2. Create the new Data Source [¶](https://www.tinybird.co/docs/about:blank#2-create-the-new-data-source)

Create a copy of the Data Source file that you want to modify and rename it.

For example, `datasources/original_ds.datasource` -> `datasources/new_ds.datasource`.

Modify the new Data Source schema in the file to make the changes you need.

Now push the new Data Source to Tinybird with `tb push datasources/new_ds.datasource`.

### 3. Backfill the new Data Source with existing data [¶](https://www.tinybird.co/docs/about:blank#3-backfill-the-new-data-source-with-existing-data)

If you want to move the existing data from the original Data Source to the new Data Source, use a Copy Pipe or a Pipe that materializes data into the new Data Source.

#### 3.1 Recommended option: Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#3-1-recommended-option-copy-pipe)

A [Copy Pipe](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes) is a Pipe used to copy data from one Data Source to another Data Source. This method is useful for one-time moves of data or scheduled executions.

Move your data using the following Copy Pipe, paying particular attention to the `TYPE`, `TARGET_DATASOURCE` and `COPY_SCHEDULE` configs at the end:

NODE copy_node
SQL >
    SELECT
        * EXCEPT (my_nullable_column),
        toInt64(coalesce(my_nullable_column,0)) as my_column -- adjust query to your changes
    FROM original_ds

TYPE COPY
TARGET_DATASOURCE new_ds
COPY_SCHEDULE @on-demand Push it to the Workspace:

tb push pipes/temp_copy.pipe And run the Copy:

tb pipe copy run temp_copy When it's done, remove the Pipe:

tb pipe rm temp_copy
#### 3.2 Alternative option: A Populate [¶](https://www.tinybird.co/docs/about:blank#3-2-alternative-option-a-populate)

Alternatively, you can create a Materialized View Pipe and run a Populate to transform data from the original schema into the modified schema of the new Data Source.

Do this using the following Pipe, paying particular attention to the `TYPE` and `DATASOURCE` configs at the end:

NODE temp_populate
SQL >
    SELECT
        * EXCEPT (my_nullable_column),
        toInt64(coalesce(my_nullable_column,0)) as my_column
    FROM original_ds

TYPE materialized
DATASOURCE new_ds Then push the Pipe to Tinybird, passing the `--populate` flag to force it to immediately start processing data:

tb push pipes/temp.pipe --populate --wait When it's done, remove the Pipe:

tb pipe rm temp At this point, review your new Data Source and ensure that everything is as expected.

### 4. Delete the original Data Source and rename the new Data Source [¶](https://www.tinybird.co/docs/about:blank#4-delete-the-original-data-source-and-rename-the-new-data-source)

You can now go to the UI, delete the original Data Source, and rename the new Data Source to use the name of the original Data Source.

By renaming the new Data Source to use the same name as the original Data Source, any SQL in your Pipes or Endpoints that referred to the original Data Source will continue to work.

If you have a Materialized View that depends on the Data Source, you must unlink the Pipe that is materializing data before removing the Data Source. You can modify and reconnect your Pipe after completing the steps above.

## Scenario 3: I need to change a Materialized View & I can't interrupt service [¶](https://www.tinybird.co/docs/about:blank#scenario-3-i-need-to-change-a-materialized-view-i-cant-interrupt-service)

This scenario assumes you want to modify a Materialized View that is actively receiving data and serving API Endpoints, *and* you want to avoid service downtime.

### Before you begin [¶](https://www.tinybird.co/docs/about:blank#before-you-begin)

Because this is a complex scenario, let's introduce some names for the example resources to make it a bit easier to follow along.

Let's assume that you have a Data Source that is actively receiving data; let's call this the `Landing Data Source` . From the `Landing Data Source` , you have a Pipe that is writing to a Materialized View; let's call these the `Materializing Pipe` and `Materialized View Data Source` respectively.

### 1. Use the CLI to pull your Tinybird resources down into files [¶](https://www.tinybird.co/docs/about:blank#1-use-the-cli-to-pull-your-tinybird-resources-down-into-files)

Use `tb pull --auto` to pull your Tinybird resources down into files. The `--auto` flag organizes the resources into directories, with your Data Sources being places into a `datasources` directory.

### 2. Duplicate the Materializing Pipe & Materialized View Data Source [¶](https://www.tinybird.co/docs/about:blank#2-duplicate-the-materializing-pipe-materialized-view-data-source)

Duplicate the `Materializing Pipe` & `Materialized View Data Source`.

For example:

pipes/original_materializing_pipe.pipe -> pipes/new_materializing_pipe.pipe
datasources/original_materialized_view_data_source.datasource -> datasources/new_materialized_view_data_source.datasource Modify the new files to change the schema as needed.

Lastly, you'll need to add a `WHERE` clause to the new `Materializing Pipe` . This clause is going to filter out old rows, so that the `Materializing Pipe` is only materializing rows newer than a specific time. For the purpose of this guide, let's call this the `Future Timestamp` . Do **not** use variable time functions for this timestamp (e.g. `now()` ). Pick a static time that is in the near future; five to fifteen minutes should be enough. The condition should be `>` , for example:

WHERE … AND my_timestamp > "2024-04-12 13:15:00"
### 3. Push the Materializing Pipe & Materialized View Data Source [¶](https://www.tinybird.co/docs/about:blank#3-push-the-materializing-pipe-materialized-view-data-source)

Push the `Materializing Pipe` & `Materialized View Data Source` to Tinybird:

tb push datasources/new_materialized_view_data_source.datasource
tb push pipes/new_materializing_pipe.pipe
### 4. Create a new Pipe to transform & materialize the old schema to the new schema [¶](https://www.tinybird.co/docs/about:blank#4-create-a-new-pipe-to-transform-materialize-the-old-schema-to-the-new-schema)

You now have two Materialized Views: the one with the original schema, and the new one with the new schema. You need to take the data from the original Materialized View, transform it into the new schema, and write it into the new Materialized View.

To do this, create a new Pipe. In this guide, it's called the `Transform Pipe` . In your `Transform Pipe` create the SQL `SELECT` logic that transforms the old schema to the new schema.

Lastly, your `Transform Pipe` should have a `WHERE` clause that only selects rows that are **older** than our `Future Timestamp` . The condition should be `<=` , for example:

WHERE … AND my_timestamp <= "2024-01-12 13:00:00"
### 5. Wait until after the Future Timestamp, then push & populate with the Transform Pipe [¶](https://www.tinybird.co/docs/about:blank#5-wait-until-after-the-future-timestamp-then-push-populate-with-the-transform-pipe)

Now, to avoid any potential for creating duplicates or missing rows, wait until after the `Future Timestamp` time has passed. This means that there should no longer be any rows arriving that have a timestamp that is **older** than the `Future Timestamp`.

Then, push the `Transform Pipe` and force a populate:

tb push pipes/new_materializing_pipe.pipe --populate --wait
### 6. Wait for the populate to finish, then change your API Endpoint to read from the new Materialized View Data Source [¶](https://www.tinybird.co/docs/about:blank#6-wait-for-the-populate-to-finish-then-change-your-api-endpoint-to-read-from-the-new-materialized-view-data-source)

Wait until the previous command has completed to ensure that all data from the original Materialized View has been written to the new `Materialized View Data Source`.

When it's complete, modify the API Endpoint that is querying the old Materialized View to query from the new `Materialized View Data Source`.

For example:

SELECT * from original_materialized_view_data_source Would become:

SELECT * from new_materialized_view_data_source
### 7. Test, then clean up old resources [¶](https://www.tinybird.co/docs/about:blank#7-test-then-clean-up-old-resources)

Test that your API Endpoint is serving the correct data. If everything looks good, you can tidy up your Workspace by deleting the original Materialized View & the new `Transform Pipe`.

## Scenario 4: It's too complex and I can't figure it out [¶](https://www.tinybird.co/docs/about:blank#scenario-4-its-too-complex-and-i-cant-figure-it-out)

If you are dealing with a very complex scenario, don't worry! Contact Tinybird support ( [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) ).

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Got your schema sorted and ready to make some queries? Understand[  how to work with time](https://www.tinybird.co/docs/docs/work-with-data/query/guides/working-with-time)  .
- Learn how to[  monitor your ingestion](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion)  .



---

URL: https://www.tinybird.co/docs/get-data-in/connectors/snowflake
Last update: 2025-01-21T08:14:33.000Z
Content:
---
title: "Snowflake Connector · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for how to use the Tinybird Snowflake Connector"
---



The Snowflake Connector is being deprecated. See [Ingest from Snowflake using AWS](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-snowflake-using-aws-s3) for an alternative.

# Snowflake Connector [¶](https://www.tinybird.co/docs/about:blank#snowflake-connector)

Use the Snowflake Connector to load data from your existing Snowflake account into Tinybird so that you can quickly turn them into high-concurrency, low-latency REST APIs.

The Snowflake Connector is fully managed and requires no additional tooling. You can define a sync schedule inside Tinybird and execution is taken care of for you.

With the Snowflake Connector you can:

- Start ingesting data instantly from Snowflake using SQL.
- Use SQL to query, shape, and join your Snowflake data with other sources.
- Use Auth tokens to control access to API endpoints. Implement access policies as you need. Support for row-level security.

Snowflake IP filtering isn't supported by the Snowflake Connector. If you need to filter IPs, use the GCS/S3 Connector.

## Load a Snowflake table [¶](https://www.tinybird.co/docs/about:blank#load-a-snowflake-table)

### Load a Snowflake table in the UI [¶](https://www.tinybird.co/docs/about:blank#load-a-snowflake-table-in-the-ui)

To add a Snowflake table as a Data Source, follow these steps.

#### Create a connection [¶](https://www.tinybird.co/docs/about:blank#create-a-connection)

Create a new Data Source using the Snowflake Connector dialog:

1. Open Tinybird and add a new Data Source by selecting**  Create new (+)**   next to the**  Data Sources**   section.
2. In the Data Sources dialog, select the Snowflake connector.
3. Enter your Snowflake Account Identifier. To find this, log into Snowflake, find the account info and then copy the Account Identifier.
4. In Tinybird, in the**  Connection details**   dialog, configure authentication with your Snowflake account. Enter your user password and Account Identifier.
5. Select the role and warehouse to access your data.
6. Copy the SQL snippet from the text box. The snippet creates a new Snowflake Storage Integration linking your Snowflake account with a Tinybird staging area for your Workspace. It also grants permission to the given role to create new Stages to unload data from your Snowflake Account into Tinybird.
7. With the SQL query copied, open a new SQL Worksheet inside Snowflake. Paste the SQL into the Worksheet query editor. You must edit the query and replace the ``   fragment with the name of your Snowflake database.
8. Select**  Run**   . The statement must be executed with a Snowflake `ACCOUNTADMIN`   role, since Snowflake Integrations operate at Account level and usually need administrator permissions.

#### Select the database, table, and schema [¶](https://www.tinybird.co/docs/about:blank#select-the-database-table-and-schema)

After running the query, the `Statement executed successfully` message appears. Return to your Tinybird tab to resume the configuration of the Snowflake connector. Set a name for the Snowflake connection and complete this step by selecting **Next**.

The Snowflake Connector now has enough permissions to inspect your Snowflake objects available to the given role. Browse the tables available in Snowflake and select the table you wish to load.

Start by selecting the database to which the table belongs, then the schema, and the table. Finish by selecting **Next**.

Maximum allowed table size is 50 million rows. The result is truncated if it exceeds that limit.

#### Configure the schedule [¶](https://www.tinybird.co/docs/about:blank#configure-the-schedule)

You can configure the schedule on which you wish to load data. By default, the frequency is set to **One-off** which performs a one-time sync of the table. You can change this by selecting a different option from the menu.

To configure a schedule that runs a regular sync, select the **Interval** option. You can configure a schedule in minutes, hours, or days by using the menu, and set the value for the schedule in the text field.

You can also select whether the sync should run immediately, or if it should wait until the first scheduled sync. The **Replace data** import strategy is selected by default. Finish by selecting **Next**.

Maximum allowed frequency is 5 minutes.

#### Complete the configuration [¶](https://www.tinybird.co/docs/about:blank#complete-the-configuration)

The final screen of the dialog shows the interpreted schema of the table, which you can change as needed. You can also modify what the name of the Data Source in Tinybird. Select **Create Data Source** to complete the process.

After you've created the Data Source, a status chart appears showing executions of the loading schedule. The Data Source takes a moment to create the resources required to perform the first sync.

When the first sync has completed, a green bar appears indicating the status. Details about the data, such as storage size and number of rows, is shown. You can also see a preview of the data.

### Load a Snowflake table in the CLI [¶](https://www.tinybird.co/docs/about:blank#load-a-snowflake-table-in-the-cli)

To add a Snowflake table as a Data Source using the Tinybird CLI, follow these steps.

#### Create a connection [¶](https://www.tinybird.co/docs/about:blank#create-a-connection)

You need to create a connection before you can load a table from Snowflake into Tinybird using the CLI. Creating a connection grants your Tinybird Workspace the appropriate permissions to view data from Snowflake.

[Authenticate your CLI](https://www.tinybird.co/docs/docs/cli/install#authentication) and switch to the desired Workspace. Then run:

tb connection create snowflake The output includes instructions to configure read-only access to your data in Snowflake. Enter your user, password, account identifier, role, warehouse, and a name for the connection.

After introducing the required information, copy the SQL block that appears.

** Creating a new Snowflake connection at the xxxx workspace.

User (must have create stage and create integration in Snowflake):
Password:
Account identifier:
Role (optional):
Warehouse (optional):
Connection name (optional, current xxxx):

Enter this SQL statement in Snowflake using your admin account to create the connection:

------
create storage integration if not exists "tinybird_integration_role"
    type = external_stage
    storage_provider = 'GCS'
    enabled = true
    comment = 'Tinybird Snowflake Connector Integration'
    storage_allowed_locations = ('gcs://tinybird-cdk-production-europe-west3/id');

grant create stage on all schemas in database  to role ACCOUNTADMIN;

grant ownership on integration "tinybird_integration_ACCOUNTADMIN" to role ACCOUNTADMIN;
------

Ready?  (y, N):

** Validating connection...
** xxxx.connection created successfully! Connection details saved into the .env file and referenced automatically in your connection file. With the SQL query copied, open a new SQL Worksheet inside Snowflake. Paste the SQL into the Worksheet query editor. You must edit the query and replace the `` fragment with the name of your Snowflake database.

Select **Run** . This statement must be executed with a Snowflake `ACCOUNTADMIN` role, since Snowflake Integrations operate at Account level and usually need administrator permissions.

The `Statement executed successfully` message appears. Return to your terminal, select **yes** (y) and the connection is created.A new `snowflake.connection` file appears in your project files. The `.connection` file can be safely deleted.

#### Create the Data Source [¶](https://www.tinybird.co/docs/about:blank#create-the-data-source)

After you've created the connection, you can create a Data Source and configure the schedule to import data from Snowflake.

The Snowflake import is configured using the following options, which you can add at the end of your .datasource file:

- `IMPORT_SERVICE`   : Name of the import service to use. In this case, `snowflake`  .
- `IMPORT_CONNECTION_NAME`   : The name given to the Snowflake connection inside Tinybird. For example, `'my_connection'`  .
- `IMPORT_EXTERNAL_DATASOURCE`   : The fully qualified name of the source table in Snowflake. For example, `database.schema.table`  .
- `IMPORT_SCHEDULE`   : A cron expression (UTC) with the frequency to run imports. Must be higher than 5 minutes. For example, `*/5 * * * *`  .
- `IMPORT_STRATEGY`   : The strategy to use when inserting data, either `REPLACE`   or `APPEND`  .
- `IMPORT_QUERY`   : (Optional) The SELECT query to extract your data from Snowflake when you don't need all the columns or want to make a transformation before ingestion. The FROM must reference a table using the full scope: `database.schema.table`  .

Note: For `IMPORT_STRATEGY` only `REPLACE` is supported today. The `APPEND` strategy will be enabled in a future release.

The following example shows a configured .datasource file for a Snowflake Data Source:

##### snowflake.datasource file

DESCRIPTION >
    Snowflake demo data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `id` Integer `json:$.id`,
    `orderid` LowCardinality(String) `json:$.orderid`,
    `status` LowCardinality(String) `json:$.status`,
    `amount` Integer `json:$.amount`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE snowflake
IMPORT_CONNECTION_NAME my_snowflake_connection
IMPORT_EXTERNAL_DATASOURCE mydb.raw.events
IMPORT_SCHEDULE */5 * * * *
IMPORT_STRATEGY REPLACE
IMPORT_QUERY >
    select
    timestamp,
    id,
    orderid,
    status,
    amount
        from
        mydb.raw.events The columns you select in the `IMPORT_QUERY` must match the columns defined in the Data Source schema. For example, if you Data Source has columns `ColumnA, ColumnB` then your `IMPORT_QUERY` must contain `SELECT ColumnA, ColumnB FROM ...` . A mismatch of columns causes data to arrive in the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine).

#### Push the configuration to Tinybird [¶](https://www.tinybird.co/docs/about:blank#push-the-configuration-to-tinybird)

With your connection created and Data Source defined, you can now push your project to Tinybird using:

tb push The first run of the import begins on the next lapse of the CRON expression.

## Iterate a Snowflake Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-a-snowflake-data-source)

### Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

Use of the CLI and the version control integration to handle your resources.

To use the advantages of version control, connect your Workspace with [your repository](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control#connect-your-workspace-to-git-from-the-cli) , and set the [CI/CD configuration](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration).

Check the [use case examples](https://github.com/tinybirdco/use-case-examples) repository where you can find basic instructions and examples to handle Snowflake Data Sources iteration using git integration, under the `iterate_snowflake` section.

To use the [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/quick-start) check its documentation.

For instance to create the connections in the main-branch Workspace using the CLI:

tb auth # use the main Workspace admin Token
tb connection create snowflake
# these prompts are interactive and will ask you to insert the necessary information You can only create connections in the main Workspace. When creating the connection in a Branch, it's created in the main Workspace and from there is available to every Branch.

For testing purposes, use different connections from main-branches workspaces.

### Add a new Snowflake Data Source [¶](https://www.tinybird.co/docs/about:blank#add-a-new-snowflake-data-source)

You can add a new Data Source directly with the UI or the CLI tool, following [the load of a Snowflake table section](https://www.tinybird.co/docs/about:blank#load-a-snowflake-table).

This works for testing purposes, but doesn't carry any connection details. You must add the connection and Snowflake configuration in the .datasource file when moving to production.

To add a new Data Source using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_snowflake).

### Update a Data Source [¶](https://www.tinybird.co/docs/about:blank#update-a-data-source)

- Snowflake Data Sources can't be modified directly from UI
- When you create a new Tinybird Branch, the existing Snowflake Data Sources won't be connected. You need to re-create them in the Branch.
- In Branches, it's usually useful to work with[  fixtures](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies#fixture-tests)   , as they'll be applied as part of the CI/CD, allowing the full process to be deterministic in every iteration and avoiding quota consume from external services.

Snowflake Data Sources can be modified from the CLI tool:

tb auth
# modify the .datasource Datafile with your editor
tb push --force {datafile}
# check the command output for errors To update it using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_snowflake).

### Delete a Data Source [¶](https://www.tinybird.co/docs/about:blank#delete-a-data-source)

Snowflake Data Sources can be deleted directly from UI or CLI like any other Data Source.

To delete it using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_snowflake).

## Logs [¶](https://www.tinybird.co/docs/about:blank#logs)

Job executions are logged in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . You can check this log directly in the Data Source view page in the UI. Filter by `datasource_id` to monitor ingestion through the Snowflake Connector from the `datasources_ops_log`:

SELECT
  timestamp,
  event_type,
  result,
  error,
  job_id
FROM
  tinybird.datasources_ops_log
WHERE
  datasource_id = 't_1234'
AND
  event_type = 'replace'
ORDER BY timestamp DESC
## Schema evolution [¶](https://www.tinybird.co/docs/about:blank#schema-evolution)

The Snowflake Connector supports backwards compatible changes made in the source table. This means that, if you add a new column in Snowflake, the next sync job automatically adds it to the Tinybird Data Source.

Non-backwards compatible changes, such as dropping or renaming columns, aren't supported and might cause the next sync to fail.

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

See [Snowflake Connector limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#snowflake-connector-limits).



---

URL: https://www.tinybird.co/docs/get-data-in/connectors/s3
Last update: 2025-01-29T13:58:37.000Z
Content:
---
title: "S3 Connector · Tinybird Docs"
theme-color: "#171612"
description: "Bring your S3 data to Tinybird using the S3 Connector."
---




# S3 Connector [¶](https://www.tinybird.co/docs/about:blank#s3-connector)

Use the S3 Connector to ingest files from your Amazon S3 buckets into Tinybird so that you can turn them into high-concurrency, low-latency REST APIs. You can load a full bucket or load files that match a pattern. In both cases you can also set an update date from which the files are loaded.

With the S3 Connector you can load your CSV, NDJSON, or Parquet files into your S3 buckets and turn them into APIs. Tinybird detects new files in your buckets and ingests them automatically. You can then run serverless transformations using Data Pipes or implement auth tokens in your API Endpoints.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

The S3 Connector requires permissions to access objects in your Amazon S3 bucket. The IAM Role needs the following permissions:

- `s3:GetObject`
- `s3:ListBucket`
- `s3:ListAllMyBuckets`

The following is an example of AWS Access Policy:

When configuring the connector, the UI, CLI and API all provide the necessary policy templates.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::",
                "arn:aws:s3:::/*"
            ],
            "Effect": "Allow"
        },
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
} The following is an example trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Principal": {
                "AWS": "arn:aws:iam::473819111111111:root"
            },
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "ab3caaaa-01aa-4b95-bad3-fff9b2ac789f8a9"
                }
            }
        }
    ]
}
## Supported file types [¶](https://www.tinybird.co/docs/about:blank#supported-file-types)

The S3 Connector supports the following file types:

| File type | Accepted extensions | Compression formats supported |
| --- | --- | --- |
| CSV | `.csv`  , `.csv.gz` | `gzip` |
| NDJSON | `.ndjson`  , `.ndjson.gz` | `gzip` |
|  | `.jsonl`  , `.jsonl.gz` |  |
|  | `.json`  , `.json.gz` |  |
| Parquet | `.parquet`  , `.parquet.gz` | `snappy`  , `gzip`  , `lzo`  , `brotli`  , `lz4`  , `zstd` |

You can upload files with .json extension, provided they follow the Newline Delimited JSON (NDJSON) format. Each line must be a valid JSON object and every line has to end with a `\n` character.

Parquet schemas use the same format as NDJSON schemas, using [JSONPath](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths) syntax.

## S3 file URI [¶](https://www.tinybird.co/docs/about:blank#s3-file-uri)

Use the full S3 File URI and wildcards to select multiple files.

The S3 Connector supports the following wildcard patterns:

- Single Asterisk ( `*`   ): matches zero or more characters within a single directory level, excluding `/`   . It doesn't cross directory boundaries. For example, `s3://bucket-name/*.ndjson`   matches all `.ndjson`   files in the root of your bucket but doesn't match files in subdirectories.
- Double Asterisk ( `**`   ): matches zero or more characters across multiple directory levels, including `/`   . It can cross directory boundaries recursively. For example: `s3://bucket-name/**/*.ndjson`   matches all `.ndjson`   files in the bucket, regardless of their directory depth.

The file extension is required to accurately match the desired files in your pattern.

### Examples [¶](https://www.tinybird.co/docs/about:blank#examples)

The following are examples of patterns you can use and whether they'd match the example file path:

| File path | S3 File URI | Will match? |
| --- | --- | --- |
| example.ndjson | `s3://bucket-name/*.ndjson` | Yes. Matches files in the root directory with the `.ndjson`   extension. |
| example.ndjson.gz | `s3://bucket-name/**/*.ndjson.gz` | Yes. Recursively matches `.ndjson.gz`   files anywhere in the bucket. |
| example.ndjson | `s3://bucket-name/example.ndjson` | Yes. Exact match to the file path. |
| pending/example.ndjson | `s3://bucket-name/*.ndjson` | No. `*`   doesn't cross directory boundaries. |
| pending/example.ndjson | `s3://bucket-name/**/*.ndjson` | Yes. Recursively matches `.ndjson`   files in any subdirectory. |
| pending/example.ndjson | `s3://bucket-name/pending/example.ndjson` | Yes. Exact match to the file path. |
| pending/example.ndjson | `s3://bucket-name/pending/*.ndjson` | Yes. Matches `.ndjson`   files within the `pending`   directory. |
| pending/example.ndjson | `s3://bucket-name/pending/**/*.ndjson` | Yes. Recursively matches `.ndjson`   files within `pending`   and all its subdirectories. |
| pending/example.ndjson | `s3://bucket-name/**/pending/example.ndjson` | Yes. Matches the exact path to `pending/example.ndjson`   within any preceding directories. |
| pending/example.ndjson | `s3://bucket-name/other/example.ndjson` | No. Does not match because the path includes directories which aren't part of the file's actual path. |
| pending/example.ndjson.gz | `s3://bucket-name/pending/*.csv.gz` | No. The file extension `.ndjson.gz`   doesn't match `.csv.gz` |
| pending/o/inner/example.ndjson | `s3://bucket-name/*.ndjson` | No. `*`   doesn't cross directory boundaries. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/*.ndjson` | Yes. Recursively matches `.ndjson`   files anywhere in the bucket. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/inner/example.ndjson` | Yes. Matches the exact path to `inner/example.ndjson`   within any preceding directories. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/ex*.ndjson` | Yes. Recursively matches `.ndjson`   files starting with `ex`   at any depth. |
| pending/o/inner/example.ndjson | `s3://bucket-name/**/**/*.ndjson` | Yes. Matches `.ndjson`   files at any depth, even with multiple `**`   wildcards. |
| pending/o/inner/example.ndjson | `s3://bucket-name/pending/**/*.ndjson` | Yes. Matches `.ndjson`   files within `pending`   and all its subdirectories. |
| pending/o/inner/example.ndjson | `s3://bucket-name/inner/example.ndjson` | No. Does not match because the path includes directories which aren't part of the file's actual path. |
| pending/o/inner/example.ndjson | `s3://bucket-name/pending/example.ndjson` | No. Does not match because the path includes directories which aren't part of the file's actual path. |
| pending/o/inner/example.ndjson.gz | `s3://bucket-name/pending/*.ndjson.gz` | No. `*`   doesn't cross directory boundaries. |
| pending/o/inner/example.ndjson.gz | `s3://bucket-name/other/example.ndjson.gz` | No. Does not match because the path includes directories which aren't part of the file's actual path. |

### Considerations [¶](https://www.tinybird.co/docs/about:blank#considerations)

When using patterns:

- Use specific directory names or even specific file URIs to limit the scope of your search. The more specific your pattern, the narrower the search.
- Combine wildcards: you can combine `**`   with other patterns to match files in subdirectories selectively. For example, `s3://bucket-name/**/logs/*.ndjson`   matches `.ndjson`   files within any logs directory at any depth.
- Avoid unintended matches: be cautious with `**`   as it can match a large number of files, which might impact performance and return partial matches.

To test your patterns and see a sample of your matching files before proceeding, use the **Preview** step in the Connector.

### Sample file URL [¶](https://www.tinybird.co/docs/about:blank#sample-file-url)

When files that match the pattern you've provided exceed the file size limits of your plan, or when the preview step reaches request limits, Tinybird prompts you to provide a sample file URL.

The sample file is used to infer the schema of the data, ensuring compatibility with the ingestion process. After the schema is inferred, all files matching the initial pattern are ingested.

A sample file URL must point to a single file and must follow the full S3 URI format, including the bucket name and directory path. For example, if the initial bucket URI is `s3://example-bucket-name/data/**/*.ndjson` then the Sample file URL would be `s3://example-bucket-name/data/2024-12-01/sample-file.ndjson`.

The following considerations apply:

- Make sure the sample file is representative of the overall dataset to avoid mismatched schemas during ingestion or quarantined data.
- When using compression format, for example .gz, make sure that the sample file is compressed in the same way as the other files in the dataset.
- After the preview, all files matching the pattern are ingested, not just the ones processed for the preview.

## Set up the connection [¶](https://www.tinybird.co/docs/about:blank#set-up-the-connection)

You can set up an S3 connection using the UI or the CLI. The steps are as follows:

1. Create a new Data Source in Tinybird.
2. Create the AWS S3 connection.
3. Configure the scheduling options and path/file names.
4. Start ingesting the data.

## Load files using the CLI [¶](https://www.tinybird.co/docs/about:blank#load-files-using-the-cli)

Before you can load files from Amazon S3 into Tinybird using the CLI, you must create a connection. Creating a connection grants your Tinybird Workspace the appropriate permissions to view files in Amazon S3.

To create a connection, you need to use the Tinybird CLI version 3.8.3 or higher. [Authenticate your CLI](https://www.tinybird.co/docs/docs/cli/install#authentication) and switch to the desired Workspace.

Follow these steps to create a connection:

1. Run `tb connection create s3_iamrole --policy read`   command and press `y`   to confirm.
2. Copy the suggested policy and replace the bucket placeholder ``   with your bucket name.
3. In AWS, create a new policy in**  IAM**  ,**  Policies (JSON)**   using the edited policy.
4. Return to the Tinybird CLI, press `y`   , and copy the next policy.
5. In AWS, go to**  IAM**  ,**  Roles**   and copy the new custom trust policy. Attach the policy you edited in the previous step.
6. Return to the CLI, press `y`   , and paste the ARN of the role you've created in the previous step.
7. Enter the region of the bucket. For example, `us-east-1`  .
8. Provide a name for your connection in Tinybird.

The `--policy` flag allows to switch between write (sink) and read (ingest) policies.

Now that you've created a connection, you can add a Data Source to configure the import of files from Amazon S3.

Configure the Amazon S3 import using the following options in your .datasource file:

- `IMPORT_SERVICE`   : name of the import service to use, in this case, `s3_iamrole`  .
- `IMPORT_SCHEDULE`   : either `@auto`   to sync once per minute, or `@on-demand`   to only execute manually (UTC).
- `IMPORT_STRATEGY`   : the strategy used to import data. Only `APPEND`   is supported.
- `IMPORT_BUCKET_URI`   : a full bucket path, including the `s3://`   protocol , bucket name, object path and an optional pattern to match against object keys. You can use patterns in the path to filter objects. For example, ending the path with `*.csv`   matches all objects that end with the `.csv`   suffix.
- `IMPORT_CONNECTION_NAME`   : name of the S3 connection to use.
- `IMPORT_FROM_TIMESTAMP`   : (optional) set the date and time from which to start ingesting files. Format is `YYYY-MM-DDTHH:MM:SSZ`  .

When Tinybird discovers new files, it appends the data to the existing data in the Data Source. Replacing data isn't supported.

The following is an example of a .datasource file for S3:

##### s3.datasource file

DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE s3_iamrole
IMPORT_CONNECTION_NAME connection_name
IMPORT_BUCKET_URI s3://bucket-name/*.csv
IMPORT_SCHEDULE @auto
IMPORT_STRATEGY APPEND With your connection created and Data Source defined, you can now push your project to Tinybird using:

tb push
## Load files using the UI [¶](https://www.tinybird.co/docs/about:blank#load-files-using-the-ui)

### 1. Create a new Data Source [¶](https://www.tinybird.co/docs/about:blank#1-create-a-new-data-source)

In Tinybird, go to **Data Sources** and select **Create Data Source**.

Select **Amazon S3** and enter the bucket name and region, then select **Continue**.

### 2. Create the AWS S3 connection [¶](https://www.tinybird.co/docs/about:blank#2-create-the-aws-s3-connection)

Follow these steps to create the connection:

1. Open the AWS console and navigate to IAM.
2. Create and name the policy using the provided copyable option.
3. Create and name the role with the trust policy using the provided copyable option.
4. Select**  Connect**  .
5. Paste the connection name and ARN.

### 3. Select the data [¶](https://www.tinybird.co/docs/about:blank#3-select-the-data)

Select the data you want to ingest by providing the [S3 File URI](https://www.tinybird.co/docs/about:blank#s3-file-uri) and selecting **Preview**.

You can also set the ingestion to start from a specific date and time, so that the ingestion process ignores all files added or updated before the set date and time:

1. Select**  Ingest since ISO date and time**  .
2. Write the desired date or datetime in the input, following the format `YYYY-MM-DDTHH:MM:SSZ`  .

### 4. Preview and create [¶](https://www.tinybird.co/docs/about:blank#4-preview-and-create)

The next screen shows a preview of the incoming data. You can review and modify any of the incoming columns, adjust their names, change their types, or delete them. You can also configure the name of the Data Source.

After reviewing your incoming data, select **Create Data Source** . On the Data Source details page, you can see the sync history in the tracker chart and the current status of the connection.

## Schema evolution [¶](https://www.tinybird.co/docs/about:blank#schema-evolution)

The S3 Connector supports adding new columns to the schema of the Data Source using the CLI.

Non-backwards compatible changes, such as dropping, renaming, or changing the type of columns, aren't supported. Any rows from these files are sent to the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources#the-quarantine-data-source).

## Iterate an S3 Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-an-s3-data-source)

To iterate an S3 Data Source, use the Tinybird CLI and the [version control integration](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) to handle your resources.

Create a connection using the CLI:

tb auth # use the main Workspace admin Token
tb connection create s3_iamrole To iterate an S3 Data Source through a Branch, create the Data Source using a connector that already exists. The S3 Connector doesn't ingest any data, as it isn't configured to work in Branches. To test it on CI, you can directly append the files to the Data Source.

After you've merged it and are running CD checks, run `tb datasource sync ` to force the sync in the main Workspace.

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

The following limits apply to the S3 Connector:

- When using the `auto`   mode, execution of imports runs once every minute.
- Tinybird ingests a maximum of 5 files per minute. This is a Workspace-level limit, so it's shared across all Data Sources.

The following limits apply to maximum file size per type:

| File type | Max file size |
| --- | --- |
| CSV | 10 GB for the Free plan, 32 GB for Developer and Enterprise |
| NDJSON | 10 GB for the Free plan, 32 GB for Developer and Enterprise |
| Parquet | 1 GB for the Free plan, 5 GB for Developer and Enterprise |



Check the [limits page](https://www.tinybird.co/docs/docs/get-started/plans/limits) for limits on ingestion, queries, API Endpoints, and more.

To adjust these limits, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

## Monitoring [¶](https://www.tinybird.co/docs/about:blank#monitoring)

You can follow the standard recommended practices for monitoring Data Sources as explained in our [ingestion monitoring guide](https://www.tinybird.co/docs/docs/monitoring/monitor-data-ingestion) . There are specific metrics for the S3 Connector.

If a sync finishes unsuccessfully, Tinybird adds a new event to `datasources_ops_log`:

- If all the files in the sync failed, the event has the `result`   field set to `error`  .
- If some files failed and some succeeded, the event has the `result`   field set to `partial-ok`  .

Failures in syncs are atomic, meaning that if one file fails, no data from that file is ingested.

A JSON object with the list of files that failed is included in the `error` field. Some errors can happen before the file list can be retrieved (for instance, an AWS connection failure), in which case there are no files in the `error` field. Instead, the `error` field contains the error message and the files to be retried in the next execution.

In scheduled runs, Tinybird retries all failed files in the next executions, so that rate limits or temporary issues don't cause data loss. In on-demand runs, since there is no next execution, truncate the Data Source and sync again.

You can distinguish between individual failed files and failed syncs by looking at the `error` field:

- If the `error`   field contains a JSON object, the sync failed and the object contains the error message with the list of files that failed.
- If the `error`   field contains a string, a file failed to ingest and the string contains the error message. You can see the file that failed by looking at the `Options.Values`   field.

For example, you can use the following query to see the sync error messages for the last day:

SELECT JSONExtractString(error, 'message') message, *
FROM tinybird.datasources_ops_log
WHERE
    datasource_id = ''
    AND timestamp > now() - INTERVAL 1 day
    AND message IS NOT NULL
ORDER BY timestamp DESC

---

URL: https://www.tinybird.co/docs/get-data-in/connectors/redpanda
Last update: 2025-01-12T22:34:39.000Z
Content:
---
title: "Redpanda Connector · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird Redpanda Connector"
---




# Redpanda Connector [¶](https://www.tinybird.co/docs/about:blank#redpanda-connector)

The Redpanda Connector allows you to ingest data from your existing Redpanda cluster and load it into Tinybird so that you can quickly turn them into high-concurrency, low-latency REST APIs.

The Redpanda Connector is fully managed and requires no additional tooling. Connect Tinybird to your Redpanda cluster, choose a topic, and Tinybird will automatically begin consuming messages from Redpanda.

The Redpanda Connector is:

- **  Easy to use**   . Connect to your Redpanda cluster in seconds. Choose your topics, define your schema, and ingest millions of events per second into a fully-managed OLAP.
- **  SQL-based**   . Using nothing but SQL, query your Redpanda data and enrich it with dimensions from your database, warehouse, or files.
- **  Secure**   . Use Auth tokens to control access to API endpoints. Implement access policies as you need. Support for row-level security.

Note that you need to grant READ permissions to both the Topic and the Consumer Group to ingest data from Redpanda into Tinybird.

## Using the UI [¶](https://www.tinybird.co/docs/about:blank#using-the-ui)

To connect Tinybird to your Redpanda cluster, click the `+` icon next to the data project section on the left navigation menu, select **Data Source** , and select **Redpanda** from the list of available Data Sources.

Enter the following details:

- **  Connection name**   : A name for the Redpanda connection in Tinybird.
- **  Bootstrap Server**   : The comma-separated list of bootstrap servers (including Port numbers).
- **  Key**   : The**  Key**   component of the Redpanda API Key.
- **  Secret**   : The**  Secret**   component of the Redpanda API Key.
- **  Decode Avro messages with schema registry**   : Optionally, you can enable Schema Registry support to decode Avro messages. You will be prompted to enter the Schema Registry URL, username and password.

Once you have entered the details, select **Connect** . This creates the connection between Tinybird and Redpanda. You will then see a list of your existing topics and can select the topic to consume from. Tinybird will create a **Group ID** that specifies the name of the consumer group this consumer belongs to. You can customize the Group ID, but ensure that your Group ID has **read** permissions to the topic.

Once you have chosen a topic, you can select the starting offset to consume from. You can choose to consume from the **latest** offset or the **earliest** offset. If you choose to consume from the earliest offset, Tinybird will consume all messages from the beginning of the topic. If you choose to consume from the latest offset, Tinybird will only consume messages that are produced after the connection is created. Select the offset, and click **Next**.

Tinybird will then consume a sample of messages from the topic and display the schema. You can adjust the schema and Data Source settings as needed, then click **Create Data Source** to create the Data Source.

Tinybird will now begin consuming messages from the topic and loading them into the Data Source.

## Using .datasource files [¶](https://www.tinybird.co/docs/about:blank#using-datasource-files)

If you are managing your Tinybird resources in files, there are several settings available to configure the Redpanda Connector in .datasource files.

See the [datafiles docs](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda) for more information.

The following is an example of Kafka .datasource file for an already existing connection:

##### Example data source for Redpanda Connector

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String
  `__headers` Map(String,String)

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

# Connection is already available. If you
# need to create one, add the required fields
# on an include file with the details.
KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
KAFKA_STORE_HEADERS true
### Columns of the Data Source [¶](https://www.tinybird.co/docs/about:blank#columns-of-the-data-source)

When you connect a Kafka producer to Tinybird, Tinybird consumes optional metadata columns from that Kafka record and writes them to the Data Source.

The following fields represent the raw data received from Kafka:

- `__value`   : A String representing the entire unparsed Kafka record inserted.
- `__topic`   : The Kafka topic that the message belongs to.
- `__partition`   : The kafka partition that the message belongs to.
- `__offset`   : The Kafka offset of the message.
- `__timestamp`   : The timestamp stored in the Kafka message received by Tinybird.
- `__key`   : The key of the kafka message.
- `__headers`   : Headers parsed from the incoming topic messages. See[  Using custom Kafka headers for advanced message processing](https://www.tinybird.co/blog-posts/using-custom-kafka-headers)  .

Metadata fields are optional. Omit the fields you don't need to reduce your data storage.

### Using INCLUDE to store connection settings [¶](https://www.tinybird.co/docs/about:blank#using-include-to-store-connection-settings)

To avoid configuring the same connection settings across many files, or to prevent leaking sensitive information, you can store connection details in an external file and use `INCLUDE` to import them into one or more .datasource files.

You can find more information about `INCLUDE` in the [Advanced Templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) documentation.

As an example, you may have two Redpanda .datasource files, which re-use the same Redpanda connection. You can create an INCLUDE file that stores the Redpanda connection details.

The Tinybird project may use the following structure:

##### Tinybird data project file structure

ecommerce_data_project/
├── datasources/
│   └── connections/
│       └── my_connector_name.incl
│   └── my_kafka_datasource.datasource
│   └── another_datasource.datasource
├── endpoints/
├── pipes/ Where the file `my_connector_name.incl` has the following content:

##### Include file containing Redpanda connection details

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password And the Redpanda .datasource files look like the following:

##### Data Source using includes for Redpanda connection details

SCHEMA >
    `value` String,
    `topic` LowCardinality(String),
    `partition` Int16,
    `offset` Int64,
    `timestamp` DateTime,
    `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/my_connection_name.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id When using `tb pull` to pull a Redpanda Data Source using the CLI, the `KAFKA_KEY` and `KAFKA_SECRET` settings will **not** be included in the file to avoid exposing credentials.

## Redpanda logs [¶](https://www.tinybird.co/docs/about:blank#redpanda-logs)



You can find global logs in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-datasources-ops-log) . Filter by `datasource_id` to select the correct datasource, and set `event_type` to `append-kafka`.

To select all Kafka releated logs in the last day, run the following query:

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC If you can't find logs in `datasources_ops_log` , the `kafka_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-kafka-ops-log) contains more detailed logs. Filter by `datasource_id` to select the correct datasource, and use `msg_type` to select the desired log level ( `info`, `warning` , or `error` ).

SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']

---

URL: https://www.tinybird.co/docs/get-data-in/connectors/msk
Last update: 2024-12-18T09:46:02.000Z
Content:
---
title: "Amazon MSK · Tinybird Docs"
theme-color: "#171612"
description: "Get Amazon MSK data into Tinybird."
---




# Amazon MSK [¶](https://www.tinybird.co/docs/about:blank#amazon-msk)

Use the Kafka Connector to ingest data streams from Amazon Managed Streaming for Apache Kafka (MSK) into Tinybird so that you can quickly turn them into high-concurrency, low-latency REST APIs.

The Kafka Connector is fully managed and requires no additional tooling. Connect Tinybird to your Kafka cluster, select a topic, and Tinybird automatically begins consuming messages from Kafka. You can transform or enrich your Kafka topics with JOINs using serverless Data Pipes. Auth tokens control access to API endpoints.

Secure connections through AWS PrivateLink or Multi-VPC for MSK are available for Enterprise customers on a Dedicated infrastructure plan. Reach out to support@tinybird.co for more information.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

Grant `READ` permissions to both the Topic and the Consumer Group to ingest data from Kafka into Tinybird.

You must secure your Kafka brokers with SSL/TLS and SASL. Tinybird uses `SASL_SSL` as the security protocol for the Kafka consumer. Connections are rejected if the brokers only support `PLAINTEXT` or `SASL_PLAINTEXT`.

Kafka Schema Registry is supported only for decoding Avro messages.

## Add a Kafka connection [¶](https://www.tinybird.co/docs/about:blank#add-a-kafka-connection)

You can create a connection to Kafka using the Tinybird CLI or the UI.

### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-cli)

Run the following commands to add a Kafka connection:

##### Adding a Kafka connection in the main Workspace

tb auth # Use the main Workspace admin Token
tb connection create kafka --bootstrap-servers  --key  --secret  --connection-name  --ssl-ca-pem 
### Using the UI [¶](https://www.tinybird.co/docs/about:blank#using-the-ui)

Follow these steps to add a new connection using the UI:

1. Go to**  Data Project**  .
2. Select the**  +**   icon, then select**  Data Source**  .
3. Select**  Kafka**  .
4. Follow the steps to configure the connection.

### Add a CA certificate [¶](https://www.tinybird.co/docs/about:blank#add-a-ca-certificate)

You can add a CA certificate in PEM format when configuring your Kafka connection from the UI. Tinybird checks the certificate for issues before creating the connection.

To add a CA certificate using the Tinybird CLI, pass the `--ssl-ca-pem ` argument to `tb connection create` , where `` is the location or value of the CA certificate.

CA certificates don't work with Kafka Sinks and Streaming Queries.

## Update a Kafka connection [¶](https://www.tinybird.co/docs/about:blank#update-a-kafka-connection)

You can update your credentials or cluster details only from the Tinybird UI. Follow these steps:

1. Go to**  Data Project**   , select the**  +**   icon, then select**  Data Source**  .
2. Select**  Kafka**   and then the connection you want to edit or delete using the three-dots menu.

Any Data Source that depends on this connection is affected by updates.

## Use .datasource files [¶](https://www.tinybird.co/docs/about:blank#use-datasource-files)

You can configure the Kafka Connector using .datasource files. See the [datafiles documentation](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda).

The following is an example of Kafka .datasource file for an already existing connection:

##### Example data source for Kafka Connector

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String
  `__headers` Map(String,String)

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

# Connection is already available. If you
# need to create one, add the required fields
# on an include file with the details.
KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
KAFKA_STORE_HEADERS true To add connection details in an INCLUDE file, see [Use INCLUDE to store connection settings](https://www.tinybird.co/docs/about:blank#use-include-to-store-connection-settings).

### Columns of the Data Source [¶](https://www.tinybird.co/docs/about:blank#columns-of-the-data-source)

When you connect a Kafka producer to Tinybird, Tinybird consumes optional metadata columns from that Kafka record and writes them to the Data Source.

The following fields represent the raw data received from Kafka:

- `__value`   : A String representing the entire unparsed Kafka record inserted.
- `__topic`   : The Kafka topic that the message belongs to.
- `__partition`   : The kafka partition that the message belongs to.
- `__offset`   : The Kafka offset of the message.
- `__timestamp`   : The timestamp stored in the Kafka message received by Tinybird.
- `__key`   : The key of the kafka message.
- `__headers`   : Headers parsed from the incoming topic messages. See[  Using custom Kafka headers for advanced message processing](https://www.tinybird.co/blog-posts/using-custom-kafka-headers)  .

Metadata fields are optional. Omit the fields you don't need to reduce your data storage.

### Use INCLUDE to store connection settings [¶](https://www.tinybird.co/docs/about:blank#use-include-to-store-connection-settings)

To avoid configuring the same connection settings across many files, or to prevent leaking sensitive information, you can store connection details in an external file and use `INCLUDE` to import them into one or more .datasource files.

You can find more information about `INCLUDE` in the [Advanced Templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) documentation.

For example, you might have two Kafka .datasource files that reuse the same Kafka connection. You can create an include file which stores the Kafka connection details.

The Tinybird project would use the following structure:

##### Tinybird data project file structure

ecommerce_data_project/
    datasources/
        connections/
          my_connector_name.incl
          ca.pem # CA certificate (optional)
        my_kafka_datasource.datasource
        another_datasource.datasource
    endpoints/
    pipes/ Where the file `my_connector_name.incl` has the following content:

##### Include file containing Kafka connection details

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password
KAFKA_SSL_CA_PEM ca.pem # CA certificate (optional) And the Kafka .datasource files look like the following:

##### Data Source using includes for Kafka connection details

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/my_connection_name.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id When using `tb pull` to pull a Kafka Data Source using the CLI, the `KAFKA_KEY`, `KAFKA_SECRET`, `KAFKA_SASL_MECHANISM` and `KAFKA_SSL_CA_PEM` settings aren't included in the file to avoid exposing credentials.

## Iterate a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-a-kafka-data-source)

The following instructions use Branches. Be sure you're familiar with the behavior of Branches in Tinybird when using the Kafka Connector - see [Prerequisites](https://www.tinybird.co/docs/about:blank#prerequisites).

Use Branches to test different Kafka connections and settings. See [Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches).

Connections created using the UI are created in the main Workspace. so if you create a new Branch from a Workspace with existing Kafka Data Sources, the Branch Data Sources don't receive that streaming data automatically. Use the CLI to recreate the Kafka Data Source.

### Update a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#update-a-kafka-data-source)

When you create a Branch that has existing Kafka Data Sources, the Data Sources in the Branch aren't connected to Kafka.

Therefore, if you want to update the schema, you need to recreate the Kafka Data Source in the Branch.

In branches, Tinybird automatically appends `_{BRANCH}` to the Kafka group ID to prevent collisions. It also forces the consumers in Branches to always consume the `latest` messages, to reduce the performance impact.

### Add a new Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#add-a-new-kafka-data-source)

To create and test a Kafka Data Source in a Branch, start by using an existing connection. You can create and use existing connections from the Branch using the UI: these connections are always created in the main Workspace.

You can create a Kafka Data Source in a Branch as in production. This Data Source doesn't have any connection details internally, so you it's useful for testing purposes.

Define the connection in the .datafile and Kafka parameters that are used in production. To move the Data Source to production, include the connection settings in the Data Source .datafile, as explained in the [.datafiles documentation](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda).

### Delete a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#delete-a-kafka-data-source)

If you've created a Data Source in a Branch, the Data Source is active until the Data Source is removed from the Branch or when the entire Branch is removed.

If you delete an existing Kafka Data Source in a Branch, it isn't deleted in the main Workspace. To delete a Kafka Data Source, do it against the main Workspace. You can also use the CLI and include it in the CI/CD workflows as necessary.

## MSK logs [¶](https://www.tinybird.co/docs/about:blank#msk-logs)



You can find global logs in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-datasources-ops-log) . Filter by `datasource_id` to select the correct datasource, and set `event_type` to `append-kafka`.

To select all Kafka releated logs in the last day, run the following query:

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC If you can't find logs in `datasources_ops_log` , the `kafka_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-kafka-ops-log) contains more detailed logs. Filter by `datasource_id` to select the correct datasource, and use `msg_type` to select the desired log level ( `info`, `warning` , or `error` ).

SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']
## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

The limits for the Kafka connector are:

- Minimum flush time: 4 seconds
- Throughput (uncompressed): 20MB/s
- Up to 3 connections per Workspace

If you're regularly hitting these limits, contact [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) for support.

## Troubleshooting [¶](https://www.tinybird.co/docs/about:blank#troubleshooting)

### If you aren't receiving data [¶](https://www.tinybird.co/docs/about:blank#if-you-arent-receiving-data)

When Kafka commits a message for a topic and a group id, it always sends data from the latest committed offset. In Tinybird, each Kafka Data Source receives data from a topic and uses a group id. The combination of `topic` and `group id` must be unique.

If you remove a Kafka Data Source and you recreate it again with the same settings after having received data, you' only get data from the latest committed offset, even if `KAFKA_AUTO_OFFSET_RESET` is set to `earliest`.

This happens both in the main Workspace and in Branches, if you're using them, because connections are always created in the main Workspace and are shared across Branches.

Recommended next steps:

- Use always a different group id when testing Kafka Data Sources.
- Check in the `tinybird.kafka_ops_log`[  Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources)   to see if you've already used a group id to ingest data from a topic.

### Compressed messages [¶](https://www.tinybird.co/docs/about:blank#compressed-messages)

Tinybird can consume from Kafka topics where Kafka compression is enabled, as decompressing the message is a standard function of the Kafka Consumer. If you compressed the message before passing it through the Kafka Producer, Tinybird can't do post-Consumer processing to decompress the message.

For example, if you compressed a JSON message through gzip and produced it to a Kafka topic as a `bytes` message, it would be ingested by Tinybird as `bytes` . If you produced a JSON message to a Kafka topic with the Kafka Producer setting `compression.type=gzip` , while it would be stored in Kafka as compressed bytes, it would be decoded on ingestion and arrive to Tinybird as JSON.



---

URL: https://www.tinybird.co/docs/get-data-in/connectors/kafka
Last update: 2024-12-18T09:46:02.000Z
Content:
---
title: "Kafka Connector · Tinybird Docs"
theme-color: "#171612"
description: "Documentation for the Tinybird Kafka Connector"
---




# Kafka Connector [¶](https://www.tinybird.co/docs/about:blank#kafka-connector)

Use the Kafka Connector to ingest data streams from your Kafka cluster into Tinybird so that you can quickly turn them into high-concurrency, low-latency REST APIs.

The Kafka Connector is fully managed and requires no additional tooling. Connect Tinybird to your Kafka cluster, select a topic, and Tinybird automatically begins consuming messages from Kafka. You can transform or enrich your Kafka topics with JOINs using serverless Data Pipes. Auth tokens control access to API endpoints.

Secure connections through AWS PrivateLink or Multi-VPC for MSK are available for Enterprise customers on an Enterprise plan. Reach out to support@tinybird.co for more information.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

Grant `READ` permissions to both the Topic and the Consumer Group to ingest data from Kafka into Tinybird.

You must secure your Kafka brokers with SSL/TLS and SASL. Tinybird uses `SASL_SSL` as the security protocol for the Kafka consumer. Connections are rejected if the brokers only support `PLAINTEXT` or `SASL_PLAINTEXT`.

Kafka Schema Registry is supported only for decoding Avro messages.

## Add a Kafka connection [¶](https://www.tinybird.co/docs/about:blank#add-a-kafka-connection)

You can create a connection to Kafka using the Tinybird CLI or the UI.

### Using the CLI [¶](https://www.tinybird.co/docs/about:blank#using-the-cli)

Run the following commands to add a Kafka connection:

tb auth # Use the main Workspace admin Token
tb connection create kafka --bootstrap-servers  --key  --secret  --connection-name  --ssl-ca-pem 
### Using the UI [¶](https://www.tinybird.co/docs/about:blank#using-the-ui)

Follow these steps to add a new connection using the UI:

1. Go to**  Data Project**  .
2. Select the**  +**   icon, then select**  Data Source**  .
3. Select**  Kafka**  .
4. Follow the steps to configure the connection.

### Add a CA certificate [¶](https://www.tinybird.co/docs/about:blank#add-a-ca-certificate)

You can add a CA certificate in PEM format when configuring your Kafka connection from the UI. Tinybird checks the certificate for issues before creating the connection.

To add a CA certificate using the Tinybird CLI, pass the `--ssl-ca-pem ` argument to `tb connection create` , where `` is the location or value of the CA certificate.

CA certificates don't work with Kafka Sinks and Streaming Queries.

#### Aiven Kafka [¶](https://www.tinybird.co/docs/about:blank#aiven-kafka)

Aiven for Apache Kafka service instances expose multiple SASL ports with two different kinds of SASL certificates: Private CA (self-signed) and Public CA, signed by Let's Encrypt.

If you are using the Public CA port, you can connect to Aiven Kafka without any additional configuration. However, if you are using the Private CA port, you need to provide the CA certificate by pointing to the path of the CA certificate file using the `KAFKA_SSL_CA_PEM` setting.

## Update a Kafka connection [¶](https://www.tinybird.co/docs/about:blank#update-a-kafka-connection)

You can update your credentials or cluster details only from the Tinybird UI. Follow these steps:

1. Go to**  Data Project**   , select the**  +**   icon, then select**  Data Source**  .
2. Select**  Kafka**   and then the connection you want to edit or delete using the three-dots menu.

Any Data Source that depends on this connection is affected by updates.

## Use .datasource files [¶](https://www.tinybird.co/docs/about:blank#use-datasource-files)

You can configure the Kafka Connector using .datasource files. See the [datafiles documentation](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda).

The following is an example of Kafka .datasource file for an already existing connection:

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String
  `__headers` Map(String,String)

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

# Connection is already available. If you
# need to create one, add the required fields
# on an include file with the details.
KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
KAFKA_STORE_HEADERS true To add connection details in an INCLUDE file, see [Use INCLUDE to store connection settings](https://www.tinybird.co/docs/about:blank#use-include-to-store-connection-settings).

### Columns of the Data Source [¶](https://www.tinybird.co/docs/about:blank#columns-of-the-data-source)

When you connect a Kafka producer to Tinybird, Tinybird consumes optional metadata columns from that Kafka record and writes them to the Data Source.

The following fields represent the raw data received from Kafka:

- `__value`   : A String representing the entire unparsed Kafka record inserted.
- `__topic`   : The Kafka topic that the message belongs to.
- `__partition`   : The kafka partition that the message belongs to.
- `__offset`   : The Kafka offset of the message.
- `__timestamp`   : The timestamp stored in the Kafka message received by Tinybird.
- `__key`   : The key of the kafka message.
- `__headers`   : Headers parsed from the incoming topic messages. See[  Using custom Kafka headers for advanced message processing](https://www.tinybird.co/blog-posts/using-custom-kafka-headers)  .

Metadata fields are optional. Omit the fields you don't need to reduce your data storage.

### Use INCLUDE to store connection settings [¶](https://www.tinybird.co/docs/about:blank#use-include-to-store-connection-settings)

To avoid configuring the same connection settings across many files, or to prevent leaking sensitive information, you can store connection details in an external file and use `INCLUDE` to import them into one or more .datasource files.

You can find more information about `INCLUDE` in the [Advanced Templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) documentation.

For example, you might have two Kafka .datasource files that reuse the same Kafka connection. You can create an include file which stores the Kafka connection details.

The Tinybird project would use the following structure:

ecommerce_data_project/
    datasources/
        connections/
          my_connector_name.incl
          ca.pem # CA certificate (optional)
        my_kafka_datasource.datasource
        another_datasource.datasource
    endpoints/
    pipes/ Where the file `my_connector_name.incl` has the following content:

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password
KAFKA_SSL_CA_PEM ca.pem # CA certificate (optional) And the Kafka .datasource files look like the following:

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/my_connection_name.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id When using `tb pull` to pull a Kafka Data Source using the CLI, the `KAFKA_KEY`, `KAFKA_SECRET`, `KAFKA_SASL_MECHANISM` and `KAFKA_SSL_CA_PEM` settings aren't included in the file to avoid exposing credentials.

## Iterate a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-a-kafka-data-source)

The following instructions use Branches. Be sure you're familiar with the behavior of Branches in Tinybird when using the Kafka Connector - see [Prerequisites](https://www.tinybird.co/docs/about:blank#prerequisites).

Use Branches to test different Kafka connections and settings. See [Branches](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/branches).

Connections created using the UI are created in the main Workspace. so if you create a new Branch from a Workspace with existing Kafka Data Sources, the Branch Data Sources don't receive that streaming data automatically. Use the CLI to recreate the Kafka Data Source.

### Update a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#update-a-kafka-data-source)

When you create a Branch that has existing Kafka Data Sources, the Data Sources in the Branch aren't connected to Kafka.

Therefore, if you want to update the schema, you need to recreate the Kafka Data Source in the Branch.

In branches, Tinybird automatically appends `_{BRANCH}` to the Kafka group ID to prevent collisions. It also forces the consumers in Branches to always consume the `latest` messages, to reduce the performance impact.

### Add a new Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#add-a-new-kafka-data-source)

To create and test a Kafka Data Source in a Branch, start by using an existing connection. You can create and use existing connections from the Branch using the UI: these connections are always created in the main Workspace.

You can create a Kafka Data Source in a Branch as in production. This Data Source doesn't have any connection details internally, so you it's useful for testing purposes.

Define the connection in the .datafile and Kafka parameters that are used in production. To move the Data Source to production, include the connection settings in the Data Source .datafile, as explained in the [.datafiles documentation](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda).

### Delete a Kafka Data Source [¶](https://www.tinybird.co/docs/about:blank#delete-a-kafka-data-source)

If you've created a Data Source in a Branch, the Data Source is active until the Data Source is removed from the Branch or when the entire Branch is removed.

If you delete an existing Kafka Data Source in a Branch, it isn't deleted in the main Workspace. To delete a Kafka Data Source, do it against the main Workspace. You can also use the CLI and include it in the CI/CD workflows as necessary.

## Kafka logs [¶](https://www.tinybird.co/docs/about:blank#kafka-logs)



You can find global logs in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-datasources-ops-log) . Filter by `datasource_id` to select the correct datasource, and set `event_type` to `append-kafka`.

To select all Kafka releated logs in the last day, run the following query:

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC If you can't find logs in `datasources_ops_log` , the `kafka_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-kafka-ops-log) contains more detailed logs. Filter by `datasource_id` to select the correct datasource, and use `msg_type` to select the desired log level ( `info`, `warning` , or `error` ).

SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']
## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

The limits for the Kafka connector are:

- Minimum flush time: 4 seconds
- Throughput (uncompressed): 20MB/s
- Up to 3 connections per Workspace

If you're regularly hitting these limits, contact [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) for support.

## Troubleshooting [¶](https://www.tinybird.co/docs/about:blank#troubleshooting)

### If you aren't receiving data [¶](https://www.tinybird.co/docs/about:blank#if-you-arent-receiving-data)

When Kafka commits a message for a topic and a group id, it always sends data from the latest committed offset. In Tinybird, each Kafka Data Source receives data from a topic and uses a group id. The combination of `topic` and `group id` must be unique.

If you remove a Kafka Data Source and you recreate it again with the same settings after having received data, you' only get data from the latest committed offset, even if `KAFKA_AUTO_OFFSET_RESET` is set to `earliest`.

This happens both in the main Workspace and in Branches, if you're using them, because connections are always created in the main Workspace and are shared across Branches.

Recommended next steps:

- Use always a different group id when testing Kafka Data Sources.
- Check in the `tinybird.datasources_ops_log`[  Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources)   to see global errors.
- Check in the `tinybird.kafka_ops_log`[  Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources)   to see if you've already used a group id to ingest data from a topic.

### Compressed messages [¶](https://www.tinybird.co/docs/about:blank#compressed-messages)

Tinybird can consume from Kafka topics where Kafka compression is enabled, as decompressing the message is a standard function of the Kafka Consumer. If you compressed the message before passing it through the Kafka Producer, Tinybird can't do post-Consumer processing to decompress the message.

For example, if you compressed a JSON message through gzip and produced it to a Kafka topic as a `bytes` message, it would be ingested by Tinybird as `bytes` . If you produced a JSON message to a Kafka topic with the Kafka Producer setting `compression.type=gzip` , while it would be stored in Kafka as compressed bytes, it would be decoded on ingestion and arrive to Tinybird as JSON.



---

URL: https://www.tinybird.co/docs/get-data-in/connectors/dynamodb
Last update: 2024-12-18T09:46:02.000Z
Content:
---
title: "DynamoDB Connector · Tinybird Docs"
theme-color: "#171612"
description: "Bring your DynamoDB data to Tinybird using the DynamoDB Connector."
---




# DynamoDB Connector [¶](https://www.tinybird.co/docs/about:blank#dynamodb-connector)

Use the DynamoDB Connector to ingest historical and change stream data from Amazon DynamoDB to Tinybird.

The DynamoDB Connector is fully managed and requires no additional tooling. Connect Tinybird to DynamoDB, select your tables, and Tinybird keeps in sync with DynamoDB.

With the DynamoDB Connector you can:

- Connect to your DynamoDB tables and start ingesting data in minutes.
- Query your DynamoDB data using SQL and enrich it with dimensions from your streaming data, warehouse, or files.
- Use Auth tokens to control access to API endpoints. Implement access policies as you need. Support for row-level security.

DynamoDB Connector only works with Workspaces created in AWS Regions.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

- Tinybird CLI version 5.3.0 or higher. See[  the Tinybird CLI quick start](https://www.tinybird.co/docs/docs/cli/quick-start)  .
- Tinybird CLI[  authenticated with the desired Workspace](https://www.tinybird.co/docs/docs/cli/install)  .
- DynamoDB Streams is active on the target DynamoDB tables with `NEW_IMAGE`   or `NEW_AND_OLD_IMAGE`   type.
- Point-in-time recovery (PITR) is active on the target DynamoDB table.

You can switch the Tinybird CLI to the correct Workspace using `tb workspace use `.

Supported characters for column names are letters, numbers, underscores, and dashes. Tinybird automatically sanitizes invalid characters like dots or dollar signs.

## Required permissions [¶](https://www.tinybird.co/docs/about:blank#required-permissions)

The DynamoDB Connector requires certain permissions to access your tables. The IAM Role needs the following permissions:

- `dynamodb:Scan`
- `dynamodb:DescribeStream`
- `dynamodb:DescribeExport`
- `dynamodb:GetRecords`
- `dynamodb:GetShardIterator`
- `dynamodb:DescribeTable`
- `dynamodb:DescribeContinuousBackups`
- `dynamodb:ExportTableToPointInTime`
- `dynamodb:UpdateTable`
- `dynamodb:UpdateContinuousBackups`

The following is an example of AWS Access Policy:

When configuring the connector, the UI, CLI and API all provide the necessary policy templates.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:Scan",
        "dynamodb:DescribeStream",
        "dynamodb:DescribeExport",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeContinuousBackups",
        "dynamodb:ExportTableToPointInTime",
        "dynamodb:UpdateTable",
        "dynamodb:UpdateContinuousBackups"
      ],
      "Resource": [
        "arn:aws:dynamodb:*:*:table/",
        "arn:aws:dynamodb:*:*:table//stream/*",
        "arn:aws:dynamodb:*:*:table//export/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::",
        "arn:aws:s3:::/*"
      ]
    }
  ]
} The following is an example trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Principal": {
        "AWS": "arn:aws:iam::473819111111111:root"
      },
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "ab3caaaa-01aa-4b95-bad3-fff9b2ac789f8a9"
        }
      }
    }
  ]
}
## Load a table using the CLI [¶](https://www.tinybird.co/docs/about:blank#load-a-table-using-the-cli)

To load a DynamoDB table into Tinybird using the CLI, create a connection and then a Data Source.

The connection grants your Tinybird Workspace the necessary permissions to access AWS and your tables in DynamoDB. The Data Source then maps a table in DynamoDB to a table in Tinybird and manages the historical and continous sync.

### Create the DynamoDB connection [¶](https://www.tinybird.co/docs/about:blank#create-the-dynamodb-connection)

The connection grants your Tinybird Workspace the necessary permissions to access AWS and your tables in DynamoDB.

To connect, run the following command:

tb connection create dynamodb This command initiates the process of creating a connection. When prompted, type `y` to proceed.

### Create a new IAM Policy in AWS [¶](https://www.tinybird.co/docs/about:blank#create-a-new-iam-policy-in-aws)

The Tinybird CLI provides a policy template.

1. Replace ``   with the name of your DynamoDB table.ç
2. Replace ``   with the name of the S3 bucket you want to use for the initial load.
3. In AWS, go to**  IAM**  ,**  Policies**  ,**  Create Policy**  .
4. Select the**  JSON**   tab and paste the modified policy text.
5. Save and create the policy.

### Create a new IAM Role in AWS [¶](https://www.tinybird.co/docs/about:blank#create-a-new-iam-role-in-aws)

1. Return to the Tinybird CLI to get a trust policy template.
2. In AWS, go to**  IAM**  ,**  Roles**  ,**  Create Role**  .
3. Select**  Custom Trust Policy**   and paste the trust policy copied from the CLI.
4. In the**  Permissions**   tab, attach the policy created in the previous step.
5. Complete the role creation process.

### Complete the connection [¶](https://www.tinybird.co/docs/about:blank#complete-the-connection)

In the AWS IAM console, find the role you've created. Copy its Amazon Resource Name (ARN), which looks like `arn:aws:iam::111111111111:role/my-awesome-role`.

Provide the following information to Tinybird CLI:

- The Role ARN
- AWS region of your DynamoDB tables
- Connection name

Tinybird uses the connection name to identify the connection. The name can only contain AlphaNumeric characters `a-zA-Z` and underscores `_` , and must start with a letter.

When the CLI prompts are completed, Tinybird creates the connection.



The CLI will generate a `.connection` file in your project directory. This file isn't used and is safe to delete. A future release will allow you to push this file to Tinybird to automate the creation of connections, similar to Kafka connections.

### Create a DynamoDB Data Source file [¶](https://www.tinybird.co/docs/about:blank#create-a-dynamodb-data-source-file)

The Data Source maps a table in DynamoDB to a table in Tinybird and manages the historical and continous sync. [Data Source files](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files) contain the table schema, and specific DynamoDB properties to target the table that Tinybird imports.

Create a Data Source file called `mytable.datasource` . There are two approaches to defining the schema for a DynamoDB Data Source:

1. Define the Partition Key and Sort Key from your DynamoDB table, and access other properties from JSON at query time.
2. Define all DynamoDB item properties as columns.

The Partition Key and Sort Key, if any, from your DynamoDB must be defined in the Data Source schema. These are the only properties that are mandatory to define, as they're used for deduplication of records (upserts and deletes).

#### Approach 1: Define only the Partition Key and Sort Key [¶](https://www.tinybird.co/docs/about:blank#approach-1-define-only-the-partition-key-and-sort-key)

If you don't want to map all properties from your DynamoDB table, you can define only the Partition Key and Sort Keys.

The entire DynamoDB item is as JSON in a `_record` column, and you can extract properties using `JSONExtract*` functions.

For example, if you have a DynamoDB table with `transaction_id` as the Partition Key, you can define your Data Source schema like this:

##### mytable.datasource

SCHEMA >
    transaction_id String `json:$.transaction_id`

IMPORT_SERVICE "dynamodb"
IMPORT_CONNECTION_NAME 
IMPORT_TABLE_ARN 
IMPORT_EXPORT_BUCKET  Replace the `` with the name of the connection created in the first step. Replace `` with the ARN of the table you'd like to import. Replace `` with the name of the S3 bucket you want to use for the initial sync.

#### Approach 2: Define all DynamoDB item properties as columns [¶](https://www.tinybird.co/docs/about:blank#approach-2-define-all-dynamodb-item-properties-as-columns)

If you want to strictly define all your properties and their types, you can map them into your Data Source as columns.

You can map properties to [any of the supported types in Tinybird](https://www.tinybird.co/docs/docs/get-data-in/data-sources#supported-data-types) . Properties can be also arrays of the previously mentioned types, and nullable. Use the nullable type when there are properties that might not have a value in every item within your DynamoDB table.

For example, if you have a DynamoDB with items like this:

{
  "timestamp": "2024-07-25T10:46:37.380Z",
  "transaction_id": "399361d5-10fc-4777-8187-88aaa4623569",
  "name": "Chris Donnelly",
  "passport_number": 4904040,
  "flight_from": "Burien",
  "flight_to": "Sanford",
  "airline": "BrianAir"
} Where `transaction_id` is the partition key, you can define your Data Source schema like this:

##### mytable.datasource

SCHEMA >
    `timestamp` DateTime64(3) `json:$.timestamp`,
    `transaction_id` String `json:$.transaction_id`,
    `name` String `json:$.name`,
    `passport_number` Int64 `json:$.passport_number`,
    `flight_from` String `json:$.flight_from`,
    `flight_to` String `json:$.flight_to`,
    `airline` String `json:$.airline`

IMPORT_SERVICE "dynamodb"
IMPORT_CONNECTION_NAME 
IMPORT_TABLE_ARN 
IMPORT_EXPORT_BUCKET  Replace `` with the name of the connection created in the first step. Replace `` with the ARN of the table you'd like to import. Replace `` with the name of the S3 bucket you want to use for the initial sync.

You can map properties with basic types (String, Number, Boolean, Binary, String Set, Number Set) at the root item level. Follow this schema definition pattern:

  `json:$.`
- `PropertyName`   is the name of the column within your Tinybird Data Source.
- `PropertyType`   is the type of the column within your Tinybird Data Source. It must match the type in the DynamoDB Data Source:
  - Strings correspond to `String`     columns.
  - All `Int`    , `UInt`     , or `Float`     variants correspond to `Number`     columns.
  - `Array(String)`     corresponds to `String Set`     columns.
  - `Array(UInt)`     and all numeric variants correspond to `Number Set`     columns.
- `PropertyNameInDDB`   is the name of the property in your DynamoDB table. It must match the letter casing.

Map properties within complex types, like `Maps` , using JSONPaths. For example, you can map a property at the first level in your Data Source schema like:

MyString String `json:$..`. For `Lists` , standalone column mapping isn't supported. Those properties require extraction using `JSONExtract*` functions or consumed after a transformation with a Materialized View.

### Push the Data Source [¶](https://www.tinybird.co/docs/about:blank#push-the-data-source)

With your connection created and Data Source defined, push your Data Source to Tinybird using `tb push`.

For example, if your Data Source file is `mytable.datasource` , run:

tb push mytable.datasource Due to how Point-in-time recovery works, data might take some minutes before it appears in Tinybird. This delay only happens the first time Tinybird retrieves the table.

## Load a table using the UI [¶](https://www.tinybird.co/docs/about:blank#load-a-table-using-the-ui)

To load a DynamoDB table into Tinybird using the UI, select the DynamoDB option in the Data Source dialog. You need an existing connection to your DynamoDB table.

The UI guides you through the process of creating a connection and finally creating a Data Source that imports the data from your DynamoDB table.

### Create a DynamoDB connection [¶](https://www.tinybird.co/docs/about:blank#create-a-dynamodb-connection)

When you create a connection, provide the following information:

- AWS region of your DynamoDB tables
- ARN of the table you want to import
- Name of the S3 bucket you want to use for the initial sync.

In the next step, provide the ARN of the IAM Role you created in AWS. This role must have the necessary permissions to access your DynamoDB tables and S3 bucket.

### Create a Data Source [¶](https://www.tinybird.co/docs/about:blank#create-a-data-source)

After you've created the connection, a preview of the imported data appears. You can change the schema columns, the sorting key, or the TTL.

Due to the schemaless nature of DynamoDB, the preview might not show all the columns in your table. You can manually add columns to the schema in the **Code Editor** tab.

When you're ready, select **Create Data Source**.

Due to how Point-in-time recovery works, data might take some minutes before it appears in Tinybird. This delay only happens the first time Tinybird retrieves the table.

## Columns added by Tinybird [¶](https://www.tinybird.co/docs/about:blank#columns-added-by-tinybird)

When loading a DynamoDB table, Tinybird automatically adds the following columns:

| Column | Type | Description |
| --- | --- | --- |
| `_record` | `Nullable(String)` | Contents of the event, in JSON format. Added to `NEW_IMAGES`   and `NEW_AND_OLD_IMAGES`   streams. |
| `_old_record` | `Nullable(String)` | Stores the previous state of the record. Added to `NEW_AND_OLD_IMAGES`   streams. |
| `_timestamp` | `DateTime64(3)` | Date and time of the event. |
| `_event_type` | `LowCardinality(String)` | Type of the event. |
| `_is_deleted` | `UInt8` | Whether the record has been deleted. |

If an existing table with stream type `NEW_AND_OLD_IMAGES` is missing the `_old_record` column, add it manually with the following configuration: `_old_record` Nullable(String) `json:$.OldImage`.

## Iterate a Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-a-data-source)

To iterate a DynamoDB Data Source, use the Tinybird CLI and the [version control integration](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control) to handle your resources.

You can only create connections in the main Workspace. When creating the connection in a Branch, it's created in the main Workspace and from there is available to every Branch.

DynamoDB Data Sources created in a Branch aren't connected to your source. AWS DynamoDB documentation discourages [reading the same Stream from various processes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html#Streams.Processing) , because it can result in throttling. This can affect the ingestion in the main Branch.

Browse the [use case examples](https://github.com/tinybirdco/use-case-examples) repository to find basic instructions and examples to handle DynamoDB Data Sources iteration using git integration.

### Add a new DynamoDB Data Source [¶](https://www.tinybird.co/docs/about:blank#add-a-new-dynamodb-data-source)

You can add a new Data Source directly with the Tinybird CLI. See [load of a DynamoDB table](https://www.tinybird.co/docs/about:blank#load-a-table-using-the-cli).

To add a new Data Source using the recommended version control workflow, see the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_dynamodb).

When you add a Data Source to a Tinybird Branch, it doesn't have any connection details. You must add the Connection and DynamoDB configuration in the .datasource Datafile when moving to a production environment or Branch.

### Update a Data Source [¶](https://www.tinybird.co/docs/about:blank#update-a-data-source)

You can modify DynamoDB Data Sources using Tinybird CLI. For example:

tb auth
# modify the .datasource Datafile with your editor
tb push --force {datafile}
# check the command output for errors When updating an existent DynamoDB Data Source, the first sync isn't repeated, only the new item modifications are synchronized by the CDC process.

To update a Data Source using the recommended version control workflow, see the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_dynamodb).

In Branches, work with [fixtures](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies#fixture-tests) , as they're be applied as part of the CI/CD, allowing the full process to be deterministic in every iteration and avoiding quota usage from external services.

### Delete a Data Source [¶](https://www.tinybird.co/docs/about:blank#delete-a-data-source)

You can delete DynamoDB Data Sources like any other Data Source.

To delete it using the recommended version control workflow, see the [examples repository](https://github.com/tinybirdco/use-case-examples/tree/main/iterate_dynamodb).

## DynamoDB logs [¶](https://www.tinybird.co/docs/about:blank#dynamodb-logs)

You can find DynamoDB logs in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-datasources-ops-log) . Filter by `datasource_id` to select the correct Data Source. Use `event_type` to select between initial synchronization logs ( `sync-dynamodb` ), or update logs ( `append-dynamodb` ).

To select all DynamoDB related logs in the last day, run the following query:

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type in ['sync-dynamodb', 'append-dynamodb']
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC
## Connector architecture [¶](https://www.tinybird.co/docs/about:blank#connector-architecture)

AWS provides two free, default functions for DynamoDB:

- DynamoDB Streams captures change events for a given DynamoDB table and provides an API to access events as a stream. This allows CDC-like access to the table for continuous updates.
- You can use Point-in-time recovery (PITR) to take snapshots of your DynamoDB table and save the export to S3. This allows historical access to table data for batch uploads.

The DynamoDB Connector uses the following functions to send DynamoDB data to Tinybird:



<-figure->
![Connecting DynamoDB to Tinybird architecture](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fassets%2Fguides%2Fingest-from-dynamodb%2Ftinybird-dynamodb-connector-arch.png&w=3840&q=75)

<-figcaption->
Connecting DynamoDB to Tinybird architecture





## Schema evolution [¶](https://www.tinybird.co/docs/about:blank#schema-evolution)

The DynamoDB Connector supports backwards compatible changes made in the source table. This means that, if you add a new column in DynamoDB, the next sync job automatically adds it to the Tinybird Data Source.

Non-backwards compatible changes, such as dropping or renaming columns, aren't supported by default and might cause the next sync to fail.

## Considerations on queries [¶](https://www.tinybird.co/docs/about:blank#considerations-on-queries)

The DynamoDB Connector uses the ReplacingMergeTree engine to remove duplicate entries with the same sorting key. Deduplication occurs during a merge, which happens at an unknown time in the background. Doing `SELECT * FROM ddb_ds` might yield duplicated rows after an insertion.

To account for this, force the merge at query time by adding `FINAL` to the query. For example, `SELECT * FROM ddb_ds FINAL` . Adding `FINAL` also filters out the rows where `_is_deleted = 1`.

## Override sort and partition keys [¶](https://www.tinybird.co/docs/about:blank#override-sort-and-partition-keys)

The DynamoDB Connector automatically sets values for the Sorting Key and the Partition Key properties based on the source DynamoDB table. You might want to override the default values to fit your needs.

To override Sorting and Partition key values, open your .datasource file and edit the values for `ENGINE_PARTITION_KEY` and `ENGINE_SORTING_KEY` . For the Sorting key, you must append the additional columns and leave `pk` and `sk` in place.

For example:

ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(toDateTime64(_timestamp, 3))"
ENGINE_SORTING_KEY "pk, sk, "
ENGINE_VER "_timestamp" Sorting key is used for deduplication of data. When adding columns to `ENGINE_SORTING_KEY` , make sure they contain the same value across record changes.

You can then push the new .datasource configuration using `tb push`:

tb push updated-ddb.datasource Don't edit the values for `ENGINE` or `ENGINE_VER` . The DynamoDB Connector requires the ReplacingMergeTree engine and a version based on the timestamp.

## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

See [DynamoDB Connector limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#dynamodb-connector-limits).



---

URL: https://www.tinybird.co/docs/get-data-in/connectors/confluent
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Confluent Connector · Tinybird Docs"
theme-color: "#171612"
description: "Use the Confluent Connector to bring data from Confluent to Tinybird."
---




# Confluent Connector [¶](https://www.tinybird.co/docs/about:blank#confluent-connector)

Use the Confluent Connector to bring data from your existing Confluent Cloud cluster into Tinybird so that you can quickly turn them into high-concurrency, low-latency REST API Endpoints and query using SQL.

The Confluent Connector is fully managed and requires no additional tooling. Connect Tinybird to your Confluent Cloud cluster, select a topic, and Tinybird automatically begins consuming messages from Confluent Cloud.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You need to grant `READ` permissions to both the Topic and the Consumer Group to ingest data from Confluent into Tinybird.

The Confluent Cloud Schema Registry is only supported for decoding Avro messages. When using Confluent Schema Registry, the Schema name must match the Topic name. For example, if you're ingesting the Kafka Topic `my-kafka-topic` using a Connector with Schema Registry enabled, it expects to find a Schema named `my-kafka-topic-value`.

## Create the Data Source using the UI [¶](https://www.tinybird.co/docs/about:blank#create-the-data-source-using-the-ui)

To connect Tinybird to your Confluent Cloud cluster, select **Create new (+)** next to the data project section, select **Data Source** , and then select **Confluent** from the list of available Data Sources.

Enter the following details:

- **  Connection name**   : A name for the Confluent Cloud connection in Tinybird.
- **  Bootstrap Server**   : The comma-separated list of bootstrap servers, including port numbers.
- **  Key**   : The key component of the Confluent Cloud API Key.
- **  Secret**   : The secret component of the Confluent Cloud API Key.
- **  Decode Avro messages with schema registry**   : (Optional) Turn on Schema Registry support to decode Avro messages. Enter the Schema Registry URL, username, and password.

After you've entered the details, select **Connect** . This creates the connection between Tinybird and Confluent Cloud. A list of your existing topics appears and you can select the topic to consume from.

Tinybird creates a **Group ID** that specifies the name of the consumer group that this Kafka consumer belongs to. You can customize the Group ID, but ensure that your Group ID has **Read** permissions to the topic.

After you've chosen a topic, you can select the starting offset to consume from. You can consume from the earliest offset or the latest offset:

- If you consume from the earliest offset, Tinybird consumes all messages from the beginning of the topic.
- If you consume from the latest offset, Tinybird only consumes messages that are produced after the connection is created.

After selecting the offset, select **Next** . Tinybird consumes a sample of messages from the topic and displays the schema. You can adjust the schema and Data Source settings as needed, then select **Create Data Source**.

Tinybird begins consuming messages from the topic and loading them into the Data Source.

## Configure the connector using .datasource files [¶](https://www.tinybird.co/docs/about:blank#configure-the-connector-using-datasource-files)

If you are managing your Tinybird resources in files, there are several settings available to configure the Confluent Connector in .datasource files. See the [datafiles docs](https://www.tinybird.co/docs/docs/cli/datafiles/datasource-files#kafka-confluent-redpanda) for more information.

The following is an example of Kafka .datasource file for an already existing connection:

##### Example data source for Confluent Connector

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String
  `__headers` Map(String,String)

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

# Connection is already available. If you
# need to create one, add the required fields
# on an include file with the details.
KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
KAFKA_STORE_HEADERS true
### Columns of the Data Source [¶](https://www.tinybird.co/docs/about:blank#columns-of-the-data-source)

When you connect a Kafka producer to Tinybird, Tinybird consumes optional metadata columns from that Kafka record and writes them to the Data Source.

The following fields represent the raw data received from Kafka:

- `__value`   : A String representing the entire unparsed Kafka record inserted.
- `__topic`   : The Kafka topic that the message belongs to.
- `__partition`   : The kafka partition that the message belongs to.
- `__offset`   : The Kafka offset of the message.
- `__timestamp`   : The timestamp stored in the Kafka message received by Tinybird.
- `__key`   : The key of the kafka message.
- `__headers`   : Headers parsed from the incoming topic messages. See[  Using custom Kafka headers for advanced message processing](https://www.tinybird.co/blog-posts/using-custom-kafka-headers)  .

Metadata fields are optional. Omit the fields you don't need to reduce your data storage.

### Use INCLUDE to store connection settings [¶](https://www.tinybird.co/docs/about:blank#use-include-to-store-connection-settings)

To avoid configuring the same connection settings across many files, or to prevent leaking sensitive information, you can store connection details in an external file and use `INCLUDE` to import them into one or more .datasource files.

You can find more information about `INCLUDE` in the [Advanced Templates](https://www.tinybird.co/docs/docs/cli/advanced-templates) documentation.

For example, you might have two Confluent Cloud .datasource files, which re-use the same Confluent Cloud connection. You can create an include file which stores the Confluent Cloud connection details.

The Tinybird project might use the following structure:

##### Tinybird data project file structure

ecommerce_data_project/
    datasources/
        connections/
          my_connector_name.incl
        my_confluent_datasource.datasource
        another_datasource.datasource
    endpoints/
    pipes/ Where the file `my_connector_name.incl` has the following content:

##### Include file containing Confluent Cloud connection details

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password And the Confluent Cloud .datasource files look like the following:

##### Data Source using includes for Confluent Cloud connection details

SCHEMA >
  `value` String,
  `topic` LowCardinality(String),
  `partition` Int16,
  `offset` Int64,
  `timestamp` DateTime,
  `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/my_connection_name.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id When using `tb pull` to pull a Confluent Cloud Data Source using the CLI, the `KAFKA_KEY` and `KAFKA_SECRET` settings aren't included in the file to avoid exposing credentials.

### Internal fields [¶](https://www.tinybird.co/docs/about:blank#internal-fields)

The `__` fields stored in the Kafka datasource represent the raw data received from Kafka:

- `__value`   : A String representing the whole Kafka record inserted.
- `__topic`   : The Kafka topic that the message belongs to.
- `__partition`   : The kafka partition that the message belongs to.
- `__offset`   : The Kafka offset of the message.
- `__timestamp`   : The timestamp stored in the Kafka message received by Tinybird.
- `__key`   : The key of the kafka message.

## Compressed messages [¶](https://www.tinybird.co/docs/about:blank#compressed-messages)

Tinybird can consume from Kafka topics where Kafka compression is enabled, as decompressing the message is a standard function of the Kafka Consumer.

However, if you compressed the message before passing it through the Kafka Producer, then Tinybird can't do post-Consumer processing to decompress the message.

For example, if you compressed a JSON message through gzip and produced it to a Kafka topic as a `bytes` message, it's ingested by Tinybird as `bytes` . If you produced a JSON message to a Kafka topic with the Kafka Producer setting `compression.type=gzip` , while it's stored in Kafka as compressed bytes, it's decoded on ingestion and arrive to Tinybird as JSON.

## Confluent logs [¶](https://www.tinybird.co/docs/about:blank#confluent-logs)



You can find global logs in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-datasources-ops-log) . Filter by `datasource_id` to select the correct datasource, and set `event_type` to `append-kafka`.

To select all Kafka releated logs in the last day, run the following query:

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC If you can't find logs in `datasources_ops_log` , the `kafka_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources#tinybird-kafka-ops-log) contains more detailed logs. Filter by `datasource_id` to select the correct datasource, and use `msg_type` to select the desired log level ( `info`, `warning` , or `error` ).

SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']

---

URL: https://www.tinybird.co/docs/get-data-in/connectors/bigquery
Last update: 2025-02-03T12:06:33.000Z
Content:
---
title: "BigQuery Connector · Tinybird Docs"
theme-color: "#171612"
description: "Use the BigQuery Connector to load data from BigQuery into Tinybird."
---




# BigQuery Connector [¶](https://www.tinybird.co/docs/about:blank#bigquery-connector)

Use the BigQuery Connector to load data from BigQuery into Tinybird so that you can quickly turn it into high-concurrency, low-latency API Endpoints. You can load full tables or the result of an SQL query.

The BigQuery Connector is fully managed and requires no additional tooling. You can define a sync schedule inside Tinybird and execution is taken care of for you.

With the BigQuery Connector you can:

- Connect to your BigQuery database with a handful of clicks. Select which tables to sync and set the schedule.
- Use an SQL query to get the data you need from BigQuery and then run SQL queries on that data in Tinybird.
- Use authentication tokens to control access to API endpoints. Implement access policies as you need, with support for row-level security.

Check the [use case examples](https://github.com/tinybirdco/use-case-examples) repository for examples of BigQuery Data Sources iteration using Git integration.

The BigQuery Connector can't access BigQuery external tables, like connected Google Sheets. If you need this functionality, reach out to [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

- Tinybird CLI. See[  the Tinybird CLI quick start](https://www.tinybird.co/docs/docs/cli/quick-start)  .
- Tinybird CLI[  authenticated with the desired Workspace](https://www.tinybird.co/docs/docs/cli/install)  .

You can switch the Tinybird CLI to the correct Workspace using `tb workspace use `.

To use version control, connect your Tinybird Workspace with [your repository](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/working-with-version-control#connect-your-workspace-to-git-from-the-cli) , and set the [CI/CD configuration](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/continuous-integration) . For testing purposes, use a different connection than in the main branches or Workspaces.

For instance to create the connections in the main branch or Workspace using the CLI:

tb auth # Use the main Workspace admin Token
tb connection create bigquery
# Prompts are interactive and ask you to insert the necessary information You can only create connections in the main Workspace. Even when creating the connection in the branch or as part of a Data Source creation flow, it's created in the main workspace and from there it's available for every branch.

## Load a BigQuery table [¶](https://www.tinybird.co/docs/about:blank#load-a-bigquery-table)

### Load a BigQuery table in the UI [¶](https://www.tinybird.co/docs/about:blank#load-a-bigquery-table-in-the-ui)

Follow these steps to load a table using the Tinybird UI:

1. Open the Tinybird UI and add a new Data Source by clicking**  Create new (+)**   next to the Data Sources section.
2. In the dialog, select the**  BigQuery**   option from the list of Data Sources.
3. Follow the instructions and configure access to your BigQuery database.
4. Access the GCP IAM Dashboard by selecting the**  IAM & Admin**   link, and use the provided principal name from the dialog.

In GCP, follow these steps:

1. In the IAM Dashboard, select**  Grant Access**  .
2. Paste the**  principal**   name you just copied into the**  New principals**   box.
3. In the**  Role**   box, find and select the role**  BigQuery Data Viewer**
4. Select**  Save**  .

The principal should now be listed in the **View By Principals** list.

To complete the setup, return to the Tinybird UI and follow these steps:

1. In the dialog, select**  Next**   . It can take a few seconds for the GCP permissions to apply.
2. Browse the tables available in BigQuery, and select the table you wish to load. Start by selecting the project that the table belongs to, then the dataset and finally the table.
3. Select**  Next**  .

Note: the maximum allowed table size is 50 million rows, the result will be truncated if it exceeds that limit.

1. Configure the schedule on which you wish to load data. You can configure a schedule in minutes, hours, or days by using the drop down selector, and set the value for the schedule in the text field. Finish by selecting**  Next**  .

Note: the maximum allowed frequency is 5 minutes.

1. Edit the schema as needed. You can also modify what the Data Source in Tinybird will be called by changing the name at the top.
2. Select**  Create Data Source**  .

You are now on the Data Source data page, where you can view the data that has been loaded and a status chart showing executions of the loading schedule.

### Load a BigQuery table in the CLI [¶](https://www.tinybird.co/docs/about:blank#load-a-bigquery-table-in-the-cli)

Before you can load a BigQuery table into Tinybird using the CLI, you need to create a connection. Creating a connection grants your Tinybird Workspace the appropriate permissions to view data from BigQuery.

1. [  Authenticate your CLI](https://www.tinybird.co/docs/docs/cli/install#authentication)   and switch to the desired Workspace. Then run:

tb connection create bigquery The output of this command includes instructions to configure a GCP principal with read only access to your data in BigQuery. The instructions include the URL to access the appropriate page in GCP's IAM Dashboard.

1. Copy the**  principal name**   shown in the output.
2. In the GCP IAM Dashboard, select the**  Grant Access**   button.
3. Paste the**  principal**   name you just copied into the**  New principals**   box.
4. In the**  Role**   box, find and select the role**  BigQuery Data Viewer**  .
5. Click**  Save**   to complete. The principal should now be listed in the**  View By Principals**   list.

Note: It can take a few seconds for the GCP permissions to apply.

1. Select**  yes**   (y) to create the connection. A new `bigquery.connection`   file is created in your project files.

Now that your connection is created, you can create a Data Source and configure the schedule to import data from BigQuery.

The BigQuery import is configured using the following options, which can be added at the end of your .datasource file:

- `IMPORT_SERVICE`   : name of the import service to use, in this case, `bigquery`
- `IMPORT_SCHEDULE`   : a cron expression (UTC) with the frequency to run imports, must be higher than 5 minutes, e.g. `*/5 * * * *`
- `IMPORT_STRATEGY`   : the strategy to use when inserting data, must be set to `REPLACE`
- `IMPORT_EXTERNAL_DATASOURCE`   : (optional) the fully qualified name of the source table in BigQuery e.g. `project.dataset.table`
- `IMPORT_QUERY`   : (optional) the SELECT query to extract your data from BigQuery when you don't need all the columns or want to make a transformation before ingestion. The FROM must reference a table using the full scope: `project.dataset.table`

Both `IMPORT_EXTERNAL_DATASOURCE` and `IMPORT_QUERY` are optional, but you must provide one of them for the connector to work.

For example:

##### bigquery.datasource file

DESCRIPTION >
    bigquery demo data source

SCHEMA >
    'timestamp' DateTime 'json:$.timestamp',
    'id' Integer 'json:$.id',
    'orderid' LowCardinality(String) 'json:$.orderid',
    'status' LowCardinality(String) 'json:$.status',
    'amount' Integer 'json:$.amount'

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE bigquery
IMPORT_SCHEDULE */5 * * * *
IMPORT_EXTERNAL_DATASOURCE mydb.raw.events
IMPORT_STRATEGY REPLACE
IMPORT_QUERY >
    select
    timestamp,
    id,
    orderid,
    status,
    amount
        from
        mydb.raw.events The columns you select in the `IMPORT_QUERY` must match the columns defined in the Data Source schema. For example, if your Data Source has the columns `ColumnA, ColumnB` then your `IMPORT_QUERY` must contain `SELECT ColumnA, ColumnB FROM ...` . A mismatch of columns causes data to arrive in the [quarantine Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-operations/recover-from-quarantine).

With your connection created and Data Source defined, you can now push your project to Tinybird using:

tb push The first run of the import will begin on the next lapse of the CRON expression.

If you need to sync your data using an incremental strategy, set up a [scheduled query](https://cloud.google.com/bigquery/docs/scheduling-queries) in BigQuery that [unloads data to S3](https://cloud.google.com/bigquery/docs/omni-aws-export-results-to-s3) , and then use our managed [S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3) . Reach out to [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) if you need help.

## Configure granular permissions [¶](https://www.tinybird.co/docs/about:blank#configure-granular-permissions)

If you need to configure more granular permissions for BigQuery, you can always grant access at dataset or individual object level.

1. Create a new role in your[  IAM & Admin Console in GCP](https://console.cloud.google.com/iam-admin/roles/create)   , and assign the `resourcemanager.projects.get`   permission.The Connector needs this permission to list the available projects the generated Service Account has access to, so you can explore the BigQuery tables and views in the Tinybird UI.
2. Grant permissions to specific datasets to the Service Account by clicking on**  Sharing**   >**  Permissions**  :
3. Select**  ADD PRINCIPAL**  .
4. Paste the**  principal**   name copied earlier into the**  New principals**   box.
5. In the**  Role**   box, find and select the role**  BigQuery Data Viewer**  .

Now the Tinybird Connector UI only shows the specific resources you've granted permissions to.

## Schema evolution [¶](https://www.tinybird.co/docs/about:blank#schema-evolution)

The BigQuery Connector supports backwards compatible changes made in the source table. This means that, if you add a new column in BigQuery, the next sync job will automatically add it to the Tinybird Data Source.

Non-backwards compatible changes, such as dropping or renaming columns, aren't supported and will cause the next sync to fail.

## Iterate a BigQuery Data Source [¶](https://www.tinybird.co/docs/about:blank#iterate-a-bigquery-data-source)

To iterate a BigQuery Data Source, use the Tinybird CLI and the version control integration to handle your resources.

You can only create connections in the main Workspace. When creating the connection in a Branch, it's created in the main Workspace and from there is available to every Branch.

### Add a new BigQuery Data Source [¶](https://www.tinybird.co/docs/about:blank#add-a-new-bigquery-data-source)

You can add a new Data Source directly with the UI or the CLI tool, following [the load of a BigQuery table section](https://www.tinybird.co/docs/about:blank#load-a-BigQuery-table).

When adding a Data Source in a Tinybird Branch, it will work for testing purposes, but won't have any connection details internally. You must add the connection and BigQuery configuration in the .datasource Datafile when moving to production.

To add a new Data Source using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples).

### Update a Data Source [¶](https://www.tinybird.co/docs/about:blank#update-a-data-source)

- BigQuery Data Sources can't be modified directly from UI
- When you create a new Tinybird Branch, the existing BigQuery Data Sources won't be connected. You need to re-create them in the Branch.
- In Branches, it's usually useful to work with[  fixtures](https://www.tinybird.co/docs/docs/work-with-data/strategies/implementing-test-strategies#fixture-tests)   , as they'll be applied as part of the CI/CD, allowing the full process to be deterministic in every iteration and avoiding quota consume from external services.

BigQuery Data Sources can be modified from the CLI tool:

tb auth
# modify the .datasource Datafile with your editor
tb push --force {datafile}
# check the command output for errors To update it using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples).

### Delete a Data Source [¶](https://www.tinybird.co/docs/about:blank#delete-a-data-source)

BigQuery Data Sources can be deleted directly from UI or CLI like any other Data Source.

To delete it using the recommended version control workflow check the instructions in the [examples repository](https://github.com/tinybirdco/use-case-examples).

## Logs [¶](https://www.tinybird.co/docs/about:blank#logs)

Job executions are logged in the `datasources_ops_log` [Service Data Source](https://www.tinybird.co/docs/docs/monitoring/service-datasources) . This log can be checked directly in the Data Source view page in the UI. Filter by `datasource_id` to monitor ingestion through the BigQuery Connector from the `datasources_ops_log`:

SELECT
  timestamp,
  event_type,
  result,
  error,
  job_id
FROM
  tinybird.datasources_ops_log
WHERE
  datasource_id = 't_1234'
AND
  event_type = 'replace'
ORDER BY timestamp DESC
## Limits [¶](https://www.tinybird.co/docs/about:blank#limits)

See [BigQuery Connector limits](https://www.tinybird.co/docs/docs/get-started/plans/limits#bigquery-connector-limits).



---

URL: https://www.tinybird.co/docs/cli/datafiles/pipe-files
Last update: 2025-01-07T15:48:10.000Z
Content:
---
title: "Pipe files · Tinybird Docs"
theme-color: "#171612"
description: "Pipe files describe your Tinybird Pipes. Define the type, Data Source, and other settings."
---




# Pipe files (.pipe) [¶](https://www.tinybird.co/docs/about:blank#pipe-files-pipe)

Pipe files describe your Pipes. You can use .pipe files to define the type, starting node, Data Source, and other settings of your Pipes. See [Data Sources](https://www.tinybird.co/docs/docs/work-with-data/query/pipes).

## Available instructions [¶](https://www.tinybird.co/docs/about:blank#available-instructions)

The following instructions are available for .pipe files.

| Instruction | Required | Description |
| --- | --- | --- |
| `%` | No | Use as the first character of a node to indicate the node uses the[  templating system](https://www.tinybird.co/docs/docs/cli/template-functions)  . |
| `DESCRIPTION ` | No | Sets the description for a node or the complete file. |
| `TAGS ` | No | Comma-separated list of tags. Tags are used to[  organize your data project](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/organizing-resources)  . |
| `NODE ` | Yes | Starts the definition of a new node. All the instructions until a new `NODE`   instruction or the end of the file are related to this node. |
| `SQL ` | Yes | Defines a block for the SQL of a node. The block must be indented. |
| `INCLUDE  ` | No | Includes are pieces of a Pipe that you can reuse in multiple .pipe files. |
| `TYPE ` | No | Sets the type of the node. Valid values are `ENDPOINT`  , `MATERIALIZED`  , `COPY`   , or `SINK`  . |
| `DATASOURCE ` | Yes | Required when `TYPE`   is `MATERIALIZED`   . Sets the destination Data Source for materialized nodes. |
| `TARGET_DATASOURCE ` | Yes | Required when `TYPE`   is `COPY`   . Sets the destination Data Source for copy nodes. |
| `TOKEN  READ` | No | Grants read access to a Pipe or Endpoint to the token named . If the token isn't specified or  doesn't exist, it will be automatically created. |
| `COPY_SCHEDULE` | No | Cron expression with the frequency to run copy jobs. Must be higher than 5 minutes. For example, `*/5 * * * *`   . If undefined, it defaults to `@on-demand`  . |
| `COPY_MODE` | No | Strategy to ingest data for copy jobs. One of `append`   or `replace`   . If empty, the default strategy is `append`  . |

## Materialized Pipe [¶](https://www.tinybird.co/docs/about:blank#materialized-pipe)

In a .pipe file you can define how to materialize each row ingested in the earliest Data Source in the Pipe query to a materialized Data Source. Materialization happens at ingest.

The following example shows how to describe a Materialized Pipe. See [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views).

##### tinybird/pipes/sales_by_hour_mv.pipe

DESCRIPTION Materialized Pipe to aggregate sales per hour in the sales_by_hour Data Source

NODE daily_sales
SQL >
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    GROUP BY day, country

TYPE MATERIALIZED
DATASOURCE sales_by_hour
## Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#copy-pipe)

In a .pipe file you can define how to export the result of a Pipe to a Data Source, optionally with a schedule.

The following example shows how to describe a Copy Pipe. See [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes).

##### tinybird/pipes/sales_by_hour_cp.pipe

DESCRIPTION Copy Pipe to export sales hour every hour to the sales_hour_copy Data Source

NODE daily_sales
SQL >
    %
    SELECT toStartOfDay(starting_date) day, country, sum(sales) as total_sales
    FROM teams
    WHERE
    day BETWEEN toStartOfDay(now()) - interval 1 day AND toStartOfDay(now())
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

TYPE COPY
TARGET_DATASOURCE sales_hour_copy
COPY_SCHEDULE 0 * * * *
## API Endpoint Pipe [¶](https://www.tinybird.co/docs/about:blank#api-endpoint-pipe)

In a .pipe file you can define how to export the result of a Pipe as an HTTP endpoint.

The following example shows how to describe an API Endpoint Pipe. See [API Endpoints](https://www.tinybird.co/docs/docs/publish/api-endpoints).

##### tinybird/pipes/sales_by_hour_endpoint.pipe

TOKEN dashboard READ
DESCRIPTION endpoint to get sales by hour filtering by date and country

TAGS sales

NODE daily_sales
SQL >
    %
    SELECT day, country, sum(total_sales) as total_sales
    FROM sales_by_hour
    WHERE
    day BETWEEN toStartOfDay(now()) - interval 1 day AND toStartOfDay(now())
    and country = {{ String(country, 'US')}}
    GROUP BY day, country

NODE result
SQL >
    %
    SELECT * FROM daily_sales
    LIMIT {{Int32(page_size, 100)}}
    OFFSET {{Int32(page, 0) * Int32(page_size, 100)}}

TYPE ENDPOINT
## Sink Pipe [¶](https://www.tinybird.co/docs/about:blank#sink-pipe)

The following parameters are available when defining Sink Pipes:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_SERVICE` | Yes | One of `gcs_hmac`  , `s3`  , `s3_iamrole`   , or `kafka`  . |
| `EXPORT_CONNECTION_NAME` | Yes | The name of the export connection. |
| `EXPORT_SCHEDULE` | No | Cron expression, in UTC time. Must be higher than 5 minutes. For example, `*/5 * * * *`  . |

### Blob storage Sink [¶](https://www.tinybird.co/docs/about:blank#blob-storage-sink)

When setting `EXPORT_SERVICE` as one of `gcs_hmac`, `s3` , or `s3_iamrole` , you can use the following instructions:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_BUCKET_URI` | Yes | The desired bucket path for the exported file. Path must not include the filename and extension. |
| `EXPORT_FILE_TEMPLATE` | Yes | Template string that specifies the naming convention for exported files. The template can include dynamic attributes between curly braces based on columns' data that will be replaced with real values when exporting. For example: `export_{category}{date,'%Y'}{2}`  . |
| `EXPORT_FORMAT` | Yes | Format in which the data is exported. The default value is `csv`  . |
| `EXPORT_COMPRESSION` | No | Compression file type. Accepted values are `none`  , `gz`   for gzip, `br`   for brotli, `xz`   for LZMA, `zst`   for zstd. Default values is `none`  . |
| `EXPORT_STRATEGY` | Yes | One of the available strategies. The default is `@new`  . |

### Kafka Sink [¶](https://www.tinybird.co/docs/about:blank#kafka-sink)

Kafka Sinks are currently in private beta. If you have any feedback or suggestions, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

When setting `EXPORT_SERVICE` as `kafka` , you can use the following instructions:

| Instruction | Required | Description |
| --- | --- | --- |
| `EXPORT_KAFKA_TOPIC` | Yes | The desired topic for the export data. |



---

URL: https://www.tinybird.co/docs/cli/datafiles/include-files
Last update: 2024-12-18T11:12:31.000Z
Content:
---
title: "Include files · Tinybird Docs"
theme-color: "#171612"
description: "Include files help you organize settings so that you can reuse them across .datasource and .pipe files."
---




# Include files (.incl) [¶](https://www.tinybird.co/docs/about:blank#include-files-incl)

Include files (.incl) help separate connector settings and reuse them across multiple .datasource files or .pipe templates.

Include files are referenced using `INCLUDE` instruction.

## Connector settings [¶](https://www.tinybird.co/docs/about:blank#connector-settings)

Use .incl files to separate connector settings from .datasource files.

For example, the following .incl file contains Kafka Connector settings:

##### tinybird/datasources/connections/kafka_connection.incl

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password While the .datasource file only contains a reference to the .incl file using `INCLUDE`:

##### tinybird/datasources/kafka_ds.datasource

SCHEMA >
`value` String,
`topic` LowCardinality(String),
`partition` Int16,
`offset` Int64,
`timestamp` DateTime,
`key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/kafka_connection.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
### Pipe nodes [¶](https://www.tinybird.co/docs/about:blank#pipe-nodes)

You can use .incl datafiles to [reuse node templates](https://www.tinybird.co/docs/docs/cli/advanced-templates#reusing-templates).

For example, the following .incl file contains a node template:

##### tinybird/includes/only_buy_events.incl

NODE only_buy_events
SQL >
    SELECT
        toDate(timestamp) date,
        product,
        color,
        JSONExtractFloat(json, 'price') as price
    FROM events
    where action = 'buy' The .pipe file starts with the `INCLUDE` reference to the template:

##### tinybird/endpoints/sales.pipe

INCLUDE "../includes/only_buy_events.incl"

NODE endpoint
DESCRIPTION >
    return sales for a product with color filter
SQL >
    %
    select date, sum(price) total_sales
    from only_buy_events
    where color in {{Array(colors, 'black')}}
    group by date A different .pipe file can reuse the sample template:

##### tinybird/pipes/top_per_day.pipe

INCLUDE "../includes/only_buy_events.incl"

NODE top_per_day
SQL >
SELECT date,
        topKState(10)(product) top_10,
        sumState(price) total_sales
from only_buy_events
group by date

TYPE MATERIALIZED
DATASOURCE mv_top_per_day
### Include with variables [¶](https://www.tinybird.co/docs/about:blank#include-with-variables)

You can templatize .incl files. For instance you can reuse the same .incl template with different variable values:

##### tinybird/includes/top_products.incl

NODE endpoint

DESCRIPTION >
    returns top 10 products for the last week

SQL >
    %
    select
        date,
        topKMerge(10)(top_10) as top_10
    from top_product_per_day

    {% if '$DATE_FILTER' == 'last_week' %}
        where date > today() - interval 7 day
    {% else %}
        where date between {{Date(start)}} and {{Date(end)}}
    {% end %}

    group by date The `$DATE_FILTER` parameter is a variable in the .incl file. The following examples show how to create two separate endpoints by injecting a value for the `DATE_FILTER` variable.

The following .pipe file references the template using a `last_week` value for `DATE_FILTER`:

##### tinybird/endpoints/top_products_last_week.pipe

INCLUDE "../includes/top_products.incl" "DATE_FILTER=last_week" Whereas the following .pipe file references the template using a `between_dates` value for `DATE_FILTER`:

##### tinybird/endpoints/top_products_between_dates.pipe

INCLUDE "../includes/top_products.incl" "DATE_FILTER=between_dates"
### Include with environment variables [¶](https://www.tinybird.co/docs/about:blank#include-with-environment-variables)

Because you can expand `INCLUDE` files using the Tinybird CLI, you can use environment variables.

For example, if you have configured the `KAFKA_BOOTSTRAP_SERVERS`, `KAFKA_KEY` , and `KAFKA_SECRET` environment variables, you can create an .incl file as follows:

##### tinybird/datasources/connections/kafka_connection.incl

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS ${KAFKA_BOOTSTRAP_SERVERS}
KAFKA_KEY ${KAFKA_KEY}
KAFKA_SECRET ${KAFKA_SECRET} You can then use the values in your .datasource datafiles:

##### tinybird/datasources/kafka_ds.datasource

SCHEMA >
`value` String,
`topic` LowCardinality(String),
`partition` Int16,
`offset` Int64,
`timestamp` DateTime,
`key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/kafka_connection.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id Alternatively, you can create separate .incl files per environment variable:

##### tinybird/datasources/connections/kafka_connection_prod.incl

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS production_servers
KAFKA_KEY the_kafka_key
KAFKA_SECRET ${KAFKA_SECRET}
##### tinybird/datasources/connections/kafka_connection_stg.incl

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS staging_servers
KAFKA_KEY the_kafka_key
KAFKA_SECRET ${KAFKA_SECRET} And then include both depending on the environment:

##### tinybird/datasources/kafka_ds.datasource

SCHEMA >
`value` String,
`topic` LowCardinality(String),
`partition` Int16,
`offset` Int64,
`timestamp` DateTime,
`key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/kafka_connection_${TB_ENV}.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id Where `$TB_ENV` is one of `stg` or `prod`.

See [deploy to staging and production environments](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/staging-and-production-workspaces) to learn how to leverage environment variables.



---

URL: https://www.tinybird.co/docs/cli/datafiles/datasource-files
Last update: 2025-01-31T08:53:28.000Z
Content:
---
title: "Datasource files · Tinybird Docs"
theme-color: "#171612"
description: "Datasource files describe your Data Sources. Define the schema, engine, and other settings."
---




# Datasource files (.datasource) [¶](https://www.tinybird.co/docs/about:blank#datasource-files-datasource)

Datasource files describe your Data Sources. You can use .datasource files to define the schema, engine, and other settings of your Data Sources. See [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources).

## Available instructions [¶](https://www.tinybird.co/docs/about:blank#available-instructions)

The following instructions are available for .datasource files.

| Declaration | Required | Description |
| --- | --- | --- |
| `SCHEMA ` | Yes | Defines a block for a Data Source schema. The block must be indented. |
| `DESCRIPTION ` | No | Description of the Data Source. |
| `TOKEN  APPEND` | No | Grants append access to a Data Source to the token named . If the token isn't specified or  doesn't exist, it will be automatically created. |
| `TAGS ` | No | Comma-separated list of tags. Tags are used to[  organize your data project](https://www.tinybird.co/docs/docs/work-with-data/organize-your-work/organizing-resources)  . |
| `ENGINE ` | No | Sets the engine for Data Source. Default value is `MergeTree`  . |
| `ENGINE_SORTING_KEY ` | No | Sets the `ORDER BY`   expression for the Data Source. If unset, it defaults to DateTime, numeric, or String columns, in that order. |
| `ENGINE_PARTITION_KEY ` | No | Sets the `PARTITION`   expression for the Data Source. |
| `ENGINE_TTL ` | No | Sets the `TTL`   expression for the Data Source. |
| `ENGINE_VER ` | No | Column with the version of the object state. Required when using `ENGINE ReplacingMergeTree`  . |
| `ENGINE_SIGN ` | No | Column to compute the state. Required when using `ENGINE CollapsingMergeTree`   or `ENGINE VersionedCollapsingMergeTree`  . |
| `ENGINE_VERSION ` | No | Column with the version of the object state. Required when `ENGINE VersionedCollapsingMergeTree`  . |
| `ENGINE_SETTINGS ` | No | Comma-separated list of key-value pairs that describe engine settings for the Data Source. |
| `INDEXES ` | No | Defines one or more indexes for the Data Source. See[  Data Skipping Indexes](https://www.tinybird.co/docs/docs/sql-reference/engines/mergetree#data-skipping-indexes)   for more information. |
| `SHARED_WITH ` | No | Shares the Data Source with one or more Workspaces. Use in combination with `--user_token`   with admin rights in the origin Workspace. |

The following example shows a typical .datasource file:

##### tinybird/datasources/example.datasource

# A comment
TOKEN tracker APPEND

DESCRIPTION >
    Analytics events **landing data source**

TAGS stock, recommendations

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"
ENGINE_SETTINGS "index_granularity=8192"

INDEXES >
    INDEX idx1 action TYPE bloom_filter GRANULARITY 3

SHARED_WITH >
    analytics_production
    analytics_staging
### SCHEMA [¶](https://www.tinybird.co/docs/about:blank#schema)

A `SCHEMA` declaration is a newline, comma-separated list of columns definitions. For example:

##### Example SCHEMA declaration

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload` Each column in a `SCHEMA` declaration is in the format `   ` , where:

- ``   is the name of the column in the Data Source.
- ``   is one of the supported[  Data Types](https://www.tinybird.co/docs/docs/get-data-in/data-sources#supported-data-types)  .
- ``   is optional and only required for NDJSON Data Sources. See[  JSONpaths](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths)  .
- ``   sets a default value to the column when it's null. A common use case is to set a default date to a column, like `updated_at DateTime DEFAULT now()`  .

To change or update JSONPaths or other default values in the schema, push a new version of the schema using `tb push --force` or use the [alter endpoint on the Data Sources API](https://www.tinybird.co/docs/docs/api-reference/datasource-api#post--v0-datasources-(.+)-alter).

### JSONPath expressions [¶](https://www.tinybird.co/docs/about:blank#jsonpath-expressions)

`SCHEMA` definitions support JSONPath expressions. For example:

##### Schema syntax with jsonpath

DESCRIPTION Generated from /Users/username/tmp/sample.ndjson

SCHEMA >
    `d` DateTime `json:$.d`,
    `total` Int32 `json:$.total`,
    `from_novoa` Int16 `json:$.from_novoa` See [JSONPaths](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-ndjson-data#jsonpaths) for more information.

### ENGINE settings [¶](https://www.tinybird.co/docs/about:blank#engine-settings)

`ENGINE` declares the engine used for the Data Source. The default value is `MergeTree`.

See [Engines](https://www.tinybird.co/docs/docs/sql-reference/engines) for more information.

## Connectors [¶](https://www.tinybird.co/docs/about:blank#connectors)

Connector settings are part of the .datasource content. You can use include files to reuse connection settings and credentials.

When working with connectors, it’s important to understand how tokens interact with .datasource files. If a token doesn’t exist or isn't explicitly specified in the .datasource, it will be automatically created. This ensures that connectors can establish a working connection by default.

However, once a token is created and associated with a connector, it's crucial to handle it with care. Avoid deleting the token or modifying its scopes, as this can break the connection and disrupt the import process. The token is a critical component for maintaining a stable connection and ensuring that data is imported correctly.

### Kafka, Confluent, RedPanda [¶](https://www.tinybird.co/docs/about:blank#kafka-confluent-redpanda)

The Kafka, Confluent, and RedPanda connectors use the following settings:

| Instruction | Required | Description |
| --- | --- | --- |
| `KAFKA_CONNECTION_NAME` | Yes | The name of the configured Kafka connection in Tinybird. |
| `KAFKA_BOOTSTRAP_SERVERS` | Yes | Comma-separated list of one or more Kafka brokers, including Port numbers. |
| `KAFKA_KEY` | Yes | Key used to authenticate with Kafka. Sometimes called Key, Client Key, or Username, depending on the Kafka distribution. |
| `KAFKA_SECRET` | Yes | Secret used to authenticate with Kafka. Sometimes called Secret, Secret Key, or Password, depending on the Kafka distribution. |
| `KAFKA_TOPIC` | Yes | Name of the Kafka topic to consume from. |
| `KAFKA_GROUP_ID` | Yes | Consumer Group ID to use when consuming from Kafka. |
| `KAFKA_AUTO_OFFSET_RESET` | No | Offset to use when no previous offset can be found, for example when creating a new consumer. Supported values are `latest`  , `earliest`   . Default: `latest`  . |
| `KAFKA_STORE_HEADERS` | No | Store Kafka headers as field `__headers`   for later processing. Default value is `'False'`  . |
| `KAFKA_STORE_BINARY_HEADERS` | No | Stores all Kafka headers as binary data in field `__headers`   as a binary map of type `Map(String, String)`   . To access the header `'key'`   run: `__headers['key']`   . Default value is `'True'`   . This field only applies if `KAFKA_STORE_HEADERS`   is set to `True`  . |
| `KAFKA_STORE_RAW_VALUE` | No | Stores the raw message in its entirety as an additional column. Supported values are `'True'`  , `'False'`   . Default: `'False'`  . |
| `KAFKA_SCHEMA_REGISTRY_URL` | No | URL of the Kafka schema registry. |
| `KAFKA_TARGET_PARTITIONS` | No | Target partitions to place the messages. |
| `KAFKA_KEY_FORMAT` | No | Format of the message value. Valid values are `avro`  , `json_with_schema`   , and `json_without_schema`   . Using `avro`   or `json_with_schema`   requires `KAFKA_SCHEMA_REGISTRY_URL`   to be set. |
| `KAFKA_VALUE_FORMAT` | No | Format of the message value. Valid values are `avro`  , `json_with_schema`   , and `json_without_schema`   . Using `avro`   or `json_with_schema`   requires `KAFKA_SCHEMA_REGISTRY_URL`   to be set. |
| `KAFKA_KEY_AVRO_DESERIALIZATION` | No | If the key of the message is serialized in avro format, allow decoding it using the Avro definition stored in `KAFKA_SCHEMA_REGISTRY_URL`   . The key is converted to JSON and stored in the `__key`   column. Defaults to `'False'`   . A deprecated option is to use the `KAFKA_KEY_FORMAT`   parameter with the `avro`   value. |
| `KAFKA_SSL_CA_PEM` | No | CA certificate in PEM format for SSL connections. |
| `KAFKA_SASL_MECHANISM` | No | SASL mechanism to use for authentication. Supported values are `'PLAIN'`  , `'SCRAM-SHA-256'`  , `'SCRAM-SHA-512'`   . Default values is `'PLAIN'`  . |

The following example defines a Data Source with a new Kafka, Confluent, or RedPanda connection in a .datasource file:

##### Data Source with a new Kafka/Confluent/RedPanda connection

SCHEMA >
  `value` String,
  `topic` LowCardinality(String),
  `partition` Int16,
  `offset` Int64,
  `timestamp` DateTime,
  `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id The following example defines a Data Source that uses an existing Kafka, Confluent, or RedPanda connection:

##### Data Source with an existing Kafka/Confluent/RedPanda connection

SCHEMA >
  `value` String,
  `topic` LowCardinality(String),
  `partition` Int16,
  `offset` Int64,
  `timestamp` DateTime,
  `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id Refer to the [Kafka Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/kafka), [Amazon MSK](https://www.tinybird.co/docs/docs/get-data-in/connectors/msk), [Confluent Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/confluent) , or [RedPanda Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/redpanda) documentation for more details.

### BigQuery [¶](https://www.tinybird.co/docs/about:blank#bigquery)

The BigQuery connector uses the following settings:

| Instruction | Required | Description |
| --- | --- | --- |
| `IMPORT_SERVICE` | Yes | Name of the import service to use. Use `bigquery`  . |
| `IMPORT_SCHEDULE` | Yes | Cron expression, in UTC time, with the frequency to run imports. Must be higher than 5 minutes. For example, `*/5 * * * *`   . Use `@auto`   to sync once per minute when using `s3`   , or `@on-demand`   to only run manually. |
| `IMPORT_CONNECTION_NAME` | Yes | Name given to the connection inside Tinybird. For example, `'my_connection'`  . |
| `IMPORT_STRATEGY` | Yes | Strategy to use when inserting data. Use `REPLACE`   for BigQuery. |
| `IMPORT_EXTERNAL_DATASOURCE` | No | Fully qualified name of the source table in BigQuery. For example, `project.dataset.table`  . |
| `IMPORT_QUERY` | No | The `SELECT`   query to retrieve your data from BigQuery when you don't need all the columns or want to make a transformation before ingest. The `FROM`   clause must reference a table using the full scope. For example, `project.dataset.table`  . |

See [BigQuery Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/bigquery) for more details.

#### BigQuery example [¶](https://www.tinybird.co/docs/about:blank#bigquery-example)

The following example shows a BigQuery Data Source described in a .datasource file:

##### Data Source with a BigQuery connection

DESCRIPTION >
    bigquery demo data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `id` Integer `json:$.id`,
    `orderid` LowCardinality(String) `json:$.orderid`,
    `status` LowCardinality(String) `json:$.status`,
    `amount` Integer `json:$.amount`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE bigquery
IMPORT_SCHEDULE */5 * * * *
IMPORT_EXTERNAL_DATASOURCE mydb.raw.events
IMPORT_STRATEGY REPLACE
IMPORT_QUERY >
    select
    timestamp,
    id,
    orderid,
    status,
    amount
        from
        mydb.raw.events
### S3 [¶](https://www.tinybird.co/docs/about:blank#s3)

The S3 connector uses the following settings:

| Instruction | Required | Description |
| --- | --- | --- |
| `IMPORT_SERVICE` | Yes | Name of the import service to use. Use `s3`   for S3 connections. |
| `IMPORT_CONNECTION_NAME` | Yes | Name given to the connection inside Tinybird. For example, `'my_connection'`  . |
| `IMPORT_STRATEGY` | Yes | Strategy to use when inserting data. Use `APPEND`   for S3 connections. |
| `IMPORT_BUCKET_URI` | Yes | Full bucket path, including the `s3://`   protocol, bucket name, object path, and an optional pattern to match against object keys. For example, `s3://my-bucket/my-path`   discovers all files in the bucket `my-bucket`   under the prefix `/my-path`   . You can use patterns in the path to filter objects, for example, ending the path with `*.csv`   matches all objects that end with the `.csv`   suffix. |
| `IMPORT_FROM_DATETIME` | No | Sets the date and time from which to start ingesting files on an S3 bucket. The format is `YYYY-MM-DDTHH:MM:SSZ`  . |

See [S3 Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/s3) for more details.

#### S3 example [¶](https://www.tinybird.co/docs/about:blank#s3-example)

The following example shows an S3 Data Source described in a .datasource file:

##### tinybird/datasources/s3.datasource - Data Source with an S3 connection

DESCRIPTION >
    Analytics events landing data source

SCHEMA >
    `timestamp` DateTime `json:$.timestamp`,
    `session_id` String `json:$.session_id`,
    `action` LowCardinality(String) `json:$.action`,
    `version` LowCardinality(String) `json:$.version`,
    `payload` String `json:$.payload`

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"
ENGINE_TTL "timestamp + toIntervalDay(60)"

IMPORT_SERVICE s3
IMPORT_CONNECTION_NAME connection_name
IMPORT_BUCKET_URI s3://my-bucket/*.csv
IMPORT_SCHEDULE @auto
IMPORT_STRATEGY APPEND

---

URL: https://www.tinybird.co/docs/api-reference/pipe-api/materialized-views
Last update: 2024-12-30T16:50:50.000Z
Content:
---
title: "Pipes API Materialized Views and Populates reference · Tinybird Docs"
theme-color: "#171612"
description: "The Pipes API enables you to manage your Pipes. Use the Materialized Views and Populates service to create, delete, or populate Materialized Views."
---

POST /v0/pipes/(.+)/nodes/(.+)/population [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-nodes-(.+)-population) Populates a Materialized View

Populating a Materialized View [¶](https://www.tinybird.co/docs/about:blank#id1) curl -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/population" \ -d "populate_condition=toYYYYMM(date) = 202203" The response will not be the final result of the import but a Job. You can check the job status and progress using the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api).

Alternatively you can use a query like this to check the operations related to the populate Job:

Check populate jobs in the datasources_ops_log including dependent Materialized Views triggered [¶](https://www.tinybird.co/docs/about:blank#id2) SELECT * FROM tinybird. datasources_ops_log WHERE timestamp > now () - INTERVAL 1 DAY AND operation_id IN ( SELECT operation_id FROM tinybird. datasources_ops_log WHERE timestamp > now () - INTERVAL 1 DAY and datasource_id = '{the_datasource_id}' and job_id = '{the_job_id}' ) ORDER BY timestamp ASC When a populate job fails for the first time, the Materialized View is automatically unlinked. In that case you can get failed population jobs and their errors to fix them with a query like this:

Check failed populate jobs [¶](https://www.tinybird.co/docs/about:blank#id3) SELECT * FROM tinybird. datasources_ops_log WHERE datasource_id = '{the_datasource_id}' AND pipe_name = '{the_pipe_name}' AND event_type LIKE 'populateview%' AND result = 'error' ORDER BY timestamp ASC Alternatively you can use the `unlink_on_populate_error='true'` flag to always unlink the Materialized View if the populate job does not work as expected.

| Key | Type | Description |
| --- | --- | --- |
| token | String | Auth token. Ensure it has the `PIPE:CREATE`   scope on it |
| populate_subset | Float | Optional. Populate with a subset percent of the data (limited to a maximum of 2M rows), this is useful to quickly test a materialized node with some data. The subset must be greater than 0 and lower than 0.1. A subset of 0.1 means a 10 percent of the data in the source Data Source will be used to populate the Materialized View. It has precedence over `populate_condition` |
| populate_condition | String | Optional. Populate with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `populate_condition='date == toYYYYMM(now())'`   it’ll populate taking all the rows from the trigger Data Source which `date`   is the current month. `populate_condition`   is not taken into account if the `populate_subset`   param is present. Including in the `populate_condition`   any column present in the Data Source `engine_sorting_key`   will make the populate job process less data. |
| truncate | String | Optional. Default is `false`   . Populates over existing data, useful to populate past data while new data is being ingested. Use `true`   to truncate the Data Source before populating. |
| unlink_on_populate_error | String | Optional. Default is `false`   . If the populate job fails the Materialized View is unlinked and new data won’t be ingested in the Materialized View. |

| Code | Description |
| --- | --- |
| 200 | No error |
| 400 | Node is not materialized |
| 403 | Forbidden. Provided token doesn’t have permissions to append a node to the pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found, Node not found |

POST /v0/pipes/(.+)/nodes/(.+)/materialization [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-nodes-(.+)-materialization) Creates a Materialized View

Creating a Materialized View [¶](https://www.tinybird.co/docs/about:blank#id6) curl \ -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/materialization?datasource=my_data_source_name&populate=true"| Key | Type | Description |
| --- | --- | --- |
| token | String | Auth token. Ensure it has the `PIPE:CREATE`   scope on it |
| datasource | String | Required. Specifies the name of the destination Data Source where the Materialized View schema is defined. If the Data Source does not exist, it creates automatically with the default settings. |
| override_datasource | Boolean | Optional. Default `false`   When the target Data Source of the Materialized View exists in the Workspace it’ll be overriden by the `datasource`   specified in the request. |
| populate | Boolean | Optional. Default `false`   . When `true`   , a job is triggered to populate the destination datasource. |
| populate_subset | Float | Optional. Populate with a subset percent of the data (limited to a maximum of 2M rows), this is useful to quickly test a materialized node with some data. The subset must be greater than 0 and lower than 0.1. A subset of 0.1 means a 10 percent of the data in the source Data Source will be used to populate the Materialized View. Use it together with `populate=true`   , it has precedence over `populate_condition` |
| populate_condition | String | Optional. Populate with a SQL condition to be applied to the trigger Data Source of the Materialized View. For instance, `populate_condition='date == toYYYYMM(now())'`   it’ll populate taking all the rows from the trigger Data Source which `date`   is the current month. Use it together with `populate=true`  . `populate_condition`   is not taken into account if the `populate_subset`   param is present. Including in the `populate_condition`   any column present in the Data Source `engine_sorting_key`   will make the populate job process less data. |
| unlink_on_populate_error | String | Optional. Default is `false`   . If the populate job fails the Materialized View is unlinked and new data won’t be ingested in the Materialized View. |
| engine | String | Optional. Engine for destination Materialized View. If the Data Source already exists, the settings are not overriden. |
| engine_* | String | Optional. Engine parameters and options. Requires the `engine`   parameter. If the Data Source already exists, the settings are not overriden.[  Check Engine Parameters and Options for more details](https://www.tinybird.co/docs/docs/api-reference/datasource-api) |

SQL query for the materialized node must be sent in the body encoded in utf-8

| Code | Description |
| --- | --- |
| 200 | No error |
| 400 | Node already being materialized |
| 403 | Forbidden. Provided token doesn’t have permissions to append a node to the pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found, Node not found |
| 409 | The Materialized View already exists or `override_datasource`   cannot be performed |

DELETE /v0/pipes/(.+)/nodes/(.+)/materialization [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+)-nodes-(.+)-materialization) Removes a Materialized View

By removing a Materialized View, nor the Data Source nor the Node are deleted. The Data Source will still be present, but will stop receiving data from the Node.

Removing a Materialized View [¶](https://www.tinybird.co/docs/about:blank#id9) curl -H "Authorization: Bearer " \ -X DELETE "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/materialization"| Code | Description |
| --- | --- |
| 204 | No error, Materialized View removed |
| 403 | Forbidden. Provided token doesn’t have permissions to append a node to the pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found, Node not found |



---

URL: https://www.tinybird.co/docs/api-reference/pipe-api/copy-pipes-api
Last update: 2024-12-30T16:50:50.000Z
Content:
---
title: "Pipes API Copy Pipes reference · Tinybird Docs"
theme-color: "#171612"
description: "The Pipes API enables you to manage your Pipes. Use the Copy Pipes service to create, delete, schedule, and trigger Copy jobs."
---

POST /v0/pipes/(.+)/nodes/(.+)/copy [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-nodes-(.+)-copy) Calling this endpoint sets the pipe as a Copy one with the given settings. Scheduling is optional.

To run the actual copy after you set the pipe as a Copy one, you must call the POST `/v0/pipes/:pipe/copy` endpoint.

If you need to change the target Data Source or the scheduling configuration, you can call PUT endpoint.

Restrictions:

- You can set only one schedule per Copy pipe.
- You can’t set a Copy pipe if the pipe is already materializing. You must unlink the Materialization first.
- You can’t set a Copy pipe if the pipe is already an endpoint. You must unpublish the endpoint first.

Setting the pipe as a Copy pipe [¶](https://www.tinybird.co/docs/about:blank#id1) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/copy" \ -d "target_datasource=my_destination_datasource" \ -d "schedule_cron=*/15 * * * *"| Key | Type | Description |
| --- | --- | --- |
| token | String | Auth token. Ensure it has the `PIPE:CREATE`   and `DATASOURCE:APPEND`   scopes on it |
| target_datasource | String | Name or the id of the target Data Source. |
| schedule_cron | String | Optional. A crontab expression. |

Successful response [¶](https://www.tinybird.co/docs/about:blank#id3) { "id": "t_3aa11a5cabd1482c905bc8dfc551a84d", "name": "my_copy_pipe", "description": "This is a pipe to copy", "type": "copy", "endpoint": null, "created_at": "2023-03-01 10:14:04.497505", "updated_at": "2023-03-01 10:34:19.113518", "parent": null, "copy_node": "t_33ec8ac3c3324a53822fded61a83dbbd", "copy_target_datasource": "t_0be6161a5b7b4f6180b10325643e0b7b", "copy_target_workspace": "5a70f2f5-9635-47bf-96a9-7b50362d4e2f", "nodes": [{ "id": "t_33ec8ac3c3324a53822fded61a83dbbd", "name": "emps", "sql": "SELECT * FROM employees WHERE starting_date > '2016-01-01 00:00:00'", "description": null, "materialized": null, "cluster": null, "mode": "append", "tags": { "copy_target_datasource": "t_0be6161a5b7b4f6180b10325643e0b7b", "copy_target_workspace": "5a70f2f5-9635-47bf-96a9-7b50362d4e2f" }, "created_at": "2023-03-01 10:14:04.497547", "updated_at": "2023-03-01 10:14:04.497547", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "dependencies": [ "employees" ], "params": [] }] } DELETE /v0/pipes/(.+)/nodes/(.+)/copy [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+)-nodes-(.+)-copy) Removes the Copy type of the pipe.
By removing the Copy type, nor the node nor the pipe are deleted.
The pipe will still be present, but will stop any scheduled and copy settings.

Unsetting the pipe as a Copy pipe [¶](https://www.tinybird.co/docs/about:blank#id4) curl -X DELETE \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/copy"| Code | Description |
| --- | --- |
| 204 | No error |
| 400 | Wrong node id |
| 403 | Forbidden. Provided token doesn’t have permissions to publish a pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found |

PUT /v0/pipes/(.+)/nodes/(.+)/copy [¶](https://www.tinybird.co/docs/about:blank#put--v0-pipes-(.+)-nodes-(.+)-copy) Calling this endpoint will update a Copy pipe with the given settings: you can change its target Data Source, as well as adding or modifying its schedule.

Updating a Copy Pipe [¶](https://www.tinybird.co/docs/about:blank#id6) curl -X PUT \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/copy" \ -d "target_datasource=other_destination_datasource" \ -d "schedule_cron=*/15 * * * *"| Key | Type | Description |
| --- | --- | --- |
| token | String | Auth token. Ensure it has the `PIPE:CREATE`   scope on it |
| target_datasource | String | Optional. Name or the id of the target Data Source. |
| schedule_cron | String | Optional. A crontab expression. If schedule_cron=’None’ the schedule will be removed from the copy pipe, if it was defined |

Successful response [¶](https://www.tinybird.co/docs/about:blank#id8) { "id": "t_3aa11a5cabd1482c905bc8dfc551a84d", "name": "my_copy_pipe", "description": "This is a pipe to copy", "type": "copy", "endpoint": null, "created_at": "2023-03-01 10:14:04.497505", "updated_at": "2023-03-01 10:34:19.113518", "parent": null, "copy_node": "t_33ec8ac3c3324a53822fded61a83dbbd", "copy_target_datasource": "t_2f046a4b2cc44137834a35420a533465", "copy_target_workspace": "5a70f2f5-9635-47bf-96a9-7b50362d4e2f", "nodes": [{ "id": "t_33ec8ac3c3324a53822fded61a83dbbd", "name": "emps", "sql": "SELECT * FROM employees WHERE starting_date > '2016-01-01 00:00:00'", "description": null, "materialized": null, "cluster": null, "mode": "append", "tags": { "copy_target_datasource": "t_2f046a4b2cc44137834a35420a533465", "copy_target_workspace": "5a70f2f5-9635-47bf-96a9-7b50362d4e2f" }, "created_at": "2023-03-01 10:14:04.497547", "updated_at": "2023-03-07 09:08:34.206123", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "dependencies": [ "employees" ], "params": [] }] } POST /v0/pipes/(.+)/copy [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-copy) Runs a copy job, using the settings previously set in the pipe.
You can use this URL to do an on-demand copy.
This URL is also used by the scheduler to make the programmed calls.

This URL accepts parameters, just like in a regular endpoint.

This operation is asynchronous and will copy the output of the endpoint to an existing datasource.

Runs a copy job on a Copy pipe [¶](https://www.tinybird.co/docs/about:blank#id9) curl -H "Authorization: Bearer " \ -X POST "https://api.tinybird.co/v0/pipes/:pipe/copy?param1=test¶m2=test2"| Key | Type | Description |
| --- | --- | --- |
| token | String | Auth token. Ensure it has the `PIPE:READ`   scope on it |
| parameters | String | Optional. The value of the parameters to run the Copy with. They are regular URL query parameters. |
| _mode | String | Optional. One of ‘append’ or ‘replace’. Default is ‘append’. |

| Code | Description |
| --- | --- |
| 200 | No error |
| 400 | Pipe is not a Copy pipe or there is a problem with the SQL query |
| 400 | The columns in the SQL query don’t match the columns in the target Data Source |
| 403 | Forbidden. The provided token doesn’t have permissions to append a node to the pipe ( `ADMIN`   or `PIPE:READ`   and `DATASOURCE:APPEND`   ) |
| 403 | Job limits exceeded. Tried to copy more than 100M rows, or there are too many active (working and waiting) Copy jobs. |
| 404 | Pipe not found, Node not found or Target Data Source not found |

The response will not be the final result of the copy but a Job. You can check the job status and progress using the [Jobs API](https://www.tinybird.co/docs/docs/api-reference/jobs-api).

Successful response [¶](https://www.tinybird.co/docs/about:blank#id12) { "id": "t_33ec8ac3c3324a53822fded61a83dbbd", "name": "emps", "sql": "SELECT * FROM employees WHERE starting_date > '2016-01-01 00:00:00'", "description": null, "materialized": null, "cluster": null, "tags": { "copy_target_datasource": "t_0be6161a5b7b4f6180b10325643e0b7b", "copy_target_workspace": "5a70f2f5-9635-47bf-96a9-7b50362d4e2f" }, "created_at": "2023-03-01 10:14:04.497547", "updated_at": "2023-03-01 10:14:04.497547", "version": 0, "project": null, "result": null, "ignore_sql_errors": false, "dependencies": [ "employees" ], "params": [], "job": { "kind": "copy", "id": "f0b2f107-0af8-4c28-a83b-53053cb45f0f", "job_id": "f0b2f107-0af8-4c28-a83b-53053cb45f0f", "status": "waiting", "created_at": "2023-03-01 10:41:07.398102", "updated_at": "2023-03-01 10:41:07.398128", "started_at": null, "is_cancellable": true, "datasource": { "id": "t_0be6161a5b7b4f6180b10325643e0b7b" }, "query_id": "19a8d613-b424-4afd-95f1-39cfbd87e827", "query_sql": "SELECT * FROM d_b0ca70.t_25f928e33bcb40bd8e8999e69cb02f94 AS employees WHERE starting_date > '2016-01-01 00:00:00'", "pipe_id": "t_3aa11a5cabd1482c905bc8dfc551a84d", "pipe_name": "copy_emp", "job_url": "https://api.tinybird.co/v0/jobs/f0b2f107-0af8-4c28-a83b-53053cb45f0f" } } POST /v0/pipes/(.+)/copy/pause [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-copy-pause) Pauses the scheduling. This affects any future scheduled Copy job. Any copy operation currently copying data will be completed.

Pauses a scheduled copy [¶](https://www.tinybird.co/docs/about:blank#id13) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/copy/pause"| Code | Description |
| --- | --- |
| 200 | Scheduled copy paused correctly |
| 400 | Pipe is not copy |
| 404 | Pipe not found, Scheduled copy for pipe not found |

POST /v0/pipes/(.+)/copy/resume [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-copy-resume) Resumes a previously paused scheduled copy.

Resumes a Scheduled copy [¶](https://www.tinybird.co/docs/about:blank#id15) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/copy/resume"| Code | Description |
| --- | --- |
| 200 | Scheduled copy resumed correctly |
| 400 | Pipe is not copy |
| 404 | Pipe not found, Scheduled copy for pipe not found |

POST /v0/pipes/(.+)/copy/cancel [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-copy-cancel) Cancels jobs that are working or waiting that are tied to the pipe and pauses the scheduling of copy jobs for this pipe.
To allow scheduled copy jobs to run for the pipe, the copy pipe must be resumed and the already cancelled jobs will not be resumed.

Cancels scheduled copy jobs tied to the pipe [¶](https://www.tinybird.co/docs/about:blank#id17) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/copy/cancel"| Code | Description |
| --- | --- |
| 200 | Scheduled copy pipe cancelled correctly |
| 400 | Pipe is not copy |
| 400 | Job is not in cancellable status |
| 400 | Job is already being cancelled |
| 404 | Pipe not found, Scheduled copy for pipe not found |

Successful response [¶](https://www.tinybird.co/docs/about:blank#id19) { "id": "t_fb56a87a520441189a5a6d61f8d968f4", "name": "scheduled_copy_pipe", "description": "none", "endpoint": "none", "created_at": "2023-06-09 10:54:21.847433", "updated_at": "2023-06-09 10:54:21.897854", "parent": "none", "type": "copy", "copy_node": "t_bb96e50cb1b94ffe9e598f870d88ad1b", "copy_target_datasource": "t_3f7e6534733f425fb1add6229ca8be4b", "copy_target_workspace": "8119d519-80b2-454a-a114-b092aea3b9b0", "schedule": { "timezone": "Etc/UTC", "cron": "0 * * * *", "status": "paused" }, "nodes": [ { "id": "t_bb96e50cb1b94ffe9e598f870d88ad1b", "name": "scheduled_copy_pipe_0", "sql": "SELECT * FROM landing_ds", "description": "none", "materialized": "none", "cluster": "none", "tags": { "copy_target_datasource": "t_3f7e6534733f425fb1add6229ca8be4b", "copy_target_workspace": "8119d519-80b2-454a-a114-b092aea3b9b0" }, "created_at": "2023-06-09 10:54:21.847451", "updated_at": "2023-06-09 10:54:21.847451", "version": 0, "project": "none", "result": "none", "ignore_sql_errors": "false", "node_type": "copy", "dependencies": [ "landing_ds" ], "params": [] } ], "cancelled_jobs": [ { "id": "ced3534f-8b5e-4fe0-8dcc-4369aa256b11", "kind": "copy", "status": "cancelled", "created_at": "2023-06-09 07:54:21.921446", "updated_at": "2023-06-09 10:54:22.043272", "job_url": "https://api.tinybird.co/v0/jobsjobs/ced3534f-8b5e-4fe0-8dcc-4369aa256b11", "is_cancellable": "false", "pipe_id": "t_fb56a87a520441189a5a6d61f8d968f4", "pipe_name": "pipe_test_scheduled_copy_pipe_cancel_multiple_jobs", "datasource": { "id": "t_3f7e6534733f425fb1add6229ca8be4b", "name": "target_ds_test_scheduled_copy_pipe_cancel_multiple_jobs" } }, { "id": "b507ded9-9862-43ae-8863-b6de17c3f914", "kind": "copy", "status": "cancelling", "created_at": "2023-06-09 07:54:21.903036", "updated_at": "2023-06-09 10:54:22.044837", "job_url": "https://api.tinybird.co/v0/jobsb507ded9-9862-43ae-8863-b6de17c3f914", "is_cancellable": "false", "pipe_id": "t_fb56a87a520441189a5a6d61f8d968f4", "pipe_name": "pipe_test_scheduled_copy_pipe_cancel_multiple_jobs", "datasource": { "id": "t_3f7e6534733f425fb1add6229ca8be4b", "name": "target_ds_test_scheduled_copy_pipe_cancel_multiple_jobs" } } ] }

---

URL: https://www.tinybird.co/docs/api-reference/pipe-api/api-endpoints
Last update: 2024-12-30T16:50:50.000Z
Content:
---
title: "Pipes API endpoints reference · Tinybird Docs"
theme-color: "#171612"
description: "The Pipes API helps you manage your Pipes. Use the API Endpoints service to publish or unpublish your Pipes as API Endpoints."
---

POST /v0/pipes/(.+)/nodes/(.+)/endpoint [¶](https://www.tinybird.co/docs/about:blank#post--v0-pipes-(.+)-nodes-(.+)-endpoint) Publishes an API endpoint

Publishing an endpoint [¶](https://www.tinybird.co/docs/about:blank#id1) curl -X POST \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/endpoint" Successful response [¶](https://www.tinybird.co/docs/about:blank#id2) { "id": "t_60d8f84ce5d349b28160013ce99758c7", "name": "my_pipe", "description": "this is my pipe description", "nodes": [{ "id": "t_bd1e095da943494d9410a812b24cea81", "name": "get_all", "sql": "SELECT * FROM my_datasource", "description": "This is a description for the **first** node", "materialized": null, "cluster": null, "dependencies": [ "my_datasource" ], "tags": {}, "created_at": "2019-09-03 19:56:03.704840", "updated_at": "2019-09-04 07:05:53.191437", "version": 0, "project": null, "result": null, "ignore_sql_errors": false }], "endpoint": "t_bd1e095da943494d9410a812b24cea81", "created_at": "2019-09-03 19:56:03.193446", "updated_at": "2019-09-10 07:18:39.797083", "parent": null } The response will contain a `token` if there’s a **unique READ token** for this pipe. You could use this token to share your endpoint.

| Code | Description |
| --- | --- |
| 200 | No error |
| 400 | Wrong node id |
| 403 | Forbidden. Provided token doesn’t have permissions to publish a pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found |

DELETE /v0/pipes/(.+)/nodes/(.+)/endpoint [¶](https://www.tinybird.co/docs/about:blank#delete--v0-pipes-(.+)-nodes-(.+)-endpoint) Unpublishes an API endpoint

Unpublishing an endpoint [¶](https://www.tinybird.co/docs/about:blank#id4) curl -X DELETE \ -H "Authorization: Bearer " \ "https://api.tinybird.co/v0/pipes/:pipe/nodes/:node/endpoint"| Code | Description |
| --- | --- |
| 200 | No error |
| 400 | Wrong node id |
| 403 | Forbidden. Provided token doesn’t have permissions to publish a pipe, it needs `ADMIN`   or `PIPE:CREATE` |
| 404 | Pipe not found |



---

URL: https://www.tinybird.co/docs/work-with-data/query/guides/working-with-time
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Working with time · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to work with time in Tinybird."
---




# Working with time [¶](https://www.tinybird.co/docs/about:blank#working-with-time)

## Overview [¶](https://www.tinybird.co/docs/about:blank#overview)

With real-time data, nothing is more important to understand than the impact of time, including time zones and Daylight Savings Time (DST).

This page explains the different functions for working with time in Tinybird when using, storing, filtering, and querying data. It also explains functions that may behave in ways you don't expect when coming in from other databases, and includes with example queries and some test data to play with.

Read Tinybird's ["10 Commandments for Working With Time" blog post](https://www.tinybird.co/blog-posts/database-timestamps-timezones) to understand best practices and top tips for working with time.

### Resources [¶](https://www.tinybird.co/docs/about:blank#resources)

If you want to follow along, you can find all the sample queries & data in the `timezone_analytics_guide` repo.

The repo contains several examples that this page walks through:



<-figure->
![Data flow showing different examples](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fworking-with-time-2.png&w=3840&q=75)


## Time-related types in Tinybird [¶](https://www.tinybird.co/docs/about:blank#time-related-types-in-tinybird)

In the companion Workspace, this section is in the Pipe `simple_types_and_transforms`.

There are basically two main Types, with several sub-types, when dealing with time in Tinybird - the first for **dates** , and the second for **specific time instants**.

For dates, you can choose one type or another depending on the range of dates you need to store.

For time instants, it will depend not only on the range, but also on the precision. The default records seconds, and you can also work with micro or nanoseconds.

**Date and time types**

| Type | Range of values | Parameters |
| --- | --- | --- |
| Date | [1970-01-01, 2149-06-06] |  |
| Date32 | [1900-01-01, 2299-12-31] |  |
| DateTime | [1970-01-01 00:00:00, 2106-02-07 06:28:15] | Time zone: More about that later. |
| DateTime64 | [1900-01-01 00:00:00, 2299-12-31 23:59:59.99999999] | Precision: [0-9]. Usually: 3(milliseconds), 6(microseconds), 9(nanoseconds). Time zone: More about that later. |

Note that this standard of appending `-32` or `-64` carries through to most other functions dealing with `Date` and `DateTime` , as you'll soon see.

## Transform your data into these types [¶](https://www.tinybird.co/docs/about:blank#transform-your-data-into-these-types)

### Date from a String [¶](https://www.tinybird.co/docs/about:blank#date-from-a-string)

Various ways of transforming a `String` into a `Date`:

SELECT
  '2023-04-05' AS date,
  toDate(date),
  toDate32(date),
  DATE(date),
  CAST(date, 'Date')| time | toDate(time) | toDate32(time) | DATE(time) | CAST(time, 'Date') |
| --- | --- | --- | --- | --- |
| 2023-04-05 | 2023-04-05 | 2023-04-05 | 2023-04-05 | 2023-04-05 |

### Store a date before 1970 [¶](https://www.tinybird.co/docs/about:blank#store-a-date-before-1970)

You can use the `toDate32` function, which has a range of [1900-01-01, 2299-12-31]. If you try to use the normal `toDate` function, you'll get the boundary of the available range - this might trip up users coming from other databases who would expect to get an Error.

select toDate('1903-04-15'), toDate32('1903-04-16')| toDate('1903-04-15') | toDate32('1903-04-16') |
| --- | --- |
| 1970-01-01 | 1903-04-16 |

### Parse other formats into DateTime [¶](https://www.tinybird.co/docs/about:blank#parse-other-formats-into-datetime)

You can use the `parseDateTimeBestEffortOrNull` function to parse a string into a `DateTime` , and if it fails for some reason, it will return a `NULL` value. This is your best default option, and there are several others which give different outcome behaviors, like `parseDateTimeBestEffort` (gives an error on a bad input), or `parseDateTimeBestEffortOrZero` (returns 0 on bad input).

Also remember to use the `-32` or `-64` suffixes for the functions that return a `DateTime` , depending on the range of dates you need to store.

Tinybird's Data Engineers know these to generally work with typical output strings experienced in the wild, such as those produced by most JavaScript libraries, Python logging, the default formats produced by AWS and GCP, and many others. But they can't be guaranteed to work, so test your data! And if something doesn't work as expected, contact Tinybird at [support@tinybird.co](https://www.tinybird.co/docs/mailto:support@tinybird.co) or in the [Community Slack](https://www.tinybird.co/docs/docs/community).

### How soon is Now()? [¶](https://www.tinybird.co/docs/about:blank#how-soon-is-now)

In the companion Workspace, this section is in a Pipe called `what_is_now`.

The function `now` gives a seconds-accuracy current timestamp. You can use the `now64` function to get precision down to nanoseconds if necessary.

They will default to the time zone of the server, which is UTC in Tinybird. You can use the `toDateTime` function to convert to a different time zone, or `toDate` to convert to a date.

Be mindful of using `toDate` and ensuring you are picking the calendar day relative to the time zone you are thinking about, otherwise you will end up with the UTC day instead by default. You might want to use the convenience functions `today` and `yesterday` instead.

SELECT 
now(),
toDate(now()),
toDateTime(now()),
toDateTime64(now(),3),
now64(),
toDateTime64(now64(),3),
toUnixTimestamp64Milli(toDateTime64(now64(),3)),
today(),
yesterday()| now() | toDate(now()) | toDateTime(now()) | toDateTime64(now(),3) | now64() | toDateTime64(now64(),3) | toUnixTimestamp64Milli(toDateTime64(now64(),3)) | today() | yesterday() |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 2021-03-19 19:15:00 | 2021-03-19 | 2021-03-19 19:15:00 | 2021-03-19 19:15:00.000 | 2021-03-19 19:15:00.478 | 2021-03-19 19:15:00.478 | 1616181300 | 2021-03-19 | 2021-03-18 |

## Filter by date in Tinybird [¶](https://www.tinybird.co/docs/about:blank#filter-by-date-in-tinybird)

In the companion Workspace, this section is in a Pipe called `filter_by_date`.

To filter by date in Tinybird, you need to make sure that you are comparing dates of the same type on both sides of the condition.

In the example Workspace, the timestamps are stored in a column named `timestamp_utc`:

2023-03-19 19:15:00
2023-03-19 19:17:23
2023-03-20 00:02:59 It is extremely common that you want to filter your data by a given day. It is quite simple, but there are a few gotchas to be aware of. Let's examine them step by step.

The first thing you could imagine is that Tinybird is clever enough to understand what you want to do if you pass a `String` containing a date. As you can see, this query:

SELECT
    timestamp_utc
FROM sales_data_raw
WHERE timestamp_utc = '2023-03-19' Produces no results. This actually makes sense when you realize you're comparing different types, a `DateTime` and a `String`.

What if you force your parameter to be a proper date? You can easily do it:

SELECT
    timestamp_utc
FROM sales_data_raw
WHERE timestamp_utc = toDate('2023-03-19') Nope. You may realize that you are comparing a `Date` and a `DateTime`.

Ok, so what if you convert the `Date` into a `DateTime`?

SELECT
    timestamp_utc
FROM sales_data_raw
WHERE timestamp_utc = toDateTime('2023-03-19') No results. This is because the `DateTime` produced has a time component of midnight, which is still just a single moment in time, not a range of time - which is exactly what a `Date` is - all the moments between the midnights on a given calendar day.

So what you really have to do is **make sure you're comparing the same kind of dates in both sides** . But it will be instructive to see the results of all the transformations so far:

SELECT
    timestamp_utc,
    toDate(timestamp_utc),
    toDate('2023-03-19'),
    toDateTime('2023-03-19')
FROM sales_data_raw
WHERE toDate(timestamp_utc) = toDate('2023-03-19') You can see that:

| timestamp_utc | toDate(timestamp_utc) | toDate('2023-03-19') | toDateTime('2023-03-19') |
| --- | --- | --- | --- |
| 2023-03-19 19:15:00 | 2023-03-19 | 2023-03-19 | 2023-03-19 00:00:00 |

In order to select the day you want, you need to convert both sides of the condition to `Date` . In this example, you do it in the `WHERE` clause.

Remember that, if filtering by date, you must have a date in both sides of the condition.

## Example of a good timestamp schema in Tinybird [¶](https://www.tinybird.co/docs/about:blank#example-of-a-good-timestamp-schema-in-tinybird)

Here's a few tips for the Types to use when storing timestamps in Tinybird:

1. It's a good idea to include the expected time zone, or any modifications like normalization, in the `timestamp`   column name. This will help you avoid confusion when you're working with data from different time zones, particularly when joining multiple data sources.
2. Your time zone names are low cardinality strings, so you can use the `LowCardinality(String)`   modifier to save space. Same goes for the offset, which is an `Int32`  .
3. You can use the `String`   type to store the local timestamp in a format that is easy to parse, but isn't going to be eagerly converted by Tinybird unexpectedly. The length of the string can vary by the precision you need, but it's a good idea to**  keep it consistent across all your Data Sources**   . There is also the `FixedString`   type, which is a fixed length string, but while it's more efficient to process it's also not universally supported by all Tinybird drivers, so `String`   is a better default.

Here's an example:

timestamp_utc [DateTime('UTC')], timezone [lowCardinalityString], offset [Int32], timestamp_local [String], 
## The details of Tinybird DateTime functions [¶](https://www.tinybird.co/docs/about:blank#the-details-of-tinybird-datetime-functions)

In the companion Workspace, this section is in a Pipe called `NittyGritty_DateTime_Operations`.

This uses more of the test data, specifically the recording of metadata about Store Hours in Tokyo, to see what happens with the various functions you would expect to use, and what the outcomes are.

### Things that work as expected [¶](https://www.tinybird.co/docs/about:blank#things-that-work-as-expected)

SELECT 
-- these are simple
timezone,
store_open_time_utc,
store_open_time_local, 
-- This next one will be converted to the Tinybird server's default time zone
parseDateTimeBestEffort(store_open_time_local) as parse_naive, 
-- This will be in the specified time zone, but it must be a Constant - you can't look it up!
parseDateTimeBestEffort(store_open_time_local, 'Asia/Tokyo') as parse_tz_lookup, 
-- this next one stringify's the DateTime so you can pick it in UTC without TZ conversion
parseDateTimeBestEffort(substring(store_open_time_local, 1, 19)) as parse_notz, 
-- These next ones work, but you must get the offsets right for each DST change!
store_open_time_utc + store_open_timezone_offset as w_plus,
date_add(SECOND, store_open_timezone_offset, store_open_time_utc) as w_date_add,
addSeconds(store_open_time_utc, store_open_timezone_offset) as w_add_seconds,
--- This gives you nice UTC formatting as you'd expect
formatDateTime(store_open_time_utc, '%FT%TZ'),
-- Because BestEffort will convert your string to UTC, you're going to get UTC displayed again here
formatDateTime(parseDateTimeBestEffort(store_open_time_local), '%FT%T')
FROM store_hours_raw
where store = 'Tokyo'
Limit 1| timezone | store_open_time_utc | store_open_time_local | parse_naive | parse_tz_lookup | w_plus | w_date_add | w_add_seconds | formatDateTime(store_open_time_utc, '%FT%TZ') | formatDateTime(parseDateTimeBestEffort(store_open_time_local), '%FT%T') |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Asia/Tokyo | 2023-03-20 00:00:00 | 2023-03-20T09:00:00+09:00 | 2023-03-20 00:00:00 | 2023-03-20 09:00:00 | 2023-03-20 09:00:00 | 2023-03-20 09:00:00 | 2023-03-20 09:00:00 | 2023-03-20T00:00:00Z | 2023-03-20T00:00:00 |

### Things that might surprise you [¶](https://www.tinybird.co/docs/about:blank#things-that-might-surprise-you)

Here you have a column with a time in UTC, and another column with the local time zone. Tinybird has an opinion that a `DateTime` column must always be one specific time zone.

It also has the opinion that you must always supply the time zone as a `String` , and can't lookup from another column on a per-row basis.

SELECT 
-- toString(store_open_time_utc, timezone) as w_tostr_lookup,   << This causes an error
-- toDateTime(store_open_time_utc, timezone),    << This also causes an error
  toString(store_open_time_utc, 'Asia/Tokyo') as w_tostr_const,  -- This is correct.
  toTimezone(store_open_time_utc, timezone) as w_totz_lookup,  -- This silently gives the wrong answer due to the bug!
  toDateTime(store_open_time_utc, 'Asia/Tokyo') as todt_const,  -- This is the best method
  toTimezone(store_open_time_utc, 'Asia/Tokyo') as w_totz_const,  -- this works, but toTimezone can fail if misused, so better to use toDateTime
-- Because BestEffort will convert your string to UTC, you might think you'll get the local time
-- But as the Tinybird column is UTC, that is what you're going to get
  formatDateTime(parseDateTimeBestEffort(store_open_time_local), '%FT%T'),
--   formatDateTime(store_open_time_utc, '%FT%T%z')
  toDate32('1903-04-15'),  -- This gives the expected date
  toDate('1903-04-15')  -- This silently hits the date boundary at 1970-01-01, so be careful
## Understand time zone handling [¶](https://www.tinybird.co/docs/about:blank#understand-time-zone-handling)

DateTimes always have a time zone. Here are the key points to note:

- A DateTime is stored as a Unix timestamp that represents the number of seconds in UTC since 1970-01-01.
- Tinybird stores a single time zone associated with a column and uses it to handle transformations during representation/export or other calculations.
- Tinybird enforces the single time zone per column rule in the behavior of native functions it provides for manipulating DateTime data.
- Tinybird assumes that any time zone offset or DST change is a multiple of 15 minutes. This is true for most modern time zones, but not for various historical corner cases.

Let's explore some areas to be aware of.

### How a time zone is selected for the Column [¶](https://www.tinybird.co/docs/about:blank#how-a-time-zone-is-selected-for-the-column)

Here are some general rules to observe when selecting a time zone for a column:

- If no time zone is specified, the column will represent the data in the Tinybird server's time zone, which is UTC in Tinybird.
- If a time zone is specified as a string, the column will represent the data as that time zone.
- If multiple time zones are in a query that produces a column without already creating the column with a time zone, the first time zone in the query wins (e.g. a CASE statement to pick different time zones).
- If the time zone isn't represented as a constant (e.g. by lookup to another column or table), you should get an error message.

### DateTime Operations without time zone conversion [¶](https://www.tinybird.co/docs/about:blank#datetime-operations-without-time-zone-conversion)

Tinybird provides some native operations that work with DateTime without handling time zone conversion. Mostly these are for adding or subtracting some unit of time. These operations behave exactly as you would expect.

Remember: You, as the programmer, are responsible for correctly selecting the amount to add or subtract for a given timestamp. Many incorrect datasets are produced here by incorrect chaining of time zone translations or handling DST incorrectly.

## How to Normalize your Data in Tinybird [¶](https://www.tinybird.co/docs/about:blank#how-to-normalize-your-data-in-tinybird)

In the companion Workspace, this section is in a Pipe called `sales_normalized`.

The process of converting timestamps on the data so that they are all in the same time zone is called "time zone normalization" or "time zone conversion". This is a common practice in data analytics to ensure that data from multiple sources, which may be in different time zones, can be accurately compared and analyzed. By converting all timestamps to a common time zone, it's possible to compare data points from different sources and derive meaningful insights from the data.

Consider a sample dataset containing sales data from multiple stores which are each in different time zones. Each store is open from 0900 to 1700 local time, and you're going to shift them all to UTC:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fworking-with-time-1.png&w=3840&q=75)


The simplest way to do this in Tinybird is to chain standard conversion functions - first, convert from your canonical stored UTC timestamp to the store-local time zone, then extract a `String` representation of that local time, then convert that back into a `DateTime` object. Tinybird will by default present this in the server time zone, so you have to know that this dataset has been normalized and name it appropriately.

In this example you are using a CASE statement to handle each store, and the result is presented as a Materialized View in Tinybird. This has the benefit of pre-calculating the time zone shifts, you could also pre-aggregate to some time interval like 5 or 15 mins, resulting in a highly efficient dataset for further analysis. In your schema you have a `String` identifying the `store` , an `Int32` for the `sale_amount` , and a `DateTime` in UTC for `timestamp_utc` of the transaction:

SELECT store, sale_amount,
  CASE 
    WHEN store = 'New York' THEN toDateTime(toString(toTimezone(timestamp_utc, 'America/New_York')))
    WHEN store = 'Tokyo' THEN toDateTime(toString(toTimezone(timestamp_utc, 'Asia/Tokyo')))
    WHEN store = 'London' THEN toDateTime(toString(toTimezone(timestamp_utc,'Europe/London')))
    WHEN store = 'Madrid' THEN toDateTime(toString(toTimezone(timestamp_utc, 'Europe/Madrid')))
    WHEN store = 'Auckland' THEN toDateTime(toString(toTimezone(timestamp_utc, 'Pacific/Auckland')))
    WHEN store = 'Chatham' THEN toDateTime(toString(toTimezone(timestamp_utc, 'Pacific/Chatham')))
    else toDateTime('1970-01-01 00:00:00')
  END AS timestamp_normalized
FROM sales_data_raw Note that the time zones used are all input as Strings, as Tinybird requires this. You use an else statement so that Tinybird doesn't mark the `timestamp_normalized` column as Nullable, which drastically impacts performance.

## Good time zone test data [¶](https://www.tinybird.co/docs/about:blank#good-time-zone-test-data)

Our users do *all sorts* of things with their data. Issues with time-based aggregations, particularly when time zone conversions are involved, are one of the most common gotchas.

The Tinybird Data Engineers amalgamated the these into a test data set, which is comprised of:

1. A facts table listing generated transactions for a set of retail stores across the world.
2. Some additional columns to aid in checking processing correctness.
3. A dimension table giving details about the store hours and time zones, again with additional information for correctness checking.

In the companion Workspace, these are the `sales_data_raw` and `store_hours_raw` tables respectively. The dataset is generated by [this Notebook](https://github.com/tinybirdco/timezone_analytics_guide/blob/main/sample_data_generator.ipynb) in the repo, and [pregenerated fixtures](https://github.com/tinybirdco/timezone_analytics_guide/tree/main/tinybird/datasources/fixtures) from it are also in the repo.

### The tricky use case [¶](https://www.tinybird.co/docs/about:blank#the-tricky-use-case)

There is also one extra store in what is deliberately the most complex case.

Let's say you have a store in the Chatham Islands (New Zealand), which typically has a +1345hr time zone. This store records the start of the new business day at 0900 each day instead of 0000 (midnight) due to some 'local regulations'. Let's also say that this store remains open 24hrs a day, and you are recording sales through a DST change.

The Chatham Islands DST changes back one hour at 0345 local time on April 2nd, so the April 2nd calendar day will have 25hrs elapsed, however due to the store business day changeover at 0900, this will actually occur during the April 1st store business day (open at 0900 Apr1, close at 0859 Apr2).

Confusing? Yes, exactly.

In order to more easily observe this behavior, our test data records a fixed price transaction at a fixed interval of every 144 seconds throughout the business day, so your tests can have patterns to observe. Why 144 seconds is a useful number here is left as an exercise for the reader.

### Why this is a good test scenario [¶](https://www.tinybird.co/docs/about:blank#why-this-is-a-good-test-scenario)

- When viewed in UTC, the DST change is at 2015hrs on April 1st, and the day has 25hrs elapsed, which may confuse an inexperienced developer.
- You also have a requirement that the business day windowing doesn't map to the calendar day (the 'local regulations' above), which mimics a typical unusual requirement in business analysis.
- You can also see that the offset changes from +1345 to +1245 between Apr1 and Apr2 - this is what trips up naive addSeconds time zone conversions twice a year, as they would only be correct for part of the day if not carefully bounded.
- In addition to this, the Chatham Islands time zone has a delightful offset in seconds which looks a lot like a typo when it changes from 49500 to 45900.
- It is also a place where almost nobody has a store as the local population is only 800 people, so great for something that can be filtered from real data.

This is what makes it an excellent time zone for test data.

### Test data schema [¶](https://www.tinybird.co/docs/about:blank#test-data-schema)

Take a look at the Schema, and why the various fields are useful. Note that you have used simple Types here to make it easier to understand, but you could also use the FixedString and other more complex types were described earlier in the use cases.

**Store Hours:**

`store` String `json:$.store` ,
`store_close_time_local` String `json:$.store_close_time_local` ,
`store_close_time_utc` DateTime `json:$.store_close_time_utc` ,
`store_close_timezone_offset` Int32 `json:$.store_close_timezone_offset` ,
`store_date` Date `json:$.store_date` ,
`store_open_seconds_elapsed` Int32 `json:$.store_open_seconds_elapsed` ,
`store_open_time_local` String `json:$.store_open_time_local` ,
`store_open_time_utc` DateTime `json:$.store_open_time_utc` ,
`store_open_timezone_offset` Int32 `json:$.store_open_timezone_offset` ,
`timezone` String `json:$.timezone` , **Sales Data:**

`store` String `json:$.store` ,
`sale_amount` Int32 `json:$.sale_amount` ,
`timestamp_local` String `json:$.timestamp_local` ,
`timestamp_utc` DateTime `json:$.timestamp_utc` ,
`timestamp_offset` Int32 `json:$.timestamp_offset`
### Schema notes [¶](https://www.tinybird.co/docs/about:blank#schema-notes)

The tables are joined on the `store` column, in this case a `String` for readability but likely to be a more efficient `LowCardinalityString` or an `Int32` in a real use case.

Local times are stored as Strings so that the underlying database is never tempted to change them to some other time representation. Having a static value to check your computations against is very useful in test data.

In the Sales Data, note that the `String` of the local timestamp in the facts table is kept. In a large dataset this would bloat the table and probably not be necessary. Strictly speaking, the offset is also unnecessary as you should be able to recalculate what it was given the timestamp in UTC and the exact time zone of the event producing service. Practically speaking however, if you are going to need this information a lot for your analysis, you are simply trading off later compute time against storage.

Note that a lot of metadata about the store business day in the Store Hours table is also kept - this helps to ensure that not only the analytic calculations are correct, but that the generated data is correct (see "Test data validation" below). This is a surprisingly common issue where test data is produced - it looks good to the known edge cases, but doesn't end up correct to the unknown edge cases. It's in exactly these scenarios that you want to know more about how it was produced, so you can fix it.

### Test data validation [¶](https://www.tinybird.co/docs/about:blank#test-data-validation)

In the [Data Generator Notebook](https://github.com/tinybirdco/timezone_analytics_guide/blob/main/sample_data_generator.ipynb) , you can also inspect the generated data to ensure that it conforms to the changes expected over the difficult time window of 1st April to 3rd April in the Chatham Islands.

Review the following output and consider what you've read so far about the changes over these few days, and then take a look at the timestamps and calculations to see how each plays out:

Day: 2023-04-01
opening time UTC: 2023-03-31T19:15:00+00:00
opening time local: 2023-04-01T09:00:00+13:45
timezone offset at store open: 49500
total hours elapsed during store business day: 25
total count of sales during business day: 625 and sales amount in cents: 31250
total hours elapsed during calendar day: 24
total count of sales during calendar day: 600 and sales amount in cents: 30000
closing time UTC: 2023-04-01T20:14:59+00:00
closing time local: 2023-04-02T08:59:59+12:45
timezone offset at store close: 45900

Day: 2023-04-02
opening time UTC: 2023-04-01T20:15:00+00:00
opening time local: 2023-04-02T09:00:00+12:45
timezone offset at store open: 45900
total hours elapsed during store business day: 24
total count of sales during business day: 600 and sales amount in cents: 30000
total hours elapsed during calendar day: 25
total count of sales during calendar day: 625 and sales amount in cents: 31250
closing time UTC: 2023-04-02T20:14:59+00:00
closing time local: 2023-04-03T08:59:59+12:45
timezone offset at store close: 45900

Day: 2023-04-03
opening time UTC: 2023-04-02T20:15:00+00:00
opening time local: 2023-04-03T09:00:00+12:45
timezone offset at store open: 45900
total hours elapsed during store business day: 24
total count of sales during business day: 600 and sales amount in cents: 30000
total hours elapsed during calendar day: 24
total count of sales during calendar day: 600 and sales amount in cents: 30000
closing time UTC: 2023-04-03T20:14:59+00:00
closing time local: 2023-04-04T08:59:59+12:45
timezone offset at store close: 45900 This test data generator has been through several iterations to capture the specifics of various scenarios that Tinybird customers have raised; hopefully you will also find it helpful.

## Query correctly with time zones and DST [¶](https://www.tinybird.co/docs/about:blank#query-correctly-with-time-zones-and-dst)

Now that you've have looked at the test data, let's work through understanding some examples of the correct way to query it, and where it can go wrong.

In the companion Workspace, this section is in the Pipe `Aggregate_Querying`.

### Naively querying by UTC [¶](https://www.tinybird.co/docs/about:blank#naively-querying-by-utc)

If you naively query by UTC here, you get an answer that looks correct - sales start at 9am and stop at 5pm when London is on winter time, which matches UTC.

SELECT
  store,
  toStartOfFifteenMinutes(timestamp_utc) AS period_utc,
  sum(sale_amount) AS sale_amount,
  count() as sale_count
FROM sales_data_raw
where store = 'London' AND toDate(period_utc) = toDate('2023-03-24')
GROUP BY
    store,
    period_utc
ORDER BY period_utc ASC| store | period_utc | sale_amount | sale_count |
| --- | --- | --- | --- |
| London | 2023-03-24 09:00:00 | 88418 | 7 |
| London | 2023-03-24 09:15:00 | 29159 | 3 |
| London | 2023-03-24 09:30:00 | 35509 | 3 |

The essential mistake here is blindly converting the UTC timestamp in `period_utc` to a date, however as the time zone aligns with UTC, it has no impact on these results.

### Naively querying a split time zone [¶](https://www.tinybird.co/docs/about:blank#naively-querying-a-split-time-zone)

However if we naively query Auckland applying a calendar day to the UTC periods, we get two split blocks at the start and end of day.

This is because you're actually getting the end of the day you want, and the start of the next day, because of the ~+13 UTC offset.

select * from sales_15min_utc_mv
where store = 'Auckland' AND toDate(period_utc) = toDate('2023-03-24')
ORDER BY period_utc ASC| store | period_utc | sale_amount | sale_count |
| --- | --- | --- | --- |
| Auckland | 2023-03-24 00:00:00 | 71538 | 6 |
| Auckland | 2023-03-24 00:15:00 | 51795 | 5 |
| Auckland | 2023-03-24 00:30:00 | 67106 | 7 |

The same mistake is made as above, with the blind conversion of the UTC timestamp to a date, however in this case it's more obvious that it's wrong as very few time zones map 0900 local exactly to midnight.

Try the same query again with the Tokyo store, and see how the mistake is easily hidden.

### Aggregating by timestamp over a period [¶](https://www.tinybird.co/docs/about:blank#aggregating-by-timestamp-over-a-period)

A convenience of modern time zone and DST definitions is that they can all be expressed in 15 minute increments (in fact, underlying Tinybird assumes this is true to drive processing efficiency), so **if you aren't sure what aggregation period is best for you, start with 15mins and see how your data set grows** , as you can query any larger period as groups of 15min periods.

To remind you again of the test data: You have a table containing sales data for a set of stores. Your data has columns for a `store` , a `timestamp` in UTC, and a `sale_amount` . In Tinybird, you can quickly and efficiently pre-calculate the 15min periods as a Materialized View with the following query:

SELECT
    store,
    toStartOfFifteenMinutes(timestamp_utc) AS period_utc,
    sumSimpleState(sale_amount) AS sale_amount,
    countState() AS sale_count
FROM sales_data_raw
GROUP BY
    store,
    period_utc
ORDER BY period_utc ASC Using this, you can then produce any larger period for a given window of that period. See the next example for how to break it into local days.

### Querying by day and time zone [¶](https://www.tinybird.co/docs/about:blank#querying-by-day-and-time-zone)

In the example Workspace, see the [sales_15min_utc_mv Materialized View](https://github.com/tinybirdco/timezone_analytics_guide/blob/main/tinybird/datasources/sales_15min_utc_mv.datasource).

Building on the previous example, this query produces a correct aggregate by the local time zone day for a given `store`, `timezone`, `start_day` and `end_day` (inclusive). We also feature use of Tinybird parameters, such as you would use in an API Endpoint.

Note that you're querying the tricky Chatham store here, and over the days with the DST change, so you can see the pattern of transactions emerge.

This example uses Tinybird parameters for the likely user-submitted values of `timezone`, `start_day` and `end_day` , and you are selecting the data from the `sales_15min_utc_mv` Materialized View described above.

%
Select
  store,
  toDate(toTimezone(period_utc, {{String(timezone, 'Pacific/Chatham')}})) as period_local,
  sum(sale_amount) as sale_amount,
  countMerge(sale_count) as sale_count
from sales_15min_utc_mv
where
  store = {{String(store, 'Chatham')}}
  and period_local >= toDate({{String(start_day, '20230331')}})
  and period_local <= toDate({{String(end_day, '20230403')}})
group by store, period_local
order by period_local asc| store | period_local | sale_amount | sale_count |
| --- | --- | --- | --- |
| Chatham | 2023-03-31 | 30000 | 600 |
| Chatham | 2023-04-01 | 30000 | 600 |
| Chatham | 2023-04-02 | 31250 | 625 |
| Chatham | 2023-04-03 | 30000 | 600 |

Note that the `DateTime` has been converted to the target time zone **before** reducing to the local calendar day using `toDate` , and that the user input is converted to the same `Date` type for comparison, and the column has been renamed to reflect this change.

Also note that the time zone parameter is a `String` , which is required by Tinybird. It's up to you to ensure you supply the right time zone string and Store filters here.

Because the Data Source in this case is a Materialized View, you should also be careful to use the correct `CountMerge` function to get the final results of the incremental field.

### Validating the results [¶](https://www.tinybird.co/docs/about:blank#validating-the-results)

You can validate the results by comparing them to the same query over the raw data, and see that the results are identical.

select 
  store,
  toDate(substring(timestamp_local, 1, 10)) as period_local,
  sum(sale_amount) as sale_amount,
  count() as sale_count
from sales_data_raw
where 
  store = 'Chatham'
  and toDate(substring(timestamp_local, 1, 10)) >= toDate('20230331')
  and toDate(substring(timestamp_local, 1, 10)) <= toDate('20230403')
group by store, period_local
order by period_local asc| store | period_local | sale_amount | sale_count |
| --- | --- | --- | --- |
| Chatham | 2023-03-31 | 30000 | 600 |
| Chatham | 2023-04-01 | 30000 | 600 |
| Chatham | 2023-04-02 | 31250 | 625 |
| Chatham | 2023-04-03 | 30000 | 600 |

### Querying with a lookup table [¶](https://www.tinybird.co/docs/about:blank#querying-with-a-lookup-table)

In our test data, the store day doesn't match the local calendar day - if you recall it was specified that it starts at 0900 local time. Fortunately, we can use a lookup table to map the store day to our UTC timestamps, and then use that to query the data. This is often a good idea if you have a lot of data, as it can be more efficient than using a function to calculate the mapping each time.

select 
  store,
  store_date,
  sum(sale_amount),
  countMerge(sale_count) as sale_count 
from sales_15min_utc_mv
join store_hours_raw using store
where period_utc >= store_hours_raw.store_open_time_utc
and period_utc < store_hours_raw.store_close_time_utc
and store_date >= toDate('2023-03-31') and store_date <= toDate('2023-04-03')
and store = 'Chatham'
group by store, store_date
order by store_date asc| store | period_local | sale_amount | sale_count |
| --- | --- | --- | --- |
| Chatham | 2023-03-31 | 30000 | 600 |
| Chatham | 2023-04-01 | 31250 | 625 |
| Chatham | 2023-04-02 | 30000 | 600 |
| Chatham | 2023-04-03 | 30000 | 600 |

Note that the extra hour of business moves to the 1st of April, as that store day doesn't close until 0859 on the 2nd of April, which is after the DST change.

## Use Materialized Views to pre-calculate the data [¶](https://www.tinybird.co/docs/about:blank#use-materialized-views-to-pre-calculate-the-data)

Several of these examples have pointed to a [Materialized View](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) , which is a pre-calculated table that is updated incrementally as new data arrives.

These are also prepared for you in the companion Workspace, specifically for the UTC 15min rollup, and the normalized 15min rollup.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read Tinybird's[  "10 Commandments for Working With Time" blog post](https://www.tinybird.co/blog-posts/database-timestamps-timezones)   to understand best practices and top tips for working with time.
- Understand[  Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views)  .
- Learn how to dynamically aggregate time series data by[  different time intervals (rollups)](https://www.tinybird.co/docs/docs/work-with-data/query/guides/dynamic-aggregation)  .
- Got a tricky use case that you want help with? Connect with us in the[  Tinybird Slack community](https://www.tinybird.co/docs/docs/community)  .



---

URL: https://www.tinybird.co/docs/work-with-data/query/guides/lambda-example-cdc
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Change Data Capture project with lambda architecture · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn a useful implementation to consume CDC streams from Kafka and get an updated view of the changes."
---




# Lambda CDC processing with Tinybird [¶](https://www.tinybird.co/docs/about:blank#lambda-cdc-processing-with-tinybird)

This guide outlines a practical implementation of CDC processing with Tinybird using a [lambda](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-architecture) approach. It produces an API that returns the freshest deduplicated view of the data by combining a scheduled batch job with new rows since the last batch.

This is more complex than a simple deduplication query or Materialized View, recommended as an optimization where the dataset or processing SLAs demand it.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This is a read-through guide, explaining an example, so you don't need an active Workspace to try it out in. Use the concepts and apply them to your own scenario.

To understand the guide, you'll need familiarity with Change Data Capture concepts and Tinybird [deduplication strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies).

## Data characteristics [¶](https://www.tinybird.co/docs/about:blank#data-characteristics)

This implementation focuses on fast filtering and aggregation of slowly-changing dimensions over a long history with high cardinality.

The test dataset is a Postgres Debezium CDC to Kafka with an event history of tens of millions of updates into ~5M active records, receiving up to 75k new events per hour. Tinybird provides low-latency, high-concurrency responses with real-time data freshness. In this example, CDC Source is configured as *partial* mode, i.e. only new and changed records are sent as data and deletes as a null. In *full* CDC you would get the old and new data in each change, which can be very helpful in OLAP processing.

The test dataset exhibits high cardinality over many years, optimized for ElasticSearch with nested JSON arrays. Updates are sparse over data dimensions and time, leading to specific decisions in this implementation. It is also worth noting that the JSON is up to 100kB per document, but for the analysis only a small part is needed.

Any given primary key of the upstream data source can be deleted or the Kafka topic compacted or reloaded, resulting in many 'null' records to handle.

## Solution features [¶](https://www.tinybird.co/docs/about:blank#solution-features)

- Lambda processing architecture
- Split data + deletes table processing
- Null events as deletes by Kafka partition key
- Batch + speed layer CDC upsert
- Full data history table
- Full delete history table
- Batch table with good sorting keys
- Latest data as reusable API

## Solution technical commentary [¶](https://www.tinybird.co/docs/about:blank#solution-technical-commentary)

This implementation doesn't use `AggregatingMergeTree` or `ReplacingMergeTree` due to sorting key limitations. Instead, it uses a `MergeTree` table with subquery deduplication. The data history and delete tables are split to avoid bloat and null processing, improving performance.

It focuses on the Kafka event timestamp and partition key for deduplication. Various Tinybird functions are used for JSON extraction, avoiding nulls to speed up processing.

## Data pipeline lineage [¶](https://www.tinybird.co/docs/about:blank#data-pipeline-lineage)

1. **  Raw Kafka Table ``**
2. **  Initial Materialized Views**
  - Data History extraction ``
  - Delete History extraction ``
3. **  Historical Data Sources**
  - All insert/update events ``
  - All delete events ``
4. **  Batching Copy Pipe ``**
5. **  Batches Data Source ``**
6. **  Lambda 'Upsert' API ``**



<-figure->
![An overview of the Data Flow, with a Kafka Data source, two Materialized Views to keep track of changes and deletes, a Copy Pipe to deduplicate in batches, and a Pipe to combine all Data Sources](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-cdc-overview.png&w=3840&q=75)

<-figcaption->
Lambda CDC overview





## Landing Data Source [¶](https://www.tinybird.co/docs/about:blank#landing-data-source)

Where the CDC events from the Kafka topic are consumed:

##### raw_kafka_ds.datasource

SCHEMA >
    `__value` String
     --`__topic` LowCardinality(String),
     --`__partition` Int16,
     --`__offset` Int64,
     --`__timestamp` DateTime,
     --`__key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(__timestamp)"
ENGINE_SORTING_KEY "__offset"

KAFKA_CONNECTION_NAME ''
KAFKA_TOPIC ''
KAFKA_GROUP_ID ''
KAFKA_AUTO_OFFSET_RESET 'earliest'
KAFKA_STORE_RAW_VALUE 'True'
KAFKA_STORE_HEADERS 'True'
KAFKA_STORE_BINARY_HEADERS 'True'
KAFKA_TARGET_PARTITIONS 'auto'
KAFKA_KEY_AVRO_DESERIALIZATION '' Helpful notes and top tips for your own implementation:

- The other Kafka metadata fields (commented above) like `__timestamp`   etc. are automatically added by Tinybird's Kafka connector.
- Always increment the `KAFKA_GROUP_ID`   if you reprocess the topic!
- The `__value`   may be `null`   in the case of a `DELETE`   for many CDC setups, so don't parse the JSON values in the raw table.
- You can get the operation ( `INSERT`  , `UPDATE`  , `DELETE`   ...) from the `KAFKA_STORE_HEADERS`   for many CDC sources and read them in the `__headers`   field, though you don't need it for this implementation as `INSERT`   and `UPDATE`   are equivalent for our purposes, and `DELETE`   is always the `null`   record.
- The sorting key should definitely be `__offset`   or `__partition`   . CDC Data can often have high density bursts of activity, which results in a lot of changes being written in a short time window. For this reason it's often better to partition and sort `raw_kafka_ds`   data by `__partition`   and /or `__offset`   to avoid the skew of using `__timestamp`  .
- Remember that you have to pair the `__key`   with the `__offset`   to get a unique pairing, as each partition has its own offsets. This is why `__timestamp`   is a good boundary for multi-partition topics.
- This implementation doesn't set a TTL as it only partially processes the `__value`   schema for the given use case. If you want to create other tables out of it you'd need the data source. This also allows reprocessing if you decide you need something else out of the raw JSON.
- You could optionally run a delete operation on every `__offset`   before the `__offset`   of the first `__value`   that isn't a `null`   in each Partition, which would effectively truncate the table of all old compactions. This can be done in the CLI with a `tb datasource truncate `   command. Remember to[  filter by partitions with no active ingest](https://www.tinybird.co/docs/docs/get-data-in/data-operations/replace-and-delete-data)  .

## Full data history [¶](https://www.tinybird.co/docs/about:blank#full-data-history)

This section contains all the `INSERT` and `UPDATE` operations.

To generate the Materialized View you'd first need a Pipe that will result in the `data_history_mv` Data Source:

##### mat_data.pipe

NODE mv
SQL >

   SELECT
      toInt64(__key) as k_key,
      __offset as k_offset,
      __timestamp as k_timestamp,
      __partition as k_partition,
      __value as k_value,
      
   FROM raw_kafka_ds
   WHERE __value != 'null'

TYPE materialized
DATASOURCE data_history_mv Notes:

- This example treats `__key`   and `__timestamp`   as the primary concern, and then parses out all the various fields the customer wants.
- You want this table to have the same extracted columns as the batch table, as the Lambda process UNIONs them.
- Use the various Tinybird functions for JSON extraction, and avoid Nullable fields as it slows processing and bloats tables.

Here is the Data Source definition of the MV:

##### data_history_mv.datasource

SCHEMA >

   `k_key` Int64,
   `k_offset` Int64,
   `k_timestamp` DateTime,
   `k_partition` Int16,
   `k_value` String
   

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(k_timestamp)"
ENGINE_SORTING_KEY "k_key, k_timestamp" Helpful notes:

- `__timestamp`   and `__key`   are critical for lambda processing, so while they aren't low cardinality they are the permanent filters in later queries.
- Because this example mostly cares about quick access to the most recent events, it uses `k_timestamp`   for partitioning.
- This example keeps the raw JSON of the rest of the record in `k_value`   against some field being wanted for later indexing. You can ignore the column in daily processing, and use it for backfill without reprocessing the entire raw topic again if you need to later. You can obviously partially extract from this for stable column indexing.
- Additional sorting keys for customer queries aren't retained because you need offset and key here, however other approaches could be considered if necessary.
- All other columns are based on fields extracted from the kafka `__value`   JSON.
- If the sorting key columns for customer queries were limited to columns that did not change during a CDC update, then a `ReplacingMergeTree`   may work here. However customer updates are often over required columns including date fields making it impractical.

## Deletes history [¶](https://www.tinybird.co/docs/about:blank#deletes-history)

This section contains the `DELETE` operations.

As above, you need a Pipe to get the deletes and materialize them:

##### mat_deletes.datasource

NODE mv
SQL >

    SELECT
      toInt64(__key) as k_key,  -- key used as deduplication identity
      __timestamp as k_timestamp -- ts used for deduplication incrementor
    FROM raw_kafka_ds
    WHERE __value = 'null'

TYPE materialized
DATASOURCE deletes_history_mv Helpful notes:

- Nothing fancy - just parses out the null records by tracking the `__key`  .
- Note converting to Int64 instead of String for better performance, as you know the key and offset are auto-incrementing integers from Postgres and Kafka respectively. This may not always be true for other CDC sources.

It results into a table with all the deletes:

##### deletes_history_mv.datasource

SCHEMA >
    `k_key` Int64,
    `k_timestamp` DateTime

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(k_timestamp)"
ENGINE_SORTING_KEY "k_timestamp, k_key" Notes:

- Sorting key order is deliberate. You want to be able to fetch only the `__timestamp`   s since the last batch was run for lambda processing, and then you want the `__key`   s that are to be deleted from the dataset as a list for delete processing.
- You don't need to have a separate deletes table for an efficient implementation, however if you have a lot of deletes (such as in compacted and reloaded Kafka Topics for CDC) then you may find this more efficient at the cost of a little more implementation complexity. A recommended approach is to start with a single history table including both data and deletes and then optimize later if necessary.

## Consolidation batches [¶](https://www.tinybird.co/docs/about:blank#consolidation-batches)

Using Copy Pipes, you can generate snapshots of the state of the original table at several points in time. This will help you speed up compute later, only having to account for changes that arrived since last snapshot/batch happened.

##### copy_batches.pipe

NODE get_ts_boundary
SQL >

    WITH (
      (SELECT max(k_timestamp) FROM data_history_mv) AS max_history,
      (SELECT max(k_timestamp) FROM delete_history_mv) AS max_deletes
    ) 
    SELECT least(max_history, max_deletes, now()) as batch_ts_

NODE dedupe_and_delete_history
SQL >

   WITH 
   latest_rows AS (
      SELECT k_key, max(k_timestamp) AS latest_ts
      FROM 
      GROUP BY k_key
   ),
   ts_boundary AS (
      SELECT batch_ts_ AS batch_ts
      FROM get_ts_boundary
   )
   SELECT f.*, assumeNotNull((SELECT batch_ts FROM ts_boundary)) AS batch_ts
   FROM data_history_mv f
   INNER JOIN latest_rows lo
   ON f.k_key = lo.k_key AND f.k_timestamp = lo.latest_ts
   WHERE f.k_key NOT IN (
      SELECT k_key 
      FROM deletes_history_mv 
      WHERE k_timestamp <= (SELECT batch_ts FROM ts_boundary)
   )
   AND f.k_timestamp <= (SELECT batch_ts FROM ts_boundary)
   

TYPE copy
TARGET_DATASOURCE batches_ds
COPY_SCHEDULE 0,30 * * * * Notes:

- You use the slowest data processing stream as the batch timestamp boundary, in case one of the Kafka partitions is lagging and other typical stream processing challenges.
- This example uses a subquery and self-join to deduplicate because testing showed it as performing the best over the dataset used. Each dataset will have unique characteristics that may drive a different approach such as `LIMIT 1 BY`   etc.
- Note the deduplication method works fine in batch, but is the same as the lambda as well.
- The schedule should be adjusted to match the customer cadence requirements.
- Note that this example uses `<=`   in the row selection here, and `>`   in the selection later to ensure it doesn't duplicate the boundary row.

##### batches_ds.datasource

SCHEMA >
   `k_key` Int64,
   `k_offset` Int64,
   `k_timestamp` DateTime,
   `k_partition` Int16,
   `k_value` String,
   `batch_ts` DateTime
   

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toDate(batch_ts)"
ENGINE_SORTING_KEY "batch_ids, "
ENGINE_TTL "batch_ts + toIntervalDay(1)" Notes:

- You can't use a TTL to simply keep the last 3 versions, so you must pick a date and monitor that batches are running as expected (Tinybird won't consider the whole table in a TTL query, just that row).
- This Table forms the bulk of the rows used for the actual query process, so it's important that the sorting keys are optimized for data results.
- The `batch_id`   remains at the head of the sorting key so you can quickly select the latest batch for use.
- The `batch_id`   is a simple timestamp boundary of all rows across all Partitions included in the batch, including all Deletes already applied to the batch. This is important when understanding the logic of the Lambda processing later.
- Partition key is by day on the `batch_ts`   so you can read as few rows as possible, but all sequentially.
- Analysis of the customer sorting keys may yield good optimization information, such as a need for controlling index granularity if they handle a lot of multi-tenant data, for example.

## Lambda Pipe [¶](https://www.tinybird.co/docs/about:blank#lambda-pipe)

Lastly, you get the latest snapshot plus all the changes since then, and consolidate. This API Endpoint can also be used as a "view" so that other Pipes query it.

##### latest_values.pipe

NODE get_batch_info
SQL >
    SELECT max(batch_ts) AS batch_ts 
    FROM batches_ds

NODE new_deletes
SQL >

    select k_key 
    FROM deletes_history_mv
    WHERE k_timestamp > (SELECT batch_ts from get_batch_info)  -- only rows since last batch

NODE filter_new_rows
SQL >

    %
    SELECT {{columns(cols, 'k_key, k_offset, k_timestamp, ')}}
    FROM data_history_mv
    WHERE 1 
    AND k_timestamp > (SELECT batch_ts from get_batch_info)  -- only rows since last batch
    AND k_key not in (select k_key from new_deletes)  -- remove newly deleted rows from new rows
    AND 

NODE dedup_new_rows_by_subquery
SQL >

    WITH latest_rows AS (
        SELECT k_key, max(k_timestamp) AS latest_ts
        FROM filter_new_rows
        GROUP BY k_key
      )
    SELECT f.*
    FROM filter_new_rows f
    INNER JOIN latest_rows lo
    ON f.k_key = lo.k_key AND f.k_timestamp = lo.latest_ts

NODE get_and_filter_batch
SQL >

    %
    SELECT {{columns(cols, 'k_key, k_offset, k_timestamp, ')}}
    FROM batches_ds
    PREWHERE batch_ts = (SELECT batch_ts from get_batch_info)  -- get latest batch
    WHERE 1
    AND k_key not in (select k_key from new_deletes)  --  filter by new deletes since last batch
    AND k_key not in (select k_key from dedup_new_rows_by_subquery)  -- omit rows already updated since batch


NODE batch_and_latest
SQL >

    SELECT * FROM get_and_filter_batch
    UNION ALL
    SELECT * FROM dedup_new_rows_by_subquery Notes:

- This is a longer Pipe which is published as an API.
- It starts by determining which batch to use, which also gives you the boundary timestamp. It then fetches and processes all new rows since the selected batch, including deletes processing and deduplication. It then backfills this with all other rows from the batch, and UNIONs the results.
- It uses the same deduplication strategy as the batch processing Pipe for consistency of results.
- Note the use of the `columns`   Parameter. This then defaults to returning all columns, but a user can specify a subset to reduce the data fetched and processed.
- This API can then be called as a Data Source by other Pipes, which can also use the same parameter names to pass through filters like columns or other customer filters that may be required.

## Conclusion [¶](https://www.tinybird.co/docs/about:blank#conclusion)

This image explains, in detail, the full overview of this approach: A Kafka Data Source, two Materialized Views to keep track of changes and deletes, a Copy Pipe to deduplicate in batches, and a Pipe to combine all Data Sources.



<-figure->
![An overview of the data flow, with a Kafka Data Source, two Materialized Views to keep track of changes and deletes, a Copy Pipe to deduplicate in batches, and a Pipe to combine all Data Sources](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-cdc-detail.png&w=3840&q=75)

<-figcaption->
Lambda CDC overview





## Possible improvements [¶](https://www.tinybird.co/docs/about:blank#possible-improvements)

This example deliberately kept the full history, but you could speed up and store less data if the history Materialized Views are `ReplacingMergeTree` , or if you add a TTL long enough to be sure the changes have been incorporated to the batch Data Source.

## Alternatives [¶](https://www.tinybird.co/docs/about:blank#alternatives)

### A simpler way of achieving the latest view [¶](https://www.tinybird.co/docs/about:blank#a-simpler-way-of-achieving-the-latest-view)

This example is solving for many peculiarities of the test dataset, like not having a simple key for deduplication, and a large number of delete operations bloating the resulting tables. As a comparison, here's a solution you can use when your CDC case is very simple.

It's possibly less formant (since you need to extract fields at query time, filter deletes over the whole data source...) but easier and perfectly functional if volumes aren't too big. Just define the Kafka Data Source as a `ReplacingMergeTree` (or create a MV from raw), query with FINAL, and exclude deletes:

##### raw_kafka_ds_rmt.datasource

SCHEMA >
    `__value` String
     --`__topic` LowCardinality(String),
     --`__partition` Int16,
     --`__offset` Int64,
     --`__timestamp` DateTime,
     --`__key` String

ENGINE "ReplacingMergeTree"
ENGINE_PARTITION_KEY ""
ENGINE_SORTING_KEY "__key"
ENGINE_VER "__offset" --or "__timestamp" An example query to consolidate latest updates and exclude duplicates:

##### raw_kafka_ds_rmt.datasource

SELECT
   *
   , JSONExtract(__value, 'field', type) as field --this for every field
FROM raw_kafka_ds_rmt FINAL
WHERE __value!='null'
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read more about using a[  lambda approach](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-architecture)   in your Tinybird architecture.
- Understand Tinybird[  deduplication strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies)  .



---

URL: https://www.tinybird.co/docs/work-with-data/query/guides/lambda-architecture
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Build a lambda architecture in Tinybird · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit."
---




# Build a lambda architecture in Tinybird [¶](https://www.tinybird.co/docs/about:blank#build-a-lambda-architecture-in-tinybird)

In this guide, you'll learn a useful alternative processing pattern for when the typical Tinybird flow doesn't fit.

This page introduces a useful data processing pattern for when the typical Tinybird flow (Data Source --> incremental transformation through Materialized Views --> and API Endpoint publication) doesn't fit. Sometimes, the way Materialized Views work means you need to use **Copy Pipes** to create the intermediate Data Sources that will keep your API Endpoints performant.

## The ideal Tinybird flow [¶](https://www.tinybird.co/docs/about:blank#the-ideal-tinybird-flow)

You ingest data (usually streamed in, but can also be in batch), transform it using SQL, and serve the results of the queries via [parameterizable](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) API Endpoints. Tinybird provides freshness, low latency, and high concurrency: Your data is ready to be queried as soon as it arrives.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-1.png&w=3840&q=75)

<-figcaption->
Data flow with Data Source and API Endpoint





Sometimes, transforming the data at query time isn't ideal. Some operations - doing aggregations, or extracting fields from JSON - are better if done at ingest time, then you can query that prepared data. [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) are perfect for this kind of situation. They're triggered at ingest time and create intermediate tables (Data Sources in Tinybird lingo) to keep your API Endpoints performance super efficient.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-2.png&w=3840&q=75)

<-figcaption->
Data flow with Data Source, MV, and API Endpoint





The best practice for this approach is usually having a Materialized View (MV) per use case:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-3.png&w=3840&q=75)

<-figcaption->
Materialized Views for different use cases





If your use case fits in these first two paragraphs, stop reading. No need to over-engineer it.

## When the ideal flow isn't enough [¶](https://www.tinybird.co/docs/about:blank#when-the-ideal-flow-isnt-enough)

There are some cases where you may need intermediate Data Sources (tables) and Materialized Views don't fit.

- Most common: Things like Window Functions where you need to check the whole table to make calculations.
- Fairly common: Needing an Aggregation MV over a deduplication table (ReplacingMergeTree).
- Scenarios where Materialized Views fit but aren't super efficient (hey `uniqState`   ).
- And lastly, one of the hardest problems in syncing OLTP and OLAP databases: Change data capture (CDC).

Want to know more about *why* Materialized Views don't work in these cases? [Read the docs.](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views#what-shouldnt-i-use-materialized-views-for)

As an example, let's look at the *Aggregation Materialized Views over deduplication DS* scenario.

Deduplication in Tinybird happens asynchronously, during merges, which you can't force in Tinybird. That's why you always have to add `FINAL` or the `-Merge` combinator when querying.

Plus, Materialized Views only see the block of data that is being processed at the time, so when materializing an aggregation, it will process any new row, no matter if it was a new id or a duplicated id. That's why this pattern fails.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-4.png&w=3840&q=75)

<-figcaption->
Aggregating MV over deduplication DS will not work as expected





## Solution: Use an alternative architecture with Copy Pipes [¶](https://www.tinybird.co/docs/about:blank#solution-use-an-alternative-architecture-with-copy-pipes)

Tinybird has another kind of Pipe that will help here: [Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes).

At a high level, they're a helpful `INSERT INTO SELECT` , and they can be set to execute following a cron expression. You write your query, and (either on a recurring basis or on demand), the Copy Pipe appends the result in a different table.

So, in this example, you can have a clean, deduplicated snapshot of your data, with the correct Sorting Keys, and can use it to materialize:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-5.png&w=3840&q=75)

<-figcaption->
Copy Pipes to the rescue





## Avoid loss of freshness [¶](https://www.tinybird.co/docs/about:blank#avoid-loss-of-freshness)

*"But if you recreate the snapshot every hour/day/whatever... Aren't you losing freshness?"* Yes - you're right. That's when the lambda architecture comes into play:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-lambda-6.png&w=3840&q=75)

<-figcaption->
Lambda/Kappa Architecture





You'll be combining the already-prepared data with the same operations over the fresh data being ingested at that moment. This means you end up with higher performance despite quite complex logic over both fresh and old data.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Check an[  example implementation of a CDC use case](https://www.tinybird.co/docs/docs/work-with-data/query/guides/lambda-example-cdc)   with this architecture.
- Read more about[  Copy Pipes](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/copy-pipes)  .
- Read more about[  Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views)  .



---

URL: https://www.tinybird.co/docs/work-with-data/query/guides/dynamic-aggregation
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Rollup aggregations with query parameters · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to dynamically aggregate time series data by different time intervals (rollups) to optimize frontend performance."
---




# Rollup aggregations with query parameters [¶](https://www.tinybird.co/docs/about:blank#rollup-aggregations-with-query-parameters)

In this guide, you'll learn how to dynamically aggregate time series data by different time intervals (rollups) to optimize frontend performance.

These days, it's not uncommon to have datasets with a per-second resolution for a few years' worth of data. This creates some demands at the storage and the query layers, and it also presents some challenges for the data consumers. Aggregating this data dynamically and on-the-fly is key for resolving the specific demands at scale.

In this guide, you'll create an API Endpoint that aggregates the data in different time-frames depending on the amount of data, so only the needed rows are sent to the frontend, thereby gaining performance and speed.

When preparing the API Endpoint you'll focus on 3 things:

1. Keep the API Endpoint interface extremely simple: "I want events data from this**  start date**   to this**  end date**   ".
2. Make the API Endpoint return enough data with adequate resolution for the selected date range.
3. Don't add logic to the frontend to do the aggregation of the returned data. You simply request the desired date range knowing that you will receive an amount of data that won't swamp your rendering pipeline.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You'll need to have at least read through the [quick start guide](https://www.tinybird.co/docs/docs/get-started/quick-start) to be familiar with the scenario. The following guide uses a Data Source called `events` with 100M rows, for a ~5-year timespan.

## 1. Build the Pipe [¶](https://www.tinybird.co/docs/about:blank#1-build-the-pipe)

In this step, you'll learn how to use Tinybird's [templating language](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) to add more logic to your API Endpoints.

In addition to the main docs on [using the templating language to pass query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) to Endpoints, you can also learn about variables and the functions that are available within templates in the [CLI > Advanced Templates](https://www.tinybird.co/docs/docs/cli/template-functions) docs.

The Endpoint created in the [quick start guide](https://www.tinybird.co/docs/docs/get-started/quick-start) returns **sales per day** of an ecommerce store, and takes a start and end date. The problem of having a fixed period of one day to aggregate data is that the amount of data transferred will vary a lot, depending on the selected dates, as well as the work that the client will have to do on the frontend to render that data. Also, if the start and end dates are close to each other, the grouping window will be too big and the backend won't return data with a high enough level of detail.

Fortunately, you can add **conditional logic** when defining API Endpoints in Tinybird and, depending on the range of dates selected, the Endpoint will return data grouped by one of these periods:

- Weekly
- Daily
- Every 4 hours
- Every 1 hour
- Every 15 minutes

To do this, you can create a new Pipe named `ecommerce_events_dynamic` . In the first Node, add the following code to 1) keep only the `buy` events, and 2) cast the `price` column to `Float32` changing its name to `buy_events_with_price`:

##### Filtering raw events Data Source to keep only BUY events

SELECT 
    date,
    product_id,
    user_id,
    toFloat32(JSONExtractFloat(extra_data, 'price')) price
FROM events
WHERE event='buy' And then, add another transformation node containing the following query. Name this node `buy_events_dynamic_agg`:

##### Using dynamic parameters to aggregate data depending on the time range

%
SELECT
    {% set days_interval = day_diff(Date(start_date, '2018-01-01'), Date(end_date, '2018-01-31')) %}
    {% if days_interval > 180 %}
        toStartOfWeek(date)
    {% elif days_interval > 31 %}
        toStartOfDay(date)
    {% elif days_interval > 7 %}
        toStartOfInterval(date, INTERVAL 4 HOUR)
    {% elif days_interval > 2 %}
        toStartOfHour(date)
    {% else %}
        toStartOfFifteenMinutes(date)
    {% end %} AS t,
    round(sum(price), 2) AS sales
FROM buy_events_with_price
WHERE date BETWEEN
    toDateTime(toDate({{Date(start_date, '2018-10-01')}})) AND
    toDateTime(toDate({{Date(end_date, '2020-11-01')}}) + 1)
GROUP BY t
ORDER BY t This query makes use of Tinybird's templating language:

- It defines a couple of Date parameters and adds some default values to be able to test the Pipe while you build it ( `{{Date(start_date, '2018-01-01')}}`   and `{{Date(end_date, '2018-12-31')}}`   ).
- It computes the number of days in the interval defined by `start_date`   and `end_date`  : `days_interval = day_diff(...)`  .
- It uses the `days_interval`   variable to decide the best granularity for the data.

Using the templating language additions might look a little complicated at a first glance, but you'll quickly become familiar with it!

## 2. Publish your API Endpoint [¶](https://www.tinybird.co/docs/about:blank#2-publish-your-api-endpoint)

Selecting the `Publish` button > `buy_events_dynamic_agg` node makes your API accessible immediately. Once it's published you can directly test it using the snippets available in the API Endpoint page or using Tinybird's REST API. Just change the `start_date` and the `end_date` parameters to see how the aggregation window changes dynamically.

##### Testing the API Endpoint using cURL

TOKEN=

curl -s "https://api.tinybird.co/v0/pipes/ecommerce_events_dynamic.json?start_date=2018-10-01&end_date=2018-11-01&token=$TOKEN" \
    | jq '.data[:2]'
[
  {
    "t": "2018-10-01 00:00:00",
    "sales": 66687.38
  },
  {
    "t": "2018-10-01 04:00:00",
    "sales": 50821.24
  }
] Use a Token with the right scope. Replace `` with a Token whose [scope](https://www.tinybird.co/docs/docs/api-reference/token-api) is `READ` or higher.

To sum up: With Tinybird, you can dynamically return different responses from your analytics API Endpoints depending on the request's parameters. This can give you more granular control over the data you send to the client, either for performance or privacy reasons.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about the[  using the templating language to pass query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters)  .
- Need to deduplicate your data? Read the[  deduplication strategies guide](https://www.tinybird.co/docs/docs/work-with-data/strategies/deduplication-strategies)  .



---

URL: https://www.tinybird.co/docs/work-with-data/query/guides/adapt-postgres-queries
Last update: 2025-02-11T13:54:28.000Z
Content:
---
title: "Adapt Postgres queries · Tinybird Docs"
theme-color: "#171612"
description: "Postgres is great as an OLTP database, but if you need real-time queries on hundreds of millions of rows, it's not going to be fast enough. Tinybird is. Most queries from Postgres will look very similar on Tinybird. Here you'll learn how to adapt most of the Postgres queries to run on Tinybird."
---




# Adapt Postgres queries [¶](https://www.tinybird.co/docs/about:blank#adapt-postgres-queries)

In this guide, you'll learn how to adapt Postgres queries to run on Tinybird, looking at a specific example.

Postgres is great as an OLTP database, but if you need real-time queries on hundreds of millions of rows, it's not going to be fast enough. Tinybird is. Most queries from Postgres will look very similar on Tinybird. This guide explains how to adapt most Postgres queries to run on Tinybird.

[Haki Benita](https://twitter.com/be_haki) has a [great blog post](https://hakibenita.com/sql-for-data-analysis) on how to do lots of operations like pivot tables, subtotals, linear regression, binning, or interpolation on Postgres, coming from a Pandas background. This guide shows how to adapt most of the Postgres queries from Haki's post to run on Tinybird.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

You don't need an active Tinybird Workspace to read through this guide, but it's a good idea to read Haki's post first so you're familiar with the examples. In addition, a working knowledge of Tinybird and SQL is required.

## Common table expressions [¶](https://www.tinybird.co/docs/about:blank#common-table-expressions)

Both the `WITH  AS ` as well as the `WITH  AS ` syntaxes are supported.

WITH emails AS (
    SELECT 'ME@hakibenita.com' AS email
)
SELECT * FROM emails
;

WITH emails AS 
    (
        SELECT 'ME@hakibenita.com' AS email
    )
SELECT *
FROM emails

Query id: 6b234b03-6dc4-4ddf-8454-b03d34b75b60

┌─email─────────────┐
│ ME@hakibenita.com │
└───────────────────┘

1 rows in set. Elapsed: 0.014 sec.
##### A CHAINED CTE ON POSTGRES

WITH emails AS (
    SELECT 'ME@hakibenita.com' AS email
),

normalized_emails AS (
    SELECT lower(email) AS email FROM emails
)

SELECT * FROM normalized_emails;

WITH
    emails AS 
    (
        SELECT 'ME@hakibenita.com' AS email
    ),
    normalized_emails AS 
    (
        SELECT lower(email) AS email
        FROM emails
    )
SELECT *
FROM normalized_emails

Query id: c511a113-1852-4a9f-90bf-99d33eba8254

┌─email─────────────┐
│ me@hakibenita.com │
└───────────────────┘

1 rows in set. Elapsed: 0.127 sec.
### On Tinybird [¶](https://www.tinybird.co/docs/about:blank#on-tinybird)

For now, Tinybird only supports the WITH  AS  syntax. The previous queries need to be rewritten like this:

##### CTES ON TINYBIRD

WITH (SELECT 'ME@hakibenita.com') AS email
SELECT email There's a difference with CTEs on Postgres VS Tinybird. In Postgres, as the original post says, "CTEs are a great way to split a big query into smaller chunks, perform recursive queries and even to cache intermediate results". On Tinybird, CTEs can only return one row, so those intermediate results can't have multiple rows. For a similar result, on Tinybird you have to use subqueries.

A common pattern is returning a tuple of groupArrays in the CTE, so you can return more than one row in the form of arrays. Then consume the results in the main query with transform for instance or arrayJoin.

In Tinybird, Pipes act like notebooks where each node is a subquery and you can refer to the results of one node in another Node. It's great to see intermediate results and reduce the complexity of your queries. If you'd like to try it out, [sign up here](https://www.tinybird.co/signup?utm_source=blog).

## Generating data [¶](https://www.tinybird.co/docs/about:blank#generating-data)

As you see in the original article, [in Postgres](https://hakibenita.com/sql-for-data-analysis#generating-data) there are several ways to do it:

## Union all [¶](https://www.tinybird.co/docs/about:blank#union-all)

This works the same in Tinybird as in Postgres:

##### UNION ALL ALSO WORKS

WITH dt AS (
   SELECT 1 AS id, 'haki' AS name
   UNION ALL
   SELECT 2, 'benita'
)
SELECT * FROM dt;

WITH dt AS (
    SELECT
        1 AS id,
        'haki' AS name
    UNION ALL
    SELECT
        2,
        'benita'
)
SELECT *
FROM dt

Query id: e755e5a5-5e5b-4e8a-a262-935f9946d45d

┌─id─┬─name───┐
│  2 │ benita │
└────┴────────┘
┌─id─┬─name─┐
│  1 │ haki │
└────┴──────┘

2 rows in set. Elapsed: 0.051 sec. The VALUES keyword won't work on Tinybird to select data, only to insert it.

## Joining data [¶](https://www.tinybird.co/docs/about:blank#joining-data)

The join syntax from Postgres will work on Tinybird, but typically the kinds of analytical data that you'll store on Tinybird will be orders of magnitude bigger than what you'd store on Postgres, and this would make your joins slow. There are ways to make JOINs faster, check the [best practices for writing faster SQL queries](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) or contact us for guidance.

## Unnest - arrayJoin [¶](https://www.tinybird.co/docs/about:blank#unnest-arrayjoin)

[arrayJoin](https://www.tinybird.co/docs/docs/sql-reference/functions/array-functions#arrayjoin) is the Tinybird equivalent of unnest on Postgres. So this Postgres query:

##### ON Tinybird, UNNEST EXPANDS AN ARRAY INTO ROWS

WITH dt AS (
    SELECT unnest(array[1, 2]) AS n
)
SELECT * FROM dt; Would be rewritten on Tinybird like this:

##### ON Tinybird, ARRAYJOIN EXPANDS AN ARRAY INTO ROWS

SELECT arrayJoin([1, 2]) AS dt
## Generating series of data [¶](https://www.tinybird.co/docs/about:blank#generating-series-of-data)

### Generate_series [¶](https://www.tinybird.co/docs/about:blank#generate-series)

The generate_series doesn't exist on Tinybird, but with the [numbers](https://www.tinybird.co/docs/docs/) function a lot can be accomplished. This is its basic usage:

##### NUMBERS PRODUCES ROWS

SELECT * FROM numbers(10) A similar result can be obtained with the [range](https://www.tinybird.co/docs/docs/sql-reference/functions/array-functions#range) function, that returns arrays. If you only provide an argument, it behaves like numbers. And with range you can also specify a start, end and step:

##### RANGE PRODUCES ARRAYS

SELECT range(10), range(0, 10, 2) This, combined with arrayJoin lets us do the same as generate_series:

##### RANGE OF INTEGERS USING START, END AND STEP

SELECT arrayJoin(range(0, 10, 2)) AS number
## Generating time series [¶](https://www.tinybird.co/docs/about:blank#generating-time-series)

generate_series can produce results with other types different than integers, while range only outputs integers. But with some smart logic you can achieve the same results. For example, on Postgres you'd generate a with a datetime for each hour in a day this way, as in the original [post](https://hakibenita.com/sql-for-data-analysis#generating-data):

##### THE GENERATE_SERIES FUNCTION OF POSTGRES

WITH daterange AS (
    SELECT *
    FROM generate_series(
        '2021-01-01 UTC'::timestamptz, -- start
        '2021-01-02 UTC'::timestamptz, -- stop
        interval '1 hour'   -- step
    ) AS t(hh)
) SELECT * FROM daterange;

           hh
────────────────────────
 2021-01-01 00:00:00+00
 2021-01-01 01:00:00+00
 2021-01-01 02:00:00+00
 ...
 2021-01-01 22:00:00+00
 2021-01-01 23:00:00+00
 2021-01-02 00:00:00+00
### Generate a time series specifying the start date and the number of intervals [¶](https://www.tinybird.co/docs/about:blank#generate-a-time-series-specifying-the-start-date-and-the-number-of-intervals)

On Tinybird, you can achieve the same this way:

##### TIME SERIES ON Tinybird GIVEN START DATE, NUMBER OF INTERVALS AND INTERVAL SIZE

WITH toDate('2021-01-01') as start
SELECT addHours(toDate(start), number) AS hh
FROM (
    SELECT arrayJoin(range(0, 24)) AS number
)
### Generate a time series specifying the start and end date and the step [¶](https://www.tinybird.co/docs/about:blank#generate-a-time-series-specifying-the-start-and-end-date-and-the-step)

Another way of doing the same thing:

##### TIME SERIES ON Tinybird GIVEN THE START AND END DATE AND THE STEP SIZE

WITH toStartOfDay(toDate('2021-01-01')) AS start,
toStartOfDay(toDate('2021-01-02')) AS end
SELECT arrayJoin(arrayMap(x -> toDateTime(x), range(toUInt32(start), toUInt32(end), 3600))) as hh
### Generate a time series using timeSlots [¶](https://www.tinybird.co/docs/about:blank#generate-a-time-series-using-timeslots)

Using the [timeSlots](https://www.tinybird.co/docs/docs/sql-reference/functions/date-time-functions#timeslots) function, you can specify the start (DateTime), duration (seconds) and step (seconds) and it generates an array of DateTime values.

##### TIME SERIES ON Tinybird USING THE TIMESLOTS FUNCTION

WITH toDateTime('2021-01-01 00:00:00') AS start
SELECT arrayJoin(timeSlots(start, toUInt32(24 * 3600), 3600)) AS hh
## Generating a random value [¶](https://www.tinybird.co/docs/about:blank#generating-a-random-value)

The [rand](https://www.tinybird.co/docs/docs/sql-reference/functions/random-functions#rand) function in Tinybird is akin to random in Postgres, with the difference that rand returns a random UInt32 number between 0 and 4294967295. So to get random floats between 0 and 1 like random, you have to divide the result by 4294967295.

##### GENERATING A RANDOM VALUE ON Tinybird WITH THE RAND FUNCTION

SELECT 
    rand() random_int,
    random_int / 4294967295 random_float To get more than one row, you'd simply do

##### GENERATING SEVERAL RANDOM VALUES ON Tinybird

SELECT 
    rand() random_int,
    random_int / 4294967295 random_float
FROM numbers(100)
## Generating random integers within a range [¶](https://www.tinybird.co/docs/about:blank#generating-random-integers-within-a-range)

You would use the floor or ceil function (not round, for the reasons explained [here](https://hakibenita.com/sql-for-data-analysis#random) ) in addition to the result of rand multiplied by the max of the range of integers you want to generate, like this:

##### GENERATING SEVERAL RANDOM INTEGERS IN A GIVEN RANGE ON Tinybird

SELECT
    ceil(rand() / 4294967295 * 3) AS n
FROM
    numbers(10) And here you can see that the distribution is uniform (this wouldn't happen if you had use round):

##### THE DISTRIBUTION IS UNIFORM

SELECT
    ceil(rand() / 4294967295 * 3) AS n,
    count(*)
FROM
    numbers(10000)
GROUP BY n
ORDER BY n
## Sampling data from a list [¶](https://www.tinybird.co/docs/about:blank#sampling-data-from-a-list)

This is how you'd take samples with replacement from a list in Postgres:

##### RANDOM CHOICE IN POSTGRES

SELECT
    (array['red', 'green', 'blue'])[ceil(random() * 3)] AS color
FROM
    generate_series(1, 5); In Tinybird, this is how you'd do it:

##### RANDOM CHOICE IN Tinybird

SELECT
    ['red', 'green', 'blue'][toInt32(ceil(rand() / 4294967295 * 3))] AS color
FROM
    numbers(5) To get only one value, you'd remove the FROM numbers(5) part. Note that to define an array on Tinybird you can do it either calling array('red', 'green', 'blue') or with ['red', 'green', 'blue'] like in the code snippet.

## Sampling data from a table [¶](https://www.tinybird.co/docs/about:blank#sampling-data-from-a-table)

Sorting data by rand() can be used to get a random sample, like here:

##### SAMPLE DATA USING RAND

SELECT *
FROM events_mat
ORDER BY rand()
LIMIT 100 But this is slow, as a full scan of the table has to be run here.

A more efficient way to do it's using the SAMPLE clause. You can pass an integer to it (should be large enough, typically above 1000000):

##### THE SAMPLE CLAUSE, PASSING AN INTEGER

SELECT * FROM events_mat SAMPLE 1000000 And you can also pass a float between 0 and 1, to indicate the fraction of the data that will be sampled.

##### THE SAMPLE CLAUSE, PASSING A FLOAT

SELECT * FROM events_mat SAMPLE 0.01
## Descriptive statistics on a numeric series [¶](https://www.tinybird.co/docs/about:blank#descriptive-statistics-on-a-numeric-series)

Tinybird also comes with lots of statistical function, like Postgres does (see [this section](https://hakibenita.com/sql-for-data-analysis#describing-a-series) of the original post). The first query, written on Postgres this way:

##### DESCRIPTIVE STATISTICS ON POSTGRES

WITH s AS (
    SELECT * FROM (VALUES (1), (2), (3)) AS t(n)
)
SELECT
    count(*),
    avg(n),
    stddev(n),
    min(n),
    percentile_cont(array[0.25, 0.5, 0.75]) WITHIN GROUP (ORDER BY n),
    max(n)
FROM
    s;

count │   avg  │   stddev  │ min │ percentile_cont │ max
──────┼────────┼───────────┼─────┼─────────────────┼─────
    3 │ 2.0000 │ 1.0000000 │   1 │ {1.5,2,2.5}     │   3 Would be done on Tinybird like this:

##### DESCRIPTIVE STATISTICS ON Tinybird

SELECT
    count(*),
    avg(n),
    stddevSamp(n),
    min(n),
    quantiles(0.25, 0.5, 0.75)(n),
    max(n)
FROM
    (SELECT arrayJoin([1,2,3]) AS n)
## Descriptive statistics on categorical series [¶](https://www.tinybird.co/docs/about:blank#descriptive-statistics-on-categorical-series)

Tinybird can also be used to get some statistics from discrete values. While on Postgres you'd do this:

##### DESCRIPTIVE STATISTICS OF CATEGORICAL VALUES ON POSTGRES

WITH s AS (SELECT unnest(array['a', 'a', 'b', 'c']) AS v)
SELECT
    count(*),
    count(DISTINCT V) AS unique,
    mode() WITHIN GROUP (ORDER BY V) AS top
FROM
    s;

 count │ unique │ top
───────┼────────┼─────
     4 │      3 │ a On Tinybird you'd do:

##### DESCRIPTIVE STATISTICS OF CATEGORICAL VALUES ON Tinybird

SELECT
    count(*) AS count,
    uniq(v) AS unique,
    topK(1)(v) top
FROM 
    (SELECT arrayJoin(['a', 'b', 'c', 'd']) AS v) [uniq](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#uniq) will provide approximate results when your data is very big. If you need exact results you can use [uniqExact](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#uniqexact) , but be aware that uniq will generally be faster than uniqExact. Check out the [topK](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#topk) docs as well if you're interested.

As a side note, if you have categorical columns, most likely you'll have [better performance](https://altinity.com/blog/2019/3/27/low-cardinality) and lower storage cost data types. The performance of using LowCardinality will be better than using the base data types even on columns with more than a few millions of different values.

## Subtotals and aggregations [¶](https://www.tinybird.co/docs/about:blank#subtotals-and-aggregations)

The same operations done in [this section](https://hakibenita.com/sql-for-data-analysis#subtotals) of Haki's post can be done with Tinybird.

Given a table that contains this data:

##### EMPLOYEES

SELECT * FROM employees Finding the number of employees with each role is straightforward, same syntax as on Postgres:

##### EMPLOYEES PER ROLE

SELECT department, role, count(*) count
FROM employees
GROUP BY department, role
### Using rollup and cube [¶](https://www.tinybird.co/docs/about:blank#using-rollup-and-cube)

The ROLLUP modifier is also available on Tinybird, although the syntax is slightly different than on Postgres. This query on Postgres:

##### GROUP BY WITH ROLLUP ON POSTGRES

SELECT department, role, COUNT(*)
FROM employees
GROUP BY ROLLUP(department, role); would be written on Tinybird like this:

##### GROUP BY WITH ROLLUP ON Tinybird

SELECT department, role, COUNT(*)
FROM employees
GROUP BY department, role WITH ROLLUP It allows you to have more subtotals (but not all). To have all the subtotals for all the possible combinations of grouping keys, you need to use the CUBE modifier:

##### GROUP BY WITH CUBE ON Tinybird

SELECT department, role, COUNT(*)
FROM employees
GROUP BY department, role WITH CUBE
## Pivot tables and conditional expressions [¶](https://www.tinybird.co/docs/about:blank#pivot-tables-and-conditional-expressions)

Pivot tables let you reshape data when you want typically a column with keys, a column with categories and a column with values, and you want to aggregate those values and use the categories column as columns of a new table.

On Postgres you could do it this way:

##### PIVOT TABLE CREATED MANUALLY ON POSTGRES

SELECT
    role,
    COUNT(*) FILTER (WHERE department = 'R&D') as "R&D",
    COUNT(*) FILTER (WHERE department = 'Sales') as "Sales"
FROM
    employees
GROUP BY
    role;

   role    │ R&D │ Sales
───────────┼─────┼───────
 Manager   │   1 │     1
 Developer │   2 │     2 On Tinybird, you could do the same this way:

##### PIVOT TABLE CREATED MANUALLY ON POSTGRES

SELECT 
    role,
    countIf(department = 'R&D') as "R&D",
    countIf(department = 'R&D') as "Sales"
FROM
    employees
GROUP BY
    role
## Running and Cumulative Aggregations [¶](https://www.tinybird.co/docs/about:blank#running-and-cumulative-aggregations)

Aggregations over sliding windows are a common solution. This can be done with Window functions.

This also can be done with the [groupArrayMovingSum](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#grouparraymovingsum) and [groupArrayMovingAvg](https://www.tinybird.co/docs/docs/sql-reference/functions/aggregate-functions#grouparraymovingavg) functions, available in stable releases since a long time ago already. This is an example of its usage:

Given this dataset:

##### GOOGLE TRENDS DATA FOR THE TERM 'AMAZON'

SELECT date, amazon as value FROM amazon_trends You can compute a 7-day moving average of value like this:

##### 7-DAY MOVING AVERAGE

SELECT * FROM 
    (SELECT
        groupArray(date) as date_arr,
        groupArray(value) as value_arr,
        groupArrayMovingAvg(7)(value) mov_avg
    FROM (SELECT date, amazon as value FROM amazon_trends))
ARRAY JOIN * The periods parameter is optional. If you omit it, all the previous rows are used for the aggregation.

## Linear regression [¶](https://www.tinybird.co/docs/about:blank#linear-regression)

Given this data

##### DATA TO PERFORM LINEAR REGRESSION

SELECT
    arrayJoin([[1.2, 1], [2, 1.8], [3.1, 2.9]])[1] x,
    arrayJoin([[1.2, 1], [2, 1.8], [3.1, 2.9]])[2] y On Postgres you can see in the [original post](https://hakibenita.com/sql-for-data-analysis#linear-regression) that you'd do linear regression like this:

##### LINEAR REGRESSION ON POSTGRES

WITH t AS (SELECT * FROM (VALUES
    (1.2, 1.0),
    (2.0, 1.8),
    (3.1, 2.9)
) AS t(x, y))
SELECT
    regr_slope(y, x) AS slope,
    regr_intercept(y, x) AS intercept,
    sqrt(regr_r2(y, x)) AS r
FROM
    t;

       slope        │      intercept       │ r
────────────────────┼──────────────────────┼───
 1.0000000000000002 │ -0.20000000000000048 │ 1 There's not a function on Tinybird like regr_r2 that gives you the R2 coefficient, but it wouldn't be hard to calculate it yourself as the formula is [simple](https://www.google.com/search?q=r2+formula).

## Filling null values. [¶](https://www.tinybird.co/docs/about:blank#filling-null-values)

This part is called " [interpolation](https://hakibenita.com/sql-for-data-analysis#interpolation) " in Haki's post. Filling null values with Pandas is a one-liner. Consider this table

##### A TABLE WITH A INT AND A STRING COLUMN, WITH SOME NULL VALUES

SELECT * FROM num_str
## Fill null values with a constant value [¶](https://www.tinybird.co/docs/about:blank#fill-null-values-with-a-constant-value)

The way to replace all the null values by a constant value is using the coalesce function, that works in Tinybird the same way it does in Postgres, using [coalesce](https://www.tinybird.co/docs/docs/sql-reference/functions/functions-for-nulls#coalesce):

##### FILLING NULL VALUES WITH A CONSTANT

SELECT
    n,
    coalesce(v, 'X') AS v
FROM
    num_str
## Back and forward filling data [¶](https://www.tinybird.co/docs/about:blank#back-and-forward-filling-data)

groupArray, as the other aggregate functions on Tinybird, skips null values. So the solution involves replacing them by another value (make sure that the new value doesn't appear in the column before). This is done with the ifNull function. Add some array magic in, and this is how you'd do it:

##### BACK AND FORWARD FILLING VALUES

SELECT
  values.1 n,
  values.2 v,
  values.3 v_ffill,
  values.4 v_bfill
FROM
  (SELECT
    arrayJoin(
      arrayZip(
        groupArray(n) AS n,
        arrayMap(x -> x != 'wadus' ? x : null, groupArray(v_nulls_replaced)) AS v,
        arrayFill(x -> x != 'wadus', groupArray(v_nulls_replaced)) AS v_ffill,
        arrayReverseFill(x -> x != 'wadus', groupArray(v_nulls_replaced)) AS v_bfill
      )
    ) values
  FROM
     (SELECT 
        *,
        ifNull(v, 'wadus') v_nulls_replaced
      FROM num_str 
      ORDER BY n ASC)
  ) To understand what's going on here, import [this Pipe](https://www.tinybird.co/docs/docs/assets/pipes/back_and_forward_fill.pipe) with a step-by-step explanation and the results of the transformations that are taking place. Tinybird lets you run each subquery in a node of a notebook-like UI. This lets you build and debug complex queries in a cleaner way. If you'd like to use it, sign up [here](https://www.tinybird.co/signup).

## Filling gaps in time-series, reshaping indexes. [¶](https://www.tinybird.co/docs/about:blank#filling-gaps-in-time-series-reshaping-indexes)

Sometimes, you'll group a time-series by a Date or DateTime column, and it can happen that the intervals between rows aren't always the same because there were no values found for some dates or datetimes. In Pandas, the solution would be creating a new date_range index and then re-indexing the original Series/DataFrame with that index.

On Tinybird, the same can be accomplished with the WITH FILL modifier. Here's a simple example of it:

##### FILLING GAPS IN TIME ON Tinybird

SELECT 
    toDate((number * 2) * 86400) AS d1,
    'string_value' as string_col,
    toInt32(rand() / exp2(32) * 100) as n
FROM numbers(10)
ORDER BY 
    d1 WITH FILL STEP 1 The `STEP 1` part isn't necessary here as it's the default, but know that you can set a different value than 1.

## Linear interpolation [¶](https://www.tinybird.co/docs/about:blank#linear-interpolation)

Imagine you have a table like this, containing a time-series with some rows missing. You could fill those missing gaps with the WITH FILL expression previously shown, but that way you'd just get zeroes when there's a missing value, while the actual missing value is probably closer to the previous and the next values than to zero.

##### TIME-SERIES WITH MISSING ROWS

SELECT *, bar(value, 0, 100, 20) 
FROM trends_with_gaps 
ORDER BY date WITH FILL STEP 1 Linear interpolation is the simplest way to fill those missing values. In it, missing values are replaced by the average of the previous and the next known values. On Postgres, Haki's post explains how to do it [here](https://hakibenita.com/sql-for-data-analysis#linear-interpolation).

On Tinybird, this can be done with arrays:

##### INTERPOLATING VALUES

SELECT
    date,
    value,
    value_interpolated,
    bar(value_interpolated, 0, 100, 20) AS value_interpolated_bar
FROM 
(
    SELECT
        groupArray(date) AS dt_arr,
        groupArray(value) AS value_arr,
        arrayFill(x -> ((x.1) > 0), arrayZip(value_arr, dt_arr)) AS value_lower,
        arrayReverseFill(x -> ((x.1) > 0), arrayZip(value_arr, dt_arr)) AS value_upper,
        arrayMap((l, u, v, dt) -> if(v > 0, v, (l.1) + ((((u.1) - (l.1)) / ((u.2) - (l.2))) * (dt - (l.2)))), value_lower, value_upper, value_arr, dt_arr) AS value_interpolated
    FROM (
            SELECT * FROM trends_with_gaps
            ORDER BY date WITH FILL STEP 1
        )
)
ARRAY JOIN
    dt_arr AS date,
    value_interpolated,
    value_arr AS value For a step-by-step explanation of how this works, and to see how you could construct this query iteratively with a notebook-like interface on Tinybird, import [this Pipe](https://www.tinybird.co/docs/docs/assets/pipes/interpolating_gaps.pipe).

## Binning and histograms [¶](https://www.tinybird.co/docs/about:blank#binning-and-histograms)

The original post talks about custom binning, equal-width and equal-height binning. The way to do custom binning is very similar on Tinybird, which also supports CASE statements. Equal height binning could be achieved with the quantiles function, already described before, in the descriptive statistics section. The most interesting use-case of equal-width binning is creating histograms, which is very easy on Tinybird. It even comes with a histogram function, which receives a number of bins and the data, and returns a list of tuples containing the lower and upper bounds of each bucket, as well as its height:

##### THE HISTOGRAM FUNCTION

SELECT histogram(10)(value) AS values FROM trends_with_gaps Then, extracting each values from the tuples and even having a visual representation of the data can be done like this:

## Running Tinybird without worrying about it [¶](https://www.tinybird.co/docs/about:blank#running-tinybird-without-worrying-about-it)

[Tinybird](https://www.tinybird.co/) lets you do real-time analytics on huge amounts of data without having to worry about scalability, hosting or maintaining any Tinybird clusters. With it, you can ingest huge datasets and streaming data, analyze it with SQL and publish dynamic API Endpoints on those queries a couple of clicks.

Tinybird is already being used by some big companies and it's recently been featured on [Techcrunch](https://techcrunch.com/2021/07/05/tinybird-turns-raw-data-into-realtime-api-at-scale) . To use Tinybird, [sign up here](https://www.tinybird.co/signup?utm_source=blog)



---

URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/materialized-views/example-mv-ui
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Example of Materialized View · Tinybird Docs"
theme-color: "#171612"
description: "The following example shows how to create a Materialized View in Tinybird."
---




# Example of Materialized View [¶](https://www.tinybird.co/docs/about:blank#example-of-materialized-view)

The following example shows how to create a Materialized View in Tinybird.

For this example, the question to answer is: "What was the average purchase price of all products in each city on a particular day?"

## Get baseline performance [¶](https://www.tinybird.co/docs/about:blank#get-baseline-performance)

First, you need to check the baseline performance before creating an Endpoint on top of the original ingested Data Source.

### Filtering and extracting [¶](https://www.tinybird.co/docs/about:blank#filtering-and-extracting)

Create a first node to filter the data to include only `buy` events. Simultaneously, normalize event timestamps to rounded days and extract `city` and `price` data from the `extra_data` column containing JSON.

##### NODE 1: FILTERING & EXTRACTING

SELECT
  toDate(date) day,
  JSONExtractString(extra_data, 'city') as city,
  JSONExtractFloat(extra_data, 'price') as price
FROM events
WHERE event = 'buy'
### Averaging [¶](https://www.tinybird.co/docs/about:blank#averaging)

Next, create a node to aggregate data and get the average purchase price by city per day:

##### Node 2: Averaging

SELECT
  day,
  city,
  avg(price) as avg
FROM only_buy_events_per_city
GROUP BY day, city
### Creating a parameterized API Endpoint [¶](https://www.tinybird.co/docs/about:blank#creating-a-parameterized-api-endpoint)

Finally, create a node that you can publish as an API Endpoint, adding a parameter to filter results to a particular day:

##### Node 3: Creating a parameterized API Endpoint

%
SELECT *
FROM avg_buy_per_day_and_city
WHERE day = {{Date(day, '2020-09-09')}}
## Create the Materialized View [¶](https://www.tinybird.co/docs/about:blank#create-the-materialized-view)

A Materialized View is formed by a Pipe that ends in the creation of a new Data Source instead of an Endpoint. Start by duplicating the existing Pipe:

1. Duplicate the `events_pipe`   so that you get a Pipe with the same nodes as the baseline.
2. Rename the Pipe to `events_pipe_mv`  .

In this case you're going to materialize the second node in the Pipe, because it's the one performing the aggregation. The third node simply provides you with a filter to create a parameterized Endpoint.

You can't use [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) in nodes that are published as Materialized Views.

To create the Materialized View:

1. Select the node options.
2. Select**  Create a Materialized View from this Node**  .
3. Update the**  View**   settings as required.
4. Select**  Create Materialized View**  .



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-2.gif&w=3840&q=75)


Your Materialized View has been created as a new Data Source. By default, the name that Tinybird gives the Data Source is the name of the materialized Pipe ode appended with `_mv` , in this case `avg_buy_per_day_and_city_mv`.

Append the names of all your Transformation Pipes and Materialized View Data Sources with `_mv` , or another common identifier.

This example uses a new Data Source. You can also select an existing Data Source as the destination for the Materialized View, but it must have the same schema as the Materialized View output. If both schemas match, Tinybird offers that Data Source as an option that you can select when you're creating a Materialized View.

### Populating Existing Data [¶](https://www.tinybird.co/docs/about:blank#populating-existing-data)

When Tinybird creates a Materialized View, it initially only populates a partial set of data from the original Data Source. This allows you to quickly validate the results of the Materialized View.

Once you have validated with the partial dataset, you can populate the Materialized View with all the existing data in the Original Data source. To do so, select **Populate With All Data**.

You now have a Materialized View Data Source that you can use to query against in your Pipes.

### Testing performance improvements [¶](https://www.tinybird.co/docs/about:blank#testing-performance-improvements)

To test how the Materialized View has improved the performance of the API Endpoint, return to the original Pipe.

In the original Pipe, do the following:

1. Copy the SQL from the original API Endpoint Node, `avg_buy`  .
2. Create a new transformation Node, called `avg_buy_mv`  .
3. Paste the query from the original API Endpoint node into your new Node.
4. Update the query to select from your new Materialized View, `avg_buy_per_day_and_city_mv`  .



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-3.gif&w=3840&q=75)


Because this query is an aggregation, you need to rewrite the query, because data in Materialized Views in Tinybird exists in intermediate states. As new data is ingested, the data in the Materialized View gets appended in blocks of partial results. A background periodically process merges the appended partial results and saves them in the Materialized View.

Because you are processing data in real time, you might not be able to wait for the background process to complete. To account for this, reaggregate in the API Endpoint query using the `-merge` combinator. This example uses an `avg` aggregation, so you need to use `avgMerge` to compact the results in the Materialized View.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-4.gif&w=3840&q=75)


When you run your the modified query, you get the same results as you got when you ran the final node against the original Data Source. This time, however, the performance has improved significantly.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-5.png&w=3840&q=75)


With the Materialized View, you get the same results, but process less data twice as fast at a fraction of the cost.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-6.png&w=3840&q=75)


### Pointing the API Endpoint at the new node [¶](https://www.tinybird.co/docs/about:blank#pointing-the-api-endpoint-at-the-new-node)

Now that you've seen how much the performance of the API Endpoint query has improved by using a Materialized View, you can easily change which node the API Endpoint uses. Select the node dropdown, and then select the new node you created by querying the Materialized View.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaterialized-views-7.gif&w=3840&q=75)


This way, you improve the API Endpoint performance while retaining the original URL, so applications which call that API Endpoint see an immediate performance boost.



---

URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/materialized-views/example-mv-cli
Last update: 2024-12-17T18:51:47.000Z
Content:
---
title: "Example of Materialized View (CLI) · Tinybird Docs"
theme-color: "#171612"
description: "The following example shows how to create a Materialized View using Tinybird CLI."
---




# Example of Materialized View (CLI) [¶](https://www.tinybird.co/docs/about:blank#example-of-materialized-view-cli)

Consider an `events` Data Source which, for each action performed in an ecommerce website, stores a timestamp, the user that performed the action, the product, which type of action - `buy`, `add to cart`, `view` , and so on - and a JSON column containing some metadata, such as the price.

The `events` Data Source is expected to store billions of rows per month. Its data schema is as follows:

##### DEFINITION OF THE EVENTS.DATASOURCE FILE

SCHEMA >
    `date` DateTime,
    `product_id` String,
    `user_id` Int64,
    `event` String,
    `extra_data` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYear(date)"
ENGINE_SORTING_KEY "date, cityHash64(extra_data)"
ENGINE_SAMPLING_KEY "cityHash64(extra_data)" You want to publish an API Endpoint calculating the top 10 products in terms of sales for a date range ranked by total amount sold. Here's where Materialized Views can help you.

## Materialize the results [¶](https://www.tinybird.co/docs/about:blank#materialize-the-results)

After doing the desired transformations, set the `TYPE` parameter to `materialized` and add the name of the Data Source, which materializes the results.

##### DEFINITION OF THE TOP PRODUCT PER_DAY.PIPE

NODE only_buy_events
DESCRIPTION >
    filters all the buy events

SQL >
    SELECT 
        toDate(date) AS date, 
        product_id, 
        JSONExtractFloat(extra_data, 'price') AS price
    FROM events 
    WHERE event = 'buy'

NODE top_per_day
SQL >
    SELECT 
        date, 
        topKState(10)(product_id) AS top_10, 
        sumState(price) AS total_sales
    FROM only_buy_events 
    GROUP BY date

TYPE materialized
DATASOURCE top_products_view Do the rest in the Data Source schema definition for the Materialized View, named `top_products_view`:

##### DEFINITION OF THE TOP PRODUCTS VIEW.DATASOURCE FILE

SCHEMA >
    `date` Date,
    `top_10` AggregateFunction(topK(10), String),
    `total_sales` AggregateFunction(sum, Float64)

ENGINE "AggregatingMergeTree"
ENGINE_SORTING_KEY "date" The destination Data Source uses an [AggregatingMergeTree](https://www.tinybird.co/docs/docs/) engine, which for each `date` stores the corresponding `AggregateFunction` for the top 10 products and the total sales.

Having the data precalculated as it gets ingested makes the API Endpoint run in real time, no matter the number of rows in the `events` Data Source.

As for the Pipe used to build the API Endpoint, `top_products_agg` , it's as follows:

##### DEFINITION OF THE TOP PRODUCTS PER DAY PIPE

NODE top_products_day
SQL >
    SELECT 
        date, 
        topKMerge(10)(top_10) AS top_10, 
        sumMerge(total_sales) AS total_sales
    FROM dev__top_products_view 
    GROUP BY date When preaggregating, the Aggregate Function uses the mode `State` , while when getting the calculation it makes use of `Merge`.

## Push to Tinybird [¶](https://www.tinybird.co/docs/about:blank#push-to-tinybird)

Once it's done, push everything to your Tinybird account:

##### PUSH YOUR PIPES AND DATA SOURCES USING THE CLI

tb push datasources/top_products_view.datasource
tb push pipes/top_product_per_day.pipe --populate
tb push endpoints/top_products_endpoint.pipe When pushing the `top_product_per_day.pipe` , use the `--populate` flag. This causes the transformation to run in a job, and the Materialized View `top_products_view` to be populated.

You can repopulate Materialized Views at any moment:

##### Command to force populate the materialized view

tb push pipes/top_product_per_day.pipe --populate --force

---

URL: https://www.tinybird.co/docs/work-with-data/process-and-copy/materialized-views/best-practices
Last update: 2025-01-28T09:52:38.000Z
Content:
---
title: "Best practices for Materialized Views · Tinybird Docs"
theme-color: "#171612"
description: "Learn how Materialized Views work and how to best use them."
---




# Best practices for Materialized Views [¶](https://www.tinybird.co/docs/about:blank#best-practices-for-materialized-views)

Read on to learn how [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) work and how to best use them in your data projects.

## How data gets into a Materialized View [¶](https://www.tinybird.co/docs/about:blank#how-data-gets-into-a-materialized-view)

Tinybird ingests data into Materialized Views in blocks. This process is presented in the following diagram. Every time new data is ingested into the origin Data Source, the materialization process is triggered, which applies the transformation Pipe over the data ingested and saves the output of that Pipe, which is a partial result, in the Materialized View.



<-figure->
![Materialization process](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaster-materialized-views-1.png&w=3840&q=75)


Data that was present in the origin Data Source prior to the Materialized View creation is inserted into the destination Materialized View through a populate operation.

### Regular materializaton [¶](https://www.tinybird.co/docs/about:blank#regular-materializaton)

Materialized Views in Tinybird are incremental and triggered upon ingestion. From the moment it's created, the new rows are inserted in the Materialized View. If an insert is too big, it's processed in blocks.

Because materializations are performed only over the new data being ingested, and not over the whole data source, avoid using the following operations: Window functions, `ORDER BY`, `Neighbor` , and `DISTINCT`.

### Populates [¶](https://www.tinybird.co/docs/about:blank#populates)

Populates move historical data from the origin Data Source into the Materialized View. There are two types: complete and partial.

If you're populating from a Data Source with hundreds of millions of rows and doing complex transformations, you might face memory errors. In this type of situation, use partial populates.

If you're using the CLI, populates are triggered using `tb pipe populate` . You can add conditions using the `--sql-condition` flag, for example, `--sql-condition='date == toYYYYMM(now())'` . If your `sql_condition` includes any column present in the Data Source `engine_sorting_key` , the populate job should process less data.

If you have constant ingest in your origin Data Source, see [Populates and streaming ingest](https://www.tinybird.co/docs/about:blank#populates-and-streaming-ingest).

## Aggregated Materialized Views [¶](https://www.tinybird.co/docs/about:blank#aggregated-materialized-views)

Sometimes a background process in Tinybird merges partial results saved in intermediate states during the ingestion process, compacting the results and reducing the number of rows. The following diagram illustrates this process in more detail through a simple example.

Let's say an eCommerce store wants to materialize the count of units sold by product. It's ingesting a JSON object every minute, with a product represented by a capital letter and the quantity sold during the last minute.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaster-materialized-views-2.png&w=3840&q=75)


The store could define in their Pipe some simple SQL to sum the count of units sold per minute as data is ingested. The Pipe is applied over each new block of appended data, and the output is immediately saved in intermediate states into the Materialized View. Every 8 or 10 minutes, the background process merges the intermediate states, completing the aggregation across the entire Data Source.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fmaster-materialized-views-3.png&w=3840&q=75)


Because they're working in real time, the store can't always wait for this background process to take place. When querying the Materialized View, they should use the proper merge combinator and `GROUP BY` clause in the query itself.

### Understanding State and Merge combinators for Aggregates [¶](https://www.tinybird.co/docs/about:blank#understanding-state-and-merge-combinators-for-aggregates)

Tracking when the background process that merges aggregate results in a Materialized View has occurred isn't always practical. Because of this, you need to store intermediate states using the `-State` suffix. If you're creating a Materialized View using the UI, this is done automatically.

Here's an example of using `-State` when defining the transformation Pipe to calculate these intermediate states:

##### USING the -State SUFFIX

NODE Avg calculation
SQL >
SELECT 
  day, 
  city,
  avgState() avg
FROM table
GROUP BY day, city You also need to specifically define the appropriate schema for the Materialized View:

##### MV SCHEMA

SCHEMA >
  day Date,
  city String,
  avg AggregateFunction(avg, Float64)

ENGINE_SORTING_KEY date, city Finally, you need to retrieve the data using the `-Merge` suffix in your API Endpoint node to make sure the merge process is completed for all data in the Materialized View:

##### USE MERGE SUFFIX IN ENDPOINT NODE

NODE endpoint
SQL > 
% 
SELECT 
  day, 
  city,
  avgMerge(avg) as avg
FROM avg_table
WHERE day > {{Date(start_date)}}
GROUP BY day, city
## Understanding the Materialized View parameters [¶](https://www.tinybird.co/docs/about:blank#understanding-the-materialized-view-parameters)

When you create a Materialized View in the UI, Tinybird automatically recommends the best parameters for most use cases. This is useful to understand parameters for more complex use cases.

### Sorting Key [¶](https://www.tinybird.co/docs/about:blank#sorting-key)

The Sorting Key defines how data is sorted and is critical for great performance when filtering. Choose the order of your sorting keys depending on how you're going to query them. Here are a few examples for a simple Materialized View containing `day`, `city` , and `avg` columns:

- You want to query the average for all cities on a particular day: the `day`   column should be the first sorting key.
- You wanted the average over the last month for a particular city: the `city`   column should be the first sorting key.

For Materialized Views containing aggregations, every column in the `GROUP BY` statement has to be in the sorting keys, and only those columns can be sorting keys.

For non-aggregated Materialized Views, you can select other columns if they fit better for your use case, but don't recommend adding too many: you get only a negligible performance boost after the fourth sorting key column.

### Partition by [¶](https://www.tinybird.co/docs/about:blank#partition-by)

A partition is a logical combination of records by a given criterion. Usually you don't need a partition key, or a partition by month is enough.

Tinybird guesses the best partition key if your materialization query has a Date or DateTime column. If there aren't any Date columns, Tinybird doesn't set a partition key. Having no partition is better than having the wrong partition. If you're comfortable with partitions and you want to group records by another criteria, you can switch to the advanced tab and add your custom partition code.

### Time To Live (TTL) [¶](https://www.tinybird.co/docs/about:blank#time-to-live-ttl)

If you have certain lifetime requirements on the data in your Materialized Views, you can specify a Time To Live (TTL) parameter when creating a Materialized View.

An example of TTL requirement is satisfying GDPR regulations. TTLs can also be useful if you only intend to query a brief history of the data. For example, if you always and only query for data from within the last week, you can set a TTL of 7 days. When a TTL is set, all rows older than the TTL are removed from the Materialized View.

### Advanced tab (UI) [¶](https://www.tinybird.co/docs/about:blank#advanced-tab-ui)

Most of the time, the defaults recommended by Tinybird are the best parameters to use. Occasionally, however, you might need to tweak these parameters. For this, you can use the **Advanced** tab in Tinybird, where you can write code that's passed to the **View** creation.

## Populates and streaming ingest [¶](https://www.tinybird.co/docs/about:blank#populates-and-streaming-ingest)

A populate is the operation of moving data that was present in the origin Data Source before the creation of the Materialized View.

The following diagram illustrates the process. At `t1` , the Materialized View is created, so new rows arriving in the origin Data Source are processed into the Materialized View. To move the data from `t0` to `t1` , launch a populate, either manually or when defining the Materialized View, at time `t2`.

All that data that arrives between `t1` and `t2` might be materialized twice: once due to the regular materialization process, at ingest time, and the other one due to the populate process.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fpopulate-duplicates-data.png&w=3840&q=75)


When you don't have streaming ingest in the origin Data Source, it's usually a safe operation, as long as no new data arrives while the populate is running.

### Backfill strategies [¶](https://www.tinybird.co/docs/about:blank#backfill-strategies)

Consider one of the following strategies for backfilling data.

#### Two Materialized View Pipes [¶](https://www.tinybird.co/docs/about:blank#two-materialized-view-pipes)

Use a timestamp in the near future to split real-time ingest and backfill.

Create the regular MV with a `WHERE` clause specifying that materialized data is newer than a certain timestamp in the future. For example, `WHERE timestamp >= '2024-01-31 00:00:00'`:

##### realtime materialized.pipe

NODE node
SQL > 
   %
   SELECT (...) 
   FROM origin_ds
   WHERE timestamp >= '2024-01-31 00:00:00'

TYPE Materialized
TARGET_DATASOURCE mv Wait until the desired timestamp has passed, and create a the backfill Materialized View Pipe with a `WHERE` clause for data before the specified timestamp. No new data is processed, as the condition can't be met.

##### populate.pipe

NODE node
SQL > 
   %
   SELECT (...) 
   FROM origin_ds
   WHERE timestamp < '2024-01-31 00:00:00'

TYPE Materialized
TARGET_DATASOURCE mv Finally, because it's now safe, run the `--populate` command.

#### Use Copy Pipes [¶](https://www.tinybird.co/docs/about:blank#use-copy-pipes)

Depending on the transformations, Copy Pipes can substitute the populate for historical data. See [Backfill strategies](https://www.tinybird.co/docs/docs/work-with-data/strategies/backfill-strategies).

These tips only apply for streaming ingest. With batch ingest, or being able to pause ingest, populates are totally safe.

## Use the same alias in SELECT and GROUP BY [¶](https://www.tinybird.co/docs/about:blank#use-the-same-alias-in-select-and-group-by)

If you use an alias in the `SELECT` clause, you must reuse the same alias in the `GROUP BY`.

Take the following query as an example:

##### Different alias in SELECT and GROUP BY

SELECT
    key,
    multiIf(value = 0, 'isZero', 'isNotZero') as zero,
    sum(amount) as amount
FROM ds
GROUP BY key, value The previous query results in the following error:

`Column 'value' is present in the GROUP BY but not in the SELECT clause`

To fix this, use the same alias in the `GROUP BY`:

##### GOOD: same alias in SELECT and GROUP BY

SELECT
    key,
    multiIf(value = 0, 'isZero', 'isNotZero') as zero,
    sum(amount) as amount
FROM ds
GROUP BY key, zero
## Don't use nested GROUP BYs [¶](https://www.tinybird.co/docs/about:blank#dont-use-nested-group-bys)

Don't use nested `GROUP BY` clauses in the Pipe that creates a Materialized View. While nested aggregations are possible, Materialized Views are processed in independent blocks, and this might yield unexpected results. Tinybird restricts these behaviors and throws an error when they're detected to avoid inaccurate results.

Consider the following query with nested `GROUP BY` clauses:

##### Nested GROUP BY in Pipe

SELECT
    product,
    count() as c
FROM (
    SELECT
        key,
        product,
        count() as orders
    FROM ds
    GROUP BY key, product
)
GROUP BY product The previous query throws the following error:

Columns 'key, product' are present in the GROUP BY but not in the SELECT clause To fix this, make sure you don't nest `GROUP BY` clauses:

##### Single GROUP BY

SELECT
    key,
    product,
    count() as orders
FROM ds
GROUP BY key, product
## Avoid big scans [¶](https://www.tinybird.co/docs/about:blank#avoid-big-scans)

Avoid big scans in Materialized Views. When using JOINs, do them with a subquery to the Data Source you join to, not the whole Data Source.



---

URL: https://www.tinybird.co/docs/publish/charts/guides/real-time-dashboard
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Build a real-time dashboard with Tremor & Next.js · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to build a user-facing web analytics dashboard using Tinybird, Tremor, and Next.js."
---




# Build a real-time dashboard [¶](https://www.tinybird.co/docs/about:blank#build-a-real-time-dashboard)

In this guide you'll learn how to build a real-time analytics dashboard from scratch, for free, using just 3 tools: Tinybird, Tremor, and Next.js. You'll end up with a dashboard and enough familiarity with Tremor to adjust the frontend & data visualization for your own projects in the future.

[GitHub Repository](https://github.com/tinybirdco/demo-user-facing-saas-dashboard-signatures) Imagine you're a [DocuSign](https://www.docusign.com/) competitor. You're building a SaaS to disrupt the document signature space, and as a part of that, you want to give your users a real-time data analytics dashboard so they can monitor how, when, where, and what is happening with their documents in real time.

In this tutorial, you'll learn how to:

1. Use Tinybird to capture events (like a document being sent, signed, or received) using the Tinybird Events API.
2. Process them with SQL.
3. Publish the transformations as real-time APIs.
4. Use Tremor components in a Next.js app to build a clean, responsive, real-time dashboard.

Here's how it all fits together:



<-figure->
![Diagram showing the data flow from Tinybird --> Next.js --> Tremor](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-real-time-dashboard-data-flow.png&w=3840&q=75)


## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you'll need the following:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. Node.js >=18
3. Python >=3.8
4. Working familiarity with JavaScript

This tutorial uses both the Tinybird web UI and the Tinybird CLI. If you're not familiar with the Tinybird CLI, [read the CLI docs](https://www.tinybird.co/docs/docs/cli/install) or just give it a go! You can copy and paste every code snippet and command in this tutorial - each step is clearly explained.

## 1. Create a Tinybird Workspace [¶](https://www.tinybird.co/docs/about:blank#1-create-a-tinybird-workspace)

Navigate to the Tinybird web UI ( [app.tinybird.co](https://app.tinybird.co/) ) and create an empty Tinybird Workspace (no starter kit) called `signatures_dashboard` in your preferred region.

## 2. Create the folder structure [¶](https://www.tinybird.co/docs/about:blank#2-create-the-folder-structure)

In your terminal, create a folder called `tinybird-signatures-dashboard` . This folder is going to contain all your code. Inside it, create a bunch of folders to keep things organized:

##### Create the folder structure

mkdir tinybird-signatures-dashboard && cd tinybird-signatures-dashboard
mkdir datagen datagen/utils app tinybird The final structure will be:

##### Folder structure

└── tinybird-signatures-dashboard
   ├── app
   ├── datagen
   │   └── utils
   └── tinybird
## 3. Install the Tinybird CLI [¶](https://www.tinybird.co/docs/about:blank#3-install-the-tinybird-cli)

The Tinybird CLI is a command-line tool that allows you to interact with Tinybird's API. You will use it to create and manage the data project resources that underpin your real-time dashboard. Run the following commands to prepare the virtual environment, install the CLI, and authenticate (the `-i` flag is for "interactive"):

##### Install the Tinybird CLI

python -m venv .venv
source .venv/bin/activate
pip install tinybird-cli
tb auth -i Choose the region that matches your Workspace region (if you're not sure which region you chose, don't worry: In the Tinybird UI, select the same of the Workspace (top left) and it will say the region under your email address). You’ll then be prompted for your [user admin Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) , which lives in the Tinybird UI under "Tokens". Paste it into the CLI and press enter.

You're now authenticated to your Workspace from the CLI, and your auth details are saved in a `.tinyb` file in the current working directory.

Your user admin Token has full read/write privileges for your Workspace. Don't share it or publish it in your application. You can find more detailed info about Static Tokens [in the Tokens docs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens).

Ensure that the `.tinyb` file and the `.venv` folder aren't publicly exposed by creating a `.gitignore` file and adding it:

##### Housekeeping: Hide your Token!

touch .gitignore
echo ".tinyb" >> .gitignore
echo ".venv" >> .gitignore
## 4. Create a mock data stream [¶](https://www.tinybird.co/docs/about:blank#4-create-a-mock-data-stream)

Now download the [mockDataGenerator.js](https://github.com/tinybirdco/demo-user-facing-saas-dashboard-signatures/blob/main/datagen/mockDataGenerator.js) file and place it in the `datagen` folder.

##### Mock data generator

cd datagen
curl -O https://raw.githubusercontent.com/tinybirdco/demo-user-facing-saas-dashboard-signatures/refs/heads/main/datagen/mockDataGenerator.js
### What this file does [¶](https://www.tinybird.co/docs/about:blank#what-this-file-does)

The `mockDataGenerator.js` script generates mock user accounts, with fields like `account_id`, `organization`, `phone_number` , and various certification statuses related to the account's means of identification:

##### Create fake account data

const generateAccountPayload = () => {
    const status = ["active", "inactive", "pending"];
    const id = faker.number.int({ min: 10000, max: 99999 });
    account_id_list.push(id);

    return {
        account_id: id,
        organization: faker.company.name(),
        status: status[faker.number.int({ min: 0, max: 2 })],
        role: faker.person.jobTitle(),
        certified_SMS: faker.datatype.boolean(),
        phone: faker.phone.number(),
        email: faker.internet.email(),
        person: faker.person.fullName(),
        certified_email: faker.datatype.boolean(),
        photo_id_certified: faker.datatype.boolean(),
        created_on: (faker.date.between({ from: '2020-01-01', to: '2023-12-31' })).toISOString().substring(0, 10),
        timestamp: Date.now(),
    }
} In addition, the code generates mock data events about the document signature process, with variable status values such as `in_queue`, `signing`, `expired`, `error` , and more:

const generateSignaturePayload = (account_id, status, signatureType, signature_id, since, until, created_on) => {
    return {
        signature_id,
        account_id,
        status,
        signatureType,
        since: since.toISOString().substring(0, 10),
        until: until.toISOString().substring(0, 10),
        created_on: created_on.toISOString().substring(0, 10),
        timestamp: Date.now(),
        uuid: faker.string.uuid(),
    }
} Lastly, the generator creates and sends a final status for the signature using weighted values:

const finalStatus = faker.helpers.weightedArrayElement([
    { weight: 7.5, value: 'completed' },
    { weight: 1, value: 'expired' },
    { weight: 0.5, value: 'canceled' },
    { weight: 0.5, value: 'declined' },
    { weight: 0.5, value: 'error' },
]) // 7.5/10 chance of being completed, 1/10 chance of being expired, 0.5/10 chance of being canceled, declined or error
### Download the helper functions [¶](https://www.tinybird.co/docs/about:blank#download-the-helper-functions)

This script also utilizes a couple of helper functions to access your Tinybird Token and send the data to Tinybird with an HTTP request using the Tinybird Events API. These helper functions are located in the `tinybird.js` file in the repo. [Download that file](https://github.com/tinybirdco/demo-user-facing-saas-dashboard-signatures/blob/main/datagen/utils/tinybird.js) and add it to the `datagen/utils` directory.

##### Helper functions

cd datagen/utils
curl -O https://raw.githubusercontent.com/tinybirdco/demo-user-facing-saas-dashboard-signatures/refs/heads/main/datagen/utils/tinybird.js The Tinybird Events API is useful for two reasons:

1. It allows for the flexible and efficient ingestion of data, representing various stages of signatures, directly into the Tinybird platform without needing complex streaming infrastructure.
2. It allows you to stream events directly from your application instead of relying on batch ETLs or change data capture which requires the events to first be logged in a transactional database, which can add lag to the data pipeline.

### Install the Faker library [¶](https://www.tinybird.co/docs/about:blank#install-the-faker-library)

Run this command:

##### Install Faker

cd datagen
npm init --yes
npm install @faker-js/faker To run this file and start sending mock data to Tinybird, you need to create a custom script in the `package.json` generated file inside `datagen` folder. Open up that file and add the following to the scripts:

##### Add seed npm script

"seed": "node data-project/mockDataGenerator.js" Note that since your code is using ES modules, you'll need to add `"type": "module"` to the `package.json` file to be able to run the script and access the modules. For more information on why, [read this helpful post](https://www.codeconcisely.com/posts/nextjs-esm/).

Your package.json should now look something like this:

##### package.json

{
  "name": "datagen",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "type": "module",
  "scripts": {
    "seed": "node ./mockDataGenerator.js"
  },
  "dependencies": {
    "@faker-js/faker": "^8.4.1"
  },
  "license": "ISC",
  "author": ""
} Okay: You're ready to start sending mock data to Tinybird. Open up a new terminal tab or window in this local project directory, in the `datagen` folder run:

##### Generate mock data!

npm run seed Congratulations! You should see the seed output in your terminal. Let this run in the background so you have some data for the next steps. Return to your original terminal tab or window and move onto the next steps.

### Verify your mock data stream [¶](https://www.tinybird.co/docs/about:blank#verify-your-mock-data-stream)

To verify that the data is flowing properly into Tinybird, inspect the Tinybird Data Sources. In the Tinybird UI, navigate to the `signatures` and `accounts` [Data Sources](https://www.tinybird.co/docs/docs/get-data-in/data-sources) to confirm that the data has been received. The latest records should be visible.

You can also confirm using the CLI, by running a SQL command on your Data Source:

tb sql "select count() from signatures" If you run this a few times, and your mock data stream is still running, you'll see this number increase. Neat.

This project uses mock data streams to simulate data generated by a hypothetical document signatures app. If you have your own app that's generating data, you don't need to do this! You can just add the helper functions to your codebase and call them to send data directly from your app to Tinybird.

## 5. Build dashboard metrics with SQL [¶](https://www.tinybird.co/docs/about:blank#5-build-dashboard-metrics-with-sql)

You now have a Data Source: Events streaming into Tinybird, which ensures your real-time dashboard has access to fresh data. The next step is to build real-time metrics using [Tinybird Pipes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes).

A Pipe is a set of chained, composable nodes of SQL that process, transform, and enrich data in your Data Sources.

Create a new Pipe in the Tinybird UI by selecting the + icon in the left-hand nav bar and selecting "Pipe". Rename your new Pipe `ranking_of_top_organizations_creating_signatures`.

Next, time to make your first Node! Remove the placeholder text from the Node, and paste the following SQL in:

%
    SELECT 
      account_id,
      {% if defined(completed) %}
        countIf(status = 'completed') total
      {% else %}
        count() total
      {% end %}
    FROM signatures
    WHERE
        fromUnixTimestamp64Milli(timestamp)
        BETWEEN {{
            Date(
                date_from,
                '2023-01-01',
                description="Initial date",
                required=True,
            )
        }}
        AND {{
    Date(
      date_to,
       '2024-01-01',
       description="End date",
        required=True
    ) 
         }}
    GROUP BY account_id
    HAVING total > 0
    ORDER BY total DESC Key points to understand in this snippet:

1. As well as standard SQL, it uses the Tinybird[  templating language and query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters)   - you can tell when query params are used, because the `%`   symbol appears at the top of the query. This makes the query*  dynamic*   , so instead of hardcoding the date range, the user can now select a range and have the results refresh in real time.
2. It has an `if defined`   statement. In this case, if a boolean tag called `completed`   is passed, the Pipe calculates the number of completed signatures. Otherwise, it calculates all signatures.

Select "Run" to run and save this Node, then rename `retrieve_signatures` . Below this Node, create a second one. Remove the placeholder text and paste the following SQL in:

##### Second Node

SELECT
  organization,
  sum(total) AS org_total
FROM retrieve_signatures
LEFT JOIN accounts ON accounts.account_id = retrieve_signatures.account_id
GROUP BY organization
ORDER BY org_total DESC
LIMIT {{Int8(limit, 10, description="The number of rows accounts to retrieve", required=False)}} Name this node `endpoint` and select "Run" to save it.

You now have a 2-Node Pipe that gets the top `` number of organizations by signatures within a date range, either completed or total depending on whether a completed query parameter is passed or not.

## 6. Publish metrics as APIs [¶](https://www.tinybird.co/docs/about:blank#6-publish-metrics-as-apis)

You're now ready to build a low-latency, high-concurrency REST API Endpoint from your Pipe - with just 2 clicks!

Select the "Create API Endpoint" button at top right, then select the `endpoint` Node.

You'll be greeted with an API page that contains a usage monitoring chart, parameter documentation, and sample usage. In addition, the API has been secured through an automatically-generated, read-only Token.

### Test your API [¶](https://www.tinybird.co/docs/about:blank#test-your-api)

Copy the HTTP API Endpoint from the "Sample usage" box and paste it directly into a new browser tab to see the response. In the URL, you can manually adjust the `date_from` and `date_to` parameters and see the different responses. You can also adjust the `limit` parameter, which controls how many rows are returned.

If you request the data in a JSON format (the default behavior), you'll also receive some metadata about the response, including statistics about the query latency:

##### Example Tinybird API statistics

"statistics": {
    "elapsed": 0.001110996,
    "rows_read": 4738,
    "bytes_read": 101594
} You'll notice that the API response in this example took barely 1 millisecond (which is... pretty fast) so your dashboards are in good hands when it comes to being ultra responsive. When building out your own projects in the future, use this metadata [and Tinybird's other tools](https://www.tinybird.co/docs/docs/monitoring/health-checks) to monitor and optimize your dashboard query performance.

### Optional: Pull the Tinybird resources into your local directory [¶](https://www.tinybird.co/docs/about:blank#optional-pull-the-tinybird-resources-into-your-local-directory)

At this point, you've created a bunch of Tinybird resources: A Workspace, a Data Source, Pipes, and an API Endpoint. You can pull these resources down locally, so that you can manage this project with Git.

In your terminal, start by pulling the Tinybird data project:

##### In the root directory

tb pull --auto You'll see a confirmation that 3 resources ( `signatures.datasource`, `accounts.datasource` , and `ranking_of_top_organizations_creating_signatures.pipe` ) were written into two subfolders, `datasources` and `pipes` , which were created by using the `--auto` flag. Move them into the `data-project` directory:

##### Move to /tinybird directory

cd tinybird
mv datasources pipes tinybird/ As you add additional resources in the Tinybird UI, use the `tb pull –auto` command to pull files from Tinybird. You can then add them to your Git commits and push them to your remote repository.

If you create data project resources locally using the CLI, you can push them to the Tinybird server with `tb push` . For more information on managing Tinybird data projects in the CLI, check out [this CLI overview](https://www.tinybird.co/docs/docs/cli/quick-start).

## 7. Create real-time dashboard [¶](https://www.tinybird.co/docs/about:blank#7-create-real-time-dashboard)

Now that you have a low-latency API with real-time dashboard metrics, you're ready to create the visualization layer using Next.js and Tremor. These two tools provide a scalable and responsive interface that integrate with Tinybird's APIs to display data dynamically. Plus, they look great.

## Initialize your Next.js project [¶](https://www.tinybird.co/docs/about:blank#initialize-your-next-js-project)

In your terminal, create a folder call `app` and inside it create your Next.js app with this command. In this tutorial you'll use plain JavaScript files and Tailwind CSS:

##### Create a Next app

cd app
npx create-next-app . --js --tailwind --eslint --src-dir --app --import-alias "@/*"
### Add Tremor to your Next.js app [¶](https://www.tinybird.co/docs/about:blank#add-tremor-to-your-next-js-app)

You're going to use Tremor to create a simple bar chart that displays the signature count for each organization. Tremor provides stylish React chart components that you can deploy easily and customize as needed.

Install Tremor with the CLI:

##### Install Tremor

npx @tremor/cli@latest init Select Next as your framework and allow Tremor to overwrite your existing `tailwind.config.js`.

### Add SWR to your Next.js app [¶](https://www.tinybird.co/docs/about:blank#add-swr-to-your-next-js-app)

You're going to use [SWR](https://swr.vercel.app/) to handle the API Endpoint data and refresh it every 5 seconds. SWR is a great React library to avoid dealing with data caching and revalidating complexity on your own. Plus, you can define what refresh policy you want to follow. Take a look to [its docs](https://swr.vercel.app/docs/revalidation) to know different revalidation strategies.

##### Install SWR

npm i swr
### Set up environment variables [¶](https://www.tinybird.co/docs/about:blank#set-up-environment-variables)

Next, you need to add your Tinybird host and user admin Token as environment variables so you can run the project locally. Create a `.env.local` file in the root directory ( `/signatures_dashboard` ) and add the following:

##### Set up environment variables

NEXT_PUBLIC_TINYBIRD_HOST="YOUR TINYBIRD API HOST" # Your regional API host e.g. https://api.tinybird.co
NEXT_PUBLIC_TINYBIRD_TOKEN="YOUR SIGNING TOKEN" # Use your Admin Token as the signing token

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

### Set up your page.js [¶](https://www.tinybird.co/docs/about:blank#set-up-your-page-js)

Next.js created a `page.js` as part of the bootstrap process. Open it in your preferred code editor and clear the contents. Paste in the snippets in order from the following sections, understanding what each one does:

### Import UI libraries [¶](https://www.tinybird.co/docs/about:blank#import-ui-libraries)

To build your dashboard component, you will need to import various UI elements and functionalities from the libraries provided at the beginning of your file. Note the use of the `use client;` directive to render the components on the client side. For more details on this, check out the [Next.js docs](https://nextjs.org/docs/app/building-your-application/rendering#network-boundary).

##### Start building index.js

"use client";

import { BarChart, Card, Subtitle, Text, Title } from "@tremor/react";
import React from "react";
import useSWR from "swr";
### Define constants [¶](https://www.tinybird.co/docs/about:blank#define-constants)

Inside your main component, define the constants required for this specific component:

##### Add environment variables and states

// Get your Tinybird host and Token from the .env file
const TINYBIRD_HOST = process.env.NEXT_PUBLIC_TINYBIRD_HOST; // The host URL for the Tinybird API
const TINYBIRD_TOKEN = process.env.NEXT_PUBLIC_TINYBIRD_TOKEN; // The access Token for authentication with the Tinybird API
const REFRESH_INTERVAL_IN_MILLISECONDS = 5000; // five seconds
### Connect your dashboard to your Tinybird API [¶](https://www.tinybird.co/docs/about:blank#connect-your-dashboard-to-your-tinybird-api)

You'll need to write a function to fetch data from Tinybird. Note that for the sake of brevity, this snippet hardcodes the dates and uses the default limit in the Tinybird API. You could set up a Tremor datepicker and/or number input if you wanted to dynamically update the dashboard components from within the UI.

##### Define query parameters and Tinybird fetch function

export default function Dashboard() {
  // Define date range for the query
  const today = new Date(); // Get today's date
  const dateFrom = new Date(today.setMonth(today.getMonth() - 1)); // Start the query's dateFrom to the one month before today
  const dateTo = new Date(today.setMonth(today.getMonth() + 1)); // Set the query's dateTo to be one month from today

  // Format for passing as a query parameter
  const dateFromFormatted = dateFrom.toISOString().substring(0, 10);
  const dateToFormatted = dateTo.toISOString().substring(0, 10);

  // Constructing the URL for fetching data, including host, token, and date range
  const endpointUrl = new URL(
    "/v0/pipes/ranking_of_top_organizations_creating_signatures.json",
    TINYBIRD_HOST
  );
  endpointUrl.searchParams.set("token", TINYBIRD_TOKEN);
  endpointUrl.searchParams.set("date_from", dateFromFormatted);
  endpointUrl.searchParams.set("date_to", dateToFormatted);

  // Initializes variables for storing data
  let ranking_of_top_organizations_creating_signatures, latency, errorMessage;

  try {
    // Function to fetch data from Tinybird URL and parse JSON response
    const fetcher = (url) => fetch(url).then((r) => r.json());

    // Using SWR hook to handle state and refresh result every five seconds
    const { data, error } = useSWR(endpointUrl.toString(), fetcher, {
      refreshInterval: REFRESH_INTERVAL_IN_MILLISECONDS,
    });

    if (error) {
      errorMessage = error;
      return;
    }
    if (!data) return;

    if (data?.error) {
      errorMessage = data.error;
      return;
    }

    ranking_of_top_organizations_creating_signatures = data.data; // Setting the state with the fetched data
    latency = data.statistics?.elapsed; // Setting the state with the query latency from Tinybird
  } catch (e) {
    console.error(e);
    errorMessage = e;
  }
### Render the Component [¶](https://www.tinybird.co/docs/about:blank#render-the-component)

Finally, include the rendering code to display the "Ranking of the top organizations creating signatures" in the component's return statement:

##### Render the dashboard component

return (
    
      Top Organizations Creating Signatures
      Ranked from highest to lowest
      {ranking_of_top_organizations_creating_signatures && (
        
      )}
      {latency && Latency: {latency * 1000} ms}
      {errorMessage && (
        

          

            Oops, something happens: {errorMessage}
          

          
Check your console for more information

        

      )}
    
  );
}
### View your dashboard! [¶](https://www.tinybird.co/docs/about:blank#view-your-dashboard)

It's time! Run `npm run dev` and navigate to `http://localhost:3000/` in your browser. You should see something like this:



<-figure->
![Diagram showing the data flow from Tinybird --> Next.js --> Tremor](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-real-time-dashboard-data-flow.png&w=3840&q=75)


Congratulations! You've created a real-time dashboard component using Tinybird, Tremor, and Next.js. You'll notice the dashboard is rendering very quickly by taking a peek at the latency number below the component. In this example case, Tinybird returned the data for the dashboard in a little over 40 milliseconds aggregating over about a million rows. Not too bad for a relatively un-optimized query!

### Optional: Expand your dashboard [¶](https://www.tinybird.co/docs/about:blank#optional-expand-your-dashboard)

You've got the basics: An active Workspace and Data Source, knowledge of how to build Pipes, and access to the [Tremor docs](https://www.tremor.so/docs/getting-started/installation) . Build out some more Pipes, API Endpoints, and visualizations!



<-figure->
![Dashboard showing more visualizations](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-real-time-dashboard-further-examples.png&w=3840&q=75)


You can also spend some time [optimizing your data project](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) for faster responses and minimal data processing using fine-tuned indexes, [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) , and more.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Investigate the[  GitHub repository for this project](https://github.com/tinybirdco/demo-user-facing-saas-dashboard-signatures)   in more depth.
- Understand today's real-time analytics landscape with[  Tinybird's definitive guide](https://www.tinybird.co/blog-posts/real-time-analytics-a-definitive-guide)  .
- Learn how to implement[  multi-tenant security](https://www.tinybird.co/blog-posts/multi-tenant-saas-options)   in your user-facing analytics.



---

URL: https://www.tinybird.co/docs/publish/charts/guides/charts-using-iframes-and-jwt-tokens
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Build charts with iframes and JWTs. · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird Charts make it easy to create interactive charts. In this guide, you'll learn how build them using iframes and JWTs."
---




# Build charts with iframes and JWTs [¶](https://www.tinybird.co/docs/about:blank#build-charts-with-iframes-and-jwts)

In this guide, you'll learn how to build Tinybird Charts using inline frames (iframes) and secure them using JSON Web Tokens (JWTs)

[Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts) make it easy to visualize your data and create interactive charts. As soon as you've published an API Endpoint, you can create a Chart from the Tinybird UI, then immediately embed it in your frontend application. Check out the [live demo](https://guide-tinybird-charts.vercel.app/) to see an example of Charts in action.

[JWTs](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#json-web-tokens-jwts) are signed tokens that allow you to securely authorize and share data between your application and Tinybird.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes that you have a Tinybird Workspace with active data and one or more published API Endpoints.

You'll need a basic familiarity with JavaScript and Python.

## 1. Create a Chart [¶](https://www.tinybird.co/docs/about:blank#1-create-a-chart)

[Create a new Chart](https://www.tinybird.co/docs/docs/publish/charts) based on any of your API Endpoints.

## 2. View the Chart component code [¶](https://www.tinybird.co/docs/about:blank#2-view-the-chart-component-code)

After saving your Chart, you're on the API Endpoint "Overview" page. Your Chart should be visible in the "Output" section.

To the view the component code for the Chart, select the code symbol ( `<>` ) above it. Select the dropdown and select the iframe example, instead of the default React one.

Now select the JWT so the Static Token shown in the iframe's URL is replaced by a `` placeholder.



<-figure->
![Get the iframe code](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-iframes-jwts__get-iframe-code.png&w=3840&q=75)


## 3. Insert your iframe into a new page [¶](https://www.tinybird.co/docs/about:blank#3-insert-your-iframe-into-a-new-page)

Now you have your Chart code, create a new `index.html` file to paste the code into:

##### index.html




    
    
    Tinybird Charts


    
    


 In the next step, you'll generate a JWT to replace the `` placeholder.

## 4. Create a JWT [¶](https://www.tinybird.co/docs/about:blank#4-create-a-jwt)

### Understanding the token exchange [¶](https://www.tinybird.co/docs/about:blank#understanding-the-token-exchange)



<-figure->
![Generate a new token](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fguides-iframes-jwts__token-exchange.png&w=3840&q=75)


For each user session (or any other approach you want to follow), your frontend application will send a request with a JWT to your backend. It can be a new or an existing one. Your backend will self-sign and return the token. From that point onwards, you can use this token for any Chart or API call made to Tinybird, directly from your frontend application.

JWTs support TTL and includes multi-tenancy capabilities, which makes them safe to use without creating any complex middleware.

### Create a JWT endpoint [¶](https://www.tinybird.co/docs/about:blank#create-a-jwt-endpoint)

Create a new endpoint that your frontend will use to retrieve your token. Remember to set your `TINYBIRD_SIGNING_KEY`:

##### Generate a new token

from flask import Flask, jsonify, render_template
import jwt
import datetime
import os

app = Flask(__name__)

# Get your Tinybird admin Token
TINYBIRD_SIGNING_KEY= # Use your admin Token as the signing key, or use process.env.TB_TOKEN / similar if you have it set locally

# Generate Token function for a specific pipe_id
def generate_jwt():
    expiration_time = datetime.datetime.utcnow() + datetime.timedelta(hours=48)
    workspace_id = "1f484a32-6966-4f63-9312-aadad64d3e12"
    token_name = "charts_token"
    pipe_id = "t_b9427fe2bcd543d1a8923d18c094e8c1"

    payload = {
        "workspace_id": workspace_id,
        "name": token_name,
        "exp": expiration_time,
        "scopes": [
            {
                "type": "PIPES:READ",
                "resource": pipe_id
            },
        ],
    }

    return jwt.encode(payload, TINYBIRD_SIGNING_KEY, algorithm='HS256')

@app.route('/generate-token', methods=['GET'])
def get_token():
    token = generate_jwt()
    return jsonify({"token": token})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5151)
## 5. Use the JWT in your iframe [¶](https://www.tinybird.co/docs/about:blank#5-use-the-jwt-in-your-iframe)

Edit your `index.html` file using JavaScript to retrieve a JWT from your API Endpoint, and include this token in your iframe.

##### Update the index.html file




    
    
    Tinybird Charts


    
    

    


## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about[  JSON Web Tokens (JWTs)](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#json-web-tokens-jwts)
- Learn more about[  Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts)
- [  Consume APIs in a Next.js frontend using JWTs](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-apis-nextjs)
- [  Add Tinybird Charts to a Next.js frontend](https://www.tinybird.co/docs/docs/publish/charts/guides/add-charts-to-nextjs)



---

URL: https://www.tinybird.co/docs/publish/charts/guides/bigquery-dashboard
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Build user-facing dashboard with BigQuery · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to build a user-facing dashboard using Tinybird and BigQuery."
---




# Build a user-facing analytics dashboard with BigQuery and Tinybird [¶](https://www.tinybird.co/docs/about:blank#build-a-user-facing-analytics-dashboard-with-bigquery-and-tinybird)

In this guide you'll learn how to take data from BigQuery and build a user-facing analytics dashboard using Tinybird, Next.js, and Tremor components. You'll end up with a dashboard and enough familiarity with Tremor to adjust the frontend & data visualization for your own projects in the future.

Google BigQuery is a serverless data warehouse, offering powerful online analytical processing (OLAP) computations over large data sets with a familiar SQL interface. Since its launch in 2010, it's been widely adopted by Google Cloud users to handle long-running analytics queries to support strategic decision-making through business intelligence (BI) visualizations.

Sometimes, however, you want to extend the functionality of your BigQuery data beyond business intelligence: For instance, real-time data visualizations that can be integrated into user-facing applications. As outlined in [the Tinybird blog post on BigQuery dashboard options](https://www.tinybird.co/blog-posts/bigquery-real-time-dashboard) , you can build Looker Studio dashboards over BigQuery data, but they'll struggle to support user-facing applications that require high concurrency, fresh data, and low-latency API responses. Tinybird is the smart option for fast and real-time.

Let's get building!

[GitHub Repository](https://github.com/tinybirdco/bigquery-dashboard)

<-figure->
![Analytics dashboard build with BigQuery data, Tinybird Endpoints, and Tremor components in a Next.js app](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-bigquery-dashboard.png&w=3840&q=75)


Imagine you're a ***huge*** baseball fan. You want to build a real-time dashboard that aggregates up-to-the-moment accurate baseball stats from teams around the world, and gives you the scoop on all your favorite players. This tutorial explains how to build a really nice-looking prototype version.

In this tutorial, you'll learn how to:

1. Ingest your existing BigQuery data into Tinybird.
2. Process and transform that data with accessible SQL.
3. Publish the transformations as real-time APIs.
4. Use Tremor components in a Next.js app to build a clean, responsive, real-time dashboard that consumes those API Endpoints.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you'll need:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. A BigQuery account
3. Node.js >=20.11

This tutorial includes a [Next.js](https://nextjs.org/) app and [Tremor](https://www.tremor.so/components) components for frontend visualization, but you don't need working familiarity with TypeScript or JavaScript - just copy & paste the code snippets.

## 1. Create a Tinybird Workspace [¶](https://www.tinybird.co/docs/about:blank#1-create-a-tinybird-workspace)

Navigate to the Tinybird web UI ( [app.tinybird.co](https://app.tinybird.co/) ) and create an empty Tinybird Workspace (no starter kit) called `bigquery_dashboard` in your preferred region.

## 2. Connect your BigQuery dataset to Tinybird [¶](https://www.tinybird.co/docs/about:blank#2-connect-your-bigquery-dataset-to-tinybird)

To get your BigQuery data into Tinybird, you'll use the [Tinybird BigQuery Connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/bigquery).

Download [this sample dataset](https://github.com/tinybirdco/bigquery-dashboard/blob/main/baseball_stats.csv) that contains 20,000 rows of fake baseball stats. Upload it to your BigQuery project as a new CSV dataset.

Next, follow the [steps in the documentation](https://www.tinybird.co/docs/docs/get-data-in/connectors/bigquery#load-a-bigquery-table-in-the-ui) to authorize Tinybird to view your BigQuery tables, select the table you want to sync, and set a sync schedule. Call the Data Source `baseball_game_stats`.

Tinybird will copy the contents of your BigQuery table into a Tinybird Data Source and ensure the Data Source stays in sync with your BigQuery table.

Tinybird can sync BigQuery tables as often as every 5 minutes. If you need fresher data in your real-time dashboards, consider sending data to Tinybird via alternative sources such as [Apache Kafka](https://www.tinybird.co/docs/docs/get-data-in/connectors/kafka), [Confluent Cloud](https://www.tinybird.co/docs/docs/get-data-in/connectors/confluent), [Google Pub/Sub](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-google-pubsub) , or Tinybird's native [HTTP streaming endpoint](https://www.tinybird.co/docs/docs/get-data-in/guides/ingest-from-the-events-api).

## 3. Create some Pipes [¶](https://www.tinybird.co/docs/about:blank#3-create-some-pipes)

In Tinybird, a [Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) is a transformation definition comprised of a series of SQL statements. You can build metrics through a series of short, composable [nodes](https://www.tinybird.co/docs/docs/work-with-data/query/pipes#nodes) of SQL. Think of Pipes as a way to build SQL queries without always needing to write common table expressions or subqueries, as these can be split out into reusable, independent nodes.

For example, here's a simple single-Node Pipe definition that calculates the season batting average for each player:

##### player_batting_percentages.pipe

SELECT
  player_name AS "Player Name",
  sum(stat_hits)/sum(stat_at_bats) AS "Batting Percentage"
FROM baseball_game_stats
GROUP BY "Player Name"
ORDER BY "Batting Percentage" DESC Create your first Pipe from your newly-created BigQuery Data Source by selecting “Create Pipe” in the top right corner of the Tinybird UI. Paste in the SQL above and run the query. Rename the Pipe `player_batting_percentages`.

Naming your Pipe something descriptive is important, as the Pipe name will be used as the URL slug for your API Endpoint later on.

## 4. Extend Pipes with Query Parameters [¶](https://www.tinybird.co/docs/about:blank#4-extend-pipes-with-query-parameters)

Every good dashboard is interactive. You can make your Tinybird queries interactive using Tinybird's templating language to [generate query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters) . In Tinybird, you add query parameters using `{{(,}}` , defining the data type of the parameter, its name, and an optional default value.

For example, you can extend the SQL query in the previous step to dynamically change the number of results returned from the Pipe, by using a `limit` parameter and a default value of 10:

##### player_batting_percentages.pipe plus query parameters

SELECT
  player_name AS "Player Name",
  sum(stat_hits)/sum(stat_at_bats) AS "Batting Percentage"
FROM baseball_game_stats
GROUP BY "Player Name"
ORDER BY "Batting Percentage" DESC
LIMIT {{UInt16(limit, 10, description="The number of results to display")}} Replace the SQL in your Pipe with this code snippet. Run the query and rename the node `endpoint`.

The `%` character at the start of a Tinybird SQL query shows there's a [dynamic query parameter](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters#define-dynamic-parameters) coming up.

## 5. Publish your Pipes as APIs [¶](https://www.tinybird.co/docs/about:blank#5-publish-your-pipes-as-apis)

The magic of Tinybird is that you can instantly publish your Pipes as fully-documented, scalable REST APIs instantly. From the Pipe definition in the Tinybird UI, select “Create API Endpoint” in the top right corner, select the `endpoint` Node.

Congratulations! You just ingested BigQuery data, transformed it, and published it as a Tinybird API Endpoint!

### Create additional Pipes [¶](https://www.tinybird.co/docs/about:blank#create-additional-pipes)

Create these additional 5 Pipes (they can also be found in the [project repository](https://github.com/tinybirdco/bigquery-dashboard/tree/main/data-project/pipes) ). Rename them as they are titled in each snippet, and call each node `endpoint` . Read through the SQL to get a sense of what each query does, then run and publish each one as its own API Endpoint:

##### batting_percentage_over_time

%
SELECT
    game_date AS "Game Date",
    sum(stat_hits)/sum(stat_at_bats) AS "Batting Percentage"
FROM baseball_game_stats
WHERE player_team = {{String(team_name, 'BOS', required=True)}}
GROUP BY "Game Date"
ORDER BY "Game Date" ASC
##### most_hits_by_type

%
SELECT player_name AS name, sum({{ column(hit_type, 'stat_hits') }}) AS value
FROM baseball_game_stats
GROUP BY name
ORDER BY value DESC
LIMIT 7
##### opponent_batting_percentages

%
SELECT game_opponent AS "Team", sum(stat_hits) / sum(stat_at_bats) AS "Opponent Batting Percentage"
FROM baseball_game_stats
GROUP BY "Team"
ORDER BY "Opponent Batting Percentage" ASC
LIMIT {{ UInt16(limit, 10) }}
##### player_batting_percentages

%
SELECT
    player_name AS "Player Name",
    sum(stat_hits)/sum(stat_at_bats) AS "Batting Percentage"
FROM baseball_game_stats
GROUP BY "Player Name"
ORDER BY "Batting Percentage" DESC
LIMIT {{UInt16(limit, 10)}}
##### team_batting_percentages

%
SELECT player_team AS "Team", sum(stat_hits) / sum(stat_at_bats) AS "Batting Percentage"
FROM baseball_game_stats
GROUP BY "Team"
ORDER BY "Batting Percentage" DESC
LIMIT {{ UInt16(limit, 10) }} 1 Data Source, 6 Pipes: Perfect. Onto the next step.

## 6. Create a Next.js app [¶](https://www.tinybird.co/docs/about:blank#6-create-a-next-js-app)

This tutorial uses Next.js, but you can visualize Tinybird APIs just about anywhere, for example with an app-building tool like [Retool](https://www.tinybird.co/blog-posts/service-data-sources-and-retool) or a monitoring platform like [Grafana](https://www.tinybird.co/blog-posts/tinybird-grafana-plugin-launch).

In your terminal, create a project folder and inside it create your Next.js app, using all the default options:

##### Create a Next app

mkdir bigquery-tinybird-dashboard
cd bigquery-tinybird-dashboard
npx create-next-app Tinybird APIs are accessible via [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) . In order to run your dashboard locally, you'll need to create a `.env.local` file at the root of your new project:

##### Create .env.local at root of my-app

touch .env.local And include the following:

##### Set up environment variables

NEXT_PUBLIC_TINYBIRD_HOST="YOUR TINYBIRD API HOST" # Your regional API host e.g. https://api.tinybird.co
NEXT_PUBLIC_TINYBIRD_TOKEN="YOUR SIGNING TOKEN" # Use your Admin Token as the signing token

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

## 7. Define your APIs in code [¶](https://www.tinybird.co/docs/about:blank#7-define-your-apis-in-code)

To support the dashboard components you're about to build, it's a great idea to create a helper file that contains all your Tinybird API references. In the project repo, that's called `tinybird.js` and it looks like this:

##### tinybird.js helper file

const playerBattingPercentagesURL = (host, token, limit) =>
    `https://${host}/v0/pipes/player_batting_percentages.json?limit=${limit}&token=${token}`

const teamBattingPercentagesURL = (host, token, limit) =>
    `https://${host}/v0/pipes/team_batting_percentages.json?limit=${limit}&token=${token}`

const opponentBattingPercentagesURL = (host, token, limit) =>
    `https://${host}/v0/pipes/opponent_batting_percentages.json?limit=${limit}&token=${token}`

const battingPercentageOverTimeURL = (host, token, team_name) =>
    `https://${host}/v0/pipes/batting_percentage_over_time.json?team_name=${team_name}&token=${token}`

const hitsByTypeURL = (host, token, hit_type) =>
    `https://${host}/v0/pipes/most_hits_by_type.json?hit_type=${hit_type}&token=${token}`

const fetchTinybirdUrl = async (fetchUrl, setData, setLatency) => {
    const data = await fetch(fetchUrl)
    const jsonData = await data.json();
    setData(jsonData.data);
    setLatency(jsonData.statistics.elapsed)
}

export {
    fetchTinybirdUrl,
    playerBattingPercentagesURL,
    teamBattingPercentagesURL,
    opponentBattingPercentagesURL,
    battingPercentageOverTimeURL,
    hitsByTypeURL
} Inside `/src/app` , create a new subfolder called `/services` and paste the snippet into a new `tinybird.js` helper file.

## 8. Build your dashboard components [¶](https://www.tinybird.co/docs/about:blank#8-build-your-dashboard-components)

This tutorial uses the [Tremor React library](https://tremor.so/) because it provides a clean UI out of the box with very little code. You could easily use [ECharts](https://echarts.apache.org/en/index.html) or something similar if you prefer.

### Add Tremor to your Next.js app [¶](https://www.tinybird.co/docs/about:blank#add-tremor-to-your-next-js-app)

You're going to use Tremor to create a simple bar chart that displays the signature count for each organization. Tremor provides stylish React chart components that you can deploy easily and customize as needed.

Inside your app folder, install Tremor with the CLI:

##### Install Tremor

npx @tremor/cli@latest init Select Next as your framework and allow Tremor to overwrite your existing `tailwind.config.js`.

### Create dashboard component files [¶](https://www.tinybird.co/docs/about:blank#create-dashboard-component-files)

Your final dashboard contains 3 Bar Charts, 1 Area Chart, and 1 Bar List. You'll use Tremor Cards to display these components, and each one will have an interactive input. In addition, you'll show the API response latency underneath the Chart (just so you can show off about how “real-timey” the dashboard is).

Here's the code for the Player Batting Averages component ( `playerBattingPercentages.js` ). It sets up the file, defines the limit parameters, then renders the Chart components:

"use-client";

import { Card, Title, Subtitle, BarChart, Text, NumberInput, Flex } from '@tremor/react'; // Tremor components
import React, { useState, useEffect } from 'react';
import {fetchTinybirdUrl, playerBattingPercentagesURL } from '../services/tinybird.js' // Tinybird API

// utilize useState/useEffect to get data from Tinybird APIs on change
const PlayerBattingPercentages = ({host, token}) => {
    const [player_batting_percentages, setData] = useState([{
        "Player Name": "",
        "Batting Percentage": 0,
    }]);

    // set latency from the API response
    const [latency, setLatency] = useState(0

    // set limit parameter when the component input is changed
    const [limit, setLimit] = useState(10);

    // format the numbers on the component
    const valueFormatter = (number) => `${new Intl.NumberFormat("us").format(number).toString()}`;

    // set the Tinybird API URL with query parameters
    let player_batting_percentages_url = playerBattingPercentagesURL(host, token, limit)


    useEffect(() => {
        fetchTinybirdUrl(player_batting_percentages_url, setData, setLatency)
    }, [player_batting_percentages_url]);

    // build the Tremor component
    return (
        
            
                

                    Player Batting Percentages
                    All Players
                

                

                    # of Results
                     setLimit(value)}
                    />
                

            
            // Build the bar chart with the data received from the Tinybird API
            
            Latency: {latency*1000} ms // Add the latency metric
        
    );
};

export default PlayerBattingPercentages; In the project repo, you'll find the 5 dashboard components you need, inside the `src/app/components` directory. Each one renders a dashboard component to display the data received by one of the Tinybird APIs.

It's time to build them out. For this tutorial, just recreate the same files in your app, pasting in the JavaScript (or downloading the files and dropping them in to your app directory). When building your own dashboard in future, use this as a template and build to fit your needs!

## 9. Compile components into a dashboard [¶](https://www.tinybird.co/docs/about:blank#9-compile-components-into-a-dashboard)

Final step! Update your `page.tsx` file to render a nicely-organized dashboard with your 5 components: Replace the contents of `page.tsx` with [this file](https://github.com/tinybirdco/bigquery-dashboard/blob/main/src/app/page.js).

The logic in this page gets your Tinybird Token from your local environment variables to be able to access the Tinybird APIs, then renders the 5 components you just built in a Tremor [Grid](https://blocks.tremor.so/blocks/grid-lists).

To visualize your dashboard, run it locally with `npm run dev` and open http://localhost:3000. You'll see your complete real-time dashboard!



<-figure->
![Analytics dashboard build with BigQuery data, Tinybird Endpoints, and Tremor components in a Next.js app](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-bigquery-dashboard.png&w=3840&q=75)


Notice the latencies in each dashboard component. This is the Tinybird API request latency. This isn't using any sort of cache or query optimization; each request is directly querying the 20,000 rows in the table and returning a response. As you interact with the dashboard and change inputs, the APIs respond. In this case, that's happening in just a few milliseconds. Now ***that's*** a fast dashboard.

### Optional: Expand your dashboard [¶](https://www.tinybird.co/docs/about:blank#optional-expand-your-dashboard)

You've got the basics: An active Workspace and Data Source, knowledge of how to build Pipes, and access to the [Tremor docs](https://www.tremor.so/docs/getting-started/installation) . Build out some more Pipes, API Endpoints, and visualizations!

You can also spend some time [optimizing your data project](https://www.tinybird.co/docs/docs/work-with-data/query/sql-best-practices) for faster responses and minimal data processing using fine-tuned indexes, [Materialized Views](https://www.tinybird.co/docs/docs/work-with-data/process-and-copy/materialized-views) , and more.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Investigate the[  GitHub repository for this project](https://github.com/tinybirdco/bigquery-dashboard)   in more depth.
- Understand today's real-time analytics landscape with[  Tinybird's definitive guide](https://www.tinybird.co/blog-posts/real-time-analytics-a-definitive-guide)  .
- Learn how to implement[  multi-tenant security](https://www.tinybird.co/blog-posts/multi-tenant-saas-options)   in your user-facing analytics.



---

URL: https://www.tinybird.co/docs/publish/charts/guides/analytics-with-confluent
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Build user-facing analytics apps with Confluent · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to build a user-facing web analytics application with Confluent and Tinybird."
---




# Build a user-facing web analytics application with Confluent and Tinybird [¶](https://www.tinybird.co/docs/about:blank#build-a-user-facing-web-analytics-application-with-confluent-and-tinybird)

In this guide you'll learn how to take data from Kafka and build a user-facing web analytics dashboard using Confluent and Tinybird.

[GitHub Repository](https://github.com/tinybirdco/demo_confluent_charts/tree/main)

<-figure->
![Tinybird Charts showing e-commerce events](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-confluent-chart-1.png&w=3840&q=75)


In this tutorial, you will learn how to:

1. Connect Tinybird to a Kafka topic.
2. Build and publish Tinybird API Endpoints using SQL.
3. Create 2 Charts without having to code from scratch.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

To complete this tutorial, you'll need:

1. A[  free Tinybird account](https://www.tinybird.co/signup)
2. An empty Tinybird Workspace
3. A Confluent account
4. Node.js >=20.11

This tutorial includes a [Next.js](https://nextjs.org/) app for frontend visualization, but you don't need working familiarity with TypeScript - just copy & paste the code snippets.

## 1. Setup [¶](https://www.tinybird.co/docs/about:blank#1-setup)

Clone the `demo_confluent_charts` repo.

## 2. Create your data [¶](https://www.tinybird.co/docs/about:blank#2-create-your-data)

### Option 1: Use your own existing data [¶](https://www.tinybird.co/docs/about:blank#option-1-use-your-own-existing-data)

In Confluent, create a Kafka topic with simulated e-commerce events data. Check [this file](https://github.com/tinybirdco/demo_confluent_charts/blob/main/tinybird/datasources/ecomm_events.datasource) for the schema outline to follow.

### Option 2: Mock the data [¶](https://www.tinybird.co/docs/about:blank#option-2-mock-the-data)

Use Tinybird's [Mockingbird](https://mockingbird.tinybird.co/docs) , an open source mock data stream generator, to stream mock web events instead.

In the repo, navigate to `/datagen` and run `npm i` to install the dependencies.

Create an `.env` and replace the default Confluent variables:

cp .env.example .env Run the mock generator script:

node mockConfluent.js
## 3. Connect Confluent <> Tinybird [¶](https://www.tinybird.co/docs/about:blank#3-connect-confluent-tinybird)

In your Tinybird Workspace, create a new [Data Source](https://www.tinybird.co/docs/docs/get-data-in/data-sources) using the native [Confluent connector](https://www.tinybird.co/docs/docs/get-data-in/connectors/confluent) . Paste in the bootstrap server, rename the connection to `tb_confluent` , then paste in your API key and secret. Select "Next".

Search for and select your topic, and select "Next". Ingest from the earliest offset, then under "Advanced settings" > "Sorting key" select `timestamp`.

Rename the Data Source to `ecomm_events` and select "Create". Your Data Source is now ready, and you've connected Confluent to Tinybird! You now have something that's like a database table and a Kafka consumer ***combined*** . Neat.

## 4. Transform your data [¶](https://www.tinybird.co/docs/about:blank#4-transform-your-data)

### Query your data stream [¶](https://www.tinybird.co/docs/about:blank#query-your-data-stream)

Your data should now be streaming in, so let's do something with it. In Tinybird, you can transform data using straightforward SQL in chained nodes that form a Pipe.

Create a new [Pipe](https://www.tinybird.co/docs/docs/work-with-data/query/pipes) and rename it `sales_trend` . In the first node space, paste the following SQL:

SELECT timestamp, sales FROM ecomm_events
WHERE timestamp >= now() - interval 7 day This gets the timestamp and sales from just the last 7 days.

Run the query and rename the node `filter_data`.

In the second node space, paste the following:

SELECT toDate(timestamp) AS ts, sum(sales) AS total_sales from filter_data
GROUP BY ts
ORDER BY ts This casts the timestamp to a date as `ts` , and sums up the sales - meaning you can get a trend of sales by day.

Run the query and rename the node `endpoint`.

### Publish your transformed data [¶](https://www.tinybird.co/docs/about:blank#publish-your-transformed-data)

In the top right of the screen, select "Create API Endpoint" and select the `endpoint` Node. Congratulations! It's published and ready to be consumed.

## 5. Create a Tinybird Chart [¶](https://www.tinybird.co/docs/about:blank#5-create-a-tinybird-chart)

In the top right of the screen, select "Create Chart".

Rename the chart "Sales Trend" and select and Area Chart. Under the "Data" tab, select `ts` as the index and `total_sales` as the category.

You should see a Chart magically appear! In the top right of the screen, select "Save".

## 6. Run an app locally [¶](https://www.tinybird.co/docs/about:blank#6-run-an-app-locally)

View the component code for the Chart by selecting the code symbol ( `<>` ) above it. Copy this code and paste into a new file in the `components` folder called `SalesTrend.tsx`.

In `page.tsx` , replace `
Chart 1
` with your new Chart `` . Save and view in the browser with `npm run dev` . You should see your Chart!

### Create a second Pipe --> Chart [¶](https://www.tinybird.co/docs/about:blank#create-a-second-pipe-chart)

Create a second Pipe in Tinybird called `utm_sales`:

SELECT utm_source, sum(sales) AS total_sales FROM ecomm_events
WHERE timestamp >= now() - interval 7 day
GROUP BY utm_source
ORDER BY total_sales DESC This gets sales by utm over the last 7 days.

Run the query and rename the node `endpoint` .... Then, you guessed it! Publish it as an Endpoint, create a Chart, and get the code. This time, create a donut Chart called "UTM Sales" with `utm_source` as the index and `total_sales` as the category.

Check the "Legend" box and play around with the colors to create clear differentiators.

Create a new component file called `UTMSales.tsx` and import in `page.tsx` replacing Chart 2. You did it!



<-figure->
![Tinybird Charts showing e-commerce events](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Ftutorial-confluent-chart-1.png&w=3840&q=75)


## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read more about[  Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts)  .
- Use Charts internally to[  monitor latency](https://www.tinybird.co/docs/docs/monitoring/latency#how-to-visualize-latency)   in your own Workspace.



---

URL: https://www.tinybird.co/docs/publish/charts/guides/add-charts-to-nextjs
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Add Tinybird Charts to a Next.js frontend · Tinybird Docs"
theme-color: "#171612"
description: "Tinybird Charts make it easy to create interactive charts. In this guide, we'll show you how to add Tinybird Charts to a Next.js frontend."
---




# Add Tinybird Charts to a Next.js frontend [¶](https://www.tinybird.co/docs/about:blank#add-tinybird-charts-to-a-next-js-frontend)

In this guide, you'll learn how to generate create Tinybird Charts from the UI, and add them to your Next.js frontend.

Tinybird Charts make it easy to visualize your data and create interactive charts. You can create a chart from the UI, and then embed it in your frontend application. This guide will show you how to add Tinybird Charts to a Next.js frontend.

You can view the [live demo](https://guide-tinybird-charts.vercel.app/) or browse the [GitHub repo (guide-tinybird-charts)](https://github.com/tinybirdco/guide-tinybird-charts).



<-figure->
![Tinybird charts demo](https://www.tinybird.co/docs/docs/_next/image?url=%2Fdocs%2Fimg%2Fcharts-demo.png&w=3840&q=75)


## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes that you have a Tinybird account, and you are familiar with creating a Tinybird Workspace and pushing resources to it.

You'll need a working familiarity with JavaScript and Next.js.

## Run the demo [¶](https://www.tinybird.co/docs/about:blank#run-the-demo)

These steps cover running the GitHub demo locally. [Skip to the next section](https://www.tinybird.co/docs/about:blank#build-from-scratch) to build the demo from scratch.

### 1. Clone the GitHub repo [¶](https://www.tinybird.co/docs/about:blank#1-clone-the-github-repo)

Clone the [GitHub repo (guide-tinybird-charts)](https://github.com/tinybirdco/guide-tinybird-charts) to your local machine.

### 2. Push Tinybird resources [¶](https://www.tinybird.co/docs/about:blank#2-push-tinybird-resources)

The repo contains a `tinybird` folder which includes sample Tinybird resources:

- `events.datasource`   : The Data Source for incoming events.
- `airline_market_share.pipe`   : An API Endpoint giving a count of bookings per airline.
- `bookings_over_time.pipe`   : An API Endpoint giving a time series of booking volume over time.
- `bookings_over_time_by_airline.pipe`   : An API Endpoint giving a time series of booking volume over time with an `airline`   filter.
- `meal_choice_distribution.pipe`   : An API Endpoint giving a count of meal choices across bookings.
- `top_airlines.pipe`   : An API Endpoint giving a list of the top airlines by booking volume.

Make a new Tinybird Workspace in the region of your choice. Then, configure the [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/install) (install and authenticate) and `tb push` the resources to your Workspace. Alternatively, you can drag and drop the files onto the UI to upload them.

### 3. Generate some fake data [¶](https://www.tinybird.co/docs/about:blank#3-generate-some-fake-data)

Use [Mockingbird](https://tbrd.co/mockingbird-tinybird-charts-guide) to generate fake data for the `events` Data Source.

Using this link ^ provides a pre-configured schema, and you'll just need to enter your Workspace Admin Token and select the Host region that matches your Workspace. When configured, select `Save` , then scroll down and select `Start Generating!`.

In the Tinybird UI, confirm that the `events` Data Source is successfully receiving data.

### 4. Install dependencies [¶](https://www.tinybird.co/docs/about:blank#4-install-dependencies)

In the cloned repo, navigate to `/app` and install the dependencies with `npm install`.

### 5. Configure .env [¶](https://www.tinybird.co/docs/about:blank#5-configure-env)

First create a new file `.env.local`

##### Create the .env.local file in /app

cp .env.example .env.local From the Tinybird UI, copy the read Token for the Charts (if you deployed the resources from this repo, it will be called `CHART_READ_TOKEN` ). Paste the Token into the `.env.local` file in your directory:

##### In the .env.local file

NEXT_PUBLIC_TINYBIRD_STATIC_READ_TOKEN="STATIC READ TOKEN"
### Run the demo app [¶](https://www.tinybird.co/docs/about:blank#run-the-demo-app)

Run it locally:

npm run dev Then open `localhost:3000` with your browser.

## Build from scratch [¶](https://www.tinybird.co/docs/about:blank#build-from-scratch)

This section will take you from a fresh Tinybird Workspace to a Next.js app with a Tinybird Chart.

### 1. Set up a Workspace [¶](https://www.tinybird.co/docs/about:blank#1-set-up-a-workspace)

Create a new Workspace. This guide uses the `EU GCP` region, but you can use any region.

Save [this .datasource file](https://github.com/tinybirdco/guide-tinybird-charts/blob/main/tinybird/datasources/events.datasource) locally, and upload it to the Tinybird UI - you can either drag and drop, or use **Create new (+)** to add a new Data Source.

You now have a Workspace with an `events` Data Source and specified schema. Time to generate some data to fill the Data Source!

### 2. Generate some fake data [¶](https://www.tinybird.co/docs/about:blank#2-generate-some-fake-data)

Use [Mockingbird](https://tbrd.co/mockingbird-tinybird-charts-guide) to generate fake data for the `events` Data Source.

Using this link ^ provides a pre-configured schema, and you'll just need to enter your Workspace Admin Token and select the Host region that matches your Workspace. When configured, select `Save` , then scroll down and select `Start Generating!`.

In the Tinybird UI, confirm that the `events` Data Source is successfully receiving data.

### 3. Create and publish an API Endpoint [¶](https://www.tinybird.co/docs/about:blank#3-create-and-publish-an-api-endpoint)

In the Tinybird UI, select the `events` Data Source and then select `Create Pipe` in the top right.

In the new Pipe, change the name to `top_airlines`.

In the first SQL Node,paste the following SQL:

SELECT
        airline,
        count() as bookings
    FROM events
    GROUP BY airline
    ORDER BY bookings DESC
    LIMIT 5 Name this node `endpoint` and select `Run`.

Now, publish the Pipe by selecting `Create API Endpoint` and selecting the `endpoint` Node. Congratulations! You have a published API Endpoint.

### 4. Create a Chart [¶](https://www.tinybird.co/docs/about:blank#4-create-a-chart)

Publishing the API Endpoint takes you to the API Endpoint overview page.

Scroll down to the `Output` section and select the `Charts` tab.

Select `Create Chart`.

On the `General` tab, set the name to `Top Airlines` then choose `Bar List` as the Chart type.

On the `Data` tab, choose the `airline` column for the `Index` and check the `bookings` box for the `Categories`.

Select Save.

### 5. View the Chart component code [¶](https://www.tinybird.co/docs/about:blank#5-view-the-chart-component-code)

After saving your Chart, you'll be returned to the API Endpoint overview page and you'll see your Chart in the `Output` section.

To the view the component code for the Chart, select the code symbol ( `<>` ) above it. You'll see the command to install the `tinybird-charts` library as well as the React component code.

### 6. Create a new Next.js app [¶](https://www.tinybird.co/docs/about:blank#6-create-a-new-next-js-app)

On your local machine, create a new working directory and navigate to it. For this example, it's called `myapp`.

##### Make a new working directory for your Next.js frontend app

mkdir myapp
cd myapp In the `myapp` dir, create a new Next.js app with the following command:

##### Initialize a new Next.js app

npx create-next-app@latest Some prompts to configure the app appears. Use the following settings:

##### Example new Next.js app settings

✔ What is your project named? … tinybird-demo
✔ Would you like to use TypeScript? … No / [Yes]
✔ Would you like to use ESLint? … No / [Yes]
✔ Would you like to use Tailwind CSS? … No / [Yes]
✔ Would you like to use `src/` directory? … No / [Yes]
✔ Would you like to use App Router? (recommended) … No / [Yes]
✔ Would you like to customize the default import alias (@/*)? … [No] / Yes After the app is created, navigate to the app directory (this will be the same as the project name you entered, in this example, `tinybird-demo` ).

cd tinybird-demo
### 7. Add the Tinybird Charts library [¶](https://www.tinybird.co/docs/about:blank#7-add-the-tinybird-charts-library)

Add the Tinybird Charts library to your project

npm install @tinybirdco/charts
### 8. Add the Chart component [¶](https://www.tinybird.co/docs/about:blank#8-add-the-chart-component)

Create a new subfolder and file `src/app/components/Chart.tsx` . This will contain the component code for the Chart.

[Copy the component from the Tinybird UI](https://www.tinybird.co/docs/about:blank#5-view-the-chart-component-code) and paste it here. It should look like this:

##### Example Chart.tsx code copied from Tinybird UI Chart

'use client'

import { BarList } from '@tinybirdco/charts'

export function TopAirlines() {
  return (
    
  )
} Save the file.

### 9. Add the Chart to your page [¶](https://www.tinybird.co/docs/about:blank#9-add-the-chart-to-your-page)

In your `src/app/page.tsx` file, delete the default contents so you have an empty file.

Then, import the `TopAirlines` component and add it to the page:

##### src/app/page.tsx

import { TopAirlines } from "./components/Chart";

export default function Home() {
  return (
    

      
    

  )
}
### 10. Run the app [¶](https://www.tinybird.co/docs/about:blank#10-run-the-app)

Run the app with `npm run dev` and open `localhost:3000` in your browser.

### 11. You're done! [¶](https://www.tinybird.co/docs/about:blank#11-youre-done)

You've successfully added a Tinybird Chart to a Next.js frontend.

Your Next.js frontend should now show a single bar line Chart. See the [live demo](https://guide-tinybird-charts.vercel.app/) and browse the [GitHub repo (guide-tinybird-charts)](https://github.com/tinybirdco/guide-tinybird-charts) for inspiration on how to combine more Chart components to make a full dashboard.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Interested in dashboards? Explore Tinybird's many applications in the[  Use Case Hub](https://www.tinybird.co/docs/docs/use-cases)  .



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/share-endpoint-documentation
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Share API Endpoints documentation · Tinybird Docs"
theme-color: "#171612"
description: "In this guide you'll learn how to share your Tinybird API Endpoint documentation with development teams."
---




# Share Tinybird API Endpoint documentation [¶](https://www.tinybird.co/docs/about:blank#share-tinybird-api-endpoint-documentation)

In this guide, you'll learn how to share your Tinybird API Endpoint documentation with development teams.

## The Tinybird API Endpoint page [¶](https://www.tinybird.co/docs/about:blank#the-tinybird-api-endpoint-page)

When you publish an API Endpoint, Tinybird generates a documentation page for you that is ready to share and OpenAPI-compatible (v3.0). It contains your API Endpoint description, information about the dynamic parameters you can use when querying this Endpoint, and code snippets for quickly integrating your API in 3rd party applications.

To share your published API Endpoint, navigate to the "Create Chart" button (top right of the UI) > "Share this API Endpoint" modal:

## Use Static Tokens to define API Endpoint subsets [¶](https://www.tinybird.co/docs/about:blank#use-static-tokens-to-define-api-endpoint-subsets)

Tinybird authentication is based on [Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens) which contain different scopes for specific resources. For example, a Token lets you read from one or many API Endpoints, or get write permissions for a particular Data Source.

If you take a closer look at the URLs generated for sharing a public API Endpoint page, you'll see that after the Endpoint ID, it includes a Token parameter. This means that this page is only accessible if the Token provided in the URL has read permissions for it:

https://api.tinybird.co/endpoint/t_bdcad2252e794c6573e21e7e?token= For security, Tinybird automatically generates a read-only Token when sharing a public API Endpoint page for the first time. If you don't explicitly use it, your Admin Token won't ever get exposed.

### The API Endpoints list page [¶](https://www.tinybird.co/docs/about:blank#the-api-endpoints-list-page)

Tinybird also allows you to render the API Endpoints information for a given Token.

https://app.tinybird.co///endpoints?token= Enter the URL above (with your Token and the provider and region where the API Endpoint is published) into the browser, and it'll return a list that shows all API Endpoints that this Token can read from.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fsharing-endpoints-documentation-with-development-teams-2.png&w=3840&q=75)

<-figcaption->
The API Endpoints list page is extremely useful for sharing your API Endpoint documentation with development teams





When integrating your API Endpoint in your applications it's highly recommend that you manage dedicated Tokens. The easiest way is creating a Token for every application environment, so that you can also track the different requests to your API Endpoints by application, and choose which API Endpoints are accessible for them.

Once you do that, you can share auto-generated documentation with ease, without compromising your data privacy and security.

API Endpoint docs pages include a read Token by default. In the "Share this API Endpoint" modal, you can also see public URLs for every Token with read permissions for your Pipe.

## Browse your docs in Swagger [¶](https://www.tinybird.co/docs/about:blank#browse-your-docs-in-swagger)

As mentioned above, all Tinybird's documentation is compatible with OpenAPI 3.0 and accessible via API. A quick way of generating documentation in Swagger is navigating to the "Create Chart" button > "Share this API Endpoint" modal > "OpenAPI 3.0" tab, copying the "Shareable link" URL, and using it in your preferred Swagger installation.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fsharing-endpoints-documentation-with-development-teams-3.png&w=3840&q=75)

<-figcaption->
You can generate as many URLs as you need by using different Tokens





If you use a Token with permissions for more than one API Endpoint, the Swagger documentation will contain information about all the API Endpoints at once.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- You've got Endpoints, now make them pretty: Use[  Tinybird Charts](https://www.tinybird.co/docs/docs/publish/charts)  .
- Learn how to[  monitor and analyze your API performance](https://www.tinybird.co/docs/docs/monitoring/analyze-endpoints-performance)  .



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/serverless-analytics-api
Last update: 2025-01-22T09:19:22.000Z
Content:
---
title: "Handling data privacy in serverless analytics APIs · Tinybird Docs"
theme-color: "#171612"
description: "Creating an Analytics Dashboard where each user is able to access only certain parts of the data is really easy with Tinybird. You don't need to build anything specific from scratch. Tinybird is able to provide dynamic API Endpoints, including specific security requirements per-user."
---




# Handling data privacy in serverless analytics APIs [¶](https://www.tinybird.co/docs/about:blank#handling-data-privacy-in-serverless-analytics-apis)

Creating an analytics dashboard where each user is able to access only certain parts of the data is really easy with Tinybird. You don't need to build anything specific from scratch. Tinybird provides dynamic API Endpoints, including specific security requirements per-user.

## The serverless approach to real-time analytics [¶](https://www.tinybird.co/docs/about:blank#the-serverless-approach-to-real-time-analytics)

Let's assume you have just two components - the simplest possible stack:

- **  A frontend application:**   Code that runs in the browser.
- **  A backend application:**   Code that runs in the server and manages both the user authentication and the authorization. Very probably, the backend will also expose an API from where the frontend fetches the information needed.

This guide covers the different workflows that will handle each user operation with the right permissions, by integrating your backend with [Static Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#what-should-i-use-tokens-for) in a very simple way.

## Create Tokens on user sign-up [¶](https://www.tinybird.co/docs/about:blank#create-tokens-on-user-sign-up)

The only thing you need (to ensure that your users have the right permissions on your data) is a created Tinybird [Static Token](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#what-should-i-use-tokens-for) every time you create a new user in your backend.

##### Creating a Token with filter scope

TOKEN=

curl -H "Authorization: Bearer $TOKEN" \
  -d "name=user_692851_token" \
  -d "scope=PIPES:READ:ecommerce_example" \
  -d "scope=DATASOURCES:READ:events:user_id=692851" \
  https://api.tinybird.co/v0/tokens/ Use a Token with the right scope. Replace `` with a Token whose [scope](https://www.tinybird.co/docs/docs/api-reference/token-api) is `TOKENS` or `ADMIN`.

This Token will let a given user query their own transactions stored in an `events` Data Source and exposed in an `ecommerce_example` API Endpoint.

Some other noteworthy things you can see here:

- You can give a `name`   to every Token you create. In this case, the name contains the `user_id`   , so that it's easier to see what Token is assigned to each user.
- You can assign as many scopes to each Token as you want, and `DATASOURCES:READ:datasource_name`   and `PIPES:READ:pipe_name`   can take an optional SQL filter (like this example does) to restrict the rows that queries authenticated with the Token will have access to.

If everything runs successfully, your call will return JSON containing a Token with the specified scopes:

##### Creating a Token with filter scope: Response

{
    "token": "p.eyJ1IjogImI2Yjc1MDEx....",
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "ecommerce_example",
            "filter": ""
        },
        {
            "type": "DATASOURCES:READ",
            "resource": "events",
            "filter": "user_id=692851"
        }
    ],
    "name": "user_692851_token"
} All the Tokens you create are also visible in your [Workspace > Tokens page](https://app.tinybird.co/tokens) in the UI, where you can create, update and delete them.

## Modify Tokens when user permissions are changed [¶](https://www.tinybird.co/docs/about:blank#modify-tokens-when-user-permissions-are-changed)

Imagine one of your users is removed from a group, which makes them lose some permissions on the data they can consume. Once that is reflected in your backend, you can [update the user admin Token](https://www.tinybird.co/docs/docs/api-reference/token-api#put--v0-tokens-(.+)) accordingly as follows:

##### Modify an existing Token

TOKEN=
USER_TOKEN=

curl -X PUT \
  -H "Authorization: Bearer $TOKEN" \
  -d "name=user_692851_token" \
  -d "scope=PIPES:READ:ecommerce_example" \
  -d "scope=DATASOURCES:READ:events:user_id=692851 and event in ('buy', 'add_item_to_cart')" \
  https://api.tinybird.co/v0/tokens/$USER_TOKEN Pass the Token you previously created as a path parameter. Replace `` by the value of `token` from the previous response, or [copy it from the UI](https://app.tinybird.co/tokens).

In this example you'd be restricting the SQL filter of the `DATASOURCES:READ:events` scope to restrict the type of events the user will be able to read from the `events` Data Source. This is the response you'd see from the API:

##### Modify an existing Token: Response

{
    "token": "p.eyJ1IjogImI2Yjc1MDEx,...",
    "scopes": [
        {
            "type": "PIPES:READ",
            "resource": "ecommerce_example",
            "filter": ""
        },
        {
            "type": "DATASOURCES:READ",
            "resource": "events",
            "filter": "user_id=692851 and event in ('buy', 'add_item_to_cart')"
        }
    ],
    "name": "user_692851_token"
}
## Delete Tokens after user deletion [¶](https://www.tinybird.co/docs/about:blank#delete-tokens-after-user-deletion)

Whenever a user is removed from your system, you should also [remove the Token from Tinybird](https://www.tinybird.co/docs/docs/api-reference/token-api#delete--v0-tokens-(.+)) . That will make things easier for you in the future.

##### Remove a token

TOKEN=
USER_TOKEN=

curl -X DELETE \
  -H "Authorization: Bearer $TOKEN" \
  https://api.tinybird.co/v0/tokens/$USER_TOKEN If the Token is successfully deleted, this request will respond with no content and a 204 status code.

## Refresh Tokens [¶](https://www.tinybird.co/docs/about:blank#refresh-tokens)

It's a good practice to change Tokens from time to time, so you can automate this in your backend as well. Refreshing a Token requires executing this request for every one of your users:

##### Refresh a token

TOKEN=
USER_TOKEN=

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  https://api.tinybird.co/v0/tokens/$USER_TOKEN/refresh
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Learn more about the[  Tokens API](https://www.tinybird.co/docs/docs/api-reference/token-api)  .
- Understand the concept of[  Static Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#what-should-i-use-tokens-for)  .



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/reliable-scheduling-with-trigger
Last update: 2025-01-08T08:08:08.000Z
Content:
---
title: "Reliable scheduling with Trigger.dev · Tinybird Docs"
theme-color: "#171612"
description: "Learn how to create complex, reliable scheduling with Trigger.dev"
---




# Reliable scheduling with Trigger.dev [¶](https://www.tinybird.co/docs/about:blank#reliable-scheduling-with-trigger-dev)

[Trigger.dev](https://trigger.dev/) is an open source background job platform. With Trigger.dev you can easily create, schedule, and manage background jobs using code.

Read on to learn how to create complex, reliable scheduling with Trigger.dev.

## Before you start [¶](https://www.tinybird.co/docs/about:blank#before-you-start)

Before you start, ensure:

- You have a[  Trigger.dev account](https://trigger.dev/)  .
- You have a[  Tinybird Workspace](https://www.tinybird.co/)  .

## Create your first trigger task [¶](https://www.tinybird.co/docs/about:blank#create-your-first-trigger-task)

The [tinybird-trigger-tasks package](https://www.npmjs.com/package/@sdairs/tinybird-trigger-tasks) implements tasks for Tinybird Copy Pipes and the Query API. You can [find the source code in the @sdairs/tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger).

1. Create a working directory, and run `npx trigger.dev@latest init`   to connect the project to Trigger.dev.
2. Inside the `trigger`   directory, install the npm package with `npm install @sdairs/tinybird-trigger-tasks`  .
3. Create a new file called `myTask.ts`   and add the following code:

import { task } from "@trigger.dev/sdk/v3";
import { tinybirdCopyTask } from "@sdairs/tinybird-trigger-tasks";

export const exampleExecutor = task({
    id: "example-executor",
    run: async (payload, { ctx }) => {
        console.log("Example executor task is running");

        // Run a copy job
        const copyResult = await tinybirdCopyTask.triggerAndWait({ pipeId:  });
        console.log(copyResult);

    },
});
1. Go to your Tinybird Workspace, and create a new Pipe. Use the following SQL:

SELECT number + 1 AS value
FROM numbers(100)
1. Name the Pipe `my_copy`   , then select `Create Copy`   from the actions menu. Follow the prompts to create the Copy Pipe.
2. Update `myTask.ts`   , replacing ``   with the name of your Pipe, `my_copy`   in this case.
3. Create a `.env`   file in your directory root.
4. Go to your Tinybird Workspace and copy the Admin Token, then add it to the `.env`   file as follows:

TINYBIRD_TOKEN=p.eyJ...
1. Run `npx trigger.dev@latest dev`   to push the task to Trigger.dev.
2. Go to your Trigger.dev dashboard, and perform a test run to trigger the task and the Copy Pipe.
3. Go to your Tinybird Workspace and check the Copy Pipe results.

## See also [¶](https://www.tinybird.co/docs/about:blank#see-also)

- [  Trigger.dev quick start](https://trigger.dev/docs/quick-start)
- [  tinybird-trigger repo](https://github.com/sdairs/tinybird-trigger)
- [  YouTube: Using Trigger.dev with Tinybird for code-first background job execution](https://www.youtube.com/watch?v=0TcQfcMrGNw)



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-apis-nextjs
Last update: 2025-01-22T09:19:22.000Z
Content:
---
title: "Consume APIs in a Next.js frontend with JWTs · Tinybird Docs"
theme-color: "#171612"
description: "In this guide, you'll learn how to generate self-signed JWTs from your backend, and call Tinybird APIs directly from your frontend, using Next.js."
---




# Consume APIs in a Next.js frontend with JWTs [¶](https://www.tinybird.co/docs/about:blank#consume-apis-in-a-next-js-frontend-with-jwts)

In this guide, you'll learn how to generate self-signed JWTs from your backend, and call Tinybird APIs directly from your frontend, using Next.js.

JWTs are signed tokens that allow you to securely authorize and share data between your application and Tinybird.

You can view the [live demo](https://guide-nextjs-jwt-auth.vercel.app/) or browse the [GitHub repo (guide-nextjs-jwt-auth)](https://github.com/tinybirdco/guide-nextjs-jwt-auth).

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes that you have a Tinybird account, and you are familiar with creating a Tinybird Workspace and pushing resources to it.

Make sure you understand the concept of Tinybird's [Static Tokens](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#what-should-i-use-tokens-for).

You'll need a working familiarity with JWTs, JavaScript, and Next.js.

## Run the demo [¶](https://www.tinybird.co/docs/about:blank#run-the-demo)

These steps cover running the GitHub demo locally. [Skip to the next section](https://www.tinybird.co/docs/about:blank#understand-the-code) for a breakdown of the code.

### 1. Clone the GitHub repo [¶](https://www.tinybird.co/docs/about:blank#1-clone-the-github-repo)

Clone the [GitHub repo (guide-nextjs-jwt-auth)](https://github.com/tinybirdco/guide-nextjs-jwt-auth) to your local machine.

### 2. Push Tinybird resources [¶](https://www.tinybird.co/docs/about:blank#2-push-tinybird-resources)

The repo includes two sample Tinybird resources:

- `events.datasource`   : The Data Source for incoming events.
- `top_airlines.pipe`   : An API Endpoint giving a list of top 10 airlines by booking volume.

Configure the [Tinybird CLI](https://www.tinybird.co/docs/docs/cli/install) and `tb push` the resources to your Workspace. Alternatively, you can drag and drop the files onto the UI to upload them.

### 3. Generate some fake data [¶](https://www.tinybird.co/docs/about:blank#3-generate-some-fake-data)

Use [Mockingbird](https://tbrd.co/mockingbird-nextjs-jwt-demo) to generate fake data for the `events` Data Source.

Using this link ^ provides a pre-configured schema, but you will need to enter your Workspace admin Token and Host. When configured, scroll down and select `Start Generating!`.

In the Tinybird UI, confirm that the `events` Data Source is successfully receiving data.

### 4. Install dependencies [¶](https://www.tinybird.co/docs/about:blank#4-install-dependencies)

Navigate to the cloned repo and install the dependencies with `npm install`.

### 5. Configure .env [¶](https://www.tinybird.co/docs/about:blank#5-configure-env)

First create a new file `.env.local`

cp .env.example .env.local Copy your [Tinybird host](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) and admin Token (used as the `TINYBIRD_SIGNING_TOKEN` ) to the `.env.local` file:

TINYBIRD_SIGNING_TOKEN="TINYBIRD_SIGNING_TOKEN>" # Use your Admin Token as the signing Token
TINYBIRD_WORKSPACE="YOUR_WORKSPACE_ID"  # The UUID of your Workspace
NEXT_PUBLIC_TINYBIRD_HOST="YOUR_TINYBIRD_API_REGION e.g. https://api.tinybird.co" # Your regional API host

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

### Run the demo app [¶](https://www.tinybird.co/docs/about:blank#run-the-demo-app)

Run it locally:

npm run dev Then open `localhost:3000` with your browser.

## Understand the code [¶](https://www.tinybird.co/docs/about:blank#understand-the-code)

This section breaks down the key parts of code from the example.

### .env [¶](https://www.tinybird.co/docs/about:blank#env)

The `.env` file contains the environment variables used in the application.

##### .env file

TINYBIRD_SIGNING_TOKEN="YOUR SIGNING TOKEN"
TINYBIRD_WORKSPACE="YOUR WORKSPACE ID"
NEXT_PUBLIC_TINYBIRD_HOST="YOUR API HOST e.g. https://api.tinybird.co"
#### TINYBIRD_SIGNING_TOKEN [¶](https://www.tinybird.co/docs/about:blank#tinybird-signing-token)

`TINYBIRD_SIGNING_TOKEN` is the token used to sign JWTs. **You must use your admin Token** . It is a shared secret between your application and Tinybird. Your application uses this Token to sign JWTs, and Tinybird uses it to verify the JWTs. It should be kept secret, as exposing it could allow unauthorized access to your Tinybird resources. It is best practice to store this in an environment variable instead of hardcoding it in your application.

#### TINYBIRD_WORKSPACE [¶](https://www.tinybird.co/docs/about:blank#tinybird-workspace)

`TINYBIRD_WORKSPACE` is the ID of your Workspace. It is used to identify the Workspace that the JWT is generated for. The Workspace ID is included inside the JWT payload. Workspace IDs are UUIDs and can be found using the CLI `tb workspace current` command or from the Tinybird UI.

#### NEXT_PUBLIC_TINYBIRD_HOST [¶](https://www.tinybird.co/docs/about:blank#next-public-tinybird-host)

`NEXT_PUBLIC_TINYBIRD_HOST` is the base URL of the Tinybird API. It is used to construct the URL for the Tinybird API Endpoints. You must use the correct URL for [your Tinybird region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) . The `NEXT_PUBLIC_` prefix is required for Next.js to expose the variable to the client side.

### token.ts [¶](https://www.tinybird.co/docs/about:blank#token-ts)

The `token.ts` file contains the logic to generate and sign JWTs. It uses the `jsonwebtoken` library to create the Token.

##### token.ts

"use server";

import jwt from "jsonwebtoken";

const TINYBIRD_SIGNING_TOKEN = process.env.TINYBIRD_SIGNING_TOKEN ?? "";
const WORKSPACE_ID = process.env.TINYBIRD_WORKSPACE ?? "";
const PIPE_ID = "top_airlines"; 

export async function generateJWT() {
  const next10minutes = new Date();
  next10minutes.setTime(next10minutes.getTime() + 1000 * 60 * 10);

  const payload = {
    workspace_id: WORKSPACE_ID,
    name: "my_demo_jwt",
    exp: Math.floor(next10minutes.getTime() / 1000),
    scopes: [
      {
        type: "PIPES:READ",
        resource: PIPE_ID,
      },
    ],
  };

  return jwt.sign(payload, TINYBIRD_SIGNING_TOKEN, {noTimestamp: true});
} This code runs on the backend to generate JWTs without exposing secrets to the user.

It pulls in the `TINYBIRD_SIGNING_TOKEN` and `WORKSPACE_ID` from the environment variables.

As this example only exposes a single API Endpoint ( `top_airlines.pipe` ), the `PIPE_ID` is hardcoded to its deployed ID. If you had multiple API Endpoints, you would need to create an item in the `scopes` array for each one.

The `generateJWT` function handles creation of the JWT. A JWT has various [required fields](https://www.tinybird.co/docs/docs/get-started/administration/auth-tokens#jwt-payload).

The `exp` field sets the expiration time of the JWT in the form a UTC timestamp. In this case, it's set to 10 minutes in the future. You can adjust this value to suit your needs.

The `name` field is a human-readable name for the JWT. This value is only used for logging.

The `scopes` field defines what the JWT can access. This is an array, which allows you create one JWT that can access multiple API Endpoints. In this case, you only have one API Endpoint. Under `scopes` , the `type` field is always `PIPES:READ` for reading data from a Pipe. The `resource` field is the ID or name of the Pipe you want to access. If required, you can also add `fixed_parameters` here to supply parameters to the API Endpoint.

Finally, the payload is signed using the `jsonwebtoken` library and the `TINYBIRD_SIGNING_TOKEN`.

### useFetch.tsx [¶](https://www.tinybird.co/docs/about:blank#usefetch-tsx)

The `useFetch.tsx` file contains a custom React hook that fetches data from the Tinybird API using a JWT. It also handles refreshing the token if it expires.

##### useFetch.tsx

import { generateJWT } from "@/server/token";
import { useState } from "react";

export function useFetcher() {
  const [token, setToken] = useState("");

  const refreshToken = async () => {
    const newToken = await generateJWT();
    setToken(newToken);
    return newToken;
  };

  return async (url: string) => {
    let currentToken = token;
    if (!currentToken) {
      currentToken = await refreshToken();
    }
    const response = await fetch(url + "?token=" + currentToken);

    if (response.status === 200) {
      return response.json();
    }
    if (response.status === 403) {
      const newToken = await refreshToken();
      return fetch(url + "?token=" + newToken).then((res) => res.json());
    }
  };
} This code runs on the client side and is used to fetch data from the Tinybird API.

It uses the `generateJWT` function from the `token.ts` file to get a JWT. The JWT is stored in the `token` state.

Most importantly, it uses the standard `fetch` API to make requests to the Tinybird API. The JWT is passed as a `token` query parameter in the URL.

If the request returns a `403` status code, the hook then calls `refreshToken` to get a new JWT and retries the request. However, note that this is a simple implementation and there are other reasons why a request might fail with a `403` status code (e.g., the JWT is invalid, the API Endpoint has been removed, etc.).

### page.tsx [¶](https://www.tinybird.co/docs/about:blank#page-tsx)

The `page.tsx` file contains the main logic for the Next.js page. It is responsible for initiating the call to the Tinybird API Endpoints and rendering the data into a chart.

##### page.tsx

"use client";

import { BarChart, Card, Subtitle, Text, Title } from "@tremor/react";
import useSWR from "swr";
import { getEndpointUrl } from "@/utils";
import { useFetcher } from "@/hooks/useFetch";

const REFRESH_INTERVAL_IN_MILLISECONDS = 5000; // five seconds

export default function Dashboard() {
  const endpointUrl = getEndpointUrl();
  const fetcher = useFetcher();

  let top_airline, latency, errorMessage;

  const { data } = useSWR(endpointUrl, fetcher, {
    refreshInterval: REFRESH_INTERVAL_IN_MILLISECONDS,
    onError: (error) => (errorMessage = error),
  });

  if (!data) return;

  if (data?.error) {
    errorMessage = data.error;
    return;
  }

  top_airline = data.data;
  latency = data.statistics?.elapsed;

  return (
    
      Top airlines by bookings
      Ranked from highest to lowest
      {top_airline && (
        
      )}
      {latency && Latency: {latency * 1000} ms}
      {errorMessage && (
        

          

            Oops, something happens: {errorMessage}
          

          
Check your console for more information

        

      )}
    
  );
} It uses [SWR](https://swr.vercel.app/) and the `useFetcher` hook from [useFetch.tsx](https://www.tinybird.co/docs/about:blank#usefetch-tsx) to fetch data from the Tinybird API.

When the API Endpoint returns data, it's rendered as bar chart using the `BarChart` component from the `@tremor/react` library.

## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Read the[  blog post on JWTs](https://www.tinybird.co/blog-posts/jwt-api-endpoints-public-beta)  .
- Explore more use cases that use this approach, like[  building a real-time, user-facing dashboard](https://www.tinybird.co/docs/docs/use-cases/user-facing-dashboards)  .



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-apis-in-a-notebook
Last update: 2025-01-20T11:46:12.000Z
Content:
---
title: "Consume APIs in a Notebook · Tinybird Docs"
theme-color: "#171612"
description: "Notebooks are a great resource for exploring data and generating plots. In this guide, you'll learn how to consume Tinybird APIs in a notebook."
---




# Consume APIs in a notebook [¶](https://www.tinybird.co/docs/about:blank#consume-apis-in-a-notebook)

Notebooks are a great resource for exploring data and generating plots. In this guide, you'll learn how to consume Tinybird APIs in a notebook.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This [Colab notebook](https://github.com/tinybirdco/examples/blob/master/notebook/consume_from_apis.ipynb) uses a Data Source of updates to Wikipedia to show how to consume data from queries. There are two options: Using the [Query API](https://www.tinybird.co/docs/docs/api-reference/query-api) , and using API Endpoints using the [Pipes API](https://www.tinybird.co/docs/docs/api-reference/pipe-api) and parameters. The full code for every example in this guide can be found in the notebook.

This guide assumes some familiarity with Python.

## Setup [¶](https://www.tinybird.co/docs/about:blank#setup)

Follow the setup steps in the [notebook file](https://github.com/tinybirdco/examples/blob/master/notebook/consume_from_apis.ipynb) and use the linked CSV file of Wikipedia updates to create a new Data Source in your Workspace.

For less than 100 MB of data, you can fetch all the data. For calls with than 100 MB of data, you need to do it sequentially, with not more than 100 MB per API call. The solution is to get batches using Data Source sorting keys. Selecting the data by columns used in the sorting key keeps it fast. In this example, the Data Source is sorted on the `timestamp` column, so you can use batches of a fixed amount of time. In general, time is a good way to batch.

The functions `fetch_table_streaming_query` and `fetch_table_streaming_endpoint` in the notebook work as generators. They should always be used in a `for` loop or as the input for another generator.

You should process each batch as it arrives and discard unwanted fetched data. Only fetch the data you need in the processing. The idea here isn't to recreate a Data Source in the notebook, but to process each batch as it arrives and write less data to your DataFrame.

## Fetch data with the Query API [¶](https://www.tinybird.co/docs/about:blank#fetch-data-with-the-query-api)

This guide uses the [requests library for Python](https://pypi.org/project/requests/) . The SQL query pulls in an hour less of data than the full Data Source. A DataFrame is created from the text part of the response.

##### DataFrame from the query API

table_name = 'wiki'
host = 'api.tinybird.co'
format = 'CSVWithNames'
time_column = 'toDateTime(timestamp)'
date_end = 'toDateTime(1644754546)'
 
s = requests.Session()
s.headers['Authorization'] = f'Bearer {token}'
 
URL = f'https://{host}/v0/sql'
sql = f'select * from {table_name} where {time_column} <= {date_end}'
params = {'q': sql + f" FORMAT {format}"}
 
r = s.get(f"{URL}?{urlencode(params)}")
df = pd.read_csv(StringIO(r.text))

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

## Fetch data from an API Endpoint & parameters [¶](https://www.tinybird.co/docs/about:blank#fetch-data-from-an-api-endpoint-parameters)

This Endpoint node in the Pipe `endpoint_wiki` selects from the Data Source within a range of dates, using the parameters for `date_start` and `date_end`.

##### Endpoint wiki

%
SELECT * FROM wiki
WHERE timestamp BETWEEN 
toInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00')}}))
AND
toInt64(toDateTime({{String(date_end, '2022-02-13 11:00:00')}})) These parameters are passed in the call to the API Endpoint to select only the data within the range. A DataFrame is created from the text part of the response.

##### Dataframe from API Endpoint

host = 'api.tinybird.co'
api_endpoint = 'endpoint_wiki'
format = 'csv'
 
date_start = '2022-02-13 10:30:00'
date_end = '2022-02-13 11:30:00'
 
s = requests.Session()
s.headers['Authorization'] = f'Bearer {token}'
 
URL = f'https://{host}/v0/pipes/{api_endpoint}.{format}'
params = {'date_start': date_start,
         'date_end': date_end
         }
        
r = s.get(f"{URL}?{urlencode(params)}")
df = pd.read_csv(StringIO(r.text))

Replace the Tinybird API hostname or region with the [API region](https://www.tinybird.co/docs/docs/api-reference#regions-and-endpoints) that matches your Workspace.

## Fetch batches of data using the Query API [¶](https://www.tinybird.co/docs/about:blank#fetch-batches-of-data-using-the-query-api)

The function `fetch_table_streaming_query` in the notebook accepts more complex queries than a date range. Here you choose what you filter and sort by. This example reads in batches of 5 minutes to create a small DataFrame, which should then be processed, with the results of the processing appended to the final DataFrame.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fconsume-apis-in-a-notebook-1.png&w=3840&q=75)

<-figcaption->
5-minute batches of data using the index





##### DataFrames from batches returned by the Query API

tinybird_stream = fetch_table_streaming_query(token,
                                       'wiki',
                                       60*5,
                                       1644747337,
                                       1644758146,
                                       sorting='timestamp',
                                       filters="type IN ['edit','new']",
                                       time_column="timestamp",
                                       host='api.tinybird.co')
 
df_all=pd.DataFrame()
for x in tinybird_stream:
   df_batch = pd.read_csv(StringIO(x))
   # TO DO: process batch and discard fetched data
   df_proc=process_dataframe(df_batch)
   df_all = df_all.append(df_proc) # Careful: appending dfs means keeping a lot of data in memory
## Fetch batches of data from an API Endpoint & parameters [¶](https://www.tinybird.co/docs/about:blank#fetch-batches-of-data-from-an-api-endpoint-parameters)

The function `fetch_table_streaming_endpoint` in the notebook sends a call to the API with parameters for the `batch size`, `start` and `end` dates, and, optionally, filters on the `bot` and `server_name` columns. This example reads in batches of 5 minutes to create a small DataFrame, which should then be processed, with the results of the processing appended to the final DataFrame.

‍The API Endpoint `wiki_stream_example` first selects data for the range of dates, then for the batch, and then applies the filters on column values.

##### API Endpoint wiki_stream_example

%
SELECT * from wiki
--DATE RANGE
WHERE timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}}))
AND toUInt64(toDateTime({{String(date_end, '2022-02-13 10:35:00', description="end")}}))
--BATCH BEGIN
AND timestamp BETWEEN toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval {{Int16(batch_no, 1, description="batch number")}}
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--BATCH END
AND toUInt64(toDateTime({{String(date_start, '2022-02-13 10:30:00', description="start")}})
              + interval ({{Int16(batch_no, 1, description="batch number")}} + 1)
              * {{Int16(batch_size, 10, description="size of the batch")}} second)
--FILTERS
{% if defined(bot) %}
 AND bot = {{String(bot, description="is a bot")}}
{% end %}
{% if defined(server_name) %}
 AND server_name = {{String(server_name, description="server")}}
{% end %} These parameters are passed in the call to the API Endpoint to select only the data for the batch. A DataFrame is created from the text part of the response.

##### DataFrames from batches from the API Endpoint

tinybird_stream = fetch_table_streaming_endpoint(token,
                                                 'csv',
                                                 60*5, 
                                                 '2022-02-13 10:15:00',
                                                 '2022-02-13 13:15:00',
                                                 bot = False,
                                                 server_name='en.wikipedia.org'
                                                )

df_all=pd.DataFrame()
for x in tinybird_stream:
    df_batch = pd.read_csv(StringIO(x))
    # TO DO: process batch and discard fetched data 
    df_proc=process_dataframe(df_batch)
    df_all = df_all.append(df_proc) # Careful: appending dfs means keeping a lot of data in memory
## Next steps [¶](https://www.tinybird.co/docs/about:blank#next-steps)

- Explore more use cases for Tinybird in the[  Use Case Hub](https://www.tinybird.co/docs/docs/use-cases)  .
- Looking for other ways to integrate? Try[  consume APIs in a Next.js frontend](https://www.tinybird.co/docs/docs/publish/api-endpoints/guides/consume-apis-nextjs)  .



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-api-endpoints-in-prometheus-format
Last update: 2024-12-18T15:47:50.000Z
Content:
---
title: "Consume API Endpoints in Prometheus format · Tinybird Docs"
theme-color: "#171612"
description: "Export Pipe endpoints in Prometheus format to integrate Tinybird data into your monitoring stack."
---




# Consume API Endpoints in Prometheus format [¶](https://www.tinybird.co/docs/about:blank#consume-api-endpoints-in-prometheus-format)

Prometheus is a powerful open-source monitoring and alerting toolkit widely used for metrics collection and visualization. You can export Pipe endpoints in Prometheus format to integrate your Tinybird data into your monitoring stack.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes you have a Tinybird Workspace with an active Data Source, Pipes, and at least one API Endpoint.

## Structure data for Prometheus [¶](https://www.tinybird.co/docs/about:blank#structure-data-for-prometheus)

To export the Pipe output in Prometheus format, data must conform to the following structure:

### Mandatory columns [¶](https://www.tinybird.co/docs/about:blank#mandatory-columns)

- `name (String)`   : The name of the metric.
- `value (Number)`   : The numeric value for the metric.

### Optional columns [¶](https://www.tinybird.co/docs/about:blank#optional-columns)

- `help (String)`   : A description of the metric.
- `timestamp (Number)`   : A Unix timestamp for the metric.
- `type (String)`   : Defines the metric type ( `counter`  , `gauge`  , `histogram`  , `summary`  , `untyped`   , or empty).
- `labels (Map(String, String))`   : A set of key-value pairs providing metric dimensions.

### Example [¶](https://www.tinybird.co/docs/about:blank#example)

Here’s an example of a Tinybird Pipe query that outputs two metrics, `http_request_count` and `http_request_duration_seconds` , in the same query. Both metrics include labels for `method` and `status_code`.

SELECT
    -- Metric 1: http_request_count
    'http_request_count' AS name,
    toFloat64(count(*)) AS value,
    'Total number of HTTP requests' AS help,
    'counter' AS type,
    map('method', method, 'status_code', status_code) AS labels
FROM
    http_requests
GROUP BY
    method, status_code

UNION ALL

SELECT
    -- Metric 2: http_request_duration_seconds
    'http_request_duration_seconds' AS name,
    avg(request_time) AS value,
    'Average HTTP request duration in seconds' AS help,
    'gauge' AS type,
    map('method', method, 'status_code', status_code) AS labels
FROM
    http_requests
GROUP BY
    method, status_code

ORDER BY
    name
## Export Tinybird Pipe endpoint in Prometheus format [¶](https://www.tinybird.co/docs/about:blank#export-tinybird-pipe-endpoint-in-prometheus-format)

Export Pipe data in Prometheus format by appending .prometheus to your API endpoint URI. For example:

https://api.tinybird.co/v0/pipes/your_pipe_name.prometheus The following is an example Prometheus output:

# HELP http_request_count Total number of HTTP requests
# TYPE http_request_count counter
http_request_count{method="PUT",status_code="203"} 1
http_request_count{method="PATCH",status_code="203"} 1
http_request_count{method="DELETE",status_code="201"} 4
http_request_count{method="POST",status_code="203"} 1
http_request_count{method="OPTIONS",status_code="203"} 1
http_request_count{method="PATCH",status_code="204"} 1
http_request_count{method="PUT",status_code="204"} 1
http_request_count{method="HEAD",status_code="203"} 1
http_request_count{method="GET",status_code="201"} 4
http_request_count{method="POST",status_code="204"} 1
http_request_count{method="GET",status_code="203"} 1
http_request_count{method="POST",status_code="201"} 4
http_request_count{method="DELETE",status_code="204"} 1
http_request_count{method="OPTIONS",status_code="201"} 4
http_request_count{method="GET",status_code="204"} 1
http_request_count{method="PATCH",status_code="201"} 4
http_request_count{method="PUT",status_code="201"} 4
http_request_count{method="DELETE",status_code="203"} 1
http_request_count{method="HEAD",status_code="201"} 4

# HELP http_request_duration_seconds Average HTTP request duration in seconds
# TYPE http_request_duration_seconds gauge
http_request_duration_seconds{method="GET",status_code="200"} 75.01
http_request_duration_seconds{method="DELETE",status_code="201"} 11.01
http_request_duration_seconds{method="POST",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="HEAD",status_code="204"} 169.01
http_request_duration_seconds{method="PATCH",status_code="204"} 169.01
http_request_duration_seconds{method="PUT",status_code="204"} 169.01
http_request_duration_seconds{method="HEAD",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="OPTIONS",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="DELETE",status_code="200"} 75.01
http_request_duration_seconds{method="OPTIONS",status_code="204"} 169.01
http_request_duration_seconds{method="GET",status_code="201"} 11.01
http_request_duration_seconds{method="PATCH",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="PUT",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="POST",status_code="204"} 169.01
http_request_duration_seconds{method="DELETE",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="PUT",status_code="200"} 75.01
http_request_duration_seconds{method="POST",status_code="200"} 75.01
http_request_duration_seconds{method="PATCH",status_code="200"} 75.01
http_request_duration_seconds{method="POST",status_code="201"} 11.01
http_request_duration_seconds{method="DELETE",status_code="204"} 169.01
http_request_duration_seconds{method="OPTIONS",status_code="201"} 11.01
http_request_duration_seconds{method="GET",status_code="204"} 169.01
http_request_duration_seconds{method="PATCH",status_code="201"} 11.01
http_request_duration_seconds{method="PUT",status_code="201"} 11.01
http_request_duration_seconds{method="GET",status_code="202"} 102.00999999999999
http_request_duration_seconds{method="HEAD",status_code="201"} 11.01
## Integrate endpoints in Prometheus-compatible tools [¶](https://www.tinybird.co/docs/about:blank#integrate-endpoints-in-prometheus-compatible-tools)

Now that you’ve structured and exported your Pipe data in Prometheus format, you can integrate it into monitoring and observability tools.

Prometheus is widely supported by various visualization and alerting platforms, making it easy to use your Tinybird data with tools like Grafana, Datadog, and more. See the documentation for these tools to integrate them with the Prometheus endpoints from Tinybird.

## Monitoring your Tinybird Organization with Grafana and Datadog [¶](https://www.tinybird.co/docs/about:blank#monitoring-your-tinybird-organization-with-grafana-and-datadog)

Check the [Tinybird Organization metrics](https://github.com/tinybirdco/tinybird-org-metrics-exporter) repository for a working example of how to consume the Prometheus endpoints in Grafana and Datadog to monitor your Tinybird Organization.



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/consume-api-endpoints-in-grafana
Last update: 2025-01-17T07:58:01.000Z
Content:
---
title: "Consume API Endpoints in Grafana · Tinybird Docs"
theme-color: "#171612"
description: "Grafana is an awesome open source analytics & monitoring tool. In this guide, you'll learn how to create dashboards consuming Tinybird API Endpoints."
---




# Consume API Endpoints in Grafana [¶](https://www.tinybird.co/docs/about:blank#consume-api-endpoints-in-grafana)

[Grafana](https://grafana.com/grafana/) is an awesome open source analytics & monitoring tool. In this guide, you'll learn how to create Grafana Dashboards and Alerts consuming Tinybird API Endpoints.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

This guide assumes you have a Tinybird Workspace with an active Data Source, Pipes, and at least one API Endpoint. You'll also need a Grafana account.

## 1. Install the Infinity plugin [¶](https://www.tinybird.co/docs/about:blank#1-install-the-infinity-plugin)

Follow the steps in [Grafana Infinity plugin installation](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/?tab=installation).

## 2. Create a Grafana data source [¶](https://www.tinybird.co/docs/about:blank#2-create-a-grafana-data-source)

Create a new Data Source using the Infinity plugin. Connections > Data sources > Add new data source > Infinity.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-ds-infinity.png&w=3840&q=75)


Edit the name and complete the basic setup:

**Authentication** : choose Bearer Token. Pick a token with access to all the needed endpoints or create different data sources per token. Feel free to also add a restrictive list of allowed hosts.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-ds-auth.png&w=3840&q=75)


That's basically it, but it's a good practice to add a **Health check** with an endpoint that the Token has access to so you verify connectivity is OK.

## 3. Configure the Query in Grafana [¶](https://www.tinybird.co/docs/about:blank#3-configure-the-query-in-grafana)

Create a new Dashboard, edit the suggested Panel, and use the Data Source you just created.

For the example you'll consume the endpoint shown in this picture. A time series of sensor temperature and humidity readings.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-endpoint-sample.png&w=3840&q=75)


So, the in the Query editor is:

- Type: JSON
- Parser: Backend (needed for alerting and JSON parsing)
- Source: URL
- Format: Table
- Method: GET
- URL: `https://api.eu-central-1.aws.tinybird.co/v0/pipes/api_readings.json`
- Parsing options & Result fields > Rows/root: `data`   Needed cause the Tinybird response includes metadata, data, statistics...
- Parsing options & Result fields > Columns: add the needed fields, select types, and adjust time formats.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-query.png&w=3840&q=75)


It is important to use the Backend Parser for the Alerts to work and for compatibility with the root and field selectors as mentioned in the [plugin docs](https://grafana.com/docs/plugins/yesoreyeram-infinity-datasource/latest/query/backend/#root-selector--field-selector).

For the Time Formats you can use plugin options. By default it uses *Default ISO* so you can simply add `formatDateTime(t,'%FT%TZ') as t` to your Tinybird API Endpoint and don't need to configure the option in Grafana.

Save the dashboard and you should now be able to see a chart.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-dashboard.png&w=3840&q=75)


## 4. Using time ranges [¶](https://www.tinybird.co/docs/about:blank#4-using-time-ranges)

When you have millions of rows it's better to filter time ranges in Tinybird than to retrieve all the data and filter later when showing the chart in Grafana. You will get faster responses and more efficient use of the resources.

Edit the Tinybird Pipe to accept `start_ts` and `end_ts` parameters.

%
SELECT formatDateTime(timestamp,'%FT%TZ') t, temperature, humidity 
FROM readings
{% if defined(start_ts) and defined(end_ts) %}
WHERE timestamp BETWEEN parseDateTimeBestEffort({{String(start_ts)}}) 
   AND parseDateTimeBestEffort({{String(end_ts)}})
{% end %}
ORDER BY t ASC In the Query editor, next to URL, click on Headers, Request params and fill URL Query Params.

Use Grafana's global variables [$__from and $__to](https://grafana.com/docs/grafana/v9.0/variables/variable-types/global-variables/#__from-and-__to) defined with the time range selector at the top right od the dashboard.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-time-ranges.png&w=3840&q=75)


As said, filtering helps to have more efficient dashboards, here you can see how using filters, the scan size decreases.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-stats-filter.png&w=3840&q=75)


## 5. Dashboard variables [¶](https://www.tinybird.co/docs/about:blank#5-dashboard-variables)

You can also define dashboard variables and use them in the query:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-dashboard-variable.png&w=3840&q=75)


That will make it interactive:



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-variables.gif&w=3840&q=75)


Note you have to edit the Pipe:

%
SELECT formatDateTime(timestamp,'%FT%TZ') t,
{% if defined(magnitude) and magnitude != 'all' %}
{{column(magnitude)}}
{% else %}
temperature, humidity 
{% end %}
FROM readings
{% if defined(start_ts) and defined(end_ts) %}
WHERE timestamp BETWEEN parseDateTimeBestEffort({{String(start_ts)}}) 
   AND parseDateTimeBestEffort({{String(end_ts)}})
{% end %}
ORDER BY t ASC
## 6. Alerts [¶](https://www.tinybird.co/docs/about:blank#6-alerts)

Infinity plugin supports alerting, so you can create your own [rules](https://grafana.com/docs/grafana/latest/alerting/alerting-rules/) . In this example alert triggers when temperature goes outside a defined range.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-alert-definition.png&w=3840&q=75)


And this is what you will see in the dashboard.



<-figure->
![](/docs/_next/image?url=%2Fdocs%2Fimg%2Fgrafana-alert-trigger.png&w=3840&q=75)


Be sure to correctly set up [notifications](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/) if needed.

## Note [¶](https://www.tinybird.co/docs/about:blank#note)

Note: a previous version of this guide referred to the [JSON API plugin](https://grafana.com/grafana/plugins/marcusolsson-json-datasource/) but it was migrated to using [Infinity plugin](https://grafana.com/grafana/plugins/yesoreyeram-infinity-datasource/) since it's now the [default supported](https://grafana.com/blog/2024/02/05/infinity-plugin-for-grafana-grafana-labs-will-now-maintain-the-versatile-data-source-plugin/) one.



---

URL: https://www.tinybird.co/docs/publish/api-endpoints/guides/advanced-dynamic-endpoints-functions
Last update: 2025-01-23T13:11:21.000Z
Content:
---
title: "Advanced template functions for dynamic API Endpoint · Tinybird Docs"
theme-color: "#171612"
description: "Learn more about creating dynamic API Endpoint using advanced templates."
---




# Advanced template functions for dynamic API Endpoint [¶](https://www.tinybird.co/docs/about:blank#advanced-template-functions-for-dynamic-api-endpoint)

The [Template functions section](https://www.tinybird.co/docs/docs/cli/template-functions) of the [Advanced templates docs](https://www.tinybird.co/docs/docs/cli/advanced-templates) explains functions that help you create more advanced dynamic templates. On this page, you'll learn about how these templates can be used to create dynamic API Endpoint with Tinybird.

## Prerequisites [¶](https://www.tinybird.co/docs/about:blank#prerequisites)

Make sure you're familiar with [template functions](https://www.tinybird.co/docs/docs/cli/template-functions) and [query parameters](https://www.tinybird.co/docs/docs/work-with-data/query/query-parameters).

## Example data [¶](https://www.tinybird.co/docs/about:blank#example-data)

This guide uses the eCommerce events data enriched with products. The data looks like this:

##### Events and products data

SELECT *, price, city, day FROM events_mat
ANY LEFT JOIN products_join_sku ON product_id = sku
## Tips and tricks [¶](https://www.tinybird.co/docs/about:blank#tips-and-tricks)

When the complexity of Pipes and API Endpoints grows, developing them and knowing what's going-on to debug problems can become challenging. Here are some useful tricks for using Tinybird's product:

### WHERE 1=1 [¶](https://www.tinybird.co/docs/about:blank#where-11)

When you filter by different criteria, given by dynamic parameters that can be omitted, you'll need a `WHERE` clause. But if none of the parameters are present, you'll need to add a `WHERE` statement with a dummy condition (like `1=1` ) that's always true, and then add the other filter statements dynamically if the parameters are defined, like you do in the [defined](https://www.tinybird.co/docs/about:blank#defined) example of this guide.

### Use the set function [¶](https://www.tinybird.co/docs/about:blank#use-the-set-function)

The [set](https://www.tinybird.co/docs/docs/cli/advanced-templates#variables-vs-parameters) function present in the previous snippet lets you set the value of a parameter in a Node, so that you can check the output of a query depending on the value of the parameters it takes. Otherwise, you'd have to publish an API Endpoint and make requests to it with different parameters.

Using `set` , you don't have to exit the Tinybird UI while creating an API Endpoint and the whole process is faster, without needing to go back and forth between your browser or IDE and Postman or cURL.

Another example of its usage:

##### Using set to try out different parameter values

%
{% set select_cols = 'date,user_id,event,city' %}
SELECT
  {{columns(select_cols)}}
FROM events_mat You can use more than one `set` statement. Put each one on a separate line at the beginning of a Node.

`set` is also a way to set defaults for parameters. If you used `set` statements to test your API Endpoint while developing, remember to remove them before publishing your code, because if not, the `set` function overrides any incoming parameter.

### Default argument [¶](https://www.tinybird.co/docs/about:blank#default-argument)

Another way to set default values for parameters is using the `default` argument that most Tinybird template functions accept. The previous code could be rewritten as follows:

##### Using the default argument

%
SELECT
  {{columns(select_cols, 'date,user_id,event,city')}}
FROM events_mat Keep in mind that defining the same parameter in more than one place in your code in different ways can lead to inconsistent behavior. Here's a solution to avoid that:

### Using WITH statements to avoid duplicating code [¶](https://www.tinybird.co/docs/about:blank#using-with-statements-to-avoid-duplicating-code)

If you plan to use the same dynamic parameters more than once in a node of a Pipe, define them in one place to avoid duplicating code. This also makes it clearer which parameters will appear in the Node. You can do this with one or more statements at the beginning of a Node, using the `WITH` clause.

The WITH clause supports CTEs. These are preprocessed before executing the query, and can only return one row. This is different to other databases such as Postgres. For example:

##### DRY with the with clause

%
{% set terms='orchid' %}
WITH {{split_to_array(terms, '1,2,3')}} AS needles
SELECT
  *,
  joinGet(products_join_sku, 'color', product_id) color,
  joinGet(products_join_sku, 'title', product_id) title
FROM events
WHERE
  multiMatchAny(lower(color), needles)
  OR multiMatchAny(lower(title), needles)
### Documenting your API Endpoints [¶](https://www.tinybird.co/docs/about:blank#documenting-your-api-endpoints)

Tinybird creates auto-generated documentation for all your published API Endpoints, taking the information from the dynamic parameters found in the Pipe. It's best practice to set default values and descriptions for every parameter in one place (also because some functions don't accept a description, for example). This is typically done in the final Node, with `WITH` statements at the beginning. See how to do it in the [last section](https://www.tinybird.co/docs/about:blank#putting-it-all-together) of this guide.

### Hidden parameters [¶](https://www.tinybird.co/docs/about:blank#hidden-parameters)

If you use some functions like `enumerate_with_last` in the example [below](https://www.tinybird.co/docs/about:blank#enumerate-with-last) , you'll end up with some variables (called `x`, `last` in that code snippet) that Tinybird will interpret as if they were parameters that you can set, and they will appear in the auto-generated documentation page. To avoid that, add a leading underscore to their name, renaming `x` to `_x` and `last` to `_last`.

### Debugging any query [¶](https://www.tinybird.co/docs/about:blank#debugging-any-query)

Tinybird has an experimental feature that lets you see how the actual SQL code that will be run on Tinybird for any published API Endpoint looks, interpolating the query string parameters that you pass in the request URL. If you have a complex query and you'd like to know what is the SQL that will be run, [contact Tinybird](https://www.tinybird.co/docs/mailto:support@tinybird.co) to get access to this feature to debug a query.

## Advanced functions [¶](https://www.tinybird.co/docs/about:blank#advanced-functions)

Most of these functions also appear in the [Advanced templates](https://www.tinybird.co/docs/docs/cli/advanced-templates?#template-functions) section of the docs. The following are practical examples of their usage so that it's easier for you to understand how to use them.

### defined [¶](https://www.tinybird.co/docs/about:blank#defined)

The `defined` function lets you check if a query string parameter exists in the request URL or not.

Imagine you want to filter events with a price within a minimum or a maximum price, set by two dynamic parameters that could be omitted. A way to define the API Endpoint would be like this:

##### filter by price

%
{% set min_price=20 %}
{% set max_price=50 %}

SELECT *, price
FROM events_mat
WHERE 1 = 1
{% if defined(min_price) %}
  AND price >= {{Float32(min_price)}}
{% end %}
{% if defined(max_price) %}
  AND price <= {{Float32(max_price)}}
{% end %} To see the effect of having a parameter not defined, use `set` to set its value to `None` like this:

##### filter by price, price not defined

%
{% set min_price=None %}
{% set max_price=None %}

SELECT *, price
FROM events_mat
WHERE 1 = 1
{% if defined(min_price) %}
  AND price >= {{Float32(min_price)}}
{% end %}
{% if defined(max_price) %}
  AND price <= {{Float32(max_price)}}
{% end %} It's also possible to provide smart defaults to avoid needing to use the `defined` function at all:

##### filter by price with default values

%
SELECT *, price
FROM events_mat_cols
WHERE price >= {{Float32(min_price, 0)}}
  AND price <= {{Float32(max_price, 999999999)}}
### Array(variable_name, 'type', [default]) [¶](https://www.tinybird.co/docs/about:blank#arrayvariable-name-type-default)

Transforms a comma-separated list of values into a Tuple. You can provide a default value for it or not:

%
SELECT
    {{Array(code, 'UInt32', default='13412,1234123,4123')}} AS codes_1,
    {{Array(code, 'UInt32', '13412,1234123,4123')}} AS codes_2,
    {{Array(code, 'UInt32')}} AS codes_3 To filter events whose type belongs to the ones provided in a dynamic parameter, separated by commas, you'd define the API Endpoint like this:

##### Filter by list of elements

%
SELECT *
FROM events
WHERE event IN {{Array(event_types, 'String', default='buy,view')}} And then the URL of the API Endpoint would be something like `{% user("apiHost") %}/v0/pipes/your_pipe_name.json?event_types=buy,view`

### sql_and [¶](https://www.tinybird.co/docs/about:blank#sql-and)

`sql_and` lets you create a filter with `AND` operators and several expressions dynamically, taking into account if the dynamic parameters in a template it are present in the request URL.

It's not possible to use Tinybird functions inside the `{{ }}` brackets in templates. `sql_and` can only be used with the `{column_name}__{operand}` syntax. This function does the same as what you saw in the previous query: filtering a column by the values that are present in a tuple generated by `Array(...)` if `operand` is `in` , are greater than (with the `gt` operand), or less than (with the `lt` operand). Let's see an example to make it clearer:

- Endpoint template code
- Generated SQL

##### SQL_AND AND COLUMN__IN

%
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE {{sql_and(event__in=Array(event_types, 'String', default='buy,view'),
                section_id__in=Array(sections, 'Int16', default='1,2'))}} You don't have to provide default values. If you set the `defined` argument of `Array` to `False` , when that parameter isn't provided, no SQL expression will be generated. You can see this in the next code snippet:

- Endpoint template code
- Generated SQL

##### defined=False

%
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE {{sql_and(event__in=Array(event_types, 'String', default='buy,view'),
                section_id__in=Array(sections, 'Int16', defined=False))}}
### split_to_array(name, [default]) [¶](https://www.tinybird.co/docs/about:blank#split-to-arrayname-default)

This works similarly to `Array` , but it returns an Array of Strings (instead of a tuple). You'll have to cast the result to the type you want after. As you can see here too, they behave in a similar way:

##### array and split_to_array

%
SELECT
    {{Array(code, 'UInt32', default='1,2,3')}},
    {{split_to_array(code, '1,2,3')}},
    arrayMap(x->toInt32(x), {{split_to_array(code, '1,2,3')}}),
    1 in {{Array(code, 'UInt32', default='1,2,3')}},
    '1' in {{split_to_array(code, '1,2,3')}} One thing that you'll want to keep in mind is that you can't pass non-constant values (arrays, for example) to operations that require them. For example, this would fail:

##### using a non-constant expression where one is required

%
SELECT
    1 IN arrayMap(x->toInt32(x), {{split_to_array(code, '1,2,3')}}) If you find an error like this, you should use a Tuple instead (remember that `{{Array(...)}}` returns a tuple). This will work:

##### Use a tuple instead

%
SELECT
    1 IN {{Array(code, 'Int32', default='1,2,3')}} `split_to_array` is often used with [enumerate_with_last](https://www.tinybird.co/docs/about:blank#enumerate-with-last).

### column and columns [¶](https://www.tinybird.co/docs/about:blank#column-and-columns)

They let you select one or several columns from a Data Source or Pipe, given their name. You can also provide a default value.

##### columns

%
SELECT {{columns(cols, 'date,user_id,event')}}
FROM events
##### column

%
SELECT date, {{column(user, 'user_id')}}
FROM events
### enumerate_with_last [¶](https://www.tinybird.co/docs/about:blank#enumerate-with-last)

Creates an iterable array, returning a Boolean value that allows checking if the current element is the last element in the array. Its most common usage is to select several columns, or compute some function over them. See an example of `columns` and `enumerate_with_last` here:

- Endpoint template code
- Generated SQL

##### enumerate_with_last + columns

%
SELECT
    {% if defined(group_by) %}
        {{columns(group_by)}},
    {% end %}
    sum(price) AS revenue,
    {% for last, x in enumerate_with_last(split_to_array(count_unique_vals_columns, 'section_id,city')) %}
        uniq({{symbol(x)}}) as {{symbol(x)}}
        {% if not last %},{% end %}
    {% end %}
  FROM events_enriched
{% if defined(group_by) %}
    GROUP BY
         {{columns(group_by)}}

    ORDER BY
        {{columns(group_by)}}
{% end %} If you use the `defined` function around a parameter it doesn't make sense to give it a default value because if it's not provided, that line will never be run.

### error and custom_error [¶](https://www.tinybird.co/docs/about:blank#error-and-custom-error)

They let you return customized error responses. With `error` you can customize the error message:

##### error

%
{% if not defined(event_types) %}
  {{error('You need to provide a value for event_types')}}
{% end %}
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE event IN {{Array(event_types, 'String')}}
##### error response using error

{"error": "You need to provide a value for event_types"} And with `custom_error` you can also customize the response code:

##### custom_error

%
{% if not defined(event_types) %}
  {{custom_error({'error': 'You need to provide a value for event_types', 'code': 400})}}
{% end %}
SELECT
  *,
  joinGet(products_join_sku, 'section_id', product_id) section_id
FROM events
WHERE event IN {{Array(event_types, 'String')}}
##### error response using custom_error

{"error": "You need to provide a value for event_types", "code": 400} **Note:** `error` and `custom_error` have to be placed at the start of a node or they won't work. The order should be:

1. `set`   lines, to give some parameter a default value (optional)
2. Parameter validation functions: `error`   and `custom_error`   definitions
3. The SQL query itself

## Putting it all together [¶](https://www.tinybird.co/docs/about:blank#putting-it-all-together)

You've created a Pipe where you use most of these advanced techniques to filter ecommerce events.

This is its code:

##### advanced_dynamic_endpoints.pipe

NODE events_enriched
SQL >

    SELECT
        *,
        price,
        city,
        day
    FROM events_mat_cols
    ANY LEFT JOIN products_join_sku ON product_id = sku



NODE filter_by_price
SQL >

    %
        SELECT * FROM events_enriched
        WHERE 1 = 1
        {% if defined(min_price) %}
          AND price >= {{Float32(min_price)}}
        {% end %}
        {% if defined(max_price) %}
          AND price <= {{Float32(max_price)}}
        {% end %}



NODE filter_by_event_type_and_section_id
SQL >

    %
    SELECT
          *
        FROM filter_by_price
        {% if defined(event_types) or defined(section_ids) %} ...
            WHERE {{sql_and(event__in=Array(event_types, 'String', defined=False, enum=['remove_item_from_cart','view','search','buy','add_item_to_cart']),
                            section_id__in=Array(section_ids, 'Int32', defined=False))}}
        {% end %}



NODE filter_by_title_or_color
SQL >

    %
    SELECT *
    FROM filter_by_event_type_and_section_id
    {% if defined(search_terms) %}
    WHERE
      multiMatchAny(lower(color), {{split_to_array(search_terms)}})
      OR multiMatchAny(lower(title), {{split_to_array(search_terms)}})
    {% end %}



NODE group_by_or_not
SQL >

    %
        SELECT
            {% if defined(group_by) %}
              {{columns(group_by)}},
              sum(price) AS revenue,
              {% for _last, _x in enumerate_with_last(split_to_array(count_unique_vals_columns)) %}
                  uniq({{symbol(_x)}}) as {{symbol(_x)}}
                  {% if not _last %},{% end %}
              {% end %}
            {% else %}
              *
            {% end %}
        FROM filter_by_title_or_color
        {% if defined(group_by) %}
            GROUP BY {{columns(group_by)}}
            ORDER BY {{columns(group_by)}}
        {% end %}



NODE pagination
SQL >

    %
    WITH
      {{Array(group_by,
      'String',
      '',
               description='Comma-separated name of columns. If defined, group by and order the results by these columns. The sum of revenue will be returned')}},
      {{Array(count_unique_vals_columns, 'String', '',
               description='Comma-separated name of columns. If both group_by and count_unique_vals_columns are defined, the number of unique values in the columns given in count_unique_vals_columns will be returned as well')}},
       {{Array(search_terms, 'String', '',
              description='Comma-separated list of search terms present in the color or title of products')}},
       {{Array(event_types, 'String', '',
               description="Comma-separated list of event name types", enum=['remove_item_from_cart','view','search','buy','add_item_to_cart'])}},
       {{Array(section_ids, 'String', '',
               description="Comma-separated list of section IDs. The minimum value for an ID is 0 and the max is 50.")}}
    SELECT * FROM group_by_or_not
    LIMIT {{Int32(page_size, 100)}}
    OFFSET {{Int32(page, 0) * Int32(page_size, 100)}} To replicate it in your account, copy the previous code to a new file called `advanced_dynamic_endpoints.pipe` locally and run `tb push pipes/advanced_dynamic_endpoints.pipe\` with the Tinybird [CLI](https://www.tinybird.co/docs/docs/cli/install) to push it to your Tinybird account.



---