Increased observability capabilities in Service Data Sources¶

For API Endpoint requests as well as for calls to the Query API you can access the amount of read data and rows, duration of the API call and error information (if relevant) in the pipe_stats_rt Service Data Source, as well as what parameters were used on each request. Now, you can do the same thing to monitor ingest and materialization usage.

Read and written bytes and rows are now also available in the datasources_ops_log Service Data Source for every ingestion operation, whether that’s new data ingested, materializations or replaces.

And lastly, it’s now possible to track your total storage in the new datasources_storage Service Data Source.

Remember that these service Data Sources can be queried from your Pipes so you can easily create API Endpoints or Time Series visualizations to analyze your own usage.

Other notable changes ¶

• The Kafka connector now supports ingestion of messages encoded with non UTF-8 message keys. Until now, we were assuming that everyone would use UTF-8 keys in their messages but… apparently not!
• Improved performance for ingestion of large JSON and Parquet files, specifically for files with a large number of columns.
• Also fixed schema detection issues when importing large Parquet files from a URL. Now, we parse the whole file and use the actual schema defined in the footer, instead of trying to infer the schema as we do with other schemaless formats. As a result of this change, the whole import process will be more accurate and predictable.
• Added support for "bytes" type in Parquet files.
• Certain situations were causing Materialized Views populate jobs to appear stuck. Various improvements to progress tracking.
• If the underlying ClickHouse® process was restarted while a Populate job was waiting for a query to finish, it would cause the job to get stuck and never finish. Now solved.
• populateview entries in the Service Data Source tinybird.datasources_ops_log were not being properly registered when pushing a Pipe with force=true. They are now.
• When creating a Materialized View, the columns in the GROUP BY need to match those in the Sorting Key of the destination data source. Tinybird validates that this is the case when you create a Materialized View, but the validation was being triggered incorrectly in some instances.
• Fixed an issue that was exposing some internal error information when some operations on the Events API failed. Now we return an error code and a generic "Internal Tinybird error" message without leaking any unnecessary internal information.
• Now, the CLI shows warnings with tips when uploading a materialization Pipe. These warnings link to documentation where you can learn how to improve your queries. Right now, there are only a few tips, but we will be adding more.
• We've released a new homepage for Tinybird’s Documentation.