October 11, 2024

Improved Data Flow experience

We added new features to the Data Flow so you can work and navigate through the resources easier:

  • Pipes can now be opened in split screen like Data Sources.
  • Filter the resources by tag, keyword or type.
  • See all the dependencies of a resource, including connectors and shared Workspaces.
  • Switch the nodes between collapsed or expanded mode.

Image

DynamoDB Connector now adds Old Image column for NEW_AND_OLD_IMAGES streams

Our native DynamoDB connector now supports OLD_IMAGE property ingestion for tables with NEW_AND_OLD_IMAGES stream type feature enabled. This update allows you to track changes in your records over time more effectively.

For new Data Sources created from scratch, you'll notice a new column automatically added: _old_record. This column will store the previous state of the record before any updates.

For existing tables, if your DynamoDB table has NEW_AND_OLD_IMAGES stream type enabled, you'll need to make a small adjustment to take advantage of this feature. Alter your Data Source by adding a new column with the following configuration: `_old_record` Nullable(String) `json:$.OldImage`

Dedicated Infrastructure Monitoring

If you're using dedicated database clusters in an infrastructure-based pricing plan, we offer three ways of monitoring the state of your clusters:

  • Organization's metric logs Service Data Source: This data source provides detailed logs for monitoring metrics across your infrastructure.
  • Prometheus endpoint: This endpoint provides Prometheus ready metrics that you can integrate with the observability platform of your choice.
  • Cluster load charts: Accessible under Organization settings, Usage, this charts provides a visual representation of your cluster load, helping you track performance at a glance.

Image

Bug fixes and improvements

  • Added a new section about using the Query API with metadata parameters to our Analyze Endpoints Performance guide.
  • Added a Gitlab .gitlab-ci.yml template to our CI/CD guide.
  • Now if a DynamoDB import job fails, the remaining files will be imported regardless of the previous failed job.
  • The Pipe url is now updated when you rename a Pipe.
  • Users can now run a Time Series without the need of changing any parameter value.
  • Fixed an issue where deleted elements in DynamoDB Data Sources were incorrectly appearing in SELECT query results. This has been resolved for newly created tables as of Sept 28th. Affected customers will be contacted through communication channels.
  • Upgraded the BI connector to use Postgres 14.13, it comes with two benefits: it's supported until Nov. 2026 and improves connection management.
View previous update