Build a user-facing web analytics application with Confluent and Tinybird¶

In this guide you'll learn how to take data from Kafka and build a user-facing web analytics dashboard using Confluent and Tinybird.

TL;DR¶

In this tutorial, you will learn how to:

1. Connect Tinybird to a Kafka topic.
2. Build and publish Tinybird API Endpoints using SQL.
3. Create 2 Charts without having to code from scratch.

Prerequisites¶

To complete this tutorial, you'll need:

1. A free Tinybird account
2. An empty Tinybird Workspace
3. A Confluent account
4. Node.js >=20.11

This tutorial includes a Next.js app for front-end visualization, but you don't need working familiarity with TypeScript - just copy & paste the code snippets.

1. Setup¶

Clone the demo_confluent_charts repo.

2. Create your data¶

Option 1: Use your own existing data¶

In Confluent, create a Kafka topic with simulated e-commerce events data. Check this file for the schema outline to follow.

Option 2: Mock the data¶

Use Tinybird's Mockingbird, an open source mock data stream generator, to stream mock web events instead.

In the repo, navigate to /datagen and run npm i to install the dependencies.

Create an .env and replace the default Confluent variables:

cp .env.example .env

Run the mock generator script:

node mockConfluent.js

3. Connect Confluent <> Tinybird¶

In your Tinybird Workspace, create a new Data Source using the native Confluent connector. Paste in the bootstrap server, rename the connection to tb_confluent, then paste in your API key and secret. Select "Next".

Search for and select your topic, and select "Next". Ingest from the earliest offset, then under "Advanced settings" > "Sorting key" select timestamp.

Rename the Data Source to ecomm_events and select "Create". Your Data Source is now ready, and you've connected Confluent to Tinybird! You now have something that's like a database table and a Kafka consumer combined. Neat.

4. Transform your data¶

Query your data stream¶

Your data should now be streaming in, so let's do something with it. In Tinybird, you can transform data using straightforward SQL in chained Nodes that form a Pipe.

Create a new Pipe and rename it sales_trend. In the first Node space, paste the following SQL:

SELECT timestamp, sales FROM ecomm_events
WHERE timestamp >= now() - interval 7 day

This gets the timestamp and sales from just the last 7 days.

Run the query and rename the Node filter_data.

In the second Node space, paste the following:

SELECT toDate(timestamp) AS ts, sum(sales) AS total_sales from filter_data
GROUP BY ts
ORDER BY ts

This casts the timestamp to a date as ts, and sums up the sales - meaning you can get a trend of sales by day.

Run the query and rename the Node endpoint.

Publish your transformed data¶

In the top right of the screen, select "Create API Endpoint" and select the endpoint Node. Congratulations! It's published and ready to be consumed.

5. Create a Tinybird Chart¶

In the top right of the screen, select "Create Chart".

Rename the chart "Sales Trend" and select and Area Chart. Under the "Data" tab, select ts as the index and total_sales as the category.

You should see a Chart magically appear! In the top right of the screen, select "Save".

6. Run an app locally¶

View the component code for the Chart by selecting the code symbol (<>) above it. Copy this code and paste into a new file in the components folder called SalesTrend.tsx.

In page.tsx, replace <p>Chart 1<p> with your new Chart <SalesTrend />. Save and view in the browser with npm run dev. You should see your Chart!

Create a second Pipe --> Chart¶

Create a second Pipe in Tinybird called utm_sales:

SELECT utm_source, sum(sales) AS total_sales FROM ecomm_events
WHERE timestamp >= now() - interval 7 day
GROUP BY utm_source
ORDER BY total_sales DESC

This gets sales by utm over the last 7 days.

Run the query and rename the Node endpoint.... Then, you guessed it! Publish it as an Endpoint, create a Chart, and get the code. This time, create a donut Chart called "UTM Sales" with utm_source as the index and total_sales as the category.

Create a new component file called UTMSales.tsx and import in page.tsx replacing Chart 2. You did it!

Next steps¶

• Read more about Tinybird Charts.
• Use Charts internally to monitor latency in your own Workspace.