May 02, 2021

Analyzing our own Nginx logs with ClickHouse on Tinybird

How we analyze our own logs in real-time using our own product.
Xoel LĂłpez
Founder at TheirStack

In a recent post we walked through the steps required to use Tinybird to analyze Nginx logs. In this one we’ll share how we are using our own product internally to analyze our own Nginx traffic in real-time and automatically, and how you can do it too.

To follow along, you’ll need 3 things:

  • A Tinybird account. Sign in if you don’t have one yet.
  • The code from this GitHub repo: it contains a Tinybird project Data source configurations and Pipes that will work out of the box.
  • tbtail, a tool to stream your logs to Tinybird, installed in the machines where your Nginx logs are generated.

With that, you can quickly get a sense of what’s going on with all your traffic in real-time, as well as examples for you to start slicing and dicing your logs in however other ways you want.

We’re still in private beta, so after you sign in with Google, Github or Microsoft authentication, your account won’t be active. Email us at hi@tinybird.co and we’ll activate it for you.

SCHEMA > Evolution
Get 10 links weekly to the Data and AI articles the Tinybird team is reading.

Run these commands to clone the data project locally, replicate it on Tinybird and have everything set and ready for tbtail to start sending logs to your account:

The last thing is installing tbtail. The details on how to do it are in its repo. We published deb and bin packages, and you can also compile it from source.

After it’s installed, you just need to run this command in the machine where the Nginx logs are created to send logs to Tinybird automatically.

Then, if you go to your dashboard, you’ll see a new pipe called query_grouped_requests. It defines an endpoint that lets you query your aggregated data. In the Pipe, you can click on the green “View API” button on the top right corner of the page to see its live documentation.

Do you like this post?

Related posts

Publish SQL-based endpoints on NGINX log analysis
A new dashboard for Tinybird Analytics
Build a Datadog alternative in 5 minutes
Operational Analytics in Real Time with Tinybird and Retool
I rebuilt the Auth0 Activity Page with webhooks and Tinybird
Tinybird Local: Build with data on your machine
Tinybird at South Summit Madrid 2019
Tinybird connects with Confluent for real-time streaming analytics at scale

Tinybird

Team

Jul 18, 2023
Ship data as you ship code: Tinybird is becoming local-first.
More Data, More Apps: Improving data ingestion in Tinybird

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Tinybird is not affiliated with, associated with, or sponsored by ClickHouse, Inc. ClickHouseÂŽ is a registered trademark of ClickHouse, Inc.