Dec 09, 2024

How to analyze usage from your MCP Server

Since Anthropic launched Model Context Protocol (MCP), thousands of developers have built MCP Servers to connect Claude AI to their unique contexts. Here's how to monitor your MCP Servers for errors and analyze usage with standard logging handlers, Tinybird, and Grafana.
Cameron Archer
Head of Marketing
Alberto Romeu
Backend Developer

Two weeks ago Anthropic AI launched Model Context Protocol (MCP), an open source framework that defines a standard server-client architecture to make LLM applications context-aware.

Since the launch, thousands (millions?) of developers have built MCP Servers.

We were one of the first adopters of MCP, and we built and tested our first server MCP Server just two days after the announcement from Anthropic. We think it can create a lot of opportunities to use Tinybird in new ways.

But vision was getting ahead of practice, and we thought it would be a good idea to put some basic observability infrastructure in place to monitor how our MCP server is being used. Something that would help us discover errors, understand how the tools were being used, and get a feel for how we could make v1 better than v0.

For now, MCP Servers are installed locally. Anthropic has mentioned they'll add SDKs for remote production servers in the future, but for now, it's local. Thus, there's nothing pre-built for remote logging collection and analysis.

So, after we built our MCP Server and started testing it locally (and encouraging our community to do the same), we quickly added logging and analysis features for metrics like:

  • Total sessions with the server
  • Total requests to the server
  • Total errors on the server
  • Requests/errors by tool
  • Error rate by tool
  • etc.

Using a combination of Python/Typescript logging libraries, Tinybird (and the newly introduced Prometheus endpoints), and Grafana, we've built a nice MCP Server Analytics tool and an observability dashboard so that we can see how our server is being used by the various people cloning and testing it on their local machines.

The Grafana dashboard we built to visualize metrics from our MCP Server

If you want to do the same for your own MCP Server, we've put all the code in this repository with an example (the Tinybird MCP Server). To deploy the analytics tool, just…

  1. Create a free Tinybird account
  2. Deploy the project from the repo to your Tinybird Workspace
  3. Configure your chosen language's built-in logging handler to send MCP server events to Tinybird
  4. Add the Tinybird token generated for you to your server code

From there you'll have several pre-built Prometheus endpoints that you can hook up to Grafana, Datadog, or wherever you do your o11y. You can then extend your metrics by adding new Prometheus endpoints in Tinybird based on what you need from your server. Anytime somebody clones and deploys your server, it will send observability logs to your Tinybird Workspace.

If you're ready to start, just go to the repository and follow the steps in the README (You'll also find the Tinybird MCP Server and an example in the root folder). 

Below I'll share a bit more about how the tool works.

How the MCP Server Analytics tool works

The MCP Server Analytics repo gives you code to configure the logging handlers in either Python or Typescript to send MCP server events to a Tinybird Workspace using the Tinybird Events API, an HTTP streaming endpoint.

Tinybird writes those events into a managed database which you can query with SQL to build various metrics. You then publish metrics as APIs in various formats. The repo uses the Prometheus endpoint format to simplify interoperability with various observability tools.

The code in the repo is set up so that you can create logging handlers to send data to a Tinybird Workspace. The project includes sample Tinybird Pipes that publish various metrics as Prometheus endpoints.

The example in the repository contains Tinybird .pipe files that define a series of SQL nodes to transform the logged events into various metrics. The Pipes included log...

  • total errors
  • errors last 5m
  • total requests
  • requests last 5m
  • total sessions
  • sessions last 5m
  • errors by tool
  • requests by tool
  • sessions by app

Write some more SQL to extend these metrics for your specific MCP Server use case.

Why use Tinybird for this?

Good question. Certainly, you could use a more specialized tool like Loki or statsd, but these are generally the domain of infra/platform teams. Tinybird is data infrastructure for software engineers, so it can be used to build and deploy applications while also integrating with common monitoring tools like Grafana.

With the Prometheus format for endpoints, you can do what we've done here (e.g. build an observability dashboard), but you can also build RESTful HTTP endpoints using the same data for other applications, like user-facing dashboards or in-product analytics.

Questions?

Check out the Tinybird MCP repo or ask to join the Tinybird Slack community. Alternatively, tag us with your questions on Twitter or LinkedIn.

Do you like this post?

Related posts

How to monitor Tinybird using Datadog with vector.dev
Building a faster CI pipeline with Turborepo and pnpm
Product design: how our SaaS integrates with git
Behind the scenes of Tinybird's big frontend refactor
We have released a stable version of the Tinybird CLI
Automating customer usage alerts with Tinybird and Make
More Data, More Apps: Improving data ingestion in Tinybird
Using custom Kafka headers for advanced message processing
Building an enterprise-grade real-time analytics platform
Horizontally scaling Kafka consumers with rendezvous hashing

Build fast data products, faster.

Try Tinybird and bring your data sources together and enable engineers to build with data in minutes. No credit card required, free to get started.
Need more? Contact sales for Enterprise support.