Consider a simple webserver. We want to record metrics for the total requests we’ve processed, as well as how many requests are in flight.
If you look at Prometheus’ docs, you will find a suggestion that “requests processed” should be recorded as a counter, and “requests in flight” should be a gauge:
A counter is a cumulative metric …whose value can only increase or be reset to zero on restart. For example, you can use a counter to represent the number of requests served, tasks completed, or errors.
A gauge is a metric …that can arbitrarily go up and down. .. Gauges are typically used for measured values like temperatures or current memory usage, but also “counts” that can go up and down, like the number of concurrent requests.
The tower_http library also provides a gauge-like metric for in-flight requests, for Rust HTTP servers.
Consider an alternative:
received, i.e. when the server begins processingcompleted, i.e. when the server stops processingThis is the same number of metrics as above, (2) but both are meaningful when used with rate.
We have not lost information: received - completed == in_flight, so we can still compute the instantaneous in-flight number
(as a recording rule, if you want it easily accessible.)
To my mind, tracking the two counters makes more sense. “In flight” is a consequence of “received but not completed”; expressing that in the metrics matches my mental model.
Consider a different example: CPU utilization. We could have a gauge for “how busy was the CPU in the last 1 second”, or a cumulative counter of “how much time has the CPU been not-idle”.
A four-second burst of activity might look like:
| time | gauge | counter |
|---|---|---|
| 0s | 1% | 0.01s |
| 1s | 100% | 1.01s |
| 2s | 100% | 2.01s |
| 3s | 100% | 3.01s |
| 4s | 100% | 4.01s |
| 5s | 1% | 4.02s |
But we can’t sample this metric continuously. If we’re only sampling it every five seconds, we would see:
| time | gauge | counter |
|---|---|---|
| 0s | 1% | 0.01s |
| … | … | … |
| 5s | 1% | 4.02s |
The gauge is misleading: it makes it look like the CPU is idle. The counter, on the other hand, shows the CPU has been busy–80% utilization.
Some people will say “all you need are traces” or “all you need are logs”. In principle, I agree; pragmatically, I often choose to only add metrics.1
An underlying argument of those articles is that we often care about events rather than state. This is an useful insight: we can often (not always) reconstruct state by replaying events. We can’t necessarily reconstruct the events that led to a particular state.2
I don’t think there’s is, or should be, a hard rule here about what to track. But these are some things I’ll be thinking about the next time I add a metric:
Which am I interested in, events or state? Is the state a consequence of some events that I care about? Or does the state stand on its own?