For as long as I’ve worked in security, there’s been a running joke about “vanity metrics” — big numbers, flashy charts, etc. They’re easy to understand and report, usually reflect well on the team, and often, utterly disconnected from anything that actually matters to the business.
But over the years, I’ve come to realize something important: what we often call vanity metrics aren’t necessarily being used because someone is trying to pad their performance report. More often than not, they’re being used because they’re the only metrics available. Or at least, the only ones people understand how to measure.
And that’s a problem.
As our field evolves — and as security becomes more data-driven, more cross-functional, and more integral to how modern businesses operate — so too must our approach to measurement. We need to think differently about what we track, why we track it, and what story it tells.
Why Vanity Metrics Persist
In the annual Voice of Security research, the data confirmed what many of us feel instinctively: security teams continue to rely heavily on metrics that don’t meaningfully reflect performance or business impact.
Why? Because many of the alternatives are difficult to access or define. Or they require the kind of statistical thinking that hasn’t historically been part of security culture.
When I was leading security operations at Coinbase, we were lucky enough to have a data scientist embedded with us. Suddenly, we had someone who could look at our fraud and abuse problems with rigor. Here was someone who understood confounding variables, who could test assumptions, and who could help design meaningful ways to measure impact. And I found myself thinking: how do I get someone like you for every part of the security team?
Security hasn’t always been required to think this way. But that’s changing — especially in a world increasingly influenced by AI, where data science is the foundation.
If we want to lead effectively in this new era, we need to level up how we think about data.
A New Framework for Security Metrics
So how should we approach measurement in security operations? I like to break it down into four categories.
1. Useless to Measure
These are metrics that simply don’t tell you anything actionable. Things like arbitrary log counts, or dashboards you inherited and never questioned. If a metric doesn’t drive any insight or decision-making, cut it.
2. Good to Measure, Not to Optimize
Some metrics are worth tracking — not because you can influence them, but because they explain context. The number of incidents your team handles is a good example. You can’t control how many attackers target your organization. But you can use that number to understand team workload, burnout, and why you might’ve missed a few OKRs last quarter.
Another good example is escalation rates from Tier 1 to Tier 2. This can help you understand the complexity of alerts or potential training gaps, but it’s not something to gamify. High escalation rates might simply reflect the reality of your threat environment, not a deficiency in Tier 1.
3. Good to Measure and Improve
Then there are the metrics you should be targeting. These are the ones your team directly controls — things like false positive rates in your alerting systems. High false positives mean alert fatigue, missed threats, and team burnout. You can’t ignore them, and you can absolutely improve them.
Automation coverage is another big one — tracking what percentage of your tasks are handled through automation rather than manually. The higher that percentage that can be automated securely, the less repetitive work your analysts have to do. That means faster responses, fewer errors, and more time for strategic projects.
These are the types of metrics we should hold ourselves accountable to. They reflect both team performance and security posture.
4. Metrics That Require Translation for Executives and Boards
Here’s where things often go sideways: taking an operational metric — like number of incidents or detection accuracy — and sending it straight to the board.
Let’s be real: no board member cares about false positives. They care about materiality. About financial impact. About whether you’ll need to file an 8-K with the SEC. So instead of forcing operational metrics upstream, build a translation layer. Connect what your team measures to what the business needs to understand.
What Comes Next
Security leaders need a new mental model for metrics — one that balances technical insight with strategic value. I’m envisioning a quadrant or maturity curve, something that maps:
- How much control you have over a metric (low to high)
- How relevant it is to business stakeholders (low to high)
It’s a way to prioritize, communicate, and ultimately drive better decisions — within the team and across the business.
If we want to be seen as partners to the business, we need to speak the business’s language. That means leaving behind the comfort of familiar metrics and embracing the challenge of more rigorous, meaningful ones. It’s not easy — but it’s essential.
Source: Security Magazine