We live in a world that – more than ever – is filled with data. Collecting data of all kinds has become fashionable, and this has led to the proliferation of key performance indicators (KPIs). We measure all kinds of things, and it has slowly but steadily become common practice to move to pay-for-performance models in every sphere of life including businesses, policing, healthcare, education, non-profits, government, and more.
The assumption is that if the KPIs are achieved and trending in the right direction (usually up and to the right corner) over time, everything is good. But in reality, the assumption is far from true. When people are paid for performance, and their performance is measured via metrics or KPIs that are far removed from reality – and those metrics are used to dole out rewards or punishment, people find creative ways to game the system. In the short term, it appears that everything is good because the goals are met and trending in the right direction, but in the medium to long term, using these metrics has an adverse impact on the organization.
We have known this intuitively for a long time. We have many examples – Enron, Wells Fargo, Volkswagen, spiraling costs of healthcare and education, policing (where big crimes are reported as smaller crimes because the frontline policing staff wants to show they are reducing the crime rate – as their jobs might depend on it), etc. We even see this happening in our homes. If you have teenagers, you know that they can be extremely creative in achieving the goals that you set for them without actually doing what you want them to do.
What can we do about this? We still need metrics to measure how we are performing so that we can continue to monitor and improve performance. And yet, we don’t want to fall in the trap of measuring either the wrong things or pushing people to game the metrics.
I recently read The Tyranny of Metrics by Jerry Z. Mueller. In the book, he elegantly explains all the different ways that metrics fixation has negatively impacted organizations and society at large and how the fixation on metrics can create chaos.
I have written about KPIs and metrics here, here, and here. I already knew that KPIs need to be defined with a lot of intention and in partnership with the people who will be held responsible for them, and I also knew about gaming of KPIs in business contexts, but I was not aware that gaming KPIs is so prevalent across all kinds of organizations.
Data vs. judgment
There are two ways to measure any particular performance: with data or by having someone who knows the job judge the performance. In the recent past, as it has become easier to collect data, we have moved towards collecting a lot of data and using it as the primary way to measure performance.
What we need to understand is that measured data is almost never an alternative to judgment. In fact, measurement demands judgment – judgment about whether to measure, what to measure, how to evaluate the significance of what is being measured, how to decide if rewards and penalties need be attached to the results, and last, judgment about who should have access to the measured data.
Guidelines on measurements
At the end of the book, Jerry Mueller provides a checklist on when and how to create performance metrics and what we need to keep in mind when analyzing them.
Below are some guidelines based on his checklist, with my perspectives added.
1. What do you plan to measure?
The kind of information you plan to measure has an important impact on how you design your metrics. The more the object to be measured resembles inanimate matter, the more likely it is measurable (natural sciences or engineering).
When the objects being measured are influenced by the process of measurement, then the measurement becomes less reliable. It’s even more unreliable if the measurement is about human activity.
People are conscious and will respond to the process of being measured. And if rewards and punishments are involved, they will react in a way to skew the validity of the measurements.
2. What will the measures be used for?
Metrics are more useful when they’re used by people to track performance to compare themselves with their peers and to identify potential areas of improvement.
When metrics are used by external parties (including senior management, government, etc.) who may not recognize their limitations to dish out rewards or punishments, this leads to behavior to game the system and are more harmful in the mid-to-long term.
It is good to offer recognition to those who excel and assistance to those who are behind.
KPIs that help accentuate the internal motivations of the people being measured, so they can continue to improve, tend to be better than KPIs designed to get people to behave in ways that they are not intrinsically motivated by or that appeal to extrinsic motivations.
Low-stakes metrics are almost always better than high-stakes metrics.
If the performance being measured does not require or has the potential for any intrinsic rewards, pay for performance might work really well.
3. How useful is the measurement?
Just because something is measurable doesn’t mean that it is worth measuring. In some cases, the ease of measurement could be inversely proportional to the significance or usefulness of what is being measured.
Is what you are measuring a proxy for what you really want to know?
If the metric is not useful nor a proxy for what you really want to know, you are better off not measuring it at all.
4. How useful are more metrics?
Performance measurements, when useful, are more effective in finding outliers, especially poor performers or true misconduct. They might not be very useful in distinguishing people in the middle or at the top of the ladder.
The more you measure, the marginal cost of measuring will become more than the marginal improvements you gain through measurement. So, the fact that metrics are helpful doesn’t mean that more metrics are more helpful.
In fact, it is easier, simpler, and faster to use fewer metrics but to use ones that are relevant to drive behavior and improve performance. You can change the KPI, depending on the maturity of performance, instead of having the same KPI to measure everyone’s performance (regardless of how mature the person or process doing the activity is).
5. What are the costs of measurement?
There is always a cost of acquiring the data for the metrics. Every moment that we devote to producing metrics is time not spent on doing what is being measured. So, before we initiate a new metric, we need to identify the costs of measuring.
We can also look to see if there are other sources of information about performance that can be used that are based on the judgment and experience of customers or someone else.
6. Who develops the metrics and what do they mean?
How and by whom are the measures of performance developed? Measurements are more likely to be beneficial if developed from the bottom up with inputs from the people who will be measured (experience and judgment).
KPIs defined by people who don’t have context on the performance being measured are typically the ones that are gamed or manipulated, as they may neither add value nor increase the difficulty of performing the activity.
In conclusion
KPIs are effective to the extent that the people being measured believe in their worth and effectiveness. We need to remember that even the best KPIs are subject to gaming and corruption. Keep an eye on the result of what is being measured.
Sometimes, recognizing the limits of the possible is the beginning of wisdom. Not all problems are solvable, and even fewer are soluble by metrics. It’s not true that everything can be improved by measurement nor that whatever can be measured can be improved.
We need to exercise judgment, which comes from experience, before introducing any new metrics. Metrics should inform judgment and vice versa.
As Jerry Mueller concludes in his book, sometimes the best use of metrics is not to use them at all.
For more insight on leadership, see Leadership And The Importance Of Being Fully Present In The Moment.
This post was originally published on “Leading Transformation” and has been republished here with permission.
The post How To Define And Use Metrics Effectively appeared first on Business Intelligence Info.