"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales."
I was too taken aback to follow up asking why # of sales can't be gamed too.
Number of sales (or customers) is a vanity metric favored by startups looking to entice investment. Revenue is a metric useful for measuring actual business health.
Any chance the interviewer was overly exit eager ?
Why is that credit? Good interview questions don't have a right or wrong answer, they have a right or wrong process of coming up with an answer. Especially for a question like "Pick a metric," almost any relevant metric is defensible and has tradeoffs, and the question should be about how the interviewee analyzes engineering tradeoffs, gathers requirements, defends their initial proposal, and is open to counterarguments.
(If there's a correct answer, you can just give them a quiz. If there's a Googleable correct answer, you can just employ them and give them Google.)
But of course, "If you can't measure it, you can't manage it", said by Drucker, says EVERYONE. Hence the cherry picking of preferential KPIs or OKRs.
But, actually, Drucker didn't say it. In fact, W. Edwards Deming said it. And what he actually said was "It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.". So, basically, the quote held up as the "reason" behind excessive over-tracking of the wrong thing... is wrong.
That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed.
Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy.
IMO there must be high-level metrics which define which low-level metrics are valuable today.
It's getting weirder when you factor in shareholder value: sacrificing the future of the company for looking good in all the fashionable metrics might be exactly the leaders job, depending on how the board is made up.
Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted).
In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.
That's why the goal in so many large public companies becomes "Make the stock price go up." If the stock price goes up, the executives and everyone else in the company with stock can unload it, take the cash, and let the new shareholders deal with any long-term consequences of their actions.
That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
the problem with figuring out how the numbers are being achieved is that, without tracking everything totally and fully, you'll be reliant on some kind of estimate/measure. Which can also then be gamed.
