Managing engineers based on what people say doesn’t cut it anymore in the era of tighter budgets. People are insanely biased, so you need data to make good decisions.
If you rely only on conversations and surveys, you’re going to have these major efficiency problems:
(1) You’ll put a lot of effort in the wrong places
We did a test at Collage.com where we surveyed the customer service team. We asked how many times they personally experienced a severe bug in the past week that corrupted customer data.
The catch is that we knew this bug had only happened once based on system logs.
The results? The combined surveys indicated that the issue had happened OVER 50 TIMES!
Availability bias is super powerful and everyone is susceptible (including leaders), so you need data to know the real impact of issues associated with strong emotions.
(2) You won’t put effort where it’s badly needed
Bias is even more dangerous when it runs the other way and people normalize major inefficiencies.
We had another bug at Collage.com where the shopping cart wouldn’t scroll and it was impossible to check out with multiple items.
The way customers describe this when they call support though, it sounds like “user error” or “browser issues.”
This went on for days and days before someone actually found the bug while testing, even though a bunch of customers had called us about it.
We had become so used to dealing with similar-sounding non-issues that we didn’t think anything of these warning signs.
As a manager, you may be shocked to see what people endure every day without thinking to speak up when you dig into the data.
(3) You’ll create a highly political organization
Not having data doesn’t prevent people from forming strong opinions.
At best, engineers will waste a ton of time arguing about things.
At worst, the loudest voice will be the wrong one, and your quieter, more talented engineers will quit.
--------
WHAT TO DO INSTEAD
It’s still important to listen to people, and the fact that someone feels bad about something is a problem itself.
But, people are very bad at estimating the MAGNITUDE of problems.
In 2024, engineering teams have to use data if they want to survive.
It’s fine to use qualitative feedback as a starting point, but you need to measure how often the things you hear about are actually happening.
Exploring that data to see what people aren’t telling you is also important, either because they’ve normalized inefficiency or can’t see the big picture from their perspective.
It’s critical to have full observability of your development systems, including Jira, GitHub, CI/CD, etc. This lets you see things bottom-up, not just top-down.
Ticket data can provide some context, but most of the information about tech debt, slow processes, poor developer experience, etc. requires looking across lower-level metrics and matching sequences of events across different systems.
Start small, start big, but start now before it's too late.