Over several years of troubleshooting issues for a SaaS company, I found severe bugs in almost every part of our software system. I was curious why I was able to find them, but the developers who nominally knew the most about the system could not. While I'm proud of what I accomplished, I don't think my ability to find these issues was the result of my superior intellect or use of black magic. Rather, I had a secret weapon that developers do not have, belief.
I believed that something was not right in our system. I believed that because of one or more complaints from serious customers and team members. Over time, I understood that customers could not tell you which part of your system was busted, or what errors to look for. Customers only know that it didn't work the way they want it to, and how that relates to the individual technical components of your system is something they are paying you to figure out.
In contrast, developers believed, very firmly, that there were no problems. Their unit tests passed. The software works when they use it. They dismiss the customer complaints as incoherent ramblings, with no clear indication of what might be wrong, and no repro steps that a serious complaint should have. In some cases, it was not even possible to identify which developer to talk to without these prerequisite facts. Asking a developer questions about how their component wasn't working seemed like an accusation of professional misconduct, a sort of "why do you hate our customers?". They were naturally defensive.
Data analysis is a good way to overcome disbelief and defensive denial. Looking at system data is more impersonal, and lets developers arrive at their own conclusions about what the data means and how that relates to what they have done. Of course, I recommend The Troubleshooting Chart for this purpose. Doing your homework making a few charts can also help you avoid false-positive reports and unnecessary interruption of developers until there is solid evidence.
Once developers believe in the problem, you will the light go on in their eyes and they will shower you with possibilities of what might be wrong, how to diagnose them, and what options might be available for fixes and workarounds. I have invariably found that developers care immensely about quality, and once they are properly mobilized they far exceeded my own abilities at finding the root cause and fixes (so much for my being smarter).
