One of my early career lessons, while I was transitioning from being “technical-focused” to being more “business-focused”, occurred when, as PM, I had to deal with a critical escalation just ahead of a platinum customer’s web-based service launch.
During performance testing (which always occurs at the last minute due to taking place on a near-live system) my project team uncovered an intermittent issue under load that caused a momentary outage of the system. We were concerned about the unpredictability of the problem, and wanted to fix it before going live. As the PM, I had to go cap-in-hand and say “Dear Mr. Customer, I advise you not to go live this week”.
The lesson learned was in what happened next.
Of course, all h$@$#l broke loose on both sides of our collaborative project teams that were rushing to complete everything before go live. Escalations ensued on both sides – customer and vendor. Which is when the executive team on the customer side came back to me with the question: “So what would happen if we went live anyways?”
This was a new thing for me at the time. I went on to write an impact analysis executive brief of the problem.
I had and still have a great respect for that customer’s executive team. I won’t even hint at who they were, but will say that they were a big player in their space with a very talented, experienced, level-headed executive team. From their perspective they needed to weigh out the business impacts of stopping/delaying the launch with the technical impacts of going live with a potentially serious technical issue. Their approach was to tackle the issue head-on, with a calm attitude and the purposefulness of intention of moving things forward. What was important was resolving the escalation at hand – not finger pointing, or bullying the vendor for a resolution. With all the facts at hand, they made decisions and clearly communicated expectations.
For someone who had spent a lot of time in the trenches dealing with the issues of bleeding-edge new software products, this was rarified air. To keep the military analogy going, it was what to me a command tent should be like. An oasis in the storm, where an objective view can be taken and then appropriate measures planned and communicated out. A style I believe all executives should learn to practice.
It taught me to be transparent and direct when discussing risks and issues with executives. (It’s also important to be tactful, timely and aware of priorities so as not to be creating tempests in teapots.) Management can deal with problems when they know about them. Problems arise when they don’t know about issues and the potential impacts. Ask any Surgeon or ER doctor.
As to the problem – the service was crashing when some particular set of circumstances came together under load. The system recovered from outages gracefully, so the service went down and then came up again, only customers had to re-login, which doesn’t sound all that drastic in current time where Apps that crash on a regular basis.
In the end, the project paused for two weeks which gave all teams a chance to be ready for the launch, a fix was found, implemented and tested for the critical problem, and the service support teams were assured of the quality of what was being handed over to them to support.
