Aug 132011

Is this your QA team?

To borrow from another phrase, “defects happen.”

Even six sigma processes are not without defects.

By definition, a six sigma [manufacturing] process is one in which 99.99966% of the products manufactured are statistically expected to be free of defects.

That amounts to an average of 3.4 defects per million opportunities.

We are left to ask how much is reasonable to invest in those last 3.4 defects…

It is not the role of QA to prevent defects

Over the last couple weeks there has been a great conversation on LinkedIn about the role of QA in preventing defects from getting into production. Please take the time to check it out and join the conversation.

In the first response to the original question, Brian Will pointed out a distinction between the startup firm or the software company releasing to 1M+ end users… and then again to the medical device startup with respect to quality assurance.

Meanwhile, a colleague and friend of mine named Tom Gagne has a blog titled “Anything worth doing, is worth doing poorly” – a quote from a bank executive known for getting things done… quickly and (as far as I am aware) effectively.

So given that six sigma doesn’t eliminate defects, and given that distinctions for quality (viz defects) appear relative, the domain of quality assurance is not specifically about eliminating defects, either.

Trust production

Producing trust: Outside of a dog, a book is a man's best friend; inside of a dog, it's too hard to read - Groucho Marx

One consistency in all matters related to quality, however, is the domain of “trust production”.

Regardless what units you use to measure quality, as soon as you assess that quality is poor you get a little more skeptical about the product or service.

That’s when you take on added costs to check and double-check. That’s when you may look for alternatives, if you didn’t already.

In “A fourth rule for agile project managers,” I mentioned the unfortunate circumstance that software/IT customers sometimes suffer from “a deeply ingrained history of unmet expectations”.

In those cases, they may actually trust that your products will be awful.

So one factor generally missing from most conversations about quality is elapsed time.

It seems we typically look at QA with regard to defects for a given software product or release. We treat the domain as if those defects exist only at a point in time.

Meanwhile, defects always appear within a history of successes and failures that produce assessments of confidence in our ability to deliver – positively or negatively. Even if this is the first release, defects start the history on a bad note.

Given time as an additional dimension of quality assurance, customer service becomes a significant factor in public assessments of quality and value. Defects happen, but how we respond when they happen has a profound impact.

In addition, thinking on a longer time horizon than one release allows notions of “good enough” or “done enough” to start producing business value quickly even when some defects are known to exist or some features are not fully implemented.

Finally, a temporal consideration for quality promotes practices that can improve consistency, repeatability, reliability and competence in all aspects of software development… not just the “gauntlet” prior to production release.

What do you think about “trust production” as a form of quality assessment that takes shape over time?

  14 Responses to “Running the QA guantlet?”

  1. On Google+, Stephan Schwab wrote:

    My approach is to get QA involved and stay involved until the problem expressed in a story is truly solved. I use a combination of the 3 amigos practice together with Acceptance Test Driven Development where even the testers help programmers write good and meaningful micro tests.

    In software I understand quality as degree of excellence and therefore testers or QA need to work with everybody to maintain a high level of excellence. They need to be on board all the time.

    • Stephan, thanks for the comment. Have you thought as well about assessments of quality we build over time, and how that might adjust our thinking about QA overall?

      • Stephan replied (on Google+):

        If I understand it correctly trust is as much a product as the software itself. Having a good or a bad history of defects defines that trust product.

        I think that’s a very good way of showing that the investment in quality is worthwhile as there is a tangible product as a result.

        • That is a powerful way to think about it. Trustworthiness is something we demonstrate every time we live up to a promise (like shipping great products)… of course, it is up to the other person to decide to “trust” us.

          Seeing our work as part of our unfolding story of fulfilling promises can help us maintain perspective that it is not all about this release (especially when we make mistakes and need to live up to them).

  2. On LinkedIn Group “project management in – depth study,” Dennis Makhari wrote:

    The omission may have been unintentional. Management of quality is an ongoing process. What happens in the future always has direct relevance to the past. Recommended management practices (ISO9001) refer to activities as a function of time.

    • Thanks for the comment, Dennis. You can also say it the other way, too… what happened in the past is relevant to the future. Our historical patterns of behavior teach others what to expect of us.

  3. On LinkedIn Group Product Design & Development, Balaji Venkatesan wrote:


    Do anyone have an elegant solution to solve this issue?

    It is typically hard to identify all the issues when the design is signed off and in many settings the designer is far removed from the use.

    Though reactive a systematic root cause analysis to identify design failures may help.

    Reminds me of this fantastic talk :

    • @Balaji… that is an AWESOME talk, and I agree that it is spot-on with my concern for trust production. Thanks for sharing it with us… it’s a great gift.

  4. In LinkedIn Group project management in – depth study, Peter Humphreys wrote:

    Ken, there is another maxim to consider – that a half-baked solution is often better than no solution at all. An organisation I used to work for suffered from a terminal cultural malaise, whereby nobody wanted to make decisions, in case they made the wrong ones, and got blamed for it. As you can expect, nothing got done, and projects routinely got canned after large amounts of money were expended.

    I called this the “Burning Building Syndrome”. Imagine you are stuck inside a burning building. The best course of action is likely to take stock quickly, and then make your way efficiently but without panic to the appropriate exit. Yes, you may choose the wrong exit. The trouble is, the parable goes, the people are so afraid of suffering after making for the wrong exit, that they stand on the spot, and burn to death anyway.

    It took a manager to make a bold and impefect decision – to stop development on a “greenfields” development and tailor a similar solution from another organisation. Yes, there were howls of protest, and it was messy. But it got everyone heading for an exit. And despite the cynicism, it was successful. The implementation was “good enough”, and years were spent afterwards tweaking it, to make it better. And all of this was far better than burning loads of money for no return, by killing the project.

    • @Peter, I agree. and thanks for pointing that out. I work routinely with companies that suffer, not just without solutions, but without even the decisions that could lead to any solution at all. Some suffer from overly democratic leadership styles where seeking consensus stalls leadership, and some suffer from cultures that punish failure to the extent that people fear stepping out to lead in the first place.

      That is not what I am getting at in trust production, though. To make a decision, even an “incorrect” one, is important for any leader to make. (I also allow for “no action” to be a decision we might choose.)

      In trust production, I am looking for us to value not only our decisions, but the explicit and implicit promises we make… and not just ones we are making in our current round of promises, but those we have made in the past that ultimately lead to future assessments of trust.

      I have worked in many organizations where IT was not highly valued, in part because they failed to deliver on the big promises they made, while they never built up a string of small wins to establish a history of sincerity, reliability and competence. Each little win can be like a deposit in our “trust” bank account… but if we don’t acknowledge wins and losses as affecting trust, we could sit and wonder why our work is not valued as much as we might like it to be.

      And if we think that fulfilling on those promises can be met by QA, or if we leave it to QA alone to “catch” defects against our promises, then we may not be taking trust production seriously.

  5. And again in LinkedIn Group project management in – depth study, Dennis Makhari wrote:

    The ISO guide dwells a great deal on repeatability. It does, however extend to project type activities. This is where project management practices like PMBOK come in. It is not so much about how to make a cup of tea, but rather what activities are in place to make sure that the solution is understood by all stakeholders to address the defined problem and it is kept in check at “short” intervals to avoid tangential recourse.

    In my experience most organisations apply the bottom-up strategy formulation. This leaves each department operating as a company within a company. Hence you end up with the detrimental Lobbying activities as a way of business.

    I have found it to be more effective to deploy a top-down approach – i.e. each departmental strategy links through to corporate strategy (historical data is important when formulating strategy). This way you have an organisation wide integrated strategy that informs all projects and initiatives that are adopted at any level and it is also clear to everyone what they aim to address.

    Lobbying for support is immaterial because each project is pegged to the integrated strategy. Progress is fed upwards and any decision to stop a project would be well informed and taken at steering meetings. You also avoid “Pet Project” syndrome.

    Each project directly impacts on one or more parallel initiatives, all of which drive the business goals at corporate level. Performance on these projects directly contributes to PMS.

    The application could get a bit complex for very large organisations but the principles do not falter.

    • @Dennis, these are great observations. I think it is in the spirit of ISO and PMBOK to produce consistency and repeatability in our processes, which definitely help us to trust how everyone is going to work together to get things accomplished. This also reduces the cost (in terms of time, energy and coordinated action) of complicated tasks.

      You also make a great point about solid governance for “steering the ship”. In an organization where changing course is strategically directed more than punitive in nature, mature governance can bring together competing interests as a “team” and increase trust as well. This is an importance factor that aligns QA as a tactical strength in an overall strategy and does not see QA as the gauntlet that defends us against bad things happening.

      Another leadership dimension of trust production is contingency planning. How are our plans for backing out a process? To what extent can we exercise a contingency to move forward a little and not “throw away” a large part of our investment. Can we score a little win, while perhaps delaying the big one for a time? Much of that can be viewed positively if contingency plans are produced in advance, rather than appearing as a last resort… In that, contingencies are choices we make intelligently, though they are not usually our first “best” path… then perhaps we can win if we “win” and alternatively win if we “lose”.

  6. On LinkedIn Group Product Design & Development, Arthur Gold wrote:


    To me quality is about continually getting better about satisfying requirements. Time is a requirement that is often excluded or overlooked. Just a random thought from a Quality professsional. [sic]


    • Thanks, Art… that’s what I was thinking, too. As soon as time enters the picture, quality professionals can’t be the only ones paying attention.

Leave a Reply