Subjective test reporting, the new normal!

When it comes to reporting the status of our testing projects, we usually rely on fact-based analysis of the available raw data, or data-driven decision-making. This often results in beautiful dashboards with dozens of charts tailored to the needs of our stakeholders. And that is exactly where we fail today, whether you like it or not!

I start the presentation by showing a famous painting „Empire of light” by René Margritte where you see day and night in one image. I ask the audience if they like the painting and what time of day it is, what season. Based on the different answers, I illustrate that subjectivity is involved.
But testers and subjectivity is like swearing. We are supposed to be objective and metric-oriented when we report the status, the progress of testing to our stakeholders.

Then I explain that we fool ourselves into believing that our current testing reports are objective and unbiased. The objective truth doesn’t exist because there are many kinds of cognitive biases. I explain what negativity, bandwagon and inattentive blindness are.
Second, we do not reach our audience through a tsunami of statistics and data. In this area, more is not better. Irrelevant statistics are reported that obscure the real message behind the data.

By showing examples of graphs, I explain that they lack the details for useful insights (e.g. showing a trend graph of test execution without explaining the minimum execution threshold we set ourselves).
So if we cannot guarantee complete objectivity, why are we not open to subjective reporting? This kind of reporting conveys information based on personal opinions, interpretations, assumptions or beliefs.

In this part of the story, I explain how subjectivity is related to value assurance, for me another part of testing alongside quality control and quality assurance. Value assurance is about product value, customer and user experience. In this context, I explain that testers tend to fall into the so-called watermelon effect (green outside, red inside -> with classical user acceptance testing, we do not adequately cover the different aspects of value assurance. Moreover, subjective reporting follows other frameworks.
Moreover, subjective reporting follows other frameworks such as ITIL, where concepts like XLA (experience level agreements) have made their appearance. This only proves that our testing focus needs to shift from IT quality to product and value quality.

Of course, subjective reporting requires different formats to present information. In this talk, I will show different concrete formats (e.g. question quadrant, persona lights) of subjective reporting and explain in which context you can use them. Finally, I end my talk by referring to the title…. even though subjective reporting is relatively new in testing, it will become the new normal.

The slidedeck of this presentation is ready and available. If you want, I can share it and you can see how my story is visually supported by the slides.