The man who mistook his wife for an actual change in performance

There was once a man who mistook his wife for a hat.

This is his wife.

From the TV programme “The Good Wife” actually

This is a hat.

Any old hat

He thought

Why did he think that?

He had a brain injury. The injury affected the way his brain processed visual signals. Looking at his wife, he saw a hat. It wasn’t something he could do anything about, his understanding of what he saw was damaged, physically.

Oliver Sacks wrote a book about him and patients like him, with bizarre neurological impairments due to brain injuries.  There was a patient who after an accident saw totally in black and white. A man who couldn’t form any memories after the late 1960s.

The essence of these stories is that people see with their brains. If what is inside their brain is telling the, “IT’S A HAT!” then they see a hat

They form the image and create the understanding of what they are seeing out there in the world using their brain not eyes.

I think this is like how people look at graphs and see totally different things.

Remember the performance reporting archetypes?

Not mine, Mr Davis Balistacci’s

These are the 6 patterns of any three numbers arranged in every order they can be, with the name used to describe it when people have to invent a fairy story around the shape these numbers make on a page.

For example, here is the fairy story around “the rebound”…


Name: THE REBOUND
What it looks like:

Commonly Interpreted: “An inexplicable decrease in performance in month 2 has been balanced with a rebound in performance in month 3 to ensure we are entering the next quarter with all our ducks lined up.”


Of course it is all total nonsense, inventing patterns and stories out of pure noise. It’s just what people do when faced with numbers or events, we first of all spot hey HERE’S something, and then say it is ALL ABOUT THIS!

It’s all very complimicated, making sense of the world without falling prey to silly ideas. This is why we invented numbers and statistics to help us.
Note: not do it for us, but help us to do it.

And sadly this is where people use numbers and statistics to muck it all up with THIS goddam awful thing….

The bleedin’ Add Trendline….

Just like Powerpoint is a failed substitute for telling a compelling story, so Excel can be a substitute for doing actual analysis.

We’ve moaned about this before, how a trendline added to some data like so…

capture

…can fool you into thinking that there is some kind of long range and continuing decline in Some Numbers.

When looked at using actual analysis, the data shows there are two different processes, both stable but with ONE step-change happening….

capture

 

The “add trendline” is a dreadful thing. If the performance archetype graphs, or one’s like them, go through a performance team, they might leave the other side with one attached to them….

This line purports to show….a trend!  It shows the way the dots are moving! The dots are going down! In the future the next dot will be even lower!

Take the same three dots in a slightly different order, stick a trendline in ET VOILA…
The trend is reversed! The dots are going up! In the future the next dot will be even higher!

I’m exaggerating for hopefully comic effect, but i see these sorts of things all the time. Excel is designed to put “analysis” just two mouse clicks away, for the unwary to click on.

However, stick these dots between two different types of line……

The lines represent the predictable upper and lower limits of the measure. We can see that the arrangement of the three dots now shows just meandering between these lines with no change in the underlying system that produces these numbers, because no change in the upper and lower limits…ie there IS NO TREND.

(Obviously 3 dots is too few to calculate upper and lower control limits. But it is also too few to make predictions from and that doesn’t stop anybody using trendlines to do it.)

This is how the unwary mistake their wife (some line pointing upwards) for a hat (an actual change in performance)..

They are looking at noise in all the wrong places. People are not idiots they “know” that there is random noise and movements in data. That not everything is signal. But what they don’t necessarily know is where this noise is, how to distinguish it from signal and even perhaps the mental model to map signal and noise onto.

Below is the same three dots with a trendline added or between some imaginary control limits. Look at the one below with the trendline added. The red arrows show the gap between actual data points and the calculated trendline.

This is assumed to be the “noise” between actual datapoints and some assumed “trajectory” that is taken to be…god i can’t believe I am typing this….the UNDERLYING  PERFORMANCE. uRRGH.

A different way of looking at the noise in the data is seeing the 3 data points in the context of the predictable performance of the system. What is predictable is that there WILL be noise, between these two lines. Within certain criteria there will be movement of data between the lines. This is a built in feature of all of reality, there is noise. And these lines help you understand where it is and therefore how to distinguish it from signal, true change of an underlying system.

In the chart above the blue arrows show where the noise is, between these two lines essentially. The three dots are relegated from “HERE’S THE NEWS! QUICK! THE LATEST DATA! WHAT’S IT SAY!!” to instead, “here’s some extra data, when we add these to the data and knowledge of the system already calculated from that data that we already have, does this tell us that anything has changed?

Or, in short, “anything we need to know?“.

I think this is a lot more useful than the incessant jumping around being fooled by randomness, mistaking wives for hats, noise for signal, jumping at every heartbeat of data. Heartbeats are predictable, they’ll keep on coming.

Data come in patterns. Notice the patterns they make, and look for any changes to the patterns.

Advertisements
This entry was posted in data, experiment, measures, systems thinking and tagged , , , , . Bookmark the permalink.

2 Responses to The man who mistook his wife for an actual change in performance

  1. ISOwatcher says:

    Nelson rules, General Electric rules, Westgard rules…ISO 17025 is decades behind therefore UKAS still want trendlines so they have noise to inspect!

    And why are EQA schemes organized by the public sector not releasing their anonymized data into the public domain? Why is no-one asking Public Health England for disclosure so that they can be analyzed critically and published? Just how reliable might EU Official Control Laboratories be?
    https://www.whatdotheyknow.com/

    “Proficiency testing evidence
    “The data available are very limited but at the other end of the scale from single‐center descriptive reports, proficiency testing schemes have the potential to give insight into the long‐term effectiveness of accreditation. Despite the metrical perfectionism of assessors, a study funded by the UK Food Standards Agency (FSA) reported that the reproducibility of microbial counts in routine enforcement examination of foods averaged ±12% and ranged up to ±41% (Jarvis et al. 2007). Results from a national proficiency testing program of over 39,500 samples in the United States, where ISO accreditation is not common, provide an imperfect control but an informative comparison in which over 5% of food pathogens failed to be detected (Snabes et al. 2013). A similar magnitude of pathogen detection failures occurred in a proficiency testing program which serves mainly accredited UK and European laboratories.
    Disclaimer: The opinions are those of the authors.
    [Correction added on 21 December 2015, after first online publication: Nita Patel and Public Health England have been removed from the above sentence and a disclaimer has been added to this current version.] These percentages varied over time and between pathogens and appear consistent with Gaussian distributions and overlapping confidence intervals. Detailed analysis that might identify notable international methodological variations or causal association with accreditation has not been published. Both enumeration and detection figures would not surprise microbiologists of the preaccreditation era. They appear unsupportive of the carefully nonspecific claims of accreditation to transform the validity of results within or between laboratories. Therefore publication of proficiency testing data is needed to determine whether ISO accreditation has any effect on the results that laboratories report. If accreditation lacks the power to affect numerical results and service quality to users, its alleged value is negated.”
    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4767433/

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s