Another story about false positives and what they really teach us.  The traditional story is of a boy who is given the responsibility to guard the village herd of sheep.  He is supposed to scream for help to notify the villagers when the sheep are threatened.  He decides he wants company and screams “wolf” so everyone mobilizes.  He does this repeatedly until, finally, the villagers stop coming and when a wolf really shows up, it is very bad for the boy and the sheep.   Aesop’s moral to this story:  “A liar will not be believed, even when he speaks the truth.”

If we tell the story from the perspective of detective controls, it comes out a bit differently.   An Enterprise (the village) identifies some valuable assets (the sheep) and sets up a monitor (the boy) to set off an alarm when there is an imminent threat (a wolf).  The Enterprise becomes weary of the false positives set off by the monitor and they begin ignoring the alerts.   The monitor may get disabled, but the Enterprise ends up defenseless.   When you look at it like that, it is the villagers, in other words the Enterprise, that shares the blame.   The moral becomes “those who depend on a liar for the truth are fools”.

Once the villagers determined that the boy could not be trusted, they were foolish to let him watch something as valuable as their sheep.  Carry that over to detective controls and alarms.  Once you are finding false positives on a regular basis, the risk is not that your alarms are not working, but that those who are in charge of heeding the alarms will tune them out.  Before they are tuned out, the alarms must be tuned.   Ideally, only alerts worth following up on should come through.  If you can get it to that point.

And sometimes you can’t.  It is not really all or nothing.  I would argue that it’s better to get more alerts and suffer some false positives than less and run the risk of missing something significant.  However, you then need to be able to explain how you distinguish between a true alert and a false positive.   Sometimes, when you ask operations folks how they know an alert is worth following up on, you get the enigmatic reply: “I just know”.   And that may be the case, but at that point you are left depending on the talents of the current staff.  As soon as someone who does not “just know” takes over, your control becomes much weaker, if not downright worthless.

There is another angle to this that is worth mentioning.   And this ties this post back to the one I wrote about Chicken Little (another case of a detective control issuing a false positive).    There is a thin line between urgency and panic.  A grey area between appropriate response and over-reaction.  Especially when there is the possibility of false positives.  And there is something to be gained by determining how to react to detective controls before they sound the alarm.  So when you’re working out your incident response procedures—that is: the way to respond when there really are wolves about to attack the sheep— it is worthwhile to take the time to develop alarm responses as well.   First rule of thumb for that: