I am a big proponent of risk management and risk-based security.  I also work (mainly) in a very specific, yet large, segment of information security that pertains to the payment card industry (PCI).  Since I’ve been involved in this space for a long time I sometimes suffer from the curse of knowledge.  This helps when analyzing information and determining which is valuable and which is not.

Two weeks ago at Mini-Metricon, Pete Lindstrom said, “we have solved the problem of information security over 200 times, the problem is we don’t know which one is right.”  He went on to explain that different people are experts in their own domain.  The curse of knowledge hits me in that of all the information available in the payment card industry, I know which is useful to me and which should be discarded or is more applicable to another individual.  I do this without thinking and as a result my mental concept of risk management is shifted from that of others in the general public.  My network includes a strong background in the PCI industry of over 6 years and the opportunity to work closely with many smart people including Alex Hutton, Adam Shostack, Branden Williams, Walt Conway, Paul Guthrie, Andrew Jamieson, Anton Chuvakin, Lucas Zaichkowski, Martin McKeay, and many many other industry experts.  Having access to this holistic source of information provides me a wealth of information that others may simply not have.  (It also helps that my job involves QA and I end up reading hundreds of reports or case studies every year.)  Also, it’s not a point in time, but I call upon these individuals all the time to help shape and crystallize my understanding of the ever changing landscape of risk.

Two years ago when I met up with Adam Shostack at RSA and as we talked about the industry he explained to me that what we need as an industry is more data in order to form proper conclusions.  The main idea being that the more data you have on a specific topic the more easily you, and everyone else, can make a rationale decision about how to best protect it.  The problem with the lack of data is the ability to trust the limited data and conclusions you want so very much to rely on.

This is why when I finally met up with Donn Parker I asked him to explain his concept of diligence-base security vs risk-based security [PDF].  In a nutshell, Donn explained that risk-based approaches are nothing more than data alchemy as there is simply not enough public data available to make any sort of statistically significant conclusion when you assume that the entire population of data breaches or security failures (realistically unknown) is vastly larger.  Indeed it is very difficult to measure and make statistical decisions about the unknown-unknowns.

The example I like to reference is that of scanning for rogue devices (i.e. wireless access points) on a computer network.  Detecting rogue devices (unknowns) is very different than examining known devices, and logic breaks down when trying to apply traditional sampling methods to this unknown landscape.  Traditionally, sampling of a population is done when the population is uniform, or in some way known.  In general, the more uniform the population the smaller the sample size may be to determine a statistical conclusion.  The problem with rogue devices is that the population is unknown.  If you try to sample from an ever changing population the results you get at any point in time may be statically non-reflective of the total population.

Mr. Parker advocates that since we do not have a population of data breaches significant to the total number, and since the total number and type are ever changing, there is no scientific way to apply risk-based controls.  Instead he advocates a diligence-based approach towards security.  Since we cannot measure and thus appropriately apply risk-based metrics we should take the agreed upon “best practice” controls we have and be diligent about their application and maintenance.

Arguably, one could take the same cynical approach towards the traditional baseline “best practice” baselines such as BS7799, ISO 27001, ISO 27005 (for that matter the entire ISO 27000 series), or even HIPAA (HHS guidelines), or GLBA (FFIEC guidelines).  How do we know that these are sound practices upon which we should build an information security program?  With technologies changing and evolving over time there are many different ways to envision security.

So if we cannot base our foundation on best practices, and we cannot apply risk-based controls, what then is left?  This is where I propose holistic information security.  The diligence method is based on factors such as budget, management directives, staff talent and availability, and organizational policies.  Although this sounds right from a business perspective, following these methods provides a ‘good enough for the current business’ which may or may not be the best direction for the business to protect itself.  Arguably, one cannot know what the best direction is for the business due to lack of data.  See also, chicken-and-the-egg.

I’ve watched over the years as analysts, experts, and individuals claim to have the correct answer, when in fact all they have is their one piece of the pie of truth.  Instead, I advocate taking a holistic approach towards security and assimilating as much data as you can before making a decision.  Talk with as many stake holders as possible so you can elevate your level of knowledge about your industry from amateur to expert.  Only by reviewing others’ piece of the pie can you approach seeing the bigger picture.  In fact, Donn Parker advocates this in his ISSA Journal 2008 paper by proposing that practitioners of the art of information security seek out other sources of information from other organizations of comparable size, type, structure, and threat exposure.

If we are actually dealing with an unknown-unknown that we cannot measure or (honestly) see the entirety of, then we are left with only one option.  The only option left is to assimilate as much of the whole as we can.  The goal should be to “seek first to understand and then to be understood“.  This approach enables us to make more informed decisions about what is valuable information and what is fodder.

Update: I also highly recommend you watch Alex Hutton’s Security B-Sides talk on, Risk Management – Time to blow it up and start over? [slides]