Chaordic Mind

I am a big proponent of risk management and risk-based security.  I also work (mainly) in a very specific, yet large, segment of information security that pertains to the payment card industry (PCI).  Since I’ve been involved in this space for a long time I sometimes suffer from the curse of knowledge.  This helps when analyzing information and determining which is valuable and which is not.

Two weeks ago at Mini-Metricon, Pete Lindstrom said, “we have solved the problem of information security over 200 times, the problem is we don’t know which one is right.”  He went on to explain that different people are experts in their own domain.  The curse of knowledge hits me in that of all the information available in the payment card industry, I know which is useful to me and which should be discarded or is more applicable to another individual.  I do this without thinking and as a result my mental concept of risk management is shifted from that of others in the general public.  My network includes a strong background in the PCI industry of over 6 years and the opportunity to work closely with many smart people including Alex Hutton, Adam Shostack, Branden Williams, Walt Conway, Paul Guthrie, Andrew Jamieson, Anton Chuvakin, Lucas Zaichkowski, Martin McKeay, and many many other industry experts.  Having access to this holistic source of information provides me a wealth of information that others may simply not have.  (It also helps that my job involves QA and I end up reading hundreds of reports or case studies every year.)  Also, it’s not a point in time, but I call upon these individuals all the time to help shape and crystallize my understanding of the ever changing landscape of risk.

Two years ago when I met up with Adam Shostack at RSA and as we talked about the industry he explained to me that what we need as an industry is more data in order to form proper conclusions.  The main idea being that the more data you have on a specific topic the more easily you, and everyone else, can make a rationale decision about how to best protect it.  The problem with the lack of data is the ability to trust the limited data and conclusions you want so very much to rely on.

This is why when I finally met up with Donn Parker I asked him to explain his concept of diligence-base security vs risk-based security [PDF].  In a nutshell, Donn explained that risk-based approaches are nothing more than data alchemy as there is simply not enough public data available to make any sort of statistically significant conclusion when you assume that the entire population of data breaches or security failures (realistically unknown) is vastly larger.  Indeed it is very difficult to measure and make statistical decisions about the unknown-unknowns.

The example I like to reference is that of scanning for rogue devices (i.e. wireless access points) on a computer network.  Detecting rogue devices (unknowns) is very different than examining known devices, and logic breaks down when trying to apply traditional sampling methods to this unknown landscape.  Traditionally, sampling of a population is done when the population is uniform, or in some way known.  In general, the more uniform the population the smaller the sample size may be to determine a statistical conclusion.  The problem with rogue devices is that the population is unknown.  If you try to sample from an ever changing population the results you get at any point in time may be statically non-reflective of the total population.

Mr. Parker advocates that since we do not have a population of data breaches significant to the total number, and since the total number and type are ever changing, there is no scientific way to apply risk-based controls.  Instead he advocates a diligence-based approach towards security.  Since we cannot measure and thus appropriately apply risk-based metrics we should take the agreed upon “best practice” controls we have and be diligent about their application and maintenance.

Arguably, one could take the same cynical approach towards the traditional baseline “best practice” baselines such as BS7799, ISO 27001, ISO 27005 (for that matter the entire ISO 27000 series), or even HIPAA (HHS guidelines), or GLBA (FFIEC guidelines).  How do we know that these are sound practices upon which we should build an information security program?  With technologies changing and evolving over time there are many different ways to envision security.

So if we cannot base our foundation on best practices, and we cannot apply risk-based controls, what then is left?  This is where I propose holistic information security.  The diligence method is based on factors such as budget, management directives, staff talent and availability, and organizational policies.  Although this sounds right from a business perspective, following these methods provides a ‘good enough for the current business’ which may or may not be the best direction for the business to protect itself.  Arguably, one cannot know what the best direction is for the business due to lack of data.  See also, chicken-and-the-egg.

I’ve watched over the years as analysts, experts, and individuals claim to have the correct answer, when in fact all they have is their one piece of the pie of truth.  Instead, I advocate taking a holistic approach towards security and assimilating as much data as you can before making a decision.  Talk with as many stake holders as possible so you can elevate your level of knowledge about your industry from amateur to expert.  Only by reviewing others’ piece of the pie can you approach seeing the bigger picture.  In fact, Donn Parker advocates this in his ISSA Journal 2008 paper by proposing that practitioners of the art of information security seek out other sources of information from other organizations of comparable size, type, structure, and threat exposure.

If we are actually dealing with an unknown-unknown that we cannot measure or (honestly) see the entirety of, then we are left with only one option.  The only option left is to assimilate as much of the whole as we can.  The goal should be to “seek first to understand and then to be understood“.  This approach enables us to make more informed decisions about what is valuable information and what is fodder.

Update: I also highly recommend you watch Alex Hutton’s Security B-Sides talk on, Risk Management – Time to blow it up and start over? [slides]

Many people debate the efficacy, effectiveness, and overall utility of the Payment Card Industry Data Security Standard (PCI DSS).  Some people involved in this debate suffer from a bounded rationality, wherein their rationality is bounded by the few articles they read online, their perspective as a merchant, or their view as an information security professional.

I’d like to outline the good, the bad, and the ugly about the PCI DSS.  I do this not to condemn or defend any one party but instead to raise the level of conversation and debate beyond some of the fallacy ridden discussions we have been having up until now.

The Good

One thing that people do not realize is that the Payment Card Industry is unlike any other.  The reason we all talk about it is because it is more of a “horizontal” than it is a “vertical” industry.  Instead of talking about the banking & finance, agriculture, or even retail industry we are talking about just about every company that utilizes credit cards in some way.  It is the case that regulatory compliance has driven the dollars behind information security since 2001, and the greatest motivator in the past few years has been the PCI DSS.  If we like it or not, this standard has been the carrot (and stick) necessary to making companies care about securing your payment card data.

Until the PCI DSS was created (and the CISP/SDP before it) there was little to no standardized way for acquiring banks to measure the risk present in their merchant population. Sure, they could have reviewed varying security reports but none of these went to the heart of the risk matter by eliminating the retention of sensitive authentication data.  In fact, it was not until the PCI mandates for not storing such data that we saw real change in the payments industry.

Thought the original PCI DSS compliance deadlines were September 30, 2004, it was not until the Visa Compliance Acceleration Program (CAP) in 2007 that substantial movement occurred.  The CAP program provided the motivation necessary to make merchants validate they were not retaining sensitive authentication data.

Information security writer and reporter, George V Hulme, gets it and says, “I still contend, PCI DSS has done much to raise merchant security from the dismal state it was in — to the better state it is in today.”  Things are certainly getting better.

The PCI DSS has given acquiring banks, merchants and service providers a method of measuring their exposure to electronic and paper data compromise, the most important of which is keeping hackers from accessing the payment card data.  This is something that no other security standard specifically calls out.  (There exist many other information security programs but none define industry terms such as “sensitive authentication data” or “cardholder data”.)

So where would we be without the specific PCI mandate to eliminate sensitive authentication data and protect the remaining cardholder data?  Well, I argue we would have even more data breaches than we see today.  In the most recent (3.31.09) Visa Inc statistics, we see that within the U.S. all of the top two merchant levels have either validated compliance or are in the process of remediation.  This is a big change from even 3-4 years ago where many merchants had only begun to protect their payment card data.

Additionally, there has been significant work done to help the medium and smaller merchant levels.  Visa spearheaded the PA-DSS program, formerly the PABP, in an effort to provide secure payment applications to merchants who may not otherwise care about security or compliance.  If we follow the 80/20 rule, it is easy to imagine that 80% of the small (Level 4) merchants use the top 20% of payment applications/terminals.  If the industry can verify these payment terminals are secure they can help reduce the risk of data loss for 80% of the small merchant market.

The Bad

The PCI SSC has put in place a structured 2-year cycle for updating and improving the PCI standards.  These changes hope to move the standard in the direction of protecting data compromises based on evolving attack and threat patterns.  In the move from v1.1 to v1.2 we saw the addition of Requirement 6.6 in direct response to the rise in the number of web application attacks.  These are very positive moves, but I would not say everything is roses.

The PCI DSS references the need for a “risk based” approach but it is buried deep in Requirement 12.1.2, which states, “[security policies must] include an annual process that identifies threats and vulnerabilities, and results in a formal risk assessment.”  I like that it is here, but I think this sentence should be the overarching measure by which all requirements are addressed.

There have been many positives moves in this direction.  In April 2008, Bank of America released a document titled Beyond Minimum Compliance: PCI Risk Management.  Though not a risk management framework, this document outlined some Top 10 PCI Best Practices that included: becoming compliant, using secure payment applications, reducing the scope of data, validate compliance, and maintain a state of security throughout the year.  In 2009, the PCI Security Standards Council released a Prioritized Approach to Pursue PCI DSS Compliance.  This takes the PCI SAP and prioritizes each requirement based on, what I assume is, that which may prevent the exposure of payment card data to the greatest number of attacks.

The executive summary of the PCI Security Audit Procedures (SAP) should clearly state the need for a risk management methodology and approach.  Moreover, it should designate a call for action that companies create and maintain a capability and maturity model (CMM) to track their progress in reducing risk.

I’ve found that one of the major roadblocks to compliance adoption is the confusion about to what degree each requirement should be implemented.  These fears could be lessened if the risk management dialogue happened at the beginning of the standard.  Similar approaches have been seen in COBIT and even the ISO27001 (with the ISO27005 framework) to name a few.

The Ugly

What I find ugly about the industry is the continual flame wars that erupt around the standard itself.  I find that many, if not all of these, have to do with the improper implementation of the standard, or the lack of a deeper insight to the industry and the history of regulatory compliance.

Anyone who has worked in a cross section of regulatory compliance arenas can tell you that there is a major difference between government regulated industries and self-regulated industries.  With the exception of NAR’s REALTOR® Secure Program, I don’t know any other industry self-regulation that exists other than the PCI DSS.  (Branden Williams, Alex Hutton and David Mortman remind me of the HiTRUST program for the health care industry.)  This is a welcome anomaly in the world of government mandates.  Those who think PCI DSS is hard to comply with may not have dealt with GLBA or SOX.  With the GLB Act, if the Fed gives you a low rating (higher number) on your bank, they don’t fine you, they just shut you down.  The payments industry is trying to enable commerce in a more secure way than it was done before.

A favorite topic of Dan Geer is that of punctuated equilibrium.  The historical context of punctuated equilibrium has shown that without incremental action, a buildup of force can result in great reactionary action.  The simplest example of this is that of a volcano or earthquake.  In public policy and the payments industry this buildup has been noted several times.  We can see this historically where the government has intervened after the industry itself could not properly self-regulate.  The Great Depression gave rise to the Securities and Exchange Commission (SEC), the once poor food standards gave rise to the Food and Drug Administration (FDA), and we are now asking if the lack of personal data privacy will give way to a government managed compliance standard.  I argue that self-regulation with continual improvement will make faster strides towards data protection than will omnibus legislation.

People complain that the 2-year life cycle of the PCI DSS is too slow, but they must not be that familiar with the 4-year life cycle for other government mandates.  Also, changing a standard any faster would cause merchants’ heads to spin!  Though the payments industry is trying to self-regulate, this never seems to be enough for some people.

The “ugly” side of the industry is that which everyone becomes a pundit, spinning the story to meet their specific pain.  People say that PCI DSS is all about “risk transference” which is a flat out fallacy of belief.  The truth is that Issuing banks have always picked up the tab for fraud, and have long been able to recover that money from merchants via the various card brand issuer reimbursement programs (i.e. Visa ADCR).  The only thing the PCI DSS ever did was give acquiring banks (and all organizations handling payment card data) a measure against which to protect the payment card data.

This angry snarl of punditry is really the sad conclusion to our state of affairs.  I fully support those who wish to improve the industry but many times people feel that change should be revolutionary and not evolutionary.  They are quick to call your baby ugly and then tell you how to parent your child.  Everyone has their own input, and I just wish people would be more constructive with their comments, feedback, and advice.  We are listening!  I am listening!

As a final note, I’d like to remind the implementers of the PCI standards (merchants and service providers) to take responsibility for their own actions.  The PCI DSS should not be used as a stand alone tool.  Companies need to wrap the DSS in a comprehensive risk management program that is measured through time by a proper capability and maturity model.  Only by staying vigilant and continuously reevaluating our current security posture can we properly protect against the ever changing attack vectors that confront us.

Additional Reading

I also highly recommend PCI Shrugged: Debunking Criticisms of PCI DSS.  Please suggest others in the comments.

Update: It should be noted that I am a believer in risk management, especially quantitative risk management, but simply want to highlight some of the effects that bounded rationality has on our ability to manage risk.  I want to push us towards a more optimized view of rationality and risk management.

When we think of how to protect our most sensitive data we have one of two approaches.  Security is a tactical approach and risk management is a strategic approach.  Security implies the implementation of sound risk management practices.  While technical people like to talk about ‘vulns’ it is the risk management people who wax philosophically about long term strategy, data centric vs system centric approaches, and drink from the fountain of Utopian kool-aid.

I too have paid my dues and talked about risk management in its perfect form.  This approach involves metrics, models, threat vectors, CIA triad, and a multitude of other factors.  Risk management was married long ago to the maiden of Capability and Maturity Models for long term vitality.  Combined, these two go hand in hand to protect data from the foes.  Or so the story goes.

Now, I’m not about to become a risk management heretic, but as Mahan Khalsa says in his infamous sales books, “let’s get real!”  One thing that risk management does not (typically) take into account is that people, humans, are irrational beings.  When it comes to assessing risk, managing hazards, making decisions, managing a crisis, navigating office politics, and altering perceptions we have a roadblock called emotion.  Within emotion are all the factors that influence our decision making capabilities, such as: fear, uncertainty, doubt, misdirection, and oh so many more.

History has shown that human fear the small possibility of a quick immediate death much more than the larger possibility of a long term slow death.  The World Health Organization (WHO) reported that from 2003 – 2009 the total number of global deaths from Avian Fluwas 257.  That’s not enough to even be a statistical anomaly but we saw it on cover of just about every magazine and newspaper around the world for a few months. The WHO does not even rank influenza, of any sort, in the top 10 causes of deathby the WHO.  In fact, chronic heart disease killed 7.2 million people in 2004, and road traffic accidents killed 1.27 million people.  We worry more about contracting a rare form of the flu and dying than we do of driving to the grocery store on a Friday night.

Proper MetriCon people might say that numbers don’t lie or have emotions, but the question is, “how good are those numbers?”  I recall one year an analyst group put out a press release saying that it costs companies $200 per lost credit card.  The following year many vendor companies ran with that number and sold their product as costing only $100 per record to protect.  This could result in a 50% savings.  The problem came the following year when the analyst firm revised their numbers to say that it only cost companies $80 per lost credit card.  (Numbers have been rounded and changed to protect their creators.)

I have been sold on the need for more metrics in risk management and security, but the problem is we need to temper our reaction to data the same way we wait for Service Pack 2 before purchasing software.

We need to temper our risk management approach to one that accepts the hesitation of people to make precise and accurate decisions, especially if they are not satisfying an immediate need.  I’ve spoken with many PCI Qualified Security Assessor (QSA) companies and many agree that companies focus on satisfying compliance first and push off risk management for a later date, that sometimes never arrives.  The economics do not even need to matter as long as the immediate need is being satisfied.

People would rather spend more money now to satisfy compliance even if they could spend less over the long term to pave the road for a sound security strategy.  Why?  Well, there are many reasons but some of them include:

• High turnover
• Annual management based objectives (MBOs)
• Immediate need for “compliance”
• Lack of enterprise visibility
• Siloed departments/divisions
• Lack of information/education

It is the lack of awareness, information, and education that causes many companies to ignore the long term death and focus on the short term threat.  This can be like putting a band-aid on a bloody stump and calling it a mere “flesh wound“.

We need to accept that people are going to make irrational decisions and devise new and creative ways to re-educate them about the decisions they are making.  I think that better and better metrics are certainly a way to get there, but we are a long way from the panacea of payment security and risk metrics.