Chaordic Mind

When measuring compliance results the data available is sparse and the analysis ranges from those who love it to those who hate it. Regardless of your personal, political or analytical beliefs the question remains about the efficacy of the practice itself.

There are several approaches and conversations surrounding the aspect of regulatory compliance, but the question remains, “What leads to excellence in organizations and the security they implement?”

The conclusion of this post is simple: Excellence within an organization is not achieved by measuring the compliance of an organization but by measuring the compliance of individuals and employees.

So how do we come to such a conclusion? To start we identify the main points for and against regulatory compliance:

• Pro: It raises the minimum level of security
• Cons: It creates a glass ceiling that either (1) prevents proactive organizations from implementing better security or (2) encourages reactive organizations to never excel above a certain point

Regardless of your opinion, both arguments focus around the precise level at which compliance measures an organization. Isn’t that interesting? Both the pro (for) and con (against) opinions seem to claim the same thing. This would be a good thing if it encouraged security without limiting organizations.

Already, PCI compliance (credit card security) has been compared with the No Child Left Behind or Elementary and Secondary Education Act (ESEA) within the United States. If this program is broken, let’s identify why, dispel false truths and come to a conclusion of how to learn from its lessons.

The fact remains, “nationwide: only 6 percent of U.S. students perform at the advanced-proficiency level in math, a share that lags behind kids in some 30 other countries, from the United Kingdom to Taiwan.”

Davi Ottenheimer wrote a summary of an article in The Atlantica titled, “Your Child Left Behind”. It goes on to dispel many myths about the No Child Left Behind Act. Eric Hanushek has spent “40 years calmly butchering conventional wisdom on education” including the following:

• More money does not tend to lead to better results
• Smaller class sizes do not tend to improve learning
• No U.S. state does very well compared with other rich nations
• Even relatively privileged students do not compete favorably with average students in other well-off countries. (In Illinois, the percentage of kids with a college-educated parent who are highly skilled at math is lower than the percentage of such kids among all students in Iceland, France, Estonia, and Sweden.)

“Per student, we now spend more than all but three other countries—Luxembourg, Switzerland, and Norway—on elementary and secondary education. And the list of countries that spend the most, notably, has little in common with the outcomes that Hanushek and his colleagues put into rank order. (The same holds true on the state level, where New York, one of the highest-spending states—it topped the list at $17,000 per pupil in 2008—still comes in behind 15 other states and 30 countries on Hanushek’s list.

If money, class size, and affluence of individuals do not impact classroom performance then why do we think that money, attack landscape, and intelligence of security professionals will impact the security of organizations?

Davi writes, “The exception to this lazy approach is the state of Massachusetts, which has followed a path that found success in other countries. It has directly intervened and introduced compliance.” But what kind of compliance? Was it simple testing that measured student performance against standardized tests? Nope.

The Atlantic gives us a glimpse into the answer. “What did Massachusetts do? Well, nothing that many countries (and industries) didn’t do a long time ago. For example, Massachusetts made it harder to become a teacher, requiring newcomers to pass a basic literacy test before entering the classroom. (In the first year, more than a third of the new teachers failed the test.) The state also required students to pass a test before graduating from high school–a notion so heretical that it led to protests in which students burned state superintendent David Driscoll in effigy. To help tutor the kids who failed, the state moved money around to the places where it was needed most. “We had a system of standards and held people to it–adults and students,” Driscoll says.”

So what works?

“Massachusetts, in other words, began demanding meaningful outcomes from everyone in the school building. Obvious though it may seem, it’s an idea that remains sacrilegious in many U.S. schools, despite the clumsy advances of No Child Left Behind. Instead, we still fixate on inputs—such as how much money we are pouring into the system or how small our class sizes are—and wind up with little to show for it. Since the early 1970s, we’ve doubled the amount of money we spend per pupil nationwide, but our high-schoolers’ reading and math scores have barely budged.”

Problem Set

My personal opinion is to focus on a capability and maturity model (CMM) of security and making regulatory compliance a natural side effect rather than an end goal. Sounds academic; so where’s the beef?

The practical implementation of this is shown in the recent Verizon PCI Compliance Report wherein it showed that organizations fail at tasks that: require human intervention or reoccurring activity. Many organizations that focus on compliance as an end-goal, fail to validate or maintain security throughout the year. No shocker there, but how do we overcome the human side of security? Security professionals have been talking about addressing the end-user for a long time, but these are not end-user problems they are security-professional problems.

Can we just throw more money at the problem? Reduce the scope of compliance? Train our security staff? Well, studies of the U.S. education system show these methods to be ineffective since they do not encourage well funded security professionals to do things like review audit logs on a daily basis. Even the most automated of systems are often times ignored for a variety of reasons.

Conclusion

Excellence within an organization is not achieved by measuring the compliance of an organization but by measuring the compliance of individuals and employees.

Building maturity models for information security implies an ever increasing maturity and level of security. This helps break the proverbial “glass ceiling” of compliance by having the security of an organization grow in proportion to the ever evolving attack landscape. This is so much easier said than done.

Our ability to achieve this goal hinges on our ability to encourage individual participation. Encouraging individual security-professionals to take action towards this goal.

So how useful is regulatory compliance? I advocate that compliance is good, but only in measuring the security of an organization at a point in time.  We need something much more than this to achieve real security.  We need something that will encourage validation and maintenance of security.

Final <rant>

Better security will not come from automation (DLP, audit log aggregation, etc.)  Better security will not come from more intelligent tools.  Better security will come from a higher standard within organizations to focus on maintaining security.  This leads to a discussion of cross-training security-professionals on conversational business-speak and helping them build measurable, results-driven risk models… but that is for another day.

Please read my guest blog post over at IT Knowledge Exchange.  It covers the topic of: Capability and Maturity Model Creation in Information Security.

The post references the following capability and maturity model (CMM) resources:

• Information Security Management Maturity Model (ISM3, or ISM-cubed)
• Systems Security Engineering Capability Maturity Model (SSE-CMM), which is ISO standard 21827
• Computer Security Handbook 4th Edition (general theory)

Also, Katie Moussouris reminded me of the Microsoft SDL Optimization Model.

Update: It should be noted that I am a believer in risk management, especially quantitative risk management, but simply want to highlight some of the effects that bounded rationality has on our ability to manage risk.  I want to push us towards a more optimized view of rationality and risk management.

When we think of how to protect our most sensitive data we have one of two approaches.  Security is a tactical approach and risk management is a strategic approach.  Security implies the implementation of sound risk management practices.  While technical people like to talk about ‘vulns’ it is the risk management people who wax philosophically about long term strategy, data centric vs system centric approaches, and drink from the fountain of Utopian kool-aid.

I too have paid my dues and talked about risk management in its perfect form.  This approach involves metrics, models, threat vectors, CIA triad, and a multitude of other factors.  Risk management was married long ago to the maiden of Capability and Maturity Models for long term vitality.  Combined, these two go hand in hand to protect data from the foes.  Or so the story goes.

Now, I’m not about to become a risk management heretic, but as Mahan Khalsa says in his infamous sales books, “let’s get real!”  One thing that risk management does not (typically) take into account is that people, humans, are irrational beings.  When it comes to assessing risk, managing hazards, making decisions, managing a crisis, navigating office politics, and altering perceptions we have a roadblock called emotion.  Within emotion are all the factors that influence our decision making capabilities, such as: fear, uncertainty, doubt, misdirection, and oh so many more.

History has shown that human fear the small possibility of a quick immediate death much more than the larger possibility of a long term slow death.  The World Health Organization (WHO) reported that from 2003 – 2009 the total number of global deaths from Avian Fluwas 257.  That’s not enough to even be a statistical anomaly but we saw it on cover of just about every magazine and newspaper around the world for a few months. The WHO does not even rank influenza, of any sort, in the top 10 causes of deathby the WHO.  In fact, chronic heart disease killed 7.2 million people in 2004, and road traffic accidents killed 1.27 million people.  We worry more about contracting a rare form of the flu and dying than we do of driving to the grocery store on a Friday night.

Proper MetriCon people might say that numbers don’t lie or have emotions, but the question is, “how good are those numbers?”  I recall one year an analyst group put out a press release saying that it costs companies $200 per lost credit card.  The following year many vendor companies ran with that number and sold their product as costing only $100 per record to protect.  This could result in a 50% savings.  The problem came the following year when the analyst firm revised their numbers to say that it only cost companies $80 per lost credit card.  (Numbers have been rounded and changed to protect their creators.)

I have been sold on the need for more metrics in risk management and security, but the problem is we need to temper our reaction to data the same way we wait for Service Pack 2 before purchasing software.

We need to temper our risk management approach to one that accepts the hesitation of people to make precise and accurate decisions, especially if they are not satisfying an immediate need.  I’ve spoken with many PCI Qualified Security Assessor (QSA) companies and many agree that companies focus on satisfying compliance first and push off risk management for a later date, that sometimes never arrives.  The economics do not even need to matter as long as the immediate need is being satisfied.

People would rather spend more money now to satisfy compliance even if they could spend less over the long term to pave the road for a sound security strategy.  Why?  Well, there are many reasons but some of them include:

• High turnover
• Annual management based objectives (MBOs)
• Immediate need for “compliance”
• Lack of enterprise visibility
• Siloed departments/divisions
• Lack of information/education

It is the lack of awareness, information, and education that causes many companies to ignore the long term death and focus on the short term threat.  This can be like putting a band-aid on a bloody stump and calling it a mere “flesh wound“.

We need to accept that people are going to make irrational decisions and devise new and creative ways to re-educate them about the decisions they are making.  I think that better and better metrics are certainly a way to get there, but we are a long way from the panacea of payment security and risk metrics.