When measuring compliance results the data available is sparse and the analysis ranges from those who love it to those who hate it. Regardless of your personal, political or analytical beliefs the question remains about the efficacy of the practice itself.

There are several approaches and conversations surrounding the aspect of regulatory compliance, but the question remains, “What leads to excellence in organizations and the security they implement?”

The conclusion of this post is simple: Excellence within an organization is not achieved by measuring the compliance of an organization but by measuring the compliance of individuals and employees.

So how do we come to such a conclusion? To start we identify the main points for and against regulatory compliance:

• Pro: It raises the minimum level of security
• Cons: It creates a glass ceiling that either (1) prevents proactive organizations from implementing better security or (2) encourages reactive organizations to never excel above a certain point

Regardless of your opinion, both arguments focus around the precise level at which compliance measures an organization. Isn’t that interesting? Both the pro (for) and con (against) opinions seem to claim the same thing. This would be a good thing if it encouraged security without limiting organizations.

Already, PCI compliance (credit card security) has been compared with the No Child Left Behind or Elementary and Secondary Education Act (ESEA) within the United States. If this program is broken, let’s identify why, dispel false truths and come to a conclusion of how to learn from its lessons.

The fact remains, “nationwide: only 6 percent of U.S. students perform at the advanced-proficiency level in math, a share that lags behind kids in some 30 other countries, from the United Kingdom to Taiwan.”

Davi Ottenheimer wrote a summary of an article in The Atlantica titled, “Your Child Left Behind”. It goes on to dispel many myths about the No Child Left Behind Act. Eric Hanushek has spent “40 years calmly butchering conventional wisdom on education” including the following:

• More money does not tend to lead to better results
• Smaller class sizes do not tend to improve learning
• No U.S. state does very well compared with other rich nations
• Even relatively privileged students do not compete favorably with average students in other well-off countries. (In Illinois, the percentage of kids with a college-educated parent who are highly skilled at math is lower than the percentage of such kids among all students in Iceland, France, Estonia, and Sweden.)

“Per student, we now spend more than all but three other countries—Luxembourg, Switzerland, and Norway—on elementary and secondary education. And the list of countries that spend the most, notably, has little in common with the outcomes that Hanushek and his colleagues put into rank order. (The same holds true on the state level, where New York, one of the highest-spending states—it topped the list at $17,000 per pupil in 2008—still comes in behind 15 other states and 30 countries on Hanushek’s list.

If money, class size, and affluence of individuals do not impact classroom performance then why do we think that money, attack landscape, and intelligence of security professionals will impact the security of organizations?

Davi writes, “The exception to this lazy approach is the state of Massachusetts, which has followed a path that found success in other countries. It has directly intervened and introduced compliance.” But what kind of compliance? Was it simple testing that measured student performance against standardized tests? Nope.

The Atlantic gives us a glimpse into the answer. “What did Massachusetts do? Well, nothing that many countries (and industries) didn’t do a long time ago. For example, Massachusetts made it harder to become a teacher, requiring newcomers to pass a basic literacy test before entering the classroom. (In the first year, more than a third of the new teachers failed the test.) The state also required students to pass a test before graduating from high school–a notion so heretical that it led to protests in which students burned state superintendent David Driscoll in effigy. To help tutor the kids who failed, the state moved money around to the places where it was needed most. “We had a system of standards and held people to it–adults and students,” Driscoll says.”

So what works?

“Massachusetts, in other words, began demanding meaningful outcomes from everyone in the school building. Obvious though it may seem, it’s an idea that remains sacrilegious in many U.S. schools, despite the clumsy advances of No Child Left Behind. Instead, we still fixate on inputs—such as how much money we are pouring into the system or how small our class sizes are—and wind up with little to show for it. Since the early 1970s, we’ve doubled the amount of money we spend per pupil nationwide, but our high-schoolers’ reading and math scores have barely budged.”

Problem Set

My personal opinion is to focus on a capability and maturity model (CMM) of security and making regulatory compliance a natural side effect rather than an end goal. Sounds academic; so where’s the beef?

The practical implementation of this is shown in the recent Verizon PCI Compliance Report wherein it showed that organizations fail at tasks that: require human intervention or reoccurring activity. Many organizations that focus on compliance as an end-goal, fail to validate or maintain security throughout the year. No shocker there, but how do we overcome the human side of security? Security professionals have been talking about addressing the end-user for a long time, but these are not end-user problems they are security-professional problems.

Can we just throw more money at the problem? Reduce the scope of compliance? Train our security staff? Well, studies of the U.S. education system show these methods to be ineffective since they do not encourage well funded security professionals to do things like review audit logs on a daily basis. Even the most automated of systems are often times ignored for a variety of reasons.

Conclusion

Excellence within an organization is not achieved by measuring the compliance of an organization but by measuring the compliance of individuals and employees.

Building maturity models for information security implies an ever increasing maturity and level of security. This helps break the proverbial “glass ceiling” of compliance by having the security of an organization grow in proportion to the ever evolving attack landscape. This is so much easier said than done.

Our ability to achieve this goal hinges on our ability to encourage individual participation. Encouraging individual security-professionals to take action towards this goal.

So how useful is regulatory compliance? I advocate that compliance is good, but only in measuring the security of an organization at a point in time.  We need something much more than this to achieve real security.  We need something that will encourage validation and maintenance of security.

Final <rant>

Better security will not come from automation (DLP, audit log aggregation, etc.)  Better security will not come from more intelligent tools.  Better security will come from a higher standard within organizations to focus on maintaining security.  This leads to a discussion of cross-training security-professionals on conversational business-speak and helping them build measurable, results-driven risk models… but that is for another day.