When we surveyed practitioners, 71% of respondents admitted that their networks are exposed to external threats due to misconfiguration issues in their security device infrastructure. Verizon reports that 79% of organizations fail to maintain PCI compliance from their prior year’s assessment. More than 50 percent told us they had no idea how many of their organizations’ internal hosts were exposed to the Internet.

We know that even in this era of constrained budgets, enterprises are spending more on network security—and yet 75% of network and security pros agree that the advantage is still on the side of the attacker. Verizon reposts that security “erosion” over the course of the year between PCI audits is the norm with most enterprises, despite the fact that we know there’s a correlation between slippage and data breaches.

Maybe it’s time to re-evaluate our priorities. As our CTO Dr. Mike points out, there’s a general consensus to focus on the core controls. If you’re already covering 90% of the basics, security pros agree it’s more wise to push for 100% versus expand the number of controls.

But if you’re focused on the core controls, how do you know what percentage level you’re at, and where the areas of exposure are? That’s where security metrics come in.

In this case, we’re referring to actionable security metrics — those that provide proactive security intelligence, a direct incentive to act. Many metrics available to security pros: number of patches; number of vulnerabilities; and the number of firewall and router config changes, are without context, or simply measure worker hours. They don’t characterize risk in a meaningful way, nor do they point towards a specific resolution.

Hitting The Books

In his seminal tome Security Metrics: Replacing Fear, Uncertainty and Doubt, Andrew Jaquith describes the value of security metrics by referencing other business disciplines. For example, freight companies know their freight cost per mile and loading factors , and those of their competitors. Management can set objectives and measure themselves against comparable companies. Choosing to be above, on, or below an industry average is a question of strategy as well as operational efficiency. For example, a freight company may be willing to have a lower load factor than its peers if that’s the tradeoff required to offer faster delivery times (for which it presumably charges a premium).

Similarly, warehousing firms measure and compare their cost/square foot and inventory turns, and e-commerce companies measure site conversion rates. Financial metrics have established for many years. Companies can therefore compare relevant metrics to those of their peers in order to evaluate internal performance.

Could such a use of metrics apply to security? Yes, but only if consistently generated within the context of a security framework.

Building Blocks

The three pillars of security, as we see it here, are visualize, comply and protect. Logically then, if we build a framework on those pillars we’ll be able to generate meaningful security metrics.

Visualize: There is wisdom in Requirement 1 of the PCI DSS, in the section entitled “Build and Maintain a Secure Network”: the requirement is to create a network diagram, and keep it current. Why? You can’t secure what you can’t see. And yet, according to Verizon Requirement 1 has the second-highest erosion factor out of the nine requirements not specific to planning and checking. When security pros can visualize the network topology—including groups that clearly identify zones (such as DMZ) and untrusted sources—they become much more effective in creating effective segmentation strategies and policies, and maintaining compliance.

Comply: Compliance refers to PCI, FINRA, FFIEC, SOX and other regulatory frameworks, of course, but also internal policies, and best practices from sources such as SANS’ 20 Critical Security Controls, Version 3.0. However, complying with regulatory and internal policies in most cases is open loop; we perform security measures in an effort to comply, but other than regulatory audits we’re mostly in the dark as to how effective our security controls remain over time. We need move from open loop security to closed loop,  with feedback controls that allow us to make continuous adjustments to thwart erosion.

Protect: The fundamental security question is whether the network is protected. How can we know what’s working, and where additional focus is required? By developing a security framework that provides security metrics — feedback controls, from which effective remediation for erosion can be devised. Security metrics enable enterprise to answer questions such as:

1. What’s my overall risk; how does it compare to yesterday, last week, last month and last year?
2. How easily can attackers get in?
3. How big is my attack surface?
4. How much of my infrastructure is undocumented?
5. Are investments and actions paying off?
6. Where do we need to improve?
7. Are we ready for our next audit?

Note that the questions above relate to actual network security, unlike, say, how many hosts were patched in the last month (time check) or how many vulnerabilities are being scanned (no context).

Comparing Models

Are these good security metrics? Let’s look at Andrew Jacquith’s definition :

1. Consistently – measured, without subjective criteria;
2. Cheap – to gather, preferably in an automated way;
3. Expressed – as a cardinal number or percentage, not high, medium and low;
4. Expressed – using at least one unit of measure, such as “number of hosts directly exposed”; and
5. Contextually – specific—relevant enough so someone can read it and take action.

The security metrics provided in RedSeal 5 satisfy all of Jacquith’s criteria for good metrics, empowering our customers to continuously monitor their network through a closed loop process and therefore address problem areas—and in doing so protect their organization’s network.

Behold, security metrics that actually work.

The fact that Symantec suffered a breach due to lax protection in someone else’s network is a significant wake-up call. It is not enough to ensure you follow best practices; in an interconnected world, you have to worry about the security of other organizations.

Your business partners and strategic customers may be friendly, but they are not going to expose specifics to you about how well they protect themselves (unless maybe you force them to, as with hosting companies or cloud providers).

This issue – needing to understand the risk of a network you cannot see – has led to standards like PCI, FISMA, and DISA STIGs, which establish accepted, measurable baselines of “basic hygiene”. As we steadily lose control of our own critical assets, and as attackers increasingly automate their attacks, we will need more baselines like this so that one organization can show another that it is handling things properly.

Now, when I wrote the above comments in a discussion, I got some push-back about the standards I chose to highlight – not everyone agrees that PCI or FISMA are good role models. Some correspondents challenged me to defend whether these regulations really work, or provide any security. What follows is my response – a take on what PCI DSS is, what it’s not, and why I think it’s good at what it aims to do:

The key thought here is “cui bono” – who benefits?

PCI is not a magic recipe for perfect security – some would thus call it “inadequate”. But that’s not what it’s for. From the point of view of credit card companies, their money sits inside other peoples’ networks (in the form of those credit card numbers). They have spent years thinking about the problem Symantec faced – what happens when you have to put your assets inside a network you can’t control? You need assurance; PCI exists to give the credit card companies some assurance that the numbers are being looked after.

However, what makes it really interesting is that they cannot simply require that all Mom & Pop corner stores will run Fort Knox levels of security – the issuers lose money if they make it too hard to work with credit cards! So they are really in a bind. Enforce too stringent a set of standards, and nobody will be able to meet them, and so nobody will be able to use credit cards. Set the bar too low, and keep on losing larger amounts of money to theft. PCI is an impressive balancing act between these pressures.

PCI is also moving – the rules change in each revision, in a process of gradually raising the bar. FISMA is changing too. It’s true that it started as a paperwork exercise, but recent changes are shaking that up. I would say the thinking behind FISMA represents some of the best work going on today on this problem – how can you motivate people to improve, without imposing too much overhead, and while staying flexible (since the threat landscape is moving).

The attackers are increasingly relying on automation, and so the big theme in FISMA reform is “continuous monitoring” – automate your defenses, as far as possible. That is, FISMA is no longer a simple checklist – it requires metrics, and it requires them to be generated so frequently that it’s not even possible to use wasteful, inaccurate human effort. Automation is the only way to meet FISMA, and defensive automation is the right next stage of evolution.

As for how the baselines should be enforced, we can already see the answer. The government focuses on regulations of defense networks (DISA STIGs), on government infrastructure (FISMA), and on critical infrastructure such as power generation (NERC/FERC) – this is an appropriate role for government. PCI, on the other hand, is not a law – it’s a commercial agreement between people whose money (in the form of credit card numbers) is held inside the networks of other people (merchants, clearing houses, etc). This is the right model.

Anyone who faces risk due to assets under someone else’s control needs to establish a yardstick that the outside entity can use to show they have taken due care. Such yardsticks cannot be impossibly hard to meet – they need to be practical and achievable (even if this means they cannot guarantee perfect security – who can?).

The yardsticks need to be measurable – opinion calls should be avoided. They need to maintain some privacy for the organization being studied – nobody will agree to expose all their infrastructure to an outside party, so the yardstick needs to summarize well. Finally, the yardstick must actually measure security effectiveness, not just busy-ness.

In today’s world, PCI is a good example of all of these properties. It’s no panacea – it doesn’t guarantee security. All it does is offer some assurance to the person taking on third party risk that the third party is at least basically competent – “you must be at least this high to ride this ride”.

Other organizations can learn lessons from PCI, and use comparable guidelines when they take on third party risk. The movement to cloud services is a great example – the financial benefits of moving to the cloud are obvious, but the risk side remains too murky.

Buyers of cloud services face the same problem Symantec faced, and that credit card companies faced. They need to demand quantified, summarized metrics of security effectiveness and security posture from their cloud service providers. Government cannot solve this (beyond prosecuting fraud), but commercial contracts can, and PCI stands as a clear example.

Each day I would look down Broadway towards the iconic cacophony of light and color making up the world famous electronic gathering place and marvel at the millions who have visited here together with the age old promise: “If you stand here long enough, you will meet everyone you have ever known.”

The sight itself is overwhelming. The colors and flashing lights create a constantly changing visual mosaic. If you pause and think for just a moment you get a glimpse of just how complex the tapestry is.

It’s not even a reasonable fraction of the complexity of your network.

Take a short imaginary journey with me.

You have been asked to figure out a network. Perhaps the network of a company you’ve just bought or one belonging to a customer who just lost their network engineer.

You have the passwords for the devices, but nothing else. There’s no one left who knows anything at all about the network.

What would you do?

Pause and think about that a second before you go on, because I’m going to tell you what I do when I’m faced with this situation… and just how complex the answer really is.

So what would I do?

It starts with my philosophy that the configurations of the network devices mean absolutely everything. It doesn’t matter what was intended. What matters is what is. What’s the reality, versus what are the policies?

So, I look at the configs.

Typically, I’ll have a spreadsheet and OmniGraffle open on my screen and then I get started. I use basic network analysis like ping and traceroute to find network devices that act as routers (these include firewalls and load balancers) and I log into them and read their config files.

I make note of their interfaces on both the spreadsheet and the diagram I’m building. I also make note of any firewall rules or ACLs that get applied.

Soon, I’ve got a list of subnets, IP addresses, and allowed and denied paths.

I keep building, but I’d better hope that it’s a small network. By the time I hit 3 or 4 devices, the possible permutations are approaching unmanageable. It doesn’t take long before I’m wishing for a database to hold all of this and to figure out what I’m finding.

Thankfully, RedSeal makes just such a product, and it works exactly like I want it to: it reads configurations, giving me the as-built network.

It analyzes all of the possible paths, taking into account NAT and rules and ACLs, ports and protocols, and delivers me a model that I can query both visually and textually.

I can even export the data to the universal network management tool: Microsoft Excel.

Most importantly, though, RedSeal tells me what is and leaves it to me to make judgments about whether or not it maps to what I want.

Sure, it gives me hints, but it never insists it knows better than me. We’ve all been frustrated by systems that do that.

Once I have an accurate and comprehensive model for my network, I can ask it questions, look at possibilities, and generally determine my security architecture from the reality of my devices.

I like that.

What do you want and need in order to really understand your networks?

Yet, as much as we expected to hear from attendees at the GFirst conference in September that there was still a good deal of confusion as to the specifics and timing of their continuous monitoring plans, the results that came back – less than half of all agencies said they feel confident they’ll be able to make the cutoff date – opened some eyes all around.

As our government advisor Major General John Casciano (USAF-Ret.) has been saying for years, we knew that various agencies’ and practitioners’ perceptions of what adopting continuous monitoring actually entails would likely be all over the map.

But, for none of the four technologies we included in the survey to emerge as a leading model, or even two, really made us shake our heads in wonder. (For the record 51 percent of respondents labeled IDS/IPS as part of their projects, followed by SIEM at 49 percent, network device audit at 43 percent and vulnerability assessment tools at 35 percent – respondents could obviously choose more than one option).

Clearly, government IT security pros are not only discouraged about their ability to get continuous monitoring in place ahead of the White House deadline, but they’re not even sure what they need to do. They still don’t feel comfortable with the specifics of what the OMB regulators are expecting of them.

Considering that the Obama administration cited improvement of cyber-security as a primary objective and former Fed CIO Vivek Kundra, along with cyber-czar Howard Schmidt, listed continuous monitoring as one of the most important programs in making progress, it would seem, given the replies, that those objectives are truly falling short.

To be fair, the whole nature of continuous monitoring is pretty complex, with NIST calling for agencies to both monitor for suspicious traffic on their networks, as well as deploy far more proactive and comprehensive means of assessing exposed vulnerabilities and other underlying points of risk.

However, I doubt anyone would have guessed that, if our results are representative of the larger community and not simply the relatively small sample of 200-plus experts that we reached, the atmosphere around continuous monitoring seems so completely out of whack.

Without getting too self-serving, we at RedSeal feel that the brand of intelligence we can provide regarding real-world network access and vulnerability exposure can help government agencies make continuous monitoring a reality, and a very valuable effort.

When In-Q-tel invested in us earlier this year to help facilitate continuous monitoring in the U.S. intelligence community we felt that was an endorsement that spoke for itself.

For all of us dependent on these hard-working government decision makers and practitioners to help keep our nation and its critical infrastructure safe, here’s hoping that 2012 proves a period of rapid acceleration of both understanding and deployment of continuous monitoring capabilities.

We all know, given both recent and historic events, there’s a heck of a lot of important matters at stake.

That new era begins today with the relaunch of RedSeal, as we introduced the most significant update to our solution ever – RedSeal 5 – and rebrand from “RedSeal Systems” to “RedSeal Networks.”

If you’re a regular visitor to the blog you likely notice that we’ve given our public image, and website, a total facelift – refocusing from the concept of security posture management into what we feel is the next big enterprise IT risk modelling – proactive security intelligence.

Our technical and marketing experts will tell you everything about RedSeal 5  in upcoming posts, and you can read about the introduction here, but long story short, this is precisely the type of release you leverage to turn a page and redefine what you do.

While RedSeal 4.2 and its predecessors delivered incredible value around assessment of network defenses and available access, as well as vulnerability exposure, RedSeal 5 is a much evolved species of the same origin.

By generating key metrics that trend actionable security performance data, RedSeal 5 is ground-breaking; its increased capabilities for predicting the impact that proposed changes will have on security infrastructure are unprecedented.

Predictive risk analysis capabilities on one end and quantitative analysis of historic performance on the other – both previously nonexistent; we know that’s a big deal.

I’ll keep my promise not to delve into product in this space, so the bigger question – why proactive security intelligence – remains.

Here’s what we know. Enterprises aren’t starved for security and risk information anymore, they’re drowning under a tidal wave of data produced by their network defenses and vulnerability assessment tools.

The “visibility” into such data provided by RedSeal was the most popular benefit cited by customers when we recently polled them. Just as today’s astronomers use intelligent tools that comb through the mountains of data generated by their infrared telescopes to discover new stars, the professionals using our product leverage automation to find those security infrastructure issues
invisible to the naked eye.

In addition to our friends at In-Q-Tel advancing our products into the U.S. government’s intelligence community, we’re seeing the concept arise everywhere, from our customer Cisco’s “open security intelligence” data analysis, to White House OMB directives on FISMA reporting guidelines.

The cut on infrastructure defenses, and, in particular, vulnerability management, has long been that the processes aren’t “proactive” enough.

That’s where you get PSI from. Preventative analysis of network security risks, it’s not particularly hard to understand.

We made the messaging change because RedSeal 5 does so much more than isolate gaps and exposed vulnerabilities. Those are still core values, but with metrics and change, RedSeal does far more.

How do we know there’s a need? Once again, our customers told us. In a survey we conducted this summer, clients unfailingly touted RedSeal’s ability to help translate their security information into something useful – deeper intelligence about security infrastructure performance and changes.

Security posture isn’t dead, it’s just a smaller piece of this larger puzzle.

We’re riding these features into a new world where we can help solve enterprise security management challenges of a higher order, the big ones that CSOs have only dreamt of.

Stopping advanced attacks, maintaining continuous compliance, trending vital security performance indicators and communicating such powerful information back to sales.

We think it’s an idea whose time is finally here, specifically realized via development of a powerful new analytical engine sitting at the middle of security infrastructure data and built to last.

Proactive security intelligence. Does that make sense or what? You tell me.

And welcome along for the ride.

It’s also clear that there is a fairly pervasive desire among the organizations I’ve visited to find ways to avoid some of the tasks that really have to be done by people, or at least to find ways of getting products to seem to do them. Key difference there.

In my previous blog post Insight Isn’t Free, I wrote about the challenge organizations often have in distinguishing between which tasks are better to be tackled by people or products.

Recently, working with a security specialist in an organization with an outsourcing relationship, inter-organizational communication showed up as a big issue. As I wrote in Network World years ago, relationships between outsourcers and their customers can be challenging. 

But now, with PCI and other compliance requirements involved, the communication and information exchange between the two organizations is even more critical.

When an organization is subject to external audit, it’s incumbent upon them to make sure that they know the status of their infrastructure. When a services organization is responsible for the configuration of that infrastructure, the flow of information is even more important. As a result, you must be certain that your communication and data exchange in such relationships is clean, clear, current, and utterly complete.

For example, when using RedSeal to analyze how your infrastructure is built, it’s essential to have configurations for the devices that impact the access paths running through your network. Prior to the need for compliance and audit management, it was common for managed service providers to maintain device configurations in a way not visible to their customers.

In some cases, they even maintained all configurations for all customers in a single configuration database, making distinguishing between customers difficult.

Nonetheless, it’s critical to be able to access your own information to make sure that the real-world iteration matches both the “as-designed” network and the policies you have in place.

This is where RedSeal comes in as such a critically important solution: it helps you see how the devices’ configurations impact your overall security standing, provides tools for analysis, and gives you the keys to the adjustments you need to ensure that your organization remains compliant.

But it all starts with communication — something a people still do better than products.

Today RedSeal published what we hope will be the first of many research reports that help bring light to current issues amongpractitioners in the enterprise IT security space.

Undertaken in partnership with Dimensional Research, our new report is titled: “Hackers Versus Enterprise Security: A Survey of IT Security Professionals.”

Based on just under 2,000 interviews with network and security pros, the major finding is that these workers feel that today’s attackers retain the upper hand in terms of automation. Over 50 percent of those surveyed were responsible for networks containing over 100 or more such devices, suggesting that the sheer size and scale of today’s security infrastructure is preventing organizations from adequately maintaining defense.

The data was culled on the show floor over the course of this summer at the Black Hat and Cisco Live conferences, and the findings are pretty thought provoking if you’re involved in managing IT security, or even just an average Joe getting your hacking geek on.

Among the specific findings:

• More than 75 percent of network management and security professionals believe that automated tools give hackers the upper hand in evading the defensive systems utilized by most enterprises to protect their critical assets and data.
• Over 71 percent of respondents admitted that their networks are exposed to external threats due to misconfiguration issues present in their security device infrastructure.
• More than 50 percent had no idea how many of their organizations’ internal hosts were actually exposed to the Internet.
• Roughly 52 percent conceded that their vulnerability management initiatives don’t allow them to prioritize remediation based on the likelihood of real-world attacks.
• While many security regulations and industry leaders have recommended for years that enterprises adopt a more metrics-driven approach toward measuring the effectiveness of security infrastructure, only 47 percent of respondents said that their employers do so today.

Clearly the results point to the true underlying value of RedSeal solutions in isolating gaps in network security infrastructure, providing visibility into vulnerability exposures and generating actionable metrics regarding the overall performance of security systems and strategies — but hopefully it’s clear that this research was straightforward and objective, informed completely by practitioners’ responses.

“Consistent application of network security controls across even medium sized networks has transcended human ability,” said Dr. Mike Lloyd, Chief Technology Officer at RedSeal. “For many years there’s been the notion of an arms race between IT security professionals and attackers; what this survey proves is that the good guys understand they’re facing a truly daunting task to
keep up.”

As CSO reporter Bill Brenner noted in his write-up, it’s hard to maintain security standing when the bad guys “have better toys.”

We think that RedSeal is one of the toys that every CSO should have in their collection, because as they say, you have to fight fire with fire. Or firepower?

Click here to read the entire report.

It’s interesting to observe how the continuous, inexorable improvement of the tools that we use today have provided security professionals with greater insight and control over so many critical aspects of our enterprise networks and systems.

However, what’s even more telling is that as those tools improve, it can become even more tempting to expect them to carry out important tasks that only we, as humans, can address.

What do I mean?:

• Only people can think independently
• Communication ties together functions within an organization
• There is no shortcut to effective communication
• Tools often bring organizational issues to the surface when dealing with complex environments
• It’s tempting to assign blame elsewhere, instead of creating paths to address problematic issues
• It’s impossible for tools alone to address many of the listed enterprise security issues above

In the work that I do with clients, the organizational divisions commonly found between network engineering and security, IT and outsourcers, and network and systems administration staff can all create the potential for conflict and disagreement.

The only solution to these divisions, as I see it, is faciliatating improved communication among all parties within the context of a clear and common sets of objectives.

Think this concept through in consideration of the unique characerisitcs of your own particular environment, and see how you can improve the flow of organizational dynamics.

One area where one sees this frequently within the context of RedSeal is in the distinction between routing and security policy – as viewed by the different perspectives of network engineering and security teams.

This common disparity shows up specifically when network engineering recognizes a route that is going to cause packets to flow a certain way under virtually all situations.

Sometimes this leads to conflict when the security team wants to apply Access Control Lists or firewall rules to paths that “should never happen”, according to routing.

Of course, routing is really effective at finding its way around failures, and so there are many times when routing determines an unexpected route to be the best option.

When that happens and there are no ACLs or firewall rules in place for the unexpected path, your security is significantly compromised.

Helping both groups to understand these issues is challenging, but it’s an issue that all of these teams must address.

Products like RedSeal can show you the issues, but understanding the implications, explaining them to the involved stakeholders, and addressing them on a practical level are unavoidable issues that will inevitably fall to you and your colleagues.

Take it on and make a difference!

Continuous monitoring, or the process of proactively scoping risks posed to critical systems and data on an ongoing basis – versus through periodic assessment, has gained overwhelming support from leading experts and practitioners in recent years, but to this point actual adoption of the practice has moved slowly.

However, with its September 14 guidance issued to agencies including the Department of Defense and the Intelligence Community, the OMB took a major step forward in requiring government stakeholders to finally get onboard – ordering them to begin to reporting their security standing on a monthly basis.

Until now federal agencies have reported this information to the OMB on an annual schedule.

The new guidance, contained in OMB Memorandum 11-33, FY 2011 Reporting Instructions for the Federal Information Security Management Act (FISMA) and Agency Privacy Management, specifically requires reporting information produced by continuous monitoring methodologies, along with setting guidelines for protecting personal privacy and reporting breaches.

As regulators appear to be taking a “crawl-walk-run” approach, this guidance is a major step (“walk”) in the right direction as it provides for more frequent and meaningful reporting of agencies’ security status comparative to quarterly, semi-annual or annual reporting. At the same time, the guidance still recognizes that many government enterprises are not yet fully equipped to perform continuous monitoring.

An unexpressed, but implicit, by-product of this reporting is that it allows security leaders to better manage many processes: plugging obvious security holes, eliminating known threats and vulnerabilities, denying unnecessary connections, keeping security policies up to date, and better enforcing security policies … to name just a few.

It’s no coincidence that I’m posting this feedback on the blog here at RedSeal, as the company’s solutions provide exactly the types of capabilities around proactive assessment and utilization of outcome-based metrics that the OMB, NIST and others have defined as requirements for continuous monitoring.

Hopefully, this guidance is a prelude to real-time (“run”), or at least daily, continuous monitoring facilitated by software such as RedSeal that analyzes output from firewalls, intrusion detection systems and other key defensive mechanisms.

Yet, while the memo also addresses the issue of privacy and breach reporting and notification, it appears not to make significant changes in those key areas. This issue alone will likely take lawyers, politicians and the courts years to sort out.

While continuous monitoring is badly needed today, it may take the government several years to get there, either due to affordability concerns or bureaucratic in-fighting.

But, all-in-all, the new OMB guidance represents tangible progress that will ultimately enable federal security leaders to better manage their overall posture.