Developing Security

Its All About Secuirty

Virtualization Security is never dull.

While much of the spotlight is on cloud, virtualization continues to invoke challenges and invite opportunity in the field of Information Security. Problems that once had few solutions now can be tackled with an ever-evolving toolbox. VMsafe may have been the first to highlight creative new ways to protect virtual machines, but certainly wont be the last.

It's hard to dissect concrete technical details from the Citrix/McAfee announcement, but I'm always intrigued by any new APIs. In Simon Crosby's blog post he says they are developing, "A hypervisor-native detection service that enables a quantum leap forward in secure virtualization, expressed via an open API to third party detection". I'm excited to see what they come up with, despite the Scott Bakula reference. Imagine my surprise when I read a quote from my humble blog at the end of his post, unfortunately I'm not insured for coffee-computer spit-takes.

I work for Trend Micro who, I'm happy to say, is leading the way in security for virtualization. I personally worked on our VMsafe-based Virtual appliance that we released in 2009 and I'm really excited about what we will soon have to offer. Without talking about things on the horizon, all I can say is this is going to be an interesting time in the history of virtualization security. (BTW: This horizon is largely what's responsible for my recent lack of post density.)

But this post isn't about responding to the announcement, the Trend Micro Cloud Security Blog (cloudsecurity.trendmicro.com) will have commentary on the Citrix announcement. The announcement reminded me of a perennial challenge with developing security software.

While all of this innovation is deeply satisfying to my inner-architect, our fight is actually on a different battlefield. We are still under siege, faced with an ever smarter enemy, and we can't be distracted by a highly disruptive home-front. Virtualization is disruptive innovation for sure, but it needn't disrupt our ability to outmaneuver our digital foe.

When developing security software we have to balance our priorities between:
- New or improved security features
- New contexts (VMsafe, Platforms, IaaS)
- Expanded ecosystems (vCenter, LDAP, SI/EM, Databases)
- Enhanced management (Configuration, Incident Response, Reporting, Metrics)
- System improvements (Performance, Scalability)

But this is a case where not all are created equal.

It's ultimately about the quality of protection being provided, not just how that protection is employed. There are many ways to solve the problem of scan storms in a VDI environment or filter packets before they enter a guest OS, but what really matters is stopping malicious activity. New contexts may be the shiny penny, but innovative security is the real jewel.

Extending VMsafe beyond ESX for Resilient Desktop Security

VMware VMsafe is a security technology for virtualized environments that can help to protect virtual infrastructures in ways not previously possible with physical machines. It has a lot of promise, but why limit VMsafe to ESX? Why not support similar hypervisor security APIs in a virtualization layer for workstations, laptops and mobile devices too? While this may sound extreme, I assure you there is good reason.

It's easiest to explain what I mean by illustrating one of VMsafe's capabilities, network introspection. Until recently there were only two vantage points to perform network introspection (host-based and physical appliances). With the advent of virtualization and VMsafe the landscape has changed. Now you have the following locations to perform packet filtration, deep packet inspection, network monitoring and more:

- Host-Based (NDIS/TDI/WPF)
- vNIC hook with tunnel to Virtual Appliance (e.g. VMsafe)
- vSwitch (e.g. Nexus 1000v)
- vSwitch bridging to Virtual Appliance (e.g. vShield Zones)
- Physical Network

I've listed these options in order of proximity to the application stack, or workload performed by the end-point. The distance from the application stack is an important factor. The further you are away, the less you see (for example node to node traffic on an internal network), yet the closer you are the more opportunity there is to subvert the introspection. It's a balance of visibility and resiliency.

If we look at network introspection as a problem where both the source and destination are untrusted environments, where is the most advantageous view point?

Locating network introspection on the host gives the highest degree of correlation with the actual workload. For instance a firewall can determine if packets should be passed based on the user or process. Unfortunately, host-based also happens to be the location most vulnerable to subversion. Malware routinely shuts down host-based controls, and drivers higher up the TCP stack can be blind to malware talking on raw sockets. It's also not uncommon for user agents to be disabled (temporarily of course). After all, when something goes wrong blame the [INSERT SECURITY PRODUCT NAME HERE].

Move one level outside the host to the new set of options provided by the virtualization layer, and you have a more complete and insulated view of the network traffic (at least until virtualization attacks are mainstream). When filtering at the vNIC or vSwitch level, the vantage point is similar to that of a network based physical appliance, with the exception that you have certainty as to the destination VM and can apply specific policies.

As we move further onto the network, policies are much harder to custom tailor to specific hosts. Physical network appliances rely on switch port, or the IP/MAC of the Ethernet frame if they need to perform specific filtration, neither of which can be guaranteed to be attached to the expected instance on the other end. End-point vendors and the Jericho Forum have been highlighting the ineffectiveness of perimeter security in an increasingly mobile world for years.

Imagine a spy on the tail of a notorious dictator who is driving to a meeting with potential buyers for some spare plutonium. If the spy gets too close they may call off the deal or worse, turn and attack. If he gets too far away he risks loosing them and seeing that all important hand-off of the merchandise. He needs to follow the car at just the right distance.

Network Introspection in the hypervisor (either the vSwitch or a vNIC hook) is a compromise between physical network and host-based that, while imperfect provides a resilient way to apply tight policies. We have this today in vSphere 4, but ESX and ESXi only run on specific hardware and are not suited to workstations, laptops and mobile devices.

VMware View provides a virtualization solution suitable for workstations and laptops. It provides for desktop consolidation, faster provisioning, and desktops independent of the device. On the hardware you install a client similar to a bootstrapper, where your desktop may then run on a server or be checked out locally to let you roam. When the desktop is running in the data center, it's protected by the various physical appliances, 3rd party vSwitches and virtual appliances. When the VM is checked out onto the laptop for roaming you lose this protection.

Base virtualization layers for all devices are coming. What I would like to see is VMsafe-like Security APIs built into virtualization solutions for user computing. When security is deployed at this layer it becomes resilient to end-users with administrator privileges and malware.

Take for example a firewall. Inside the OS it is vulnerable to alteration of the ACLs, subversion by unfiltered frame types, or being disabled by administrators or malware. Removed from this layer it becomes impervious to attempts to disable or bypass protection.

No doubt, there would be complications rolling out VMsafe-like APIs for this type of environment (especially if you extend the inspection abilities to memory, CPU and disk). For one, there would have to be a way to address this layer in order to update software or profiles. Running virtual appliances wouldn't be acceptable under the resources available on these devices so the entire security stack would have to be executed in the hypervisor layer. It would also require specialized support for hardware such as wireless cards.

I challenge VMware, Citrix or Microsoft to introduce a common technique for running security agents outside of the virtual machine where the policy and even the security software could be carried with the VM (in a OVF or similar). Extend this support to a wide range of physical hardware and delivery models (including IaaS) and we could have VMs of any nature (desktop or server) runnable in a wide variety of environments with closely coupled, but isolated security in tow.

Social Security

For my 10th post I thought I would chronicle the amazing experience I have had launching this blog and becoming a more active member of the information security community.

Alan Shimel coined the term 'Social Security' and despite its unfortunate overlap with benefits for the retired and disabled, it really fits. To me 'Social Security' is the act of sharing information security knowledge for the common good. The 200+ bloggers of the Security Bloggers Network give freely of their time and knowledge and many of them have to do so outside of normal work hours in order to keep their posts free of sales pitches ;) Many of the same bloggers are highly accessible on another great tool of the Social Security space, Twitter. Specifically the ever growing list of Security Twits carefully maintained by @quine who's members are a fountain of information.

One month ago today I launched my blog, Developing Security. I'm a Software Architect and I wanted to share my experiences developing (secure) security software. Alan Shimel was gracious enough to add me to SBN providing for an instant readership (and of course, pressure!). I also joined Twitter at the same time and followed many of the members of the Security Twits list. One of the people who noticed that I followed him was Rafal Los. He was setting up an OWASP tour and noticed that my location was on the stop. He asked me if I was aware of the event and going to attend. Long story short, I ended up being involved hosting the meeting and got a great opportunity to meet Raf. Had it not be for Twitter and 'Social Security' that's likely the first of many opportunities that would have passed me by.

Every day, the power of Twitter really continues to surprise me. Once I asked about Javascript exploits (one of my favorite topics) and within a minute I was pointed down another direction that aided my research. Another time I complained about the excessive sound effects on the OWASP podcast and Jim Manicode, who produces the podcast, saw this. After an email exchange Jim agreed with my opinion, and now the latest podcasts aren't loaded with effects. Would I have complained before Twitter? Likely not. By following the Security Twits I have discovered so many more podcasts, webcasts, groups and articles. Often this information appears on Twitter before you can find it on other sources. I've also been pleasantly surprised by the people who choose to tweet about blog posts I have written. The positive comments are really appreciated.

One month and 10 posts in, I'm really enjoying this new world. For anyone remotely thinking about starting an information security blog or joining the Security Twits, I would highly recommend it. Its definitely been a highly valuable excursion for me.

For the next couple of weeks I'll be on the road demonstrating Third Brigade's VMsafe-based Virtual Appliance. If you happen to be at the VMware Partner Exchange in Orlando (April 14 - 16), please drop by the vCloud pavilion. I've been working with a great team on our Virtual Appliance and it's exciting to get a chance to show it off. I'll be writing more about VMsafe and other efforts to apply security at the virtualization layer in the future. It's a project that I have been deeply involved with and it's very interesting subject matter.

The following week I'll be at RSA either at the sessions or at booth 538. I'm also really exited to be attending the Security Bloggers Meetup. These people are celebrities to me and I look forward to meeting them! If you would like to find me at RSA just message me on Twitter @justin_foster.

Learning from the Real World

With the birth of the personal computer, early software engineers adopted real-world systems as a basis for the electronic equivalent. Organizing data on the hard disk in folders and files was an easy to grasp concept for users. When we evolved from the command line, the natural paradigm of the desktop fit well. Users immediately understood the concept of a two dimensional workspace they could organize as they saw fit. With the explosion of the Internet, new paradigms needed a model to give users an inherent understanding. Email took its pattern from a real-world model though oddly enough not mail, but rather the office memo (I guess 'E-memo' didn't sound as good). It was so effective in borrowing the concepts of TO:, CC:, and BCC: that email quickly became widely adopted.

When working on information security systems, I too look to the real world for inspiration. However, not all real-world examples are useful...

A real life firewall doesn't present a great wealth of information on how a packet filter should function. Besides keeping the engine from turning us into a messy spot in an accident, it bears little significance on the complexities of a modern, stateful firewall. Deep Packet Inspection can't look back at a time when legions of humans manually sifted through Ethernet frames. If we were to apply the airport security model to DPI, we'd have to find novel ways of generating latency, removing any liquid-y bits and racially profiling the headers.

What real-world security examples can we learn from and apply to network security?

Neighborhood watch contains a wealth of examples for good practices in network security. It relies on the fact that those monitoring have a closer proximity and familiarity to the subject matter. It's also a low cost method of enhancing the watchful eye using something already available to the police... community residents. If we apply this logic to network security, it makes sense to locate compensating controls as close to the assets as possible. Software on end-points compliments the heavy duty hardware appliances in central points within the network. Knowing what each system is suppose to do and configuring the protection accordingly, greatly enhances our chances of detecting something suspicious.

Wiretapping uses a system of keywords to reduce the volume of conversation an analyst needs to listen to. Perhaps in resource constrained IPS systems a quick pre-match for potential suspicious content could be employed before deeper inspection takes place. New traffic sources could be given a high level of suspicion that reduces over time, decreasing the depth of the checks.

The secret service has an advanced team that surveys the destination of a dignitary in advance. In network security we use tools like Capture-HPC to visit potentially dangerous URLs, presumably to add them to a black list. Perhaps other scouting technologies could proactively assess resources ahead of users.

A Casino uses the eye in the sky to centrally monitor for cheaters. Not only is this efficient, but the nature of the observation (from above) provides an unobstructed and untamperable way of watching for wrongdoings. I would liken this to the external network monitoring provided by VMware in the VMsafe network introspection API. The monitor can watch from a location impervious to the tampering that may occur to the guest OS, but it is close enough to the guest to catch everything.

There are countless other real-world examples that one can apply to network security. Each bring years of proven experience that can inspire the information security systems of today and tomorrow.