Developing Security

Scalability Testing In The Cloud

Not long ago, we set out on a mission to perform a full scalability test on one of our products (Trend Micro Deep Security). After some quick, back-of-the-napkin calculations we discovered that we needed somewhere in the order of 35 Dell 710′s with virtualization to complete our test. Finding that many available servers is a tall order for any company, and buying that many servers for a month long test was completely out of the question (try asking your managers for 35 servers and see how pale they go!).

Naturally we turned to the cloud to help us out. Amazon Web Services (AWS) was a good fit to provide the amount of smaller scale resources we needed. (In our case micro and small instances were perfect for simulating a large manager/agent architecture, with each instance simulating dozens of agents).

One thing to be aware of, you can’t simply open an account and request 1000 micro instances. The AWS capacity team works with you, via good old e-mail, to plan the right mixture of instance types, platforms, availability zones and regions that work for both your project and AWS. Once the configuration was settled we designed the tools we needed to rapidly scale up and down our test environment. This included custom AMI (templates) and tools that leveraged the APIs for discovery and resource monitoring.

We ran into our share of quirks on the AWS platform including time skew issues when using high CPU, invalid CPU information for micro instances on CloudWatch and of course, inevitable price wars over spot instances! Due to the nature of the tests, not everything went to schedule. On occasion our plan to scale up was met with error messages from the AWS API of “Insufficient Capacity”. It helps to have backup plans when a particular instance type or region is in high demand.

Once we had addressed our various issues, AWS proved to be an incredible resource for finding scalability issues and quickly testing improvements. The ability to rapidly provision hundreds of VMs from a single AMI allowed us to scale up and down depending on the requirements of the test.

In the end we met our scalability goals and spent a fraction of what it would have cost in-house. We met our goals, and our manager kept his rosy complexion.

Reprint from Cloud Security Blog by Trend Micro

London Bridge is Falling Down

Reprinted from a piece I wrote on the Cloud Security Blog by Trend Micro.

Everyone is familiar with the traditional nursery rhyme, “London Bridge is Falling Down.” However, few know that it traces its roots back to a factual wonder of the medieval world.

In 1209 a massive stone bridge was opened over the river Thames. Quite different than the modern London Bridge we know today, this colossal structure was an engineering marvel of its day and included a chapel at the apex of the bridge.

It didn’t take long for people to realize the potential of this new prime real estate and by the late 1200’s the bridge was completely lined with multi-story structures straddling each side of the bridge (some extending out over the water). The bottom floor was comprised of businesses while the upper floors became the most desirable housing in medieval London.

Somehow this sounds oddly familiar: Taking a piece of shared infrastructure and placing business and consumer services upon it, enabling a new way of doing business and living our lives.

The problem, as you can likely tell by the sole musical remnant of this once massive structure, is that it quite literally crumbled.

The original London Bridge struggled from a problem facing the cloud computing model… overcommit. Resource pooling provides fantastic economies of scale, but what happens when everyone needs the resources at the same time?  The bridge could support the houses or a massive amount of traffic, just not both.

As an example, Amazon Web Services reserves memory for each instance, however CPU may be reserved or shared based on the instance type. VMware vCloud has a configurable Allocation Model allowing service providers to control the percentage of resource guarantee.  

When resources are reserved, rejections happen cleanly when provisioning instances. In the overcommit model there is a risk of unpredictable failures at the application layer during a ‘perfect storm’ of resource usage. In the physical world, too many heavy carts crossing the bridge at once results in structural failure; in the virtual world overcommitment could mean data loss, data corruption or lack of availability.

So this begs the question: Are you running on a “London Cloud,” my fair lady?  Make sure you know what resource allocation model your cloud uses.

Cloud Security Alliance Congress 2010

The Cloud Security Alliance kicked off its first major solo event November 16-17, 2010 in Orlando, Florida. The CSA Congress 2010 successfully hosted 370 people with talks covering all aspects of cloud security over two days.

I recently wrapped up a four part series covering the event for the Trend Micro Cloud Blog:

Part 1 - Charney, Hoff and & Mogull
Part 2 - Hoff & Mogull
Part 3 - Jones-Harbour, Cannon
Part 4 - Sutton, Hubbard & CSA Futures

This was my first attempt at pseudo-journalism, and unless my fingers uncramp from taking 13 pages of notes, my last :)

Security B-Sides Ottawa - Wrap Up

Last week Ottawa was host to the first Security B-Sides conference outside of the United States. Over 120 people attended the event held at Tuscon's over two days.

B-Sides was born out of the need to share ideas unbound by the typical mores of the big conferences. Speakers for B-Sides are chosen for their ideas and contributions to the community rather than their notoriety. Attendees of B-Sides are unconfined by budget as B-Sides is free for all.

We chose a live entertainment bar to bring a social and interactive atmosphere, and did it ever work. There was a great deal of audience interaction, and the 'hallway track' was among the best of any event. Every conference should have tables instead of rows of isolating chairs!

As one of the co-organizers of the event, I was extremely pleased with the feedback we received. We had excellent speakers over the two days cover everything from authentication to emergency response and electronic medical records to fuzzing. They came from all over North America and invoked a lively debate that continued well after the talks ended. This wasn't your typical conference as beer-in-hand speakers and heckling audience members were the rule rather than the exception. We also had a good time enhancing (or perhaps destroying) Canadian/US relations with some good natured jabs in both directions.

I want to thank my co-organizers Andrew Hay and Peter Hillier, our volunteers Bryan Tice, Keli Hay, Carl Anctil and Norbert Griffin. I also want to thank our very generous sponsors. Without them the event wouldn't have been possible. Tripwire was even good enough to give one lucky attendee a Macbook Air!

I also have to thank the B-Sides founders and global team, Mike, Chris and especially Jack Daniel for joining us and hosting the very entertaining panel session.

Originally B-Sides paralleled major conferences, but the idea is strong enough to stand on its own, as the attendees of B-Sides Ottawa can attest. I see B-Sides as more of a movement than the sum of its collective parts. It connects people with ideas and each other and I was happy to be a small part of that.

Links:
Security B-Sides Ottawa Homepage
Twitter Stream for #BSidesOttawa
Photos by Jack Daniel
Summary by Pete Hillier
Summary by Dan Menard
Summary by Rafal Los

Security B-Sides Ottawa - Speakers Selected

B-Sides Ottawa is fast approaching and today we can share the schedule of superb talks that cover a broad spectrum of Information Security subjects. We had an amazing set of talk submissions from speakers, both local and from around North America. Narowing the over 25 talks to fit the two day event was no easy task!

If you are interested in attending B-Sides Ottawa, the event is filling up fast, so please register (it's free!) by visiting the official site.

We are still looking for sponsors to support the event, if you know someone who may be interested, please have them contact bsidesottawa@gmail.com.

Day 1 - Friday, November 12th

9:00am - 9:20am	Day 1 Opening Remarks
9:30am - 10:20am	My Life on the Information Security D-List - Andrew Hay, Senior Analyst, The 451 Group, @andrewsmhay
10:30am - 11:20am	Using ISO 27005 for Risk Assessment - Benoît H. Dicaire, InfoSec Strategist, INFRAX, @BDicaire
11:30am - 12:20pm	Vendor Sponsored Lunch
12:30pm - 1:20pm	Myths, Mistakes and Outright Lies (when it comes to your computer security) - Kellman Meghu, Security Engineering Manager Check Point Canada @kellman
1:30pm - 2:20pm	So I've adopted an EMR; What's the worst that can happen? - Peter Hillier, CISO MD Physician Services Inc. (A CMA Company), @DeathwishDuck
2:30pm - 2:50pm	Break
3:00pm - 3:50pm	The Evolving Authentication Landscape - Eric Skinner, CTO, Entrust @EricSkinner
4:00pm - 4:50pm	InfoSec Speed Debates - Jack Daniel and panelists TBD
5:00pm onwards	Vendor Sponsored Dinner, drinks, and shenanigans

Day 2 - Saturday, November 13th

8:00am - 8:50am	Vendor Sponsored Breakfast
9:00am - 9:20am	Day 2 Opening Remarks
9:30am - 10:20am	A new approach to preventing injection attacks on the Web Application Stack - Ahmed Masud CEO/ CTO Trustifier Inc.
10:30am - 11:20am	Fuzzing Cows - Karim Nathoo & Mike Sues
11:30am - 12:20pm	Vendor Sponsored Lunch
12:30pm - 1:20pm	Into the Rabbithole Evolved Web Application Security Testing - Rafal Los @rafallos
1:30pm - 2:20pm	The Unintended Consequences of Beating Users with Carrot Sticks: Radical Thoughts on Security Reform - Ben Tomhave @falconsview
2:30pm - 2:50pm	Break
3:00pm - 3:50pm	CERTs or CIRTs in Canada - Adrien de Beaupré, EWA-Canada, isc.sans.edu
4:00pm - 4:50pm	The Nmap Scripting Engine: Making Nmap work for you! - Ron Bowes @iagox86
4:50pm - 5:00PM	Closing Remarks - Andrew Hay & Justin Foster
5:00pm onwards	Dinner and drinks downtown

More information here.

Security B-Sides Ottawa – Friday, November 12th & Saturday, November 13th, 2010

Andrew Hay and I are organizing a Security B-Sides two day, free event in Ottawa, Canada this November. It about time that the hottest thing in Information Security made its way north of the border.

If you are unfamiliar with B-Sides, it is a community-driven event built for and by information security community members.  The goal is to expand the spectrum of conversation beyond the traditional confines of space and time.  It creates opportunities for individuals to both present and participate in an intimate atmosphere that encourages collaboration. 

More information from Andrew's Blog:

Well we finally decided on Friday, November 12th & Saturday, November 13th, 2010 for the first Security BSides Ottawa conference. We’re still finalizing the venue but we’re quite close to locking it down.

Information about BSides Ottawa can be found here.

The Call For Papers (CFP) can be found here and is open to all.

Remember, this is a free event and we expect numerous speakers and attendees from Government, Education, Defence, Healthcare, Financial Services, and Technology sectors. It’s win-win.

Invite your friends by posting this on Twitter: “#BSidesOttawa Friday, November 12th & Saturday, November 13th 2010: Discover the next big thing! http://bit.ly/BSidesOttawa”

In addition to presenting, we are also looking for volunteers and sponsors. If you think you can help out, please let us know.

Where else can you find a security with scheduled shenanigans? :)

Write Once, Run Any Cloud

Java, and other languages, gave us platform independence long ago. Application writers could ignore the underlying operating system (to a large extent) and focus on the work at hand. While this reduced the need for OS and architecture-specific code, the underlying operating system still required a lot of care and feeding.

Recently, Platform as a Service took over management of the underlying operating system and infrastructure, but early PaaS offerings required moving your application and data out to a service provider and risking lock-in.

Last month VMware announced a partnership with Salesforce.com to deliver a new way to run apps written for the Spring Framework. VMforce opens up new context to run Java enterprise applications, hosted by force.com. These applications are built with the same tools used to run applications within vSphere or vCloud.

Today Google made a big announcement that their popular App Engine framework will now be host-able on VMware vSphere, vCloud partners, VMforce, or other infrastructures such as Amazon EC2.

This is write once, run anywhere on steroids.

This new flexibility is a great way to bridge the gap between private and public clouds:

• For applications that require a large degree of elasticity, multi-context frameworks are a boon to cloud-bursting (extending to a public provider during times of peak usage)
• Disaster Recovery can be much more cost-effective if the answer during an outage is to run the same application on Google App Engine, Force.com, Azure or others
• For applications dealing with sensitive data, public PaaS environments can be used as a test-bed during development (especially scalability testing) before building out the private cloud infrastructure required to host the production application
• Applications can take advantage of frameworks and tools that make scalable applications easier to develop, without locking into a particular provider
• Portability gives additional options for security. It is difficult to apply external security controls (WAF, DAM, DLP, IDS/IPS) to a public PaaS environment, but applications can be guarded by physical or virtual appliances in your private cloud

While it will take some time before development with these frameworks dominate enterprise applications, there is no doubt this will be a factor in the evolution of the datacenter. We are moving, more rapidly than I would have expected, towards decreased relevance of the virtual machine and operating system for server workloads.

This new-found Application and Data-centricity is a welcome step forward. I hope Google's announcement is a sign of more to come.

Its All About Secuirty

Virtualization Security is never dull.

While much of the spotlight is on cloud, virtualization continues to invoke challenges and invite opportunity in the field of Information Security. Problems that once had few solutions now can be tackled with an ever-evolving toolbox. VMsafe may have been the first to highlight creative new ways to protect virtual machines, but certainly wont be the last.

It's hard to dissect concrete technical details from the Citrix/McAfee announcement, but I'm always intrigued by any new APIs. In Simon Crosby's blog post he says they are developing, "A hypervisor-native detection service that enables a quantum leap forward in secure virtualization, expressed via an open API to third party detection". I'm excited to see what they come up with, despite the Scott Bakula reference. Imagine my surprise when I read a quote from my humble blog at the end of his post, unfortunately I'm not insured for coffee-computer spit-takes.

I work for Trend Micro who, I'm happy to say, is leading the way in security for virtualization. I personally worked on our VMsafe-based Virtual appliance that we released in 2009 and I'm really excited about what we will soon have to offer. Without talking about things on the horizon, all I can say is this is going to be an interesting time in the history of virtualization security. (BTW: This horizon is largely what's responsible for my recent lack of post density.)

But this post isn't about responding to the announcement, the Trend Micro Cloud Security Blog (cloudsecurity.trendmicro.com) will have commentary on the Citrix announcement. The announcement reminded me of a perennial challenge with developing security software.

While all of this innovation is deeply satisfying to my inner-architect, our fight is actually on a different battlefield. We are still under siege, faced with an ever smarter enemy, and we can't be distracted by a highly disruptive home-front. Virtualization is disruptive innovation for sure, but it needn't disrupt our ability to outmaneuver our digital foe.

When developing security software we have to balance our priorities between:
- New or improved security features
- New contexts (VMsafe, Platforms, IaaS)
- Expanded ecosystems (vCenter, LDAP, SI/EM, Databases)
- Enhanced management (Configuration, Incident Response, Reporting, Metrics)
- System improvements (Performance, Scalability)

But this is a case where not all are created equal.

It's ultimately about the quality of protection being provided, not just how that protection is employed. There are many ways to solve the problem of scan storms in a VDI environment or filter packets before they enter a guest OS, but what really matters is stopping malicious activity. New contexts may be the shiny penny, but innovative security is the real jewel.

Cloudamorphosis

Cloud computing and mobile devices have revolutionized our personal and professional lives. These innovations have unlocked a new age of elasticity and mobility. Along with this digital revolution, an unexpected transformation is taking place at the heart of server workloads and mobile devices.

We are witnessing the decline of the general purpose operating system.

The once static datacenter has transformed into a highly agile virtual datacenter, and is once again transforming thanks to cloud computing. First generation migrations to cloud, using IaaS, are facing tough competition from PaaS frameworks designed to take advantage of the rapid elasticity and scalability the cloud model provides.

A similar change is taking place with client devices. The once ubiquitous laptop is being supplanted by highly specialized and proprietary devices like Smart phones, iPads and Netbooks running Google’s Chrome OS. As these devices become more capable, the need for a general purpose Operating Systems like Windows or Mac OSX fades.

While the agility and financial benefits of this metamorphosis can’t be denied, this transformation is not without cost. As we move away from the open general purpose operating system, we lose the ability to deploy host-based controls. The lack of host-based control brings a lack of control and visibility.

The more diversified our IT infrastructure becomes, the more information security is left in the dark.

We are seeing this today, with many organizations grappling with how to secure and monitor all of the places where their data now lives. Breaches through lost laptops of the past have turned into hijacked cloud SaaS and PaaS resources of the future.

These fundamental changes are creating new challenges, but they are also creating new opportunities. As security emerges from the cocoon of the past, a new generation of cloud-focused solutions will unify the diverse mixture of assets, restoring the control we once had and embracing the agility of the new model.

Metamorphosis is an opportunity to change behavior. We can take this opportunity to rethink how and where we secure data.

The changes to the delivery model have made some security aspects easier. For example, one of the advantages of PaaS is the reduced OS footprint needed to support the singular purpose of each instance. Just-enough-OS running below the platform services have a significantly reduced attack surface and require less maintenance. New applications can by patched by simply re-deploying them from an updated template.

In some cases, the new delivery models require new means of employing security. PaaS, for instance requires special consideration when deploying applications directly exposed to the internet. Without host-based controls, PaaS-built applications must rely on filtered network traffic, embedded security modules or other means to augment the applications' resiliency.

As the workloads become more diverse, identity and encryption take on a new importance. With the always-on connectivity we move away from mobile devices having large amounts of data, but it becomes even more critical that we authorize and encrypt data changing hands. Data moving between different cloud resources also needs special care as we enter a future of multi-provider, geographically diverse IT-as-a-Service.

With all of this change, we have to remember that effective security management requires unified visibility and control across the spectrum of traditional assets, mobile devices and cloud computing resources. The next generation of security solutions need to bridge this gap and let our data safely take flight.

Reprinted from the Cloud Security Blog by Trend Micro

Developers: “IaaS? No thanks, I’ll PaaS”

Below is an another excerpt from a post I wrote for the Cloud Security Blog by Trend Micro:

---

As new applications are developed based on the cloud model, developers are turning to Platform-as-a-Service (PaaS) to simplify application development and deployment. After all, babysitting the operating systems, data stores, messaging queues and application containers running below the application is complicated and costly. The promise of PaaS is the delivery of an application infrastructure, where the provider handles the care and feeding of the underlying stack.

Sounds great, until you consider how much control you are really giving up from a security perspective...

Read the rest...