Developing Security

Scalability Testing In The Cloud

Not long ago, we set out on a mission to perform a full scalability test on one of our products (Trend Micro Deep Security). After some quick, back-of-the-napkin calculations we discovered that we needed somewhere in the order of 35 Dell 710′s with virtualization to complete our test. Finding that many available servers is a tall order for any company, and buying that many servers for a month long test was completely out of the question (try asking your managers for 35 servers and see how pale they go!).

Naturally we turned to the cloud to help us out. Amazon Web Services (AWS) was a good fit to provide the amount of smaller scale resources we needed. (In our case micro and small instances were perfect for simulating a large manager/agent architecture, with each instance simulating dozens of agents).

One thing to be aware of, you can’t simply open an account and request 1000 micro instances. The AWS capacity team works with you, via good old e-mail, to plan the right mixture of instance types, platforms, availability zones and regions that work for both your project and AWS. Once the configuration was settled we designed the tools we needed to rapidly scale up and down our test environment. This included custom AMI (templates) and tools that leveraged the APIs for discovery and resource monitoring.

We ran into our share of quirks on the AWS platform including time skew issues when using high CPU, invalid CPU information for micro instances on CloudWatch and of course, inevitable price wars over spot instances! Due to the nature of the tests, not everything went to schedule. On occasion our plan to scale up was met with error messages from the AWS API of “Insufficient Capacity”. It helps to have backup plans when a particular instance type or region is in high demand.

Once we had addressed our various issues, AWS proved to be an incredible resource for finding scalability issues and quickly testing improvements. The ability to rapidly provision hundreds of VMs from a single AMI allowed us to scale up and down depending on the requirements of the test.

In the end we met our scalability goals and spent a fraction of what it would have cost in-house. We met our goals, and our manager kept his rosy complexion.

Reprint from Cloud Security Blog by Trend Micro

London Bridge is Falling Down

Reprinted from a piece I wrote on the Cloud Security Blog by Trend Micro.

Everyone is familiar with the traditional nursery rhyme, “London Bridge is Falling Down.” However, few know that it traces its roots back to a factual wonder of the medieval world.

In 1209 a massive stone bridge was opened over the river Thames. Quite different than the modern London Bridge we know today, this colossal structure was an engineering marvel of its day and included a chapel at the apex of the bridge.

It didn’t take long for people to realize the potential of this new prime real estate and by the late 1200’s the bridge was completely lined with multi-story structures straddling each side of the bridge (some extending out over the water). The bottom floor was comprised of businesses while the upper floors became the most desirable housing in medieval London.

Somehow this sounds oddly familiar: Taking a piece of shared infrastructure and placing business and consumer services upon it, enabling a new way of doing business and living our lives.

The problem, as you can likely tell by the sole musical remnant of this once massive structure, is that it quite literally crumbled.

The original London Bridge struggled from a problem facing the cloud computing model… overcommit. Resource pooling provides fantastic economies of scale, but what happens when everyone needs the resources at the same time?  The bridge could support the houses or a massive amount of traffic, just not both.

As an example, Amazon Web Services reserves memory for each instance, however CPU may be reserved or shared based on the instance type. VMware vCloud has a configurable Allocation Model allowing service providers to control the percentage of resource guarantee.  

When resources are reserved, rejections happen cleanly when provisioning instances. In the overcommit model there is a risk of unpredictable failures at the application layer during a ‘perfect storm’ of resource usage. In the physical world, too many heavy carts crossing the bridge at once results in structural failure; in the virtual world overcommitment could mean data loss, data corruption or lack of availability.

So this begs the question: Are you running on a “London Cloud,” my fair lady?  Make sure you know what resource allocation model your cloud uses.

Cloud Security Alliance Congress 2010

The Cloud Security Alliance kicked off its first major solo event November 16-17, 2010 in Orlando, Florida. The CSA Congress 2010 successfully hosted 370 people with talks covering all aspects of cloud security over two days.

I recently wrapped up a four part series covering the event for the Trend Micro Cloud Blog:

Part 1 - Charney, Hoff and & Mogull
Part 2 - Hoff & Mogull
Part 3 - Jones-Harbour, Cannon
Part 4 - Sutton, Hubbard & CSA Futures

This was my first attempt at pseudo-journalism, and unless my fingers uncramp from taking 13 pages of notes, my last :)

Write Once, Run Any Cloud

Java, and other languages, gave us platform independence long ago. Application writers could ignore the underlying operating system (to a large extent) and focus on the work at hand. While this reduced the need for OS and architecture-specific code, the underlying operating system still required a lot of care and feeding.

Recently, Platform as a Service took over management of the underlying operating system and infrastructure, but early PaaS offerings required moving your application and data out to a service provider and risking lock-in.

Last month VMware announced a partnership with Salesforce.com to deliver a new way to run apps written for the Spring Framework. VMforce opens up new context to run Java enterprise applications, hosted by force.com. These applications are built with the same tools used to run applications within vSphere or vCloud.

Today Google made a big announcement that their popular App Engine framework will now be host-able on VMware vSphere, vCloud partners, VMforce, or other infrastructures such as Amazon EC2.

This is write once, run anywhere on steroids.

This new flexibility is a great way to bridge the gap between private and public clouds:

• For applications that require a large degree of elasticity, multi-context frameworks are a boon to cloud-bursting (extending to a public provider during times of peak usage)
• Disaster Recovery can be much more cost-effective if the answer during an outage is to run the same application on Google App Engine, Force.com, Azure or others
• For applications dealing with sensitive data, public PaaS environments can be used as a test-bed during development (especially scalability testing) before building out the private cloud infrastructure required to host the production application
• Applications can take advantage of frameworks and tools that make scalable applications easier to develop, without locking into a particular provider
• Portability gives additional options for security. It is difficult to apply external security controls (WAF, DAM, DLP, IDS/IPS) to a public PaaS environment, but applications can be guarded by physical or virtual appliances in your private cloud

While it will take some time before development with these frameworks dominate enterprise applications, there is no doubt this will be a factor in the evolution of the datacenter. We are moving, more rapidly than I would have expected, towards decreased relevance of the virtual machine and operating system for server workloads.

This new-found Application and Data-centricity is a welcome step forward. I hope Google's announcement is a sign of more to come.

Cloudamorphosis

Cloud computing and mobile devices have revolutionized our personal and professional lives. These innovations have unlocked a new age of elasticity and mobility. Along with this digital revolution, an unexpected transformation is taking place at the heart of server workloads and mobile devices.

We are witnessing the decline of the general purpose operating system.

The once static datacenter has transformed into a highly agile virtual datacenter, and is once again transforming thanks to cloud computing. First generation migrations to cloud, using IaaS, are facing tough competition from PaaS frameworks designed to take advantage of the rapid elasticity and scalability the cloud model provides.

A similar change is taking place with client devices. The once ubiquitous laptop is being supplanted by highly specialized and proprietary devices like Smart phones, iPads and Netbooks running Google’s Chrome OS. As these devices become more capable, the need for a general purpose Operating Systems like Windows or Mac OSX fades.

While the agility and financial benefits of this metamorphosis can’t be denied, this transformation is not without cost. As we move away from the open general purpose operating system, we lose the ability to deploy host-based controls. The lack of host-based control brings a lack of control and visibility.

The more diversified our IT infrastructure becomes, the more information security is left in the dark.

We are seeing this today, with many organizations grappling with how to secure and monitor all of the places where their data now lives. Breaches through lost laptops of the past have turned into hijacked cloud SaaS and PaaS resources of the future.

These fundamental changes are creating new challenges, but they are also creating new opportunities. As security emerges from the cocoon of the past, a new generation of cloud-focused solutions will unify the diverse mixture of assets, restoring the control we once had and embracing the agility of the new model.

Metamorphosis is an opportunity to change behavior. We can take this opportunity to rethink how and where we secure data.

The changes to the delivery model have made some security aspects easier. For example, one of the advantages of PaaS is the reduced OS footprint needed to support the singular purpose of each instance. Just-enough-OS running below the platform services have a significantly reduced attack surface and require less maintenance. New applications can by patched by simply re-deploying them from an updated template.

In some cases, the new delivery models require new means of employing security. PaaS, for instance requires special consideration when deploying applications directly exposed to the internet. Without host-based controls, PaaS-built applications must rely on filtered network traffic, embedded security modules or other means to augment the applications' resiliency.

As the workloads become more diverse, identity and encryption take on a new importance. With the always-on connectivity we move away from mobile devices having large amounts of data, but it becomes even more critical that we authorize and encrypt data changing hands. Data moving between different cloud resources also needs special care as we enter a future of multi-provider, geographically diverse IT-as-a-Service.

With all of this change, we have to remember that effective security management requires unified visibility and control across the spectrum of traditional assets, mobile devices and cloud computing resources. The next generation of security solutions need to bridge this gap and let our data safely take flight.

Reprinted from the Cloud Security Blog by Trend Micro

Developers: “IaaS? No thanks, I’ll PaaS”

Below is an another excerpt from a post I wrote for the Cloud Security Blog by Trend Micro:

---

As new applications are developed based on the cloud model, developers are turning to Platform-as-a-Service (PaaS) to simplify application development and deployment. After all, babysitting the operating systems, data stores, messaging queues and application containers running below the application is complicated and costly. The promise of PaaS is the delivery of an application infrastructure, where the provider handles the care and feeding of the underlying stack.

Sounds great, until you consider how much control you are really giving up from a security perspective...

Read the rest...

Cloud Computing Standards, Dream vs. Reality

Below is an another excerpt from a post I wrote for the Cloud Security Blog by Trend Micro:

---

Portability and interoperability in cloud computing may seem tangential to security, but avoiding vendor lock-in is about more than having access to competitive pricing or better service. When relying on a single provider there is inherent risk, especially in the availability of the service and data.

Throughout history the need for portability and interoperability has usually been dealt with through standardization. Standard railroad gauges enabled cross continental travel, just as TCP/IP unlocked worldwide communications. It's not surprising then, that many people look at cloud computing and assume we need standards before lock-in can be avoided. But do we really need widely-adopted standards? While not ideal, interoperability can still be achieved through abstraction (or brokering) and portability through conversion in an environment with many standards.

Read the rest...

Why aren't cloud services secured as a service?

Below is an excerpt from my first (hopefully not last) post for the Cloud Security Blog by Trend Micro:

---

Cloud-based security as a service offerings have seen a steady increase in popularity, due to the benefits that the deployment model provides. Security as a service enables rapid provisioning, cost savings and enhanced security through real-time updates and the community effect.

With the explosive adoption of public cloud computing it's time we apply the techniques used to provide security FROM the cloud, to provide security FOR the cloud.

In public cloud environments like Amazon Web Services (AWS), the Elastic Compute Cloud (EC2) instances only provide firewall as a service. It’s up to the customer to harden the operating system and the applications running within the virtual machine instance. Ongoing patching helps reduce the attack surface, however patching alone doesn’t maintain a secure environment. The only currently viable option to augment the security posture is host-based controls deployed and managed by the customer. Host-based agents can provide Anti-Malware, IDS/IPS, WAF, DLP, Integrity Monitoring and other capabilities, but it’s up to the end-user to purchase, deploy, configure and monitor these countermeasures.

This presents an opportunity for service providers to offer security as a service, designed to protect the instances their users spin up. With the introduction of premium pay-per-use security services, customers could choose the countermeasures they require on a feature by feature basis. Adding malware scanning of your virtual machines could be a single check box away.

Read the rest...

Security as a Service and the question of Privacy

During a panel session at the RSA conference, Eva Chen (CEO of Trend Micro) and Jay Chaudhry (CEO of Zscaler) had a lively debate about the role of Cloud computing in providing Security as a Service. Trend Micro recently released the Smart Protection Network, which uses an end-point agent that works closely with a Cloud service to provide web, email and file reputation services. Zscaler by contrast, proxies all web traffic through the service requiring no end-point agent. The debate really focused around how much of the security service should be located externally. There were a lot of points debated including latency, bandwidth, and reliability but one of the most contentious issues was that of data privacy.

Before I elaborate, it's important to mention why an organization would want to consider a Cloud offering for outsourcing aspects of security. One of the earliest Security as a Service offerings was Postini in early 2000 (acquired by Google in 2007). Moving Anti-Spam and Email Anti-Malware outside of the organization was a simple matter of changing the MX record. Organizations could rapidly acquire these services without the costs and complexities of a physical appliance or software solution, and the per user pricing ensured a fair cost for organizations of all sizes.

So clearly, cost (both OPEX and CAPEX) was an initial motivator but over time providers have utilized the central nature to provide significant benefits other than cost. The providers can be much more responsive with real time services replacing staged updates, and multi-tenant data correlation enhances everyone's service. The central infrastructure also proves more fault tolerant than individual appliances or software solutions for the most part, though when service provider failures do occur they have a much broader impact. In short, moving to the Cloud enhances security providing for much bigger and more intangible benefits.

Recently a new generation of Security as a Service Anti-Malware offerings have emerged. Trend Micro, Panda Security and McAfee have all recently released Cloud-based services. They all use a driver at the end points which instead of scanning against a set of signatures, creates a hash of the file. The hash is looked up in the Cloud and the service returns a result of good, bad or unknown. In the case of unknown files, the contents are then fully scanned (either locally or uploaded to the Cloud depending on the provider and circumstance). A local cache of hashes allows future full scans to determine safe files much faster than signature matches. The collective nature means that 99.9% of the files on your system have likely already been scanned and the hash categorized by someone. There is actually a lot of overlap here with another type of security control: File Integrity Monitoring, but I'll leave that for another day. These capabilities are augmented by other reputation and URL blocking services to create a strong shield against web threats. As with Cloud-based email proxies, because hashes (and sometime content) are being sent to the Cloud, there is a measure of trust required in the provider.

Zscaler, Purewire and others offload all of the work to the Cloud by providing the various security services (URL Filtering, Anti-Malware, DLP, etc.) as a web proxy/gateway. The advantage, of course is the lack of end-point agents to deploy and maintain as well as platform neutral protection for many devices. Though I suspect for many organizations the idea of all of their web traffic going through an external provider may be a barrier.

Other Cloud-based scanning services, like Qualys' Vulnerability Scanner or HP's Application Security do not deal with raw data from an organization, though the findings from there are still very sensitive.  

Clearly, the architecture of the solution is important. The more of the stack we move to the Cloud the less there is to deploy and manage locally, but the more trust we have to put in the provider and the more reliant we are on Internet connectivity and availability of the provider. The axiom "Trust but verify" is hard to apply in these environments because of the lack of visibility into the data end-to-end. The Cloud Security Alliance is working hard to address the complex issues of shifting layers of the stack to providers in all areas of Cloud computing.

As much as organizations may hesitate on Security as a Service, it is too important of an advancement to simply ignore. The enhanced security services and significant cost savings are going to inevitably pull us closer to the Cloud. It is key to evaluate the architecture and determine the acceptable risk level. Trusting a provider's service quality, up time or ROI is nothing new, but trusting a provider with our precious data does require a new level of something else usually associated with the sky... Faith. If we put the faith in the providers we can realize significant benefits in return. Other SaaS offerings like CRM giant SalesForce.com have done a good job of instilling the level of trust needed to be successful and I suspect the Security as a Service market will do the same.

I'm interested to see where this goes and how many of our current 'point solutions' end up in the Cloud (or partially in the Cloud). I'm also interested to see how this impacts how the Cloud itself is secured. It seems only a natural fit that Security as a Service will ultimately have a large role in other SaaS, PaaS and IaaS offerings both external and internal to the organization.