Thursday, February 23, 2012

OpenStack Swift’s New Wins at HP, SoftLayer, Wikimedia - a Tipping Point

OpenStack Swift is an open-source object-storage software that can be used to create an Amazon S3 like private or public cloud storage implementation. It started as an internal Rackspace project. Once Rackspace open-sourced Swift, under the OpenStack umbrella, the first wave of adopters was Korea Telecom, Internap, and UCSD (see my blog on UCSD’s implementation).

The second set of wins is with HP, SoftLayer, and Wikimedia (operator of Wikipedia). This is not an incremental progress, in my view Swift has practically won. First, let’s analyze these three wins, then look at why I believe Swift is going to be the winner, and finally see what it means for the industry at large.

DISCLAIMER: The views expressed here are my own and don't necessarily represent my employer Emulex's positions, strategies or opinions.

HP Cloud (Public Cloud Storage)

HP has introduced, since late 2011, a private beta for HP cloud services; essentially an IaaS compute and storage offering. In full disclosure, I am a beta tester and plan to blog about my experience shortly. HP cloud services are based on OpenStack, specifically Swift for Object Storage. Needless to say, this is a tremendous validation point for OpenStack especially since HP has stated that they are committed to contributing back code to OpenStack. HP’s object storage is also connected to their CDN service. Pricing has not been announced.

HP is claiming the following benefits for their object storage service – storage capacity, scale on-demand, security, and reliability. The use-cases HP states are archiving and backup, static content serving, and storing large private or public data-sets. This is pretty much in-line with my blog on use-cases. HP is also stating additional use-cases when compute & storage are used together such as log-file storage/ analysis, audio/ video transcoding, and map-reduce.

SoftLayer Object Storage (Public Cloud Storage)

SoftLayer announced an Object Storage service based on OpenStack Swift last week, another IaaS service. For some interesting reason they don’t call it Cloud Storage. Putting that aside, this is another great validation for Swift being used for Public Cloud Storage. SoftLayer also has connected their Object Storage service to their CDN. Priced at $0.12/GB/month, this service seems to have a number of key differentiators e.g. meta-data indexing & querying, free inbound public bandwidth, and world-wide replication/ transfer capability (coming soon). SoftLayer is targeting the archival/ long-term-storage use-case.

I’m fairly intrigued by the meta-data capability. This is one key area of innovation for Object Storage. I always wondered what it would be like to index all the meta-data in a NoSQL database such as MongoDB. With Meta-data querying (in the generic sense, I have not tried SoftLayer’s implementation), essentially you don’t need to browse through stuff to find an object, you can simply query. For example, you could say “give me a list of X-rays of patients who got treated at the San Jose Kaiser facility”. Or you could say “give me a list of all objects that can accept a slow access time” for the purpose of migrating them to a spun-down SATA zone. In my copious spare time, I need to test drive this capability on SoftLayers Object Storage.

Wikimedia Media Storage (Private Cloud Storage)

Wikimedia Foundation, the foundation that runs Wikipedia, announced a couple weeks ago that they have chosen OpenStack Swift to store media files. The new architecture will store millions of images, sound clips, and other media files across Wikimedia projects. To date the solution had been local storage, but OpenStack Swift was chosen due to performance, reliability, and scalability. The current cluster is 88TB with two proxy nodes, but I’m sure it will grow a lot more as original media is migrated over.

In addition to UCSD, this is another validation for OpenStack Swift being used as a private cloud storage solution.

Swift’s victory fait accompli?
I think yes. At this time, my guess is that Swift probably has more storage under management than all other object storage systems put together – although I have no way to validate or prove this. The open-source development around Swift with large corporations like RackSpace, HP, Citrix, Dell, Cisco, AT&T etc. contributing means that very few private companies will be able to compete with another open-source or, even worse, a proprietary solution.

What does this mean?
If you are end-customer thinking of a public or private cloud storage implementation - you should be looking at OpenStack Swift very seriously. There really are no other stable open-source solutions. There are indeed proprietary ones, but none have done that well. So why bother with them?

If you are an proprietary Object Storage vendor (there are many - NTAP Bycast, EMC Atmos, Caringo, Scality, Cleversafe, Amplidata etc.) what can I say, take cover? Unless you change your gig dramatically, I can’t see how you can beat Swift’s momentum.

If you are a traditional storage vendor and want to play in the Object Storage space, you may need to look at a Swift strategy as opposed to creating a proprietary Object Storage product (as per the comment above).

In my view the latest string of designs for OpenStack Swift represent a very favorable tipping point. OpenStack Swift is now poised to be the leading software stack for Cloud Storage.

1 comment:

  1. Do you still think that OpenStack Swift is a better technology choice than Cleversafe?