Monday, August 27, 2012

Cold Storage Using OpenStack Swift vs. AWS Glacier

Amazon announced their latest IaaS service last week called Glacier, which is intended for cold storage of data. It is 10x cheaper than Amazon S3 (ignoring access charges). Amazon S3 is already ridiculously cheap as compared to enterprise storage and Glacier takes it to the next level. With Glacier, retrieval needs to be infrequent and can take hours. This restriction is what makes it "cold" storage. It seems like tape-as-a-service to me even though Amazon doesn't use this word at all. However if it walks like a duck and talks like a duck, it must be a duck. In this case tape.

The cost equation is amazing. OpenStack does not have an answer. Does this mean a problem for OpenStack? I don't think so, rather I think this is an opportunity. A combination of OpenStack Swift and Linear Tape File System (LTFS) can not only match, but leapfrog AWS Glacier.

DISCLAIMER: The views expressed here are my own and don't necessarily represent my employer Emulex's positions, strategies or opinions.

LTFS is a relatively new beast, out only since 2010. It makes a tape look like a filesystem similar to a memory stick. This takes tape, which is a complex sequential block device, and makes it look like an easy to use, well understood filesystem.

So can one just take Swift and replace xfs with LTFS? While the devil is in the details, I would say in principle yes. A lot of things would have to be looked at ranging from new REST APIs, a potential caching layer on disk, metadata (or thumbnails) on disk vs. data on LTFS split, to the ring and auditors potentially working slightly differently. However none of this is rocket science and presumably a motivated group (or company with LTFS expertise e.g. IBM, HP, Oracle, Quantum etc.) could get this done pretty quickly.

The cost numbers published by Jack Clark in his blog are pretty amazing. If they are indeed accurate, you could do a Glacier like service for private or public cloud at a substantial discount using the Swift + LTFS combination!

Maybe a birds-of-a-feather session for the San Diego summit?

1 comment:

  1. Intriguing idea... worth a brainstorm with a white board. I'll be at San Diego with 2 other people from CERN.