As "big data" becomes valuable for business intelligence, enterprises are searching for the best means of storing that data. Enterprises with long-term, large-volume storage requirements can choose between cloud-based storage systems, such as Amazon Web Services' Simple Storage Service (S3) or a range of on-premises storage options, which include storage pods. Storage pods are low-cost, low-feature disk arrays that store hundreds of...
terabytes, recently popularized by the backup vendor Backblaze. Are they threatening the business value cloud brings to large-volume storage?
Storage pods are inexpensive and simple to build and maintain, so why not opt for them over cloud storage? Cloud migration is not always easy, so enterprises looking for pooled, elastic resources that fit with business economics and costs may be drawn to storage pods, which offer many of the same benefits of cloud without some of the headaches.
Market need leads to storage pod emergence
In an era of cloud computing, when owning your own hardware is optional and Infrastructure as a Service (IaaS) providers such as Amazon, Google and Microsoft seem to be steadily driving down prices, it may seem like a step in the wrong direction to build your own design and hardware. At the same time there is a trend to offload infrastructure management and there are also cases of companies building custom infrastructure.
The advent of storage pods indicates the need for a low-cost technology that was not offered by the market, including in cloud computing. Backblaze followed a similar path chosen by Netflix and Facebook. Netflix found a need for management tools to support its cloud video streaming service, so it built its own tools and shared them with the cloud community. Facebook, to drive down the cost of building and maintaining data centers, designed its own server hardware and started the Open Compute Project, which now has open designs for servers, storage, networking, motherboards and racks.
Unlike proprietary storage systems, storage pods are commodity storage systems built using a publicly available design from Backblaze. Because Blackblaze offers unlimited backup storage for $5 per month, a commercial storage solution is not viable for customer needs. Backblaze designed its own storage system built from a custom metal case, a motherboard, SATA cards, cables, a power supply, RAM and commodity disk drives.
Businesses with engineering staff to build and maintain storage pods may choose the DIY route and shop for parts. For those that want the advantage of storage pods without building from scratch, vendors such as 45Drives offer storage pods based on the Backblaze design. 45Drives also maintains a support forum and community wiki for others building storage pods.
When a storage pod makes sense over cloud
Clearly, there are many things that can go wrong even with a storage system designed as simply as the storage pod, so do not jump to a quick decision of storage pods over cloud storage. As always, carefully consider your options before making any decision.
When planning for high-volume storage, consider your performance requirements. For example, you probably would not use a storage pod for online transaction processing applications. Reading and writing data to a bunch of commodity drives will not provide the low latencies expected for interactive applications. AWS Elastic Block Storage (EBS) offers provisioned levels of I/O performance, so you can meet performance requirements without delving into the technical details of how to implement it.
Data stored in cloud storage services like AWS S3 are automatically replicated. If you work with storage pods, you assume responsibility for managing your own disaster recovery controls.
When building your own storage system, you do not offload the need for implementation skills. You'll likely learn a lot about configuring RAID, the challenges of getting the right version of firmware for your system and that interoperable components are not always as interoperable as we would like.
One way to minimize the challenges that come with building and maintaining storage pods is to strictly follow the design specification. The Backblaze blog compares the reliability of different types of drives and other guidance for storage pod users. There is even a discussion about the reliability of different types of cables.
About the author:
Dan Sullivan holds a Master of Science degree and is an author, systems architect and consultant with more than 20 years of IT experience. He has had engagements in advanced analytics, systems architecture, database design, enterprise security and business intelligence, and worked in a broad range of industries, including financial services, manufacturing, pharmaceuticals, software development, government, retail and education. Dan has written extensively about topics that range from data warehousing, cloud computing and advanced analytics to security management, collaboration and text mining.