One of life’s great luxuries is time. In the case of storage systems, time spent on the problem leads to refinement. At GreenBytes, we have spent the last four years focused on developing a high-performance and stable storage system that delivers the performance of flash storage and the economy of magnetic storage.
The road to refinement has not been straight or smooth. Along our journey, we have learned a lot, and changed a lot, in the storage kernel. This blog installment describes, in some detail, GreenBytes’ Hybrid Storage Architecture (HSA), how it achieves high performance, and why it is unique.
Hybrid Cars Are Slow, Hybrid Storage is Fast
We have all sat in the left lane behind a hypermiling hybrid owner. Or perhaps we have been the one squeezing that last inch of mileage out of our cars through the careful application of the accelerator pedal. Who can blame us, with the cost of fuel at record highs?
Owning a storage system is like driving a hybrid – instead of maximum MPG, we are trying for maximum IOPS or maximum effective utilization. Today’s storage hypermiler wants to go far and go fast with less. Enter flash, a rapidly maturing technology that promises much and asks much (at least in dollars/GB) in return.
At GreenBytes, we include flash in every appliance we sell. But we use it in some unexpected ways.
Throughput is Easy, Low Latency is Hard
Anyone with a desktop RAID array can tell you that by using the right combination of disk striping and large IO sizes, even a small number of drives can swallow data in a hurry. The problem is that these large sequential writes represent only a small fraction of typical IO loads. The rest of us worry about IOPS more than throughput. We worry because IOPS are an indication of the latency in our storage and hence the time our users have to wait for something. Although “good things come to those who wait,” try telling that to your MS Exchange admin!
So, if latency is the enemy, how do we vanquish it? The traditional answer has been non-volatile RAM. Almost all traditional arrays have NVRAM cache that buffers reads and writes to the hard-drives. By writing the data first to NVRAM, storage arrays seek to acknowledge a write OP as quickly as possible. Why NVRAM? Well, ordinary RAM is much cheaper, but it isn’t safe. If we acknowledge a write in RAM and then lose power, data is lost and even worse, client caches go out of sync, leading to new and exciting data corruption after the array is revived. A UPS is not adequate protection, unless you own a mythical bug free and unbreakable device.
So safe write logging is important. Run without it and prepare to experience a lot of frustration. But write logging is only part of the picture.
Today’s virtual storage systems store a lot of information used for bookkeeping. This metadata maintains everything from ACLS in a filer to virtual block lists in a SAN. If you have a lot of storage, you need a proportional amount of metadata. If you want to do fancy stuff like deduplication, you need even more metadata.
Metadata goes in one of two places: chained on the magnetic disks (slow) or in NVRAM (expensive). Ever wonder why one of those fancy dedupe appliances from a major vendor costs so much? It’s not just big company margin requirements – it’s metadata: expensive NVRAM sits at the heart of most of those systems. That NVRAM is the true limiter of the amount of physical storage addressable by such systems.
At GreenBytes, we wanted low latency (high IOPS), high throughput and scalability without the NVRAM tax. How do we do this? Flash memory!
Meet the DCache
The latest GreenBytes File System (GBFS) kernel is unique and divergent from other attempts at hybrid storage. While some arrays use flash storage as a read/write cache and others as a transaction log, we have unified these functions into a single entity called the DCache. What is it?
The DCache is a top-level storage element consisting of one or more redundant sets of flash storage devices. This DCache acts as both a transaction log (somewhere for writes to land and provide low service times) and a persistent metadata repository. All storage system metadata, from virtual block addresses to deduplication lookup tables, resides on the DCache.
And the DCache scales nicely: more storage elements provide lower service times to more requests. A larger DCache provides a larger physical addressable storage space for deduplication. The DCache can also be expanded without bringing down the storage system – try that with NVRAM cards! Below, see a typical pool with a 3-disk single parity DCache:
What the DCACHE Isn’t
Many of our readers are aware that the GBFS is based on a branch of ZFS created back in 2007. The approach to hybrid storage is one area where we have diverged significantly from the vanilla ZFS you find elsewhere. The DCache is not a LOG device. It performs that function, but scales better and doesn’t require dedicated flash for that single function. It is not a renamed CACHE drive. The ZFS cache is read-only and unfortunately doesn’t survive reboot. A volatile cache is a poor place to stow deduplication metadata. Once you reboot, a cold cache takes hours to warm up, and in the interim, deduplication is virtually useless due to poor performance.
What the DCACHE Is
1. Low Latency: More DCache elements (RAID sets) scale to provide more aggregate IOPS
2. Persistent: A first class citizen integral to the storage pool, the DCache survives reboot and is protected by all of the built-in RAID options
3. Cost Effective: The DCache is designed to utilize enterprise class MLC flash with over-provisioning to ensure write endurance that exceeds the appliance lifetime
4. Unique: GBFS has been developed over four years with millions of R&D dollars. The DCache is protected by several pending patent applications
Next time, we take the DCache for a spin with some benchmarks!
