SSD RAID

There are many opinions on the benefits and liabilities of SSD RAID floating around the Internet; ION’s experience with SSD RAID differs dramatically from some of it.  ION’s SR-71mach4 SpeedServer™ is a server and storage platform optimized for and with SSD RAID.  This article shares some observations on the performance and endurance of SSDs in RAID usage, and examines and compares performance in different RAID levels.

There are a number of articles and statements that one can find discussing the “problems” with RAID 5 for SSD. Let us examine some of them.

Endurance
The proposition is that the Read / Modify / Write cycle of random writes to RAID 5 arrays can severely and prematurely shorten the life of SSDs and put data at risk.  The Read part of that process has no impact on endurance, so the only factor that really affects wear of the SSDs is that both the changed data and the changed parity block must be written for each random write.  Because of this, RAID 5 doubles the number of write cycles incurred by the SSDs in the array.

How does that work out in practical terms? ION’s standard SR-71mach4 SpeedServer,has (24) 800GB Intel S3500 SSDs distributed across three RAID controllers.  In a standard ION RAID 5 configuration that yields three arrays of 4.5TB each, or 13.5TB total.  The Intel 800GB S3500 SSD is specified by Intel for 450TB of writes. 450TB/SSD × 24 SSDs = 10,800TB.  In other words, the rated endurance would be reached by randomly writing the entire capacity 800 times.

You may have noticed though, that the provisioned RAID capacity of 13.5TB is something less than the raw RAID 5 capacity of 21 × 800GB.  This over-provisioning also increases the endurance of each SSD.  ION follows Intel’s recommendation of reserving an additional 20% of each SSD’s capacity in return for an increase in endurance of 3.5×.  Applying that to the 800 random write cycles then delivers an endurance of up to 2800 write cycles, or randomly re-writing the entire capacity twice each day for almost 4 years. That is not infinite, but it is also not bad.

Consider also the alternatives.  RAID 0 has no overhead of additional writes, but it also has no redundancy and even with the incredible reliability of SSDs, if the data is worth writing, it is probably worth saving with drive redundancy.  RAID 10, also incurs writes to two SSDs for each block written to the array, so that does not improve endurance over RAID 5.  And RAID 6 actually makes the situation worse, because every random write adds writes to two more SSDs for parity.

One of the factors that limits the reliability of RAID 5 in disk based solutions is the Unrecoverable Bit Error Rate.  This rate affects the probability of a second drive failure during the time it takes to rebuild a first failure in a RAID 5 array.  In an enterprise near-line SATA disk that rate is 1 sector in 10^15 reads.  The best enterprise disks have a rate of 1 unrecoverable read error per 10^16 sectors read.  For the Intel S3500 SSDs used in ION’s SR-71mach4 server, that rate is 1 in 10^17 sectors read, or 1/10 of the rate for enterprise SAS disks. Coupled with a RAID 5 rebuild time of well under one hour for the SSD system and the chance of data loss in an SSD RAID system due to multiple drive failures is much, much lower than for a system based on rotating disks.

One obeservation on endurance with RAID 5.  In earlier SR-71 systems that ION has retained access to, benchmarking and evaluation units, all of the SSDs checked by ION have retained more than 90% endurance remaining as reported by the drive’s own S.M.A.R.T. utility.  In some case, that is despite a year of benchmarking and several evaluations at customer sites.  In our experience, real world observations seem to match predictions.

Performance
When it comes to SSD RAID, performance is relative.  SSD RAID 5 does follow the same Read / Modify / Write cycle for random writes as arrays of spinning disks.  And so, performance is proportional to the performance of a single SSD, not multiplied by the number of SSDs in the array.  But a quick look at the RAID 5 performance measured on ION’s SR-71mach4 system shows very significant performance with 20,000 IOPS of 64kB random writes and 262,000 IOPS for the 8kB OLTP mix of 1/3 random writes and 2/3 random reads. Interestingly, RAID 10 performance is not that much higher, with 25,000 64kB random write IOPS and 277,000 IOPS for the 8kB OLTP test.  With less than 300 random IOPS per 15k spinning disk, on can guess that no disk-based RAID 5 system could ever reach 262,000 IOPS doing the 8kB OLTP test.  Theoretically, a system of 1200-1800 spinning disks in RAID 10 arrays might reach that level of performance.

Capacity
RAID, and RAID 5 in particular, is really a big benefit for SSDs.  At 800GB, an SSD is only 20% as big as the 4TB near-line enterprise drives, so the only problem is not using RAID.  Using RAID 5, a single 2U SR-71mach4 SpeedServer delivers 13.5TB of usable, optimally provisioned SSD RAID storage.  Adding external enclosures of SSDs in the SR-71mach4 Configurator, that can scale up to 150TB while preserving the performance of the system.  While 150TB may not be data warehouse capacity by current standards, it is a lot of space for the servers and applications that need that level of performance.

Summary
Contrary to many opinions, RAID 5 seems to deliver an excellent mix of performance, capacity and endurance for all-SSD deployments in optimized systems like ION’s SR-71mach4 SpeedServer.  Comparing observations with SSD RAID 5 versus SSD RAID 10, the RAID 5 system ranges from 75% to 160% of the performance of the RAID 10 system.  So, there may be some corner cases where the RAID 10 performance advantage is worth losing 43% of capacity, but that is likely rare.