Storage Technology Evaluations and Results

During FY 2003, more storage vendors started to introduce storage systems supporting 2 Gb/s FC ports and an increased number of ports for SAN connectivity. Some of these storage systems include EMC's CX 600, Yotta Yotta's NetStorager GSX 2400, 3PARdata's S400 and S800, and DataDirect Networks' S2A 8500. All of these systems can support eight or more 2 Gb/s FC ports, with a possible 16 Gb/s or more peak aggregate performance.

In FY 2003, we tested several of these new storage systems, including the newly acquired CX 600 and an S400 evaluation system from 3PARdata. We also continued our evaluation of the Yotta Yotta NetStorager GSX 2400 system.

In this section, we present a high-level summary of the storage technology evaluations we conducted during FY 2003. These include evaluations of the EMC CX 600, Yotta Yotta NetStorager, and 3PARdata 2 Gb/s FC storage devices. For the purpose of comparison, we also include an evaluation of 1 Gb/s Fibre Channel storage devices.

1         1 Gb/s FC Storage Performance

In order to provide a baseline for understanding the performance and scalability characteristics of the 2 Gb/s Fibre Channel storage devices, we conducted an evaluation of the 1 Gb/s FC storage devices that were already connected to the testbed. At the beginning of FY 2003, the GUPFS testbed had a Fibre Channel SAN with the following 1 Gb/s storage devices:

·       A dual-controller DotHill 7124 RAID subsystem, with an expansion cabinet

·       A dual-controller Silicon Gear Mercury II RAID subsystem

·       A single-controller Chaparral A8526 RAID subsystem with attached storage

Each of the RAID controllers had dual Fibre Channel ports for connecting to the switch, both of which could be used simultaneously. All three storage devices supported RAID 0, 1, 3, and 5 configurations and used similar 10,000 RPM 73 GB U160 SCSI or FC-AL disk drives. The DotHill contained 20 drives, the Silicon Gear 12 drives, and the Chaparral 10 drives. Total unformatted SAN-attached storage capacity was approximately 3.1 TB, with a nominal maximum of 2.7 TB of formatted RAID 5 storage.

These storage devices were originally selected when the project was initially targeted toward early evaluations of the Sistina Global File System (GFS) system, which at that time utilized the novel and promising Device Memory Export Protocol (DMEP) SCSI extension in its distributed lock management. However, Sistina has since moved away from DMEP and has adopted a new IP-based Global Universal Lock Manager (GULM) implementation. Without the DMEP support, these storage devices are no different from any other storage devices and therefore can only be used for file system and fabric technology evaluations. We believe these storage devices are representative of traditional 1 Gb/s RAID disk storage systems. The MPTIO benchmark was run to obtain the baseline performance of these 1Gb/s storage devices.

 

Figure 1. DotHill 1 Gb/s FC performance: in-cache (IC) vs. out-of-cache (OC).

Figure 1 shows the MPTIO results of the DotHill storage device, specifically DotHill's 1 Gb/s FC port performance. The results indicate very little performance scalability on the DotHill storage. The in-cache (IC) performance was about the same as the out-of-cache (OC) performance, and the best performance was about 57 MB/s for writes and 93 MB/s for reads. The MPTIO tests were also run on the Silicon Gear and Chaparral storage devices, and the results also indicated that these 1 Gb/s storage devices had no performance scalability when the number of clients was increased. The DotHill results are representative of the performance and scalability of these storage devices.

The lack of scalability makes these 1 Gb/s devices of very little use for file system evaluation; therefore, we have determined that they are unsuitable for future GUPFS deployment. These 1 Gb/s FC storage devices are now mainly being used for test development and/or the functionality evaluation of file system and fabric technologies.

2         EMC CX 600 Performance

The EMC CX 600 was added to the GUPFS testbed in FY 2003. Figure 2 shows performance of a single EMC CX 600 2 Gb/s FC port, using the PIORAW test with four processes accessing files of different file sizes (FS) and using different I/O sizes.

 

 

 

 

 

 

 

 

 

 

 

 

Figure 2. CX 600 2 Gb/s performance: Small File (FS = 128 MB) vs. Large File (FS = 16 GB or 30 GB).

Three file sizes were selected to study how I/O performance may be affected by the cache in the CX 600 controllers. The FS = 128 MB results show CX 600 performance for in-cache reads and writes. For I/O sizes larger than 256 KB, the small file (FS = 128 MB), in-cache read/write performance is about 200 MB/s. The results indicate that the CX 600 controller is capable of sustaining I/O throughput that is very close to what a 2 GB/s FC port can sustain. However, the read/write performance for the large file tests (FS = 16 GB and 30GB) was only about 150 MB/s, for I/O sizes larger than 256 KB. This seems to indicate that, for writes, the controller was not able to flush the data to the backend disks fast enough to match the writes to its front-end FC ports, and for reads, the controller read-ahead rate was not fast enough to match the front-end reads. This may indicate that the CX 600 storage has a very limited backend disk performance.

Figure 3. EMC CX 600 Scalability: in-cache (IC) vs. out-of-cache (OC).

Figure 3 shows the best in-cache (IC) and out-of-cache (OC) I/O performance scalability on the EMC CX 600 system, scaling from one client to four clients using the PIORAW benchmark, with 1 MB I/O size. The file size for the IC tests was 128 MB and for the OC tests was 4,231 MB, using four individual RAID-5 volumes. Each RAID-5 volume was created using five 73-GB disks.

The I/O scalability of the EMC CX 600 was very disappointing. Except for the in-cache (IC) reads, the CX 600 box showed very poor scalability. Even for the in-cache reads, the performance was limited at about 600 MB/s. For out-of-cache I/Os, the performance was much lower, at about 200 MB/s. All these results make the EMC CX 600 and storage with similar architecture less attractive as a possible choice for the underlying storage of shared file systems.

 

 

Figure 4. EMC CX 600 performance with different Qlogic drivers.

We have also noticed some performance differences for out-of-cache reads when different Qlogic drivers were used with the CX 600 system. The v6.1b2 release of the Qlogic driver has been the default driver used in our testbed environment for I/O benchmarks. However, EMC also shipped a modified Qlogic driver with the CX 600 system. Figure 4 shows in-cache (IC) and out-of-cache (OC) performance over a single 2 Gb/s FC port using different Qlogic drivers.

The results show no performance difference for writes when different drivers were used. However, for OC reads with larger I/O sizes, the read performance was lower (92 MB/s vs. 143 MB/s) with the default Qlogic v6.1b2 driver. This demonstrates the importance of determining the baseline storage performance and picking good device drivers before starting any file system and fabric evaluations, since the I/O performance of the same storage device may be different with different device drivers.

3         Yotta Yotta NetStorager Performance

Since May 2002, we have been beta testing YottaYotta's NetStorager system. The evaluation has been very successful. The NetStorager has improved substantially and is now delivering very good scalable performance. In 2003, a Yotta Yotta NetStorager GSX 2400 system was added to the GUPFS testbed to facilitate the evaluation of file system and fabric technologies.

Figure 5 shows the best in-cache (IC) and out-of-cache (OC) I/O performance scalability on the Yotta Yotta's GSX 2400 NetStorager system, scaling from one client to sixteen clients using the MPTIO benchmark, with 1 MB I/O size. The file size for the IC tests was 2,048 MB and for the OC tests was 32,768 MB, using a single shared 1-TB RAID-0 volume over 32 disks.

 

 

Figure 5. Yotta Yotta GSX 2400 performance.

The overall performance of the GSX 2400 was very good when it was used as the underlying storage for the evaluation of file system and fabric technologies. Being able to support shared access to the same LUNs from multiple ports makes GSX 2400 a viable storage technology for GUPFS. However, the lack of support for any RAID implementation except RAID-0 (striping) makes GSX 2400 less attractive. Another weakness of GSX 2400 is its slow GUI and command line interface (CLI) — it often takes several seconds for each step to complete. For operations (e.g., defining a new LUN) that require multiple steps, its slowness can be very annoying.

Storage devices like the GSX 2400 (which can provide acceptable scalability while exporting the same LUN through multiple interfaces and can also approach the aggregate performance of their interfaces) are beginning to appear. These represent a different class of storage devices from the traditional 1 and 2 Gb/s devices and the middle tier CX600. We believe this new class of storage devices is what will be required for the deployment of a block-based shared file system solution for GUPFS. Other types of shared file system solutions, although not requiring such storage devices, would benefit from their high performance and their ability to effectively service multiple hosts.

4         3PAR S400 Performance

A single 3PAR system can be configured as a cluster of two to eight controller nodes. Each node can scale from four to twenty-four 2-Gb/s full-bandwidth Fibre Channel ports. Controller nodes connect to drive chassis and to hosts via Fibre Channel, and to each other via a high-bandwidth, low-latency backplane. Each 3PAR system features a full-mesh, passive system backplane that provides a dedicated 1 GB/s link between each controller node. This low-latency backplane enables tight and rapid coordination among the controller nodes, allowing them to form a single, cache-coherent system. As a result, all volumes can be exported by any or all controller nodes simultaneously and coherently. In the event of a node failure, its work is transferred to another node in the cluster.

Figure 6. 3PARdata S400 performance.

Figure 6 shows the best in-cache (IC) and out-of-cache (OC) I/O scalability on the 3PAR S400 storage system, scaling from one to eight clients using the MPTIO benchmark, with 1 MB I/O size. The file size used for the IC tests was 2,100 MB and for the OC tests was 50,400 MB, using a single shared 1-TB RAID-5 volume.

Similar to the Yotta Yotta GSX 2400 storage, the 3PARdata S400 storage also shows very good I/O scalability. Both Yotta Yotta GSX 2400 and 3PARdata S400 are far better storage systems than the traditional 1 GB/s storage or EMC CX 600 when used as the underlying storage for the shared file systems like GUPFS.

If we compare the 3PARdata results with the YottaYotta results in the previous section, both storage systems show very good scalability for in-cache (IC) reads and writes. For in-cache I/Os, with the exception of 3PARdata for in-cache writes, both systems are capable of sustain an I/O throughput higher than 1 GB/s (or 8 Gb/s or 1000 MB/s). For out-of-cache (OC) reads and writes, the performance on both storage systems seemed to be limited by their bandwidth to their backend disks, which was about 800 MB/s. The reason that 3PARdata was slower for writes may be because the LUN tested was defined as a RAID-5 volume while, in the Yotta Yotta case, the LUN was a RAID-0 volume.

5         Storage Performance and Scalability

Storage can be a performance bottleneck in any file system. While a storage device may be able to sustain very good single-port performance, having good single-port performance is not sufficient for a shared disk file system like GUPFS. For GUPFS, the underlying storage devices must also demonstrate a very good scalability when the number of clients increases (to thousands or tens of thousands). A shared file system will not scale if the underlying storage does not scale.

Figure 7 shows how storage devices scale when the number of clients increases. The figure gives the results of four storage devices: Yotta Yotta GSX 2400 (YottaYotta), 3PARdata S400 (3PARdata), Silicon Gear Mercury II (SilconGear), and DotHill SANnet (DotHill). Both Silicon Gear and Dot Hill have only two 1 Gb/s front-end ports. Yotta Yotta has eight 2 Gb/s ports, and 3PARdata has sixteen 2 Gb/s ports (although only eight were used during the test). On each storage system, with the exception of Yotta Yotta, which only supports RAID-0 volumes, we created a single RAID-5 volume for shared access by multiple clients.

 

 

 

 

 

 

 

Figure 7. Storage scalability.

The figure indicates that both the Silicon Gear and Dot Hill devices did not scale when the number of clients increased. Silicon Gear performance actually dropped when the number of clients increased. On the other hand, both the Yotta Yotta storage and the 3PARdata storage did scale very well when the number of clients increased.

Figure 8 shows the aggregate performance of the four storage devices: DotHill, Silicon Gear, Yotta Yotta, and 3PARdata, using the MPTIO benchmark with different test conditions. The results indicate that the Yotta Yotta and 3PARdata storage systems will be able to sustain higher performance than Silicon Gear or Dot Hill in a shared file system. These performance results indicate that Yotta Yotta and 3PARdata, and storage systems with similar performance characteristics would potentially be the choices of the underlying storage for shared file systems.

 

Figure 8. Storage aggregate performance.

6         Summary

The results of the storage evaluations we conducted indicated that the performance of traditional 1 Gb/s Fibre Channel storage devices is limited and not scalable, and that such storage systems are unsuitable for use in the future GUPFS deployment. Our testing of the EMC CX 600 showed that this 2 Gb/s storage device, which is representative of the evolution of the traditional 1 Gb/s storage devices with some added middle-tier functionality, has poor scalability as the number of clients increases, except in the case of in-cache writes. We believe this is due to an architectural limitation caused by an imbalance between the bandwidth available to host connections and the device's back-end disks, which its caching algorithms were unable to overcome. Given the limited performance and poor scalability of the EMC CX 600, we have determined that it is unsuitable for use in the GUPFS production environment, as are the related traditional middle-tier 2 Gb/s FC storage devices.

Both the Yotta Yotta GSX 2400 and 3PARdata S400 show good I/O performance and scalability. They represent a new class of storage — they can provide acceptable scalability in performance, they are capable of exporting the same storage to many hosts through multiple interfaces, and their performance approaches the aggregate performance of their interfaces. We believe this new class of storage devices is what will be required for the successful deployment of a GUPFS solution, including block-based shared file system solutions.

With the appearance of the scalable storage devices, storage technologies are generally on track to be ready for the GUPFS deployment.