In the last few weeks I’ve had to find an alternate solution for storing and serving VMware images to a big ESX cluster at work. I looked at a lot of open source tools and options, and some commercial ones, and have settled on the NexentaStor.

Our original solution is about 2 years old, and is a Dell NX1950 system running Windows Unified Storage Services 2003.  It had a single MD3000 head unit connected to the NX1950, and two MD1000’s chained off of the MD3000.  The MD3000 and one MD1000 each had 15 400GB 15K SAS drives in them for a total of 30 drives, and the remaining MD1000 had 15 1TB 7500RPM SATAu drives in it.

The first two trays were setup to serve out virtual disks via the iSCSI target provided in Windows Unified Storage Services.  The last tray of drives was for file shares and backups.

Suffice to say, there were many, many problems with performance, reliability, and interoperability.  We had at least 3 major outages attributed to the iSCSI target freezing or becoming overwhelmed.  Several other situations arose when drives failed in the array, and the horrible PERC+MD3000 combo tried to use a spare drive, which drug the usability of the array to a crawl.  In general, my experiences with Windows Unified Storage Services has proven to me that it’s definitely not worth the money at all, especially given that you can only get it (AFAIK) from OEMs.

The iSCSI piece is absolutely horrible in it – slower than anything else I benchmarked, and really quite buggy.  NFS is not even worth deploying as it’s at least twice as slow as iSCSI, and not even on the charts compared to traditional NFS.  Plus it has so many odd quirks about configuring it, that it makes it a big barrier to actually use with ESX.

We have several other solutions at work already – NetApp, EqualLogic, and various types of NFS/iSCSI hybrids.  I went about testing the following solutions before making my decision:

In the end, NexentaStor was the fastest, most reliable, and had all of the options I needed for my environment.  It’s based on OpenSolaris with a bit of Debian, but what I really was impressed with is ZFS and all of the benefits it provides.  By far, it’s NFS and iSCSI implementations were faster than any of the other vendors, even the purpose built EqualLogic array.

EqualLogic was the fastest next to Nexenta, but it’s really expensive, and only iSCSI.  I wanted the option to do NFS too.  NetApp is slightly slower than EqualLogic, and well, is by far highway robbery in terms of cost.  OpenSolaris is unstable as all get out, unless you go back far enough, and then there are missing pieces of functionality, such as iSCSI.  OpenFiler is really nice, and we use it in some other areas already, but it’s RAID technology is just as slow as Dell PERC or other crappy controllers.  It’s NFS and iSCSI implementations are fast, but not as fast as NetApp/EqualLogic and nowhere near NexentaStor.

Nexenta is still small compared to some of the other solutions, but it’s very well thought out.  I am looking forward to a lot more interesting things from their product line.  I’ll write more about the specifics later…