Starting to compose blog this on my train ride up to Boston on Tuesday night form NYC, I am sure I will actually finish it later which is why I mention this (it is now Friday and I am finally completing this blog on my train ride into NYC, correction it is now Tuesday afternoon and I am finally finishing the blog, I was sidetracked by a few other activities and a more factual blog – worked on it on Friday but did not finish).  The train ride between NY and Boston provides plenty of captive time to catchup on industry happenings. 

I am a fairly regular reader of both Dave Hitz’s blog, great content (since I will most likely not drive a Tesla anytime in the near future I need to read Dave’s blog to gleam some insight) I also happen to think that he is a very intelligent individual – not alone there.  Also from time to time when I am board I also read Jonathan Schwarz’s blog, he is a bit maniacal IMO but he also has some good tidbits which can be gleamed from the piles and piles of rhetoric, for instance reporting earnings on your blog c’mon, sometimes it is just a bit over the top.  I also wish someone would create a JAVA sucks, kill it now petition, I would be more than happy to be the first person to sign it – sometimes portability is just not worth the anguish (With that said SUN had done an incredible job proliferating this utterly painful technology.  Tonight I read both Dave’s and Jonathan’s blog hoping to gain some additional insight into the litigation saga.  BTW – .Net and Mono provide a pretty decent solution for a good looking, decent performing, portable, etc… alternative to JAVA.  Now I am off on a tangent, as a long time Linux user and long time VMware user can someone please explain to me why the hell VMware would remove Linux support for the ESX console in ESX 3.x?  I understand that the bulk of the desktop users are running Windows so Linux support may seem insignificant but I should remind everyone that Linux users needing to run Windows were some of the early adopters of VMware’s technology – show us some love.  The sad part is that the VMware console is built on .Net, some planning and Mono compliance would have given them portability to multiple platforms.  Anyway thought I would through that in there.  Onward….

The litigation situation between SUN and NetApp is what prompted this unorganized stream of consciousness.  It got me to thinking about how much innovation has slowed due to the need to “litigate vs. innovate” (taken from Shwarz’s blog – obviously written during one of his cosmic moments of consciousness).  SUN has changed a lot from the days of Khosla, Bechtolsheim (happy to see him back), Joy and McNealey.

I really enjoy technology history for a couple of reasons, it is factual, finite and tangible, it is well documented with little speculation over what actually happened.  Historical accounts of technology lack the personal opinions that taint world history, religious history and most other historical happenings.  Why is this relevant you will see in a moment – hopefully…

If I were to mention the name Dan Bricklin and Bob Frankston most people will respond with a blank stare and possible a who?  In fact Dan Briklan and Bob Frankston are two of the greatest innovators in personal computing history they wrote and application called VisiCalc which was originally released for the Apple II in 1979.  Interestingly enough the creators of VisiCalc did not pursue a patent for their work, while today I am sure that Dan and Bob would file for a patent back in the day almost no one filed for patents related to software inventions.  A few other software inventions not protected by patent law are word wrapping, cut and paste, the word processing ruler, sorting and compression algorithms, hypertext linking, and compiler techniques.  Could you imagine how innovation would have slowed if these technologies were all patented…  The ability to leverage each others inventions dramatically accelerates the innovation cycle.  The open source community is proof positive that this does work, the problem is that so many of the market leaders (e.g. – Microsoft) would rather protect their market share by  locking the doors rather than just being the best and forcing them selves to out innovate their competition.  It is a complex problem and not one that I have answer for, with that said I would like choice, I would like an open document format that works seamlessly across multiple word processors, etc…  Stifling competitive innovation through lock out is not the right answer.

Another name that most would struggle with is Gary Kildall who was one of the key pioneers of the PC industry who was “upstaged” by Microsoft.  Gary Kildall was a hobbyist and the creator of the MS-DOS predecessor CP/M.  Back in the day when Microsoft was headquarter in a strip mall in Texas developing Basic for the Altair, Gary Killdal was in Seattle running a company called Digital Research and writing a hobbyist operating system called CP/M.  CP/M would be the code base that would eventually become MS-DOS and propel Microsoft to heights beyond their wildest dreams.

While arguably the developers of the modern operating system and the modern spreadsheet application stood by as Microsoft and Lotus became multi-billion dollar companies and continued to innovate today two industry giants like NetApp and SUN would rather take a position just to take a position…  The sad part is that corporations have so much control today that the Bricklins, Frankstons and Kildalls of today, the Torvalds, Raymonds, etc… have a significantly larger hill to climb.

Sometime I actually feel pain when I read Jonathan Shwarz’s blog, with that said WAFL, ZFS, who cares… move on… NetApp, ZFS is not even close to a threat, SUN is not and never will be a storage company.  SUN how about spending less time pulling patents and reviewing them to see what you can litigate against.  I just wish someone would focus their time, money and energy on building something new and innovative rather than trying to keep current technology alive by boxing each other out.

I for sure do not know everything but I do know that the rate of development and innovation has slowed dramatically over the past 30 years.

Following a fit of rage last night after I inadvertently deleted 2 hours worth of content I have now calmed down enough to recreate the post.

The story starts out like this, a customer who recently installed a EMC CX3–80 was working on a backup project roll out, the plan was to leverage ATA capacity in the CX3–80 as a backup-to-disk (B2D) target.  Once they rolled out the backup application they were experiencing very poor performance for the backup jobs that were running to disk, additionally the customer did some file system copies to this particular device and the performance appeared to slow.

The CX3–80 is actually a fairly large array but for the purposes of this post I will focus on the particular ATA RAID group which was the target of the backup job where the performance problem was identified.

I was aware that the customer only had on power rail due to some power constraints in their current data center.  The plan was to power up the CX using just the A side power until they could de-commission some equipment and power on the B side.  My initial though was that cache the culprit but I wanted to investigate further before drawing a conclusion.

My first step was to log into the system and validate that cache was actually disabled, which it was.  This was due to the fact that the SPS (supplemental power supply) only had one power feed and the batteries where not charging.  In this case write–back cache is disabled to protect from potential data loss.  Once I validated that cache was in fact disabled I thought that I would take a scientific approach to resolving the issue by base lining the performance without cache and then enabling cache and running the performance test again.

The ATA RAID group which I was testing on was configured as a 15 drive R5 group with 5 LUNs (50 – 54) ~ 2 TB in size.

Figure 1:  Physical disk layout

My testing was run against drive f: which is LUN 50 which resides on the 15 drive R5 group depicted above.  LUNs 51, 52, 53 and 54 were not being used so the RG was only being used by the benchmark I was running on LUN 50.

Figure 2:  Benchmark results before cache was enabled

As you can see the performance for writes is abysmal.  I will focus on the 64k test as we progress through the rest of this blog.  You will see above that the 64k test only push ~ 4.6 MB/s.  Very poor performance for a 15 drive stripe.  I have a theory for why this is but I will get to that later in the post.

Before cache couple be enabled we needed to power the second power supply on the the SPS, this was done by plugging the B power supply on the SPS into the A side power rail.  Once this was complete and the SPS battery was charged cache was enabled on the CX and the benchmark was run a second time.

Figure 3:  Benchmark results post cache being enabled (Note the scale on this chart differs from the above chart)

As you can see the performance increased from ~ 4.6 MB/s for 64k writes to ~ 160.9 MB/s for 64k writes.  I have to admit I would not have expected write cache to have this dramatic of an effect.

After thinking about it for a while I formulated some theories that I hope to fully prove out in the near future.  I believe that the performance characteristics that presented themselves in this particular situation was a combination of a number of things, the fact that the stripe width was 15 drives and cache being disabled created the huge gap in performance.

Let me explain some RAID basics so hopefully the explanation will become a bit clearer.

A RAID group had two key components that we need to be concerned with for the purpose of this discussion:

1. Stripe width – which is typically synonymous with the number of drives in the the raid group
2. Stripe depth – which is the size of the write that the controller performs before it round robin to the next physical spindle (Depicted in Figure 4)

Figure 4: Stripe Depth

The next concept is write cache, specifically two features of write cache know as write-back cache and write-gathering cache.

First lets examine the I/O pattern without the use of cache.  Figure 5 depicts a typical 16k I/O on an array with and 8k stripe depth and a 4 drive stripe width, with no write cache.

Figure 5:  Array with no write cache

The effect of no write cache is two fold.  First there is no write-back so the I/O needs to be acknowledge by the physical disk, this is obviously much slower that and ack from memory.  Second, because there is no write-gathering full-stripe writes can not be facilitated which means more back-end I/O operations, affectionately referred to as the Read-Modify-Write penalty.

Now lets examine the same configuration with write-cache enabled.  Depicted in Figure 6.

Figure 6:  Array with write cache enabled

Here you will note that acks are sent back to the host before they are written to physical spindles, this dramatically improves performance.  Second write-gathering cache is used to facilitate full-stripe writes which negates the read-modify-write penalty.

Finally my conclusion is that the loss of write cache could be somewhat negated by reducing stripe widths from 15 drives to 3 or 4 drives and creating a meta to accommodate larger LUN sizes.  With a 15 drive raid group the read-modify-write penalty can be severe as I believe we have seen in Figure 2.  This theory needs to be test, which I hope to do in the near future.  Obviously write-back cache also had an impact but I am not sure that is was as important as write-gathering in this case.  I could have probably tuned the stripe-depth and file system I/O size to improve the efficiency without cache as well.