Disaster Recovery Strategies for Exchange 2007

Should have called this session Site Recovery Manager brainwashing revisited.

I find it interesting that we are still spending time talking about Replistor and MirrorView as viable Exchange replication technologies.  I thought we determined years ago that they just don’t work.

It would make more sense to me when talking about Exchange 2007 and replication to spend more time on Local Continuous Replication (LCR), Cluster Continuous Replication (CCR) and Single Copy Clusters (SCC).  With applications like Exchange enterprises will more to adopt the technology with the tightest application integration (e.g. – Oracle RAC).  I would have preferred a discussion focused on application integration.

Just my 2 cents.

Hello from EMC World 2008

I have been really busy the past few weeks so the blog has slowed to a snail crawl, although I do have an interesting blog that compares some of the more popular Windows data migration tools so stay tuned for that.  This week I will try to report live from EMC World and the sessions that I attend, note that I will be listening, typing and publishing so please excuse spelling and grammar mistakes as I will not be spending time proofreading.

I am now sitting in a session entitled “Simplifying Disaster Recovery with VMware and Celerra Replicator”  This session should be entitled “VMware Site Recovery Manager and Celerra Replicator”

Since I have been typing this blog I may have missed a few details but here is what i have heard thus far:

Site Recovery Manager (SRM) is supported in the following environments:

  • >= VMware 3.0.2 and Virtual Center 2.5
  • >= DART 5.6
  • iSCSI Only (No NFS Support)

One topic that I feel the need to address is the thought that SRM replaces Disaster Recovery (DR) Runbooks.  I could not disagree more, under ideal conditions SRM automates the tasks traditionally documented in the DR Runbook but it is important to remember that SRM is another piece of software that is added to an already complex mountain of software and process, I do not believe this is a replacement for the DR Runbook but rather another infrastructure component that needs to be documented as part of the DR process.

Note:  VMware will only support RDM devices as beta?  What the heck does that mean?

SRM only support crash consistent copies, essentially no support for application awareness and Replication Manager.

I asked the question with EMC and the application vendors (Microsoft) pushing for application (Exchange, SQL) awareness via VSS and other application aware APIs what is the plan for SRM and application awareness?  The answer I received nebulous at best, to paraphrase the presenter I think he stated that you would need to determine if Exchange is core to your business; last time I checked Email is a fairly critical component of doing business in 2008.

Needless to say to this point SRM is not making a good case for jettisoning the DR Runbook.

Talk to you from Tucci’s Keynote next.

The pondering is over…

EMC has acquired iomega for a cool 213 Million or 3.85  share.  This is a significant premium for a company that has been struggling for a while.  Back in the days before every computer has a CD/RW and before we all carried a 4 GB USB drive on our key chain the primary primary medium for transferring data between computers was the 5.25 floppy, then the 3.5 floppy and then iomega with the 100 MB Zip drive which revolutionized my life and had to be the the gadget of the year.  Every techie I know had one and it dramatically simplified our lives, who could forget hooking up you 100 MB Zip drive to your PCs parallel port, nowadays it’s hard to find a PC with a parallel port.  Fast forward 10 years, just about every computer ships with a DVD/RW which can store 4+ GB of data, a 4GB USB thumb drive is ~ $25, translation the Zip drive is died a grim death.

What does this acquisition mean… hmmmm…  I could speculate that EMC has acquired iomega to continue their down market push with the obvious connection between iomega and Retrospect, which BTW I think is true.  iomega has some nice disk based products that play well in this space.  I also think that while the Zip drive is not really of much value these days iomega has a heritage of pioneering a storage technology which could help EMC realize the vision of displacing tape as the primary media for offsite storage in the SMB.  iomega’s iStorage also seems to dovetail nicely into EMC’s cloud computing initiative.  My thought here is that iStorage pairs nicely with Mozy, Pi, EMC’s Cloud Infrastructure initiative, etc…  Then of course there is the iomega Jaz the 2GB removable cartridge that never really took off but could EMC look to increase density and really target tape in the SMB with a technology like this?  The speculator is me says, maybe.  If you look around at companies like rdx are on the right track but may not have the market muscle, presence or heritage to capture significant market share and change the market paradigm, you can almost visualize how EMC could and might parlay the iomega acquisition.

Thoughts on Super Bowl XLII

imageCongratulations to the NY Giants and NY Giants fans. Unbelievable showing!!!! Yes, the Giants looked possessed on defense but did the Patriots look like an 18-0 team? I have a theory on this; read on.? Regardless I could have cared less who won the game, neither team is my team and I did not have money on

the game, but everyone loves the underdog and frankly I am so freakin tired of the the Patriots. Congratulations to Eli Manning MVP of Super Bowl XLII. I would like to offer up that there should be a co-MVP or at least adefensive player of the game should be named… can

you guess who? image Gisele Bundchen co-MVP Super Bowl XLII  Maybe next year Tom Brady should leave Gisele at home. While she is not a bad trophy / consolation prize it may be a bit easier for Eli to share his trophy (The Vince Lombardi Tropy, shown here on the right) with the team and the fans. Congrats NY!

OS X as Vista VMware Guest OS

I am now successfully running OS X as a VMware Guest OS. This was just a Sunday afternoon project to kill some free time… The UI is a bit slow and the networking requires a little tweaking to get it working but other than that everything works OK out of the box. To get more information on how to do this check out this link.

Here a screen shot of my Vista desktop running OS X:

image

The perception is emerging but the reality is legacy…

The market perception seems to be that iSCSI is gaining tremendous steam and many customers who would have adopted Fibre Channel as their interconnect a year ago are now adopting iSCSI. I would agree that this is the case but iSCSI is not an emerging interconnect and might be better classified as a legacy interconnect which is now experiencing a newsworthy adoption rate. An early concern of iSCSI was the performance penalty associated with TCP operations and the impact of software based initiators, this spawned the TOE (TCP Offload Engine) which would offload TCP calculation from the system CPU to a dedicated onboard processor dedicated to TCP operations, today most iSCSI implementations leverage software initiators (today system CPU resources the most part are so under utilized that most environments will never notice a 10% CPU utilization increase that may be associated with iSCSI). Some vendors addressed the TCP concern by modifying the iSCSI protocol to ride on UDP as opposed to TCP thus increasing performance via proprietary protocols which resemble iSCSI (i.e.- LeftHand Networks, HammerStorage and Zetera). It is important to note that LeftHand has pretty much abandoned the proprietary protocol they started with and has now adopted the iSCSI standard, it is also interesting to note that their adoption rate seems to have increased since doing this, when you are a new player I think evangelizing your protocols superiority over the standard is probably a tall order. LeftHand has also changed morphed their business into a software play and embraced the VMware Virtual Appliance Markerplace as a way to propagate their technology.

There are a number of emerging interconnects that technologically out shine iSCSI the question is how quickly can the market makers move to adopt these technologies, are the market makers interested in accelerating the adoption curve? This is a complicated question, on one hand you could argue that technologies are more stable once they have been around longer (I remember reading papers on iSCSI in 2001, did it really take 7+ years to get iSCSI to where is today?) on the other hand if the market makers validate these technologies too early they run the risk of fierce competition from more nimble startups. It is a complex problem, my feeling is that for the most part the adoption cycle is slowed by the market makers as a way to recoup development cost and and slow competition. IMO the by product of this is a slower innovation cycle.

AoE and HyperSCSI both offer the interconnect price point of iSCSI without the performance burden associated with TCP. AoE and HyperSCSI ride on Layer 2 and do not experience the protocol overhead associated with Layer 3 protocols. SoIP (Storage over IP) uses UDP as the transport protocol as opposed to TCP. We are also seeing the emergence of iSER and iWARP, next generation TCP technology that closely resembles Infiniband. iWARP (Internet Wide Area RDMA Protocol) is a superset of the VI architecture and is aimed at reducing the overhead associated with TCP, iSER (iSCSI Extensions for RDMA) maps the iSCSI protocol over RDMA networks like Infiniband or iWARP. iSER address the overhead associated with TCP and out-of-order packet delivery. How many of us hear about these protocols?

It seems that the majority of marchitecture effort is being put into FCoE (Fibre Channel over Ethernet). Major players such as Brocade, Cisco, EMC, Emulex, QLogic, IBM, Intel, Sun and Mellanox have all gotten behind FCoE in a big way so most likely this will be the next big thing in storage interconnects. The emergence of technologies that further leverage Ethernet as the interconnect will change the game, it is hard to imagine that Cisco’s dominance will will not continue to grow. It is likely that more storage services (i.e. – replication, snapshots, etc…) will be handled at the network layer, as these services move into the network layer we will continue to see the further commoditization of the storage market. Should be interesting to watch over the next few years.

While on the surface protocols that sit on top of Layer 2 (AoE and FCoE) may seem to be superior there is a tremendous amount of functionality that is provided at Layer 3 so it is not a forgone conclusion that FCoE will will the battle. Right now the only forgone conclusion I can see it that Cisco wins regardless, the others will be battling for a piece of the pie. But who knows anything can happen, after all this is technology.

Windows and mount points…

Those of us who used CP/M and DOS in the early days became accustomed to drive letters which BTW was a good design when most systems has two maybe three devices. The same computer enthusiasts who used CP/M and DOS most likely through education or professional experience were introduced to UNIX at some point. If you are like me this was probably during your college years, we began to realize how much more elegant the UNIX operating system was with novel ideas such as “mount points”, well Microsoft sorta figured this out a few years ago and integrated mount points into Windows. To me there is absolutely no reason that anyone in the server space should be using drive letters (Excluding A:,B:,C: and D: of course) unless legacy applications are hard coded to use drive letters and the move to mount points is just too painful (unfortunately in the past drive letters were the only way to address storage devices, if you are still doing this for new applications shame, shame!). One issue with mount points is an inability to easily determine total, used, and free space for a physical device from Windows Explorer. While I use mount points almost exclusively, many of my customers complain of the need to drop to the CLI (only a complaint you would hear in the Windows world…. although I agree it is nice to have a single view of all physical devices, like that provided by Windows Explorer). They could open Disk Management but that too is kind of cumbersome to just view available space. Here is a small VB script that I wrote that will provide total, used and free space for physical devices by enumerating the devices that match a specific drive label:

——— SCRIPT STARTS HERE ———

WScript.Echo “B2D Capactiy Reporter – ” & Date
Wscript.Echo “RJB – 1/2/2008”
Wscript.Echo “———————————–”
Wscript.Echo “———————————–”

Dim totalB2D, totalUSED, totalFREE
totalB2D = 0
totalUSED = 0
totalFREE = 0

strComputer = “.”
Set objWMIService = GetObject(“winmgmts:” _
& “{impersonationLevel=impersonate}!\\” & strComputer & “\root\cimv2”)

Set colItems = objWMIService.ExecQuery(“Select * from Win32_Volume where label like ‘b2d%‘”)

For Each objItem In colItems
WScript.Echo “Label: ” & objItem.Label
WScript.Echo “Mount Point: ” & objItem.Name
WScript.Echo “Block Size: ” & (objItem.BlockSize / 1024) & “K”
WScript.Echo “File System: ” & objItem.FileSystem
WScript.Echo “Capacity: ” & round(objItem.Capacity / 1048576 / 1024,2) & ” GB”
WScript.Echo “Used Space: ” & round((objItem.Capacity – objItem.FreeSpace) / 1048576 / 1024,2) & ” GB”
WScript.Echo “Free Space: ” & round(objItem.FreeSpace / 1048576 / 1024,2) & ” GB”
WScript.Echo “Percent Free: ” & round(objItem.FreeSpace / objItem.Capacity,2) * 100 & ” %”
totalB2D = totalB2D + (objItem.Capacity / 1048576 / 1024)
totalFREE = totalFREE + (objItem.FreeSpace / 1048576 / 1024)
totalUSED = totalUSED + ((objItem.Capacity – objItem.FreeSpace)/ 1048576 / 1024)
Wscript.Echo “———————————–”
Next

WScript.Echo “———————————–”
WScript.Echo “Total B2D Capacity: ” & round(totalB2D / 1024,2) & ” TB”
WScript.Echo “Total Used Capacity: ” & round(totalUSED / 1024,2) & ” TB”
WScript.Echo “Total Free Capacity: ” & round(totalFREE / 1024,2) & ” TB”
WScript.Echo “Total Percent Free: ” & round(totalFREE / totalB2D,2) * 100 & ” %”

——— SCRIPT ENDS HERE ———

This script was originally written to report the utilization of devices that were being used as Backup-to-Disk targets, hence the name B2D Capacity Reporter. The scripts keys on the disk label so it is important to label the disks properly so that it will report accurately (this can be done from Disk Management or the Command line). I have bolded above the only change that really needs to be made to make the script function properly. This script could easily be modified to report across multiple systems which could be useful if you are looking to tally all the space across multiple servers.

I have only tested this on Windows 2003 so I am not sure how it will function on other versions of Windows. Enjoy!

Oh, one more thing. When you save the script be sure to run it with cscript not wscript (e.g. – cscript diskspace.vbs).

Sharing the Windows Console

Ever have the need to share a Windows Remote Desktop session? Of course you have, WebEx, LogMeIn, GoToMyPC and other Web based collaboration suites are quite popular for this purpose. What many don’t know is that Windows Terminal Services actually has some functionality built-in to enable this. Here is how a user can “shadow” a console session being controlled by another user:

  • The primary user opens a remote console session [START->RUN->] the following command:
    • mstsc -v:servername /F -console
      • This opens a remote desktop session console session to a specified servername in full screen mode
  • The second user opens a remote session [START->RUN->] the following command:
    • mstsc -v:servername /F
      • This opens a remote desktop session to a specified servername in full screen mode
    • Next open a command window
      • START->RUN->cmd.exe
      • type “shadow 0” (omit the quotes)

This will prompt the primary user stating that you would like to connect to their console session, once they accept the connection you will be sharing the same session. For more information refer to this (http://support.microsoft.com/kb/278845) Microsoft knowledge base article.

The effect of latency and packet loss on effective throughput

One of my customers who was running three replication technologies, XOsoft, PlateSpin and EMC MirrorView started experiencing issue when we relocated their DR equipment from the production facility where we staged and tested the applications to their DR location.

While having the Producion and DR environments on the LAN we successfully replicated Exchange with XOsoft, Operating Systems with PlateSpin and Data with MirrorView, once we moved DR infrastructure these components all failed. This prompted us to perform network remediation. The following are the results of a simple ping test that was performed using WinMTR.

image

The above chart shows the output from output from a simple test which shows packets, hops, latency and % packet loss. We ran this tests a number of times from different source and destination hosts with similar results. The originating host for this for this host was 192.168.123.6 in the chart below 192.168.120.1 is the first hop.

NOTE: the destination 192.168.121.52 was sent 414 packets but only received 384 (NOTE: this number worsens over time). This is consistent with the behavior that XOsoft, PlateSpin and MV are experiencing.

The graph below represent the impact of packet loss on bandwidth. As you can see 1% packet loss has a dramatic affect on relative bandwidth.

image

Using a bandwidth calculator found here http://www.wand.net.nz/~perry/max_download.php, we calculated the relative bandwidth using the metrics we observed.

  • An effective speed of ~ 30KB/s was calculated using 7% packet loss, a 10Mbit link and 90ms RTT
    • with 15ms RTT the effective speed is ~ 185 KB/s
  • An effective speed of ~ 128KB/s was calculated using 1% packet loss, a 10Mbit link and 90ms RTT
    • with 15ms RTT the effective speed is ~ 700 KB/s

These number are dramatic when compared to the expected 9 Mbit or 1152 KB/s

In conclusion a clean network is critical, especially with technologies like replication that rely on little or to no packet loss and the use of all the bandwidth available. Interestingly enough we seem to be seeing these sort of problems more and more often. My hypothesis is that more and more organizations are looking to implement DR strategies and data replication in a critical component to these strategies, understanding this I believe this problem will get worse before it gets better. For years many of these organizations have used applications and protocols which are tolerant of packet loss, congestion, collisions, etc… protocols like http, ftp, etc… Data replication technologies are far less forgiving so network issues that have most likely existed for sometime are rearing their head at inopportune times.

My perspective on VMW

So the blog has bee a bit quiet lately but I promise I have a few meaty posts which I have been working on.  In my typical fashion I will most likely post them in rapid succession.  Anyway on to this mornings post…  Yesterday I read a fairly well know storage analyst’s take on the royal a$$ pounding that VMware has been taking over the past few days. 

image

This analyst points out a number of good points like while VMW missed the top line revenue goal by only 10 million dollars they increased revenue by 80% and grew profit margins.  This particular analyst takes the position of screw the street, CEO should ignore the street and just run their business.  While I enjoy the idealism, let’s face it when the company went public (a choice they freely made) they relinquished control and the ultimate destiny of their company to the street, running the company has now morphed from building great products to keeping the street happy, a very different paradigm for VMW I am sure.  From a different perspective I agree with the public flogging that VMW has been taking, this virtualization market while hot is still in it’s embryonic stage with < 5% of the servers in the market virtualized.  VMW by no means owns the market, there is still plenty of green field.  So, if I am looking at the market and companies like Microsoft and Citrix are targeting VMW doing business is going to get harder not easier.  With > 95% of the market up for grabs and Microsoft’s already dominant OS position how hard do we really believe it is going to be for the "market leader" in this space.  There is no doubt this is a land grab which means that VMware needs to grow top line revenues (which BTW I understand that they did by 80%) to demonstrate to the street that they are widening the gap.  A companies valuation is all about the streets perception of this companies value in the future.  IMO VMware has enjoyed a overly inflated valuation based on their present day value and market hype, their value in the future will be greatly diminished as players like Microsoft and Citrix truly compete for this business.  At ~ 54 with a P/E of ~109 I think the price correction will continue.