WAAS “Tales from the Field” – Episode 1

I had the pleasure of working with an application (referred to in the blog as APPX) that could use some serious TLC about a month ago. Apperently APPX is in wide use by isurance agencies and brokers. I waas-perfnow know more about the interworkings of APPX than I want to know – and for the life of me can not figure out why the heck anyone would write an enterprise application ontop of the JET database, the answer I was given by the application developer is not everyone has MS SQL Server? OK – but if you are small enough not to have MS SQL Server (BTW – thats pretty freakin small) download MSDE it is free – or better yet why not use MySQL, Postgres, etc… anything but JET – ODBC, JDBC people it’s not 1972 :(. Here is how the saga began. A customer of ours was looking to remove file servers and print servers from 8 branch locataions. The articulated applications were standard CIFS shares and print services – a perfect fit for Cisco WAAS (Wide Area Application Services), a product which uses WAN accerleration technology and caching to remove WAN latency thus providing LAN like performane over the WAN.

The plan was to migrate the data from the 8 branch locations to a core location (the datacenter) and have the users access the data over the WAN. The customer would then be able to remove the server infrastructure and all associated management from the edge locations. Does not get any simpler than this, or so we thought.

A bit of background information. The depoloyment model for the CISCO WAAS gear was to use WCCP NOT inline cards to route the appropriate traffic through the WAE. Pretty early in the process we realized that the end users sitting in the remote location and accessing a SMB/CIFS share in core location were expeiencing huge delays from within the APPX, which was run from a network share and sort of a hyrbid client/server/web application. Fast forward 3 weeks, I finally show up on site to try and resolve a ghostly performance issue. At this point acceleration and caching of FTP, HTTP, CIFS, etc… had been fully vetted and verified as working properly. BUT APPX was still expreiencing significant performance issues. Immediately I began to believe it was an application specific issue – in the end the assuption was correct but lets explore how we identified the issue.

Step 1: Contact the application developer and understand exactly waas-topologywhat the application does and the chronology of the steps.

The following are the notes/bullets taken away from the conversation with APPX developer:

  • Documents produced and mdb files can in fact be stored on different paths – the applcation uses a .mdb (JET database – HINT #1 PROBABLE LOCKING ISSUE) and .dot word templates to create form letters.
  • When APPX creates files it prefixes the documents with “s” or “m” – “S” refers to schedule and M refers to memo
  • APPX also suffixes the document with a numeric value – this value is the record locator
  • The path to templates and proposals are stored in proposal.mdb
  • Document locataions are strored in APPX personalization

As I mentioned above the APPX installation including the .dot and .mdb files were stored on a network share.

Step 2: Plan of attack

  • Test the performance with the APPX folder (application) stored on the to local path (c: root) – Run wizard.exe to change the path
    • Write the new template path. Writes the template path to the proposal.mdb
  • If this worked the path changes could be changed via the registry. Which meant that we could automate the changes so that the users will not need to run the wizard.exe process. This will simplify a complete rollout of the local APPX installation.

NOTE: Currently (prior to the WAAS deployment) when a new template is created the APPX directory is robocopied to network shares at the 8 remote locations. The new process would be to use a login script to copy the APPX install to the local workstations when an change is made. Pre-positioning could be used in this case to increase the prerformace of the initial load. Not only will this solve the performance issue but it also represents a significant improvement to the currrent process. The automation of updating the templates using login scripts removes the robocopy responsibility from the IT staff and ensures that the users are always operating from the latest database and templates.

NOTE: reg key location for the APPX ditro directory – HKCU/Software/APPX/

Step 3: The actions taken to resolve the issue

Creation of scripts listed below to automate the required changes:

 

  • update_distro.bat – creates APPX distribution as appxload.exe
  • logon.bat – user logon script (encompasses commands from upappx.bat and regchg.bat)
  • upappx.bat – Standalone script to updates users local c:\appx install
  • regchg.bat – Standalone script to modifies APPX db directory location reg key
  • appx_redirect_to_C.reg – reg key

All scripts housed in centralized network location ie \\core\share\waas_appx_scripts.

The logon scripts are located at \\core\netlogon

Ultimately to increase the speed of APPX the APPX Word document templates and mdb lookup databases will be moved from \\core\share\appx\database to c:\appx\database on the users localhost.

These changes dramatically increased the performance of the APPX, the belief is that performance problems were due to jet database related locking and lock release issues that were worsened due to WAN latency. During the automation process I did in fact verify that locks are placed on mdb files on a fairly regular basis. Initially I was told by the application developer, that the mdb files are not changed and locks are not placed on these files but this is not the case. Interestingly enough the APPX folks stated that most users are going to a more distributed model vs. a consolidated model. After understanding the application architecture it is not hard to understand why.

The process going forward would be the following is the following:

  • Users in core will continue to run from the LAN location – While performance could probably be faster if they followed the same deployment model as listed above because the LAN locking is most likely slowing performance the application response time is adquate so it was decided that it would be left as is.
  • There is a single administrator in the core location who modifies the templates – this typically happens weekly but may happen more frequently
  • In the past these changes would be robocopied by IT to local shares in each remote office on Sunday nights. The new automated user update process removes the need for IT to manage the distribution of the APPX changes on a weekly basis. Users will now have access to updated APPX on a daily basis vs. weekly. Updates will be performed when the user logs in.
  • There is now a scheduled task that has been created on \\core that performs the following tasks:
    • The scheduled task name is appx_load
      • Copies the APPX directory from d:\share\appx to d:\share\waas_appx_scripts\appx
      • This step is required because there are locks on the mdb files in the \\core\share\appx directory and the files were skipped by 7z during compression.
    • Once the copy of APPX is complete the APPX directory is packed into a self-executable 7z files name appxload.exe
    • The scheduled task runs at 2AM S,M,T,W,T,F
    • The appx_load scheduled task runs update_distro.bat which contains the following two commands:
      • xcopy /e /y d:\share\appx d:\share\waas_appx_scripts\appx
      • “c:\program files\7-zip\7z.exe” a -sfx appxload d:\share\waas_appx_scripts\appx
    • This step is required because the compressed file transfer is far faster than using robocopy to update the users local APPX installation.
  • The logon.bat script has been modified to copy the appxload.exe to c:\appxload.tmp on the users local machine
  • Next appx.exe is run and unpacks the the APPX distribution to c:\appx
    • NOTE: There is no measurable difference in user logon time. The customer is very happy with the process
    • NOTE: The robocopy process was taking between 4 minutes for the initial copy and 30 seconds for updates
  • Last the logon script modifies the db directory reg key to point to the new appx location
  • The commands that are run in the logon.bat are also separated into two standalone .bat files
    • upappx.bat and regchg.bat

NOTE: 7-Zip has also been installed on the \\core server and is required

NOTE: Once all the users have logged on and their reg keys modified the regedit command can be removed from the logon.bat script.

Script Details:

logon.bat:

rem update local APPX distro
@echo off
mkdir c:\appxload.tmp
copy \\core\share\waas_appx_scripts\appxload.exe c:\appxload.tmp
c:\appxload.tmp\appxload.exe -oc:\ -y
rem update APPX reg key
regedit.exe /s \\core\share\waas_appx_scripts\appx_redirect_to_C.reg

update_distro.bat

xcopy /e /y d:\share\appx d:\share\waas_appx_scripts\appx
“c:\program files\7-zip\7z.exe” a -sfx appx d:\vol15\waas_appx_scripts\appx

appx_redirect_to_C.reg

Windows Registry Editor Version 5.00
[HKEY_CURRENT_USER\Software\APPX]
“Database Directory”=”c:\\APPX\\DATABASE\\”

upappx.bat

@echo off
mkdir c:\appxload.tmp
copy \\core\share\waas_appx_scripts\appxload.exe c:\appxload.tmp
c:\appxload.tmp\appxload.exe -oc:\ -y

regchg.bat

regedit.exe /s \\core\share\waas_appx_scripts\appx_redirect_to_C.reg

So that more detail than I thought I would write. Obviously the actual application is not listed in the blog, but if you are interested in that app please send me an Email rich@bocchinfuso.net. I would be more than happy yo share more of the detail with you. Also if you find any typos, etc… in the post please leave a comment.

BTW- The customer has now been running for about 30 days and they are happy with the performance.

Best practices for building and deploying virtual appliances – from EMC World 2007

High-level guidelines for building a Virtual Appliance:

  • Start with a new VM
    • Use SCSI VMs for portability and performance
    • Split into 2GB files for size and portability
    • Allocate minimum required mem to the vm (clearly document this)
    • Disable snapshots, do not include snapshot files
    • Remove unused hardware devices
      • i.e – USB, audio, floppy, etc…
      • start devices such as floppy, cd, etc… disconnected
    • Disable “Shared Folders”
    • Avoid the use of serial and parallel ports or other specialty features
    • Chose the proper network type – Bridged is the the default but there may be a situation to use NAT
  • Install “JEOS” – Just Enough OS
    • Select the Linux distribution of your choice (pick a supported guest to make life easier on yourself – Note: if you use RHEL or SLED you will need to get permission to distribute)
      • Minimize he footprint
        • only install and run necessary services
        • only open necessary ports
        • more secure and less to patch
      • Install VMware Tools
        • Improved performance, optimized drivers for virtual hardware, etc…
        • Hooks to management tools
        • Fully redistribute able inside Linux guests
      • Include or enable users to add a second virtual disk
        • put config information: user data, logs files, etc… on second disk
        • easier to update and backup the virtual appliance
  • Install the applications stack
  • Configure the appliance for first boo
    • Console experience
      • Accept EULA
      • “Zero-Configuration”
      • Present management URL
    • Web management interface
      • Configure networking
      • Configure security
    • Monitor and manage performance of the solution
    • Test everything! Test first boot/complete execution on different machines
  • Value-Add
    • Expose logging from underlying services
    • Support SNMP
    • Provide audit hooks
    • Allow users to backup configurations and/or restore to factory default settings
  • Package the virtual appliance
    • Copy appliance to new directory
    • Remove unecessary files
      • log files, nvram file, etc..
    • Add your “Getting Started Guide” or “ReadMe”
    • Compress the entire directory
      • 7-Zip or RAR work very well
      • Use cross-platform compression technology
    • Create a web page to host your virtual appliance download
    • Certify your virtual appliance
    • Create a listing on http://vam.vmware.com
  • Provide a patch mechanism for your virtual appliance
    • leverage the default packaging technology of the OS (i.e. .deb or .rpm) or build your own packaging and update technology
    • Support direct online updates from your servers
    • support offline patching

Available tools, frameworks and services:

  • VMware Virtual Appliance Development Kit
    • under development – used as part of Ace 2.0
    • Web-based virtual appliance development tool
    • Includes rPath Linux distro
    • Includes its own patching solution
    • Service provider
    • Developing their own framework
    • Service provider

PSTs on network file shares…

Not a good idea. Someone I know implemented Cisco WAAS and is experiencing performance problems. Specifically the CPU on the WAE device is spiked to 100% utilized. They are attempting to per-position 80GB of data including large 1-2GB PSTs to the WAE (Wide Area Application Engine). In my opinion this is a recipe for disaster. First off Microsoft does not support PST access via network shares, and once you read why you will understand why caching PSTs is an even worse idea.

http://logs.technet.com/askperf/archive/2007/01/21/network-stored-pst-files-don-t-do-it.aspx

http://support.microsoft.com/kb/297019

I ran my own little test by moving a PST to a share, starting wireshark (ethereal) and watching the activity. Accessing the PST over the network makes CIFS alone look very efficient.

Two must have utilities for Windows

Admittedly I still sparingly use Internet Explorer but I found a cool little plug-in that allows me to now run windows updates from a non-IE browser, further limiting my need for IE. Check out it out at http://windowsupdate.62nds.com.

The second utility removes the Windows Genuine Advantage (WGA) from your computer, another fine piece of Microshaft nuisanceware. The original location of the “removewga” was here http://www.firewallleaktester.com/removewga.htm but it appears Microsoft forced the developer to disable the links. You can still grab the utility here http://www.softpedia.com/get/Tweak/Uninstallers/RemoveWGA.shtml .

Preliminary results: rsych to replicate virtual machines

So a couple of months ago or so I had the idea to test and document the use of rsych to replicate VMware virtual machines. Unfortunately my machine is running ESX 2.5.x and I had yet to upgrade to 3.0 so this information is a bit depreciated but I feel that it will be indicative of what I will see on ESX 3.0 (aka – VI3). On ESX 3.0 the process actually becomes much easier because all of the files (.vmx, .vmdk, .nvram) are contained in the same directory structure. So here is a simplistic representation of the commands required to replicate a VM on ESX 2.5, I am also in the process of building an automation script for ESX 3.0:

************ CREATE .REDO LOG ************
ware-cmd ~bocchrj/vmware/rh62_1/linux.vmx addredo scsi0:0

************ STARTING REPLICATION ************
rsync –verbose –progress –stats –compress /vmfs/VMs/rh62_1.vmdk esx2::rsyncVMs

************ REPLICATE VM CONFIG FILES ************
rsync –verbose –progress –stats –compress /home/bocchrj/vmware/rh62_1/* esx2::vmconfig/rh62_1

************ ADDING REDO.REDO LOG ************
vmware-cmd ~bocchrj/vmware/rh62_1/linux.vmx addredo scsi0:0

************ COMMITING REDO LOGS *************
vmware-cmd ~bocchrj/vmware/rh62_1/linux.vmx commit scsi0:0 1 0 1
vmware-cmd ~bocchrj/vmware/rh62_1/linux.vmx commit scsi0:0 0 0 0

************ DONE ************

The process worked well. I captured the output of the initial rsync and the second rsych cycle below:

Initial rsync cycle

Number of files: 1
Number of files transferred: 1
Total file size: 419430912 bytes
Total transferred file size: 419430912 bytes
Literal data: 419430912 bytes
Matched data: 0 bytes
File list size: 30
File list generation time: 0.152 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 85826493
Total bytes received: 38

sent 85826493 bytes received 38 bytes 432375.47 bytes/sec
total size is 419430912 speedup is 4.89

Second rsync cycle

Number of files: 1
Number of files transferred: 1
Total file size: 419430912 bytes
Total transferred file size: 419430912 bytes
Literal data: 1864192 bytes
Matched data: 417566720 bytes
File list size: 30
File list generation time: 0.135 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 344586
Total bytes received: 143405

sent 344586 bytes received 143405 bytes 4337.70 bytes/sec
total size is 419430912 speedup is 859.51

Looking forward to getting sometime to play with this on ESX 3.0, I am lobbying hard for the 48hr day 🙂

Mounting .vmdk files in Windows

Ever feel the need to mount a Windows or Linux .vmdk on windows, I have. Here are a couple of utilities that make it possible to open Linux and Windows .vmdks on Windows in read-only mode or read/write mode.

Virtual Disk Driver for Windows (http://chitchat.at.infoseek.co.jp/vmware/vdk.html)
Ext2IFS – Ext2/3 file system driver for windows. Required to mount Linux Virtual disks on windows (http://www.fs-driver.org/index.html)

Collecting system information with SYDI

I have been using SYDI for quite a while now. The 2.0 release of sydi-server has added some nice features, the ability to generate Word output from .xml files is huge. The only think that I found to be annoying is that the sydi-transform.vbs script does allow the input of an entire directory.

cscript.exe sydi-transform.vbs -xServer.xml -sServerhtml.xsl -oServer.html

This can make the transformation process from .xml to .html fairly labor intensive for a large environment. This little script solves that problem.

#!/usr/bin/perl -w

# syntax: perl transform.pl dir_where_xml_files_live
# e.g. – perl transform.pl ./xmlfiles
# Will walk the file system and output filename.xml.html for each .xml file
#
# Edit these variables to match the location on your system
$pathtotransform=”g:/sydi/tools/sydi-transform.vbs”;
$pathtoxsl=”g:/sydi/xml/serverhtml.xsl”;

#DO NOT EDIT BELOW THIS LINE
$dirtoget=”$ARGV[0]”;

opendir(IMD, $dirtoget) || die(“Cannot open directory”);
@thefiles= readdir(IMD);
closedir(IMD);

foreach $f (@thefiles)
{
unless ( ($f eq “.”) || ($f eq “..”) )
{
system “cscript $pathtotransform -x$f -s$pathtoxsl -o$dirtoget\/$f.html”;
}
}

MSSQL 2000 install on Windows 2003

I had the need to load a MSSQL 2000 instance on a Windows 2003 box when I encountered the following error “Unable to validate product key”. This was the MSDN SQL 2000 CD that I had used on numerous W2K and WXP boxes so I had no idea what was going on. After some researching I found the fix.

A simple registry change. The hack can be found here.

I hope this helps someone else out.

-RJB