High-level guidelines for building a Virtual Appliance:

• Start with a new VM
  • Use SCSI VMs for portability and performance
  • Split into 2GB files for size and portability
  • Allocate minimum required mem to the vm (clearly document this)
  • Disable snapshots, do not include snapshot files
  • Remove unused hardware devices
    • i.e - USB, audio, floppy, etc…
    • start devices such as floppy, cd, etc… disconnected
  • Disable “Shared Folders”
  • Avoid the use of serial and parallel ports or other specialty features
  • Chose the proper network type - Bridged is the the default but there may be a situation to use NAT
• Install “JEOS” - Just Enough OS
  • Select the Linux distribution of your choice (pick a supported guest to make life easier on yourself - Note: if you use RHEL or SLED you will need to get permission to distribute)
    • Minimize he footprint
      • only install and run necessary services
      • only open necessary ports
      • more secure and less to patch
    • Install VMware Tools
      • Improved performance, optimized drivers for virtual hardware, etc…
      • Hooks to management tools
      • Fully redistribute able inside Linux guests
    • Include or enable users to add a second virtual disk
      • put config information: user data, logs files, etc… on second disk
      • easier to update and backup the virtual appliance
• Install the applications stack
• Configure the appliance for first boo
  • Console experience
    • Accept EULA
    • “Zero-Configuration”
    • Present management URL
  • Web management interface
    • Configure networking
    • Configure security
  • Monitor and manage performance of the solution
  • Test everything! Test first boot/complete execution on different machines
• Value-Add
  • Expose logging from underlying services
  • Support SNMP
  • Provide audit hooks
  • Allow users to backup configurations and/or restore to factory default settings
• Package the virtual appliance
  • Copy appliance to new directory
  • Remove unecessary files
    • log files, nvram file, etc..
  • Add your “Getting Started Guide” or “ReadMe”
  • Compress the entire directory
    • 7-Zip or RAR work very well
    • Use cross-platform compression technology
  • Create a web page to host your virtual appliance download
  • Certify your virtual appliance
  • Create a listing on http://vam.vmware.com
• Provide a patch mechanism for your virtual appliance
  • leverage the default packaging technology of the OS (i.e. .deb or .rpm) or build your own packaging and update technology
  • Support direct online updates from your servers
  • support offline patching

Available tools, frameworks and services:

• VMware Virtual Appliance Development Kit
  • under development - used as part of Ace 2.0
• rPath Builder
  • Web-based virtual appliance development tool
  • Includes rPath Linux distro
  • Includes its own patching solution
• VirtualAppliances.net
  • Service provider
  • Developing their own framework
• jump box
  • Service provider

I recently listened to a podcast that mentioned my “Cisco WAAS performance benchmarks” and “PSTs on network file shares…” posts.  Have a listen the mention happens about 31 minutes into the podcast.  I also posted a comment to provide some rudimentary clarification on a few items discussed in the podcast.  Guys thanks for the mention!

It’s that time of year again… EMC World 2007 kicked off yesterday with a keynote by Joe Tucci.  How the industry has changed over the past 6 years.  Discussions which 6 years ago centered around back-end infrastructure, performance, SLAs, etc… all supporting B2B applications and corporate infrastructure have been replaced by discussions primarily focused on consumer applications such as YouTube, MySpace, Facebook, etc…  I am looking forward to Mark Lewis’ keynote tomorrow which he entitles “living in a 2.0 world”.  The need has always been created bu the consumer space but never in history has the consumer had the level of visibility into the back-end infrastructure that they do today.  Users today immediately know when site like YouTube and Facebook are offline, when their cell phones gps is down, when IM is offline, when they can’t download music from iTunes and the list goes on and on.  Social networking sites have proven that they will become if not already a primary communication medium, they are the new brick and mortar businesses.  Welcome to the 2.0 world.

A final thought, last night I had dinner at Charley’s Steak House, my good friend at recoverymonkey.org exercised his poetic culinary license last night.  I suggest checking it out.

Recent performance benchmarks gathered from an actual WAAS PoC deployment.  Feel free to Email me rjb@mti.com for more details.  Unfortunately I can not publish everything due to the need to protect the innocent.  All of the details below (IP addresses, locations, etc…) have been changed but the performance benchmark data is actual data collected from and actual deployment.

Deployment details:

• Core:
• Core WAE: 172.16.1.10
• Central Manager: 172.16.1.20
• CIFS server: 172.16.1.20
• CIFS share created \\172.16.1.30\waas located at d:\waas
• Default gateway: 172.16.1.254
• Net mask: 255.255.255.0
• DNS server: 172.16.1.3
• Edge:
• Edge WAE: 192.168.1.10
• Client machine: 192.168.1.100
• CIFS share \\172.16.1.30\waas mounted on z:

Topology:

Initial observations:

Some larger problematic .xls files to the WAAS accelerated file share and asked the Austin user to test the performance. The initial impression and observations were the following:

• Opening and saving a large 4MB PowerPoint:
• What took 50 seconds to open now takes around 2-3
• 13MB adobe file open 2 sec, took 2min pre-WAAS
• 35 seconds for 4.3MB excel, now 2s
• 2s for 8MB excel file

Benchmarks:

Two benchmarks were completed. The Cisco MS office benchmark and a file copy and create benchmark.

Office benchmarks:

File copy and creation benchmarks:

Quick little script to automate an FTP test:

#!/usr/local/bin/perl
###
#waastest.pl
#ftp test procedure automation for waas
#v.099b - 2007.05.05 - Rich Bocchinfuso
#@COPYLEFT 2007 - MTI CTA Team - ALL WRONGS RESERVED
#requires: creatfil.exe ncftpget.exe ncftpput.exe
###

if ($ARGV[0] eq “-help”) {
&usage;
}
if ($ARGV[0] eq “-o”) {
&cmdparms;
}
else{
&menu;
}

sub menu {
&copyleft;
print “Enter target ftp host/ip (e.g. - 192.168.1.10): “
$target = <>
chomp $target;
print “Enter output filename: “
$output = <>
chomp $output;
print “Enter test description: “
$desc = <>
chomp $desc;
print “Enter starting file size in megabytes: “
$startmb = <>
chomp $startmb;
print “Enter ending file size in megabytes: “
$endmb = <>
chomp $endmb;
print “Enter incraments in megabytes: “
$incmb = <>
chomp $incmb;
print “Enter test type [p]ut, [g]et, [a]ll: “
$test = <>
chomp $test;
print “Enter username [e.g. - foo\@bar.com]: “
$user = <>
chomp $user;
print “Enter password: “
$password = <>
chomp $password;

&copyleft;
&beginstamp;
&convert;
if ($test eq “p” || $test eq “put”) {
&put;
}
if ($test eq “g” || $test eq “get”) {
&get;
}
if ($test eq “a” || $test eq “all”) {
&put;
&get
}
else{
&error;
}
&endstamp;
}

sub cmdparms {
$target=$ARGV[2];
$user=$ARGV[3];
$password=$ARGV[4];
$startmb=$ARGV[5];
$endmb=$ARGV[6];
$incmb=$ARGV[7];
$output=$ARGV[8];
$desc=$ARGV[9];
&copyleft;
&beginstamp;
&convert;
if ($ARGV[1] eq “-p”) {
&put;
}
if ($ARGV[1] eq “-g”) {
&get;
}
if ($ARGV[1] eq “-a”) {
&put;
&get;
}
&endstamp;
}

sub put {
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
print LOG “\n…initiating ftp PUT test…\n”
close (LOG);
for ($count = $start; $count <= $end; $count+=$inc) {
print “…creating $count byte file named file_$count.dat…\n”
system “creatfil file_$count.dat $count >> $output”
print “…putting $count byte file named file_$count.dat to $target…\n”
system “ncftpput -v -u $user -p $password $target . file_$count.dat 2>> $output”
print “…removing $count byte seed file named file_$count.dat…\n”
system “del file_$count.dat >> $output”
}
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
print LOG “…ftp PUT test complete…\n”
close (LOG);
}

sub get {
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
print LOG “\n…initiating ftp GET test…\n”
close (LOG);
for ($count = $start; $count <= $end; $count+=$inc) {
print “…getting $count byte file file_$count.dat from $target…\n”
system “ncftpget -v -u $user -p $password $target . file_$count.dat 2>> $output”
print “…removing $count byte file named file_$count.dat…\n”
system “del file_$count.dat >> $output”
}
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
print LOG “…ftp GET test complete…\n”
close (LOG);
}

sub convert {
# Convert MB to bytes
$start = $startmb*1024;
$end = $endmb *1024;
$inc = $incmb*1024;
}

sub beginstamp {
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
$datetime = localtime();
printf “WASS benchmark initiatied $datetime\n\n”
printf LOG “WASS benchmark initiatied $datetime\n”
print LOG “\nDescription: $desc\n\n”
close (LOG);
print “…testing and logging latency and packet loss…may take a minute…\n\n”
system “ping -n 10 -l 1024 $target >> $output”
}

sub endstamp {
open(LOG,”>>$output”) || die(”Can’t open output file: $!”);
$datetime = localtime();
printf “\nWASS benchmark completed $datetime\n\n”
printf LOG “\nWASS benchmark completed $datetime\n\n”
close (LOG);
}

sub copyleft {
system “cls”
print “\nWAAS FTP benchmark and test utility\n”
print “v.099 - 2007.05.05\n”
print “\@COPYLEFT 2007 - MTI CTA Team - ALL WRONGS RESERVED\n\n”
}

sub usage {
&copyleft;
print “\nUsage: waastest -help {…displays this help message}\n”
print “Usage: waastest {prompt based interface}\n\n”
print “Usage: waastest -o [command line parameters]\n”
print “waastest -o [-p | -g | -a] [target hostname/ip] [username] [password] [starting file size in MB] [ending file size in MB] [inrement by MB] [output log filename] [\"test description\"]\n”
print “-p … run put tests only\n”
print “-g … run get tests only\n”
print “-a … runs put and get tests\n”
exit;
}

sub error {
print “!error! !error!\n”
print “\”waastest -help\” - for usage instructions\n\n”
exit;
}

It seems as if one of the most debated topics related to VMware and I/O performance is the mystery sounding the relative performance characteristics of vmfs volumes and rdm (Raw Device Mode) volumes.

Admittedly it is difficult to argue with the flexibility and operational benefits of vmfs volumes but I wanted to measure the characteristics of each approach and provide some documentation that could be leveraged when making the decision to use vmfs or rdm.? By no means are these test concluded but I thought as a gathered the data I would blog it so it could be used prior to me completing the whitepaper which all these tests will be part of.

Benchmark configuration:
The benchmarks contained in this document were performed in a lab environment with the following configuration:

• Physical Server:? Dell dual CPU 2850 w/ 4 GB RAM
  • Windows 2003 SP2 Virtual Machine
  • Single 2.99 Ghz CPU
  • 256 MB RAM (RAM configured this low to remove the effects of kernel file system caching)
• Disk array
  • EMC CLARiiON CX500
  • Dedicated RAID 1 Device
  • 2 LUNs Created on the RAID 1 Storage Group
  • Two dedicated 10 GB file systems
    • c:\benchmark\vmfs
      • 10 GB .vmdk created and vmfs and NTFS file system created
    • c:\benchmark\rdm
      • 10 GB rdm volume mapped to VM and NTFS file system created?

Benchmark tools:
Benchmark tests thus far were run using?two popular?disk and file system benchmarking tools.

• IOzone (http://www.iozone.org/)
  • Iozone -Rab output.wks
• HDtune (http://www.hdtune.com/)

IOzone Benchmarks:

HDtune benchmarks:

HD Tune: VMware Virtual disk Benchmark
Transfer Rate Minimum : 54.1 MB/sec
Transfer Rate Maximum : 543.7 MB/sec
Transfer Rate Average : 476.4 MB/sec
Access Time : 0.4 ms
Burst Rate : 83.3 MB/sec
CPU Usage : 36.9%

HD Tune: DGC RAID 1 Benchmark
Transfer Rate Minimum : 57.1 MB/sec
Transfer Rate Maximum : 65.3 MB/sec
Transfer Rate Average : 62.4 MB/sec
Access Time : 5.4 ms
Burst Rate : 83.9 MB/sec
CPU Usage : 13.8%

One thing that is very obvious is that VMFS makes extensive use of system/kernel cache.? This is most obvious in the HDtune benchmarks.? The increased CPU utilization is a bit of a concern, most likely due to the caching overhead.? I am going to test small block random writes while monitoring CPU overhead, my gut tells me that small block random writes to a VMFS volume will tax the CPU.? More to come….

Not a good idea. Someone I know implemented Cisco WAAS and is experiencing performance problems. Specifically the CPU on the WAE device is spiked to 100% utilized. They are attempting to per-position 80GB of data including large 1-2GB PSTs to the WAE (Wide Area Application Engine). In my opinion this is a recipe for disaster. First off Microsoft does not support PST access via network shares, and once you read why you will understand why caching PSTs is an even worse idea.

http://logs.technet.com/askperf/archive/2007/01/21/network-stored-pst-files-don-t-do-it.aspx

http://support.microsoft.com/kb/297019

I ran my own little test by moving a PST to a share, starting wireshark (ethereal) and watching the activity. Accessing the PST over the network makes CIFS alone look very efficient.

I have just been so darn busy that the blog has suffered, I have about 15 half-started blogs that need to be completed and posted. Today I plan to try to work on a number of them so hope fully I can get some of them posted. Next week I will be at EMC World, the sessions provide a great blogging opportunity so I should be fairly active next week as well. Keep the faith!

Admittedly I still sparingly use Internet Explorer but I found a cool little plug-in that allows me to now run windows updates from a non-IE browser, further limiting my need for IE. Check out it out at http://windowsupdate.62nds.com.

The second utility removes the Windows Genuine Advantage (WGA) from your computer, another fine piece of Microshaft nuisanceware. The original location of the “removewga” was here http://www.firewallleaktester.com/removewga.htm but it appears Microsoft forced the developer to disable the links. You can still grab the utility here http://www.softpedia.com/get/Tweak/Uninstallers/RemoveWGA.shtml .