NetWorker Daily Status Report

Unhappy with the overly verbose native EMC NetWorker savegroup completion reporting and the limited formatting options available in the native html output provided by gstclreport I decided to quickly create a powershell script to produce a a better looking daily status email.  Below is ver 0.1 of the script.

Note:  In the powershell source included below all the variables are in the source for simplicity purposes. In my implementation a large number of the defined variables are used across a wider library of powershell scripts so the variables are actually contained in a separate variable file which I dot source to set the variables in the global scope.

https://gist.github.com/rbocchinfuso/06ad8a9aef58598f3660

The script delivers via email a message which contains a formatted html report in the body of the email as well as a csv attachment containing the raw data.  The following is a sample email:nwreport1

 

EMC Forum 2014 New York – It’s all about the cloud and SDW!!

This years EMC Forum 2014 New York is approaching quickly, October 8th is just around the corner and I am really excited!!!

Over the years we (FusionStorm NYC) have typically prepped a demo for EMC Forum and rolled a 20U travel rack complete with networking, servers, storage arrays, and whatever else we needed for the demo to EMC Forum.  In the past we’ve done topics like WAN Optimization, VMware SRM, VMware vCOps, and last year XtremSW.  As a techie it’s always been cool to show the flashing lights, how things are cabled, etc… but this year it’s all about the cloud, commodity compute and and SDW (Software Defined Whatever) and elasticity which is why there will be no 20U travel rack, nothing more than a laptop and an ethernet cable that will connect to a ScaleIO 1.3 AWS (Amazon Web Services) implementation.  The base configuration that I have built in AWS specifically for EMC Forum looks like this:

  • 10 x SDS (ScaleIO Data Server) Nodes in AWS (SLES 11.3)
    • Each node has 1 x 100 GB EBS SSD attached
  • 1 x SDC (ScaleIO Data Client) Node in AWS (Windows 2008 R2)
    • Using IOmeter and vdbench on SDC to generate workload
  • Single Protection Domain:  awspdomain01
  • Single Pool:  awspool01
  • 40GB awsvol01 volume mapped to Windows SDC

image

Terminology:

  • Meta Data Manager (MDM) – Configures and monitors the ScaleIO system. The MDM can be configured in a redundant Cluster Mode with three members on three servers, or in a Single Mode on a single server.
  • ScaleIO Data Server (SDS) – Manages the capacity of a single server and acts as a backend for data access. The SDS is installed on all servers that contribute storage devices to the ScaleIO system. These devices are accessed through the SDS.
  • ScaleIO Data Client (SDC) – A lightweight device driver that exposes ScaleIO volumes as block devices to the application residing on the same server on which the SDC is installed.

New Features in ScaleIO v1.30:
ScaleIO v1.30 introduces several new features, listed below. In addition, it includes internal enhancements that increase the performance, capacity usage, stability, and other storage aspects.

Thin provisioning:
In v1.30, you can create volumes with thin provisioning. In addition to the on-demand nature of thin provisioning, this also yields much quicker setup and startup times.
Fault Sets: You can define a Fault Set, a group of ScaleIO Data Servers (SDSs) that are likely to go down together (For example if they are powered in the same rack), thus ensuring that ScaleIO mirroring will take place outside of this fault set.
Enhanced RAM read cache: This feature enables read caching using the SDS server RAM.
Installation, deployment, and configuration automation:  Installation, deployment, and configuration has been automated and streamlined for both physical and virtual environments. The install.py installation from previous versions is no longer supported.
This is a significant improvement that dramatically improves the installation and operational management.

image

image

image

VMware management enhancement: A VMware, web-based plug-in communicates with the Metadata Manager (MDM) and the vSphere server to enable deployment and configuration directly from within the VMware environment.

GUI enhancement: The GUI has been enhanced dramatically. In addition to monitoring, you can use the GUI to configure the backend storage elements of ScaleIO.

GUI enhancements are big!!

clip_image002

clip_image002[5]

Active Management (huge enhancement over the v1.2 GUI):

imageimage

Smile Of course you can continue to use the CLI Smile:

  • Add Protection Domain and Pool:
    • scli –mdm_ip 172.31.43.177 –add_protection_domain –protection_domain_name awspdomain01
    • scli –mdm_ip 172.31.43.177 –add_storage_pool –protection_domain_name awspdomain01 –storage_pool_name awspool01
  • Add SDS Nodes:
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.177 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws177sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.178 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws178sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.179 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws179sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.180 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws180sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.181 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws181sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.182 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws182sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.183 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws183sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.184 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws184sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.185 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws185sds
    • scli –mdm_ip 172.31.43.178 –add_sds –sds_ip 172.31.43.186 –protection_domain_name awspdomain01 –device_path /dev/xvdf –storage_pool_name awspool01 –sds_name aws186sds
  • Add Volume:
    • scli –mdm_ip 172.31.43.178 –add_volume –protection_domain_name awspdomain01 –storage_pool_name awspool01 –size 40 –volume_name awsvol01
  • Add SDC:
    • Add Windows SDC (would look different on Linux / Unix):
      • C:\Program Files\EMC\ScaleIO\sdc\bin>drv_cfg.exe –add_mdm –ip 172.31.43.177,172.31.43.178
        Calling kernel module to connect to MDM (172.31.43.177,172.31.43.178)
      • ip-172-31-43-178:~ # scli –query_all_sdc
        Query all SDC returned 1 SDC nodes.
        SDC ID: dea8a08300000000 Name: N/A IP: 172.31.43.7 State: Connected GUID: 363770AA-F7A2-0845-8473-158968C20EEF
        Read bandwidth:  0 IOPS 0 Bytes per-second
        Write bandwidth:  0 IOPS 0 Bytes per-second
  • Map Volume to SDC:
    • scli –mdm_ip 172.31.43.178 –map_volume_to_sdc –volume_name awsvol01 –sdc_ip 172.31.43.7

REST API:  A  Representational  State  Transfer  (REST)  API can be used to expose monitoring and provisioning via the REST interface.
OpenStack support:  ScaleIO includes a Cinder driver that interfaces with OpenStack, and presents volumes to OpenStack as block devices which are available for block storage. It also  includes an OpenStack Nova driver, for handling compute and instance volume-related operations.
Planned shutdown of a Protection Domain:  You can simply and effectively shut down an entire Protection Domain, thus preventing an unnecessary rebuild/rebalance operation.
Role-based access control: A role-based access control mechanism has been introduced.
Operationally Planned shutdown of a Protection Domain is a big enhancement!!

IP roles: For each IP address associated with an SDS, you can define the communication role that the IP address will have: Internal—between SDSs and MDMs; External—between ScaleIO Data Clients (SDCs) and SDSs; or both. This allows you to define virtual subnets.
MDM— IP address configuring: You can assign up to eight IP addresses to primary, secondary, and tie-breaker MDM servers, thus enhancing MDM communication redundancy. In addition, you can configure a specific IP address that the MDM will use to communicate with the management clients. This enables you to configure a separate management network so you can run the GUI on an external system.

Note:  The above new features where taken from the Global Services Product Support Bulletin ScaleIO Software 1.30 (Access to this document likely requires a support.emc.com login).  I have played with most of the new features but not all of them, IMO v1.3 provides a major a leap forward in usability.

While big iron (traditional storage arrays) probably are not going away very soon the cool factor just doesn’t come close to SDW (Software Defined Whatever) so stop by the FusionStorm booth at EMC Forum and let’s really dig into some very cool stuff.

If you are really interesting in digging into to ScaleIO one-on-one please email me (rbocchinfuso@fusionstorm.com) or tweet me @rbocchinfuso and we can setup a time where we can focus, maybe a little more than will be possible at EMC Forum.

Looking forward to seeing at the FusionStorm booth at EMC Forum 2014 New York on October 8th.  If you haven’t registered for EMC Forum you should register now, Forum is only 12 days away.

Repurposing old EMC Centera nodes

This is the first in a multi-part series on repurposing old EMC equipment.  I recently acquired six  EMC Centera nodes, two of the nodes with 4x1TB SATA drives and four of the nodes with 4x500GB SATA drives so I started thinking what can I do with these Pentium based machines with 1 GB of RAM and a boat load of storage.  An idea hit me to create a NAS share leveraging a global file system to aggregate the capacity and performance across all the Centera nodes.  Seemingly simple there was a challenge here, most modern day global file systems like GlusterFS or GFS2 require a 64 bit processor architecture, the Centera nodes use 32 bit Pentium processors.  After spending a vacation day researching I identified two possible Global file systems as potential options, XtreemFS and FraunhoferFS (fhgfs).  I discovered fhgfs first and it looked pretty interesting, a fairly traditional Global File System consisting of metadata nodes and storage nodes (I came across this presentation which provides a good overview of the FraunhoferFS.  While fhgfs provided the basics of what I was looking for the missing link was how I was going to protect the data, fhgfs for the most part relied on hardware RAID for node survivability, because the Centera nodes are built to run EMC’s CentraStar an OS which leverages RAIN (Redundant Array of Independent Nodes) no redundancy is built in at the node level.  EMC acquired Centera and CentraStar from a Belgian company named FilePool in 2002.  As I thought trough some possible workarounds I stumbled across XtreemFS an interesting object based global file system, what was most interesting was the ability to replicate objects for redundancy.  At this point I decided to attempt to move forward with XtreemFS, my single node install went well, no issues  to really speak of as I moved towards the multi node configuration I was beginning to get core dumps when starting the daemons, is was at this point that I decided to give fhgfs a try, thinking that in phase 2 of the project I could layer on either rsync of drbd to protect the data (not there yet so not sure how well this theory will play out).  The fhgfs installed fairly easily and is up and running, the rest of this blog will walk you though the steps that I took to prepare the old Centera nodes, install Ubuntu and configure Ubuntu server, install an configure fhgfs.

Because the Centera nodes came out of a production environment they were wiped prior to leaving the production data center (as a side note DBAN booted form a USB key was used to perform the wipe of each node).  So with no data on the four Centera node internal drives the first step was to install a base OS on each node.  Rather than use a USB CD-ROM (only had one) I decided to build a unattended PXE boot install.

Phase 1:  Basic Environment Prep:

Step 1:  Build PXE Server (because this is not a blog on how to build a PXE server I suggest doing some reading).  The following two links should be very helpful:  https://help.ubuntu.com/community/PXEInstallServer, https://help.ubuntu.com/community/PXEInstallMultiDistro. I built my PXE boot server on Ubuntu 12.04 server but the process in pretty much as documented in the above two links.  You can also Google “ubuntu pxe boot server”.

Note:  One key is to be sure to install Apache and copy your Ubuntu distro to a http accessible path.  This is important when creating your kickstart configuration file (ks.cfg) so you can perform a completely automated install.  My ks.cfg file.

Step 1A:  Enter BIOS on each Centera and reset to factory defaults, make sure that each node has PXE boot enabled on the NICs.

Note:  I noticed on some of the nodes that the hardware NIC enumeration does not match Ubuntu’s ETH interface enumeration (i.e. – On 500GB nodes ETH0 is NIC2) just pay attention to this as it could cause some issues, if you have the ports just cable all the NICs to make life a little easier.

Step 1B:  Boot servers and watch the magic of PXE.  Ten minutes from now all the servers will be booted and at the “kickstart login:” prompt.

Step 2:  Change hostname and install openssh-server on each node.  Login to each node, vi /etc/hostname and update to “nodeX”, also execute apttidude install openssh-server (openssh-server will be installed form the PXE server repo, I only do this no so I can do the rest of the work remotely instead of sitting at the console).

Step 3:  After Step 2 is complete reboot the node.

Step 4:  Update /etc/apt/sources.list

Step 4 Alternative:  I didn’t have the patience to wait for the repo to mirror but you may want to do this and copy you sources.list.orig back to sources.list at a later date.

Note:  If you need to generate a a sources.list file with the appropriate repos check out http://repogen.simplylinux.ch/

Step 4A:  Add the FHGFS repo to the /etc/sources.list file

deb http://www.fhgfs.com/release/fhgfs_2011.04 deb6 non-free

Step 4A:  Once you update the /etc/sources.list file run an apt-get update to update the repo, followed by an apt-get upgrade to upgrade distro to latest revision.

Step 5:  Install lvm2, default-jre, fhgfs-admon packages

aptitude install lvm2
aptitude install default-jre
aptitude install fhgfs-admon

Phase 2:  Preparing Storage on each node:

Because the Centera nodes use JBOD drives I wanted to get the highest performance by striping within the node (horizontally) and across the nodes (vertically).  This section focuses on the configuration of horizontal striping on each node.

Note:  I probably could have taken a more elegant approach here, like boot for USB key and use the entire capacity of the four internal disks for data but this was a PoC so didn’t get overly focused on this.  Some of the workarounds I use below could have probably been avoided.

  1. Partition the individual node disks
    1. Run fdisk –l (will let you see all disks and partitions)
    2. For devices that do not have partitions create a primary partition on each disk with fdisk (in my case /dev/sda1 contained my node OS, /dev/sda6 was free, /dev/sdb, /dev/sdc and /dev/sdd had no partition table so I created a primary partition dev/sdb1, /dev/sdc1 and /dev/sdd1)
  2. Create LVM Physical Volumes (Note: If you haven’t realized it yet /dev/sda6 will be a little smaller than the other devices, this will be important later.)
      1. pvcreate /dev/sda6
      2. pvcreate /dev/sdb1
      3. pvcreate /dev/sdc1
      4. pvcreate /dev/sdd1
  3. Create a Volume Group that contains the above physical volumes
    1. vgcreate fhgfs_vg /dev/sda6 /dev/sdb1 /dev/sdc1 /dev/sdd1
    2. vgdisplay (make sure the VG was created)
  4. Create Logical Volume
    1. lvcreate -i4 -I4 -l90%FREE -nfhgfs_lvol fhgfs_vg –test
      1. Above command runs a test, notice the –I90% flag, this says to only use 90% of each physical volume.  Because this is a stripe and the available extents differ on /dev/sda6 we need to equalize the extents by consuming on 90% of the available exents.
    2. lvcreate -i4 -I4 -l90%FREE -nfhgfs_lvol fhgfs_vg
      1. Create the logical volume
    3. lvdisplay (verify that the lvol was created)
    4. Note:  The above commands performed on a node with 1TB drives, I also have nodes with 500GB drives in the same fhgfs cluster.  Depending on the the drive size in the nodes you will need to make adjustments so that the extents are equalized across the physical volumes.  As an example on the nodes with the 500GB drives the lvcreate commands looks like this lvcreate -i4 -I4 -l83%FREE -nfhgfs_lvol fhgfs_vg.
  5. Make a file system on the logical volume
    1. lvcreate -i4 -I4 -l83%FREE -nfhgfs_lvol fhgfs_vg
  6. Mount newly created file system and create relevant directories
    1. mkdir /data
    2. mount /dev/fhgfs_vg/fhgfs_lvol /data
    3. mkdir /data/fhgfs
    4. mkdir /data/fhgfs/meta
    5. mkdir /data/fhgfs/storage
    6. mkdir /data/fhgfs/mgmtd
  7. Add file system mount to fstab
    1. echo “/dev/fhgfs_vg/fhgfs_lvol     /data     ext4     errors=remount-ro     0     1” >> /etc/fstab

Note:  This is not a LVM tutorial, for more detail Google “Linux LVM”

Enable password-less ssh login (based on a public/private key pair) on all nodes

  1. On node that will be used for management run ssh-keygen (in my environment this is fhgfs-node01-r5)
    1. Note:  I have a six node fhgfs cluster fhgfs-node01-r5 to fhgfs-node06-r5
  2. Copy the ssh key to all other nodes.  From fhgfs-node01-r5 run the following commands:
    1. cat ~root/.ssh/id_dsa.pub | ssh root@fhgfs-node02-r5 ‘cat >> .ssh/authorized_keys’
    2. cat ~root/.ssh/id_dsa.pub | ssh root@fhgfs-node03-r5 ‘cat >> .ssh/authorized_keys’
    3. cat ~root/.ssh/id_dsa.pub | ssh root@fhgfs-node04-r5 ‘cat >> .ssh/authorized_keys’
    4. cat ~root/.ssh/id_dsa.pub | ssh root@fhgfs-node05-r5 ‘cat >> .ssh/authorized_keys’
    5. cat ~root/.ssh/id_dsa.pub | ssh root@fhgfs-node06-r5 ‘cat >> .ssh/authorized_keys’
  3. Note:  for more info Google “ssh with keys”

Configure FraunhoferFS (how can you not love that name)

  1. Launch the fhgfs-admon-gui
    1. I do this using Cygwin-X on my desktop, sshing to the fhgfs-node01-r5 node, exporting the DISPLAY back to my desktop and then launch the fhgfs-admon-gui.  If you don’t want to install Cygwin-X Xmingis a good alternative.
      1. java -jar /opt/fhgfs/fhgfs-admon-gui/fhgfs-admon-gui.jar
    2. Note:  This is not a detailed fhgfd install guide, reference the install guide for more detail http://www.fhgfs.com/wiki/wikka.php?wakka=InstallationSetupGuide
  2. Adding Metadata servers, Storage servers, Clients
    1. SNAGHTML12587da
  3. Create basic configuration
    1. SNAGHTML12f1d17
  4. Start Services
    1. SNAGHTML13277e3
  5. There are also a number of CLI command that can be used
    1. image
    2. e.g. – fhgfs-check-servers
      1. image
  6. If all works well a “df –h”yield the following
    1. image
    2. Note the /mnt/fhgfs mount point (pretty cool)

Creating a CIFS/NFS share

  1. Depending on how you did the install of you base Ubuntu system you likely need to load the Samba and NFS packages (Note:  I only loaded these on my node01 and node02 nodes, using these nodes as my CIFS and NFS servers respectively)
    1. aptitude install nfs-server
    2. aptitude install samba
  2. Configure Samba and/or NFS shares from /mnt/fhgfs
    1. There are lot’s or ways to do this, this is not a blog on NFS or Samba so refer to the following two links for more information:
      1. NFS:  https://help.ubuntu.com/community/SettingUpNFSHowTo
      2. Samba/CIFS:  http://www.samba.org/
    2. As a side note I like to load Webmin on the for easy web bases administration of all the nodes, as well as NFS and Samba
      1. wget http://downloads.sourceforge.net/project/webadmin/webmin/1.590/webmin_1.590_all.deb?r=http%3A%2F%2Fwww.webmin.com%2F&ts=1345243049&use_mirror=voxel
      2. Then use dpkg –i webmin_1.590_all.deb to install
      3. image

Side note:  Sometime when installing a debian package using dpkg you will have unsatisfied dependencies.  To solve this problem just follow the following steps:

  1. dpkg –i webmin_1.590_all.deb
  2. apt-get -f –force-yes –yes install

Performance testing, replicating, etc…

Once I finished the install it was time to play a little.  From a windows client I mapped to the the share that I created from the fhgfs-node01-r5 and started running some I/O to the FraunhoferFS….. I stared benchmarking using with IOzone, my goal is to compare and contrast my FraunhoferFS and NAS performance with other NAS products like NAS4Free, OpenFiler, etc… I also plan to do some testing with Unison, rsync and drdb for replication.

This is a long post so I decided to create a separate post for performance and replication.  To wet your appetite here are some the early numbers from the FhGFS NAS testing.

image

image

image

image

Created the above output quickly, In my follow-up performance post I will document the test bed, publish all the test variants and platform comparisons.  Stay tuned…

Avamar sizing brain dump

Avamar DS18 = Utility Node + Spare Node + 16 Active Data Nodes

For a 3.3. TB Gen-3 Grid

  • Raw Capacity ~102 TB
  • Spare Node ~6 TB
  • RAID5 ~15 TB
  • Checkpoint  / GC ~28 TB
  • RAIN ~3 TB
  • Available for Active Backups ~49 TB

RAID Configuration:

  • RAID 1 for 3.3 TB node
  • RAID 5 for 2 TB nodes
  • RAID 1 for 1 TB nodes

How to calculate the required capacity:

  • Seed (Initial backups)
    Daily Change * Rentention in Days
    +RAIN = GSAN Utilization

 

  • Need min available space for 4 checkpoints
  • 3 checkpoints maintained by default

Data Gathering

Note:  Agent only vs. data store depends on the desired RPO

  • xfer_rate = Gb/hr * .70
  • date_size = total of the data set to be backed up
  • restore_time = data_size x .65 / xfer_rate

If RTO < restore_rate then data store else agent only

Always use 3.3 TB nodes when configuring unless additional nodes are required to increase the ingestion rate.

Use the default de-dupe rate unless a POC or assessment has been performed.

Sizing Considerations:

  • Data Types
    • File Systems
    • Databases
    • Large Clients > 2 TB
    • Dense File Systems (excluding EMC Celerra and NetApp)
  • Organic Growth
  • RTO
  • Replication Window
  • Maintenance Window

Non-RAIN node must be replicated this includes single node Avamar deployments and 1×2 (1 utility node and 2 data store nodes – this is non-RAIN config) configurations.

**** Remember this: As a general rule it seems that transactional databases are better suited to be backed up to Data Domain and NOT with the Avamar as the hashing of databases is generally very slow.

VMware (specifically using the VMware Storage APIs) and CIFS are well suited for Avamar

Data save rates:

  • 100 – 150 GB/hr per avtar stream on latest server types
    • Note:  it is possible to launch multiple avtar daemons with some tweaking, but an out of the box install only launches a single avtar process.
  • VMguest backups can be slower (very scientific, these are backups that
  • Default assumption is chuck-compress-hash process runs at a rate of 100 GB/hr
    • This is the process that bottlenecks database backups (ideally is seems that the avtar stream rate should match the check-compress-hash process)

Scan rate:

  • ~ 1 million files per hour
    • 1 TB of file data will take about 1 hour to backup
    • 1 TB DB will take ~ 10 hours to complete

Performance:

  • 1 TB/hr per node in the grid (all file data)
  • 80% file (800 GB file) and 20% DB (200 GB DB) and the performance level drops off to .5 TB/hr
  • E.g. – DS18 perf will be ~ 15-16 TB/hr
  • Per node ingest rate ~ 8GB/hr

Restores:

Data Fetch Process

  • Per node assumption
    • Chuck size 24kb
    • each chunk is referenced in a hash index stripe
    • Speed:
      • 5 MB/s
      • 18 GB/hr (compressed chunk)
      • 25 GB/hr (rehydrated chunk)
  • E.g. – A DS18 will restore at a rate of .5 TB/hr

NDMP Sizing:

  • Size of the NDMP data set
  • Type of filer (Celerra or NetApp)
  • Number of volumes, file systems, qtrees
  • Size of volumes
  • Number of files per volume / file system

L-0 Fulls on happen once (we don’t want to size for them)

Size for L-1 incremental which will happen in perpetuity following the completion of the L-0 full.

  • Important L-1 sizing data
    • Number of files in the L-1 backup
    • Backup window

2 Accelerator Nodes

Config Max Files   Max Data   Max Streams  
  Celerra NetApp Celerra NetApp Celerra NetApp
6 GB 5 m 30 m 4-6 TB 4-6 TB 1-2 1-2
36 GB 40 m 60 m 8-12 TB 8-12 TB 4 4

NDMP throughput ~ 100 – 150 TB/hr

Assumed DeDupe Rates:

  • File data
    • Initial backup:  70% commonality (30% of the data is unique)
      • e.g. – 30% of 10 TB = 3 TB stored
    • Subsequent backups:  .3% daily change
      • e.g. – .3% of 10 TB = 30 GB stored per day
  • Database data
    • Initial backup:  35% commonality (65% of the data is unique)
      • e.g. – 65% of 10 TB = 6.5 TB stored
    • Subsequent backups:  4% daily change
      • e.g. – 4% of 10 TB = 400 GB stored per day

Tip:  Based on scan rate and the amount of data stored for DB backups you can see why Avamar may not be the best choice for DB backups.

NDMP Tips:

  • Avamar NDMP accelerator node should be on the same LAN segment as the filer and the same switch when possible
  • No Include/Exclude rules are supported
  • Able to run up to 4 NDMP backups simultaneously
    • most effective with large files
    • min of 4GB of memory per accelerator node per stream
    • 4 NDMP simultaneously scheduled as groups backups

Desktop / Laptop

Sizing:

  • Number of clients
  • Amount of data per client
    • user files
    • DB/PST files

DS18 can support ~ 5000 clients

Number of streams per node default is 18 (17 are usable, one should be reserved for restores).

That completes the brain dump.  Wish I had more but that is all for now.

EMC CX3-80 FC vs EMC CX4-120 EFD

This blog is a high level overview of some extensive testing conducted on the EMC (CLARiiON) CX3-80 with 15K RPM FC (fibre channel disk) and the EMC (CLARiiON) CX4-120 with EFD (Enterprise Flash Drives) formerly know as SSD (solid state disk).

Figure 1:  CX4-120 with EFD test configuration.

image

Figure 2:  CX3-80 with 15K RPM FC rest configuration.

image

Figure 3:  IOPs Comparison

image

Figure 4:  Response Time

image

Figure 5:  IOPs Per Drive

image

Notice that the CX3-80 15K FC drives are servicing ~ 250 IOPs per drive, this exceeds 180 IOPs per drive (the theoretical maximum for a 15K FC drive is 180 IOPs) this is due to write caching.  Note that cache is disabled for the CX4-120 EFD tests, this is important because high write I/O load can cause something known as a force cache flushes which can dramatically impact the overall performance of the array.  Because cache is disabled on EFD LUNs forced cache flushes are not a concern.

Table below provides a summary of the test configuration and findings:

Array CX3-80 CX4-120
Configuration (24) 15K FC Drives (7) EFD Drives
Cache Enabled Disabled
Footprint   ~42% drive footprint reduction
Sustained Random Read Performance   ~12x increase over 15K FC
Sustained Random Write Performance   ~5x increase over 15K FC

In summary, EFD is a game changing technology.  There is no doubt that for small block random read and write workloads (i.e. – Exchange, MS SQL, Oracle, etc…) EFD dramatically improves performance and reduces the risk of performance issues.

This post is intended to be an overview of the exhaustive testing that was performed.  I have results with a wide range of transfer sizes beyond the 2k and 4k results shown in this posts, I also have Jetstress results.  If you are interested in data that you don’t see in this post please Email me a rbocchinfuso@gmail.com.

Benchmarking De-Duplication and with Databases

In the interest of benchmarking de-duplication rates with databases I created a process to build a test database, load test records, dump the database and perform a de-dupe backup using EMC Avamar on the dump files.  The process I used is depicted in the flowchart below.

image

1.  Create a DB named testDB
2.  Create 5 DB dump target files – testDB_backup(1-5)
3.  Run the test which inserts 1000 random rows consisting of 5 random fields for each row.  Once the first insert is completed a dump is performed to testDB_backup1.  Once the dump is complete a de-dupe backup process is performed on the dump file.  This process is repeated 4 more times each time adding an additional 1000 rows to the database and dumping to a new testDB_backup (NOTE:  this dump includes existing DB records and the newly inserted rows) file and performing the de-dupe backup process.

Once the backup is completed a statistics file is generated showing the de-duplication (or commonality) ratios.  The output from this test is as follows:

image

You can see that each iteration of the backup shows an increase in the data set size with increasing commonality and de-dupe rations.  This test shows that with 100% random database data using a DB dump and de-dupe backup strategy can be a good solution for DB backup and archiving.