Avamar DS18 = Utility Node + Spare Node + 16 Active Data Nodes

For a 3.3. TB Gen-3 Grid

• Raw Capacity ~102 TB
• Spare Node ~6 TB
• RAID5 ~15 TB
• Checkpoint  / GC ~28 TB
• RAIN ~3 TB
• Available for Active Backups ~49 TB

RAID Configuration:

• RAID 1 for 3.3 TB node
• RAID 5 for 2 TB nodes
• RAID 1 for 1 TB nodes

How to calculate the required capacity:

• Seed (Initial backups)
  Daily Change * Rentention in Days
  +RAIN = GSAN Utilization

• Need min available space for 4 checkpoints
• 3 checkpoints maintained by default

Data Gathering

Note:  Agent only vs. data store depends on the desired RPO

• xfer_rate = Gb/hr * .70
• date_size = total of the data set to be backed up
• restore_time = data_size x .65 / xfer_rate

If RTO < restore_rate then data store else agent only

Always use 3.3 TB nodes when configuring unless additional nodes are required to increase the ingestion rate.

Use the default de-dupe rate unless a POC or assessment has been performed.

Sizing Considerations:

• Data Types
• File Systems
• Databases
• Large Clients > 2 TB
• Dense File Systems (excluding EMC Celerra and NetApp)
• Organic Growth
• RTO
• Replication Window
• Maintenance Window

Non-RAIN node must be replicated this includes single node Avamar deployments and 1×2 (1 utility node and 2 data store nodes – this is non-RAIN config) configurations.

**** Remember this: As a general rule it seems that transactional databases are better suited to be backed up to Data Domain and NOT with the Avamar as the hashing of databases is generally very slow.

VMware (specifically using the VMware Storage APIs) and CIFS are well suited for Avamar

Data save rates:

• 100 – 150 GB/hr per avtar stream on latest server types
• Note:  it is possible to launch multiple avtar daemons with some tweaking, but an out of the box install only launches a single avtar process.
• VMguest backups can be slower (very scientific, these are backups that
• Default assumption is chuck-compress-hash process runs at a rate of 100 GB/hr
• This is the process that bottlenecks database backups (ideally is seems that the avtar stream rate should match the check-compress-hash process)

Scan rate:

• ~ 1 million files per hour
• 1 TB of file data will take about 1 hour to backup
• 1 TB DB will take ~ 10 hours to complete

Performance:

• 1 TB/hr per node in the grid (all file data)
• 80% file (800 GB file) and 20% DB (200 GB DB) and the performance level drops off to .5 TB/hr
• E.g. – DS18 perf will be ~ 15-16 TB/hr
• Per node ingest rate ~ 8GB/hr

Restores:

Data Fetch Process

• Per node assumption
• Chuck size 24kb
• each chunk is referenced in a hash index stripe
• Speed:
• 5 MB/s
• 18 GB/hr (compressed chunk)
• 25 GB/hr (rehydrated chunk)
• E.g. – A DS18 will restore at a rate of .5 TB/hr

NDMP Sizing:

• Size of the NDMP data set
• Type of filer (Celerra or NetApp)
• Number of volumes, file systems, qtrees
• Size of volumes
• Number of files per volume / file system

L-0 Fulls on happen once (we don’t want to size for them)

Size for L-1 incremental which will happen in perpetuity following the completion of the L-0 full.

• Important L-1 sizing data
• Number of files in the L-1 backup
• Backup window

2 Accelerator Nodes

NDMP throughput ~ 100 – 150 TB/hr

Assumed DeDupe Rates:

• File data
• Initial backup:  70% commonality (30% of the data is unique)
• e.g. – 30% of 10 TB = 3 TB stored
• Subsequent backups:  .3% daily change
• e.g. – .3% of 10 TB = 30 GB stored per day
• Database data
• Initial backup:  35% commonality (65% of the data is unique)
• e.g. – 65% of 10 TB = 6.5 TB stored
• Subsequent backups:  4% daily change
• e.g. – 4% of 10 TB = 400 GB stored per day

Tip:  Based on scan rate and the amount of data stored for DB backups you can see why Avamar may not be the best choice for DB backups.

NDMP Tips:

• Avamar NDMP accelerator node should be on the same LAN segment as the filer and the same switch when possible
• No Include/Exclude rules are supported
• Able to run up to 4 NDMP backups simultaneously
• most effective with large files
• min of 4GB of memory per accelerator node per stream
• 4 NDMP simultaneously scheduled as groups backups

Desktop / Laptop

Sizing:

• Number of clients
• Amount of data per client
• user files
• DB/PST files

DS18 can support ~ 5000 clients

Number of streams per node default is 18 (17 are usable, one should be reserved for restores).

That completes the brain dump.  Wish I had more but that is all for now.

This blog is a high level overview of some extensive testing conducted on the EMC (CLARiiON) CX3-80 with 15K RPM FC (fibre channel disk) and the EMC (CLARiiON) CX4-120 with EFD (Enterprise Flash Drives) formerly know as SSD (solid state disk).

Figure 1:  CX4-120 with EFD test configuration.

Figure 2:  CX3-80 with 15K RPM FC rest configuration.

Figure 3:  IOPs Comparison

Figure 4:  Response Time

Figure 5:  IOPs Per Drive

Notice that the CX3-80 15K FC drives are servicing ~ 250 IOPs per drive, this exceeds 180 IOPs per drive (the theoretical maximum for a 15K FC drive is 180 IOPs) this is due to write caching.  Note that cache is disabled for the CX4-120 EFD tests, this is important because high write I/O load can cause something known as a force cache flushes which can dramatically impact the overall performance of the array.  Because cache is disabled on EFD LUNs forced cache flushes are not a concern.

Table below provides a summary of the test configuration and findings:

In summary, EFD is a game changing technology.  There is no doubt that for small block random read and write workloads (i.e. – Exchange, MS SQL, Oracle, etc…) EFD dramatically improves performance and reduces the risk of performance issues.

This post is intended to be an overview of the exhaustive testing that was performed.  I have results with a wide range of transfer sizes beyond the 2k and 4k results shown in this posts, I also have Jetstress results.  If you are interested in data that you don’t see in this post please Email me a rbocchinfuso@gmail.com.

In the interest of benchmarking de-duplication rates with databases I created a process to build a test database, load test records, dump the database and perform a de-dupe backup using EMC Avamar on the dump files.  The process I used is depicted in the flowchart below.

1.  Create a DB named testDB
2.  Create 5 DB dump target files – testDB_backup(1-5)
3.  Run the test which inserts 1000 random rows consisting of 5 random fields for each row.  Once the first insert is completed a dump is performed to testDB_backup1.  Once the dump is complete a de-dupe backup process is performed on the dump file.  This process is repeated 4 more times each time adding an additional 1000 rows to the database and dumping to a new testDB_backup (NOTE:  this dump includes existing DB records and the newly inserted rows) file and performing the de-dupe backup process.

Once the backup is completed a statistics file is generated showing the de-duplication (or commonality) ratios.  The output from this test is as follows:

You can see that each iteration of the backup shows an increase in the data set size with increasing commonality and de-dupe rations.  This test shows that with 100% random database data using a DB dump and de-dupe backup strategy can be a good solution for DB backup and archiving.

I just completed a fairly comprehensive EMC CLARiiON AX demo video. The demo video is available on YouTube and I am also hosting a high quality video here.

EMC CLARiiON AX Demo