In part 1 we talked about data deduplication technology in general and how live filesystem deduplication works on Netapp filers. Lets have a look how this really works.

The license to use ASIS is free should be free but ask your Netapp dealer. For the sake of this demonstration I made a copy of one of our VMware NFS volumes so we can watch how the whole process works.

nas6070> df -h vmware_data_asistest
Filesystem               total       used      avail capacity  Mounted on
/vol/vmware_data_asistest/     1200GB      504GB      695GB      42%  /vol/vmware_data_asistest/
/vol/vmware_data_asistest/.snapshot      300GB       50GB      249GB      17%  /vol/vmware_data_asistest/.snapshot

The volume has a total capacity of 1500 GB with 1200 GB available for data and the standard 20% snapshot reserve. A VMware cluster uses the volume as a datastore and there are approximately 35 virtual machines (both Windows and Linux) inside with disk sizes from 5-25 GB using a total of 504 GB. This is our inital situation before we start the deduplication process.

Initial deduplication

To enable deduplication and deduplicate existing data two simple commands on the filers command line are used:

nas6070> sis on /vol/vmware_data_asistest
SIS for "/vol/vmware_data_asistest" is enabled.
Already existing data could be processed by running "sis start -s /vol/vmware_data_asistest".       

nas6070> sis start -s /vol/vmware_data_asistest
The file system will be scanned to process existing data in /vol/vmware_data_asistest.
This operation may initialize related existing metafiles.
Are you sure you want to proceed with scan (y/n)? y
Fri Apr 18 06:13:23 CEST [nas6070: wafl.scan.start:info]: Starting SIS volume scan on volume vmware_data_asistest.
The SIS operation for "/vol/vmware_data_asistest" is started.

The initial scan of the volume will build the fingerprint database and then deduplicate data. This is a long background process which can be monitored:

nas6070> sis status
Path                           State      Status     Progress
/vol/vmware_data_asistest      Enabled    Active     47 GB Scanned

For this particular volume the first scan and the deduplication needed 4h 45 minutes. Remember - this is one-time process as all subsequent deduplication runs are incremental. Of course data can be accessed and changed during deduplication. Lets see the results using the df -s command which shows the savings:

nas6070> sis status
Path                           State      Status     Progress
/vol/vmware_data_asistest      Enabled    Idle       Idle for 05:28:35      

nas6070> df -sh vmware_data_asistest
Filesystem                used      saved       %saved
/vol/vmware_data_asistest/      219GB      378GB          63%

At first sight I thought: 63% savings? Nice, but not that spectacular. But remember - the WAFL file system will never overwrite existing data on your disk, so if you had snapshots before the deduplication started all the “old” data is frozen and still takes up space - normally only in the area reserved for snapshots but here lots of data was changed so the 20% were not enough. Lets look at the amount of space currently used by the snapshot and compare to the original state above.

nas6070> df -h vmware_data_asistest
Filesystem               total       used      avail capacity  Mounted on
/vol/vmware_data_asistest/     1200GB      219GB      980GB      18%  /vol/vmware_data_asistest/
/vol/vmware_data_asistest/.snapshot      300GB      388GB        0GB     130%  /vol/vmware_data_asistest/.snapshot

You can now either wait until the old snapshots expire or delete them manually (only if you don’t need them). I did exactly that for the purpose of demonstration and here is how the volume looks like now:

nas6070> df -sh vmware_data_asistest
Filesystem                used      saved       %saved
/vol/vmware_data_asistest/      130GB      378GB          74%

Before we used 509 GB of storage and now it’s only 130 GB - 74% saved. This additional space is immediately available for more VMware data inside the datastore.

VMware servers are not even aware of the process: their view of the NFS datastore is exactly the same as before deduplication and reads deliver the same data. If new data is written changes go to disk normally and deduplicated in regular intervals during (no inline deduplication). There is a schedule for every deduplicated volume:

nas6070> sis status -l
Path:                    /vol/vmware_data_asistest
State:                   Enabled
Status:                  Idle
Progress:                Idle for 07:46:29
Type:                    Regular
Schedule:                sun-sat@0
Last Operation Begin:    Sat Apr 19 00:59:00 CEST 2008
Last Operation End:      Sat Apr 19 00:59:52 CEST 2008
Last Operation Size:     23221 KB
Last Operation Error:    -

Here you can see the default schedule (Sat+Sun @ midnight). You can change it to suit your needs (useful in combination with backup snapshots).

Of course you can replicate your deduplicated volumes to another machine using SnapMirror and you will see the same savings on the target side. Caveat: if you backup to tape using NDMP you’ll get the original size.

In my opinion VMware and filesystem deduplication is a disruptive technology: with 60-90% savings the amount of data you can store on the filer changes dramatically - and there is no new software, no new appliances or additional complexity (see above - thats all!). It also saves energy. Beginning with Ontap 7.3 the fingerprint database is per-aggregate instead of per-volume so there will be additional savings if you have multiple datastore volumes (we do).

In part 3 we’ll discuss how ASIS can help to optimize thin provisioning. Stay tuned…

Do you use ASIS with VMware? What savings do you see? Comments welcome! (moderated).

Deduplication is a technology used to save precious space on your storage systems. In short, deduplication removes redundant data but at the same time leaves the logical view intact and is thus transparent to all applications.

Redundant data is everywhere - on many file servers each member of a project team has her own copy of a document which only marginally differs from other versions. If we compare two Windows servers they contain lots of identical data: the whole operating system, many applications… and - at the disk level - even the free space is often identical! A third area which originally sparked interest in deduplication is backup: if you backup hundreds of almost identical workstation PCs there is a lot of potential for deduplication because many files are identical.

Data deduplication vs. data compression

The idea of removing redundancy is also used in data compression: if you consider the string

ABCDDDDDDEEEEFAAAAAAABBBBBB

you can save space by writing it as 

ABC,6xD,4xE,7xA,6xB

This is a very primitive example of run-length compression but you get the idea. Another possibility is to find frequent patterns in data and then assign shortcuts to them. Unfortunately data compression has three disadvantages for filesystem applications:

• “finding patterns” is extremely time-intensive for 100s of GB of data as you have to remember lots of possible combinations to be efficient
• data has to be de-compressed before it can be used (performance)
• it is difficult to change something “in the middle” as everything starts moving around

Because of these problems - which converge to bad performance - transparent compression is not widely used in filesystems - that means it’s available (e.g. NFTS) but hardly utilized for production data. 

How deduplication works

Deduplication divides up data into chunks, finds all chunks which are identical and then stores only one copy of each chunk and all locations where it appeared. To identify chunks hash algorithms like MD5 or SHA can be applied which generate a digital “fingerprint” for a given piece of data. If the fingerprints of two chunks are equal we can compare the chunks and hopefully find a replicate. The trick is that comparing fingerprints (which are small) is much faster than comparing the data itself - it’s basically a “did I see that fingerprint before?”.

The size of a “chunk” is the magic key to efficiency: one of the few filesystems which use deduplication is Microsofts Single Instance Storage in Windows Server 2003 R2 and there a “chunk” is a whole file. That means if two files are identical Microsoft SIS can detect that but if they differ only in one single bit you’ll not see any savings.

Lets assume two file are identical - what can we do now? Microsoft’s SIS basically removes all but one duplicate files and replaces them with a hardlink to the “original”. In this way all duplicates are still visible but don’t take up space on disk. In fact some virtualization systems (e.g. Virtuozzo) can do the same thing for multiple copies of virtual environments (VEs) where multiple VEs share a single copy of the operating system. The “hardlink” analogy is however problematic if you consider write operations to a duplicate: we dont’t want to overwrite the original but to obtain a separate copy. Thats what differenciates Microsofts SIS from normal hardlinks.

Netapp’s ASIS

Lets go back to Netapp’s deduplication system called ASIS (Advanced Single Instance Storage). It is integrated into the WAFL file system used on all Netapp filers and used the 4 kB filesystem block as a chunk. If multiple 4 kB blocks are equal ASIS will find them using a fingerprint database. Of course this is much more efficient than Microsoft’s solution because small differences between files leave a large part of savings intact.

ASIS deduplicates in the background - new data written to the filesystem is not deduplicated immediately but only in scheduled intervals. Read performance is comparable to a normal volume without ASIS. I won’t dive into the gory technical details how this is done as there is already material available on the net:

• Netapp patents 
  • SYSTEM AND METHOD FOR MANAGING DATA DEDUPLICATION OF STORAGE SYSTEMS UTILIZING PERSISTENT CONSISTENCY POINT IMAGES
  • SYSTEM AND METHOD FOR RETRIEVING AND USING BLOCK FINGERPRINTS FOR DATA DEDUPLICATION
• The technical report TR-3505

VMware datastores and deduplication

When you look into a typical VMware datastore with dozens of virtual disks you can already imagine the potential for deduplication: many virtual disks are based on the same templates and thus per definition largely identical. ASIS can help to recover the wasted space.

Part 2 of this series will contain a short hand-on-session using a VMware datastore and a Netapp. Stay tuned…

Comments welcome! (moderated)