VAAI for Dummies - AKA VAAI for Me! - Part I

Hi There,

I've been away for awhile because I've been doing a lot of testing and document writing.  I'm currently working on updating a VAAI document with all the new VMware and NetApp goodies!  For those who aren't familiar with VAAI it is the vStorage APIs for Array Integration.  A few years back there was a joint effort among VMware, and some storage companies to allow storage to talk to VMware through API's, SCSI and primitive commands that normally get communicated over the wire.  By doing this a lot of work can be optimized because the storage would handle it.  Want to use VAAI?  Make sure your storage vendor supports them and that you're on a version of vSphere that does as well.

Now for some reason I really had a tough time with VAAI.  Not the concept, but the actual practical usage of the different APIs.  To add to the confusion in ESXi 5.1 VMware added more functionality!  So what did I find difficult?  Well for starters, how the heck do I tell if the thing is on and working!  A lot of the abilities are just "on" and start working.  I'm hoping that my confusion and deep dive into the technology will help you from going through similar difficulties and confusion!

Since VAAI is broken up into multiple capabilities, I felt a single article would not do it justice, therefore I'm going to write a series of VAAI blogs, unless they are wildly unpopular!  :-)  Big thanks to Cormac Hogan and Peter Learmonth who's papers were invaluable while exploring the deep and dark recesses of VAAI!!  Enjoy!

Today were going to talk about Dead Space Reclamation this is one of those new primitives I was telling you about. Why start with this one?  It's unique that you have to kick a command off to get it to work.  Dead Space Reclamation will clean up block storage if a virtual machine is stored on a thin provisioned datastore on a LUN and the machine is either deleted or moved to another datastore.  What? What? What?

Okay, say you create 100G LUN on your NetApp and turn on thin provisioning.  You assign the LUN to vSphere and install a 20G machine, but the machine is only using 5 gig.  At the vSphere and NetApp layer you'd only use up 5 gigs.  Now, I decide I need to either delete the machine or move it to another datastore.  After the delete/move vSphere reports all 5 gigs are back and ready for usage, but if you checked your storage it still thinks 5 gigs are used up because storage doesn't look into LUNs.  This is where this feature is useful, you tell your ESXi server to create a file that will consume a certain amount of space and give it back to the controller once you're done!

One thing you need to be careful is how much you tell ESXi to grow this file.  Remember, you'll probably have other data on this datastore, so using 90 or 100% would be a bad idea. :-)  In Cormac's document he uses an example of 60% and I like that number!  If you're worried about a performance impact, start with smaller numbers and slowly ramp up to larger percentages to avoid swamping your controllers.(Thanks to Jenn Schrie for this input!!)  So how do you do this?

1. Here's a LUN that has thin provisioning and is using about 7G of storage:
f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:    6.7g (7168028672)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled

2. Check your LUN on the ESXi side to make sure it has this VAAI capability and is thin provisioned:

esxcli storage core device list
naa.60a9800032466635635d414c45554356
   Display Name: NETAPP iSCSI Disk (naa.60a9800032466635635d414c45554356)
   Has Settable Display Name: true
   Size: 102656
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.60a9800032466635635d414c45554356
   Vendor: NETAPP
   Model: LUN
   Revision: 811a
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
   Other UIDs: vml.020003000060a9800032466635635d414c455543564c554e202020
   Is Local SAS Device: false
       Is Boot USB Device: false
   
esxcli storage core device vaai status get -d naa.60a9800032466635635d414c45554356
naa.60a9800032466635635d414c45554356
   VAAI Plugin Name: VMW_VAAIP_NETAPP
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

3.  Delete or migrate the virtual machine.

4.  Check your LUN again:
f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:    6.7g (7168028672)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled
As you can see, still using up 6.7G.

5. Log onto your ESXi machine where the LUN is mounted and change directory into the /vmfs/volumes directory where the datastore is located.

6. Run the reclamation command: 
vmkfstools -y 60
Attempting to reclaim 60% of free capacity 99.1 GB (59.4 GB) on VMFS-5 file system 'vaai_iscsi2' with max file size 64 TB.
Creating file .vmfsBalloonLrxcCO of size 59.4 GB to reclaim free blocks.
Done.

7. Check on your storage again:
 f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:   87.3m (91533312)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled

I hope this blog was helpful and that you'll enjoy what I've got planned for the others!

Until Next Time!
-Brain