VAAI for Dummies - AKA VAAI for Me! - Part I

Hi There,

I've been away for awhile because I've been doing a lot of testing and document writing.  I'm currently working on updating a VAAI document with all the new VMware and NetApp goodies!  For those who aren't familiar with VAAI it is the vStorage APIs for Array Integration.  A few years back there was a joint effort among VMware, and some storage companies to allow storage to talk to VMware through API's, SCSI and primitive commands that normally get communicated over the wire.  By doing this a lot of work can be optimized because the storage would handle it.  Want to use VAAI?  Make sure your storage vendor supports them and that you're on a version of vSphere that does as well.

Now for some reason I really had a tough time with VAAI.  Not the concept, but the actual practical usage of the different APIs.  To add to the confusion in ESXi 5.1 VMware added more functionality!  So what did I find difficult?  Well for starters, how the heck do I tell if the thing is on and working!  A lot of the abilities are just "on" and start working.  I'm hoping that my confusion and deep dive into the technology will help you from going through similar difficulties and confusion!

Since VAAI is broken up into multiple capabilities, I felt a single article would not do it justice, therefore I'm going to write a series of VAAI blogs, unless they are wildly unpopular!  :-)  Big thanks to Cormac Hogan and Peter Learmonth who's papers were invaluable while exploring the deep and dark recesses of VAAI!!  Enjoy!

Today were going to talk about Dead Space Reclamation this is one of those new primitives I was telling you about. Why start with this one?  It's unique that you have to kick a command off to get it to work.  Dead Space Reclamation will clean up block storage if a virtual machine is stored on a thin provisioned datastore on a LUN and the machine is either deleted or moved to another datastore.  What? What? What?

Okay, say you create 100G LUN on your NetApp and turn on thin provisioning.  You assign the LUN to vSphere and install a 20G machine, but the machine is only using 5 gig.  At the vSphere and NetApp layer you'd only use up 5 gigs.  Now, I decide I need to either delete the machine or move it to another datastore.  After the delete/move vSphere reports all 5 gigs are back and ready for usage, but if you checked your storage it still thinks 5 gigs are used up because storage doesn't look into LUNs.  This is where this feature is useful, you tell your ESXi server to create a file that will consume a certain amount of space and give it back to the controller once you're done!

One thing you need to be careful is how much you tell ESXi to grow this file.  Remember, you'll probably have other data on this datastore, so using 90 or 100% would be a bad idea. :-)  In Cormac's document he uses an example of 60% and I like that number!  If you're worried about a performance impact, start with smaller numbers and slowly ramp up to larger percentages to avoid swamping your controllers.(Thanks to Jenn Schrie for this input!!)  So how do you do this?

1. Here's a LUN that has thin provisioning and is using about 7G of storage:
f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:    6.7g (7168028672)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled

2. Check your LUN on the ESXi side to make sure it has this VAAI capability and is thin provisioned:

esxcli storage core device list
naa.60a9800032466635635d414c45554356
   Display Name: NETAPP iSCSI Disk (naa.60a9800032466635635d414c45554356)
   Has Settable Display Name: true
   Size: 102656
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.60a9800032466635635d414c45554356
   Vendor: NETAPP
   Model: LUN
   Revision: 811a
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
   Other UIDs: vml.020003000060a9800032466635635d414c455543564c554e202020
   Is Local SAS Device: false
       Is Boot USB Device: false
   
esxcli storage core device vaai status get -d naa.60a9800032466635635d414c45554356
naa.60a9800032466635635d414c45554356
   VAAI Plugin Name: VMW_VAAIP_NETAPP
   ATS Status: supported
   Clone Status: supported
   Zero Status: supported
   Delete Status: supported

3.  Delete or migrate the virtual machine.

4.  Check your LUN again:
f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:    6.7g (7168028672)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled
As you can see, still using up 6.7G.

5. Log onto your ESXi machine where the LUN is mounted and change directory into the /vmfs/volumes directory where the datastore is located.

6. Run the reclamation command: 
vmkfstools -y 60
Attempting to reclaim 60% of free capacity 99.1 GB (59.4 GB) on VMFS-5 file system 'vaai_iscsi2' with max file size 64 TB.
Creating file .vmfsBalloonLrxcCO of size 59.4 GB to reclaim free blocks.
Done.

7. Check on your storage again:
 f35*> lun show -v /vol/vaai_iscsi2/vaai_iscsi2
        /vol/vaai_iscsi2/vaai_iscsi2  100.2g (107642617856)  (r/w, online, mapped)
                Comment: "The Provisioning and Cloning capability created this lun at the request of Administrator"
                Serial#: 2Ff5c]ALEUCV
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: vmware
                Maps: rcu_generated=3
                Occupied Size:   87.3m (91533312)
                Creation Time: Thu Apr 11 13:38:15 PDT 2013
                Alignment: aligned
                Cluster Shared Volume Information: 0x0
                Space_alloc: enabled

I hope this blog was helpful and that you'll enjoy what I've got planned for the others!

Until Next Time!
-Brain
 

Didja Know? Functional Alignment in VSC 4.x!

Ahoy,

Since I've been on an alignment kick, I thought I'd dive into one of the new features available in our Virtual Storage Console (VSC).  We all hate misalignment, it creeps into our virtual environments and causes all kinds of havoc, like termites or roaches!  In the past the only way to solve misalignment is you have to shut down the virtual machine, there was NO way around it.  There's all kinds of great tools out there that can get you 90+% there without an outage, but just like fumigating those nasty buggers you've gotta vacate the house and let the exterminator do his job!  Until now!  Take that you little beasties!

VSC has a super cool new feature called Optimization and Migration.  This feature lets you logically align your misaligned virtual machines while they're running.  NO DOWNTIME!  Okay, okay, you're thinking, "Awww, there's the catch, logically align!"  So what do we do?  We create a new datastore that's shimmed to bump the alignment off to push the alignment of the unaligned machine to logically aligned.  Than we do a Storage vMotion to move the VM to the new datastore.  Just like in Algebra, two negatives make a positive!  There's also another catch, currently the feature is only block supported.  But no fears NAS fans, if you want to use the feature, just contact your account representative and let them know you want to give it a try.

Here's the part I like the best, the detailed steps!

 1.  So here we have our misaligned machine.  We see it's misaligned when we try to do a rapid clone.

2.  So next we go back to our Home screen in vCenter and choose the NetApp N.
3.  Once there select the Optimization and Migration tab and select Scan Manager.  Select the datastore the misaligned machine lives on and click Scan selected.

4. You'll notice the Scanner status is RUNNING.

5.  Once the Scanner status is IDLE you're ready to begin smooshing those nasty misaligned machines!

6.  Click on Virtual Machine Alignment link, expand the Misaligned folder and click on the datastore you scanned earlier.  You should now see your misaligned machines.  Here mine is WinXPiSCSI.  Select the machine(s) you want to logically align and click on the Migrate link.

7.  You'll now be presented with a very similar set of screens when you create a new datastore in VSC.  Select storage controller and Vserver (if you have one).

8.  Here's where things are a little different.  You might see this screen and want to use an existing datastore, but the key here is VSC actually creates a new datastore that has the shim offset built-in.  So if you don't one of these special datastores created yet, you'll need to let VSC create a new one for you.
 

 9.  Next tell VSC you'll be using VMFS. (Remember if you want to use NAS contact your account team!)

10.  Select the Protocol, Size of the new datastore,  the datastore's name, if you want VSC to create a new volume or not, the aggregate, whether to use thin provision and the block size.

11.  You'll be presented with a summary screen.  If everything looks good, click Finish!
12.  Once the migration is complete go back into Optimization and Migration and you'll notice your new datastore has been created and is labeled as Optimized - Yes.
 
13.  Go back into Virtual Machine Alignment and you'll notice your virtual machine is now in the Functionally aligned folder!

Now the only bummer is if you go to clone this virtual machine, you'll still get a misaligned warning.  You have to remember that the virtual machine IS still misaligned, but NetApp has shimmed it's storage to cancel out the misalignment nastiness.  If you want to physically align the machine, use any one of the many tools out there.  My last article showed you how to align with VMware Converter, but there's also a converter that you can download from within VSC called MBRAlign.  It's under the Tools link inside the Monitoring and Host Configuration tab in VSC.

Metrics Time! Please Vote!

Hi All,

I'm very interested in knowing how you found my blog.  If you wouldn't mind, please let me know!  The vote is now open and can be found on the right hand side of the page.

Thanks in advance,
Neil

Citrix on NetApp Clustered ONTAP - Best Practices and More!

Ahoy Ahoy,

I have some very exciting news!


Rachel Zhu and I would like to present our latest Citrix paper! This was a huge endeavor of testing XenDesktop, XenApp, Profile Management, Backup and Recovery, Performance Analysis, and the kitchen sink all on Clustered ONTAP! HUGE thanks to Will Strickland who did all of our performance testing!!

Learn the best practices of Citrix VDI on NetApp Clustered ONTAP as well as how to backup and recover your user's data using NetApp tools.

The paper is available here! 

http://www.netapp.com/us/System/pdf-reader.aspx?m=tr-4138.pdf

We hope you enjoy it and that it helps answer questions you have regarding NetApp Clustered ONTAP in Citrix VDI environments!

Didja Know? VMware Converter for Correcting Misalignment!

Ahoy,

Another in my series of "Didja Know?"  This might be old news to you, but I just found out that version 5 of the VMware vCenter Converter will fix virtual machine misalignment!  So what is misalignment and why should you care?  That's a blog all in its own, but basically it's when the guest file system doesn't match the block boundaries of the storage system.  Okay, so what?  The problem is data gets split between storage blocks and you get partial reads and writes to many more blocks than you would need if the storage and OS were aligned.  This can result in a lot more work on your storage device than is necessary.  For those that don't know, this is much more common than you realize and just until recently OS's had alignment issues with many storage vendors.  Here's a really good paper that explains misalignment in great detail!

There are a lot of tools to correct misalignment and the best practice is to fix it before you even put a machine into production.  Today I'm going to show how VMware vCenter Converter can fix alignment.  VMware vCenter Converter is normally a tool to convert physical machines into virtual machines, but I was delighted to hear that they added this feature!

Below I'm using the NetApp Virtual Storage Console to clone a Windows XP machine.  The VSC tells me the machine is misaligned and if I'd like to proceed with the clone?  If I proceed, I will carry over the misalignment to the clone.  VSC has tools built in to correct misalignment as well, but that's a demonstration for another day.

So let's fix the problem!  I run the VMware Converter and tell it that I want to Convert a machine.

 Select where the misaligned machine lives and the machine itself.

Enter the credentials of the vCenter you'd like the converted machine to go to.

Give the new machine a name and where it should be located.

Select the datastore the machine should live on and what virtual machine version you'd like.

Click on "Data to copy".


Here's where things change!  Choose, "Select volumes to copy". 

Ensure the "Create optimized partition layout" is checked.  This will correct the misalignment on the clone.

Once the convert is completed use the VSC to check for misalignment.  Here I do a "Create Rapid Clones".  This time you can see the converted virtual machine past the misalignment scan because it is now aligned!

I hope you enjoyed this "Didja Know" and that it will help keep your environment misalignment free!

Until Next Time!

Let's Talk about Linked Clones

Hi All,

Today I thought I'd address a topic I've had many questions about in the past.  The mystery behind VMware View's Linked Clones.  The technology is based upon their snapshot technology and is quite ingenious!  It is very storage efficient, creates clones fast and is easy to manage.  Unfortunately we started to see problems in the field when a large number of Linked Clones were deployed.  The controller would be getting slammed and as a result the customer's desktops would suffer.

The first I heard of the problem I was new to VDI and was told Linked Clones are misaligned, so I created a bunch, logged into a Windows 7 clone and ran an "msinfo32".  Nope, divisible by 4096, not misaligned!  I was told it's not the master image that's misaligned, but the "delta" disk.  What's a "delta" disk?!  The delta disk is where all of the changes a user makes to their desktop get stored.  Since the master image is read-only, writes get written to the delta disk.  Now this is the part that didn't make sense to me.  The delta disk is NOT misaligned, what happens is it can write data as small as 512 bytes.  Now this wouldn't be a problem if every write Windows did was 512 bytes, but it's not.  You get a mix of 512, 4K, 16K, 32K, etc.

Data ONTAP WAFL blocks have 4K boundaries and if Windows wrote everything in 512 bytes, everything would be cool!(8 512 bytes make up 1 4K).  Here in lies the problem.  Say I'm writing to the disk, I write 512 bytes of data, that data is aligned, no problem!  Okay, but the next chunk of data I write is 4K.  Ugh oh!  It won't fit because I only have 3584 bytes left, so I need to carry over the last 512 bytes from the 4K write to another 4K block which requires additional I/O.  Now I go and read that 4K of data and instead of being able to read just one block, I now have to read from two because the data was split over two NetApp 4K blocks.  You got it, more I/O!  Take a look at the picture, it should help.

What I've drawn is 512 bytes of data being written. Next 4K of data and you can see that last 512 bytes gets written to a brand new NetApp block.  And last I write 1K of data.  If I access that 512 bytes of data or that 1K of data, no problem.  The problem is larger chunks of data that can get truncated and placed onto multiple blocks which will increase I/O when it's written AND when it's read.  Is this behavior unique to NetApp?  Nope, it will effect any storage vendor that doesn't use 512 bytes as their block size.

So what can be done about this?  In vSphere 5.1 , VMware introduced a new filesystem format called SESparse.  VMware View 5.2 is the first VMware product that uses the new SESparse disk type.  SESparse has a couple of functions, it can clean up unused space in clones, but what really excites me is the smallest chunk of data it can write to disk is 4K enabling the delta disk to stay block aligned!

Here I have another 512 byte chunk of data being written to disk, and than it jumps to the next 4K boundary and is ready to write the next chunk of data, which I've drawn as 4K.  This does use a little more space, but this will help immensely if you're having performance issues due to the partial reads and writes.  If you're okay with traditional Linked Clones, you don't need to use SESparse, you can stick with VMFSSparse.  There are also some caveats to look out for, so check the VMware site for best practices.

I hope this helped clear up some of the confusion around Linked Clones and why I'm excited about the new version of View!

Until Next Time!

NFS vs. Block (FC/iSCSI) Protocol

Hi All,

Protocol seems to be one of those topics you don't discuss over dinner, like politics!  Storage folks tend to be one or the other NFS or Block and I've heard some very heated debates on this topic.  Yep, nerd fights!  I used to be a die hard block guy, especially for enterprise storage.  NFS was cool for file sharing or home directories, but BLOCK was king when it came to databases, mission critical applications, etc.  When I went to work for NetApp my opinion quickly changed, especially when I began working on virtualization products like VMware.

When I'm talking with customers I'm frequently asked, "What is better block or NFS?"  It's a bit loaded question because the customer usually has an idea of what they like better, plus they've probably already spent a ton of money on a new infrastructure.  The last thing I want to do is call their baby ugly!  I usually tell them *I* prefer NFS for these reasons, and I list out some reasons.  But in case they're a block shop, I remind them that NetApp can do both NFS and block concurrently. :-)

I was reading our internal discussion groups this morning and saw a great post by Nick Triantos.  For those of you that don't know Nick, he's a brilliant guy and an avid blogger.  I'm always amazed how quickly and precisely he knows the answers to things that would take me much more time to find out!  He answered a question on NFS vs. iSCSI for VMware this morning and I'd like to share it with you.  Enjoy!


"Datastore resizing is another difference...With NFS you can resize up or down on the fly. with VMFS you can only increase the size of the datastore. That means if you ever need to rebalance a VMFS datastore by Storage VMotioning VMs to another datastore, you now have captive storage you can't reuse, unless you create a new datastore of the required size, move your VMs into it and destroy the old one. Is it a hard thing to do? No. Does it require additional steps on the server and the storage side? Yes.

Deduplication, post process dedup that is, over NFS, allows a vmware admin to immediately realize the space savings on the host without any additional work. The same is not true for block protocols as additional work needs to be done on the storage array.

No limitation on NFS as to the datastore size. The limit is whatever the file server support. The same is not true for block protocols where the datastore size imposed is at 64TB. Although I find hard to believe one will create a 64TB datastore anyway.

Also,  each virtual disk file on NFS has its own I/O queue directly managed by the NFS server. This is not true for block protocols which have per LUN queues and can become a point of IO contention. All that translate to higher fan-in ratios in terms of the number of VM to an NFS datastore vs VMFS datastore. Partners have been telling me for a long time they have customers with 250 and 300 VMs in a single NFS datastore. The only way you can pack that many VMs in a single VMFS datastore without issues is if they are powered off. :-)

The benefit of NFS is day to day operational efficiency and granularity. The architecture does require some thinking upfront and is largely dependent on the switching infrastructure, but once you lay it down, everything else is a breeze.

In the interest of full disclosure...there are some caveats with NFS...No support by Microsoft for Exchange deployments, although at VMworld, last year, I met with customers that ignored the support statement and have been running with it with no issues. You also can't use Microsoft Failover Clusters (you can't use it with Native vSphere iSCSI either). So as long as these are not required, NFS is the right choice, IMO.

Last but not least, considering that Virtual Machines are comprised of a bunch of files, why would use a block protocol to manage them to begin with, if you had a choice?"

Where Are My Aggregates in VSC for Clustered ONTAP?

Hi Friends,


So you've setup your Vserver, you've done your discovery in vCenter with VSC and added your Clustered ONTAP system, but when you try to provision a datastore your aggregates are no where to be found?!  Yep, another one I ran into that I hope I can spare you some of my frustration. :-)
 
 Here we can see the clustered ONTAP management console for our cluster has been added to VSC.


 But when we try to provision storage, no aggregates show up?

The reason is actually by design.  Remember, everything now happens at the Vserver level, I can do all kinds of cool stuff including migrating a volume from one aggregate to another, as long as that aggregate has been added to the Vserver.  So here's what we've missed.  After we created the Vserver, we forgot to assign aggregates to this Vserver!

Take a look at the available aggregates.  You'll see we have agg0 and aggr1.  Just like in 7-Mode, we don't want to mess with aggr0 since that's where the OS lives.  So lets add all of the aggr1 aggregates from all of the nodes to our Vserver so we can start provisioning storage!

Remember the cool thing about clustered ONTAP is you can migrate volumes and this is one of the building blocks.

Let's go back to VSC and create that datastore!

This time we can see the aggregates, so we're in business and if/when we need to migrate to a volume on another aggregate, the Vserver is aware of them.

Something easy to overlook, but easy to fix!  I hope this was helpful.

Until Next Time!

Look Mom, I'm on YouTube!

Hi All,

Recently I filmed a couple of videos with Citrix and they are ready for viewing!

The first is a panel discussion at a Citrix SE event where we discussed the common storage myths and how NetApp vanquishes those myths!

http://www.youtube.com/watch?v=dMSmG_M0Ca4&list=UUwDXCIzgP3jg6Sm4ZTrmpxA

The second is the first Tech Talk in a series with a Citrix field expert and me.  We discuss what Citrix is doing in the VDI realm and how write optimizations on NetApp will help make a customer’s VDI implementation successful.

http://www.youtube.com/watch?v=wN-F_nFTrDY&list=UUwDXCIzgP3jg6Sm4ZTrmpxA&index=2

I hope you enjoy them!