Got a really awesome blog for you today, but first, credit where credit is due! Big thanks to Rob and Tomer for their ideas and help on this blog!!
Have you ever had a "noisy" neighbor? You know the type, parties all hours, loud music, drunk people asleep on your lawn... Hey, I'm all for having a good time, but just because you love your music, doesn't mean you have to share it with everybody else. :-)
Have you ever wished you could just turn down the noise level while letting them party away? This way, no one gets upset, no yelling, no police, everybody is happy! Well, with Tintri QoS, you can do just that!
So let me back up a little bit. I was doing some performance test work and I ummm, well, accidentally left my IOmeter blasts running at full volume. Yep, I was the noisy neighbor... Rob noticed a lot of network latency on Tintri Global Center on one of our VMstores and he was able to quickly and easily pin point which virtual machine(s) were going crazy and that I was the culprit!
If you're not familiar with Tintri Global Center, you MUST check it out! You can monitor up to 32 Tintri VMstores through a single GUI. Pretty cool!
Anywho, back to me being a bad neighbor... Rob was nice enough to not throttle me, but he could have if he wanted to! So not only can you find the miscreant, you can do something about it!!
I set up a quick test to show you how cool this feature is. Now remember, this feature doesn't just have to be about noisy neighbors, you can also use it to put maximum and minimums on important VMs, or just VMs that don't need as much horse power as production. (Development and Test perhaps???)
So here we have two servers running IOmeter. NG-Server1 is the noisy guy and NG-Server4 is the innocent bystander that gets bottle necked by NG-Server1's intense partying. NG-Server1 represents a dev/test machine that's pushing some big blocks down the pipe and it's affecting everyone else including NG-Server4 who represents my production box. His response time is pretty terrible and the backup starts to build up!
Here's NG-Server4 who's running a respectable workload, but is getting majorly throttled due to NG-Server1's partying.
In the real world if I was a System Administrator, this is how things would play out...
<<RING RING>> (Neil's Cell Phone Goes Off at 3AM)
NG-Server4's Application Owner: "Hey Neil, the performance of my application sucks! This is affecting production and costing the company tons of money! Fix it!!!"
Neil: "Ummm, who is this?"
NG-Server4's Application Owner: "Just fix it!!!"
In the past, a great hunt for the bottleneck would begin, which would normally wind up with me looking high and low for the culprit and usually hours of investigation and a sleepless night.
Let's head over to Tintri Global Center to see how easy it is to not only track down the offender, but resolve the issue!
This view is pretty cool, I can see all of my VMstores in a single glance and here's the VMstore that's having trouble, but what about the virtual machine(s) that are causing the headache? Let's click on Virtual Machines to find out who's causing Neil to lose sleep!
Woo Hoo, there's our noisy neighbor!!
Oh yeah, and check out the cool metrics on the side. Sheesh, I'm so pre-occupied with finding the noisy guy, I didn't show you all the cool information you can get about your virtual machines and the environment they're living in. More on that later!!
Okay, so we found our noisy guy, time to call the owner of NG-Server1.
Neil: "Hey, application owner of NG-Server1, you're crushing production, can I shut down your VM or move you to another host?"
NG-Server1 Application Owner: "What?!?! NO!!!! I'm running some really cool program that will find alien life!"
Neil: "Ummm, okay... Then is it okay for me to throttle you a bit so we can do business?"
NG-Server1 Application Owner: "UGH! I suppose..."
Let's head over to our VMstore and throttle this sucker!! Remember we know which VMstore AND VM to focus on. Right click on the VM, click Configure QoS and...
While finding alien life is pretty cool, keeping my job is even better! So here I'm throttling this VM to 100 IOPS maximum. I know, that's mean, but hey, it's 3AM!!!
Let's go take a look at IOmeter now.
Here's NG-Server1. Notice the IOPS have dropped substantially AND the latency is WAY up. Sorry, not going to find aliens tonight.
Let's take a peak at our production server NG-Server4.
Wow, what a difference!!! From 4K IOPS to 50K IOPS! And the latency has dropped substantially too.
Now remember, with all things in life, your mileage will vary, but pretty cool huh?!?!
Neil calls app owner of NG-Server4
<<Ring Ring>>
NG-Server 4 Application Owner: "I don't know what you did Neil, but things are running GREAT! You deserve a raise and a promotion!"
Neil: "Wow, thanks!"
Okay all joking aside, finding bottlenecks is extremely difficult and I spent MANY hours in the past trying to find them. Sometimes you can NEVER find them and they just disappear when the machine causing havoc finishes it's workload. I WISH I had this capability when I was a System Administrator, it would have made my life a ton easier while helping the company that much faster.
The great thing is this technology exists NOW with Tintri!
Until Next Time
-Brain
