CrashLoopBackoff

It has been almost 1 year since I started looking at the esxcfg-* commands. It initially came as a look at the first part of the Enterprise Administration Exam’s Blueprint very first bullet point. In that post I talked about using esxcfg-mpath to identify which luns are fiber, iSCSI, NFS or local.

Today lets look a little bit deeper at the command and how it can be used to help you day to day.

We can always use esxcfg-mpath -l to list all of the luns and their paths. A good thing to check here is that you have the same number of paths to each datastore that comes from the SAN. You may have a zoning issue if a certain lun can only be seen from 1 path rather than all of them. In general each hba will see the lun through each controller in an active active type fiber channel SAN. So hba A should see the lun from Controller A and B. Likewise, hba B should see Controller A and B for a total of 4 paths. If you are using fixed or MRU as the pathing policy only one will be active but esxcfg-mpath -l will show four paths.

Of course if you have more hbas and controllers you will have more paths.

You can follow the examples given by the esxcfg-mpath -h to get help. One useful tool is to create a crude script using esxcfg-mpath –policy with the –lun tag to change the policy from say MRU to Fixed. I am not a perl scripter and I sure someone already has a real shell or perl script to set the policy but I do like to prepare multiple command in notepad then paste them into the cli.

esxcfg-mpath –policy=fixed –lun=vmhba0:0:1
esxcfg-mpath –policy=fixed –lun=vmhba0:0:2
and so on…

Then try to feel really smart by alternating the paths so every fourth lun will use different paths.

esxcfg-mpath –path=vmhba1:0:1 –lun=vmhba0:0:1 –state=on
esxcfg-mpath –path=vmhba2:0:1 –lun=vmhba0:0:2 –state=on
esxcfg-mpath –preferred –path=vmhba1:0:1 –lun=vmhba0:0:1
esxcfg-mpath –preferred –path=vmhba2:0:1 –lun=vmhba0:0:2

The first two lines set the path for the respective luns to different hbas. The last two lines set the preferred path to that same port on the hba. So when there is failover the path will fail back to your set config when all is well.

Renewing the Push

Well I passed the VCP 4 and my CCNA expired (can’t get around to renewing it). At work I did the VTSP as required by the partner program. Since I am in a test taking grove I think I need to push to passing the Enterprise Administration Exam. There was a series I started a while back on command line management of ESX. So like several others I will set my next goal at the Enterprise Administration Exam.

Here are some good sites I have recently seen on studying for the Exam.

VCDX Study Guides
Simon Long’s Site
follow @vcdx001 on twitter he is giving hints about the design defense, great info, but don’t want to get too far ahead of myself.

I think for this first test you just have to know what you are doing with VI 3. My big problem is having worked with vSphere for a couple months now I hope that doesn’t hurt me.

: It isn’t time to freak out. Yet.

VMworld Wednesday Morning – Keynote and Virtualizing Exchange Session

This morning’s keynote was more geared toward the engineer at least I felt more interested. Not to say yesterday was bad, today played more to my interest level.

The Wyse Pocket Cloud is one thing I will be sure to check out. Actually how usable it will be on AT&T shotty 3g network remains to be seen but if you are in range of a good wifi, you probably have your laptop around, but hey it is still freaking cool. Having tried RDP and Citrix via Mobile PC a couple years ago I am interested to see how far the usability has come. Before the usability was equal to zero so there was nowhere to go but up.

Springsource was on the stage again. I believe VMware is trying to show their relevance in the direction they are planning.

Attended a really good Exchange 2007 on vSphere session. Most of it I was familiar with having tried to keep up with the whitepapers on Exchange over the last year. It was good to see a customer success story. Technically the focused more on topology and server role layouts than any kind of “set this setting to x, to make it fast”. From what I can see you need a 3rd party snapshot (san level) software to protect the database from corruption. All other methods do not keep multiple copies of the data. It was good to see a baseline of 1500 users per mailbox server on a single vCPU VM when using the Intel 5500 series procs. That will be good ammo to build one vCPU box and scale out and use FT for hardware failure.

VMworld Day 2 – All your vCloud are belong to us

Note: I rarely do posts like this. I would rather explain an admin problem and solution. I hope this doesn’t scare too many away. I am in a rant mood.

Day 2 was a great day at VMworld. The key-note today combined with the announcement after the keynote sparked a couple thoughts. I would bet I am not the only one that noticed. VMware basically put Microsoft, Google and Amazon on notice. VMware now has the tools to make a challenge to these previously unchecked organizations.

Microsoft and Google
First the giant world domination bent company based in Redmond and its information hording rival from Nor Cal. The significance of the Springsource merger/purchase and vCloud API’s is telling everyone use your existing apps in the private/public cloud, also your custom developed applications will soon fly into the cloud. Nothing we didn’t already know but now they are all supplied by VMware. Virtualization in general can make Microsoft mortal, and who would use Google apps if the apps they actually know and like could be highly available in a per month charge model.

Amazon
The vCloud Express being made available means VMware can provide virtual services on demand to anyone with a credit card. All at the same time letting the hosting companies front the major expenses for datacenter buildout. The software gets sold no matter how successful everything is. How will Amazon be able to turn a profit when competing against the most proven Enterprise platform for providing virtual servers? Pretty hard to do in my opinion.

Now before being called a VMware fanboy. I think VMware is starting a game they better win. There may be a glimmer of hope that competition will benefit the consumer. The bullying by software vendors of their customers should turn into innovation to set themselves apart. The first company to become complacent will lose. At this point who would know what is going to happen.

VMworld Sunday and Monday – Monkeys Fly

This is the disjointed cliffs notes but too long for twitter version of my VMworld experience so far. Monkey comment is to distinguish this post from the other 11billion being posted already. Thanks.

Sunday

Arrived unhurt from Memphis. Met a few other VMworlders on the plane.

Cabby sped away without giving me change, I am a big tipper I guess.

Discovered shorts and flip flops are not San Francisco attire. Needed to get my jeans and fleece out. Stopped washing my hair to fit in too.

Registered for the conference picked up my sweet bag and goodies.

Went to Taylor’s Automatic Refreshment down on the embarcadero for lunch. Very good.

Went and studied for a few hours for the VCP 4

Attended the VMworld Underground Warm-Up Extravaganza at the Thirsty Bear. I had a great time. I met several people that I only knew via twitter and blogs and message boards. It was great to put names to faces.

Monday

Could not sleep much past 5:30 PDT this morning. Got up did some additional VCP studying. Headed out to breakfast

Went into SRM troubleshooting lab at 7:30 besides some technical burps it went well, good to troubleshoot intentional and unintentional technical problems. I learned what to look for in the log files which was the main reason I elected this lab.

Had extra time and felt like there was no way I could get more vSphere info into my head I sat for the VCP 4 an hour early. I am glad that I did. Must have been too many people testing at once or the test software is just no good. The test would pause for 30-60 seconds between each question, slowing down my rhythm and frustrating me a little. I kept doing minute per question calculations in my head. All technical problems are forgiven though because I passed. Mainly glad I don’t have to do that again for a while. Hopefully the exam engine or whatever was wrong will be ok next time.

Getting ready for the Welcome Reception tonight and catching up on some email and other business. Thank you to everyone with all the kind words about me passing the VCP 4. Woo woo!

Using iSCSI to get some big ole disk in a Virtual Machine

First, I have lived in the South too long, because I said “Big ole disk” and couldn’t think of a more appropriate phrase. Now someone rescue me if I start to tell you to “mash” the power button on your server or SAN. I kid.

I am sure everyone out there has used this before but I like to document these things just case someone else needs help.

A coworker and I were installing a vSphere environment last week to support some new software for a customer. The software vendor required approximately 30 x 146GB drives in a Raid 5 to store images. Never would guess the software vendor happens to sell SANs too! I exaggerate it actually called for 3TB of usable space.

So my thought was to get over 2TB limit of VMFS we would need to use the MS iSCSI initiator inside the VM. Then my coworker thought we could enable MPIO using two virtual Nics with vmxnet3. We tied each vmxnet3 nic to a separate port group and assigned one of the 2 physical NICs to each port group. Additionally vmxnet3 lets you enable jumbo frames and the physical nics were already set to mtu 9000 because this was on the software iscsi vswitch. So we were able to get multiple paths from the VM to the network and have jumbo frames all the way through.

Next we presented the iSCSI volume of 3TB to the Windows machines. Of course at first it sees it as a couple of smaller volumes. Convert the disk to GPT and align to 64k, then format with NTFS. Just like that a 3TB disk inside a Virtual Machine.

Now we saw IOMETER push better sequential IO than an RDM that was set up for Round Robin, but not quite as good in the Random IO department as a RDM.

The main gain here is to get a file bigger than 2TB minus 512B. Useful for the scan/image servers that store a tons of files for a long time.

To sum up and make it clear.

1. Use the Microsoft iSCSI initiator and MPIO inside the VM when you need more than 2TB on a disk.

2. Use 2 port groups and bind them to separate physical nics to let the MPIO actually work over 2 nics.

3. With vSphere use the VMXNET3 driver for network to use jumbo frames, the E1000 driver does not support this.

VMworld 2009 – In-n-Out Burger Run

After you run the 10k on Monday evening what else do you have to do in the evening at VMworld? I know many of you will be shining your shoes and getting your top hats ready putting on your Ball Gowns for some big vendor party. Before that though, on Tuesday or Wednesday at 5:30 or 6 pm I will try to lead an expedition to a California classic, In-n-Out burger. For those of us exiled from California it is common to make In-n-Out a destination when we visit. Sound like you are a native Californian and order something not on the menu. Like a “4×4 Animal Style” – 4 beef patties, 4 pieces of cheese and all the onions and whatnot grilled to perfection.

So let me know if you are interested and while try to organize a meetup and trolley/train/cab/bus/walk over.

Things a VMware Consultant should not Tweet.

So I don’t have any real content today. So I will make up things you should not tweet from a customer site.

1. @duncanYB How do I add a host to vCenter again?

2. Just overwrote dudes LUNS, hope there wasn’t anything on them.

3. Licenses? Everyone can use my file.

4. Just P2V’d dudes server in the middle of the day. Hope no one noticed the reboot.

5. iSCSI, Service Console, VM Network, DMZ, VMotion. All on one NIC Sweet!
(don’t ruin the effect by commenting about 10GigE)

6. Exchange and SQL on SATA? No better use for a single 1 TB drive!

7. Just finished my first ESX install on a real server. Faster than it was in Workstation.

8. @Texiwill is there a way to blank the root password for the Service Console?

9. Failover is for dummies that don’t do it right the first time.

10. Can’t wait to leave here so I can finish building my life size Yoda robot.

Add your suggestions in the comments or tweet them to me @2vcps