mainesysadmin.com

This vague message turned into a nightmare for myself this past Sunday.  I was able to get the problem fixed with minimal downtime as the issue only came up during a Sunday maint. window (ironic).

I had patched my ESX hosts to Update2 (patched version) and started receiving the error “VMware HA Agent has an error” in my virtual infrastructure client.  First step is to make sure you’re running a compliant version of Virtual Center with your ESX.  Ok, Check.

Next check your host files in each esx host located here:  /etc/hosts

You should see the default line of 127.0.0.1 for localhost, below there needs to be at least one linewith the host your working on.

192.168.1.103     esx9.yourdomain.com     esx9

Over the past 3 years I’ve seen conflicting guidance over the configuration of the host files .  In 3.0.2 I was required to enter all my hosts in each host file, but now that seems to be resolved and only requires the local details of the host you’re working with.  In short make sure that you can ping your hosts from each host with the FQDN and/or friendly name and you should be all good there.

The next steps were found on some vmtn.net postings, and I lost the URL amongst all my troubleshooting but I saved the resolution.  For myself I found the following steps helpful in resolving the problem….

The problem in my situation was specifically around the Virtual Center agents and conflicting versions.

• Check the vpxa version on your host:  rpm -qa |grep vpxa
• That will give you the current version of vpxa that you’re running. eg:  VMware-vpxa-2.5.0-104215
• Stop the VMware mgmt service:  service vmware-mgmt stop
• Stop the vpx agent:  /etc/init.d/vmware-vpxa stop
• The vpx agent error can be ignored (warning: /etc/vmware/vpxa.cfg saved as /etc/vmware/vpxa.cfg.rpmsave)
• Remember your vpx version from the first step and use it here to remove the vpx agent.
• rpm -e VMware-vpxa-2.5.0-104215
• Switch over to your Virtual Center client and remove the host you just modified (guest vms will remain)
• Reboot the host (vm’s will go down)
• After boot reconnect the host to VC and the latest vpx agent will be intstalled.
• Enable HA and the error should disappear.

Luckily my event occurred on a Sunday.  This was the first time that the guest vm’s experienced unplanned downtime without being able to vmotion in 3 years.  Be careful with those VMware patches!

Scott featured a nice write up comparison of ESX nic utilization and vSwitch configuration the other day.  The article is specifically around the Guest VM vSwitch and compares the differences between Link Aggregation and the opposite.  It’s a useful reminder of the load balancing characterists of the different settings in the vSwitch.

This was a two part series on his webpage:

Part 2: http://blog.scottlowe.org/2008/10/08/more-on-vmware-esx-nic-utilization/

Part 1: http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/

The purpose of this shootout was to analyze already backed up images of VM guests sitting idle on disk. We like to keep 7 days online, so I was curious as to how the two compared in our environment.

Compressed vs Deduplicated

COMPRESSED: (backed up via vRanger Pro over VCB, size on win NTFS volume)
citrix_server = 4.29gb
win_sql_server_64bit = 9.76gb
citrix_server2 = 6.14gb
win2k3_member = 4.97gb

compressed vm total = 25.16gb

UNCOMPRESSED:

citrix_server = 10gb
win_sql_server_64bit = 20gb
citrix_server2 = 10gb
win2k3_member = 35gb

uncompressed total = 75gb

DEDUPED VS COMPRESSED:

deduped     = 54.29% savings (NetApp cmd output below)
compressed     = 66.45% savings

Here’s the actual output from the NetApp find_space exe:
A-SIS Deduplication Space Savings estimate.

Name of Fingerprint File:          d:\esx_deduped.txt
Total Number of Directories:       5
Total Number of Files:             58
Total Number of 4K Blocks:         19662390
Total Number of Duplicate Blocks:  10673928
Percentage of Duplicate Data:      54.29
Scan Start Time:                   Wed, Jul 30 2008 3:40:16 PM
Scan End Time:                     Wed, Jul 30 2008 4:22:25 PM

I’ll be honest, I was expecting deduplication to blow compression out of the water even with only 4 vm’s. I’ll go out on a limb and venture that deduplication would provide more results as you toss more vm’s into the picture.

I’ve been doing a little testing with NetApp Deduplication lately but only in a limited test environment.  I know one thing for certain, deduplication and compression do not mix at all!  When you start to look at this technology make sure you’re dealing with uncompressed data.  This posting is more or less a bookmark for:

“Scott’s Quick Guide to Setting up Netapp Deduplication”

NetApp does provide a useful windows exe called “find_space” which you can run in report mode against any windows data volume.

I came into work this morning and one of the ESX hosts had HA disabled.  The network engineers replaced the core switch and I have to assume the NIC connectivity caused HA to disable since they were all down at once.  One command helped me get HA enabled:

SSH to the host that is disabled and as root execute:

service mgmt-vmware restart

Then enable HA after successfully running the command.

I was reading the old vmblog.com today and ran across a new tool that appears useful. I’m going to be upgrading from 3.0.2 -> 3.5 soon and I can see an immediate use for this tool. RVTools can be used to read the vmware tools version installed on all your Vmware guest servers if you point the tool at your Virtual Center server. There are a few other fields that are brought down as well. I use the notes field to populate backup information for the guest and that came down nicely with the tool.

After you read the version installed you can even select which VM’s if any you would like to upgrade which looks very convenient. Here’s the post I read…

http://vmblog.com/archive/2008/05/12/3rd-party-rvtools-for-vmware-updated-to-version-1-1.aspx

or go to the horse’s mouth right here:

http://rvtools.deveij.com/

Use putty or ssh to connect one of the hosts that can access the files of the vm guest you’d like to work with.  Browse to the location of the vm guest on your SAN or disk.

Only 1, yes one command on the vmware side to extend or expand the disk.  Please power your VM guest down before attempting this…

vmkfstools -X 15G disk1.vmdk (where 15G is the final size you want the disk to expand too).

Now if this is the O/S drive for a guest VM you’ll need to use the VI3 client to attach the VMDK to another VM as a second hard drive.  This is only for a few minutes so that you can run diskpart from within windows.  Diskpart is not allowed to run on an O/S partition.

After you’ve added the drive as a second drive to another helper VM you can run the following commands.  If the drive is already a second drive you don’t have to add it to a helper VM.

Open a windows cmd prompt and enter the following…

1)diskpart.exe
2)list disk
3)list volume
4)select volume=#   (where # = the volume you need to expand)
5)extend

Don’t forget to detach the vmdk from your helper vm!