mainesysadmin.com

This vague message turned into a nightmare for myself this past Sunday.  I was able to get the problem fixed with minimal downtime as the issue only came up during a Sunday maint. window (ironic).

I had patched my ESX hosts to Update2 (patched version) and started receiving the error “VMware HA Agent has an error” in my virtual infrastructure client.  First step is to make sure you’re running a compliant version of Virtual Center with your ESX.  Ok, Check.

Next check your host files in each esx host located here:  /etc/hosts

You should see the default line of 127.0.0.1 for localhost, below there needs to be at least one linewith the host your working on.

192.168.1.103     esx9.yourdomain.com     esx9

Over the past 3 years I’ve seen conflicting guidance over the configuration of the host files .  In 3.0.2 I was required to enter all my hosts in each host file, but now that seems to be resolved and only requires the local details of the host you’re working with.  In short make sure that you can ping your hosts from each host with the FQDN and/or friendly name and you should be all good there.

The next steps were found on some vmtn.net postings, and I lost the URL amongst all my troubleshooting but I saved the resolution.  For myself I found the following steps helpful in resolving the problem….

The problem in my situation was specifically around the Virtual Center agents and conflicting versions.

• Check the vpxa version on your host:  rpm -qa |grep vpxa
• That will give you the current version of vpxa that you’re running. eg:  VMware-vpxa-2.5.0-104215
• Stop the VMware mgmt service:  service vmware-mgmt stop
• Stop the vpx agent:  /etc/init.d/vmware-vpxa stop
• The vpx agent error can be ignored (warning: /etc/vmware/vpxa.cfg saved as /etc/vmware/vpxa.cfg.rpmsave)
• Remember your vpx version from the first step and use it here to remove the vpx agent.
• rpm -e VMware-vpxa-2.5.0-104215
• Switch over to your Virtual Center client and remove the host you just modified (guest vms will remain)
• Reboot the host (vm’s will go down)
• After boot reconnect the host to VC and the latest vpx agent will be intstalled.
• Enable HA and the error should disappear.

Luckily my event occurred on a Sunday.  This was the first time that the guest vm’s experienced unplanned downtime without being able to vmotion in 3 years.  Be careful with those VMware patches!

For a task that is REALLY easy it took a more googling than I had anticipated to come up with the solution.  Once your volume has been delivered to the server it is important to initialize the disk but not to create any partitions.

1. Open Windows Disk management.
2. Click Action -> Rescan disks to make your new disk appear, if greater than 2tb it will appear as 2 partitions on 1 disk
3. Right click on the disk and initialize the disk
4. Open a windows command prompt
5. type “diskpart”
6. type “list disk”
7. select the new disk that’s over 2tb
8. convert gpt
9. Now from windows Disk Management you should see a single disk greater than 2tb

Information on the diskpart utility:   http://support.microsoft.com/default.aspx/kb/300415

The most important prereqs include Windows 2k3 is up to at least sp1 and that no partitions exist on the disk.

I came into work this morning and one of the ESX hosts had HA disabled.  The network engineers replaced the core switch and I have to assume the NIC connectivity caused HA to disable since they were all down at once.  One command helped me get HA enabled:

SSH to the host that is disabled and as root execute:

service mgmt-vmware restart

Then enable HA after successfully running the command.

If you’d like to delete a service from your view from within windows server 2003 you can use the following commands.

First top the service:

net stop “service name”

the next command will delete it…

sc delete “service name”
***The “service name” isn’t always the name listed under names when you bring up the services console gui.  Select the service you’d like to delete and review the properties.  The actual windows registry service name will be listed at the top of the properties sheet for that service.

Use putty or ssh to connect one of the hosts that can access the files of the vm guest you’d like to work with.  Browse to the location of the vm guest on your SAN or disk.

Only 1, yes one command on the vmware side to extend or expand the disk.  Please power your VM guest down before attempting this…

vmkfstools -X 15G disk1.vmdk (where 15G is the final size you want the disk to expand too).

Now if this is the O/S drive for a guest VM you’ll need to use the VI3 client to attach the VMDK to another VM as a second hard drive.  This is only for a few minutes so that you can run diskpart from within windows.  Diskpart is not allowed to run on an O/S partition.

After you’ve added the drive as a second drive to another helper VM you can run the following commands.  If the drive is already a second drive you don’t have to add it to a helper VM.

Open a windows cmd prompt and enter the following…

1)diskpart.exe
2)list disk
3)list volume
4)select volume=#   (where # = the volume you need to expand)
5)extend

Don’t forget to detach the vmdk from your helper vm!

While working with vmware support on the phone they required me to run a job from Virtual center to gather diagnostic log data from our VI3 cluster.  You can try this by opening your, “Virtual Infrastructure Client” and selecting the “Administration” menu and then “Export Diagnostic Data”.  When I run this command for as little or much information required I ALWAYS get the following error, “Failed to create diagnostic bundle.”

If you ssh to your host directly you can run the following command from each host and sftp the logs from the host afterwards.

vm-support

From a windows command prompt you can get a shutdown gui.  This is cool because you can use it to execute remote shutdowns with ease and not have to remember all the flags.

start->run->  cmd

c:\shutdown.exe -i

The gui should be launched.  I find it handy when a server is supposed to reboot and gets “stuck”.  Some windows servers just don’t shutdown cleanly and hang on the process.  If you execute another remote shutdown command you can force the server out of the endless shutdown, and save yourself from a drive into the server room.