Today I had an issue here an ESX host became unresponsive in vCenter, yet the VMs that were running on the host were fine. The normal remedy for this issue would be to restart the management agent on the ESX host via the Service Console:
However, this did not work. The mgmt-vmware restart command hung while stopping the "VMware ESX Server Host Agent". Ten minites after executing mgmt-vmware restart, I decided to break out of the process by pressing Ctrl+z.
Clearly, there was a problem with the existing running instance of the management agent, vmware-hostd. The only way to get this working without a host reboot, is to find the PID for vmware-hostd and kill it:
To locate the PID for the running vmware-hostd process execute:
ps -auxwww |grep vmware-hostd
You will see output similar to: (I've marked the PID in BOLD text)
root 13089 1.3 2.6 179080 6988 ? S 2008 1695:23 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u
To kill the running process, execute:
kill -9 <PID> (I had to run "kill -9 13089")
Once vmware-hostd is no longer running, you can restart the management agent by running:
service mgmt-vmware restart
For those of you going to VMworld Europe 2009, spare some time to visit the Veeam Software booth at VMworld Europe this year. Veeam is a leader in the virtual systems management market and I'm sure there will be some interesting solutions on display. Their booth will be #56, so be sure to drop by.
About Veeam Software:
Veeam Software, a VMware Technology Alliance Premier partner, helps organizations safeguard their investment in virtual infrastructure by providing innovative systems management software designed to reduce costs, increase productivity and mitigate risk.
Veeam offers a full suite of VMware management tools, including Veeam Backup, the 2-in-1 backup and replication solution; Veeam Reporter, to document virtual environments for capacity planning and chargeback; Veeam Configurator, to manage “configuration drift;” and Veeam Monitor, for performance monitoring and alerting across multiple VirtualCenters.
With its acquisition of nworks, Veeam expanded its product line to include connectors that incorporate VMware events, status and performance data into Microsoft System Center Operations Manager and HP Software Operations Manager. This product set includes the Smart Plug-in for VMware - an HP Certified SPI; and the Management Pack for VMware - for Microsoft System Center Operations Manager. These innovative connectors solve the need of large organizations to smoothly integrate VMware into their enterprise-wide systems monitoring architecture.
It's been a while now that I've been wanting to start a Virtualization Wiki Project. There's so much information available on virtualization and I think it would be great if it can all be in one simple location, LOL! There are numbers of virtualization vendors about now with many different products to choose from. We can't all be experts on every single virtualization platform. There's just not enough time in a day to learn and know everything.
This is why I've decided to finally go ahead and start an Open Virtualization Wiki, a site where anyone can write information on virtualization, not just on VMware but on any virtualization technology known to man kind! I've just started the site up on my web server a couple of weeks ago, but didn't really have the time to fill it with loads of information yet. Originally, I wanted to wait until there was enough information posted on the wiki before I changed the DNS settings to point to my server, but I've decided to make it live now anyway. The wiki is based on MediaWiki, the engine originally writen for Wikipedia.
Feel free to have a look at the wiki and please update it as you like. It needs your help!
You can find the wiki at http://www.vi-pedia.com. If you're unable to access the site, give it some time, we need to wait for DNS to update around the world.
I thought that it would be a good idea to post an update on my progress of the book/whitepaper/guide that I am writing on building a low cost VI lab. Life has been very busy in the past couple of weeks. I have therefore not been able to post a lot of articles to this site. I normally make a note of interesting things I find and then blog on them the same day. Lately I've been so busy (and on holiday last week I must add) that when I finally decide to blog on them, it's old news and not worth blogging over again. Most things are covered by Scott Lowe, Mike Laverick, Jason Boche and other top bloggers anyway.
For now I've decided to concentrate on the low cost lab book. Writing the book is taking longer that what I first anticipated because I'm doing a lot more testing than what I first thought would be necessary. But this is a good thing, as it's against my beliefs to state anything technical as facts if not tested. I don't like theories, I like facts. To also make sure that everything in the book is as true and accurate as possible, I will be passing it to a couple of peers for a peer review before it's released for download, so please bear with me.
Originally I only wanted to write a proper blog post on "Building a low cost VI lab", but I soon realised that it would be too much to fit in a single blog post, so I thought I could write a document on it. Now it's turned into a little book, rather than just a document. Most of the easy stuff is done, including hardware options and ESX installation and configuration options for non-HCL hardware and basic network configuration tasks. I've also completed most of the storage and shared storage sections of the book, but I've now turned my attention to fine-tuning iSCSI. I will then focus on advanced network configurations, like setting up secure hosting with DMZ configurations using cheap non-managed switches.
I have also decided to include service console commands for every change made within the VI Client. Also, as the whole point of the book is to illustrate a "low cost lab", I will cover configurations for both ESX 3.5 as well as ESXi 3.5 installable.
I'm just fighting the urge to start on the ESX 4 version, but I think it's best to finish this one for ESX 3.5 first. I don't really want to say "THIS BOOK WILL BE AVAILABLE ON SO AND SO DATE", but I don't like making promises that I can't keep. I do however thing the end of Feb should be a good time for the peers to have their review of the book. And yes, you will be able to download a free, full version of the book.
Thank you for your patience, I'll keep you posted.
I know it's got nothing to do with virtualization, but I felft like posting it anyway. The South of England is covered in snow this morning. We are about to leave on holiday for the week, but I now realise that it's going to be an interesting drive! Here's a picture from just outside my house in Hertfordshire:
I've come across an insanely brilliant blog post on iSCSI. One of the best technical blogs I've seen in a long time. Actually one of the best technical blogs I've ever seen! I took some time out and sat down with a mug of coffee to read the post in detail. I found this blog post every interesting and in good time as I've actually started on the iSCSI shared storage section of my book on "Building a Low cost VI Environment" just yesterday.
Ok, I do have to say that a lot of the stuff in the blog post is "too heavy" for my new book as the aim for my book is to build a low cost lab VI environment, not an enterprise VI environment. However I do think that a lot of what they have in the blog post can be used to optimize even a low cost lab environment. I've been using Openfiiler for a while now and I will certainly be looking at improving my home VI environment's iSCSI implementation. In my new book, the shared storage section is based iSCSI provided by Openfiler 2.3. I will definitely look into improving the iSCSI section of the book by looking into some things like "jumbo frames" mentioned in the blog post!
Thanks to Chad Sakac (EMC), Andy Banta (VMware), Vaughn Stewart (NetApp), Eric Schott (Dell/EqualLogic), and Adam Carter (HP/Lefthand) and David Black (EMC) for this great post.
Administrators that's got some experience in running Linux guest operating systems in virtual machines, may know that the time of the guest OS can drift by several hours per day. Normally, in my experience the time in the guest OS runs too fast. This can have serious implications on some services that these servers provide, such as Mail/Spam relays and web servers.
VMware has now released a very good KB article on best practises for Linux Guest OS time keeping, with a table of kernels and their parameters.
The KB article is a must read for those who have Linux guests. You can find VMware KB Article 1006427 here
VMware has a new KB up on how to "Verify the health of ESX Server Operating System". The symptoms that the KB Article covers are:
- The ESX host seems slow
- There are a lot of errors in the log files
- Accessing the ESX host over a SSH session is slow
- Operations performed on the ESX host time out
These issues are now covered by KB Article 1004019
I'm currently writing a document that I will publish soon on building a low cost lab with VI3. While I was doing some research on VMFS partition alignment, I found that I was unable to create (or delete) partitions on the /dev/sda disk that ESX is installed on. I could make the changes, but as I tried to write the changes to disk, fdisk would come back with the following error:
SCSI disk error : host 1 channel 0 id 0 lun 0 return code = be0000
I/O error: dev 08:00, sector 40355342
Device busy for revalidation (usage=8)
I know that I could just use the VI client to create this partition, but as I'm busy writing this fairly detailed document, I wanted to show potential readers how to create such an aligned local VMFS3 partition using fdisk and vmkfstools on the /dev/sda device.
Then I also had the question in my mind... "If the VI client can do it, why can't I?"
Anyway, after playing around with ESX a little, I found a fix. Use with caution though!
You need to remove the lock that ESX has on the device! The esxcfg-advcfg command can do this.
Before running fdisk, execute the following command at the service console:
esxcfg-advcfg -s 0 /Disk/PreventVMFSOverwrite
With PreventVMFSOverwrite now switched off, you can use fdisk to write the partition changes without the error. You will still get "device or resource busy" but it will write the changes.
After you have saved the partition changes using fdisk, run the following commands:
esxcfg-advcfg -s 1 /Disk/PreventVMFSOverwrite
In the past year, I’ve been doing a lot of testing on running VMware ESX 3.5 on non-supported hardware (hardware that is not included on the VMware HCL). I’ve now got my home lab to an acceptable stage where I can do testing and studying. I’ve spent many hours on forums and blogs trying to find components that are cheap, quiet, fast-enough and most importantly, components that work with ESX 3.5!
Now that I’ve been through the trouble of finding out what works and what doesn’t, I thought it may be a good idea to blog on it and give others that would like to run ESX on cheap hardware a heads up before they go and waste money on stuff that does not work!
Although, I have blogged on this topic a couple of times before in brief, but I think it’s time for a proper post that describes the trails’ and tribulations of building a cheap ESX environment in more detail, with all the do's and don't's!
I've started writing the article as a blog, but I now realise that because of the amount detail I'm putting into it, it's going to cover a lot of pages. I therefore decided to blog some of it on the site, as well as provide a full ebook in PDF format which will contain all the information you'll ever need on running ESX on low cost hardware.
I'm hoping to have the eBook ready for download by early next week.
Thanks for reading!