I've seen cases where a newly extended VMFS datastore fluctuates between the old size and the new size. Virtual Machines running on th newly extended datastore prevents the correct size of the datastore from displaying.
I found the solution to the problem in KB Article 1002558
Products: VMware ESX and ESXi
The solution is:
- Shut down or VMotion virtual machines running on the extended datastore to the ESX host that created the extent.
- On all other ESX hosts other than the one that created the datastore, run vmkfstools -V to re-read the volume information.
- Power on or VMotion the Virtual Machines back to the original ESX hosts.
This is not only on VI3 hosts but on:
VMware ESX 2.0.x
VMware ESX 2.1.x
VMware ESX 2.5.x
VMware ESX 3.0.x
VMware ESX 3.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
Now this may be old news, but I decided to blog it anyway for my own reference.
Simon Long has written a good article on how to improve VMotion performance when performing mass migrations. This is very handy when you are putting a host into maintenance mode. Thanks to Jason Boche for blogging it first!
I'll set the scene a little...
I’m working late, I’ve just installed Update Manager and I‘m going to run my first updates. Like all new systems, I’m not always confident so I decided “Out of hours” would be the best time to try.
I hit “Remediate” on my first Host then sat back, cup of tea in hand and watch to see what happens….The Host’s VM’s were slowly migrated off 2 at a time onto other Hosts.
“It’s gonna be a long night” I thought to myself. So whilst I was going through my Hosts one at time, I also fired up Google and tried to find out if there was anyway I could speed up the VMotion process. There didn’t seem to be any article or blog posts (that I could find) about improving VMotion Performance so I created a new Servicedesk Job for myself to investigate this further.
3 months later whilst at a product review at VMware UK, I was chatting to their Inside Systems Engineer, Chris Dye, and I asked him if there was a way of increasing the amount of simultaneous VMotions from 2 to something more. He was unsure, so did a little digging and managed to find a little info that might be helpful and fired it across for me to test.
After a few hours of basic testing over the quiet Christmas period, I was able to increase the amount of simultaneous VMotions…Happy Days!!
But after some further testing it seemed as though the amount of simultaneous VMotions is actually set per Host. This means if I set my vCenter server to allow 6 VMotions, I then place 2 Hosts into maintenance mode at the same time, there would actually be 12 VMotions running simultaneously. This is certainly something you should consider when deciding how many VMotions you would like running at once.
Here are the steps to increase the amount of Simultaneous VMotion Migrations per Host.
1. RDP to your vCenter Server.
2. Locate the vpdx.cfg (Default location “C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter”)
3. Make a Backup of the vpdx.cfg before making any changes
4. Edit the file in using WordPad and insert the following lines between the <vpdx></vpdx> tags;
5. Now you need to decide what value to give “maxCostPerHost”.
A Cold Migration has a cost of 1 and a Hot Migration aka VMotion has a cost of 4. I first set mine to 12 as I wanted to see if it would now allow 3 VMotions at once, I now permanently have mine set to 24 which gives me 6 simultaneous VMotions per Host (6×4 = 24).
I am unsure on the maximum value that you can use here, the largest I tested was 24.
6. Save your changes and exit WordPad.
7. Restart “VMware VirtualCenter Server” Service to apply the changes.
Now I know how to change the amount of simultaneous VMotions per Host, I decided to run some tests to see if it actually made any difference to the overall VMotion Performance.
I had 2 Host’s with 16 almost identical VM’s. I created a job to Migrate my 16 VM’s from Host 1 to Host 2.
Both Hosts VMotion vmnic was a single 1Gbit nic connected to a CISCO Switch which also has other network traffic on it.
The Network Performance graph above was recorded during my testing and is displaying the “Network Data Transmit” measurement on the VMotion vmnic. The 3 sections highlighted represent the following;
Section 1 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 6 simultaneous VMotions.
Time taken = 3.30
Section 2 - This was not a test, I was simply just migrating the VM’s back onto the Host for the 2nd test (Section 3).
Section 3 - 16 VM’s VMotioned from Host 1 to Host 2 using a maximum of 2 simultaneous VMotions.
Time taken = 6.36
Time Different = 3.06
3 Mins!! I wasn’t expecting it to be that much. Imagine if you had a 50 Host cluster…how much time would it save you?
I tried the same test again but only migrating 6 VM’s instead of 16.
Migrating off 6 VM’s with only 2 simultaneous VMotions allowed.
Time taken = 2.24
Migrating off 6 VM’s with 6 simultaneous VMotions allowed.
Time taken = 1.54
Time Different = 30secs
It’s still an improvement all be it not so big.
Now don’t get me wrong, these tests are hardly scientific and would never have been deemed as completely fair test but I think you get the general idea of what I was trying to get at.
I’m hoping to explore VMotion Performance further by looking at maybe using multiple physical nics for VMotion and Teaming them using EtherChannel or maybe even using 10Gbit Ethernet. Right now I don’t have the spare Hardware to do that but this is definitely something I will try when the opportunity arises.
As we are now busy doing P2V conversion of most of our physical servers I needed to use the Consolidation function in vCenter (previously known as VirtualCenter) to assess the workload of the physical servers (CPU and Memory Usage).
I was surprised to see that the "Consolidation" button disappeared from my VI Client! I checked if it was installed as a plugin and it was. I also checked if it was enabled and again, it was. However, the button was not there!
I found that although the plugin can be installed and enabled in the VI client, it is also a requirement to have VMware Capacity Planner installed on the vCenter server. I checked my vCenter server and confirmed that it was indeed already installed.
Here's the fix:
It seems like the Consolidation Function for vCenter is disabled by default in the vpxd.cfg file. To enable the consolidation function, do the following:
On the vCenter server, edit c:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg
(I'm still using VirtualCenter 2.5.0 Build 119598, so your path could be ...\vCenter\cpxd.cfg? I'm not sure!)
You will find the following three lines:
Change them to read:
Restart the VirtualCenter service using the Services.msc MMC snapin in Windows. The Consolidation Button should now be available in your VI, Providing that that it's installed and enabled and that VMware Capacity Planner is installed on the vCenter server.
I’ve had a quick play with Veeam Monitor for VMware Infrastructure 3 Free Edition. Veeam Monitor looks like a good product and should come in handy if you have lots of things in your VI to monitor. It is better than the performance monitors in VMware vCenter, but then again it doesn’t take much to beat vCenter’s performance counters in detail.
I’ve installed Veeam Monitor on a Windows Server 2003 box. The installation wizard will give you an option to choose an existing SQL Database Server or to install SQL Express locally. I opted to install it on an existing SQL server. If you’re going to run Veeam Monitor in a production environment, I think it will make more sense to go with this option as well. The installation then went on and automatically created a new database on my SQL server called “Monitor”, so no need to create a database before kicking off the installation.
At first glance, Veeam Monitor looks and feels like a professional application. The GUI is clean and not cluttered with an overload of information. The graphs are easy to read and understand.
Veeam Monitor allows you to add a standalone ESX Server or a VirtualCenter or “vCenter” to monitor. In my case, I pointed it to my vCenter Server and as expected, it pulled the entire VI configuration from the VC, complete with Datacenters, Clulsters, Hosts and Resource Pools.
Now, let’s look at the information it returns. For this article I’m only going to mention the information it returned for a single “Datacenter”, defined in my VC. However, Veeam Monitor will report at a Datacenter, cluster, host, resource pool and Virtual Machine level. When looking at a virtual machine level, Veeam monitor will allow you to access the VM’s console as well as the processes inside the VM’s Guest OS** (**However the processes feature is not included in the Free Edition).
So at the datacenter level, it will return the following sets of information: Summary, Overall, CPU, Memory, Networking, Disk, Swap, Tops, Alarms and Events.
Here’s what they'll all tell you:
The Summary tab provides:
|Events Count||The number of events logged for the Datacenter|
|Performance data period||The time period that performance data are reported on|
The Overall tab provides graphs for:
CPU Usage, Memory Consumed, Network Usage, Disk Usage and Swap Usage.
The CPU, Memory, Networking, Disk and Swap tabs provide information using 3D graphs (providing you selected 3D in the Veeam Monitor Client Options) on datacenter wide utilization. Much the same as vCenter does. It just looks a lot better and easier to read. As expected, you also have the option to select the counters that you would like to see for each graph.
Sorry that I’ve jumped past the CPU, Memory, Networking, Disk and Swap tabs without providing much detail, but it really is very similar to vCenter, at a datacenter level anyway. The Tops tab is the one I really liked. It displays the 3 Virtual Machines that are the top resource users for the last 5 minutes. They are showed as follows:
• Top CPU Users in Percentage
• Top Network Users in KBps
• Top Memory Users in MB
• Top Swap Users in MB
• Top Disk Users in KBps
The Alarms tab displays the alarms defined in VC for the selected VI object. It also shows a history of triggered alarms and gives you the option to create new alarms as well. Another function is to resolve a triggered alarm by adding a comment to the alarm. There’s also a button to clear triggered alarms when “Resolved, later than 1 day, later than 1 week, later than 1 month or all”.
Although I didn’t give an in-depth review of Veeam Monitor (Because I just don’t have all the time in the world right now), I do think it’s a good product.
Just note that there are a few features that are not available in the Free Edition. The ones I came across are:
• No Alarm Modelling
• No reporting
• Only 24 hours of history displayed
• No access to the Guest OS Processes running inside the VM.
All in all, as with Veeam FastSCP, Veeam has got my thumbs up for Veeam Monitor!
Here's an interesting article on the demand for virtualization in the data centre. I've copied a paragraph from the article here, as I would like to throw a comment out on it:
"VMware remains the market leader but Microsoft's Hyper-V is closing the gap. In response to a question in which we asked customers to select ALL the virtualization products they plan to use/deploy in 2009, nearly 60% said they plan to deploy VMware's ESX Server (29%) or the free VMware Server (30%). However, 52% of the respondents indicated they will use Microsoft's Hyper-V or the older Microsoft Virtual Server; 37% of respondents indicated they will opt for Hyper-V compared with the remaining 15% who said they will use the older Virtual Server. This percentage is double the number of survey respondents who indicated they would use Microsoft hypervisor solutions when we polled users in a 2007 survey"
Well, I'm not too sure about those figures. I mean come on... 52% of the respondents will use Microsoft Hyper-V? Oh no, sorry it's 37%. We'll have to see how long they stick to that. Honestly, I've not used Hyper-V in a production environment myself, so I can't really comment, but from what I've heard................................. well let's just see what happens. BTW. Why do I never get forwarded one of those surveys??
Anyway, read the full article here:
HEYHEY!! I've just received my first Christmas gift!
This one is from Veeam! I've been waiting for this gift to arrive, as I've registered for the gift a few days/weeks ago.
I'm glad to announce that I've unwrapped it and that it contained Veeam Monitor, "FREE Real time Monitoring for your ESXi".
Apparently the rest of the world that has not registered for the product will hear about it in a few hours. Just hang in there!
The news is out that VMware has decided to rename what would have been VMware Virtual Infrastructure 4 or VI4 to VMware "vSphere". This is the latest of a series of product name changes that VMware has been making in the past couple of weeks. A while ago, VMware renamed VMware Virtual Desktop Manager to VMware View.
I can't really figure out why they went for the name vSphere. The only reason I can think of is that maybe they felt the need to rename the product to try and make it sound as if it includes the a sphere of products for the entire data centre? Anyway, we just better get used to it. VI4 is out and vSphere is in.
VMware has also announced that vSphere's maximum host RAM will be 1TB and 250GB for the Guest's RAM. A nice addition, however I think it will be a while before we get our servers to have 1TB RAM.
VMware and HP are to integrate HP's BTO (Business Technology Optimization) software with VMware vCenter Lab Manager.
This will support VMware’s contemplated VDC-OS (virtual datacenter operating system) by integrating HP Discovery and dependency Mapping, BTO’s application and infrastructure discovery technologies.
HP’s BTO helps you make sure that every dollar invested in IT, every resource allocated and every application in development or production meets your business goals.
It is an approach to IT management that reduces costs and produces business outcomes by helping your IT leaders:
- Allocate IT spend and resources based on business priorities
- Automate key processes across IT strategy, applications and operations
- Measure IT effectiveness and efficiency from a business perspective
VMware’s VDC-OS is aimed at enabling the use of all types of hardware, including servers, storage and networks in an internal enterprise cloud that acts like one super computer that could automatically move workloads to external clouds when additional capacity is needed.
There is a problem with the with HA in VMware ESX 3.5 Update 3. Virtual Machines may reboot unexpectedly when migrated with VMotion or after a Power On operation. This is only when the Virtual Machine is running on an ESX 3.5 Update 3 Host and the ESX Host has VMware HA enabled with "Virtual Machine Monitoring" option active.To work around this problem:
Option 1: Disable Virtual Machine Monitoring
1. Select the VMware HA cluster and choose Edit Settings from the right-click menu.
2. In the Cluster Settings dialog box, select VMware HA in the left column.
3. Un-Check the Enable virtual machine monitoring check box.
4. Click OK.
Option 2: Set hostd hearbeat delay to 0
1. Disconnect the host from VC (Right click on host in VI Client and select "Disconnect" )
2. Login as root to the ESX Server with SSH.
3. Using a text editor such as nano or vi, edit the file /etc/vmware/hostd/config.xml
4. Set the "heartbeatDelayInSecs" tag under "vmsvc" to 0 seconds as shown here:
5. Restart the management agents for this change to take effect.
service mgmt-vmware restart
6. Reconnect the host in VC ( Right click on host in VI Client and select "Connect" )
To install the VMware tools on a Netware 6.5 Virtual Machine follow these easy steps:
1. Power on the virtual machine.
2. Choose VM > Install VMware Tools.
3. Inside the guest OS,Load the CD-ROM driver if it's not already loaded: In the system console for a NetWare 6.5 virtual machine, type LOAD CDDVD
4. Run setup.ncf from the VMWTOOLS volume. Type: vmwtools:\setup.ncf
5. Restart the guest operating system. In the system console, type restart server
If the VMware Tools install ISO is still mounted, click: VM > End VMware Tools Install