Quick Tip: Cannot delete ISO on VMware datastore

James Green · Aug 12, 2015 ·

There’s a VMware KB article that talks about trying to delete a file from a VMFS datastore, only to find that it’s locked by a particular ESXi host. The article walks through identifying which host as the file locked via a message in the vmkernel log file. By parsing the log message, one can deduce the MAC address of the host locking the file. From there the lock can be removed and the file deleted. The message in the logs looks something like this:

cpu6:5455056)FS3: 1393: vol ‘datastore_name‘, lock at 173475840: [Req mode: 1] Not free:

cpu6:5455056)[type 10c00004 offset 173475840 v 158, hb offset 3338752gen 4307, mode 1, owner 51b23791-f35c96e9-44ed-3c4a92f61680 mtime 2278287

cpu6:5455056)Res3: 5696: Rank violation threshold reached: cid 0xc1d0000c, resType 3, cnum 1410

The bolded section is a MAC address and will help chase down the host in question. But there’s something I’ve run into a number of times when trying to get a file deleted that I thought I’d share. When trying to delete an ISO (specifically an ISO) from a datastore, you’ll get the same error message about the file being locked. You search for the error you’re having and the KB article I referenced turns up. So you go look at vmkernel.log and find this:

2015-08-12T13:19:21.219Z cpu4:3476013)[type 10c00001 offset 201531392 v 18641, hb offset 3956736

gen 1051, mode 2, owner 00000000-00000000-0000-000000000000 mtime 44516

Now, what in the world does one do when the MAC address portion is empty? Well, after banging my head against my desk for about 5 minutes more times than I care to admit, I usually figure it out. I hope that after posting this I’ll remember that this means the ISO is mounted to a VM somewhere! Duh. If I can get this post to rank well in search engine results, maybe I’ll save someone from 4 minutes of those 5 minutes of head/desk banging agony. 🙂

PowerCLI – Configure Syslog for All Hosts

James Green · Aug 10, 2015 ·

Here’s a quick bit of PowerCLI to configure syslog server on all hosts, place each hosts logs in a unique directory, and then enable the firewall exception to allow it to be used. This will do every host in the vCenter you’re connected to. I highly recommend that anyone managing a vSphere environment set up a syslog destination for ESXi. There’s nothing more frustrating than attempting root-cause analysis on a failure when logs aren’t persistent and in a central location.

Cheers!

[code language=”ps”] #Get all ESXi hosts
$hosts = Get-VMHost

#Update Syslog configuration
$hosts | Set-VMHostAdvancedConfiguration Syslog.global.logHost -Value 0.0.0.0
$hosts | Set-VMHostAdvancedConfiguration Syslog.global.logDirUnique -Value True

#Enable firewall exception
$hosts | Get-VMHostFirewallException | where {$_.Name.StartsWith(‘syslog’)} |
Set-VMHostFirewallException -Enabled $true
[/code]

Vagrant Series » Multi-Machine Environments (Part 6)

James Green · Aug 7, 2015 ·

Vagrant is neat if you need a whip up a machine to quickly test a script or check compatibility on an operating system that is different than what you normally run. And the value of that isn’t to be minimized. But some of the real power of Vagrant happens when you start to stand up full environments (think 3-tier applications or large scale web server farms) in a repeatable, effortless way. In Hashicorp Land, this is specifically referred to as a “multi-machine” environment. Any Vagrantfile that defines and controls multiple guest machines qualifies for this descriptor.

Defining Multi-Machine Environments

Building a Vagrantfile that will configure a multi-machine environment is as simple as defining multiple VMs with the parameter config.vm.define. In the below example nabbed from Hashicorp documentation, one case see that an Apache front end and MySQL back end are both defined.

Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"

  config.vm.define "web" do |web|
    web.vm.box = "apache"
  end
  config.vm.define "db" do |db|
    db.vm.box = "mysql"
  end
end

Once the VMs are defined, configuration takes place by using the inner variable with the same name. In the example, web.anything and db.anything will apply configurations specifically to that machine. Global configurations can still be defined and will apply to all VMs. When configuring a multi-machine environment, it’s important to consider the load order of a Vagrantfile. I covered this in Part 3 of this series, but just to quickly recap, the load order is:

Vagrantfile packaged with the box that is to be used for a given machine.
Vagrantfile in your Vagrant home directory (defaults to ~/.vagrant.d). This lets you specify some defaults for your system user.
Vagrantfile from the project directory. This is the Vagrantfile that you’ll be modifying most of the time.
Multi-machine overrides if any.
Provider-specific overrides, if any.

As the Vagrantfile grows and a large number of machines are defined, it can become a bit of a mess. Scott Lowe wrote about an idea he found over on this blog that breaks all the configuration data out into a separate YAML file. You could also use JSON or XML if you’re more comfortable with that, as in this example.

Accessing Multi-Machine Environments

When using a multi-machine environment, some vagrant commands need to become more specific. If you issue vagrant sshin a multi-machine environment, which machine should be connected to? The commands in a multi-machine environment that only make sense for one machine need to have the machine specified in the command. So in the case of SSH, the command would be vagrant ssh web. In the case of a command like reload, you could issue vagrant reload and reload the whole environment, or you could specify one VM, as in vagrant reload db. Since sometimes one needs to be specific but there’s also a large number of VM’s, the Vagrantfile will also parse regex when it identifies a regular expression for the name, such as /web[A-Za-z0-9]/.

Correct Misaligned VMDK

James Green · Aug 6, 2015 ·

This post details the process for aligning VMDK files properly. There’s a great post from Duncan Epping on why you need to do this here. In these notes, we’re dealing with NetApp storage. Duncan’s post details some other tools for other storage vendors (like UberAlign from Nick Weaver).

Before beginning this process, ensure that:
– VM with misaligned VMDK is powered off
– VM has no snapshots
[Read more…] about Correct Misaligned VMDK

AutoLab 2.6 on Ravello – Build the vCenter & Hosts (Part 3)

James Green · Jul 28, 2015 ·

Part 3 of this series has been a long time coming. I’d been running into some serious problems getting vCenter to build properly, and then I ran out of hours on my promotional hours allotment from the kind folks at Ravello. So I had to wait until I got some fresh hours. To the right, you can see the sorts of problems I was having. My hosts would frequently PSOD either while building, or while booting. I’m really not sure what’s going on, and I’ve been pressed for time, so I didn’t open a case with Ravello to have them check it out. My feeling is that it’s probably just kinks that need worked out from what amounts to what’s been lovingly referred to as hypervisor Inception. (via HVX)

My advice after having sorted through this, which is also in the AutoLab 2.6 Deployment Guide, is to just keep trying. Eventually, I did get it to work as intended, and I didn’t do anything different. It just worked one time. So, after a long wait, here’s a few notes on the steps required to get the hosts and vCenter built.

Build the Hosts

The hosts and vCenter can be built at the same time. The deployment guide says to start them at the same time. Because the hosts are required for the vCenter build to finish (assuming you’ll run the AddHosts script as a part of the build) I opted to get the hosts started before getting the vCenter build started. There’s a trick, and I can’t remember where I heard this – select all three hosts and start them at once as opposed to starting them one at a time. The reason for this is that starting them in one command will launch them on the same back-end hypervisor, causing performance of communication between the three hosts to be improved. Again, I can’t remember where I got that tip, and I can’t find it in the deployment guide, so correct me if I’m wrong 🙂

Assuming the DC build completed successfully, it’s running a PXE server that the hosts will boot an install wizard from. From the menu, on each host, select ‘ESXi [version] automated builds’ and then the corresponding build (e.g. Host1 Automated Install). This will build the respective host with the correct settings like hostname, IP address, and so on.

Once you’ve selected a host to build and the process completes, you’ll be left with a fully configured ESXi host. All that’s left is to get them connected to an added vCenter server. Powering on the vCenter server will get the build started, and it generally takes around an hour to complete. An option was set during the DC build to add the hosts to vCenter during the build, so first the OS customization script will run and set up the box, install vCenter, and dependencies/utilities (utilities being things like 7-zip, PuTTy, etc). Once the OS is ready, vCenter is installed, and the script gets PowerCLI installed, it will kick off the script that customizes vCenter. This builds out everything like the datacenter, cluster, adds hosts, adds datastores, and more.

Once the vCenter script is totally finished (it might be best not to watch it like I did – feels like an eternity!) the final action will be to install VMware Tools and reboot, just like the DC did. Once the reboot is complete and the OS is automatically logged in again, open the AutoLab Script Menu from the shortcut on the desktop and validate the build just as was done on the DC.

And there you have it! Assuming the vCenter is built, you can log in to vCenter via the vSphere Client using credentials in the ‘as-built’ documentation. You will find a fully configured environment with as many hosts as you chose to build. In my build for this post, I just had it build 2.