Category Archives: VMware

Monitor VMware ESXi hardware without root (Nagios)

Download and configure the plugin: http://exchange.nagios.org/directory/Plugins/Operating-Systems/*-Virtual-Environments/VMWare/check_esxi_hardware-2Epy/

– Create a new user in ESXi with no access privilege, you need to login to the ESXi directly to do that.

user

esxi_access

– Enable SSH, and add nagios user to root group:
# vi /etc/group
root:x:0:root,nagios

– Check from the command line, if it works
./check_esxi_hardware.py --host https://esxihost:5989 --user file:credentials.txt --pass file:credentials.txt
OK - Server: Cisco Systems Inc.....

– Configure the credentials files to use the nagios user credentials.

Could not connect to https://vcenter_address:7331/

This usually happens from the vSphere web client while opening a console session with virtual machine.

and the log (/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log) shows something like:

[ERROR] Thread-42 System.err
INFO:oejsh.ContextHandler:started o.e.j.w.WebApp Context{/console,file:/tmp/jetty-0.0.0.0-7331-console.war-_console-any-/webapp/},/usr/lib/vmware-vsphere-client/server/work/tmp/console-distro/webapps/console.war

To fix this set the environment variable VMWARE_JAVA_HOME to proper path:

– SSH to vcenter
# vi /usr/lib/vmware-vsphere-client/server/wrapper/conf/wrapper.conf

– Under Environment variables add:
set.default.VMWARE_JAVA_HOME=/usr/java/jre-vmware

– Restart vsphere-client
# /etc/init.d/vsphere-client restart
Stopping VMware vSphere Web Client...
Stopped VMware vSphere Web Client.
Starting VMware vSphere Web Client...
Intializing registration provider...
Getting SSL certificates
Service with name was updated.
Return code is: Success
Waiting for VMware vSphere Web Client......
running: PID:

Reference: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2060604

vCenter Converter

Convert Linux Physical Server to VMware virtual machine

Download and install vCenter Converter on a windows machine.
http://www.vmware.com/products/converter
Unfortunately this tool does not have a Linux / MAC version.

vCenter Converter

vCenter Converter

In case you see an error: “Permission to perform this operation was denied”, right click and run the program as Administrator.

Permission to perform this operation was denied
Provide the source and destination information, the source is the physical server to be converted and the destination vCenter.
vCenter Converter

vCenter Converter

vCenter Converter

vCenter Converter

Follow the steps, to do the conversion , a temporary OS will be started on the destination, by default it try to get an IP address from the DHCP server so that it can connect to the source machine and fetch the files required. But in case if you don’t have DHCP server you might see error like: “Unable to obtain the IP address of the helper virtual machine” . Fix this issue by setting up a static IP to the helper virtual machine during the conversion setup. Basically the helper VM IP should be able to communicate with the source machine which needs to be migrated

vCenter Converter Static IP

vCenter Converter Static IP

Proceed with the conversion, the duration will be based on the size of the VM and the connectivity if it belongs to another site/LAN.

You may need to change the network configuration (eg: HWADDR) and the MAC address mapping (/etc/udev/rules.d) to get it connected.

ESXi host fails with a purple diagnostic screen PSOD

This happened while converting KVM VMs to VMware and power them on (method used:http://arunnsblog.com/2013/06/10/migrate-kvm-virtual-machines-to-vmware-esxi/) . It works for a while but then the ESXi crashes with PSOD.

Version : 5.1.0-799733

There were two sort of PSOD messages observed:
1) Crashed while the VM was running

 VMware NOT_IMPLEMENTED bora/vmkernel/sched/memsched.c:17724
 Code start: 0x41802b200000 VMK uptime: 10:19:25:27.335
 cpu4:8243)0x412200cdbaf0:[0x41802b27abff]PanicvPanicInt@vmkernel#nover+0x56 stack: 0x3000000008
 cpu4:8243)0x412200cdbbd0:[0x41802b27b4a7]Panic@vmkernel#nover+0xae stack: 0x100000000000000
 cpu4:8243)0x412200cdbc50:[0x41802b3d88eb]MemSched_WorldCleanup@vmkernel#nover+0x426 stack: 0x4100018a4fb0
 cpu4:8243)0x412200cdbef0:[0x41802b3033b8]WorldCleanup@vmkernel#nover+0x1cb stack: 0x4700cdbf40
 cpu4:8243)0x412200cdbf60:[0x41802b303829]WorldReap@vmkernel#nover+0x318 stack: 0x0
 cpu4:8243)0x412200cdbff0:[0x41802b2483c8]helpFunc@vmkernel#nover+0x517 stack: 0x0
 cpu4:8243)0x412200cdbff8:[0x0] stack: 0x0
 cpu4:8243)base fs=0x0 gs=0x418041000000 Kgs=0x0

VMWare_ESXi_PSOD

VMWare_ESXi_PSOD

2) Crashed during ESXi reboot.

#PF Exception 14 in world 8243:helper13-1 IP 0x41802b880a1e addr 0x410401503020

VMWare_ESXi_PSOD

VMWare_ESXi_PSOD

This seems to be a known issue in VMware ESXi 5.1 and is resolved in patch ESXi510-201212401-BG (Build 914609).
Ref: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038767

To work around this issue, SSH to the ESXi host and increase the MinZeroCopyBufferLength to 512.

# esxcli system settings advanced set -o /BufferCache/MinZeroCopyBufferLength -i 512

To verify that the setting has been updated, run this command:

# esxcli system settings advanced list --option /BufferCache/MinZeroCopyBufferLength

Before and after change

Before and after change

ESXi host fails with a purple diagnostic screen PSOD

This happened while converting KVM VMs to VMware and power them on (method used: http://arunnsblog.com/2013/06/10/migrate-kvm-virtual-machines-to-vmware-esxi/) . It works for a while but then the ESXi crashes with PSOD.

Version : 5.1.0-799733

There were two sort of PSOD messages observed:
1) Crashed while the VM was running

 VMware NOT_IMPLEMENTED bora/vmkernel/sched/memsched.c:17724
 Code start: 0x41802b200000 VMK uptime: 10:19:25:27.335
 cpu4:8243)0x412200cdbaf0:[0x41802b27abff]PanicvPanicInt@vmkernel#nover+0x56 stack: 0x3000000008
 cpu4:8243)0x412200cdbbd0:[0x41802b27b4a7]Panic@vmkernel#nover+0xae stack: 0x100000000000000
 cpu4:8243)0x412200cdbc50:[0x41802b3d88eb]MemSched_WorldCleanup@vmkernel#nover+0x426 stack: 0x4100018a4fb0
 cpu4:8243)0x412200cdbef0:[0x41802b3033b8]WorldCleanup@vmkernel#nover+0x1cb stack: 0x4700cdbf40
 cpu4:8243)0x412200cdbf60:[0x41802b303829]WorldReap@vmkernel#nover+0x318 stack: 0x0
 cpu4:8243)0x412200cdbff0:[0x41802b2483c8]helpFunc@vmkernel#nover+0x517 stack: 0x0
 cpu4:8243)0x412200cdbff8:[0x0] stack: 0x0
 cpu4:8243)base fs=0x0 gs=0x418041000000 Kgs=0x0
VMWare_ESXi_PSOD

VMWare_ESXi_PSOD

2) Crashed during ESXi reboot.

#PF Exception 14 in world 8243:helper13-1 IP 0x41802b880a1e addr 0x410401503020
VMWare_ESXi_PSOD

VMWare_ESXi_PSOD

This seems to be a known issue in VMware ESXi 5.1 and is resolved in patch ESXi510-201212401-BG (Build 914609).
Ref: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2038767

To work around this issue, SSH to the ESXi host and increase the MinZeroCopyBufferLength to 512.

# esxcli system settings advanced set -o /BufferCache/MinZeroCopyBufferLength -i 512

To verify that the setting has been updated, run this command:

# esxcli system settings advanced list --option /BufferCache/MinZeroCopyBufferLength
Before and after change

Before and after change

 

 

Migrate KVM virtual machines to VMware ESXi

– Shutdown the KVM guest
– convert the QCOW2 or RAW format to VMDK format

# qemu-img convert image.img -O vmdk image.vmdk

– Upload this image to datastore

– Create a new virtual machine with this disk image

– There might be issues with network interface mapping, fix the network mapping at /etc/udev/rules.d/70-persistent-net.rules

Create NAS/SAN storage with openfiler, work with VMware ESXi as shared storage

– Download the openfiler installation ISO , download link

I have downloaded the Installation ISO image (x86/64).

Basically we need to create open filer as a virtual machine with say 20 GB thin provisioning, and attach another disk/datastore to the virtual machine to configure it as SAN.

– Download and install  the Installation ISO image (x86/64) as a virtual machine, noting fancy here just do a normal installation.

– Once rebooted you get a web interface to login on port 446, https://< ip > :446/

– Login with username: openfiler, and password: password

Network access configuration:

set up network access configuration, enter the Network/Host , who are allowed to access.

System –> Network Access Configuration

Network Access Config

Network Access Config

Network Access Config

Network Access Config

Create a new physical volume

Volumes –> Block devices

Create Physical Volume

Create Physical Volume

Click on Edit devices on the hard disk where we going to create new physical volume, this is the extra hard drive we added to the virtual machine at the beginning. (/dev/sdb)

 

Create Physical Volume

Create Physical Volume

Select Physical volume as your partition type ( assuming that you are not using RAID), and set the mode to Primary, click create.

Create Physical Volume

Create Physical Volume

Create new Volume Group

Let us create a volume group for the physical volumes

Volumes –> Volume groups , enter a group name and select the physical drive and click Add volume

Create volume group

Create volume group

 

Create volume group

Create volume group

Create the Volume

Volume –> Add Volume

Enter Volume Name , Description, required space and select File System block.

Create Volume

Create Volume

Create Volume

Create Volume

Now Enable and Add  iSCSI Target

Services –> iSCSI Target Enable, Start

Start iSCSI target

Start iSCSI target

Volumes –> iSCSI Targets –> Target Configuration –> Add new iSCSI Target

Add new iSCSI target

Add new iSCSI target

Setup the LUN Mapping and allow access to iSCSI target

Setup LUN Mapping

Setup LUN Mapping

 

Allow access to iSCSI target

Allow access to iSCSI target

Now the open filer is ready to use.

In VMware ESXi

Create new VMKernel network adapter to use for iSCSI connection

VMKernel Adapter

VMKernel Adapter

VMKernel Adapter

VMKernel Adapter

VMKernel Adapter

VMKernel Adapter

Click on storage adapter –> Add new

Add storage adapter

Add storage adapter

Click on properties, add the VMkernel switch and the discover the iSCSI target.

Add VMKernel to iSCSI adapter

Add VMKernel to iSCSI adapter

Dynamic discovery

Dynamic discovery

Now your ESXi will show the openfiler as a datastore 🙂

Openfiler datastore

Openfiler datastore

More information about Openfiler:

http://www.openfiler.com/products