Background
Virtual machines created from the same VM template will behave identically unless they are passed additional configuration information.
For instance, CloudStack uses a single, generic VM template as the basis of every system VM. When a VM is created from this template, it is passed additional information that determines what services it will run. Depending on the configuration, the VM might become a virtual router, a secondary storage VM, or a proxy for VM console access.
Problem
The CloudStack system VM template does not harvest configuration when it is running on Hyper-V 2012. There needs to be a mechanism whereby this information is passed to the VM and harvested by the VM at start up.
Solution
Use the Hyper-V KVP Data Exchange to pass data to VMs at startup. Update the template with integration services that write KVP data to disk. Finally, update the System VM start up to harvest KVP data.
Let's elaborate on what each step involves...
Hyper-V KVP Data Exchange using WMI
KVP stands for Key Value Pair. These key / value pairs originally modeled information that appears in a Windows registry entry: registry entries have a name (a key), and they contain data (a value). However, KVPs are operating system agnostic. For example, it is just as easy to store a set of key value pairs on disk in Linux as it is to put them in the registry on Windows.KVP Data Exchange involves transmitting data between the host and the guest OS over the VMBus. "VMBus" is the term Hyper-V uses for its inter-partition communication channel. Specifically, the data exchanged consists of KVPs. Depending on the source of the KVP, their data is transmitted from host to guest or guest to host.
KVP Data Exchange is accessed through the WMI APIs exposed by Hyper-V. First, KVP data is placed in an Msvm_KvpExchangeDataItem. Place the key in the "Name" filed, the value in the "Data" field, and set the "Source" field to 0 to indicate we are pushing KVP from host to guest. E.g.
KvpExchangeDataItem kvpItem = KvpExchangeDataItem.CreateInstance(); kvpItem.LateBoundObject["Name"] = "cloudstack-vm-userdata"; kvpItem.LateBoundObject["Data"] = "username=root;password=1pass@word1"; kvpItem.LateBoundObject["Source"] = 0;
The example above uses C# WMI wrappers generated using visual Studio. If you must insist, source is available at. Another alternative is to use PowerShell commands.
Next, register the KVP object with Hyper-V using the AddKvpItems method of the Msvm_VirtualSystemManagementService class I.e.
uint32 AddKvpItems( [in] CIM_ComputerSystem REF TargetSystem, [in] string DataItems[], [out] CIM_ConcreteJob REF Job );
If you are new to WMI, let me explain further: The aim of WMI is to provide APIs that are platform agnostic. Therefore, parameters for the method call have to be serialisable in a cross platform manner. As a result, object references look more like a URI than a memory address. E.g. here is a sample CIM_ComputerSystem reference used to call AddKvpItems:
\\CC-SVR10\root\virtualization\v2:Msvm_ComputerSystem.CreationClassName="Msvm_ComputerSystem",Name="3969FFB5-5341-422A-B8D6-A60ECB6D73D6"
Indeed, WMI objects refer to such a reference as the WMI object path
Likewise, the DataItems parameter is an array of serialised Msvm_KvpExchangeDataItem objects serialised according to a WMI-specific format. When it comes to serialisation, WMI follows standards set by the DTMF. Specifically, the format used to encode a WMI object is CIM XML DTD 2.0. Therefore, the DataItems parameter is an array of XML-serialized data. E.g.
<INSTANCE CLASSNAME="Msvm_KvpExchangeDataItem"> <PROPERTY NAME="Caption" PROPAGATED="true" TYPE="string"></PROPERTY> <PROPERTY NAME="Data" TYPE="string"> <VALUE>username=root;password=1pass@word1</VALUE> </PROPERTY> <PROPERTY NAME="Description" PROPAGATED="true" TYPE="string"></PROPERTY> <PROPERTY NAME="ElementName" PROPAGATED="true" TYPE="string"></PROPERTY> <PROPERTY NAME="InstanceID" PROPAGATED="true" TYPE="string"></PROPERTY> <PROPERTY NAME="Name" TYPE="string"> <VALUE>cloudstack-vm-userdata</VALUE> </PROPERTY> <PROPERTY NAME="Source" TYPE="uint16"> <VALUE>0</VALUE> </PROPERTY> </INSTANCE>
In C#, the System.Management namespace provides objects to facilitate to create this XML for us. Specifically, ManagementObjectBase.GetText performs the nescessary serialisation.
Following on from the C# above, we call AddKvpItems in this snippet:
System.Management.ManagementBaseObject kvpMgmtObj = kvpItem.LateBoundObject;
System.Management.ManagementPath jobPath;
String kvpStr = kvpMgmtObj.GetText(System.Management.TextFormat.CimDtd20);
uint ret_val = vmMgmtSvc.AddKvpItems(new String[] { kvpStr },
vm.Path, out jobPath);
Again, C# wrappers hide much complexity.
Adding a KVP item with a Source '0' item will cause the Hyper-V host to send data; however, the Guest needs to be setup to receive KVP data.
hv_kvp_daemon the KVP Daemon
KVP data is transfered to the file system through the collaboration of a kernel driver and a user mode daemon.The KVP driver code, hv_kvp.c (http://lxr.linux.no/linux+v3.11/drivers/hv/hv_kvp.c), is compiled into the hv_utils kernel module (Source http://lxr.linux.no/linux+v3.11/drivers/hv/Makefile).
Since the driver is part of the Linux kernel code, it is provided by default with recent versions of common Linux distributions. E.g.
[root@centos6-4-hv ~]# cat /etc/*-release CentOS release 6.4 (Final) CentOS release 6.4 (Final) CentOS release 6.4 (Final) [root@centos6-4-hv ~]# modinfo -F filename hv_utils /lib/modules/2.6.32-358.el6.i686/kernel/drivers/hv/hv_utils.ko
However, it is the usermode daemon, hv_kvp_daemon, that copies KVP data to the system. On startup, hv_kvp_daemon creates files to store kvp data under
/var/lib/hyperv (source http://lxr.linux.no/linux+v3.11/tools/hv/hv_kvp_daemon.c).
Each file is known as a 'pool', and there is a file for each data pool. E.g.
[root@centos6-4-hv hyperv]# ls -al /var/lib/hyperv/ total 36 drwxr-xr-x. 2 root root 4096 Sep 11 21:33 . drwxr-xr-x. 16 root root 4096 Sep 10 13:59 .. -rw-r--r--. 1 root root 2560 Sep 10 17:05 .kvp_pool_0 -rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_1 -rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_2 -rw-r--r--. 1 root root 28160 Sep 10 14:02 .kvp_pool_3 -rw-r--r--. 1 root root 0 Sep 10 14:02 .kvp_pool_4
The prefix of each file is the pool number. This correspoinds to the KVP source. E.g. remember that source '0' is used for transmitting data from host to guest? That means our KVP data is in /var/lib/hyperv/.kvp_pool_0. E.g.
[root@centos6-4-hv hyperv]# cat /var/lib/hyperv/.kvp_pool_0 cloudstack-vm-userdatausername=root;password=1pass@word1[root@centos6-4-hv hyperv]#
Aside: there are five pools (source http://lxr.linux.no/linux+v3.11/include/linux/hyperv.h), but only four sources listed for KVP data exchange.
It appears that pool '3' contains predefined KVPs sent by the Host such as the host machine's name.
With this in mind, it is important that hv_kvp_daemon be installed and an init script added.
Microsoft provides the daemon, hv_kvp_daemon, in a package suited for RHEL (http://dlafferty.blogspot.co.uk/2013/07/installing-hyper-v-pv-drivers-for-linux.html).
However, it may be easier to search for a package specific to your distro. E.g. using rpmfind.net I found hv_kvp_daemon in the hypervkvpd package. E.g. for CentOS 6.4 simply call:
yum install hypervkvpd
Unfortunately, I have not found a package for Debian. An alternative is to compile the source and write an init script. There is a sample here.
Once you are able to create the file, you need to be able to parse it...
Harvesting Hyper-V KVP Data in Linux
KVP data files contains an array of key / value pairs. Each is a byte array of a fixed size. Source/* * Maximum key size - the registry limit for the length of an entry name * is 256 characters, including the null terminator */ #define HV_KVP_EXCHANGE_MAX_KEY_SIZE (512) /* * bytes, including any null terminators */ #define HV_KVP_EXCHANGE_MAX_VALUE_SIZE (2048)
The byte array contains a UTF-8 encoded string, which is padded out to the max size with null characters. However, Null string termination is not guaranteed (see kvp_send_key).
Provided there is only one key and the key name known, the easiest way to parse the file is to use sed. To remove null characters and the key name used in our example, you would use the following:
[root@centos6-4-hv hyperv]# cat /var/lib/hyperv/.kvp_pool_0 | sed 's/\x0//g' | sed 's/cloudstack-vm-userdata//g' > userdata [root@centos6-4-hv hyperv]# more userdata username=root;password=1pass@word1
