Surgient VQMS 5.3 SP1 NAIL Server and VMware ESX Server 3.0.1
Introduction
NAIL is an acronym for Network Abstraction and Isolation Layer. The purpose of the NAIL Server is keep the Pool VMs of a Test Environment (or virtual lab or active reservation) isolated from those of another Test Environment. (However, whether a Pool VM will be isolated or not depends on how the corresponding Server Configuration was configured.) The NAIL Server also facilitates communication between the isolated Pool VMs and remote TCP/IP Hosts by acting as the Gateway of these Pool VMs and performing Network Address Translation (NAT).
The NAIL Server operates in either one of two modes: Standard and Advanced. If all the Pool VMs in each Test Environment will be running on the same ESX Server, then the NAIL Server can be running in Standard Mode, which is the default mode. Otherwise, the NAIL Server must be running in Advanced Mode.
NAIL Server in Standard mode
For NAIL Server in Standard mode, the ESX Server must be configured as follows:
* One virtual machine portgroup (referred to as the "default network") must be created on a virtual switch.
* This portgroup must be configured without any VLAN ID.
* The port on the physical switch (cabled to the outbound vmnic associated with the virtual switch) must be an access port. (If packets arriving to this port from the ESX Server are required to be in a VLAN, then the tagging of the packets has to be done at the physical switch - see further below.)
* Spanning Tree Protocol (STP) must be enabled on the access port. The root bridge priority of the (physical) access port must be greater than the NAIL Server's root bridge priority. This will cause the NAIL Server to disable its bridge interface and so preventing spanning tree loops. Note: the NAIL Server's bridge port is its eth1 interface; on which STP is enabled. The priority # of this port is 32999; the higher the number, the lower the priority.
When an "isolated" Test Environment is running on the ESX Server (with the NAIL Server operating in Standard mode), a virtual switch with the name of <ESX Server Hostname>-IsolatedNet is created with a virtual machine portgroup having the same name. This virtual switch is configured with a VLAN ID of ALL (i.e. 4095) and it has no outbound vmnics.
Note: for NAIL Server in Standard mode, no VLAN ID network resources are required; unlike the case for NAIL Server in Advanced mode.
NAIL Server in Advanced mode
For NAIL Server in Advanced mode, the ESX Server must be configured as above for the case of NAIL Server in Standard mode, plus as follows:
* A second virtual switch must be created with a virtual machine portgroup.
* This portgroup (referred to as the "trunked network") must be configured with a VLAN ID of 4095.
* The port on the physical switch (cabled to the outbound vmnic associated with the virtual switch) must be a trunked port configured to accept packets tagged with a "specified" range of VLAN IDs. This "specified" range of VLAN IDs must coincide with {or at lease include} the range(s) of VLAN IDs network resource(s) associated with the Pool that the ESX Server belongs to. If security is not an issue, then you may configure this port to accept packet tagged with all VLAN IDs.
* The port must be configured with BPDU filtering enabled.
Notes:
1. Using NAIL Server in Advanced mode requires a 1 GB Ethernet card in the ESX Server.
2. On an ESX Server, NAIL will fail unless TCPIP task offloading is disabled.
3. VLAN Trunking Protocol (VTP) is a layer 2 messaging protocol that maintains VLAN configuration consistency thought the network. There are two protocols for VLAN Tagging. One is Cisco's proprietary Inter Switch Link (ISL), which adds a 26 byte header, called the ISL header, to the Ethernet frame. This header includes the VLAN ID of the destination VLAN of the packet. The other mechanism is the IEEE 802.1Q Trunking Protocol, which actually modifies the Ethernet header so that it contains an extra four bytes inserted after the source and the destination MAC Addresses. In the four byte 802.1Q tag, the first two bytes (which are 0x8100) are an indicator that the following frame is an 802.1Q frame and the next two bytes are for the VLAN tag (3 bits for priority bits, 1 bit for Canonical Format Indicator, and the last 12 bits are for the VLAN ID). ESX Server supports only the IEEE 802.1Q) Trunking Protocol.
4. The VLAN ID range defined in the IEEE 802.1Q specification is from 1 to 4094.
5. VLAN ID 0 is reserved for user priority data, which is not supported by ESX Server. VLAN ID 1 is usually used for management.
6. The 802.1Q specification does make 4095 an invalid value for VLANs on the wire. However, if you use 4095, as the VLAN ID for a virtual machine portgroup, then this portgroup will see traffic on any and all VLANs. That is the value that is used to indicate VGT (i.e. Virtual Guest Tagging - or "trunk port" in Cisco speak). VGT describes the situation where packets are tagged/untagged by a driver in the Guest OS inside the Virtual Machine.
6. Besides VGT, ESX Server supports Virtual Switch Tagging (VST) and External Switch Tagging (EST). In VST, the tagging/untagging of packets occur inside the virtual switch; i.e. it is performed by the vmkernel. In EST, the tagging/untagging of packets occur inside the physical switch.
Only one node (i.e. VGT or VST or EST) is permitted; network communication breaks down if you have configured tagging/untagging to occur at two or all three of the following places - inside a VM, at the virtual switch (to which the VM is connected), and at the physical switch port (to which this virtual switch outbound vmnic is cabled).
Some procedures
If virtual switches have been created after the Surgient Agent has been installed and running on the console of ESX Server, then the VCS (via the Agent) must be informed of these new virtual switches. To do this:
1. Open a DOS window on the VCS Server.
2. Change your current directory to C:\Program Files\Surgient\VCS
3. Run the vcsadmin CLU by entering the command:
vcsadmin -a admin -p <your admin password>
4. Run the command from the vcsadmin prompt:
commandrun <ESX Server Hostname> poll
As mentioned above, the default mode for the NAIL Server is Standard. To change the Nail Server mode to Advanced, do the following:
1. Open the Management Console and click the Settings tab
2. Click the System Environment icon
3. In the System Environment Details area, click the Edit link for the NAIL Server Mode option
4. Change the mode to Advanced.
It is better to change to Advanced mode before the ESX Server is placed in a Pool. If the ESX Server has been pooled and then afterwards the NAIL Server is switched to Advanced mode, then edit the Pool utilization so that the new virtual networks are selected. To do so:
1. Open the Management Console
2. Click on the resources tab
3. Click on the Hosts icon.
4. Select the ESX Server from the list
5. Click the Show Host Utilization icon.
6. Click the edit icon on the Host Utilization dialog box.
7. On the Edit Pool Settings dialog box, from the Trunked Network list, select the "trunked network" VM portgroup.
General Information
When an ESX Server is being added to a Pool for the very first time, (amongst other things) a NAIL Server VM, with the name of <ESX Server Hostname>-NailServer-1, is being created and registered on the ESX Server. The virtual disk file (NailServer.vmdk) for the NAIL Server VM is copied from the Surgient Library. The OS in the NAIL Server VM is OpenSuSE Linux. The NAIL Server VM is then powered on. Inside the NAIL Server VM is an Agent that communicates with VCS Server. The VCS Server manages the NAIL Server. The NAIL Server does not communicate with the Console Operating System (COS) of ESX Server. If the agent fails to register itself with VCS Server after 600 seconds, then the NAIL Server is powered down.
The first interface of the NAIL Server, i.e. eth0, is connected to the Default network. The IP Address of eth0 is either an IP Address from the pool of IP Addresses network resource or a DHCP assigned address; depending on how the NAIL Server Address is configure. The second and the third interfaces (i.e. eth1 and eth2, respectively) are configured for manual startup; they are dynamically managed so their status will change according to deployments of Application Configurations. They operate in promiscuous mode. In particular, when a Pool VM (whose corresponding Server Configuration had its interface configured as NAIL) is first deployed to the ESX Server, the eth2 will be started and be connected to the Trunked network. eth1 is the bridge interface, br0. The MAC Addresses of the NAIL Server's interfaces are generated and are in the 00:0c:29:X:Y:Z range, while the MAC Addresses of Pool VMs are static and are in the range 00:50:56:00:00:00 to 00:50:56:3F:FF:FF
The NAIL Server communicates via eth0 with the VCS Server using outbound http traffic on port 80 and listens for inbound http traffic from the VCS Server on port 4277. The VCS Server orchestrates changes on the ESX Server by communicating to the Surgient Console Agent (running in the COS of the ESX Server). The VCS Server sends provisioning instructions (i.e. which VLAN IDs to use, interface(s) to be NATed, etc.) to the NAIL Server Agent that are consistent with the Application Configuration that is being deployed.
Every 600 seconds the NAIL Server sends status updates to the VCS Server. The VCS Server pools the NAIL Server repeatedly. If the NAIL Server is down when it should be up, then the VCS Server will restart it. When the NAIL server comes up, the VCS Server will send it pool info (e.g. NATed IP Addresses, etc)
NAIL Server is a Surgient-provided virtual appliance whose login is restricted. Informational messages appear on the console of the NAIL Server. Troubleshooting of NAIL server is generally done from the VCS, including capturing log files. In the NAIL Server VM directory there is the <ESX Server Hostname>-NailServer-1-guestagentlog.txt log file. The COM1 port interface (/dev/ttyS1) of the NAIL Server is piped to this file. Also, the vcsadmin utility gives operational information about the NAIL Server (see below).
Application Configuration deployment and virtual switch/portgroup creation
When an Application Configuration is being deployed, the following occurs:
a) A Server Configurations whose interface is configured as DHCP/Static (and moreover, this interface is configured in the Application Configuration to connect to a non-isolated virtual network) will have the corresponding Pool VM's interface connected to the Default network.
b) A Server Configurations whose interface is configured as NAIL (and moreover, this interface is configured in the Application Configuration to connect to a non-isolated virtual network) will have the corresponding Pool VM's interface connected to a newly created virtual machine portgroup called <ESX Server Hostname>-PoolNet-N, where N is an available integer. This portgroup will be on the same virtual switch containing the Trunked network. This portgroup will be configured with an available VLAN ID (say VLAN ID M, where M is an available integer) from the pool of VLAN IDs network resources.
The NAIL Server will create a sub-interface of eth2, called eth2.M, which will pretend to be the Gateway of the Pool VMs connected to the <ESX Server Hostname>-PoolNet-N portgroup with VLAN ID M.
The aliasing is probably being performed by the following command (or something similar):
"ifconfig eth2:M hw ether <MAC ADDRESS> <IP ADDRESS> netmask <SUBNET MASK>"
where
<MAC ADDRESS> is a MAC address in the 00:0c:29:X:Y:Z range
<IP ADDRESS> and <SUBNET MASK> are the Gateway IP Address and the subnet mask of the Pool VMs connected to "<ESX Server Hostname>-PoolNet-N" for which this eth2:M sub-interface is associated.
c) A Server Configuration, irrespective of whether its interface is configured as NAIL or DHCP/Static (and moreover, this interface is configured in the Application Configuration to connect to an isolated virtual network) will have the corresponding Pool VM's interface connected to a newly created virtual machine portgroup called <ESX Server Hostname>-PoolNet-N (where N is an available integer). This portgroup is configured with a VLAN ID of ALL and is on a newly created virtual switch having the same name of <ESX Server Hostname>-PoolNet-N and having no outbound vmnics. No NATed IP Address will be assigned to the Pool VM, even if the corresponding Server Configuration had had its network interface configured as NAIL (rather than DHCP/Static).
Configuring some Server Configurations in an Application Configuration to be connected to an isolated virtual network is achieved as follows: when creating the Application Configuration, in the Network Screen of the Wizard, configure the corresponding Server Configurations to be on the same virtual network (if necessary, you may add a virtual network) and specify that this virtual network is isolated by placing a check in the Isolated check box. When the Application Configuration is deployed, the Pool VMs arising from these Server Configurations will all be running on the same ESX Server.
Network communication of a Pool VM connected to a non-isolated <ESX Server Hostname>-PoolNet-N portgroup
For a Pool VM connected to a non-isolated <ESX Server Hostname>-PoolNet-N portgroup, whenever it sends out a broadcast packet, the virtual switch will
a) receive the packet as it leave the Pool VM's interface,
b) forward the packet to other Pool VM connected to the same <ESX Server Hostname>-PoolNet-N portgroup, and
c) 1) tag the packet with VLAN ID M (where M is the VLAN ID of this portgroup); this is VST- Virtual Switch Tagging, and
2) send the packet to
i) the physical network via the outbound vmnic associated with the virtual switch, and
ii) the Trunked portgroup
The reason why the Trunked portgroup will receive the packet is because the VLAN ID of the Trunked portgroup is 4095 and this reserved VLAN ID value of 4095 means that this portgroup will see traffic on any and all VLANs. The NAIL Server, with its eth2:M sub-interface on the Trunked portgroup, will receive the packet.
When the Pool VM first boots up (after the Application Configuration is deployed) and the IP protocol initializes, the VM sends an ARP request containing its own MAC Address and IP Address so that other TCP/IP hosts can update their ARP caches and also to see if another TCP/IP host (in the same broadcast domain) is using the same IP Address. The NAIL Server will receive this packet and if the IP Address matches the IP Address in the corresponding Server Configuration, then the NAIL Server will assign an available IP Address (from the range of IP Addresses network resource associated with the pool) to this Pool VM. This will be the NATed or public or external-facing IP Address of the Pool VM. This assigned NATed IP Address is seen in the Management Console in the page for this active reservation. If, however, the Pool VM has a different IP Address (whether static or DHCP Assigned or Autoconfigured) from the one specified in the corresponding Server Configuration, then this Pool VM will not be assigned a NATed IP Address.
When the Pool VM needs to communicate with any remote TCP/IP Host (i.e. a TCP/IP Host that is not within its own subnet) for the first time, it will send out an ARP request to get the MAC Address of its Gateway. Irrespective of whether the Gateway IP Address in the APR packet matches or doesn't match the Gateway IP Address in the corresponding Server Configuration, then the NAIL Server will respond and claim to be the owner of this (i.e. Gateway) IP Address. The response packet will be tagged by the NAIL Server (this is VGT - Virtual Guest Tagging) with a VLAN ID of M and sent out to the Trunked network. The virtual switch will examine the packet and send it directly to the Pool VM, since the destination MAC Address in the response packet is the MAC Address of the Pool VM. The source destination MAC Address is in the MAC address of the eth2:M sub-interface. Even if another Pool VM connected to the same <ESX Server Hostname>-PoolNet-N portgroup has a different Gateway IP Address in the Guest OS image, then the NAIL Server will respond with be the owner of this different Gateway IP Address but with the same the same MAC Address of eth2:M. In other words, when you examine the ARP table in the Guest OS of the two Pool VMs, you'll see the same MAC address being associated with the different (Gateway) IP Addresses.
However, for another Pool VM belonging to a different Test Environment (and so connected to a different <ESX Server Hostname>-PoolNet-N portgroup; i.e. the value of N will be different), the NAIL Server will respond with a different MAC Address (i.e. the MAC Address of the sub-interface of eth2 associated with this other portgroup) when claiming to be the Gateway of this Pool VM.
In other words, the NAIL Server (faking as the different Gateways), will have a MAC Address that is unique to each "<ESX Server Hostname>-PoolNet-N", even if
1) Pool VMs connected to the same "<ESX Server Hostname>-PoolNet-N" have different gateway IP Addresses.
or
2) Pool VMs from different Test Environments have the same Gateway IP Address
Moreover, all the NAIL Servers on all the ESX (VM Host) Servers that have an eth2:M sub-interface will use the same MAC Address for this sub-interface.
Thus when a Pool VM needs to communicate with a remote TCP/IP host, it will send out packets with a destination MAC address of the NAIL Server's appropriate sub-interface of eth2. The virtual switch will tag the packets with the appropriate VLAN ID and route them directly to the NAIL Server. The NAIL server will
- a) untag the packets
- b) NAT the packet; i.e. replace the source IP Address in the packet (i.e. the IP Address in the Guest OS image) with the assigned pool IP Address (the NATed or external or public-facing IP Address for this Pool VM)
- c) replace the source MAC in the packet with the MAC Address of the NAIL Server's eth0 interface
- d) replace the destination MAC in the packet with the MAC Address of the Gateway for this range of pool IP Addresses. (Remember, when creating a range of IP Addresses network resource, you specify the range of IP Addresses, the subnet mask and the IP Address of the Gateway. The NAIL Server would have done an ARP request to get the MAC Address of the Gateway.)
- e) send the packets out along its eth0 interface connected to the Default network
The ESX Server will then send the packets out to the physical network (assuming that the Gateway is not a VM connected to a portgroup on the virtual switch containing the "Default network" portgroup).
If this Pool VM needs to communicate with another Pool VM (that is in the same Test Environment and whose corresponding Server Configuration's interface is configured as NAIL), then the ESX Server will
- 1) send the packets unmodified and directly to the other Pool VM, if the two Pool VMs are running on the same ESX Server.
or
- 2) tag the packets with the appropriate VLAN ID and send them out to the physical network, if the two Pool VMs are running on different ESX Servers.
Because the Server Configurations' interfaces are configured as NAIL, even if you have two Server Configurations from different Application Configurations using the same Guest OS image in the Library, when the Application Configurations are deployed concurrently, you will not have issues with duplicate NETBIOS names and duplicate IP Addresses, because each Test Configuration will be in its own VLAN ID and also because each Pool VM (whether using the same image or not) will have its own NATed IP address. The same thing can be said if a single Application Configuration is deployed multiple times concurrently.
Network communication of a Pool VM connected to the "Default network" portgroup
Pool VMs that are connected to the Default network are in the same Ethernet broadcast domain. They are not in any VLAN, unless VLAN tagging is configured at the physical switch port cabled to the vmnic associated with the virtual switch containing the Default network portgroup. Thus if two Pool VMs are running off the same image, you'll have duplicate NETBIOS name conflict. If the image has a static IP Address, then you'll also have duplicate IP Address conflict.
The Pool VM communicates with remote TCP/IP Hosts via the Gateway specified in its image, and not via the NAIL Server.
vcsadmin commands involving the NAIL Server
The following are three vcsadmin commands that give information about the NAIL Server. Each option is immediately preceded by a dash (e.g. -option and not - option).
nailcmdlist [options]
This command displays a list of NAIL Server commands that have been executed. An example of a command, with its description, is the following:
#88532,523,402,3910: ip route del 10.250.18.101/32 dev eth2.2014 src 10.250.1.1 table vlan.2014
"delete the host route entry for 10.250.18.101 via eth2.2014 from routing table for VLAN 2014"
The (non-required) options are:
deploymentgroupid <deploymentgroupid> - retrieves only commands belonging to the specified deployment group
scheduleddeploymentid <scheduleddeploymentid> - retrieves only commands belonging to the specified scheduled deployment
serverid <serverid> - retrieves only commands belonging to the specified server
grep <grep> - displays only commands that contain the provided string
descriptions - displays the description of each command
results - displays command execution results, if applicable
naildiagnostics [options] <NAIL Server Name>
This command retrieves various NAIL-related diagnostics for a NAIL Server. The <NAIL Server Name> is <ESX Server Hostname>-NailServer-1.
At least one option (except the -file option) is mandatory. The options are:
route - displays routing table information
ip - displays IP information, including info about the interfaces and the ARP Table.
sys - displays system information, including the NAIL Server RPM package installed in the NAIL Server VM, the output of the dmesg command and the output of the netstat command. The dmesg output gives info about the creation and the destruction of the sub-interfaces of eth2.
ver - displays version information, including info about the linux kernel running in the NAIL Server VM and info about the nail-server RPM package installed in the VM.
bridge - displays bridge information, including STP info.
rule - displays iptables/ebtables firewall rules
all - displays all the above information
file <file> - specifies the full path of a file to write the output to; otherwise, the output is written to the DOS Screen.
nailtcpcap [options] <NAIL Server Name> outputFile
This command retrieves a tcpdump capture from the specified NAIL Server and outputs it into the specified file. The non-required options are:
pktcnt <pktcnt> - the number of packets to capture; defaults to 1024
device <device> - the device from which to capture network traffic; defaults to br0
Advanced Configurations Settings
The Advanced Configurations settings are by default not editable through the Management Console interface; they are provided for diagnostic purposes. To enable editing, launch the vcsadmin utility as mentioned above. At the vcsadmin prompt, enter the following command:
configset AdvancedConfiguration.AllowEditing=true
The Edit icon (a pencil) now appears in the upper right corner of the Advanced Configuration page.
Alternatively, you can edit individual advanced configuration settings using the vcsadmin utility and not enable editing through the Management
The following gives the advanced configuration settings relating to the NAIL Server in the following format: Name \ Value \ Description
Nail.DefaultNetworkName \ Default Network \ The name of the default network
Nail.DefaultVlanName \ Virtual LAN A \ The string used for default VLAN network names
Nail.EnableLegacyMode \ false \ Enable or Disable Legacy NAIL Mode
Nail.ExternalInterface \ eth1 \ The device acting as the external bridge port of the standard NAIL server.
Nail.GatewayAddress \ <IP Address(es)> \ The gateway server IP address to be assigned to server configurations with NAIL enabled. <IP Address(es)> can be multiple comma-separated IP Addresses.
Nail.InternalInterface \ eth2 \ The device acting as the internal (trunked) bridge port of the standard NAIL server.
Nail.ServerAddress \ \ For standard mode NAIL Server configuration, determine the location of the NAIL Server.
Nail.ServerMode \ standard or advance \ Determines whether the NAIL configuration for this installation is in standard or advanced mode. Default is standard.
Nail.SwitchesPerVm \ 2 \ The number of virtual switches to pool per VM.
NailServerAgentMonitor.MaxFailures \ 3 \ The maximum number of communication failure occurences (inclusive) tolerated before performing error state handling.
NailServerAgentMonitor.PollInterval \ 300 \ The time in seconds between each attempt by the NAIL Server agent monitor script to contact all NAIL Server agents.
NIC Teaming in ESX
The preceding text refers to the case when each virtual switch has one outbound vmnic.
NIC Teaming or bonding is having two or more vmnics associated with a virtual switch, so as to provide failover, and (depending on configuration) load balancing. There are four load balancing modes that are supported in ESX Server. The load balancing mode configured at the virtual switch level can be over-ridden at the Service Console port level or the vmkernel port level or the virtual machine portgroup level. The Load Balancing mode determines which outbound vnmic will be used for which outbound packet. The vmnic which is used for an incoming packet is determined by the physical switch.
The following are the four Load Balancing modes:
- Route based on the originating virtual port ID - Choose a vmnic based on the ID of the virtual port where the traffic entered the virtual switch. This method is simple and fast and does not require the vmkernel to examine the frame for addresses. In this case the outbound packets from a VM will always be sent to the same vmnic; while this vmnic is still functional. When the VM is powered down and is powered on again, it may be connected to the virtual switch via a different port and so its outbound packets may now flow through a different vmnic.
- Route based on source MAC hash - Choose a vmnic based on a hash of the source ethernet MAC Address. In this case the outbound packets from a VM will always be sent to the same vmnic; while this vmnic is still functional. When the VM is powered down and is powered on again, its outbound packets will flow through the same vmnic (assuming that the MAC address of the VM was not changed).
- Use explicit failover order - Always use the first vmnic listed in the section of Active Adapters in the VI Client. In this case the outbound packets from all VMs will always be sent to the same vmnic; while this vmnic is still functional.
- Route based on ip hash - Choose a vmnic based on a hash of the source and the destination IP Addresses of each packet. For non-IP packets, whatever bytes are at the corresponding offset are used to compute the hash. In this mode the outbound packets from a VM will be split across the vmnics based on the destination IP Address. This load balancing mode (of all the four modes) is closest to "true load balancing". This mode is close to but is not "true load balancing" where all vmnics transmit the same amount of data. This mode (of all the four modes) requires the greatest CPU processing of the ESX Server. It requires that the physical switch to support 802.3ad link aggregation; since the ports (cabled to the vmnics in the team) on the physical switch must be configured in an EtherChannel Link. The 802.3ad static NIC Team is supported; but not the 802.3ad dynamic (also known as LACP) NIC Teaming.
You can configure EtherChannel connections with or without IEEE 802.1Q trunking. After the formation of a channel, the configuration of any port in the channel as a trunk applies the configuration to all ports in the channel. Identically configured trunk ports can be configured as an EtherChannel. 802.1Q encapsulation, if enabled, takes place independently of the source/destination load-balancing mechanism of the EtherChannel link. The VLAN ID has no influence on the link that a packet takes. If trunking is not enabled, all ports associated with the Fast EtherChannel must belong to the same VLAN