Integrating vCenter with DMF

This chapter describes integrating VMware vCenter with the DANZ Monitoring Fabric (DMF) and monitoring Virtual Machines (VM) in the vCenter.

Overview

The DANZ Monitoring Fabric (DMF) allows the integration and monitoring of VMs in a VMware vCenter cluster. After integrating a vCenter with the DMF fabric, use DMF policies to select different types of traffic from specific VMs and apply managed services, such as deduplication or header slicing, to the selected traffic.

Currently, DMF supports the following versions of VMware vCenter for monitoring:

  • vCenter Server 6.5.0
  • vCenter Server 6.7.0
  • vCenter Server 7.0.0
  • vCenter Server 8.0.0

The DANZ Monitoring Fabric provides two options to monitor a VMware vCenter cluster:

  • Monitoring using span ports: This method monitors VMware vCenter clustering using a separate monitoring network. The advantage of this configuration is that it has no impact on the production network and has a minimal effect on compute node CPU performance. However, in this configuration, each compute node must have a spare NIC to monitor traffic.

    The following figure illustrates the topology used for local SPAN configuration:

    Figure 1. Mirroring on a Separate SPAN Physical NIC (SPAN)
  • Monitoring using ERPAN/L2GRE tunnels: Use Remote SPAN (ERSPAN) to monitor VMs running on the ESX hosts within a vCenter instance integrated with DMF. ERSPAN monitors traffic to and from VMs anywhere in the network and does not require a dedicated physical interface card on the ESX host. However, ERSPAN can affect network performance, especially when monitoring VMs connected to the DMF Controller over WAN links or production networks with high utilization.

Using SPAN to Monitor VMs

This section describes the configuration required to integrate the DANZ Monitoring Fabric (DMF) Controller with one or more vCenter instances and to monitor traffic from VMs connected to the VMware vCenter after integration.

The following figure illustrates the topology required to integrate a vCenter instance with the monitoring fabric and deliver the traffic selected by DMF policies to specified delivery ports connected to different monitoring tools.

Figure 2. VMware vCenter Integration and VM Monitoring

When integrated with vCenter, the DMF Controller uses Link Layer Discovery Protocol (LLDP) to automatically identify the available filter interfaces connected to the vCenter instance.

Using ERSPAN to Monitor VMs

Use Remote SPAN (ERSPAN) to monitor VMs running on the ESX hosts within a VMware vCenter instance integrated with the DANZ Monitoring Fabric (DMF). ERSPAN monitors traffic to and from VMs anywhere in the network and does not require a dedicated physical interface card on the ESX host. However, ERSPAN can affect network performance, especially when monitoring VMs connected to the DMF Controller over WAN links or production networks with high utilization.
Figure 3. Using ERSPAN to Monitor VMs

The procedure for deploying ERSPAN is similar to SPAN but requires an additional step to define the tunnel endpoints used on the DMF network to terminate the ERSPAN session.

Configuration Summary for vCenter Integration

The following procedure summarizes the high-level steps required to integrate the vCenter and monitor traffic to or from selected VMs:

  1. (For ERSPAN only) Define the tunnel endpoint.
    Identify a fabric interface connected to the vCenter instance for the tunnel endpoint by entering the tunnel-endpoint command in config mode. To define the tunnel endpoint, refer to the Defining a Tunnel Endpoint section.
  2. Provide the vCenter address and credentials.

    The vSphere extension on the DANZ Monitoring Fabric (DMF) Controller discovers an inventory of VMs and the associated details for each VM.

  3. Select the VMs to monitor on the DMF Controller.

    The DMF Controller uses APIs to invoke the vSphere vCenter instance.

    vSphere calls the DVS to create a SPAN session. The preferred option is to SPAN on a separate physical NIC. However, the option exists to also use ERSPAN by tunneling to the remote interface.

  4. Create policies in DMF to filter, replicate, process, and redirect traffic to tools.

    When using tunnels with ERSPAN, DMF terminates the tunnels using the specified tunnel endpoint. A DMF policy for monitoring VM traffic using a SPAN session must include the required information regarding the vCenter configuration. All match conditions, including User-Defined ofFsets (UDFs), are supported.

    The policy for selecting VM traffic to monitor is similar to other DMF policies, except that the filtering interfaces are orchestrated automatically (filter interfaces are auto-discovered and cannot be specified manually). All managed-service actions are supported.

Defining a Tunnel Endpoint

Predefine the tunnel endpoints for creating tunnels when monitoring VMware vCenter traffic using either the GUI or the CLI.

GUI Procedure

To manage tunnel endpoints in the GUI, select Monitoring > Tunnel Endpoints.

Figure 4. Monitoring > Tunnel Endpoints

This page lists the tunnel endpoints that are already configured and provides information about each endpoint.

To create a new tunnel endpoint, click the provision (+) control in the Tunnel Endpoints table.
Figure 5. Create Tunnel Endpoint
To create the tunnel endpoint, enter the following information and click Save:
  • Name: Type a descriptive name for the endpoint.
  • Switch: Select the DMF switch from the selection list for the configured endpoint interface.
  • Interface: Select the interface from the selection list for the endpoint.
  • Gateway: Type the address of the default gateway.
  • IP Address: Type the endpoint IP address.
  • Mask: Type the subnet mask for the endpoint.

CLI Procedure

To configure a tunnel endpoint using the CLI, enter the tunnel-endpoint command from config mode using the following syntax: 
controller-1(config)# tunnel-endpoint <name> switch <switch> <interface> ip-address <address> mask
<mask> gateway <address>
For example, the following command defines ethernet24 on F-SWITCH-1 as a tunnel endpoint named OSEP1: 
controller-1(config)# tunnel-endpoint ERSPAN switch CORE-SWITCH ethernet7 ip-address 172.27.1.1
mask 255.255.255.0 gateway 172.27.1.2

The IP address assigned to this endpoint is 172.27.1.1, and the next hop address for connecting to the vCenter via ERSPAN is 172.27.1.2.

Using the GUI to Integrate a vCenter Instance

To integrate a vCenter instance with DANZ Monitoring Fabric (DMF) to begin monitoring VMs, select Integration > vCenter from the DMF menu bar.
Figure 6. Integration > vCenter

This page displays information about the vCenter instances integrated with DMF. To add a vCenter instance for integration with DMF, perform the following steps:

  1. Click the provision control (+) in the table.
    Figure 7. Create vCenter: Info
  2. Type an alphanumeric identifier for the vCenter instance, and (optionally) add a description in the fields provided.
  3. Identify the vCenter hostname to be integrated.
  4. Enter the vCenter username and password for authenticating to the vCenter instance.

    These credentials are used by the DMF Controller when communicating with the vCenter host.

  5. Click Next.
    Figure 8. Create vCenter: Options (page 2)
    This page defines the mirror type as SPAN or ERSPAN. When selecting ERSPAN, the following additional fields complete the ERSPAN configuration:
    • Cluster Tunnel Endpoints (optional)
    • Default Tunnel Endpoint (required)
    • Sampling Rate (optional)
    • Mirrored Packet Length (optional)
    • Create Wildcard Tunnels(optional)

    Use Cluster Tunnel Endpoints to specify a common tunnel endpoint for all the ESXi hosts in the cluster. Use Default Tunnel Endpoint to specify a common tunnel endpoint for all the ESXi hosts regardless of the cluster. When configuring both cluster and default tunnel endpoints, all hosts in clusters form tunnels using the cluster-specific configuration, and all the other hosts that are not a part of any cluster use the default configuration to form tunnels.

  6. Click Next.
    Figure 9. Create vCenter/VMs
  7. To add a VM for monitoring, click the provision control (+).
    Figure 10. Configure vCenter VM

    Select VMs from the selection list after integrating vCenter and discovering the VMs, or manually add the VM hostname.

  8. After identifying the VM to monitor, click Append.
  9. On the VMs of the Create vCenter dialog, click Save.

Using a vCenter Instance as the Traffic Source in a DMF Policy

To identify a vCenter instance integrated with the DANZ Monitoring Fabric (DMF) Controller as the traffic source for a DMF policy, click the VMware vCenter tab on the Integration page. Locate the vCenter instance name.
Figure 11. VMware vCenter Name

Proceed to the Monitoring > Policies page.

Figure 12. DMF Policies
Click the + Create Policy button to add a policy.
Figure 13. Create Policy
Enter a Name and Description for the vCenter policy. From the Traffic Sources column, select + Add Ports(s).
Figure 14. Traffic Sources - Add Ports
Click vCenters.
Figure 15. vCenters
Available vCenter instances display. Select the required vCenter instance which then appears in the Selected traffic Sources panel.
Figure 16. vCenter Instance
Click Add 1 Source. The vCenter instance appears in the Traffic Sources column.
Figure 17. vCenter Traffic Sources
From the Destination Tools column, select + Add Ports(s). Select the interface under Destination Tools.
Figure 18. Destination Tools - Add Ports
Click the Add 1 Interface button. The interface appears under the Destination Tools column.
Figure 19. Add Interface
Click Create Policy. The new vCenter policy appears in the DMF Policies dashboard.
Figure 20. Create vCenter Policy

Using the CLI to Integrate a vCenter Instance

Refer to the following topics to monitor VMs using Encapsulated Remote SPAN (ERSPAN) or Switch Port Analyzer (SPAN) on a locally connected vCenter instance and VMs on a second locally connected vCenter instance.

VMs using ERSPAN on a Locally Connected vCenter Instance

To configure the DANZ Monitoring Fabric Controller for monitoring VMs using ERSPAN on a locally connected vCenter instance, perform the following steps:

  1. Add the vCenter instance details by entering the following commands. 
    controller-1(config)# vcenter vc-1
controller-1(config-vcenter)# host-name 10.8.23.70
controller-1(config-vcenter)# password 094e470e2a121e060804
controller-1(config-vcenter)# user-name root
  2. Specify the mirror type by entering the following commands. 
    controller-1(config-vcenter)# mirror-type erspan
controller-1(config-vcenter)# sampling-rate 60
controller-1(config-vcenter)# mirrored-packet-length 60

    The sampling-rate and mirrored-packet-length commands are optional.

  3. ERSPAN mirroring requires a tunnel endpoint configuration. Use the cluster command to specify a common tunnel endpoint for all the ESXi hosts in the cluster. Use the default-tunnel-endpoint command to specify a common tunnel endpoint for all the ESXi hosts regardless of the cluster. When using both the cluster and default-tunnel-endpoint commands, all hosts in clusters form tunnels using the cluster-specific configuration, and all the other hosts not a part of any cluster use the default configuration to form tunnels. 
    controller-1(config-vcenter)# default-tunnel-endpoint VCEP1
controller-1(config-vcenter)# cluster <cluster-name> tunnel-endpoint <tunnel-endpoint-name>

    Using the tab auto-complete feature with the cluster suggests existing cluster names associated with the vCenter.

  4. Add a static route to the default or cluster tunnel-endpoint in each ESXI host. 
    esxcli network ip route ipv4 add -n <network> -g <gateway> 
Example: esxcli network ip route ipv4 add -n 192.168.200.0/24-g 192.168.150.1
  5. Add the VMs to monitor by entering the following commands. 
    controller-1(config-vcenter)# vm-monitoring
controller-1(config-vcenter-vm-monitoring)# vm vm-2001
controller-1(config-vcenter-vm-monitoring)# vm vm-2002
  6. Receive-only GRE tunnel-interfaces will be auto-configured under switch for all the hosts belonging to vc-1 that have a route to the default or cluster tunnel-endpoint. 
    ! switch
switch DMF-RU34
mac 94:8e:d3:fd:6b:96
!
gre-tunnel-interface vcenter-abd08a18
direction receive-only
local-ip 192.168.200.254 mask 255.255.255.0 gateway-ip 192.168.200.1
origination vc8--interface
parent-interface ethernet55
remote-ip 192.168.150.27
gre-key-decap 33554432
!
gre-tunnel-interface vcenter-abd08a37
direction receive-only
local-ip 192.168.200.254 mask 255.255.255.0 gateway-ip 192.168.200.1
origination vc8--interface
parent-interface ethernet55
remote-ip 192.168.150.28
gre-key-decap 33554432
!
gre-tunnel-interface vcenter-abd08a56
direction receive-only
local-ip 192.168.200.254 mask 255.255.255.0 gateway-ip 192.168.200.1
origination vc8--interface
parent-interface ethernet55
remote-ip 192.168.50.29
gre-key-decap 33554432
  7. Enter the show running-config vcenter command to view the vCenter configuration. 
    controller-1# show running-config vcenter
! vcenter
vcenter vc-1
hashed-password 752a3a3211040e0200090409090611
host-name 10.8.23.70
mirror-type erspan
mirrored-packet-length 60
sampling-rate 60
user-name This email address is being protected from spambots. You need JavaScript enabled to view it.
!
vm-monitoring
vm vm-2001
vm vm-2002
  8. Configure the policies specifying the match rules and delivery interfaces. 
    controller-1(config)# policy dmf-policy-with-vcenter
controller-1(config-policy)# action forward
controller-1(config-policy)# filter-vcenter vc-1
controller-1(config-policy)# 1 match any
controller-1(config-policy)# delivery-interface TOOL-PORT-03
  9. Enter the show running-config policy command to view the automatically assigned filter interfaces. 
    controller-1# show running-config policy dmf-policy-with-vcenter
! policy
policy dmf-policy-with-vcenter
action forward
delivery-interface TOOL-PORT-03
filter-interface DMF-RU34-filter-vcenter-abd08a18 vc-1--interface
filter-interface DMF-RU34-filter-vcenter-abd08a37 vc-1--interface
filter-interface DMF-RU34-filter-vcenter-abd08a56 vc-1--interface
filter-vcenter vc-1
1 match any
    All the host tunnels belonging to vc-1 will become the filter interfaces. If new hosts are added, deleted, or modified, policies will be recomputed with the new interfaces.

VMs using SPAN on a Locally Connected vCenter Instance

To configure the DANZ Monitoring Fabric Controller for monitoring VMs using SPAN on a locally connected vCenter instance, perform the following steps:
  1. Add the vCenter instance details by entering the following commands. 
    controller-1(config)# vcenter vc-1
controller-1(config-vcenter)# host-name 10.8.23.70 
controller-1(config-vcenter)# password 094e470e2a121e060804
controller-1(config-vcenter)# user-name root
  2. Specify the mirror type by entering the following commands. 
    controller-1(config-vcenter)# mirror-type span
controller-1(config-vcenter)# sampling-rate 60 
controller-1(config-vcenter)# mirrored-packet-length 60
    The sampling-rate and mirrored-packet-length commands are optional.
  3. Add the VMs to monitor by entering the following commands. 
    controller-1(config-vcenter)# vm-monitoring 
controller-1(config-vcenter-vm-monitoring)# vm vm-2001 
controller-1(config-vcenter-vm-monitoring)# vm vm-2002
  4. To view the vCenter configuration, enter the show running-config vcenter command as in the following example. 
    controller-1# show running-config vcenter 
! vcenter
vcenter vc-1
hashed-password 752a3a3211040e0200090409090611
host-name 10.8.23.70
mirror-type span
mirrored-packet-length 60
sampling-rate 60
user-name This email address is being protected from spambots. You need JavaScript enabled to view it.
!
vm-monitoring
vm vm-2001
vm vm-2002
  5. Configure the policies specifying the match rules and delivery interfaces. 
    controller-1(config)# policy dmf-policy-with-vcenter
controller-1(config-policy)# action forward
controller-1(config-policy)# filter-vcenter vc-1
controller-1(config-policy)# 1 match any
controller-1(config-policy)# delivery-interface TOOL-PORT-03
    Note: LLDP automatically learns the filter interfaces. All the hosts belonging to vc-1 that have physical connections to DMF switches become the filter interfaces. If new connections are made later (or existing connections are changed), policies will be recomputed with the new interfaces.
  6. To view the automatically assigned filter interfaces, enter the show running-config policy command. 
    controller-1# show running-config policy dmf-policy-with-vcenter
! policy
policy dmf-policy-with-vcenter
action forward
delivery-interface TOOL-PORT-03
filter-interface vc-filter-1 origination vc-10-9-19-7--filter-interface
filter-interface vc-filter-3 origination vc-10-9-19-7--filter-interface
filter-vcenter vc-1
1 match any

VMs on a Second Locally Connected vCenter Instance

To configure the DMF Controller for monitoring VMs on a second locally connected vCenter instance, perform the following steps:
  1. Add the VMs to monitor and configure the DMF policies to specify the match rules and delivery interfaces. 
    (config)# vcenter vc-2
(config-vcenter)# host-name 10.8.23.71
(config-vcenter)# password 094e470e2a121e060804
(config-vcenter)# user-name root
(config-vcenter)# mirror-type span | erspan
(config-vcenter)# sampling-rate 60
(config-vcenter)# mirrored-packet-length 60
(config-vcenter)# vm-monitoring
(config-vcenter-vm-monitor)# vm vm-1001
(config-vcenter-vm-monitor)# vm vm-1002
  2. Configure the policy for the second vCenter instance. 
    (config)# policy dmf-policy-with-vcenter-2
(config-policy)# vcenter vc-2
(config-policy)# 1 match any
(config-policy)# delivery-interface TOOL-PORT-02

Using the GUI to View vCenter Configuration

After integrating a vCenter instance, click the link in the Name column in the vCenter table to view vCenter activity.
Figure 21. VMware vCenter Instance Name

DANZ Monitoring Fabric (DMF) displays the vCenter Info page.

Figure 22. VMware vCenter Configuration
The Info page displays information about the configuration of the vCenter instance. To view information about vCenter resources, scroll down to the following sections:
  • Hosts
  • Virtual Switches
  • Physical Connections
  • Virtual Machines
  • Network Host Connection Details
Figure 23. Hosts, Virtual Switches, and Physical Connections
Figure 24. Virtual Machines and Network Host Connection Details

Using the CLI to View vCenter Configuration

To view the vCenter configuration in the CLI, use the show vcenter command, as in the following examples: 
controller-1# show vcenter
#vCenter Name vCenter Host Name or IP Last vCenter Update Time Detail State vSphere Version
--|------------|-----------------------|------------------------------|----------------------------|---------------|
1vc-10-9-0-75 10.9.0.75 2017-09-0918:02:35.980000 PDTConnected and authenticated. 6.5.0
2vc-10-9-0-76 10.9.0.76 2017-09-0918:02:36.488000 PDTConnected and authenticated. 6.5.0
3vc-10-9-0-77 10.9.0.77 2017-09-0918:02:35.908000 PDTConnected and authenticated. 6.0.0
4vc-10-9-0-78 10.9.0.78 2017-09-0918:02:33.507000 PDTConnected and authenticated. 6.5.0
5vc-10-9-0-79 10.9.0.79 2017-09-0918:02:32.248000 PDTConnected and authenticated. 6.5.0
6vc-10-9-0-80 10.9.0.80 2017-09-0918:02:32.625000 PDTConnected and authenticated. 6.0.0
7vc-10-9-0-81 10.9.0.81 2017-09-0918:02:34.672000 PDTConnected and authenticated. 6.0.0
8vc-10-9-0-82 10.9.0.82 2017-09-0918:02:33.008000 PDTConnected and authenticated. 6.0.0
9vc-10-9-0-83 10.9.0.83 2017-09-0918:02:30.011000 PDTConnected and authenticated. 6.0.0
10 vc-10-9-0-84 10.9.0.84 2017-09-0918:02:33.024000 PDTConnected and authenticated. 6.5.0
11 vc-10-9-0-85 10.9.0.85 2017-09-0918:02:34.827000 PDTConnected and authenticated. 6.0.0
12 vc-10-9-0-86 10.9.0.86 2017-09-0918:02:35.164000 PDTConnected and authenticated. 6.0.0
13 vc-10-9-0-87 10.9.0.87 2017-09-0918:02:38.042000 PDTConnected and authenticated. 6.5.0
14 vc-10-9-0-88 10.9.0.88 2017-09-0918:02:37.212000 PDTConnected and authenticated. 6.0.0
15 vc-10-9-0-89 10.9.0.89 2017-09-0918:02:33.436000 PDTConnected and authenticated. 6.5.0
controller-1#

controller-1# show vcenter vc-10-9-0-75
#vCenter Name vCenter Host Name or IP Last vCenter Update Time Detail State vSphere Version
--|------------|-----------------------|------------------------------|----------------------------|---------------|
1vc-10-9-0-75 10.9.0.75 2017-09-0918:02:44.698000 PDTConnected and authenticated. 6.5.0
controller-1#

controller-1# show vcenter vc-10-9-0-75 detail
vCenter Name : vc-10-9-0-75
vCenter Host Name or IP : 10.9.0.75
Last vCenter Update Time : 2017-09-09 18:02:49.463000 PDT
Detail State : Connected and authenticated.
vSphere Version : 6.5.0
controller-1#

controller-1# show vcenter vc-10-9-0-75 error
vCenter Name : vc-10-9-0-75
vCenter Host Name or IP : 10.9.0.75
State : connected
Detail State : Connected and authenticated.
Detailed Error Info :
controller-1#

Integrating vCenter with DMF using Mirror Stack

DANZ Monitoring Fabric (DMF) vCenter integration supports mirroring from vCenter hosts using the default TCP/IP stack. However, this can result in traffic drops and affect production traffic since mirror traffic can conflict with production traffic. DMF vCenter integration with Mirror Stack provides the functionality to use the mirror TCP/IP stack for mirror sessions. Mirror stack in the ESXi host allows decoupling the traffic and keeps the production traffic unaffected.

vCenter configurations in DMF will use a mirror stack by default; however, if upgrading from previous DMF versions, the already configured vCenter will be set to use the default TCP/IP stack.

Platform Compatibility

vCenter integration with Mirror Stack requires an extra NIC on the ESXi host with following versions:
  • vCenter Server 7.0.x
  • vCenter Server 8.0.x

vCenter Configuration

DMF vCenter integration with Mirror Stack requires a mirror stack configuration on the ESXi host and vCenter.

Perform the following steps to configure the mirror stack on vCenter.

Repeat the steps for each ESXi host containing VMs to be monitored.

  1. Enable the mirror stack in the ESXi host if not already enabled.
    1. Use the esxcli network ip netstack list command to review the current network stacks. 
      [root@ESX33:~] esxcli network ip netstack list
defaultTcpipStack
 Key: defaultTcpipStack
 Name: defaultTcpipStack
 State: 4660

mirror
 Key: mirror
 Name: mirror
 State: 4660
      To view the TCP/IP configuration from vCenter UI, navigate to Host > Configure > TCP/IP.
      Figure 25. TCP/IP Configuration
    2. If the mirror stack is not configured, use the esxcli network ip netstack set -N mirror command to enable it.
      Note: The mirror setting is required to enable the Mirror TCP/IP stack and DMF integration.
  2. From vCenter create a VMkernel adapter with the mirror stack.
    Figure 26. VMKernel Network Adapter

    Select the appropriate network using the Browse option.

    Figure 27. Browse
    Click Next and select Port properties and choose mirror.
    Figure 28. Port Properties - Mirror
    Figure 29. Mirror Stack Added

    Add the IPv4 address and the Default gateway address according to your local network requirements.

    Figure 30. IP Address and Gateway Address
    Click Next.
    Figure 31. VMkernel Adapters
  3. Based on the networking requirements, configure the default gateway of the mirror stack in the host's TCP/IP configuration or a static route entry in the ESXi host to the DMF tunnel endpoint. The following example illustrates adding a static route entry to the DMF tunnel endpoint. 
    [root@ESX33:~] esxcli network ip route ipv4 add -n 192.168.200.0/24-g 192.168.150.1 -N mirror

[root@ESX31:~] esxcli network ip route ipv4 list -N mirror
NetworkNetmaskGatewayInterfaceSource
------------------------------------------------------
192.168.150.0255.255.255.00.0.0.0vmk2 MANUAL
192.168.200.0255.255.255.0192.168.150.1vmk2 MANUAL
  4. Navigate to Configure > TCP/IP Configuration > Select mirror stack > IPv4 Routing Table to view the routes.
    Figure 32. TCP/IP Configuration & IPv4 Routing Table
    Figure 33. Virtual Switch

Configuring DMF

Using the CLI

From the DMF Controller configure the TCP/IP stack using the tcp-ip-stack option in the vCenter config. The default and recommended value is mirror-stack. 
dmf-controller-1(conf)# vcenter vc8
dmf-controller-1(config-vcenter)# tcp-ip-stack
default-stack mirror-stack
dmf-controller-1(config-vcenter)# tcp-ip-stack mirror-stack

Using the GUI

To configure TCP/IP Stack, navigate to Integration > VMware vCenter. While adding or editing a vCenter configuration, select the appropriate choice using TCP/IP Stack. Default Stack and Mirror Stack are the options.

Figure 34. Create vCenter TCP/IP Stack
Attention: Encapsulated Remote mirroring with Default Stack is not recommended. Use Mirror Stack for optimal performance.

Show Commands

Use the show running-config command to view the tcp-ip-stack configuration.
Note: If mirror-stack is configured, it will only show when using the details token.
dmf-controller-1(config-vcenter)# show running-config vcenter vc8 details

! vcenter
vcenter vc8
default-tunnel-endpoint r34-lag-leaf1b
hashed-password <hashed-password>
host-name <ip-address>
mirror-type encapsulated-remote
tcp-ip-stack mirror-stack
user-name This email address is being protected from spambots. You need JavaScript enabled to view it.
View the existing mirror stack NICs and IPs of the host using the show vcenter vCenter name inventory command.
Note: v8 is an example vCenter name.
dmf-controller-1# show vcenter vc8 inventory
# vCenter ESXi Host Host DNS Name Cluster Product Name Hardware Model CPU Usage (%) Memory Usage (%) Virtual switches Mirror Stack VMkernel Adapter VMkernel Adapter IP Address
-|-------|-------------|-----------------------------------|---------------|--------------------------------|--------------|-------------|----------------|----------------|-----------------------------|---------------------------|
1 vc8 10.240.166.27 ESX27.qa.bsn.sjc.aristanetworks.com BSN-NSX-1 VMware ESXi 8.0.2 build-22380479 PowerEdge R430 2 15 3vmk1192.168.60.27
2 vc8 10.240.166.28 ESX28.qa.bsn.sjc.aristanetworks.com BSN-NSX-2 VMware ESXi 8.0.2 build-223804790 44
3 vc8 10.240.166.29 ESX29.qa.bsn.sjc.aristanetworks.com EdgeVMware ESXi 8.0.0 build-20513097 PowerEdge R430 4 23 3
4 vc8 10.240.166.33 ESX33.qa.bsn.sjc.aristanetworks.com vc8-mixed-stack VMware ESXi 8.0.2 build-223804790 63vmk1192.168.60.33
5 vc8 10.240.166.35 ESX35.qa.bsn.sjc.aristanetworks.com MGMTVMware ESXi 7.0.2 build-17867351 PowerEdge R430 2623 2
6 vc8 10.240.166.38 ESX38.qa.bsn.sjc.aristanetworks.com vc8-mixed-stack VMware ESXi 8.0.2 build-223804791 23 3vmk1192.168.60.38
dmf-rack#

Troubleshooting

Use the show fabric errors and show fabric warnings commands to troubleshoot and verify that everything is functioning as expected.

In the following example, the error message indicates that DMF could not find a route from the ESXi host to the DMF tunnel endpoint. 
dmf-controller-1# show fabric errors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ vCenter related error ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#vCenter Name Error
--|------------|--------------------------------------------------------------------------------------------------------------------------------------------|
1vc701Unable to locate a matching route for Mirror TCP/IP stack in host ESX37.qa.bsn.sjc.aristanetworks.com for DMF endpoint 192.168.200.254

Limitations

  • A port mirroring session remains on the original distributed virtual switch (DVS) when a VM migrates between DVSs.
  • Port mirroring sessions will persist on the DVS if a VM is renamed in vCenter while being monitored by DMF.
  • DMF cannot create a port mirroring session in the DVS if a conflicting session with the same VM exists in the DVS. This is not a limitation in vCenter 7.
  • When using mirror stack configuration in DMF, mirror sessions may still be created on the DVS for the ESXi host that doesn’t have a mirror stack configuration. This will result in no traffic being mirrored from the VM.
  • Auto-generated filter interfaces by vCenter integration should not be deleted from the policy. If they are deleted manually from the policy, they will not be automatically re-added.
  • DMF cannot monitor VMkernel adapters.

Wildcard Tunnels for VMware vCenter Monitoring

The current implementation of VMware vCenter creates one tunnel interface from every ESXi host to DMF.

Using a wildcard tunnel on DMF for VMware vCenter reduces the number of tunnels created.

Platform Compatibility

This feature is only compatible with switches that support wildcard tunneling.

Configuration

Configure wildcard tunnels using the CLI or the GUI.

Using the CLI to Create Wildcard Tunnels

The CLI construct wildcard-tunnels is available as a configuration option when configuring a VMware vCenter in DANZ Monitoring Fabric (DMF), as shown below:

Table 1. Commands
cluster Configure tunnel-endpoint for cluster
default-tunnel-endpoint Configure tunnel endpoints
description Describe this vCenter
hashed-password

Set the vCenter password (to log into vCenter)
host-name Set the vCenter hostname
mirror-type

Set the vCenter vm monitoring mode
mirrored-packet-length

Set the mirrored packet length
password

Set the vCenter password (to log into vCenter)
sampling-rate Set the packet sampling rate
user-name

Set the vCenter user name (to log into vCenter)
vm-monitoring Enter vm-monitoring config submode
wildcard-tunnels Enable wildcard tunnels

Enable wildcard tunnels by setting the above leaf parameter, as shown in the following example of vCenter configuration on the Controller node.

dmf-controller-1(config)# vcenter VC1
dmf-controller-1(config-vcenter)# wildcard-tunnels 
dmf-controller-1(config-vcenter)# show this
! vcenter
vcenter VC1
wildcard-tunnels
dmf-controller-1(config-vcenter)#

Similarly, disable wildcard tunnels by issuing the no command as shown below:

dmf-controller-1(config-vcenter)# show this
! vcenter
vcenter VC1
wildcard-tunnels
dmf-controller-1(config-vcenter)# no wildcard-tunnels 
dmf-controller-1(config-vcenter)# show this
! vcenter
vcenter VC1
dmf-controller-1(config-vcenter)#

Show Commands

There is no specific show command for wildcard tunnels; however, check them in the vCenter running config. In addition, the show tunnels command shows the tunnels created for the selected vCenter configuration with a wildcard remote IP address.

Troubleshooting

Verify errors and warnings are clear using the show fabric errors and show fabric warnings commands. The show tunnels command displays tunnels created based on the vCenter configuration on the Controller with a wildcard remote IP address. Use the show switch <name> table gre-tunnel command to display tunnels programmed on the switch.

Using the GUI to Create Wildcard Tunnels

Use the DANZ Monitoring Fabric (DMF) GUI to create wildcard tunnels as outlined below.

Navigate to the Integration > VMware vCenter page.
Figure 35. VMware vCenter Add/Edit

Click the Menu icon.

As part of the Options step of the Add/Edit vCenter workflow, enable wildcard tunnels using the Create Wildcard Tunnels toggle input. By default, the feature is disabled.
Figure 36. VMware vCenter Create vCenter Options

Limitations

Select Broadcom® switch ASICs support wildcard tunnels; ensure your switch model supports this feature before configuring it for vCenter.

Please refer to the Platform Compatibility section for more information.

Minimum Permissions for Non-admin Users

For a non-admin user to add, remove, edit, or monitor a vCenter via the DANZ Monitoring Fabric (DMF), the privilege level assigned to the non-admin user is VSPAN operation. To assign VSPAN operation privileges to a user, perform the following steps:

  1. From the vCenter GUI, navigate to Menu > Administration.
  2. Once on the page, click on the Users and Groups link in the navigation bar on the left.
    Figure 37. Users and Groups
  3. Click on the Users tab and ensure the appropriate domain is selected (in this case, the domain is vsphere.local).
    Figure 38. Domain Selection
  4. Next, click on the ADD USER link and create the desired user. (In the example below, a user called dmf-aliceis created.)
    Figure 39. Add a New User
  5. Verify that the newly created user is on the Users and Groups page.
    Figure 40. Verify User Created
  6. After creating the desired user, create and assign a role to this user. Click on Roles under Access Control in the navigation bar on the left. Next, click on the + sign to add a new role.
    Figure 41. Add a New Role
  7. In the New Role pop-up dialog, select Distributed Switch from the left and then scroll down to find and select VSPAN operation as the role. Click Next and give the new role a new name. (In the example below the new role monitor-dmf is created.) Click Finish to create the new role.
    Figure 42. Select Role Type
    Figure 43. Save New Role
  8. Verify the creation of the new role on the Roles page.
    Figure 44. Verify New Role Created
  9. To assign the new role to the new user, click the Global Permissions link in the navigation bar on the left. Next, click on the + sign to assign the new role.
    Figure 45. Global Permissions
  10. In the Add Permission dialog, type the newly created username and select the newly created role, as shown in the figure below.
    Note:Do not forget to check mark the Propagate to children checkbox.
    Figure 46. Assign Role to User
  11. Verify assigning the newly created role to the newly created user.
    Figure 47. Verify Role Assignment to User

Monitor VMware vCenter Traffic by VM Names

Match VMware vCenter-specific information in the policy. Specifically, this feature matches traffic using VMware vCenter Virtual Machine (VM) names and requires DANZ Monitoring Fabric (DMF) vCenter integration.

Using the CLI to Monitor vCenter Traffic by VM Names

Configuration

This feature works with vCenter integration; therefore, configure vCenter Integration in DANZ Monitoring Fabric (DMF). Configure vCenter mapping in the policy, then define a policy match using VM names in the vCenter as illustrated in the following configuration example: 
dmf-controller-1(config)# policy v1
dmf-controller-1(config-policy)# action forward
dmf-controller-1(config-policy)# filter-interface filter-interface
dmf-controller-1(config-policy)# delivery-interface delivery-interface
dmf-controller-1(config-policy)# filter-vcenter vcenter-name
dmf-controller-1(config-policy)# 1 match ip src-vm-name vm-name dst-vm-name vm-name
dmf-controller-1(config-policy)# 2 match ip6 src-vm-name vm-name

Show Commands

Enter the show running-config policy policy name command to display the configuration. 
dmf-controller-1# show running-config policy v1

! policy
policy v1
action forward
delivery-interface delivery-interface
filter-interface filter-interface
filter-vcenter vcenter-name
1 match ip src-vm-name vm-name dst-vm-name vm-name
2 match ip6 src-vm-name vm-name
The show policy policy name command displays the policy information, including stats. 
dmf-controller-1# show policy v2
Policy Name: v2
Config Status: active - forward
Runtime Status : installed
Detailed Status: installed - installed to forward
Priority : 100
Overlap Priority : 0
# of switches with filter interfaces : 1
# of switches with delivery interfaces : 1
# of switches with service interfaces: 0
# of filter interfaces : 1
# of delivery interfaces : 1
# of core interfaces : 0
# of services: 0
# of pre service interfaces: 0
# of post service interfaces : 0
Push VLAN: 5
Post Match Filter Traffic: -
Total Delivery Rate: -
Total Pre Service Rate : -
Total Post Service Rate: -
Overlapping Policies : none
Component Policies : none
Installed Time : 2023-12-21 19:00:39 UTC
Installed Duration : 50 minutes, 11 secs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Match Rules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Rule
-|--------------------------------------------------------------------------|
1 1 match ip src-vm-name DMF-RADIUS-SERVER-1 dst-vm-name DMF-TACACS-SERVER-1

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Filter Interface(s)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF Switch IF NameState Dir Packets Bytes Pkt Rate Bit Rate Counter Reset Time
-|----------------|----------|----------|-----|---|-------|-----|--------|--------|------------------------------|
1 span_from_arista DELL-S4048 ethernet20 uprx0 0 0-2023-12-21 19:00:39.941000 UTC

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Delivery Interface(s)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF Switch IF NameState Dir Packets Bytes Pkt Rate Bit Rate Counter Reset Time
-|------------|----------|------------|-----|---|-------|-----|--------|--------|------------------------------|
1 ubuntu-tools DELL-S4048 ethernet50/2 uptx0 0 0-2023-12-21 19:00:39.941000 UTC
~ Service Interface(s) ~
None.
~ Core Interface(s) ~
None.
~ Failed Path(s) ~
None.
The show vcenter vcenter name endpoint command displays the vCenter VM information, including networks. 
dmf-controller-1# show vcenter vcenter1 endpoint 
#vCenter Name VM Name ESXi Host Name Network Interface Name MAC AddressIP Address Virtual Switch Portgroup Power State 
--|------------|---------|--------------|----------------------|--------------------------|------------------------------------------|--------------|-------------|-----------|
1vcenter1 ub-11-216 10.240.155.216 Network adapter 100:50:56:8b:4d:03 (VMware) 1.1.11.216/24, fe80::250:56ff:fe8b:4d03/64 DVS-DMFvlan11powered-on
2vcenter1 ub-12-216 10.240.155.216 Network adapter 100:50:56:8b:72:a0 (VMware) 1.1.12.216/24, fe80::250:56ff:fe8b:72a0/64 DVS-DMFvlan12powered-on
3vcenter1 ub-13-216 10.240.155.216 Network adapter 100:50:56:8b:c0:06 (VMware) 1.1.13.216/24, fe80::250:56ff:fe8b:c006/64 DVS-DMFvlan-10 powered-on
4vcenter1 ub-14-216 10.240.155.216 Network adapter 100:50:56:8b:d1:d9 (VMware) 1.1.14.216/24, fe80::250:56ff:fe8b:d1d9/64 DVS-DMFvlan-10 powered-on

Using the GUI to Monitor vCenter Traffic by VM Names

Configure vCenter VM name matches under the DANZ Monitoring Fabric (DMF) policies match rules section. For example:
  1. In the DMF GUI, navigate to the Monitoring > Policies page.
    Figure 48. DMF Policies
  2. Click Create Policy to create a new policy or edit an existing one by selecting a row from the Policies Table and clicking Edit.
    Figure 49. Create / Edit Policy
  3. Navigate to the Match Traffic tab.
    Figure 50. Match Traffic
  4. Click Configure a Rule to configure a custom match rule.
    Figure 51. Configure a Rule
  5. Set the EtherType to IPv4 or IPv6.
  6. Add the Source IP Address as the vCenter VM name. Select the Virtual Machine option from the Source IP Address drop-down and select a virtual machine from the VM Name drop-down.
    Figure 52. Source IP Address VM Name
  7. Add the Destination IP address as the vCenter VM name. Select the Virtual Machine option from the Destination IP Address drop-down and select a virtual machine from the VM Name drop-down.
    Figure 53. Destination IP VM
    Note: If the VM Name drop-down shows No Data, ensure only one vCenter is affiliated with the policy (under Traffic Sources).
  8. Click Add Rule to add the match rule to the policy.
  9. After entering other inputs as required, click Create Policy (or Save Policy) to save the configuration.

Troubleshooting

Fabric errors and warnings are very useful for troubleshooting this feature.

When using the show fabric warnings command, the following validation message displays when the vCenter integration cannot resolve the IP address for the VM name used in the policy. 
dmf-controller-1# show fabric warnings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Policy related warning~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Policy Name Warning
-|-----------|------------------------------------------------------------------------------------------------------------|
1 v1No IP found for VMs [ub-15-216, ub-216-multinic, ub-217-vlan10, ub-14-216, ub-11-216] associated with policy

When VM names used in a policy are matched, the following validation message content appears when a vCenter instance is not associated with the policy.

dmf-controller-1# show fabric warnings 
~~~~~~~~~~~~~~~~~~~ Policy related warning ~~~~~~~~~~~~~~~~~~~
# Policy Name Warning 
-|-----------|-----------------------------------------------|
1 v1No vCenter associated to policy with VM matches

Limitations

  • This feature only works with vCenter integration and a direct Switch Port Analyzer (SPAN) from a switch with ESXi traffic.
  • VM interface IP addresses connected to dvs will only be added to policy matches.
  • The system may use extra TCAM entries if the management network uses dvs.
  • Vmkernal names cannot be matched in the policy.
  • When a VM name with multiple vNICs (multiple IP addresses) matches the policy, a TCAM entry is added for all the IP addresses.
  • VM Names cannot be matched with the MAC option in the policy.
  • If the vCenter becomes disconnected, policies associated with the VM names may not get correct matches or traffic.