DHCPv4-Based First-boot for DMF Controller and Managed Appliances

This chapter outlines a solution for provisioning DANZ Monitoring Fabric (DMF) appliances via PXE and automating the configuration of the first boot parameters using Ansible.

Introduction

Typically, the deployment of DANZ Monitoring Fabric (DMF) on supported hardware appliances involves two steps:
  • Installing an appropriate image.
  • Configuration of firstboot parameters such as IP address (DHCP/Static), DNS and NTP server address, admin password(s), cluster information etc.

In this context, Pave refers to the automation of the first-time installation of a DMF hardware appliance. This involves installing a DMF image on a hardware appliance and completing the first-boot configuration. First-boot configuration uses an Ansible playbook as an automation tool. In contrast, Repave refers to automating the re-installation of DMF images on supported DMF appliances. This involves the automated process of re-installing a DMF image on a DMF hardware appliance and completing the first-boot configuration.

Several prerequisites are required. The production/lab environment should have:
  • A DHCP server that supports the configuration of DHCP options 66 and 67.
  • A TFTP server that can serve the bootloader and the corresponding configuration.
  • An NFS server to serve the net-bootable appliance image.
  • A server with ansible-playbook and Arista-supported playbook modules installed.
  • Supported DMF hardware appliances with preset boot order settings and a PXE-enabled management port.
Note: In the case of Repave action, this new feature introduces a new command (boot pxe) to change the boot order to PXE boot. On reboot, the appliance will automatically perform image re-installation.

Prepare Services (TFTP/NFS) for PAVE/REPAVE Operation

The following steps must be completed before deploying the DANZ Monitoring Fabric (DMF) ISO image on a TFTP server.

Create a directory by name images on the TFTP server and copy the ISO image to the TFTP server.

  1. SSH to the server with sudo privileges and create a temporary directory using the command below.
    $ mktemp -d /tmp/tmp.2syaj0amL7
  2. Mount the ISO image to the directory created above.
    $ mount /images/*.iso /tmp/tmp.2syaj0amL7
  3. Copy the following files to the root TFTPboot directory.
    $ cp /usr/lib/PXELINUX/pxelinux.0 /var/lib/tftpboot
    $ cp /usr/lib/syslinux/modules/bios/ldlinux.c32 /var/lib/tftpboot
    If the above files are unavailable on the system, obtain them via an apt-get or yum command.
    $ apt-get install pxelinux
  4. Update the TFTP_DIRECTORY variable with the parent directory created to store bootloader files.
    $ sed -i "/^TFTP_DIRECTORY=/c\TFTP_DIRECTORY=/var/lib/tftpboot" /etc/default/tftpd-hpa
  5. Create an appropriate folder under the TFTP root directory for each DMF appliance type.
    $ mkdir /var/lib/tftpboot/dmf-controller/
    $ mkdir /var/lib/tftpboot/dmf-service-node/
    $ mkdir /var/lib/tftpboot/dmf-analytics-node/
    $ mkdir /var/lib/tftpboot/dmf-recorder-node/
  6. Create an appropriate folder on the NFS root directory for each DMF appliance type.
    $ mkdir path_to_NFS_root_directory/dmf-controller/
    $ mkdir path_to_NFS_root_directory/dmf-service-node/
    $ mkdir path_to_NFS_root_directory/dmf-analytics-node/
    $ mkdir path_to_NFS_root_directorydmf-recorder-node/
  7. Copy the files from the folder mounted earlier in Step 3.
    $ cp “/tmp/tmp.2syaj0amL7/casper/vmlinuz” "/var/lib/tftpboot/dmf-controller"
    $ cp "/tmp/tmp.2syaj0amL7/casper/initrd.lz" "/var/lib/tftpboot/dmf-controller"
    $ cp -r "$/tmp/tmp.2syaj0amL7/." "path_to_NFS_root_directory/dmf-controller"
  8. Update the ownership of the tftp parent directory with TFTP user account and restart the tftp service.
    $ chown -R tftp:tftp /var/lib/tftpboot
    $ systemctl restart tftpd-hpa
  9. Copy kernel configurations that assist the DMF appliance booting with UEFI mode to locate the bootloader and vmlinuz files from the TFTP and NFS server. Arista recommends defining a name for menu entry so that it is easily remembered and prefixed with appliance type for differentiation, i.e., DCA-DM-CDL-dmf-8.4.0.
    Note:In the currently supported UEFI boot mode, Arista recommends creating a PXE configuration file for each hardware MAC address, i.e., instead of the default grub.cfg use grub.cfg-01-xx-xx-xx-xx-xx-xx . Specify the MAC address in all lower-case with : (colon) replaced by - (dash).
    To perform the action detailed in the note above, obtain the MAC / Hardware address of the management interfaces. Obtaining the MAC address can be done in two ways, as described below:
    1. If this is the first time the appliance is being installed, either:
      • Use the Integrated Dell Remote Access Controller (iDRAC) web console to determine the MAC address of the management interface.

        Figure 1. Using the iDRAC menu to obtain the Management Interface MAC Address
      • Enter the device BIOS menu during boot-up and determine the MAC address of the management interface.

        Figure 2. Using the BIOS menu to obtain the Management Interface MAC Address
    2. When performing a re-installation of DMF appliances, obtain the MAC address of the appliance via the CLI command.
      • For DMF Controllers with SKU DCA-DM-C450 or DMF Analytics Nodes with SKU DCA-DM-AN450, execute the following commands on the Controller to obtain the relevant MAC address.

        DCA-DM-C450# show local-node interfaces eno8303
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstateCarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno8303bond0 d0:8e:79:d4:1e:56 (Dell)d0:8e:79:d4:1e:56 (Dell)up upactive
        ~ Address ~
        None.
        DCA-DM-C450# show local-node interfaces eno8403
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstat CarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno8403d0:8e:79:d4:1e:57 (Dell)d0:8e:79:d4:1e:57 (Dell)downdown
        ~ Address ~
        None.
        DCA-DM-C450#
      • For DMF Controllers with SKU DCA-DM-CDL, including all DMF Recorder Nodes, execute the following commands on the device to obtain the relevant MAC address.

        DCA-DM-CDL# show local-node interfaces eno1
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstateCarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno1 bond0 d0:94:66:21:b9:45 (Dell)d0:94:66:21:b9:45 (Dell)upup active
        ~ Address ~
        None.
        DCA-DM-CDL# show local-node interfaces eno2
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstateCarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno2 bond0 d0:94:66:21:b9:45 (Dell)d0:94:66:21:b9:46 (Dell)downdown backup
        ~ Address ~
        None.
        DCA-DM-CDL#
      • For all DMF Service Nodes, execute the following commands on the device to obtain the relevant MAC address.

        dmf-service-node# show local-node interfaces eno3
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstateCarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno3 bond0 78:ac:44:8a:59:52 (Dell)78:ac:44:8a:59:52 (Dell)upup active
        ~ Address ~
        None.
        dmf-service-node# show local-node interfaces eno4
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interfaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        InterfaceMasterHardware addressPermanent hardware addressOperstateCarrierBond modeBond role
        --------- |------ |------------------------ |-------------------------- |--------- |------- |--------- |--------- |
        eno4 78:ac:44:8a:59:53 (Dell)78:ac:44:8a:59:53 (Dell)downdown
        ~ Address ~
        None.
        dmf-service-node#
    • If both the management ports of the appliance are connected to the top of the rack management switch, Arista recommends generating a separate PXE configuration file for both management ports.
      cat << EOF | sudo tee /var/lib/tftpboot/boot/grub/grub.cfg-01-0a-0b-0c-0d-0e-0f
      set default="0"
      loadfont unicode
      set gfxmode=auto
      insmod all_video
      insmod gfxterm
      serial --unit=0 --speed=115200
      serial --unit=1 --speed=115200
      terminal_input console
      terminal_input --append serial_com0
      terminal_input --append serial_com1
      terminal_output gfxterm
      terminal_output --append serial_com0
      terminal_output --append serial_com1
      set timeout=5
      menuentry "Install <INSERT_USER_FRIENDLY_NAME>" {
      linux dmf-controller/vmlinuz boot=casper netboot=nfs nfsroot=<ip-address of PXE server>
      :/srv/install/dmf-controller toram noprompt ip=dhcp -- unattended_installation autofirstboot_
      via_ssh
      initrd dmf-controller/initrd.lz
      }
      EOF

DHCPv4-Based First-boot

- DHCPv4-based first-boot for DMF Controller and Managed Appliances

There are two main steps involved in accomplishing the automatic installation of an image on a supported DANZ Monitoring Fabric (DMF) hardware appliance and the completion of the first-boot configuration.

Auto Installation of Images

Auto-installation of a DMF image uses well-known services like DHCP, TFTP, NFS, and PXE. DMF images are PXE bootable, and using the services above, perform the auto-installation of images on the DMF appliances.

A high-level procedure for auto-installation is described below.

  • Configure a DHCP server to provide a DHCP IP address.
  • Enter the next-server IP address (This is the TFTP server IP address specified by Option 66 configuration on the DHCP server).

    next-server <TFTP_SERVER_IP>;
    class "pxeclient" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
    filename "boot/grub/x86_64-efi/core.efi"; # x86 EFI
    }
  • Arista recommends binding a static IP to the hardware MAC address of the DMF appliance. The following is a sample configuration using a Linux-based DHCP service.

    group {
    host <DMF_APPLIANCE_HOSTNAME> {
    hardware ethernet <DMF_APPLIANCE_MAC>;
    fixed-address <DESIRED_IP_ADDRESS>;
    }
  • On a DMF HW appliance using PXE boot, enable PXE on the management interface NIC card.
    Figure 3. Enable interfaces in BIOS to be PXE bootable
  • For the initial PAVE action, press the F12 key during the initial boot process to manually trigger the PXE boot. This step is typically not required when reinstalling (REPAVE) the DMF appliance images.
    Figure 4. Manual PXE Boot by pressing F12
  • Power cycle the DMF HW appliance.
  • The DHCP client on the NIC card sends a DHCP discovery request and gets an IP address and next-server (TFTP server) IP address.
  • The management NIC gets the bootloader via PXE boot using a TFTP server. A bootloader config file with a filename based on the MAC address of the appliance is configured and saved on the DHCP server.
  • The appliance obtains configuration parameters like the NFS mount, where appliance ISO images are stored.
  • The appliance boots from the boot loader and obtains the DMF appliance ISO image from the NFS server.
  • The appliance boots the ISO image and starts the installation process without user input.
  • On reboot, the appliance again acquires a DHCP-based management IP address, sets up initial default user credentials, and waits for auto first-boot configuration via Ansible.
    Note: For repave, the procedure is the same except that the DMF appliance is already running a DMF image and needs to boot from PXE again for re-installation.

Auto Configuration of First-boot

Auto-configuring of first-boot parameters uses Ansible and is accomplished by interacting with auto-firstboot-cloud-plugin, which is available in hardware appliances.

  • Begin the initial configuration playbook in Ansible. The following is a sample YAML-based Ansible playbook file.
    - name: Test Autofirstboot Properties Provider
    gather_facts: false
    hosts: Controllers
    connection: "local"
    run_once: true
    tasks:
    - name: Provide Autofirstboot Properties to Cluster
    arista.dmf.provisioner:
    config_json: "<Pave-Repave-config.json>"
    timeout: 3600
    log_dir: "logs"
  • The configuration playbook obtains the JSON-based configuration file from the server executing the playbook. The JSON file contains specific sections for each DMF appliance that is being installed/re-installed. The following is an example:
    {
    "<ACTIVE-CONTROLLER-IP>": {
    "initial-admin-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-PASSWD>",
    "initial-config-password": "pxe_temp_password",
    "dhcp-ip": "10.240.156.82",
    "appliance-type": "BMF",
    "firstboot-properties": {
    "admin-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-PASSWD>",
    "recovery-password": "<ACTIVE-CONTROLLER-CLEAR-TEXT-RECOVERY-PASSWD>",
    "hostname": "<ACTIVE-CONTROLLER-HOSTNAME>",
    "cluster-name": "<CLUSTER-NAME>",
    "cluster-description": "<CLUSTER-DESCRIPTION>",
    "ip-stack": "ipv4",
    "ntp-servers": ["<CLUSTER-NTP-SERVER-1>","<CLUSTER-NTP-SERVER-2>"],
    "dns-servers": ["<CLUSTER-DNS-SERVER-1>","<CLUSTER-DNS-SERVER-2>"]
    }
    },
    "<STANDBY-CONTROLLER-IP>": {
    "initial-config-password": "pxe_temp_password",
    "dhcp-ip": "<STANDBY-CONTROLLER-DHCP-IP>",
    "appliance-type": "BMF",
    "firstboot-properties": {
    "admin-password": "<STANDBY-CONTROLLER-CLEAR-TEXT-PASSWD>",
    "recovery-password": "<STANDBY-CONTROLLER-CLEAR-TEXT-RECOVERY-PASSWD>",
    "hostname": "<STANDBY-CONTROLLER-HOSTNAME>",
    "cluster-name": "<CLUSTER-NAME>",
    "cluster-to-join": "<ACTIVE-CONTROLLER-IP>",
    "ip-stack": "ipv4"
    }
    },
    "<MANAGED-APPLIANCE-IP>": {
    "initial-config-password": "pxe_temp_password",
    "dhcp-ip": "<STANDBY-CONTROLLER-DHCP-IP>",
    "appliance-type": "BMFSN",
    "firstboot-properties": {
    "admin-password": "<MANAGED-APPLIANCE-CLEAR-TEXT-PASSWD>",
    "recovery-password": "<MANAGED-APPLIANCE-CLEAR-TEXT-RECOVERY-PASSWD>",
    "hostname": "<MANAGED-APPLIANCE-HOSTNAME>",
    "ip-stack": "ipv4",
    "controller_ip": "<ACTIVE-CONTROLLER-IP>",
    "ntp-servers": ["<NTP-SERVER-1>","<NTP-SERVER-2>"],
    "dns-servers": ["<DNS-SERVER-1>","<DNS-SERVER-2>"]
    }
    Note: The ansible script uses the initial-config-password to access the DMF appliance via SSH for the first time.
  • Ansible pushes initial configuration data to the appliance via SSH using the default credential configured in the JSON file in the steps above.
  • The appliance completes the initial configuration (first-boot).
  • The appliance confirms to Ansible that the initial configuration is complete.
The following is an example run of the Ansible playbook:
ansible-playbook sample_playbook.yml --limit="10.240.156.82,10.240.156.84" -v
[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting
with Ansible 2.12. Current version: 2.7.18 (default, Jul 1 2022, 12:27:04) [GCC 9.4.0].
This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
/root/.cache/pypoetry/virtualenvs/arista-dmf-vQFV2Q4_-py2.7/lib/python2.7/site-packages/
ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 2 is no longer
supported by the Python core team. Support for it is now deprecated in cryptography, and
will be removed in the next release.
from cryptography.exceptions import InvalidSignature
Using /etc/ansible/ansible.cfg as config file
PLAY [Test Autofirstboot Properties Provider]
*************TASK [Provide Autofirstboot Properties to Cluster]
*************Targeting all hosts specified in config [u'10.240.156.84', u'10.240.156.82']
[10.240.156.82 BMF (active)]: Waiting for SSH connection.
[10.240.156.84 BMF (standby)]: Waiting for SSH connection.
[10.240.156.84 BMF (standby)]: Established SSH connection.
[WARNING]: It is NOT recommended to disable SSL verification. Please upload a valid SSL
certificate to the https://10.240.156.84:8443 API server.
[10.240.156.82 BMF (active)]: Established SSH connection.
[WARNING]: It is NOT recommended to disable SSL verification. Please upload a valid SSL
certificate to the https://10.240.156.82:8443 API server.
[WARNING]: [10.240.156.82 BMF (active)]: Detected BIOS firmware. The BIOS PXE boot trigger has
been found to be unreliable. Upgrade to UEFI is recommended.
[WARNING]: [10.240.156.84 BMF (standby)]: Appliance already configured. Attempting PXE boot to
repave.
[WARNING]: [10.240.156.82 BMF (active)]: Appliance already configured. Attempting PXE boot to
repave.
[10.240.156.84 BMF (standby)]: Expecting appliance to reboot with IP = 10.240.156.84.
[10.240.156.82 BMF (active)]: Expecting appliance to reboot with IP = 10.240.156.82.
[10.240.156.84 BMF (standby)]: Verified repave succeeded.
[10.240.156.84 BMF (standby)]: Waiting for active to be configured.
[10.240.156.82 BMF (active)]: Verified repave succeeded.
[10.240.156.82 BMF (active)]: Successfully provided config to appliance.
[10.240.156.82 BMF (active)]: Waiting for appliance to apply config. Expecting final IP = 10.
240.156.82.
[10.240.156.82 BMF (active)]: Successfully retrieved firstboot logs.
[10.240.156.82 BMF (active)]: Config applied.
[10.240.156.82 BMF (active)]: Attempting to reset recovery password.
[10.240.156.82 BMF (active)]: Successfully reset recovery password.
[10.240.156.84 BMF (standby)]: Active configured. Proceeding with configuration.
[10.240.156.84 BMF (standby)]: Successfully provided config to appliance.
[10.240.156.84 BMF (standby)]: Waiting for appliance to apply config. Expecting final IP = 10.
240.156.84.
[10.240.156.84 BMF (standby)]: Successfully retrieved firstboot logs.
[10.240.156.84 BMF (standby)]: Config applied.
[10.240.156.84 BMF (standby)]: Attempting to reset recovery password.
[10.240.156.84 BMF (standby)]: Successfully reset recovery password.
changed: [10.240.156.82] => {"changed": true, "success": true, "summary": {"10.240.156.82": {
"appliance-type": "BMF", "expected-dhcp-ip": "10.240.156.82", "role": "active"}, "10.240.156.
84": {"appliance-type": "BMF", "expected-dhcp-ip": "10.240.156.84", "role": "standby"}}}
PLAY RECAP
******************56.82 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Assumption and Trust Model

PXE boot is fundamentally incompatible with zero-trust environments, and as a result, assumptions are made to establish the trust required to authenticate the appliances. This section provides a summary of the assumptions that underpin the security of this design.

The management network is trusted, such that:

  • DHCP is secured.
    • The DHCP server (or DHCP relaying) is secure.
    • Rogue DHCP packets are dropped/blocked.
  • Impersonation by MAC or IP spoofing is not possible due to either:
    • Assumption: Machines on the same L2 network are trusted not to impersonate other machines by presenting false identities (MAC addresses or IP addresses).
    • Guarantee: Machines in the same L2 network cannot impersonate other machines by presenting themselves with false identities as enforced by network admins who may, for example:
      • Pin a MAC address to a specific switch and switch port.
      • Pin an IP address to a MAC address statically (i.e., static ARP entry).
  • Routers between the PXE TFTP/HTTP server and the target machine are secure/trusted to forward packets to the rightful owners (e.g., having correct routing tables and MAC address tables).
  • The PXE (TFTP/NFS) server is secure and cannot be compromised, resulting in an attacker providing a malicious image.