Using the DMF Recorder Node
- Installing and configuring DMF Recorder Node: DANZ Monitoring Fabric Deployment Guide
- Integrating DMF Recorder Node with Analytics: Arista Analytics User Guide
- Stenographer Queries: Stenographer Reference for DMF Recorder Node
- DMF Recorder Node REST API: DMF Recorder Node REST APIs
Overview
The DANZ Monitoring Fabric (DMF) Recorder Node is integrated with the DANZ Monitoring Fabric for single-pane-of-glass monitoring. A single DMF Controller can manage multiple Recorder Nodes, delivering packets for recording through Out-of-Band policies. The DMF Controller also provides central APIs for packet queries across one or multiple recorder nodes and for viewing errors, warnings, statistics, and the status of connected recorder nodes.
A DMF out-of-band policy directs matching packets to be recorded to one or more recorder nodes. A recorder node interface identifies the switch and port used to attach the recorder node to the fabric. A DMF policy treats these as delivery interfaces and adds them to the policy so that flows matching the policy are delivered to the specified recorder node interfaces.
Configuration Summary
At a high level, follow the below three steps to use the recorder node.
Step 1: Define a recorder node.
Step 2: Define a DANZ Monitoring Fabric (DMF) policy to select the traffic to forward to the recorder node.
Step 3: View and analyze the recorded traffic.
- Name: Each recorder node requires a name that is unique among recorder nodes in the connected fabric. If the name is removed, all configuration for the given recorder node is removed.
- Management MAC address: Each recorder node must have a management MAC address that is unique in the connected fabric.
- Packet removal policy: This defines the behavior when the recorder node disks reach capacity. The default policy causes the earliest recorded packets to be overwritten by the most recent packets. The other option is to stop recording and wait until space is available.
- Record enable or Record disable: Recording of packets is enabled by default, but it can be enabled or disabled for a specific recorder node.
- Static auth tokens: Static auth tokens are pushed to each recorder node as an alternative form of authentication in headless mode, when the DMF Controller is unreachable, or by third-party applications that do not have or do not need DMF Controller credentials.
- Controller auth token: The recorder node treats the controller as an ordinary client and requires it to present valid credentials in the form of an authentication token. The DMF Controller authentication token is automatically generated but can be reset upon request.
- Pre-buffer: This buffer, which is defined in minutes, is used for proactive network monitoring without recording and retaining unnecessary packets. Once the buffer is full, the oldest packets are deleted.
- Maximum disk utilization: This defines the maximum disk utilization in terms of a percentage between 5% and 95%. When the configured utilization is reached, the packet removal policy is enforced. The default maximum disk utilization is 95%.
- Maximum packet age: This defines the maximum age in minutes of any packet in the recorder node. It can be used in combination with the packet removal policy to control when packets are deleted based on age rather than disk utilization alone. When not set, the maximum packet age is not enforced and packets are kept until the maximum disk utilization is reached.
Indexing Configuration
The recorder node indexing configuration defines the fields that can be used to query packets on the recorder node. By default, all indexing fields are enabled in the indexing configuration. You can selectively disable indexing fields you do not wish to use in recorder node queries.
Disabling indexing fields has two advantages. First, it reduces the index space required for each packet recorded. Second, it improves query performance by reducing unnecessary overhead. It is recommended that unnecessary indexing fields be disabled.
- MAC Source
- MAC Destination
- VLAN 1: Outer VLAN ID
- VLAN 2: Inner/Middle VLAN ID
- VLAN 3: Innermost VLAN ID
- IPv4 Source
- IPv4 Destination
- IPv6 Source
- IPv6 Destination
- IP protocol
- Port Source
- Port Destination
- MPLS
- Community ID
- MetaWatch Device ID
- MetaWatch Port ID
To understand how indexing configuration can be leveraged to your advantage, consider the following examples:
Example 1: To query packets based on applications defined by unique transport ports, disable all indexing fields except source and destination transport ports. This results in only transport ports being saved as meta data for each packet recorded. This greatly reduces per-packet index space consumption and also increases the speed of recorder-node queries.
However, you will not be able to effectively query on any other indexing field because that meta data was not saved when the packets were recorded.
Example 2: The recorder node supports community ID indexing, which is a hash of IP addresses, IP protocol, and transport ports that can be used to identify a flow of interest. If the recorder node use case is to query based on community ID, it might be redundant to index on IPv4 source and destination addresses, IPv6 source and destination addresses, IP protocol, and transport port source and destination addresses.
Pre-buffer Configuration and Events
The recorder node pre-buffer is a circular buffer in which packets to be recorded are received. When enabled, the pre-buffer feature allows for retention of the packets received by the recorder node for a specified length of time prior to an event that triggers recording of buffered and future packets to disk. In the absence of an event, the recorder node will record into this buffer, deleting the oldest packets in the buffer when the buffer reaches capacity. When a recorder node event is triggered, the packets in the pre-buffer are saved to disk, and the packets received from the time of the event trigger to the time of the event termination are saved directly to disk upon termination of the event, received packets are received and retained in the pre-buffer until the next event. By default, the pre-buffer feature is disabled, indicated by a value of zero minutes.
For example, if you configure the pre-buffer to thirty minutes, up to thirty minutes of packets will be received by the buffer. When you trigger an event, the packets currently in the buffer are recorded to disk, and packets newly received by the recorder node bypass the buffer and are written directly to disk until the event is terminated. When you terminate the event, the pre-buffer resets, accumulating received packets for up to the defined thirty-minute pre-buffer size.
The packets associated with an event can be queried, replayed, or analyzed using any type of recorder node query. Each triggered event is identified by a unique, user-supplied name, which can be used in the query to reference packets recorded in the pre-buffer prior to and during the event itself.
Using an Authentication Token
When using a DANZ Monitoring Fabric (DMF) Controller authentication token, the recorder node treats the DMF Controller as an ordinary client and requires it to present valid credentials either in the form of an HTTP basic username and password or an authentication token.
Static authentication tokens are pushed to each recorder node as an alternative form of authentication in headless mode, when the DMF Controller is unreachable, or by third-party applications that do not have or do not need Controller credentials.
Using the GUI to Add a Recorder Device
Configuring a Node to Use Local Storage
To configure a node to use local storage, use the following steps:
Configuring a Node to Use External Storage
Configuring a Recorder Node Interface
Using the GUI to Assign a Recorder Interface to a Policy
To forward traffic to a recorder node, include one or more recorder node interfaces as a delivery interface in a DANZ Monitoring Fabric (DMF) policy.
Using the GUI to Define a Recorder Query
- Window: Retrieves the timestamps of the oldest and most recent packets recorded on the recorder.
- Size: Provides the number of packets and their aggregate size in bytes that match the filter criteria specified.
- Application: Performs deep packet inspection to identify applications communicating with the packets recorded and that match the filter criteria specified.
- Packet-data: Retrieves all the packets that match the filter criteria specified.
- Packet-object: The packet object query extracts unencrypted HTTP objects from packets matching the given stenographer filter.
- HTTP, HTTP Request, and HTTP Stat: Analyzes HTTP packets, extracting request URLs, response codes, and statistics.
- DNS: Analyzes any DNS packets, extracting query and response meta data.
- Replay: Replays selected packets and transmits them to the specified delivery interface.
- IPv4: Identifies and dissects distinct IPv4 flows.
- IPv6: Identifies and dissects distinct IPv6 flows.
- TCP: Identifies and dissects distinct TCP flows.
- TCP Flow Health: Analyzes TCP flows for information such as maximum RTT, retransmissions, throughput, etc.
- UDP: Identifies and dissects distinct UDP flows.
- Hosts: Identifies all the unique hosts that match the filter criteria specified.
- RTP Stream: Characterizes the performance of Real Time Protocol streaming packets.
- Relative Time: A time range relative to the current time in which look for packets.
- Absolute Time: A specific time range in which to look for packets.
- Any IP: Include packets with the specified IP address in the IP header (either source or destination).
- Directional IP: Include packets with the specified source and/or destination IP address in the IP header.
- Src Port: Include packets with the specified protocol port number in the Src Port field in the IP header.
- Dst Port: Include packets with the specified protocol port number in the Dst Port field in the IP header.
- IP Protocol: Select the IP protocol from the selection list or specify the numeric identifier of the protocol.
- Community ID:Select packets with a specific BRO community ID string.
- Src Mac: Select packets with a specific source MAC address.
- Dst Mac: Select packets with a specific destination MAC address.
- VLAN: Select packets with a specific VLAN ID.
- Filter Interfaces: Click the provision (+) control and, in the dialog that appears, enable the checkbox for one or more filter interfaces to which the query should be restricted. To add interfaces to the dialog, click the provision (+) control on the dialog and select the interfaces from the list that is displayed.
- Policies: Click the provision (+) control and, in the dialog that appears, enable the checkbox for one or more policies to which the query should be restricted. To add policies to the dialog, click the provision (+) control on the dialog and select the policies from the list that is displayed.
- Max Bytes: This option is only available for packet queries. Specify the maximum number of bytes returned by a packet query in a PCAP file.
- Max Packets: This option is only available for packet queries. Specify the maximum number of packets returned by a packet query in a PCAP file.
- MetaWatch Device ID: Filter packets with the specified MetaWatch device ID.
- MetaWatch Port ID: Filter packets with the specified MetaWatch port ID.
Viewing Query History
You can view the queries that have been submitted to the recorder node using the GUI or CLI.
The Query History section displays the queries submitted to each recorder node and the status of the query.
To download the query results, select Download Results from the Menu control for a specific query. To export the query history, click the Export control at the top of the table (highlighted in the figure above, to the right of the Refresh control).
controller-1> show recorder-node query-history
# Packet Recorder Query Type StartDuration
---|---------------|-----------------------------------------------------------------------|------------------------|----------------------------|--------|
1 HW-PR-2 after 10m ago analysis-hosts 2019-03-20 09:52:38.021000 PDT 3428
2 HW-PR-1 after 10m ago analysis-hosts 2019-03-20 09:52:38.021000 PDT 3428
3 HW-PR-2 after 10m ago abort2019-03-20 09:52:40.439000 PDT 711
4 HW-PR-1 after 10m ago abort2019-03-20 09:52:40.439000 PDT 711
---------------------------------------------------------------------output truncated---------------------------------------------------------------------
Using the CLI to Manage the DMF Recorder Node
Basic Configuration
Authentication Token Configuration
Static authentication tokens are pushed to each recorder node as an alternative form of authentication in headless mode, when the DANZ Monitoring Fabric (DMF) Controller is unreachable, or by third-party applications that do not have or do not need DMF controller credentials in order to query the recorder node.
controller-1(config)# recorder-node auth token mytoken
Auth : mytoken
Token : some_secret_string <--- secret plaintext token displayed once here
controller-1 (config)# show running-config recorder-node auth token
! recorder-node
recorder-node auth token mytoken $2a$12$cwt4PvsPySXrmMLYA.Mnyus9DpQ/bydGWD4LEhNL6xhPpkKNLzqWS <---hashed token shows in running config
controller-1(config)# recorder-node auth generate-controller-token
Configuring the Pre-buffer
controller-1(config)# recorder-node device <name>
controller-1(config-recorder-node)# pre-buffer <minutes>
Replace name with the name of the recorder node. Replace minutes with the number of minutes to allocate to the pre-buffer.
Triggering a Recorder Node Event
To trigger an event for a specific recorder node, enter the following command from enable mode:
controller-1# trigger recorder-node <name> event <event-name>
Replace name with the name of the recorder node and replace event-name with the name to assign to the current event.
Terminating a Recorder Node Event
controller-1# terminate recorder-node <name> event <event-name>
Replace name with the name of the recorder node and replace event-name with the name of the recorder node event to terminate.
Viewing Recorder Node Events
controller-1# show recorder-node events
# Packet Recorder Time Event
-|---------------|------------------------------|-------------------------------------------------------------------|
1 pkt-rec-740 2018-02-06 16:21:37.289000 UTC Pre-buffer event my-event1 complete. Duration 3 minute(s)
2 pkt-rec-740 2018-02-06 20:23:59.758000 UTC Pre-buffer event event2 complete. Duration 73 minute(s)
3 pkt-rec-740 2018-02-07 22:39:15.036000 UTC Pre-buffer event event-02-7/event3 complete. Duration 183 minute(s)
4 pkt-rec-740 2018-02-07 22:40:15.856000 UTC Pre-buffer event event5 triggered
5 pkt-rec-740 2018-02-07 22:40:16.125000 UTC Pre-buffer event event4/event-02-7 complete. Duration 1 minute(s)
6 pkt-rec-740 2018-02-22 06:53:10.216000 UTC Pre-buffer event triggered
Using the CLI to Run Recorder Node Queries
Packet Replay
controller-1# replay recorder-node <name> to-delivery <interface> filter <stenographer-query>
[realtime | replay-rate <bps> ]
- name: Specify the recorder node for which you wish to replay the recorded packets from.
- interface: The name of the DMF delivery interface to which the packets should be delivered.
- stenographer-query: The filter used to look up desired packets.
- (Optional) real-time: Replay the packets at the original rate recorded by the specified recorder node. The absence of this parameter will result in a replay up to the line rate of the recorder node interface.
- (Optional) replay-rate bps: Specify the number of bits per second to be used for replaying the packets recorded by the specified recorder node. The absence of this parameter will result in a replay up to the line rate of the recorder node interface.
controller-1# replay recorder-node packet-rec-740 to-delivery eth26-del filter 'after 1m ago'
controller-1#
Replay policy details:
controller-1# show policy-flow | grep replay
1 __replay_131809296636625 packet-as5710-2 (00:00:70:72:cf:c7:cd:7d) 0 0 6400 1
in-port 47 apply: name=__replay_131809296636625 output: max-length=65535, port=26
Packet Data Query
You can use a packet query to search the packets recorded by a specific recorder node. The operation uses a Stenographer query string to filter only the interesting traffic. The query returns a URL that can be used to download and analyze the packets using Wireshark or other packet-analysis tools.
switch # query recorder-node <name> packet-data filter <stenographer-query>
- name: Identify the recorder instance.
- packet-data filter stenographer-query: Look up only the packets that match the specified Stenographer query.
Packet Object Query
switch# query recorder-node bmf-integrations-pr-1 packet-object filter 'after 5m ago'
switch# query recorder-node bmf-integrations-pr-1 packet-object filter 'after 1m ago'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Packet Object Query Results ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Coalesced URL : /pcap/__packet_recorder__/coalesced-bmf-2022-11-21-14-27-56-67a73ea9.tgz
Individual URL(s) : /pcap/__packet_recorder__/bmf-integrations-pr-1-2022-11-21-14-27-55-598f5ae7.tgz
Untar the folder to extract the HTTP objects.
Size Query
You can use a size query to analyze the number of packets and the total size of the packets recorded by a specific recorder node. The operation uses a Stenographer query string to filter only the interesting traffic.
# query recorder-node <name> size filter <stenographer_query>
- name: Identify the recorder node.
- size filter stenographer-query: Analyze only the packets that match the specified Stenographer query.
switch# query recorder-node <hq-bmf-packet-recorder-1> size filter "after 1m ago and src host 8.8.8.8"
~ Summary Query Results ~
# Packets : 66
Size: 7.64KB
~ Error(s) ~
None.
Window Query
You can use a window query to analyze the oldest available packet and most recent available packet recorded by a specific recorder node.
switch# query recorder-node <name> window
- name: Identify the recorder node.
switch# query recorder-node hq-bmf-packet-recorder-1 window
~~~~~~~~~~~~~ Window Query Results ~~~~~~~~~~~~~
Oldest Packet Available : 2020-07-30 05:01:08 PDT
Newest Packet Available : 2020-10-19 08:14:21 PDT
~ Error(s) ~
None.
Stopping a Query
controller-1# abort recorder-node <name> filter <string>
controller-1# abort recorder-node hq-bmf-packet-recorder-1 filter ""
Abort any request with the specified filter? This cannot be undone. enter "yes" (or "y") to
continue:
yes
Result : Success
~ Error(s) ~
None.
Using RBAC to Manage Access to the DMF Recorder Node
You can use Role-Based Access Control (RBAC) to manage access to the DANZ Monitoring Fabric (DMF) Recorder Node by associating a recorder node with an RBAC group.
To restrict access for a specific recorder to a specific RBAC group, use the CLI or GUI as described below.
RBAC Configuration Using the CLI
RBAC Configuration Using the GUI
Using the CLI to View Information About a Recorder Node
This section describes how to monitor and troubleshoot recorder node status and operation. The recorder node stores packets on the main hard disk and the indices on the SSD volumes.
Viewing the Recorder Node Interface
controller-1(config)# show topology recorder-node
# DMF IF Switch IFName State SpeedRate Limit
-|------------|----------|----------|-----|------|----------|
1 RecNode-Intf Arista7050 ethernet1up25Gbps -
Viewing Recorder Node Operation
controller-1# show recorder-node device packet-rec-740 interfaces stats
Packet Recorder Name Rx Pkts Rx BytesRx DropRx Errors Tx PktsTx Bytes Tx Drop Tx Errors
---------------|----|-------------|---------------|--------|---------|--------|----------|-------|---------|
packet-rec-740pri1 2640908588614 172081747460802 84204084 0 24630503 3053932660 0 0
Ctrl-2(config)# show policy PR-policy
Policy Name : PR-policy
Config Status : active - forward
Runtime Status : installed
Detailed Status : installed - installed to forward
Priority : 100
Overlap Priority : 0
# of switches with filter interfaces : 1
# of switches with delivery interfaces : 1
# of switches with service interfaces : 0
# of filter interfaces : 1
# of delivery interfaces : 1
# of core interfaces : 0
# of services : 0
# of pre service interfaces : 0
# of post service interfaces : 0
Push VLAN : 1
Post Match Filter Traffic : 1.51Gbps
Total Delivery Rate : 1.51Gbps
Total Pre Service Rate : -
Total Post Service Rate : -
Overlapping Policies : none
Component Policies : none
Installed Time : 2023-09-22 12:16:55 UTC
Installed Duration : 3 days, 4 hours
~ Match Rules ~
# Rule
-|-----------|
1 1 match any
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Filter Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF Switch IF Name State Dir Packets Bytes Pkt Rate Bit Rate Counter Reset Time
-|-----------|-------------------|---------|-----|---|-----------|--------------|--------|--------|------------------------------|
1 Lab-traffic Arista-7050SX3-T3X5 ethernet7 up rx 97831460642 51981008309480 382563 1.51Gbps 2023-09-22 12:16:55.738000 UTC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Delivery Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF Switch IF Name State Dir Packets Bytes Pkt Rate Bit Rate Counter Reset Time
-|---------------|-------------------|----------|-----|---|-----------|--------------|--------|--------|------------------------------|
1 PR-intf Arista-7050SX3-T3X5 ethernet35 up tx 97831460642 51981008309480 382563 1.51Gbps 2023-09-22 12:16:55.738000 UTC
~ Service Interface(s) ~
None.
~ Core Interface(s) ~
None.
~ Failed Path(s) ~
None.
Ctrl-2(config)#
Viewing Errors and Warnings
- show fabric errors
- show fabric warnings
- show recorder-node errors
- show recorder-node warnings
Type | Condition | Cause | Resolution |
---|---|---|---|
Error | Recorder Node (RN) management link down | RN has not received controller LLDP | Wait 30s if the recorder node is newly configured. Verify it is not connected to a switch port that is a DANZ Monitoring Fabric (DMF) interface. |
Error | RN fabric link down | Controller has not received RN LLDP | Wait 30s if recorder node is newly configured. Check it is online otherwise. |
Warning | Disk/RAID health degraded | Possible hardware degradation | Investigate specific warning reported. Could be temperature issue. Possibly replace indicated disk soon. |
Warning | Low disk space | Packet or index disk space has risen above threshold | Prepare for disk full soon |
Warning | Disk full | Packet or index disk space is full. Packets are being dropped or rotated depending on removal policy. | Do nothing if removal policy is rolling-FIFO. Consider erasing packets to free up space otherwise. |
Warning | Recorder misconfiguration on a DMF interface | A recorder node has been detected in the fabric on a switch interface that is configured as a filter or delivery interface. | Remove the conflicting interface configuration, or re-cable the recorder node to a switch interface not defined as a filter or delivery interface. |
Using the GUI to view Recorder Node Statistics
Recorder node statistics can be viewed by clicking on the recorder node alias from the
page.The recorder node shows health statistics for the following:
CPU: CPU health displays the compute resource utilization of the recorder node.
Memory: Memory related stats are displayed, such as total memory, used, free, available, etc.
Virtual Disks: Virtual Disks health stats displays the Index and Packet virtual disks size, state, health and RAID level configuration.
- File Descriptors (current): Current number of files open in the entire system.
- Max System File Descriptors: Highest number of open files allowed on the entire system.
- Max Stenographer File Descriptors: Highest number of open files allowed for Stenographer application.
Stenographer: Stenographer Statistics are displayed as follows:
- Initialized: Displays the Stenographer application running state. A green check mark indicates that the application was initialized successfully. When the Stenographer application is starting up, a red x mark is expected. During this time, recording and querying is disallowed.
- Tracked Files: Tracked files are the total number of files stored under each CPU instance thread.
- Cached Files: Cached files are the number of files that are open and have a file descriptor.
- Max Cached Files: Maximum cached files is the total number of files that are allowed to be open.
Changing the Recorder Node Default Configuration
controller-1(config)# recorder-node device <instance>
Replace instance with the alias you want to use for the recorder node. This alias is associated with the MAC hardware address, using the mac command.
- banner: Set the recorder node pre-login banner message
- mac: Configure the MAC address for the recorder node
- ntp: Configure recorder node to override default timezone and NTP parameters.
- snmp-server: Configure recorder node SNMP parameters and traps.
- logging: Enable recorder node logging to Controller.
- tacacs: Set TACACS defaults, server IP address(es), timeouts and keys.
- ntp override-global: Override global time configuration with recorder node time configuration.
- snmp-server override-global: Override global SNMP configuration with recorder node SNMP configuration.
- snmp-server trap override-global: Override global SNMP trap configuration with recorder node SNMP trap configuration.
- logging override-global: Override global logging configuration with packet recorder logging configuration.
- tacacs override-global: Override global TACACS configuration with recorder node TACACS configuration.
- ntp merge-global: Merge global time configuration with recorder node time configuration.
- snmp-server merge-global: Merge global SNMP configuration with recorder node SNMP configuration.
- snmp-server trap merge-global: Merge global SNMP trap configuration with recorder node SNMP trap configuration.
- logging merge-global: Merge global logging configuration with recorder node logging configuration.
TACACS configuration does not have a merge option. It can either be inherited completely from the DMF Controller or overridden to use only the recorder node specific configuration.
Large PCAP Queries
To run large PCAP queries to the recorder node, access the recorder node via a web browser. This allows you to run packet queries directly to the recorder node without specifying the maximum byte or packet limit for the PCAP file (which is required if the query is executed from the DANZ Monitoring Fabric (DMF) Controller).
- Recorder Node IP Address: Enter the IP address of the target recorder node.
- DMF Controller Username: Provide the DMF Controller username.
- DMF Controller Password: Provide the password for authentication.
- Stenographer Query Filter: The query filter can be used to filter the query results to look for specific packets. For example, to search for packets with a source IP address of 10.0.0.145 in the last 10 minutes, use the following filter:
after 10m ago and src host 10.0.0.145
- Stenographer Query ID: Starting in DMF 8.0, a Universally Unique Identifier (UUID) is required to run queries. To generate a UUID, run the following command on any Linux machine and use the result as the Stenographer query ID:
$ uuidgen b01308db-65f2-4d7c-b884-bb908d111400
- Save pcap as: Provide the file name to be used for this PCAP query result.
- Submit Request: Click on Submit Request. This will send a query to the specified recorder node, and it will save the PCAP file with the provided file name to the default download location for the browser.
Recorder Node Management Migration L3ZTN
pre-configure
.To migrate management to a new Controller, follow the steps below:
Recorder Node CLI
The following commands are available from the recorder node:
RecNode(config)# show version
Controller Version : DMF Recorder Node 8.1.0 (bigswitch/enable/dmf-8.1.x #5)
RecNode(config)#
RecNode(config)# show controllers
controllerRole State Aux
---------------------|------|---------|---|
tcp://10.106.8.2:6653 master connected 0
tcp://10.106.8.3:6653 slaveconnected 0
tcp://10.106.8.3:6653 slaveconnected 1
tcp://10.106.8.3:6653 slaveconnected 2
tcp://10.106.8.2:6653 master connected 1
tcp://10.106.8.2:6653 master connected 2
RecNode(config)#
Multiple Queries
The GUI can be used to run multiple recorder node queries
To run queries on recorded packets by the recorder node, navigate to the
page.Ability to Deduplicate Packets - Query from Recorder Node
For Recorder Node queries, the recorded packets matching a specified query filter may contain duplicates when packet recording occurs at several different TAPs within the same network; i.e., as a packet moves through the network, it may be recorded multiple times. The dedup feature removes duplicate packets from the query results. By eliminating redundant information, packet deduplication improves query results' clarity, accuracy, and conciseness. Additionally, the dedup feature significantly reduces the size of query results obtained from packet query types.
Using the CLI to Deduplicate Packets
In the DANZ Monitoring Fabric (DMF) Controller CLI, packet deduplication is available for the packet data, packet object, size, and replay query types. Deduplication is turned off by default for these queries. To enable deduplication, “dedup” must be added to the end of the query command after all optional values have been selected (if any).
The following are command examples of enabling deduplication.
Enabling deduplication for a size query:
controller# query recorder-node rn size filter “before 5s ago” dedup
Enabling deduplication for a packet data query specifying a limit for the size of the PCAP file returned in bytes:
controller# query recorder-node rn packet-data filter “before 5s ago” limit-bytes 2000 dedup
Enabling deduplication for a replay query:
controller# replay recorder-node rn to-delivery dintf filter “before 5s ago” dedup
Enabling deduplication for a replay query specifying the replay rate:
controller# replay recorder-node rn to-delivery dintf filter “before 5s ago” replay-rate 100 dedup
A time window (in milliseconds) can also be specified for deduplication. The time window defines the time required between timestamps of identical packets to no longer be considered duplicates of each other. For example, for a time window of 200 ms, two identical packets with timestamps that are 200 ms (or less) apart are duplicates of each other. In contrast, if the two identical packets had timestamps more than 200 ms apart, they would not be duplicates of each other.
The time window must be an integer between 0 and 999 (inclusive) with a default time window of 200 ms when deduplication is enabled and no set time window value.
To configure a time window value, dedup-window must be added after dedup and followed by an integer value for the time window.
controller# query recorder-node rn size filter “before 5s ago” dedup dedup-window 150
Using the GUI to Deduplicate Packets
- Set the toggle switch deduplication to Yes in the query submission window.
- Specify an optional time window (in milliseconds) as required by entering an integer between 0 and 999 (inclusive) into the Deduplication Time Window field. The time window will default to 200 ms if no time window value is set.
- Click Submit to continue.
Limitations
Expect a query with packet deduplication enabled to take longer to complete than with packet deduplication disabled. Hence, packet deduplication, by default, is disabled.
The maximum time window value permitted is 999 ms to ensure that TCP retransmissions are not regarded as duplicates, assuming that the receive timeout value for TCP retransmissions (of any kind) is at least 1 second. If the receive timeout value is less than 1 second (particularly, exactly 999 ms or less), then it is possible for TCP retransmissions to be regarded as duplicates when the time window value used is larger than the receive timeout value.
Due to memory constraints, removing some duplicates may not occur as expected. This scenario is likely to occur if a substantial amount of packets match the query filter, which all have timestamps within the specified time window from each other. We refer to this scenario as the query having exceeded the packet window capacity. To mitigate this from occurring, decrease the time window value or use a more specific query filter to reduce the number of packets matching the query filter at a given time.
Enabling Egress sFlow on Recorder Node Interfaces
The egress sFlow feature is available from the DANZ Monitoring Fabric (DMF) 8.5 release onward. Enable egress sFlow to send sampled packets collected on the Recorder Node switch attachment point to a configured sFlow collector. Examining these sampled packets enables the identification of post-match-rule flows recorded by the DMF Recorder Nodes (RNs) without performing a query against the RNs. While not explicitly required, Arista Networks highly recommends using the DMF Analytics Node (AN) as the configured sFlow collector because it can automatically identify packets sampled utilizing this feature.
Use the CLI or GUI to enable or disable the feature.
Using the CLI to Enable Egress sFlow
The egress sFlow feature requires a configured sFlow collector. After configuring the sFlow collector, enter the following command from the config mode to enable the feature:
Controller-1(config)# recorder-node sflow
To disable the feature, enter the command:
Controller-1(config)# no recorder-node sflow
Using the GUI to Enable Egress sFlow
Using the GUI
After configuring the fabric for sFlow and setting up the sFlow collector, navigate to the Global Configuration section, click the Configure global settings button.
page. Under theIn the Configure Global Settings pop-up window, enable the sFlow setting and click Submit.
Analytics Node
When using a DMF Analytics Node as the sFlow collector, it has a dashboard to display the results from this feature. To access the results:
- Navigate to the sFlow dashboard from the Fabric dashboard.
- Select the disabled RN Flows filter.
- Select the option to Re-enable the filter, as shown below.
Troubleshooting Egress SFlow Configurations
Switches not associated with a sFlow collector (either a global sFlow collector or a switch-specific sFlow collector) do not have an active feature even if the feature is enabled. Ensure the fabric has been configured for sFlow and a sFlow collector has been configured. To verify that a configured global sFlow collector exists, use the command:
Controller-1# show sflow default
A configured collector appears as an entry in the table under the column labeled collector. Alternatively, to verify a configured collector exists for a given switch, use the command:
Controller-1#show switch <switch-name> table sflow-collector
This command displays a table with one entry per configured collector.
A feature-unsupported-on-device warning appears when connecting an EOS switch to an RN. The feature do not sample packets passing to an RN from an EOS switch. View any such warnings using the GUI or using the following CLI command:
Controller-1#show fabric warnings feature-unsupported-on-device
To verify the feature is active on a given switch, use the command:
Controller-1#show switch <switch-name> table sflow-sample
If the feature is enabled, the entry values associated with the ports connected to an RN would include an EgressSamplingRate(number)
with a number
greater than 0
. The following example shows Port(1)
on <switch-name>
connecting to an RN.
Controller-1# show switch <switch-name> table sflow-sample
#Sflow-sample Device nameEntry key Entry value
--|------------|---------------|---------|----------------------------------------------------------------------------------|
5352 <switch-name>Port(1) SamplingRate(0), EgressSamplingRate(10000), HeaderSize(128), Interval(10000)
Guidelines and Limitations for Enabling Egress sFlow
Below are the guidelines and limitations to consider while enabling Egress sFlow :
- The Egress sFlow support for the Recorder Nodes (RN) feature requires a configured sFlow collector in a fabric configured to allow sFlows.
- If a packet enters a switch through a filter interface with sFlow enabled and exits through a port connected to an RN while the feature is enabled, only one sFlow packet (i.e. the ingress sFlow packet) is sent to the collector.
- The Egress sFlow feature does not identify which RN has recorded a given packet in a fabric when there are multiple RNs. This is fine in a normal case as the queries are issued to the RNs in aggregate rather than to individual RNs and hence the information that any RN has received a packet is sufficient. In some cases, it may be possible to make that determination from the outport of the sFlow packet, but that information may not be available in all cases. This is an inherent limitation of egress sFlow.
- In DMF 8.5.0, an enabled egress sFlow feature captures the packets sent to the RN regardless of whether the RN is actively recording or not.