controller-1# show managed-service-device <SN-Name> interfaces
controller-1# show managed-service-device <SN-Name> stats

For example, the following command shows the managed services handled by the Service Node Interface (SNI):

The Load column shows no, low, moderate, high, and critical health indicators. These health indicators are represented by green, yellow, and red under DANZ Monitoring Fabric > Managed Services > Devices > Service Stats. They reflect the processor load on the service node interface at that instant but do not show the bandwidth of the respective data port (SNI) handling traffic, as shown in the following sample snapshot of the Service Stats output.

The IPXIF template consists of the key element IDs representing IP flow, field element IDs representing actions the exporter has to perform over IP flows matching key element IDs, the template ID number for uniqueness, collector information, and eviction timers.

To define a template, configure keys of interest representing the IP flow and fields that identify the values measured by the exporter, the exporter information, and the eviction timers. To define the template, select the Monitoring > Managed Service > IPFIX Template option from the DANZ Monitoring Fabric (DMF) GUI or enter the ipfix-template template-name command in config mode, replacing template-name with a unique identifier for the template instance.

A field defines each value updated for the packets the generator receives that match the specified keys. For example, include fields in the template to record the number of packets, the largest and smallest packet sizes, or the start and end times of the flows. To see a listing of the fields supported in the current release of the DANZ Monitoring Fabric (DMF) Service Node, select the Monitoring > Managed Service > IPFIX Template option from the DMF GUI, or type help in config-ipxif-template submode. The following are the fields supported:

• flow-end-milliseconds
• flow-end-reason
• flow-end-seconds
• flow-start-milliseconds
• flow-start-seconds
• maximum-ip-total-length
• maximum-layer2-total-length
• maximum-ttl
• minimum-ip-total-length
• minimum-layer2-total-length
• minimum-ttl
• octet-delta-count
• packet-delta-count
• tcp-control-bits

After the number of minutes specified by the active timer, the flow set is closed and forwarded to the IPFIX collector. The default active timer is one minute. During the number of seconds set by the inactive timer, if no packets that match the flow definition are received, the flow set is closed and forwarded without waiting for the active timer to expire. The default value for the inactive time is 15 seconds.

The following is a Wireshark view of an IPFIX flowset.

The following is a running-config that shows the IPFIX template used to generate this flowset.

! ipfix-template
ipfix-template Perf-temp
template-id 22222
key destination-ipv4-address
key destination-transport-port
key dot1q-vlan-id
key source-ipv4-address
key source-transport-port
field flow-end-milliseconds
field flow-end-reason
field flow-start-milliseconds
field maximum-ttl
field minimum-ttl
field packet-delta-count

Select IPFIX from the Action selection list on the Create Managed Service > Action page.

After completing the configuration, click Next, and then click Save.

controller(config)# managed-service MS-IPFIX-SERVICE
controller(config-managed-srv)# 1 ipfix TO-DELIVERY-INTERFACE
controller(config-managed-srv-ipfix)# collector 10.106.1.60
controller(config-managed-srv-ipfix)# template IPFIX-TEMPLATE

The active-timeout and inactive-timeout commands are optional

controller1# show running-config managed-service MS-IPFIX-ACTIVE
! managed-service
managed-service MS-IPFIX-ACTIVE
service-interface switch CORE-SWITCH-1 ethernet13/1
!
1 ipfix TO-DELIVERY-INTERFACE
collector 10.106.1.60
template IPFIX-TEMPLATE

config# show running-config ipfix-template
! ipfix-template
ipfix-template IPFIX-IP
template-id 1974
key destination-ipv4-address
key destination-ipv6-address
key ethernet-type
key source-ipv4-address
key source-ipv6-address
field flow-end-milliseconds
field flow-end-reason
field flow-start-milliseconds
field minimum-ttl
field tcp-control-bits
------------------------output truncated------------------------

Destination MAC rewrite for the records-per-interface NetFlow and IPFIX feature is the default setting and applies to switches running Extensible Operating System (EOS) and SWL and is supported on all platforms.

A configuration option exists for using src-mac when overwriting the dst-mac isn't preferred.

c1(config)# filter-mac-rewrite rewrite-src-mac
c1(config)# filter-mac-rewrite rewrite-dst-mac

c1(config)# managed-service ms1
c1(config-managed-srv)# 1 netflow
c1(config-managed-srv-netflow)# collector 213.1.1.20 udp-port 2055 mtu 1024 records-per-interface

c1(config)# ipfix-template i1
c1(config-ipfix-template)# field maximum-ttl 
c1(config-ipfix-template)# key records-per-dmf-interface
c1(config-ipfix-template)# template-id 300

c1(config)# managed-service ms1
c1(config-managed-srv)# 1 ipfix
c1(config-managed-srv-ipfix)# template i1

c1(config)# show running-config managed-service 
! managed-service
managed-service ms1
!
1 netflow
collector 213.1.1.20 udp-port 2055 mtu 1024 records-per-interface

c1(config)# show ipfix-template i1
~~~~~~~~~~~~~~~~~~ Ipfix-templates~~~~~~~~~~~~~~~~~~
# Template Name KeysFields
-|-------------|-------------------------|-----------|
1 i1records-per-dmf-interface maximum-ttl

c1(config)# show running-config managed-service 
! managed-service
managed-service ms1
!
1 ipfix
template i1

From the Arista Analytics Node dashboard, apply filter rules to display specific flow information.

The show switch all property command provides upper and lower bounds of packet slicing action’s offset. If bounds are present, the feature is supported; otherwise, the switch does not support the packet slicing feature.

The show fabric errors managed-service-error command provides information when DANZ Monitoring Fabric (DMF) fails to install a configured packet slicing managed service on a switch.

Use the following commands to troubleshoot packet-slicing issues.

Controller# show fabric errors managed-service-error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Managed Service related error~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Error Service Name
-|---------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
1 Pre-service interface 7280-Ethernet10/1-to-managed-service on switch 7280 is inactive; Service interface Ethernet10/1 on switch 7280 is downpacket-slicing-7280
2 Post-service interface 7280-Ethernet10/1-to-managed-service on switch 7280 is inactive; Service interface Ethernet10/1 on switch 7280 is down packet-slicing-7280

The show switch switch name interface interface name dmf-stats command provides Rx and Tx rate information for the managed service interface.

Controller# show switch 7280 interface Ethernet10/1 dmf-stats
# Switch DPID Name State Rx Rate Pkt Rate Peak Rate Peak Pkt Rate TX Rate Pkt Rate Peak Rate Peak Pkt Rate Pkt Drop Rate
-|-----------|------------|-----|-------|--------|---------|-------------|-------|--------|---------|-------------|-------------|
1 7280Ethernet10/1 down- 0128bps0 - 0128bps0 0

The show switch switch name interface interface name stats command provides Rx and Tx counter information for the managed service interface.

Controller# show switch 7280 interface Ethernet10/1 stats
# Name Rx Pkts Rx Bytes Rx Drop Tx Pkts Tx Bytes Tx Drop
-|------------|-------|--------|-------|-------|--------|-------|
1 Ethernet10/1 22843477 0 5140845937 0

There are no syslog messages associated with this feature.

The show switch all property command provides the Strip Header Supported property of the switch. If the value BSN_STRIP_HEADER_CAPS_VXLAN is present, the feature is supported; otherwise, the switch does not support this feature.

The show fabric warnings feature-unsupported-on-device command provides information when DMF fails to enable strip-vxlan on an unsupported switch.

The show switch switch-name table strip-vxlan-header command provides the gentable details.

The following are examples of several failure cases:

1. The filter interface is down.
2. The interface with strip-vxlan is neither a filter interface nor a filter-and-delivery interface.
3. The switch does not support strip-vxlan.
4. Tunneling / UDF is enabled simultaneously with strip-vxlan.
5. Unsupported pipeline mode with strip-vxlan enabled (strip-vxlan requires a specific pipeline mode strip-vxlan-match-push-vlan).

• When configured for a supported switch, the filter interface for decap-vxlan action can only be a physical interface or a LAG.
• It is not possible to enable strip-vxlan simultaneously with tunneling / UDF.
• When enabling strip-vxlan on one or more switch interfaces on the same switch, other filter interfaces on the same switch cannot be matched on the VXLAN header.

Configure the feature through the Controller in managed services.

Follow the configuration steps described in the services earlier to configure app-id-filter and app-id together. However, in this case, app-id should use a higher seq num than app-id-filter. Thus, the traffic is processed through the app-id-filter policy first, then through app-id.

! managed-service
managed-service MS1
service-interface switch CORE-SWITCH-1 ethernet2
!
!
1 app-id-filter
app facebook
filter-mode forward
!
2 app-id
collector 1.1.1.1
l3-delivery-interface L3-INTF-1

Syslog messages for configuring the app-id and app-id-filter services appear in a service node’s syslog through journalctl.

A Service Node syslog registers events for the app-id add, modify, and delete actions.

These events contain the keywords dpi and dpi-filter, which correspond to app-id and app-id-filter.

For example:

Adding dpi for port, 
Modifying dpi for port, 
Deleting dpi for port,
Adding dpi filter for port, 
Modifying dpi filter for port, 
Deleting dpi filter for port, 
App appname does not exist - An invalid app name was entered.

The addition, modification, or deletion of app names in an app-id-filter managed-service in the Controller node’s CLI influences the policy refresh activity, and these events register in floodlight.log.

• Max concurrent sessions are currently set to permit less than 200,000 active flows per core. Performance may drop the more concurrent flows there are. This value is a maximum value to prevent the service from overloading. Surpassing this threshold may cause some flows not to be processed, and the new flows will not be identified or filtered. Entries for inactive flows will time out after a few minutes for ongoing sessions and a few seconds after the session ends.
• If there are many inactive sessions, DMF holds the flow contexts, reducing the number of available flows used for DPI. The timeouts are approximately 7 minutes for TCP sessions and 1 minute for UDP.
• Heavy application traffic load degrades performance.

• When using a drop filter, a few packets may slip through the filter before determining an application ID for a flow, and when using a forward filter, a few packets may not be forwarded. Such a small amount is estimated to be between 1 and 6 packets at the beginning of a flow.
• When using a drop filter, add the unknown app ID to the filter list to drop any unidentified traffic if these packets are unwanted.
• The Controller must be connected to a Service Node for the app-id-filter app list to appear. If the list does not appear and the application names are unknown, use the app-id to send reports to the analytics node. Use the application names seen there to configure an app-id-filter. The name must match exactly.
• Since app-category does not currently show the applications included in that category, do not use it when targeting specific apps. Categories like basic, which include all basic networking protocols like TCP and UDP, may affect all flows.
• For app-id, a report is only generated for a fully classified flow after that flow has been fully classified. Therefore, the number of reported applications may not match the total number of flows. These reports are sent after enough applications are identified on the Service Node. If many applications are identified, DMF sends the reports quickly. However, DMF sends these reports every 10 seconds when identifying only a few applications.
• DMF treats a bidirectional flow as part of the same n-tuple. As such, generated reports contain the client's source IP address and the server's destination IP address.
• While configuring many ports with the app-id, there may occasionally be a few Rx drops on the 16 port machines at a high traffic rate in the first couple of seconds.
• The feature uses a cache that maps dest ip and port to the application. Caching may vary the performance depending on the traffic profile.
• The app-id and app-id-filter services are more resource-intensive than other services. Combining them in a service chain or configuring many instances of them may lead to degradation in performance.
• At scale, such as configuring 16 ports on the R740 DCA-DM-SEL, app-id may take a few minutes to set up on all these ports, and this is also true when doing a dynamic signature update.
• The show analytics app-info command only works in push-per-filter VLAN mode.

Troubleshooting

• If the number of packets forwarded by the Service Node interfaces is few, the max-tokens and tokens-per-refresh values likely need to be higher.
• If fewer packets than the tokens-per-refresh value forward, ensure the max-tokens value is larger than the tokens-per-refresh value. The system discards any surplus refresh tokens above the max-tokens value.
• When all traffic forwards, the initial max-tokens value is too large, or the tokens refreshed by tokens-per-refresh are higher than the packet rate.
• When experiencing packet drops after the first 10ms post commencement of traffic, it may be due to a low tokens-per-refresh value. For example, calculate the minimum value of max-tokens and tokens-per-refresh that would lead to forwarding all packets.

Calculation Example

Traffic Rate : 400 Mbps
Packet Size - 64 bytes
400 Mbps = 400000000 bps
400000000 bps = 50000000 Bps
50000000 Bps = 595238 pps (Includes 20 bytes of inter packet gap in addition to the 64 bytes)
1000 ms = 595238 pps
1 ms = 595.238 pps
10 ms = 5952 pps
max-tokens : 5952 (the minimum value)
tokens-per-refresh : 5952 ( the minimum value)

dmf-controller-1(config-managed-srv-flow-diff)# tap-point-pair source <Tab>
filter-interface policy-namefilter-interface-group

Both source and destination tap points must use identifiers compatible; policy-name cannot be used with filter-interface or filter-interface-group.

The filter-interface-group option takes in any configured filter interface group used to represent a collection of tap points in push-per-filter mode. This is an optional command to use when a group of tap points exists, all expecting traffic from the same source or group of source tap points for ease of configuration. For example:

1. Instead of having two separate tap-point-pairs to represent A → B, A → C, use a filter-interface-group G = [B, C], and only one tap-point-pair A → G.

   dmf-controller-1(config-managed-srv-flow-diff# tap-point-pair source type A destination filter-interface-group G

2. With a topology like A → C and B → C, configure a filter-interface-group G = [A, B], and tap-point-pair G → C.

   dmf-controller-1(config-managed-srv-flow-diff # tap-point-pair source filter-interface-group G destination type C

There are some restrictions to keep in mind while configuring tap-point-pairs:

• source and destination must exist and cannot refer to the same tap point
• You can only configure a maximum of 1024 tap points, therefore a maximum of 512 tap-point-pairs. These limits are not for a single flow-diff managed service but across all managed services with flow-diff action.
• filter-interface-group must not overlap with other groups within the same managed service and cannot have more than 128 members.
• When configuring multiple tap-point-pairs using filter-interface and filter-interface-group, an interface part of a filter interface group cannot be used simultaneously as an individual source and destination within the same managed service.

There are several caveats on what is accepted based on the VLAN mode:

Push per Filter

• Use the filter-interface option to provide the filter interface name.
• Use the filter-interface-group option to provide the filter interface group name.
• The policy-name identifier is invalid in this mode.

Push per Policy

• Accepts a policy-name identifier as a tap point identifier.
• The filter-interface and filter-interface-group identifiers are invalid in this mode.
• Policies configured as source and destination tap points within a tap-point-pair must not overlap.

Irrespective of the VLAN mode, configure a policy or policies so that the same packet can be tapped from two independent points in the network and then sent to the Service Node.

After creating a policy, add the managed service with flow-diff action as shown below:

dmf-controller-1 (config-policy)# use-managed-service service name sequence 1

There are several things to consider while configuring policies depending on the VLAN mode:

Push per Filter

• Only one policy can contain the flow-diff service action.
• A policy should have all filter-interfaces and filter-interface-groups configured as tap points in the flow-diff configuration. Missing filter interfaces and groups from policy may result in drops being reported when in reality we simply won’t be forwarding the packets from one end of the tap-point-pair to the Service Node.
• It’s also advisable to not add any filter interfaces (groups) that are not in the tap-point-pairs as their latency and drop analysis will not be done, so it will cause unnecessary packets forwarded to the Service Node, which will be reported as unexpected.

Push per Policy

• Add the flow-diff service to two policies when using policy-name as source and destination identifiers. In this case, there are no restrictions on how many filter interfaces a policy can have.
• A policy configured as one of the tap points will fail if it overlaps with the other policy in a tap-point-pair, or if the other policy does not exist.

dmf-controller-1 (config-policy)# use-timestamping

The following show commands provide helpful information.

dmf-controller-1(config)# show running-config managed-service flow-diff 
! managed-service
managed-service flow-diff
service-interface switch DCS-7050CX3-32S ethernet2/4
!
1 flow-diff
collector 192.168.1.1
l3-delivery-interface AN-Data
tap-point-pair source filter-interface ingress destination filter-interface egress

dmf-controller-1(config)# show managed-services flow-diff 
# Service Name SwitchSwitch Interface Installed Max Post-Service BW Max Pre-Service BW Total Post-Service BW Total Pre-Service BW 
-|------------|---------------|----------------|---------|-------------------|------------------|---------------------|--------------------|
1 flow-diffDCS-7050CX3-32S ethernet2/4True25Gbps25Gbps 624Kbps 432Mbps

The show running-config policy policy command checks whether the policy flow-diff service exists, whether use-timestamping is enabled, and the use of the correct filter interfaces.

The show policy policy command provides detailed information about a policy and whether any errors are related to the flow-diff service. The Service Interfaces tab section shows the packets transmitted to the Service Node and IPFIX packets received from the Service Node.

dmf-controller-1 (config)# show policy flow-diff-1
Policy Name: flow-diff-1
Config Status: active - forward
Runtime Status : installed
Detailed Status: installed - installed to forward
Priority : 100
Overlap Priority : 0
# of switches with filter interfaces : 1
# of switches with delivery interfaces : 1
# of switches with service interfaces: 1
# of filter interfaces : 1
# of delivery interfaces : 1
# of core interfaces : 4
# of services: 1
# of pre service interfaces: 1
# of post service interfaces : 1
Push VLAN: 2
Post Match Filter Traffic: 215Mbps
Total Delivery Rate: -
Total Pre Service Rate : 217Mbps
Total Post Service Rate: -
Overlapping Policies : none
Component Policies : none
Runtime Service Names: flow-diff
Installed Time : 2023-11-16 18:15:27 PST
Installed Duration : 19 minutes, 45 secs
~ Match Rules ~
# Rule
-|-----------|
1 1 match any
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Filter Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IF Switch IF NameState Dir PacketsBytes Pkt Rate Bit Rate Counter Reset Time 
-|------|--------|----------|-----|---|--------|-----------|--------|--------|------------------------------|
1 BP17280SR3E Ethernet25 uprx24319476 27484991953 23313215Mbps2023-11-16 18:18:18.837000 PST
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Delivery Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# DMF IFSwitchIF NameState Dir Packets BytesPkt Rate Bit Rate Counter Reset Time 
-|-------|---------|----------|-----|---|-------|------|--------|--------|------------------------------|
1 AN-Data 7050SX3-1 ethernet41 uptx81117222 0-2023-11-16 18:18:18.837000 PST
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Service Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Service name Role SwitchIF Name State Dir PacketsBytes Pkt Rate Bit Rate Counter Reset Time 
-|------------|------------|---------------|-----------|-----|---|--------|-----------|--------|--------|------------------------------|
1 flow-diffpre-serviceDCS-7050CX3-32S ethernet2/4 uptx23950846 27175761734 23418217Mbps2023-11-16 18:18:18.837000 PST
2 flow-diffpost-service DCS-7050CX3-32S ethernet2/4 uprx81 1175460-2023-11-16 18:18:18.837000 PST
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Core Interface(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# SwitchIF NameState Dir PacketsBytes Pkt Rate Bit Rate Counter Reset Time 
-|---------------|----------|-----|---|--------|-----------|--------|--------|------------------------------|
1 7050SX3-1 ethernet7uprx23950773 27175675524 23415217Mbps2023-11-16 18:18:18.837000 PST
2 7050SX3-1 ethernet56 uprx81 1172220-2023-11-16 18:18:18.837000 PST
3 7050SX3-1 ethernet56 uptx23950773 27175675524 23415217Mbps2023-11-16 18:18:18.837000 PST
4 7280SR3EEthernet7uptx24319476 27484991953 23313215Mbps2023-11-16 18:18:18.837000 PST
5 DCS-7050CX3-32S ethernet28 uptx81 1175460-2023-11-16 18:18:18.837000 PST
6 DCS-7050CX3-32S ethernet28 uprx23950846 27175761734 23418217Mbps2023-11-16 18:18:18.837000 PST
~ Failed Path(s) ~
None.

The Flow Diff Latency and Drop Analysis feature does not create Syslog messages.

• Only 512 tap-point-pairs and 1024 distinct tap points are allowed.
• filter-interface-group used as a tap point must not overlap with any other group within the same managed service and must not have more than 128 members.
• A filter interface cannot be used individually as a tap point and as a part of a filter-interface-group simultaneously within the same managed service.
• There is no chaining if a packet flows through three or more tap points in an A->B->C->...->Z topology. The only computed latency reports will be for tap-point-pairs A->B, A->C, …, and A->Z if these links are specified, but B->C, C->D, etc., will not be computed.
• Hardware RSS firmware in the Service Node currently cannot parse L2 header timestamps, so packets for all L2 header timestamps are sent to the same lcore; however, RSS does distribute packets correctly to multiple lcores when using src-mac timestamping.
• PTP timestamping doesn’t allow rewrite-dst-mac, so filter interfaces cannot be used as tap points in push-per-policy mode.
• In push-per-policy, a policy can only have one filter interface.
• Each packet from the L3 header and onwards gets hashed to a 64-bit value; if two packets hash to the same value, we assume the underlying packets are the same.
• Currently, on the flow-diff action in the Service Node, if packets are duplicated so that N copies of the same packet are received:
  • N-1 latencies are computed.
  • The ingress identifier is the earliest timestamp.
• The system reports timestamps as unsigned 32-bit values, with the maximum timestamp being 2^32-1, corresponding to approximately 4.29 seconds.
• Only min/mean/max latencies are currently reported.
• Packets are hashed from the L3 header onwards, meaning if there is any corrupted data past the L3 header, it will lead to drop reports. The same packet must appear at two tap points to generate a computed latency report.
• In A->B, if B packets appear before A, an unexpected type report is generated.
• At whichever tap point the packet first appears with the earliest timestamp, it is considered the source.
• While switching the latency configuration, the system may generate a couple of unexpected or drop reports at the beginning.
• Users must have good knowledge of network topology when setting up tap points and configuring timeout or sample threshold values. Improper configuration may lead to drop or unexpected reports.