NG Firewall Filter Apps

This section discusses the following topics:

Ad Blocker

Ad Blocker allows you to block most advertising content delivered to users on web pages that they request.

Ad Blocker uses downloadable filter subscriptions from various sources, containing lists of websites and extensions typically used to deliver advertising.

Warning: Ad Blocker is similar to browser plugins that block ads (like AdBlockPlus Ghostery). Like those plugins, blocking ads and tracking can sometimes interfere with the proper functioning of online websites or media. However, Ad Blocker blocks the gateway, and the browser user cannot see that something has been blocked. It does not have any way to temporarily disable or bypass blocking if it interferes with the proper functioning of online services. As such, Ad Blocker can cause issues, and the browser user has no recourse. Running Ad Blocker can impose administrative overhead dealing with issues and is ideal for some environments (small sites, homes, enthusiasts, etc.) but is not recommended for others like schools, businesses, and large sites.
Warning: With the increasing adoption of SSL, ad blockers can do very little without SSL inspection, but running SSL inspection is not ideal for many organizations.
Warning: When the update button is pressed, it pulls new signatures directly from third-party sources. We have not tested any future updates and can not guarantee they will work correctly. Updates may interfere with the proper functioning of websites and potentially cause massive problems.
Warning: Unlike other apps, Ad Blocker is off by default after installation. Only enable Ad Blocker once you read the above warnings.

Settings

This section discusses the different settings and configuration options available for Ad Blocker.

Status

This displays the current status and some statistics.

Options

If Block Ads is enabled, then Ad Blocker will block web requests that it determines are for advertisements. It will return an HTTP 403 permission denied to the client that requests the ad.

If Block Tracking & Ad Cookies is enabled, Ad Blocker will block cookies used for advertising or behavior tracking purposes.

The Update button will update the ad filter signatures.

Warning: When the update button is pressed, it pulls new signatures directly from 3rd party sources. We have not tested any future updates and can not guarantee any future updates will work correctly. Updates may interfere with the proper functioning of websites and potentially cause massive problems.

Ad Filters

Ad Blocker's Standard Filters list will populate with many entries to match common ad serving strings. This list can not be modified, but you can enable or disable the rules here. You are free to add to and edit rules in the User Defined Filters tab, as you see fit - simply click Add and enter a description to match. Be careful when selecting blocking criteria, as you may block much more content than planned if your criteria are not carefully specified.

Cookie Filters

The Standard Cookie Filters list is populated with entries to match common cookie domains. This list can not be modified, but you can enable or disable the rules here. You can add to and edit rules in the User Defined Cookie Filters tab as you see fit.

Pass Lists

Pass Lists are used to pass content that would have otherwise been blocked. This can be useful for "unblocking" sitesyou don't want blocked or allowing certain users special privileges.
  • Passed Sites: Any domains you add to the Passed Sites list will allow ads and cookies, even if blocked by an existing filter - just add the domain and save. Unchecking the pass option will block ads as if the entry was not present.
  • Passed Client IPs: If you add an IP to this list, Ad Blocker will not block any ads or cookies from that IP. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list.

Ad Blocker Reports

The Reports tab provides a view of all reports and events for all traffic handled by Ad Blocker.

Reports

You can access this application via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports and custom reports created will be listed.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Ad Blocker Summary A summary of ad blocker actions.
Ads Blocked The amount of detected and blocked ads over time.
Top Blocked Ad Sites The number of blocked ads grouped by website.
All Ad Events All HTTP requests are scanned by Ad Blocker.
Blocked Ad Events HTTP requests blocked by Ad Blocker.
Blocked Cookie Events Requests blocked by cookie filters.
The tables queried to render these reports:

Related Topics

Application Control

Application Control leverages the Network Application Visibility Library (NAVL) from Procera Networks https://www.sandvine.com/ to perform deep packet (DPI) and deep flow (DFI) inspection of network traffic. This allows the server to accurately identify thousands of today's common applications such as Social Networking, P2P, Instant Messaging, Video Streaming, File Sharing, Enterprise Applications, Web 2.0, and much more.


About Application Control

For most common applications, you can go to the list on the Applications tab and check Block for anything you want to stop. Then, Application Control will take care of the rest. You can use the Rules tab to create custom rules that target more complex traffic patterns if you need more control.

How It Works

Application Control feeds each chunk of data to a classification engine as it passes through the application. The classification engine continues to analyze the traffic flow and keeps properties of the session, such as the Application property. Each time the classification of the Application property is updated, the Applications settings are checked to see if that application is allowed. The data is blocked If the application is configured to be blocked in the settings. If not, the process continues until the session reaches a fully classified state, where the classification engine believes no more session classification is possible. At this point, the Rules are evaluated, and the session is ultimately blocked or passed based on the rules you've configured.

Settings

This section describes the different settings and configuration options available for Application Control.

Status

The Status tab displays a summary of traffic and configuration information. The Traffic Statistics section displays the total number of sessions that have been scanned and the number of those that were allowed, flagged, or blocked. The Application Statistics section shows you the total number of applications that can be detected by the application, along with the number of those protocols that will be flagged and/or blocked. Rules Statistics allows you to quickly see how many custom rules you have configured and how many of those rules are active.

Figure 1. Application Control Status

Applications

The Applications tab is the primary and preferred way for using Application Control to manage network traffic. Find the application you want to target and use the block and flag checkboxes as appropriate. You can sort the list on any of the columns displayed, which should help you find and manage the protocols you want to target. Check Block to stop these applications or Flag to allow them, silently filing them as violations in the Application Control Reports. Use the following definitions to set up the Applications tab for your organization:
  • Application: The unique identifier for the application.
  • Block: Enable this checkbox to block/reset this application's sessions. For TCP, this will actively reset the connection. The packet will be dropped for UDP, and the session will be killed.
  • Tarpit: Enable this checkbox to block/tarpit sessions of this application. For TCP, this makes it appear to both the client and the server that the other party is receiving the data, but it is not responsive. It silently drops the data. For UDP, it is identical in behavior to block, except the connection is kept open so that the next packet will be dropped instead of recategorized as a new session.
  • Flag: Enable the checkbox to flag the traffic. It will be flagged as a violation in Application Control Reports.
  • Name: The standard name for the application.
  • Category: A fairly general and high-level category for the application.
  • Productivity: Productivity is best thought of as an index value between 1 and 5 that rates the potential for each application to improve or increase the overall productivity of your network users, assuming, of course, that listening to music and playing online games is not in their job description. So, applications with a low Productivity index (e.g., MySpace, Hulu, Zynga Games) can be expected to hurt productivity. Items with a high value (e.g., Active Directory or Network File System) are critical for maintaining or improving productivity.
  • Risk: Risk is another index value between 1 and 5 that rates the potential for each protocol or application to allow nasty stuff onto your network. The higher the risk index, the greater the chance of letting in something that could be dangerous or destructive. So low-risk items (e.g., Active Directory, Oracle, LDAP) are generally no cause for concern, while applications rated with a high risk (e.g., BitTorrent, Pando, Usenet) increase the possibility you'll find yourself spending long nights deleting pirated software and cleaning up viruses and other exploits that find their way into your infrastructure.
  • Description: This section provides a more detailed description of each application on the list. In some cases, the description is much larger than will fit within the grid column, so you can click any description to see a pop-up window with the full text displayed.
    Figure 2. Application Control Applications

Rules

If the traffic you need to manage can't be handled via the Applications tab, you can create custom rules to analyze and control traffic based on more complex patterns and conditions. For each session, the rules are only evaluated once after the classification engine has completed an analysis of the traffic. The rules are then evaluated in order until the first match is found; at this point, the configured action will be performed. If there are no matches, the session will be tagged as allowed, the traffic will flow unimpeded, and no further analysis of that traffic will occur.

Note: These rules are evaluated when the classification engine has completed all analyses, which usually occurs after a few packets have passed. This means the rules are useful because enough has been learned about the session that was not known at the session creation time to have powerful rules, such as HTTP or protocol/application information. If the full classification is not completed after 15 chunks of data, then the rules are evaluated given the current information.

If an application is blocked or tarpitted in the Applications tab, it will be blocked immediately when identified before the engine has completed the analysis. In this case, the rules will have NO EFFECT because the sessions are blocked before the rules are evaluated.

Application Control Rules are a very powerful feature for controlling application usage. However, understanding how and when the rules are evaluated is essential for their use.

Figure 3. Application Control Rules



Anatomy of a Rule

An Application Control Rule is a standard rule documented in the Rules documentation. We'll use one of the default rule entries for Ultrasurf to help explain how Rules work. This is exactly the kind of traffic the Rules engine created to seek and destroy. For this particular rule, the objective is to block all traffic that:
  1. Uses port 443.
  2. Looks like valid HTTPS traffic.
  3. Don't use a valid SSL certificate.
To accomplish this, we created four matches:
  1. The first matcher makes sure the rule only looks at TCP traffic.
  2. The second causes the rule to only look at traffic with a destination port 443.
  3. The third matcher is where the real magic starts. In this case, we created a Glob matcher that looks for the /SSL tag anywhere in the Application Control/ProtoChain. (Don't worry, we'll cover globs and chains below!)
  4. The fourth matcher is the frosting on the cake. We tell the rule to look at the Application Control/Detail parameter. This is where the server name from the SSL certificate will be located when an SSL-encrypted session is detected. In this case, we left the Value field empty since we're looking for cases where there is no valid certificate.

Application Detail

The Detail field will contain different types of [#Is there a list of session properties? | information] depending on the protocols detected during session classification. The Detail field will be empty for matcher conditions other than those listed below.

Matcher Detail Contents Example
Application: FBOOKAPP The name of the Facebook Application that is being accessed. wordswithfriends
Application: HTTP The contents of the Content-Type header in the session data coming from the server. image/jpg
ProtoChain: */SSL* The server name extracted from the SSL certificate used to encrypt the session. www.gmail.com
Actions
  • Allow: Allow the traffic.
  • Block: When this option is selected, traffic in both directions will be silently dropped, but the session will remain active.

Application Control Reports

The Reports tab provides a view of all reports and events for all traffic handled by Application Control.

Reports

This application's reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Application Control Summary A summary of Application Control actions.
Top Applications Usage The amount of bandwidth per top application.
Scanned Sessions (all) The amount of scanned, flagged, and blocked sessions over time.
Scanned Sessions (flagged) The amount of flagged, and blocked sessions over time.
Scanned Sessions (blocked) The amount of flagged, and blocked sessions over time.
Top Categories (by sessions) The number of sessions grouped by category.
Top Applications (by sessions) The number of sessions grouped by application.
Top Applications (by size) The number of bytes grouped by application.
Top Flagged Applications The number of flagged sessions grouped by application.
Top Blocked Applications The number of blocked sessions grouped by application.
Top Flagged Hostnames The number of flagged sessions grouped by hostname.
Top Blocked Hostnames The number of blocked sessions grouped by hostname.
Top Flagged Clients The number of flagged sessions grouped by client.
Top Blocked Clients The number of blocked sessions grouped by client.
Top Flagged Usernames The number of flagged sessions grouped by username.
Top Blocked Usernames The number of blocked sessions grouped by username.
Classified Sessions All sessions matching an application control signature.
Flagged Sessions All sessions matching an application control signature and flagged.
Blocked Sessions All sessions matching an application control signature and blocked.
All Sessions All sessions scanned by Application Control.
The tables queried to render these reports:

Application Control Lite

Application Control Lite scans sessions for the specified signatures and will log and/or block sessions based on their content. Many applications can be hard to block based on port, as modern applications will find and use open ports automatically. Application Control Lite provides a basic functionality to block sessions based on the content (data) in the session.

Settings

This section reviews the different settings and configuration options for Application Control Lite.

Status

The Status tab will show you current information on signatures- Available, Logged, and Blocked.


Signatures

The Signatures tab shows the list of current signatures. Signatures are regular expressions written to match known protocols as accurately as possible.

As the early data in each session goes from the server to the client and the client to the server, it is stored in a buffer. As each chunk of data arrives, the data is evaluated against any enabled signatures. If the signature is checked as "log," the session will be tagged and logged as having matched the specified signature. If the signature is checked as "block," it will be logged, and the session will immediately be closed.

Writing custom signatures can be dangerous and difficult. Usually, one of several outcomes will happen when writing a block signature:
  • It will not match anything. In this case, the signature needs to be fixed.
  • It will block the desired protocol/application and nothing else. This is ideal.
  • It will only partially block the protocol. Many multi-session protocols only have some sessions identified. This can have varying effects depending on the application.
  • It will block the protocol and other things (false positives), which can cause major problems with the network.
  • It will block the protocol, and the application will adapt and use an alternative protocol to communicate. Many applications will try alternative techniques to avoid blocking.

As such, great care and expertise are involved in writing signatures to achieve the desired effect.

Note: Application Control Lite, while powerful, can be difficult, time-consuming, and dangerous to configure correctly. Application Control is recommended for most users as it comes preloaded with hundreds of maintained and current behavioral signatures and a commercial third-party application identification engine.
Warning: There was a default signature set in previous and older versions of the NG Firewall. However, enabling block on some default signatures caused false positives and blocked legitimate network traffic. Despite big warnings in the user interface and help documentation, we found users often misconfigured Application Control Lite anyway and experienced network problems. To avoid this issue, there are now no default signatures. You can download the original list of signatures if you have read this warning and understand that misconfiguring Application Control Lite will cause major network connectivity issues.

Reporting

The Reports tab provides a view of all reports and events for all traffic handled by Application Control Lite.

Application Control Lite Reports

The Reports tab provides a view of all reports and events for all traffic handled by Application Control Lite.

Reports

You can access the applications reports via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Application Control Lite Summary A summary of Application Control Lite actions.
Detection Statistics The number of logged and blocked sessions over time.
Top Blocked Protocols The top blocked sessions by protocol.
Top Logged Protocols The top logged sessions by protocol.
Top Blocked Hosts The top blocked sessions by host.
Top Logged Hosts The top logged sessions by host.
Top Blocked Users The top blocked sessions by user.
Top Logged Users The top logged sessions by user.
All Events All sessions are scanned by Application Control Lite.
Blocked Events All sessions matching an application signature and blocked.
The tables queried to render these reports:

SSL Inspector

The SSL Inspector is a special application that allows other NG Firewall applications that process HTTP traffic to process encrypted HTTPS traffic and applications that process SMTP to also process SMTP over SSL. It does this by performing man-in-the-middle decryption and encryption of SSL traffic, passing the unencrypted traffic through the NG Firewall server for inspection by other applications and services.


About SSL Inspector

When a client makes an HTTPS request, the Inspector first initiates a secure SSL connection with the external server on behalf of the client. While this session is being established, the inspector captures the server SSL certificate information. Once the server session is active, the Inspector uses the details from the server certificate to create a new certificate that will be used to encrypt the session between the inspector and the client. This certificate is generated or loaded on the fly and created using the same subject details in the server certificate. The certificate is signed by the internal CA on the NG Firewall server and is used to establish a secure connection between the inspector and the client. Creating the certificate this way is necessary to eliminate security warnings on the client. Still, it does require a few extra steps to configure the client computers and devices on your network properly. See the SSL Certificates section below for details.

SSL Certificates

SSL Certificates serve two primary purposes. They allow traffic between the client and server to be encrypted, and they allow the client to validate the server's authenticity. There are two main ways the client checks the authenticity of the server certificate. The first is validating the server certificate to ensure it has been issued or signed by a known and trusted third-party certificate authority. Once that trust has been established, the client checks the server name portion of the target URL to ensure it matches the server name registered in the certificate presented by the server. If either of these checks fails, the client will typically display a warning, indicating that the connection's security may be compromised.

When the NG Firewall server is initially installed, a default Certificate Authority is automatically created and used to sign the man-in-the-middle certificates created by the SSL Inspector. To view or make changes to the internal Certificate Authority, check out the Certificates tab of the Config/Administration page.

Config > Administration > Certificates

Client Configuration

For the client authenticity checks to be successful, the client must be configured to trust the root certificate used by the NG Firewall server to sign the man-in-the-middle certificates described above. To configure clients, you must first use the Download Root Certificate button located on the Configuration tab of the SSL Inspector Settings page to download the root certificate. You must then install this certificate to correct the client's location.

Another way to download the root certificate is to access a special URL using the IP address of the NG Firewall server:

http://yourserver/cert

Simply replace 0.0.0.0 with the IP address of your NG Firewall server. This method is especially useful when using mobile devices. For example, accessing this URL on an iPad or iPhone will download and display the certificate and provide an option to install and trust the certificate directly on the device.

Below are basic instructions for installing the root certificate on some common client platforms. If yours needs to be listed, or you need any help, consult the reference material for the target platform for further information.

Internet Explorer or Google Chrome on Microsoft Windows

Follow the below steps:
  1. Log into the NG Firewall server running SSL Inspector.
  2. Go to Config > Administration > Certificates and download the certificate using the "Download Root Certificate Authority (CA)" button.
  3. Copy the root_authority.crt you just downloaded to the Windows client computer.
  4. From a command prompt or Start/Run, run the command "certmgr. msc".
  5. Open the "Trusted Root Certification Authorities" tree in the panel on the left.
  6. Right-click on Certificates and select All Tasks > Import .
  7. Proceed with the Certificate Import Wizard, selecting the root_authority.crt file.

Firefox on Microsoft Windows

Follow the below steps:
  1. Log into the NG Firewall server running SSL Inspector.
  2. Go to Config > Administration > Certificates and download the certificate using the "Download Root Certificate Authority (CA)" button.
  3. Copy the root_authority.crt you just downloaded to the Windows client computer.
  4. Launch Firefox.
  5. From the Tools menu, go to Options > Privacy & Security .
  6. Click the Import button and select the root_authority.crt file.
  7. Enable the "Trust this CA to identify websites" checkbox and click the OK button.

Opera on Microsoft Windows

Follow the below steps:
  1. Log into the NG Firewall server running SSL Inspector.
  2. Go to Config > Administration > Certificates and download the certificate using the Download Root Certificate Authority (CA) button.
  3. Copy the root_authority.crt you just downloaded to the Windows client computer.
  4. Launch Opera.
  5. From the Tools menu, go to Preferences > Advanced > Security and click Manage Certificates.
  6. Select the Authorities tab, click Import, and select the root_authority.crt file.
  7. Click Install and click OK if you want to trust the certificate.

Group Policy Distribution

Suppose you have a fully deployed and implemented Active Directory infrastructure. In that case, you can leverage the Group Policy model to distribute the NG Firewall root certificate to your client computers. This is outside our expertise, so we can only help or assist. Still, we have compiled links to some TechNet articles with instructions for several common versions of Windows Server.

Windows Server 2003

Windows Server 2008

Windows Server 2012

Settings

This section describes the different settings and configuration options available for SSL Inspector.

Status

This displays the current status and some statistics.

Figure 4. SSL Inspector Status

Configuration

Execute the following configurations:

Download Root Certificate

As described above, client computers and devices on your network need to be configured to trust the root certificate of the NG Firewall server. Clicking this button will allow you to download the root certificate. Once downloaded, you need to install it in the Trusted Authorities certificate store on your client computers and devices. Note that this is the same root certificate that can be downloaded from the Config > Administration > Certificates page. The download link is included on the SSL Inspector Configuration page for convenience.

Alternatively, you can download the certificate from a client system by navigating to http://yourserver/cert.

Enable SMTPS Traffic Processing

This option is enabled by default and allows the SSL Inspector to work cooperatively with the other applications that act on SMTP mail traffic. When enabled, port 25 mail sessions that use STARTTLS will be decrypted inbound, allowing the clear traffic to pass through all other applications, and re-encrypted again before passing outbound.

Enable HTTPS Traffic Processing

This option is enabled by default and allows the SSL Inspector to work cooperatively with the other applications that act on HTTP web traffic. When enabled, port 443 web sessions that use SSL/TLS will be decrypted inbound, allowing the clear traffic to pass through all other applications and then re-encrypted again before passing outbound.

Block Invalid HTTPS Traffic

When processing a new HTTPS session, the inspector first analyzes the initial client computer request to see if it contains a valid SSL negotiation message. If not, the session will be ignored by default, and the traffic will flow directly between the client and server without inspection. By enabling this checkbox, you can change the default behavior and effectively block any port 443 traffic that does not contain a valid HTTPS signature.

Client/Server Connection Protocols

This section includes checkboxes for turning SSL and TLS protocols on and off to negotiate secure HTTPS and SMTPS inbound and outbound connections. The client protocols are used when the server is communicating with a client. The server protocols are used when the server is communicating with a server.
  • SSLv2Hello - This is really a legacy handshake protocol that is used between a client and server when deciding which encryption protocol to use. This means it's possible to enable SSLv2Hello and still have a TLSv1.x connection negotiated. While there are no known security issues, we still recommend leaving this disabled unless you specifically need this legacy support.
  • SSLv3 - This is an older protocol that is been deprecated since the discovery of the POODLE vulnerability. For that reason, we recommend this be disabled for maximum security.
  • TLSv1 - This is an older protocol that has some known weaknesses. These can be mitigated if the other side of the connection forces certain secure ciphers to be used. However, since this can only be guaranteed, the best practice is to disable this protocol if it is required to support connections with legacy clients or servers.
  • TLSv1.1 - This is a modern protocol that is generally regarded as secure and is used as a fallback for 1.2 or in older browsers.
  • TLSv1.2 - This is the most common and recommended TLS version.
  • TLSv1.3- This is the most recent version of the TLS protocol and offers the highest security but may have issues with some websites.

Trust All Server Certificates

Normally, when establishing an SSL connection with an external web server, the inspector will authenticate the server certificate against a standard list of trusted certificate authorities. If this trust cannot be established, the inspector will end the session. By enabling this checkbox, you can force the inspector to blindly trust all external server certificates.

Note that we DO NOT recommend running with this option enabled, as it exposes all HTTPS traffic to significant security risks.

The standard list of trusted certificates used by NG Firewall is generated from the standard ca-certificates package. It includes, among others, certificate authorities used by Mozilla's browsers. Please note that Edge Threat Management staff can neither confirm nor deny whether the certificate authorities whose certificates are included in this list have in any way been audited for trustworthiness or RFC 3647 compliance. The local system administrator is fully responsible for assessing them.

Upload Trusted Certificate

Note:This setting applies to all policies when using SSL Inspector with Policy Manager.

The inspector emulates a web browser when it makes outbound connections to external web servers. Just like a web browser, it must verify the authenticity of the server certificate before it will trust the connection and allow traffic to flow freely. As mentioned above, the inspector uses a standard list of known certificate authorities to validate server certificates. However, it's also possible you have servers in your network that use certificates that can't be authenticated this way. Perhaps you have your certificate authority or use self-signed certificates. Whatever the reason, you can use this section of the configuration page to upload additional certificates you want the inspector to trust.

Figure 5. SSL Inspector Configuration

Rules

The Rules tab allows you to specify explicit rules to Inspect or Ignore HTTPS traffic that crosses the NG Firewall. By default, many common HTTPS sites (Google, YouTube, Yahoo, and so on) are inspected, but not all HTTPS. This provides a safe default which provides HTTPS inspection on those sites without interfering with other HTTPS communications. It can easily be configured to inspect all HTTPS by enabling the "Inspect All Traffic" rule.

The Rules describe how rules work and how they are configured. SSL Inspector uses rules to determine if it should inspect or ignore traffic for the specific session.

In addition to all the common rule types, three are unique to the SSL Inspector, and these can be very useful for ignoring traffic that you don't want to inspect, or that isn't compatible with the SSL Inspector.

HTTPS: SNI Hostname

Most web browsers and many client applications include the destination hostname in the initial packet of an HTTPS session. The mechanism is called the Server Name Indication or the SNI extension to the TLS protocol. The main purpose is to allow a single web server to host multiple secure web sites. By analyzing the SNI hostname in the client request, the server can decide which SSL certificate to use to encrypt the session. This extension is necessary because the encryption must be established long before the server ever sees the HTTP request, and by then, it would be too late to use a different certificate.

Creating ignore rules based on the SNI hostname is an effective way to have the SSL Inspector ignore incompatible traffic. A prime example is the default rule for Microsoft Update. The Microsoft Update client checks the server certificate to ensure it was signed by a specific authority. Since it doesn't trust the Root Authority the SSL Inspector uses to generate certificates on the fly, Microsoft Update will fail with an error. The default rule allows this traffic to be detected and ignored, allowing Microsoft Update to work properly.

HTTPS: Certificate Subject and HTTPS: Certificate Issuer

These two rule conditions are useful when dealing with client applications that don't use SNI and aren't compatible with SSL Inspector. An excellent example is the Dropbox client utility, for which there is also a default rule. Like Microsoft Update, the Dropbox client will reject SSL certificates it doesn't explicitly trust.

Using either of these rule conditions, you can match traffic on any portion of the Subject or Issuer Distinguished Name (DN) included in the server certificate. In both cases, the information in the match string includes the standard information fields commonly stored within the SSL certificates, such as CN (common name), C (country), ST (state), L (locality), O (organization), and OU (organizational unit). Each of these is appended to the match string and separated by commas. Not all fields are required in all certificates, and some certificates may have others not listed. The order they occur in the match string is also not guaranteed.

The Subject DN generally includes information about the company to which the certificate was issued. Here is an example Certificate Subject: 
CN=*.dropbox.com, O="Dropbox, Inc.", L=San Francisco, ST=California, C=US
The Issuer DN generally includes information about the company that issued and authenticated the certificate. Here is an example Certificate Issuer: 
CN=Thawte SSL CA, O="Thawte, Inc.", C=US
Rule Actions
  • Inspect: Causes the traffic that matches the rule to be decrypted and passed along to other applications and services for further inspection, classification, and possible action.
  • Ignore: Causes the traffic that matches the rule to be ignored by the SSL Inspector.
    Figure 6. SSL Inspector Rules

SSL Inspector Reports

The Reports tab provides a view of all reports and events for all traffic handled by HTTPS Inspector.

Reports

This application's reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports and custom reports created will be listed.

You can search and defined Reports using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
SSL Inspector Summary A summary of SSL Inspector actions.
Sessions Scanned The amount of SSL sessions over time.
Sessions Inspected The amount of inspected SSL sessions over time.
Top Inspected Sites The number of inspected sessions grouped by site.
Top Ignored Sites The number of ignored sessions grouped by site.
All Sessions All sessions detected by SSL Inspector.
Inspected Sessions Events where traffic was fully processed by the inspector, and all traffic was passed through all the other applications and services.
Ignored Sessions Events where traffic was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
Blocked Sessions Events where traffic was blocked because it did not contain a valid SSL request and the Block Invalid Traffic option was enabled.
Untrusted Sessions Events where traffic was blocked because the server certificate could not be authenticated.
Abandoned Sessions Events where traffic was blocked due to an underlying problems with the SSL session.

The tables queried to render these reports:

Status

The status of the session that generated the event.
  • INSPECTED means the session was fully processed by the inspector, and all traffic was passed through all the other applications and services.
  • IGNORED means the session was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
  • BLOCKED means the traffic was blocked because it did not contain a valid HTTPS request, and the Block Invalid Traffic option was enabled.
  • UNTRUSTED means the traffic was blocked because the server certificate could not be authenticated.
  • ABANDONED means the connection failed because an underlying SSL connection problem. Usually, the client abandoned the connection because the certificate was not trusted.

Detail

Extra details about the session, with the exact content dependent on the event status.

For INSPECTED and UNTRUSTED sessions, this field will include the SNI hostname extracted from the initial message sent from the client to the server. If the SNI information is not available, the server IP address will be used instead.

For BLOCKED or IGNORED sessions, this field will contain the description of the rule that matched and was applied to the session.

For ABANDONED sessions, the detail will usually record information about the error that caused the inspection to fail. For SSL exceptions, this will include the Client or Server to indicate the session endpoint for which traffic was being processed. It will also include Encrypt or Decrypt to indicate the state of traffic inspection when the exception occurred. If available, the SSL error message will also be included. The following table lists the most common error messages and detailed information about each one.

Table 1. SSL Exception Messages
SSL Exception Messages Description
unexpected_message An inappropriate message was received. This alert is always fatal and should never be observed in communication between proper implementations.
bad_record_mac This alert is returned if a record is received with an incorrect MAC. This alert also MUST be returned if an alert is sent because a TLSCiphertext decrypted in an invalid way: either it wasn't an even multiple of the block length, or its padding values, when checked, weren't correct. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decryption_failed This alert was used in some earlier versions of TLS, and may have permitted certain attacks against the CBC mode [CBCATT]. It MUST NOT be sent by compliant implementations.
record_overflow A TLSCiphertext record was received that had a length more than 2^14+2048 bytes, or a record decrypted to a TLSCompressed record with more than 2^14+1024 bytes. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decompression_failure The decompression function received improper input (e.g., data that would expand to excessive length). This message is always fatal and should never be observed in communication between proper implementations.
handshake_failure Reception of a handshake_failure alert message indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. This is a fatal error.
no_certificate This alert was used in SSLv3 but not any version of TLS. It MUST NOT be sent by compliant implementations.
bad_certificate A certificate was corrupt, contained signatures that did not verify correctly, etc.
unsupported_certificate A certificate was of an unsupported type.
certificate_revoked A certificate was revoked by its signer.
certificate_expired A certificate has expired or is not currently valid.
certificate_unknown Some other (unspecified) issue arose in processing the certificate, rendering it unacceptable.
illegal_parameter A field in the handshake was out of range or inconsistent with other fields. This message is always fatal.
unknown_ca A valid certificate chain or partial chain was received, but the certificate was not accepted because the CA certificate could not be located or couldn't be matched with a known, trusted CA. This message is always fatal.
access_denied A valid certificate was received, but when access control was applied, the sender decided not to proceed with negotiation. This message is always fatal.
decode_error A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decrypt_error A handshake cryptographic operation failed, including being unable to correctly verify a signature or validate a Finished message. This message is always fatal.
export_restriction This alert was used in some earlier versions of TLS. It MUST NOT be sent by compliant implementations.
protocol_version The protocol version the client has attempted to negotiate is recognized but not supported. (For example, old protocol versions might be avoided for security reasons.) This message is always fatal.
insufficient_security Returned instead of handshake_failure when a negotiation has failed specifically because the server requires ciphers more secure than those supported by the client. This message is always fatal.
internal_error An internal error unrelated to the peer or the correctness of the protocol (such as a memory allocation failure) makes it impossible to continue. This message is always fatal.
user_canceled This handshake is being canceled for some reason unrelated to a protocol failure. If the user cancels an operation after the handshake is complete, just closing the connection by sending a close_notify is more appropriate. This alert should be followed by a close_notify. This message is generally a warning.
no_renegotiation Sent by the client in response to a hello request or by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation; when that is not appropriate, the recipient should respond with this alert. At that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate is where a server has spawned a process to satisfy a request; the process might receive security parameters (key length, authentication, etc.) at startup, and it might be difficult to communicate changes to these parameters after that point. This message is always a warning.
unsupported_extension sent by clients that receive an extended server hello containing an extension that they did not put in the corresponding client hello. This message is always fatal.

Related Topics

Spam Blocker

Spam Blocker is an intelligent email filter that identifies and handles spam (unsolicited bulk email). It leverages technology from the https://spamassassin.apache.org/ project and improves upon it by integrating a commercial spam engine. It can scan any email that is transported via SMTP.

Spam Blocker transparently scans email transported over SMTP on the way to your mail server (or outbound if configured so). It does not require reconfiguring your DNS MX records or the email server. Any SMTP traffic going through the NG Firewall server will be scanned.

Settings

This section discusses the different settings and configuration options available for Spam Blocker.

Status

This displays the current status and some statistics.


Email

These settings apply only to the SMTP protocol.
  • Scan SMTP: This turns SMTP scanning on or off.
  • Strength: If the Spam Score of a message is equal to or greater than this setting, your chosen action will be taken regarding the message. Higher values make Spam Blocker more sensitive to spam.
  • Action: The action was taken regarding the message if the spam score is high enough.
  • If set to Mark, "[Spam]..." will be prepended to the email subject line and delivered. If set to Pass, the message will be delivered as originally sent. The drop will inform the sending server the mail was successfully delivered, but the NG Firewall will drop the mail, so it is never delivered. Quarantine will send the mail to users' email quarantine for them to release or delete as they see fit. For more information, refer to Quarantine.
  • Drop Super Spam: If this option is enabled, any emails with a score greater than the Super Spam score will be dropped.
  • Super Spam Score: The score emails must reach to be dropped as Super Spam.
  • Advanced SMTP Configuration
    Important: The default values are the suggested values. Changing and customizing settings can cause Spam Blocker to perform less than optimally.
    • Enable tarpitting: This option enables Tarpit. If enabled, when an SMTP session is first caught, the Spam Blocker will check if the client's IP is on a Domain Name System BLocklist (DNSBL). If it is, the session is rejected before the remote server can send the email. This increases the capacity of a given server by quite a bit and can also save bandwidth. Still, it can increase false positives if the remote email server has mistakenly been put on a blacklist. This setting will not increase spam scanning accuracy - it will decrease it as it will prevent valuable super-spam training data from reaching the spam engine. Enabling this feature gives you lower spam accuracy but increased email scanning capacity.
    • Enable greylisting: This option enables greylisting. If enabled, each time a new sender tries to send mail to a specific receiver, it will receive a "421 Please try again" error. The second time, mail will be allowed. Greylisting will reduce spam because spammers often won't retry transmissions as they should, or the extra time delay will increase the chances of the spam engine properly identifying new spam waves. However, greylisting adds a delay to all legitimate emails. This setting is not suggested for most sites because of the complications.
    • Add email headers: When enabled, NG Firewall adds information about the Spam Score and the test run to get that score to the message's headers.
    • Close connection on scan failure: This option will close the connection if the scan fails, so the message will be resent and retested. If disabled, a scan failure will allow the email to be delivered without being scanned.
    • Scan outbound (WAN) SMTP: If unchecked, outbound mail (mail-in sessions going out a WAN interface) will not be scanned. If checked, outbound mail will be scanned just like incoming mail.
    • Allow and ignore TLS sessions: This option controls the allowance of TLS sessions. If unchecked (the default), the TLS advertisement (if present) is removed from the server advertisements, and TLS is not allowed in any scanned sessions. If checked, the TLS advertisement is allowed, and if the client initialized TLS, the message will pass through completely unscanned, even if it is spam.
    • CPU Load Limit: If your CPU load exceeds this number, incoming connections will be stopped until the load decreases. This is specified so that spam scanning can not monopolize the server resources.
    • Concurrent Scan Limit: This is the maximum number of messages that can be scanned simultaneously. It is specified so that spam scanning does not monopolize server resources.
    • Message Size Limit: This option allows you to change the maximum size of a message that will be scanned for spam. The default maximum size is 256KB. Spam will typically be much smaller, as spammers rely on the number of messages sent.
      Note: This does not control the message size limit of messages passed through the NG Firewall. It does not affect the maximum size of the message your server will accept, only the limit on the size of the message that will be checked for spam.

Spam Blocker Reports

The Reports tab provides a view of all reports and events for all traffic Spam Blocker handles.

Reports

You can access the application's reports via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

You can search and defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Spam Blocker Summary A summary of spam blocking actions for email activity.
Email Usage (all) The amount of scanned, clean, and spam email over time.
Email Usage (scanned) The amount of scanned email over time.
Email Usage (clean) The amount of clean email over time.
Email Usage (spam) The amount of spam email over time.
Spam Ratio The ratio of spam (true) to ham (false)
Top Spam Recipients The number of email addresses with spam.
Top Spam Sender Addresses The number of IP addresses sending spam.
All Email Events All emails are scanned by Spam Blocker.
All Spam Events All emails are marked as Spam.
Quarantined Events All emails are marked as Spam and quarantined.
Tarpit Events All email sessions that were tarpitted.
The tables queried to render these reports:

Spam Blocker Lite

Spam Blocker Lite is an intelligent email filter that identifies spam (unsolicited bulk email). It leverages technology from the https://spamassassin.apache.org/ project. It can scan any email that is transported via SMTP.

Spam Blocker Lite transparently scans email transported over SMTP on the way to your mail server (or outbound if configured so). It does not require any reconfiguration of DNS MX records or any email servers. Any SMTP traffic going through the NG Firewall server will be scanned.

Settings

This section discusses the different settings and configuration options available for Spam Blocker Lite.

Status

This displays the current status and some statistics.


Email

These settings apply only to the SMTP protocol.
  • Scan SMTP: This turns SMTP scanning on or off.
  • Strength: If the spam score of a message is equal to or greater than this setting, your chosen action will be taken regarding the message. Higher values make Spam Blocker more sensitive to spam.
  • Action: The action was taken regarding the message if the spam score is high enough.
  • If set to Mark, "[Spam]..." will be prepended to the email subject line and delivered. If set to Pass, the message will be delivered as originally sent. The drop will inform the sending server the mail was successfully delivered, but the NG Firewall will drop the mail, so it is never delivered. Quarantine will send the mail to users' email quarantine for them to release or delete as they see fit. For more information, refer to Quarantine.
  • Drop Super Spam: If this option is enabled, any emails with a score greater than the Super Spam score will be dropped.
  • Super Spam Score: The score emails must reach to be dropped as Super Spam.
  • Advanced SMTP Configuration:
    Note: The default values are the suggested values. Changing and customizing settings can cause Spam Blocker to perform less than optimally.
    1. Enable tarpitting: This option enables Tarpit. If enabled, when an SMTP session is first caught, the Spam Blocker will check if the client's IP is on a Domain Name System BLocklist (DNSBL). If it is, the session is rejected before the remote server can send the email. This increases the capacity of a given server by quite a bit and can also save bandwidth. Still, it can increase false positives if the remote email server has mistakenly been put on a blacklist. This setting will not increase spam scanning accuracy - it will decrease it as it will prevent valuable super-spam training data from reaching the spam engine. Enabling this feature gives you lower spam accuracy but increased email scanning capacity.
    2. Add email headers: When enabled, NG Firewall adds information about the Spam Score and the test run to get that score to the message's headers.
    3. Close connection on scan failure: This option will close the connection if the scan fails so that the message will be resent and retested. If disabled, a scan failure will allow the email to be delivered without being scanned.
    4. Scan outbound (WAN) SMTP: If unchecked, outbound mail (mail-in sessions going out a WAN interface) will not be scanned. If checked, outbound mail will be scanned just like incoming mail.
    5. Allow and ignore TLS sessions: This option controls the allowance of TLS sessions. If unchecked (the default), the TLS advertisement (if present) is removed from the server advertisements, and TLS is not allowed in any scanned sessions. If checked, the TLS advertisement is allowed, and if the client initializes TLS, the message will pass through completely unscanned, even if it is spam.
    6. CPU Load Limit: If your CPU load exceeds this number, incoming connections will be stopped until the load decreases. This is specified so that spam scanning can not monopolize the server resources.
    7. Concurrent Scan Limit: This is the maximum number of messages that can be scanned simultaneously. It is specified so that spam scanning does not monopolize server resources.
    8. Message Size Limit: This option allows you to change the maximum size of a message that will be scanned for spam. The default maximum size is 256KB. Spam will typically be much smaller, as spammers rely on the number of messages sent.
      Note: This does not control the message size limit of messages passed through the NG Firewall. It does not affect the maximum size of message your server will accept, only the limit on the size of message that will be checked for spam.

Spam Blocker Lite Reports

The Reports tab provides a view of all reports and events for all traffic handled by Spam Blocker Lite.

Reports

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Spam Blocker Lite Summary A summary of spam blocking actions for email activity.
Email Usage (all) The amount of scanned, clean, and spam email over time.
Email Usage (scanned) The amount of scanned email over time.
Email Usage (clean) The amount of clean email over time.
Email Usage (spam) The amount of spam email over time.
Spam Ratio The ratio of spam (true) to ham (false)
Top Spam Recipients The number of email addresses with spam.
Top Spam Sender Addresses The number of IP addresses sending spam.
All Email Events All emails are scanned by Spam Blocker.
All Spam Events All emails are marked as Spam.
Quarantined Events All emails are marked as Spam and quarantined.
Tarpit Events All email sessions that were tarpitted.
The tables queried to render these reports:

Web Filter

Web Filter monitors HTTP and HTTPS traffic on your network to filter and log web activities and block inappropriate content. Web Filter also appeals to customers who require an added level of protection or are subject to regulations; for example, Web Filter helps libraries comply with the Children's Internet Protection Act). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.


About Web Filter

  • Real-time classification and updates: When users visit a site, NG Firewall sends the URL to the Webroot BrightCloud® to be categorized. When the data is returned, the NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is auto-categorized by our partners at Webroot and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention; plus, you can request https://www.brightcloud.com/tools/url-ip-lookup.php of sites.
  • HTTPS Filtering: Web Filters have multiple techniques to deal with HTTPS SSL-encrypted HTTP. HTTPS traffic is encrypted, so only some information is visible, and this information is used to categorize the session. More information on how this works is below.
  • Detailed categorization: Web Filter offers 79 categories and tens of billions of URLs. The Web Filter database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope—maybe you want to flag websites related to nudity but allow sites dealing with sexual education.
  • Advanced features: Force safe Search on search engines, filter and log user searches, restrict Google domains, and more!

Traffic Flow

When scanning traffic, Web Filter evaluates the pass lists, block lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the client receives the request and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of filtering and control over both requested resources and Content returned in response.

HTTP Request

When evaluating HTTP requests, Web Filter applies the configured rules and lists in the following order:
  1. A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Filter as well as other applications.
  2. The request's source IP is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  3. The request's destination site is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  4. If 'Restrict Google applications' is enabled, the appropriate header is added to the request using the 'Allowed Domains' configured.
  5. If 'Pass if a referrer matches any Pass Sites' is enabled, the referrer is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  6. If the Unblock option is enabled, the request's destination site and source IP are checked against the unblock list. If a match is found, the traffic is allowed.
  7. If 'Block pages from IP only hosts' is enabled, the request will be evaluated and blocked if the destination is an IP address.
  8. The request's destination site is checked against the Block Sites list. If a match is found, the traffic is blocked.
  9. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.
  10. The category determined in Step 1 is compared to the Categories list, and the traffic is allowed, flagged, or blocked based on the corresponding match. If the category cannot be determined, the traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Filter applies the configured rules and lists in the following order:
  1. The request's source IP is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  2. The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  3. If the Unblock option is enabled, the site from which the response was received and the client IP are checked against the Unblock list. If a match is found, the traffic is allowed.
  4. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.

Settings

This section discusses the different settings and configuration options available for Web Filter.

Status

This displays the current status and some statistics.

Figure 7. Web Filter Status

Categories

Categories enable you to customize which categories of sites will be blocked or flagged. Blocked categories will display a block page to the user; flagged categories will allow the user to access the site but will be silently flagged as a violation for event logs and Reports. These block/flag actions operate similarly for all Web Filter options.

Figure 8. Web Filter Categories

Search Terms

Search Terms filtering enables you to flag or block specific search terms your users perform on popular search sites, including Google, Bing, Ask, Yahoo, and YouTube. For example, if someone searches Google and includes the word "suicide" or searches for "twerking" videos on YouTube, you can have these activities flagged or blocked.

Under Search Filter, you can add terms you want to be blocked or flagged. Search Filter terms use the Glob Matcher syntax.

Figure 9. Web Filter Search Terms

In many cases, you may have an existing list of search terms that you want to import. For example, you can find several banned words on Facebook, YouTube, WordPress, and other sources. Full List of Bad Words and Top Swear Words Banned by Google Block Facebook, YouTube. The import feature lets you import these lists from a comma-separated or newline delimited file. You can also import it in JSON format if you transfer a list from another NG Firewall.

Note: SSL Inspector must be installed and enabled to use search filtering.

Site Lookup

Site Lookup allows you to find a URL's categorization. Clicking it brings up a dialog. In Site URL, specify the URL to find and click Search to find the URL's categorization.

If you feel the current categorization is incorrect, check Suggest a different category, select a new category from the list, and click Suggest to submit the category change for consideration.

Note: This is only a suggestion. It will trigger the URL categorization provider to recategorize all categories for the URL.
Figure 10. Web Filter Site Lookup

Block Sites

Under Block Sites, you can add individual domain names you want to be blocked or flagged - enter the domain name (e.g., youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

Figure 11. Web Filter Block Sites

Pass Sites

Pass Sites is used to pass Content that would have otherwise been blocked. This can be useful for "unblocking" sites you don't want to be blocked according to block settings. Any domains you add to the Passed Sites list will be allowed, even if blocked by category or by individual URL - add the Domain and save. Unchecking the pass option will allow the site to be blocked as if the entry was not present. This list uses URL Matcher syntax.

Figure 12. Web Filter Pass Sites

Pass Clients

If you add an IP address to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

Consider using pass lists if you have a few users who must bypass Web Filter controls completely. If you have users needing different Web Filter settings, you should set up a separate policy using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up a Static IP or a Static DHCP Lease for the machine in question.

Figure 13. Web Filter Pass Clients

Rules

The Rules tab allows you to specify rules to Block or Flag traffic that passes through Web Filter.

The Rules describe how rules work and how they are configured. Web Filter uses rules to determine whether to block or flag a specific session. Flagging a session marks it in the logs for review in the event logs or reports but has no direct effect on the network traffic.

Figure 14. Web Filter Rules

Rule Actions
  • Flag: This allows the traffic that matches the rule to flow and flags the traffic for easier viewing in the event log.
  • Block: Blocks the traffic that matches the rule.

Rule Types

In previous versions of Web Filter, there were dedicated lists for blocking certain file extensions or MIME types. This capability is still available when using more flexible filter rules. To block specific file extensions, you can create a rule with the condition HTTP: Response File Extension that has a comma-separated list of the extensions to block in the Value field. You would create a rule for blocking MIME types with the condition HTTP: Content Type with a comma-separated list of the content types to block in the Value field.

Below are tables listing the default file extensions and MIME types available in previous versions. Note that these lists are not exhaustive but are included here to simplify the creation of such rules by copying and pasting the values in the tables.

Extension Category Description
exe executable an executable file format
ocx executable an executable file format
dll executable an executable file format
cab executable an ActiveX executable file format
bin executable an executable file format
com executable an executable file format
jpg image an image file format
png image an image file format
gif image an image file format
jar java a Java file format
class java a Java file format
swf flash the flash file format
mp3 audio an audio file format
wav audio an audio file format
wmf audio an audio file format
mpg video a video file format
mov video a video file format
avi video a video file format
hqx archive an archived file format
cpt compression a compressed file format
Content Category Description
application/octet-stream unspecified data byte stream
application/x-msdownload Microsoft download executable
application/exe executable executable
application/x-exe executable executable
application/dos-exe DOS executable executable
application/x-winexe Windows executable executable
application/msdos-windows MS-DOS executable executable
application/x-msdos-program MS-DOS program executable
application/x-oleobject Microsoft OLE Object executable
application/x-java-applet Java Applet executable
audio/mpegurl MPEG audio URLs audio
audio/x-mpegurl MPEG audio URLs audio
audio/mp3 MP3 audio audio
audio/x-mp3 MP3 audio audio
audio/mpeg MPEG audio audio
audio/mpg MPEG audio audio
audio/x-mpeg MPEG audio audio
audio/x-mpg MPEG audio audio
application/x-ogg Ogg Vorbis audio
audio/m4a MPEG 4 audio audio
audio/mp2 MP2 audio audio
audio/mp1 MP1 audio audio
application/ogg Ogg Vorbis audio
audio/wav Microsoft WAV audio
audio/x-wav Microsoft WAV audio
audio/x-pn-wav Microsoft WAV audio
audio/aac Advanced Audio Coding audio
audio/midi MIDI audio audio
audio/mpeg MPEG audio audio
audio/aiff AIFF audio audio
audio/x-aiff AIFF audio audio
audio/x-pn-aiff AIFF audio audio
audio/x-pn-windows-acm Windows ACM audio
audio/x-pn-windows-pcm Windows PCM audio
audio/basic 8-bit u-law PCM audio
audio/x-pn-au Sun audio audio
audio/3gpp 3GPP audio
audio/3gpp-encrypted encrypted 3GPP audio
audio/scpls streaming mp3 playlists audio
audio/x-scpls streaming mp3 playlists audio
application/smil SMIL audio
application/sdp Streaming Download Project audio
application/x-sdp Streaming Download Project audio
audio/amr AMR codec audio
audio/amr-encrypted AMR encrypted codec audio
audio/amr-wb AMR-WB codec audio
audio/amr-wb-encrypted AMR-WB encrypted codec audio
audio/x-rn-3gpp-amr 3GPP codec audio
audio/x-rn-3gpp-amr-encrypted 3GPP-AMR encrypted codec audio
audio/x-rn-3gpp-amr-wb 3gpp-AMR-WB codec audio
audio/x-rn-3gpp-amr-wb-encrypted 3gpp-AMR_WB encrypted codec audio
application/streamingmedia Streaming Media audio
video/mpeg MPEG video video
audio/x-ms-wma Windows Media video
video/quicktime QuickTime video
video/x-ms-asf Microsoft ASF video
video/x-msvideo Microsoft AVI video
video/x-sgi-mov SGI movie video
video/3gpp 3GPP video video
video/3gpp-encrypted 3GPP encrypted video video
video/3gpp2 3GPP2 video video
audio/x-realaudio RealAudio audio
text/vnd.rn-realtext RealText text
audio/vnd.rn-realaudio RealAudio audio
audio/x-pn-realaudio RealAudio plug-in audio
image/vnd.rn-realpix RealPix image
application/vnd.rn-realmedia RealMedia video
application/vnd.rn-realmedia-vbr RealMedia VBR video
application/vnd.rn-realmedia-secure secure RealMedia video
application/vnd.rn-realaudio-secure secure RealAudio audio
audio/x-realaudio-secure secure RealAudio audio
video/vnd.rn-realvideo-secure secure RealVideo video
video/vnd.rn-realvideo RealVideo video
application/vnd.rn-realsystem-rmj RealSystem media video
application/vnd.rn-realsystem-rmx RealSystem secure media video
audio/rn-mpeg MPEG audio audio
application/x-shockwave-flash Macromedia Shockwave multimedia
application/x-director Macromedia Shockwave multimedia
application/x-authorware-bin Macromedia Authorware binary multimedia
application/x-authorware-map Macromedia Authorware shocked file multimedia
application/x-authorware-seg Macromedia Authorware shocked packet multimedia
application/futuresplash Macromedia FutureSplash multimedia
application/zip ZIP archive
application/x-lzh LZH archive archive
image/gif Graphics Interchange Format image
image/png Portable Network Graphics image
image/jpeg JPEG image
image/bmp Microsoft BMP image
image/tiff Tagged Image File Format image
image/x-freehand Macromedia Freehand image
image/x-cmu-raster CMU Raster image
image/x-rgb RGB image image
text/css cascading style sheet text
text/html HTML text
text/plain plain text text
text/richtext rich text text
text/tab-separated-values tab separated values text
text/xml XML text
text/xsl XSL text
text/x-sgml SGML text
text/x-vcard vCard text
application/mac-binhex40 Macintosh BinHex archive
application/x-stuffit Macintosh Stuffit archive archive
application/macwriteii MacWrite Document document
application/applefile Macintosh File archive
application/mac-compactpro Macintosh Compact Pro archive
application/x-bzip2 block compressed compressed
application/x-shar shell archive archive
application/x-gtar gzipped tar archive archive
application/x-gzip gzip compressed compressed
application/x-tar 4.3BSD tar archive archive
application/x-ustar POSIX tar archive archive
application/x-cpio old cpio archive archive
application/x-bcpio POSIX cpio archive archive
application/x-sv4crc System V cpio with CRC archive
application/x-compress UNIX compressed compressed
application/x-sv4cpio System V cpio archive
application/x-sh UNIX shell script executable
application/x-csh UNIX csh script executable
application/x-tcl Tcl script executable
application/x-javascript JavaScript executable
application/x-excel Microsoft Excel document
application/mspowerpoint Microsoft Powerpoint document
application/msword Microsoft Word document
application/wordperfect5.1 Word Perfect document
application/rtf Rich Text Format document
application/pdf Adobe Acrobat document
application/postscript Postscript document

Advanced

The Advanced section enables you to configure additional web filter options.
  • Secure Name Indication Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in HTTPS Options.
  • Process HTTPS traffic by hostname in server certificate when SNI information is not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server, and the server name on the certificate will be used for categorization and filtering purposes.
  • Process HTTPS traffic by server IP if SNI and certificate hostname information are unavailable: If this option is enabled and neither of the previous options works, HTTPS traffic will be categorized using the IP address. More details in HTTPS Options.
Safe Browsing
  • Enforce safe Search on popular search engines: When this option is enabled, safe Search will be enforced on all searches using supported search engines: Google, Yahoo!, Bing, and Ask.
  • Enforce restrict mode on YouTube: When this option is enabled, restrict mode will be enforced on all YouTube content.
  • Force searches through kid-friendly search engines: When this option is enabled, all searches in popular search engines will be redirected through https://www.kidzsearch.com/. KidzSearch is a visual child-safe search engine and web portal powered by Google Custom Search with academic auto-complete that emphasizes safety for children.
Note: SSL Inspector must be installed and enabled to use all Safe browsing options.
Block Options
  • Block QUIC Sessions (UDP port 443): Web Filter prevents browsers from using the QUIC protocol if enabled. The Chrome browser uses QUIC to access many types of Google applications and services. By allowing QUIC, NG Firewall has less visibility and control over this type of traffic.
  • Block pages from IP-only hosts: When this option is enabled, users entering an IP address rather than a domain name will be blocked.
  • Pass if the referrer matches any Pass Sites: When this option is checked, if a site allowed via a Pass Site entry links to external Content using a referrer, that external Content will be passed regardless of Category settings. Please note that this setting requires SSL Inspector to work with HTTPS sites; many sites no longer use referrers.
  • Close connection for blocked HTTPS sessions without redirecting to the block page: If enabled, secure sites blocked by Web Filter do not redirect the user to a denial page and close the connection without any notice to the user. This is useful when you are not using SSL Inspector, and the server's root certificate authority is not installed on the client device.
Google Restrictions
  • Restrict Google applications: Only domains listed in Domain can access Google applications such as Gmail when this option is enabled. Google blocks all others. Multiple domains can be specified and separated by commas, such as google.com and domain.com. This adds an X-GoogApps-Allowed-Domains header to web requests, which is enforced on Google's servers. More information on this feature can be found https://support.google.com/a/answer/1668854?hl=en.
Note: SSL Inspector must be installed and enabled to restrict Google applications.

Custom Block Page

  • Custom block page URL: Set an external location where to redirect users when they are denied access to a web site by Web Filter. This is useful if you want to use your own server to process the denial in a different way than the built-in denial options.

    The following query string variables are passed to the forward location so the receiving system can process the information.

    Variable Description
    reason The reason the user was denied access.
    appname The NG Firewall app that is responsible for the denial.
    appid The ID of the NG Firewall app that is responsible for the denial.
    clientAddress The IP Address of the denied device.
    url The denied URL that the user requested.
Unblock Options
  • Unblock: This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.

    If Unblock is set to None, no users will be allowed to bypass the block page. If Unblock is set to Temporary, users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to Permanent and Global, then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.

    You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the NG Firewall or you can set a new, separate password only for the Unblock feature.

  • Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.
    Figure 15. Web Filter Advanced Tab

HTTPS Options

There are many ways to handle HTTPS.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Filter processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.
  • Process HTTPS traffic by SNI (Server Name Indication) if present.
  • Process HTTPS traffic by hostname in server certificate when SNI information not present
  • Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

If the SNI-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, "wellsfargo.com" is seen as the SNI information. If SNI-based categorization is enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If Web Filter is configured to flag "Financial Services" , then "https://wellsfargo.com/welcome" will be flagged, unless "wellsfargo.com" is in the pass list or the client IP address is in the client IP pass list.

If No SNI information is present and Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled, then the hostname will be pulled from the certificate presented to the client.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

If no SNI or certificate information was available and Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available the session will be processed and categorized by IP address. If the IP-based processing and categorization of the web requests determines the session should be flagged, the session is reset and no more processing of this session will be done. If the IP-based processing and categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS processing (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required.

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Filter Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Filter.

Reports

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Web Filter Summary A summary of web filter actions.
Web Usage The amount of total, flagged, and blocked web requests over time.
Web Usage (scanned) The amount of total, flagged, and blocked web requests over time.
Web Usage (flagged) The amount of flagged, and blocked web requests over time.
Web Usage (blocked) The amount of flagged, and blocked web requests over time.
Top Categories (by request) The number of web requests grouped by category.
Top Categories (by size) The sum of the size of requested web content grouped by category.
Top Flagged Categories The number of flagged web requests grouped by category.
Top Blocked Categories The number of blocked web requests grouped by category.
Top Sites (by request) The number of web requests grouped by website.
Top Sites (by size) The sum of the size of requested web content grouped by website.
Top Flagged Sites The number of flagged web requests grouped by website.
Top Blocked Sites The number of blocked web requests grouped by website.
Top Domains (by request) The number of web requests grouped by domain.
Top Domains (by size) The sum of the size of requested web content grouped by domain.
Top Flagged Domains The number of flagged web requests grouped by domain.
Top Blocked Domains The number of blocked web requests grouped by domain.
Top Domains Usage The amount of web requests per top domain.
Top Hostnames (by requests) The number of web requests grouped by hostname.
Top Hostnames (by size) The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames The number of flagged web request grouped by hostname.
Top Blocked Hostnames The number of blocked web request grouped by hostname.
Top Clients (by requests) The number of web requests grouped by client.
Top Clients (by size) The sum of the size of requested web content grouped by client.
Top Flagged Clients The number of flagged web request grouped by client.
Top Blocked Clients The number of blocked web request grouped by client.
Top Usernames (by requests) The number of web requests grouped by username.
Top Usernames (by size) The sum of the size of requested web content grouped by username.
Top Flagged Usernames The number of flagged web request grouped by username.
Top Blocked Usernames The number of blocked web request grouped by username.
Top Content (by request) The number of web requests grouped by category.
Top Content (by size) The sum of the size of requested web content grouped by category.
Top Searches The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches The number of flagged search queries grouped by term.
Top Blocked Searches The number of blocked search queries grouped by term.
All Web Events Shows all scanned web requests.
Flagged Web Events Shows all flagged web requests.
Blocked Web Events Shows all blocked web requests.
All HTTP Events Shows all scanned unencrypted HTTP requests.
All HTTPS Events Shows all encrypted HTTPS requests.
Unblocked Web Events Shows all unblocked web requests
All Search Events Shows all search queries processed by Web Filter.
Flagged Search Events Shows flagged search queries processed by Web Filter.
Blocked Search Events Shows blocked search queries processed by Web Filter.
The tables queried to render these reports:

Related Topics

Web Monitor

Web Monitor monitors HTTP and HTTPS traffic on your network to log web activities and flag inappropriate content.


About Web Monitor

  • Real-time classification and updates: When users visit a site, NG Firewall sends the URL to the Webroot BrightCloud® to be categorized. When the data is returned, the NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or See rule description. Users access the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is auto-categorized by our partners at Webroot and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention; plus, you can request https://www.brightcloud.com/tools/url-ip-lookup.php of sites.
  • HTTPS Filtering: Web Monitor has multiple techniques for dealing with HTTPS SSL-encrypted HTTP. Because HTTPS traffic is encrypted, only some information is visible, and this information is used to categorize the session. More information on how this works is below.
  • Detailed categorization: Web Monitor offers 79 categories and tens of billions of URLs. The Web Monitor database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope—maybe you want to flag websites related to nudity but allow sites dealing with sexual education.

Traffic Flow

When scanning traffic, Web Monitor evaluates the pass lists, flag lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the client computer receives the request and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of monitoring over both requested resources and content returned in response.

HTTP Request

When evaluating HTTP requests, Web Monitor applies the configured rules and lists in the following order:
  1. A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Monitor as well as other applications.
  2. The request's source IP is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  3. The request's destination site is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  4. The request's destination site is checked against the Flag Sites list. If a match is found, the traffic is flagged.
  5. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the matched rule.
  6. The category determined in step #1 is compared to the Categories list, and based on the corresponding match, traffic is allowed and possibly flagged. If the category cannot be determined, traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Monitor applies the configured rules and lists in the following order:
  1. The request's source IP is checked against the Pass Clients list. If a match is found, the traffic is allowed.
  2. The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
  3. The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the matched rule.

Settings

This section discusses the different settings and configuration options available for Web Monitor.

Status

This displays the current status and some statistics.

Figure 16. Web Monitor Status

Categories

Categories allow you to customize which categories of sites will be flagged. Flagged categories allow the user to access the site but will be silently flagged as a violation of event logs and Reports. These flag actions operate similarly for all the different Web Monitor options.

Figure 17. Web Monitor Categories

Flag Sites

Under Flag Sites, you can add individual domain names you want to be flagged - enter the domain name (e.g., youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

Figure 18. Web Monitor Flag Sites

Pass Sites

Pass Sites is used to pass content that would have otherwise been flagged. This can be useful for "unflagging" sites that you don't want to be flagged according to flag settings. Any domains you add to the Passed Sites list will be allowed, even if flagged by category or individual URL - add the domain and save. Unchecking the pass option will flag the site as if the entry was absent. This list uses URL Matcher syntax.

Figure 19. Web Monitor Pass Sites

Pass Clients

If you add an IP address to this list, Web Monitor will not flag any traffic from that IP regardless of the flagged categories or sites. Just add the IP and save. Unchecking the pass option will have the flag/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

Consider using pass lists if you have a few users who need to bypass Web Monitor controls completely. You should set up a separate rack using Policy Manager if users need different Web Monitor settings. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up a Static IP or a Static DHCP Lease for the machine in question.

Figure 20. Web Monitor Pass Clients

Rules

The Rules tab allows you to specify rules for flag traffic that passes through Web Monitor.

The Rules section describes how rules work and how they are configured. Web Monitor uses rules to determine when to flag specific sessions. Flagging a session marks it in the logs for review in the event logs or reports but has no direct effect on network traffic.

Figure 21. Web Monitor Rules

Rule Actions
  • Flag: Allows the traffic which matched the rule to flow, and flags the traffic for easier viewing in the event log.

Rule Types

In previous versions of Web Monitor, there were dedicated lists for flagging certain file extensions or MIME types. This capability is still available when using more flexible filter rules. For flagging specific file extensions, you can create a rule with the condition Web Filter: Response File Extension that has a comma-separated list of the extensions to flag in the Value field. For flagging MIME types, you would create a rule with the condition Web Filter: Response Content Type that has a comma-separated list of the content types to flag in the Value field.

Below are tables listing the default file extensions and MIME types available in previous versions. Note that these lists are not exhaustive but are included here to simplify the creation of such rules by copying and pasting the values in the tables.

Extension Category Description
exe executable an executable file format
ocx executable an executable file format
dll executable an executable file format
cab executable an ActiveX executable file format
bin executable an executable file format
com executable an executable file format
jpg image an image file format
png image an image file format
gif image an image file format
jar java a Java file format
class java a Java file format
swf flash the flash file format
mp3 audio an audio file format
wav audio an audio file format
wmf audio an audio file format
mpg video a video file format
mov video a video file format
avi video a video file format
hqx archive an archived file format
cpt compression a compressed file format
Content Category Description
application/octet-stream unspecified data byte stream
application/x-msdownload Microsoft download executable
application/exe executable executable
application/x-exe executable executable
application/dos-exe DOS executable executable
application/x-winexe Windows executable executable
application/msdos-windows MS-DOS executable executable
application/x-msdos-program MS-DOS program executable
application/x-oleobject Microsoft OLE Object executable
application/x-java-applet Java Applet executable
audio/mpegurl MPEG audio URLs audio
audio/x-mpegurl MPEG audio URLs audio
audio/mp3 MP3 audio audio
audio/x-mp3 MP3 audio audio
audio/mpeg MPEG audio audio
audio/mpg MPEG audio audio
audio/x-mpeg MPEG audio audio
audio/x-mpg MPEG audio audio
application/x-ogg Ogg Vorbis audio
audio/m4a MPEG 4 audio audio
audio/mp2 MP2 audio audio
audio/mp1 MP1 audio audio
application/ogg Ogg Vorbis audio
audio/wav Microsoft WAV audio
audio/x-wav Microsoft WAV audio
audio/x-pn-wav Microsoft WAV audio
audio/aac Advanced Audio Coding audio
audio/midi MIDI audio audio
audio/mpeg MPEG audio audio
audio/aiff AIFF audio audio
audio/x-aiff AIFF audio audio
audio/x-pn-aiff AIFF audio audio
audio/x-pn-windows-acm Windows ACM audio
audio/x-pn-windows-pcm Windows PCM audio
audio/basic 8-bit u-law PCM audio
audio/x-pn-au Sun audio audio
audio/3gpp 3GPP audio
audio/3gpp-encrypted encrypted 3GPP audio
audio/scpls streaming mp3 playlists audio
audio/x-scpls streaming mp3 playlists audio
application/smil SMIL audio
application/sdp Streaming Download Project audio
application/x-sdp Streaming Download Project audio
audio/amr AMR codec audio
audio/amr-encrypted AMR encrypted codec audio
audio/amr-wb AMR-WB codec audio
audio/amr-wb-encrypted AMR-WB encrypted codec audio
audio/x-rn-3gpp-amr 3GPP codec audio
audio/x-rn-3gpp-amr-encrypted 3GPP-AMR encrypted codec audio
audio/x-rn-3gpp-amr-wb 3gpp-AMR-WB codec audio
audio/x-rn-3gpp-amr-wb-encrypted 3gpp-AMR_WB encrypted codec audio
application/streamingmedia Streaming Media audio
video/mpeg MPEG video video
audio/x-ms-wma Windows Media video
video/quicktime QuickTime video
video/x-ms-asf Microsoft ASF video
video/x-msvideo Microsoft AVI video
video/x-sgi-mov SGI movie video
video/3gpp 3GPP video video
video/3gpp-encrypted 3GPP encrypted video video
video/3gpp2 3GPP2 video video
audio/x-realaudio RealAudio audio
text/vnd.rn-realtext RealText text
audio/vnd.rn-realaudio RealAudio audio
audio/x-pn-realaudio RealAudio plug-in audio
image/vnd.rn-realpix RealPix image
application/vnd.rn-realmedia RealMedia video
application/vnd.rn-realmedia-vbr RealMedia VBR video
application/vnd.rn-realmedia-secure secure RealMedia video
application/vnd.rn-realaudio-secure secure RealAudio audio
audio/x-realaudio-secure secure RealAudio audio
video/vnd.rn-realvideo-secure secure RealVideo video
video/vnd.rn-realvideo RealVideo video
application/vnd.rn-realsystem-rmj RealSystem media video
application/vnd.rn-realsystem-rmx RealSystem secure media video
audio/rn-mpeg MPEG audio audio
application/x-shockwave-flash Macromedia Shockwave multimedia
application/x-director Macromedia Shockwave multimedia
application/x-authorware-bin Macromedia Authorware binary multimedia
application/x-authorware-map Macromedia Authorware shocked file multimedia
application/x-authorware-seg Macromedia Authorware shocked packet multimedia
application/futuresplash Macromedia FutureSplash multimedia
application/zip ZIP archive
application/x-lzh LZH archive archive
image/gif Graphics Interchange Format image
image/png Portable Network Graphics image
image/jpeg JPEG image
image/bmp Microsoft BMP image
image/tiff Tagged Image File Format image
image/x-freehand Macromedia Freehand image
image/x-cmu-raster CMU Raster image
image/x-rgb RGB image image
text/css cascading style sheet text
text/html HTML text
text/plain plain text text
text/richtext rich text text
text/tab-separated-values tab separated values text
text/xml XML text
text/xsl XSL text
text/x-sgml SGML text
text/x-vcard vCard text
application/mac-binhex40 Macintosh BinHex archive
application/x-stuffit Macintosh Stuffit archive archive
application/macwriteii MacWrite Document document
application/applefile Macintosh File archive
application/mac-compactpro Macintosh Compact Pro archive
application/x-bzip2 block compressed compressed
application/x-shar shell archive archive
application/x-gtar gzipped tar archive archive
application/x-gzip gzip compressed compressed
application/x-tar 4.3BSD tar archive archive
application/x-ustar POSIX tar archive archive
application/x-cpio old cpio archive archive
application/x-bcpio POSIX cpio archive archive
application/x-sv4crc System V cpio with CRC archive
application/x-compress UNIX compressed compressed
application/x-sv4cpio System V cpio archive
application/x-sh UNIX shell script executable
application/x-csh UNIX csh script executable
application/x-tcl Tcl script executable
application/x-javascript JavaScript executable
application/x-excel Microsoft Excel document
application/mspowerpoint Microsoft Powerpoint document
application/msword Microsoft Word document
application/wordperfect5.1 Word Perfect document
application/rtf Rich Text Format document
application/pdf Adobe Acrobat document
application/postscript Postscript documen

Advanced

The Advanced section allows you to configure additional Web Monitor options.
  • Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream if present. More details in HTTPS Options.
  • Process HTTPS traffic by hostname in server certificate when SNI information is not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server, and the server name on the certificate will be used for categorization and filtering purposes.
  • Process HTTPS traffic by server IP if SNI and certificate hostname information are unavailable: If this option is enabled and neither of the previous options works, HTTPS traffic will be categorized using the IP address. More details in HTTPS Options.
  • Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache, all new web visits will be viewed fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.

HTTPS Options

There are many ways to handle HTTPS.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Monitor processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.
  • Process HTTPS traffic by SNI (Server Name Indication) if present.
  • Process HTTPS traffic by hostname in server certificate when SNI information not present
  • Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

If the SNI-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, it will see "wellsfargo.com" as the SNI information. If enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If "Banking" is flagged it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list. If "wellsfargo.com" is flaggeed it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list.

If No SNI information is present and Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled, then the hostname will be pulled from the certificate presented to the client.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

If no SNI or certificate information was available and Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available the session will be processed and categorized by IP address. If the IP-based processing and categorization of the web requests determines the session should be flagged, the session is reset and no more processing of this session will be done. If the IP-based processing and categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS process (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required.

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Monitor Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Monitor.

Reports

You can access the applications reports via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

You can search and defined Reports using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.

Pre-defined report queries:

Report Entry Description
Web Monitor Summary A summary of web monitor actions.
Web Usage The amount of total and flagged web requests over time.
Web Usage (scanned) The amount of total web requests over time.
Web Usage (flagged) The amount of flagged web requests over time.
Top Categories (by request) The number of web requests grouped by category.
Top Categories (by size) The sum of the size of requested web content grouped by category.
Top Flagged Categories The number of flagged web requests grouped by category.
Top Sites (by request) The number of web requests grouped by website.
Top Sites (by size) The sum of the size of requested web content grouped by website.
Top Flagged Sites The number of flagged web requests grouped by website.
Top Domains (by request) The number of web requests grouped by domain.
Top Domains (by size) The sum of the size of requested web content grouped by domain.
Top Flagged Domains The number of flagged web requests grouped by domain.
Top Domains Usage The amount of web requests per top domain.
Top Hostnames (by requests) The number of web requests grouped by hostname.
Top Hostnames (by size) The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames The number of flagged web request grouped by hostname.
Top Clients (by requests) The number of web requests grouped by client.
Top Clients (by size) The sum of the size of requested web content grouped by client.
Top Flagged Clients The number of flagged web request grouped by client.
Top Usernames (by requests) The number of web requests grouped by username.
Top Usernames (by size) The sum of the size of requested web content grouped by username.
Top Flagged Usernames The number of flagged web request grouped by username.
Top Content (by request) The number of web requests grouped by category.
Top Content (by size) The sum of the size of requested web content grouped by category.
Top Searches The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches The number of flagged search queries grouped by term.
All Web Events Shows all scanned web requests.
Flagged Web Events Shows all flagged web requests.
All HTTP Events Shows all scanned unencrypted HTTP requests.
All HTTPS Events Shows all encrypted HTTPS requests.
All Search Events Shows all search queries processed by Web Monitor.
Flagged Search Events Shows flagged search queries processed by Web Monitor.
The tables queried to render these reports:

Related Topics