NG Firewall Filter Apps

Contents

Ad Blocker
Application Control
Application Control Lite
SSL Inspector
Spam Blocker
Web Filter
Web Monitor

Ad Blocker

Ad Blocker allows you to block a majority of advertising content that is delivered to users on web pages that they request.

Ad Blocker uses downloadable filter subscriptions from a variety of sources which contain lists of web sites and extensions that are typically used to deliver advertising.

Warning: Ad Blocker is similar in nature to browser plugins that block ads (like AdBlockPlus, Ghostery). Like those plugins, sometimes blocking ads and tracking can interfere with the proper functioning of online websites or media. However, Ad Blocker blocks at the gateway the browser user has no visual indication that something has been blocked nor have any way to temporarily disable or bypass the blocking if it interferes with the proper functioning of online services. As such, Ad Blocker can cause issues and the browser user has no recourse. Running Ad Blocker can impose administrative overhead dealing with issues and is ideal for some environments (small sites, home, enthusiasts, etc) but not recommended in others like schools, businesses, and large sites.

Warning: With the increasing adoption of SSL ad blocker can do very little without SSL inspection, but running SSL inspection is not ideal for many organizations.

Warning: When the update button is pressed it pulls new signatures directly from 3rd party sources. We have not tested any future updates and can not guarantee any future updates will work correctly. Updates may interfere with the proper functioning of websites and potentially cause massive problems.

Warning: Unlike other apps, Ad Blocker is off by default after installation. Do not enable Ad Blocker until reading the above warnings.

Settings

This section reviews the different settings and configuration options available for Ad Blocker.

Status

This displays the current status and some statistics.

Options

If Block Ads is enabled, then Ad Blocker will block web requests that it determines are for advertisements. It will return an HTTP 403 permission denied to the client that requests the ad.

If Block Tracking & Ad Cookies is enabled Ad Blocker will block cookies that are used for advertising or behavior tracking purposes.

The Update button will update the ad filters signatures.

Warning: When the update button is pressed, it pulls new signatures directly from 3rd party sources. We have not tested any future updates and can not guarantee any future updates will work correctly. Updates may interfere with the proper functioning of websites and potentially cause massive problems.

Ad Filters

Ad Blocker's Standard Filters list will populate with many entries to match common ad serving strings. This list can not be modified, but you can enable or disable the rules here. You are free to add to and edit rules in the User Defined Filters tab, as you see fit - simply click Add and enter a description to match. Be careful when selecting blocking criteria as you may block much more content than planned if your criteria is not carefully specified.

Cookie Filters

The Standard Cookie Filters list is populated with entries to match common cookie domains. This list can not be modified, but you can enable or disable the rules here. You can add to and edit rules in the User Defined Cookie Filters tab as you see fit.

Pass Lists

Pass Lists are used to pass content that would have otherwise been blocked. This can be useful for "unblocking" sites that you don't want blocked or allowing certain users special privileges.

Passed Sites: Any domains you add to the Passed Sites list will allow ads and cookies, even if blocked by an existing filter - just add the domain and save. Unchecking the pass option will allow ads to be blocked as if the entry was not present.
Passed Client IPs: If you add an IP to this list, Ad Blocker will not block any ads or cookies from that IP. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list.

Reports

The Reports tab provides a view of all reports and events for all traffic handled by Ad Blocker.

This applications reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Reports can be searched and further defined using the time selectors and the Conditions window at the bottom of the page. The data used in the report can be obtained on the Current Data window on the right.


Pre-defined report queries: report Entry	Description
Ad Blocker Summary	A summary of ad blocker actions.
Ads Blocked	The amount of detected and blocked ads over time.
Top Blocked Ad Sites	The number of blocked ads grouped by website.
All Ad Events	All HTTP requests are scanned by Ad Blocker.
Blocked Ad Events	HTTP requests blocked by Ad Blocker.
Blocked Cookie Events	Requests blocked by cookie filters.

The tables queried to render these reports:

Database Schema

Ad Blocker Reports

The Reports tab provides a view of all reports and events for all traffic handled by Ad Blocker.

Reports

This application's reports can be accessed via the Reports tab at the top or the Reports tab within the settings. All pre-defined reports will be listed along with any custom reports that have been created.

Pre-defined report queries:


Report Entry	Description
Ad Blocker Summary	A summary of ad blocker actions.
Ads Blocked	The amount of detected and blocked ads over time.
Top Blocked Ad Sites	The number of blocked ads grouped by website.
All Ad Events	All HTTP requests are scanned by Ad Blocker.
Blocked Ad Events	HTTP requests blocked by Ad Blocker.
Blocked Cookie Events	Requests blocked by cookie filters.

The tables queried to render these reports:

Events

Application Control

Application Control leverages the Network Application Visibility Library (NAVL) from Procera Networks [1] to perform deep packet (DPI) and deep flow (DFI) inspection of network traffic. This allows the server to accurately identify thousands of today's common applications such as Social Networking, P2P, Instant Messaging, Video Streaming, File Sharing, Enterprise Applications, Web 2.0, and much more.

About Application Control

For most common applications, you can simply go to the list on the Applications tab and check Block for anything you want to stop. Then, Application Control will take care of the rest. If you need a higher degree of control, you can use the Rules tab to create custom rules that target more complex traffic patterns.

How It Works

Application Control feeds each chunk of data to a classification engine as it passes through the application. The classification engine continues to analyze the traffic flow and keeps properties of the session, such as the Application property. Each time the classification of the Application property is updated, the Applications settings are checked to see if that application is allowed. The data is blocked If the application is configured to be blocked in the settings. If not, the process continues until the session reaches a fully classified state, where the classification engine believes no more classification of the session is possible. At this point, the Rules are evaluated, and the session is ultimately blocked or passed based on the rules you've configured.

Settings

This section describes the different settings and configuration options available for Application Control.

Status

The Status tab displays a summary of traffic and configuration information. The Traffic Statistics section displays the total number of sessions that have been scanned, along with the number of those sessions that were allowed, flagged, or blocked. The Application Statistics section shows you the total number of applications that can be detected by the application, along with the number of those protocols that will be flagged and/or blocked. Rules Statistics allows you to quickly see how many custom rules you have configured, as well as how many of those rules are active.

Applications

The Applications tab is the primary and preferred way for using Application Control to manage network traffic. Simply find the application you want to target and use the block and flag checkboxes as appropriate. You can sort the list on any of the columns displayed, which should help you find and manage the protocols you want to target. Simply check Block to stop these applications or Flag to allow them, silently filing them as violations in the Reports. Use the following definitions to set up the Applications tab for your organization:

Application: The unique identifier for the application.
Block: Enable this checkbox to block/reset sessions of this application. For TCP, this will actively reset the connection. For UDP, the packet will be dropped, and the session will be killed.
Tarpit: Enable this checkbox to block/tarpit sessions of this application. For TCP, this makes it appear to both the client and the server that the other party is receiving the data, but it is not responsive. It silently drops the data. For UDP, it is identical in behavior to block, except the connection is kept open, so that the next packet will be dropped instead of recategorized as a new session.
Flag: Enable the checkbox to flag the traffic. It will be flagged as a violation in Reports.
Name: The standard name for the application.
Category: A fairly general and high level category for the application.
Productivity: Productivity is best thought of as an index value between 1 and 5 that rates the potential for each application to improve or increase the overall productivity of your network users, assuming, of course, that listening to music and playing online games is not in their job description. So, applications with a low Productivity index (e.g. MySpace, Hulu, Zynga Games) can be expected to have a negative impact on productivity. Items with a high value (e.g. Active Directory, Network File System) can generally be considered critical for maintaining or improving productivity.
Risk: Risk is another index value between 1 and 5 that rates the potential for each protocol or application to allow nasty stuff onto your network. The higher the risk index, the greater the chance of letting in something that could be dangerous or destructive. So low risk items (e.g. Active Directory, Oracle, LDAP) are generally no cause for concern, while applications rated with a high risk (e.g. BitTorrent, Pando, Usenet) increase the possibility you'll find yourself spending long nights deleting pirated software and cleaning up viruses and other exploits that find their way into your infrastructure.
Description: Provides a more detailed description of each application on the list. In some cases the description is much larger than will fit within the grid column, so you can click on any description to see a pop-up window with the full text displayed.

Figure 2. Application Control Applications

Rules

If the traffic you need to manage can't be handled via the Applications tab, you can create custom rules that will allow you to analyze and control traffic based on much more complex patterns and conditions. For each session, the rules are only evaluated once after the classification engine has completed an analysis of the traffic. The rules are then evaluated in order until the first match is found; at this point, the configured action will be performed. If there are no matches, the session will be tagged as allowed, the traffic will flow unimpeded, and no further analysis of that traffic will occur.

Important: These rules are evaluated when the classification engine has completed all analyses. This usually occurs after a few packets have passed. This means the rules are useful because enough has been learned about the session that is not known at the session creation time to have powerful rules, such as HTTP information or protocol/application information. If the full classification is not completed after 15 chunks of data, then the rules are evaluated given the current information.

If an application is blocked or tarpitted in the Applications tab, it will be blocked immediately when identified before the engine has completed the analysis. In this case, the rules will have NO EFFECT because the sessions are blocked before the rules are evaluated.

Application Control Rules provide a very powerful feature that is used to control application usage. However, understanding how and when the rules are evaluated is essential in their usage.

Anatomy of a Rule

An Application Control Rule is a standard rule documented in the Rules documentation. We'll use one of the default rule entries for Ultrasurf to help explain how Rules work. This is exactly the kind of traffic that the Rules engine was created to seek and destroy. For this particular rule, the objective is to block all traffic that: a) uses port 443, b) looks like valid HTTPS traffic, and c) doesn't use a valid SSL certificate. To accomplish this, we created four matchers:

The first matcher makes sure the rule only looks at TCP traffic.
The second causes the rule to only look at traffic with a destination port 443.
The third matcher is where the real magic starts. In this case, we created a Glob matcher that looks for the /SSL tag anywhere in the Application Control/ProtoChain. (Don't worry, we'll cover globs and chains below!)
The fourth matcher is the frosting on the cake. We tell the rule to look at the Application Control/Detail parameter. This is where the server name from the SSL certificate will be located when an SSL encrypted session is detected. In this case, we left the Value field empty since we're looking for cases where there is no valid certificate.

Application Detail

The Detail field will contain different types of [#Is there a list of session properties? | information] depending on the protocols detected during session classification. For matcher conditions other than those listed below, the Detail field will be empty.


Matcher	Detail Contents	Example
Application: FBOOKAPP	The name of the Facebook Application that is being accessed.	wordswithfriends
Application: HTTP	The contents of the Content-Type header in the session data coming from the server.	image/jpg
ProtoChain: /SSL	The server name extracted from the SSL certificate used to encrypt the session.	www.gmail.com

Actions

Allow: Allow the traffic.
Block: When this option is selected, traffic in both directions will be silently dropped, but the session will remain active.

Reports

The Reports tab provides a view of all reports and events for all traffic handled by Application Control.

Pre-defined report queries:


Report Entry	Description
Application Control Summary	A summary of Application Control actions.
Top Applications Usage	The amount of bandwidth per top application.
Scanned Sessions (all)	The amount of scanned, flagged, and blocked sessions over time.
Scanned Sessions (flagged)	The amount of flagged, and blocked sessions over time.
Scanned Sessions (blocked)	The amount of flagged, and blocked sessions over time.
Top Categories (by sessions)	The number of sessions grouped by category.
Top Applications (by sessions)	The number of sessions grouped by application.
Top Applications (by size)	The number of bytes grouped by application.
Top Flagged Applications	The number of flagged sessions grouped by application.
Top Blocked Applications	The number of blocked sessions grouped by application.
Top Flagged Hostnames	The number of flagged sessions grouped by hostname.
Top Blocked Hostnames	The number of blocked sessions grouped by hostname.
Top Flagged Clients	The number of flagged sessions grouped by client.
Top Blocked Clients	The number of blocked sessions grouped by client.
Top Flagged Usernames	The number of flagged sessions grouped by username.
Top Blocked Usernames	The number of blocked sessions grouped by username.
Classified Sessions	All sessions matching an application control signature.
Flagged Sessions	All sessions matching an application control signature and flagged.
Blocked Sessions	All sessions matching an application control signature and blocked.
All Sessions	All sessions are scanned by Application Control.

The tables queried to render the following reports:

Application Control Reports

The Reports tab provides a view of all reports and events for all traffic handled by Application Control.

Reports

Pre-defined report queries:


Report Entry	Description
Application Control Summary	A summary of Application Control actions.
Top Applications Usage	The amount of bandwidth per top application.
Scanned Sessions (all)	The amount of scanned, flagged, and blocked sessions over time.
Scanned Sessions (flagged)	The amount of flagged, and blocked sessions over time.
Scanned Sessions (blocked)	The amount of flagged, and blocked sessions over time.
Top Categories (by sessions)	The number of sessions grouped by category.
Top Applications (by sessions)	The number of sessions grouped by application.
Top Applications (by size)	The number of bytes grouped by application.
Top Flagged Applications	The number of flagged sessions grouped by application.
Top Blocked Applications	The number of blocked sessions grouped by application.
Top Flagged Hostnames	The number of flagged sessions grouped by hostname.
Top Blocked Hostnames	The number of blocked sessions grouped by hostname.
Top Flagged Clients	The number of flagged sessions grouped by client.
Top Blocked Clients	The number of blocked sessions grouped by client.
Top Flagged Usernames	The number of flagged sessions grouped by username.
Top Blocked Usernames	The number of blocked sessions grouped by username.
Classified Sessions	All sessions matching an application control signature.
Flagged Sessions	All sessions matching an application control signature and flagged.
Blocked Sessions	All sessions matching an application control signature and blocked.
All Sessions	All sessions scanned by Application Control.

The tables queried to render these reports:

Application Control Lite

Application Control Lite scans sessions for the specified signatures and will log and/or block sessions based on their content. Many applications can be hard to block based on port, as modern applications will find and use open ports automatically. Application Control Lite provides a basic functionality to block sessions based on the content (data) in the session.

Settings

This section reviews the different settings and configuration options available for Application Control Lite.

Status

The Status tab will show you current information on signatures Available, Logged and Blocked.

Signatures

The Signatures tab shows the list of current signatures. Signatures are regular expressions written to match known protocols as accurately as possible. New custom signatures can be designed to match certain applications or sessions, or signatures can often be found for many existing protocols L7-filter Supported Protocols.

As the early data in each session goes from the server to the client and the client to the server, it is stored in a buffer. As each chunk of data arrives, the data is evaluated against any enabled signatures. If the signature is checked "log" then the session will be tagged and logged as having matched the specified signature. If the signature is checked "block" it will be logged, and the session will immediately be closed.

Writing custom signatures can be dangerous and difficult. Usually, one of several outcomes will happen when writing a block signature:

It will not match anything. In this case, the signature needs to be fixed.
It will block the desired protocol/application and nothing else. This is ideal.
It will only partially block the protocol. Many multi-session protocols only have some sessions identified. This can have varying effects depending on the application.
It will block the protocol and block other things too (false positives). This can cause major problems with the network.
It will block the protocol, and the application will adapt and use an alternative protocol to communicate. Many applications will try alternative techniques to avoid blocking.

As such, great care and expertise are involved in writing signatures to achieve the desired effect.

Note: Application Control Lite, while powerful, can be difficult, time consuming, and dangerous to configure correctly. Application Control is recommended for most users as it comes preloaded with hundreds of maintained and current behavioral signatures and a commercial third-party application identification engine.

Warning:In previous and older versions of NG Firewall, there was a default signature set. However, enabling block on some of the default signatures caused false positives and blocked legitimate network traffic. Despite big warnings in the user interface and help documentation, we found users often misconfigured Application Control Lite anyway and experienced network problems. To avoid this issue, there are now no default signatures. If you have read this warning and understand that misconfiguring Application Control Lite will cause major network connectivity issues, then you can download the original list of signatures

here and import them into Signatures.

Reporting

The Reports tab provides a view of all reports and events for all traffic handled by Application Control Lite.

Reports

Pre-defined report queries:


Report Entry	Description
Application Control Lite Summary	A summary of Application Control Lite actions.
Detection Statistics	The number of logged and blocked sessions over time.
Top Blocked Protocols	The top blocked sessions by protocol.
Top Logged Protocols	The top logged sessions by protocol.
Top Blocked Hosts	The top blocked sessions by host.
Top Logged Hosts	The top logged sessions by host.
Top Blocked Users	The top blocked sessions by user.
Top Logged Users	The top logged sessions by user.
All Events	All sessions scanned by Application Control Lite.
Blocked Events	All sessions matching an application signature and blocked.

The tables queried to render these reports:

Application Control Lite Reports

The Reports tab provides a view of all reports and events for all traffic handled by Application Control Lite.

Reports

Pre-defined report queries:


Report Entry	Description
Application Control Lite Summary	A summary of Application Control Lite actions.
Detection Statistics	The number of logged and blocked sessions over time.
Top Blocked Protocols	The top blocked sessions by protocol.
Top Logged Protocols	The top logged sessions by protocol.
Top Blocked Hosts	The top blocked sessions by host.
Top Logged Hosts	The top logged sessions by host.
Top Blocked Users	The top blocked sessions by user.
Top Logged Users	The top logged sessions by user.
All Events	All sessions are scanned by Application Control Lite.
Blocked Events	All sessions matching an application signature and blocked.

The tables queried to render these reports:

SSL Inspector

The SSL Inspector is a special application that allows other NG Firewall applications that process HTTP traffic to also process encrypted HTTPS traffic and applications that process SMTP to also process SMTP over SSL. It does this by performing man-in-the-middle decryption and encryption of SSL traffic, passing the unencrypted traffic through the NG Firewall server for inspection by other applications and services.

About SSL Inspector

When a client makes an HTTPS request, the Inspector first initiates a secure SSL connection with the external server on behalf of the client. While this session is being established, the inspector captures information about the server SSL certificate. Once the server session is active, the Inspector uses the details from the server certificate to create a new certificate that will be used to encrypt the session between the inspector and the client. This certificate is generated or loaded on the fly, and is created using the same subject details contained in the actual server certificate. The certificate is then signed by the internal CA on the NG Firewall server and is used to establish a secure connection between the inspector and the client. Creating the certificate this way is necessary to eliminate security warnings on the client, but it does require a few extra steps to properly configure the client computers and devices on your network. See the SSL Certificates section below for details.

SSL Certificates

SSL Certificates serve two primary purposes. They allow traffic between the client and server to be encrypted, and they allow the client to validate the authenticity of the server. There are two main ways the client checks the authenticity of the server certificate. The first is by validating the server certificate to ensure it has been issued or signed by a known and trusted third party certificate authority. Once that trust has been established, the client checks the server name portion of the target URL to ensure it matches the server name registered in the certificate presented by the server. If either of these checks fail, the client will typically display a warning, indicating that the security of the connection may be compromised.

When the NG Firewall server is initially installed, a default Certificate Authority is created automatically and used to sign the man-in-the-middle certificates created by the SSL Inspector. To view or make changes to the internal Certificate Authority, check out Certificates tab of the Config/Administration page.

Config > Administration > Certificates

Client Configuration

For the client authenticity checks to be successful, the client must be configured to trust the root certificate used by the NG Firewall server to sign the man-in-the-middle certificates described above. To configure clients, you must first use the Download Root Certificate button located on the Configuration tab of SSL Inspector Settings page to download the root certificate. You must then install this certificate to correct the location on the client.

Another way to download the root certificate is to simply access a special URL using the IP address of the NG Firewall server:

http://yourserver/cert

Simply replace 0.0.0.0 with the IP address of your NG Firewall server. This method is especially useful when using mobile devices. For example, accessing this URL on an iPad or iPhone will download and display the certificate, and provide an option to install and trust the certificate directly on the device.

Below are basic instructions for installing the root certificate on some common client platforms. If yours is not listed, or you have any difficulty, consult the reference material for the target platform for further information.

Internet Explorer or Google Chrome on Microsoft Windows

Follow the below steps:

Log into the NG Firewall server running SSL Inspector.
Go to Config > Administration > Certificates and download the certificate using the "Download Root Certificate Authority (CA)" button.
Copy the root_authority.crt you just downloaded to the Windows client computer.
From a command prompt, or from Start/Run, run the command "certmgr.msc".
Open the "Trusted Root Certification Authorities" tree in the panel on the left.
Right click on "Certificates" and select All Tasks > Import .
Proceed with the Certificate Import Wizard, selecting the the root_authority.crt file.

Firefox on Microsoft Windows

Follow the below steps:

Log into the NG Firewall server running SSL Inspector.
Go to Config > Administration > Certificates and download the certificate using the "Download Root Certificate Authority (CA)" button.
Copy the root_authority.crt you just downloaded to the Windows client computer.
Launch Firefox
From the Tools menu, go to Options > Privacy & Security .
Click the Import button and select the root_authority.crt file.
Enable the "Trust this CA to identify websites" checkbox and click the OK button.

Opera on Microsoft Windows

Follow the below steps:

Log into the NG Firewall server running SSL Inspector.
Go to Config > Administration > Certificates and download the certificate using the Download Root Certificate Authority (CA) button.
Copy the root_authority.crt you just downloaded to the Windows client computer.
Launch Opera
From the Tools menu, go to Preferences > Advanced > Security and click Manage Certificates.
Select the Authorities tab, click Import, and select the root_authority.crt file.
Click Install and click OK when asked if you want to trust the certificate.

Group Policy Distribution

If you have a fully deployed and implemented Active Directory infrastructure, you can leverage the Group Policy model to distribute the NG Firewall root certificate to all of your client computers. This is way outside our area of expertise, so we can't provide much help or assistance, but we have compiled links to some TechNet articles with instructions for several common versions of Windows Server.

Windows Server 2003

Windows Server 2008

Windows Server 2012

Settings

This section describes the different settings and configuration options available for SSL Inspector.

Status

This displays the current status and some statistics.

Configuration

Execute the following configurations:

Download Root Certificate

As described above, client computers and devices on your network need to be configured to trust the root certificate of the NG Firewall server. Clicking this button will allow you to download the root certificate. Once downloaded, you need to install it in the Trusted Authorities certificate store on your client computers and devices. Note that this is the same root certificate that can be downloaded from the Config > Administration > Certificates page. The download link is included on the SSL Inspector Configuration page for convenience.

Alternatively, you can download the certificate from a client system by navigating to http://yourserver/cert.

Enable SMTPS Traffic Processing

This option is enabled by default and allows the SSL Inspector to work cooperatively with the other applications that act on SMTP mail traffic. When enabled, port 25 mail sessions that use STARTTLS will be decrypted inbound, allowing the clear traffic to pass through all other applications, and the re-encrypted again before passing outbound.

Enable HTTPS Traffic Processing

This option is enabled by default, and allows the SSL Inspector to work cooperatively with the other applications that act on HTTP web traffic. When enabled, port 443 web sessions that use SSL/TLS will be decrypted inbound, allowing the clear traffic to pass through all other applications, and then re-encrypted again before passing outbound.

Block Invalid HTTPS Traffic

When processing a new HTTPS session, the first thing the inspector does is analyze the initial client request to see if it contains a valid SSL negotiation message. If not, by default the session will be ignored, and the traffic will flow directly between the client and server with no inspection performed. By enabling this checkbox, you can change the default behavior and effectively block any port 443 traffic that does not contain a valid HTTPS signature.

Client/Server Connection Protocols

This section includes checkboxes for turning on and off the SSL and TLS protocols that will be used when negotiating secure HTTPS and SMTPS inbound and outbound connections. The client protocols are used when the server is communicating with a client. The server protocols are used when the server is communicating with a server.

SSLv2Hello - This is really a legacy handshake protocol that is used between a client and server when deciding which encryption protocol to use. This means it's possible to enable SSLv2Hello and still have a TLSv1.x connection negotiated. While there are no known security issues, we still recommend leaving this disabled unless you specifically need this legacy support.
SSLv3 - This is an older protocol that is been deprecated since the discovery of the POODLE vulnerability. For that reason, we recommend this be disabled for maximum security.
TLSv1 - This is an older protocol that has some known weaknesses. These can be mitigated if the other side of the connection forces certain secure ciphers to be used. However, since this can't be guaranteed, best practice is to disable this protocol unless it is required to support connections with legacy clients or servers.
TLSv1.1 - This is a modern protocol that is generally regarded as secure and is used as a fallback for 1.2 or in older browsers.
TLSv1.2 - This is the most common and recommended TLS version.
TLSv1.3- This is the most recent version of the TLS protocol and offers the highest security but may have issues with some websites.

Trust All Server Certificates

Normally, when establishing an SSL connection with an external web server, the inspector will authenticate the server certificate against a standard list of trusted certificate authorities. If this trust cannot be established, the inspector will end the session. By enabling this checkbox, you can force the inspector to blindly trust all external server certificates.

Please note that we DO NOT recommend running with this option enabled, as it exposes all HTTPS traffic to significant security risks.

The standard list of trusted certificates used by NG Firewall is generated from the standard ca-certificates package. It includes, among others, certificate authorities used by Mozilla's browsers. Please note that Edge Threat Management staff can neither confirm nor deny whether the certificate authorities whose certificates are included in this list have in any way been audited for trustworthiness or RFC 3647 compliance. Full responsibility to assess them belongs to the local system administrator.

Upload Trusted Certificate

Note:This setting applies to all policies when using SSL Inspector with Policy Manager.

The inspector emulates a web browser when it makes outbound connections to external web servers. Just like a web browser, it must verify the authenticity of the server certificate before it will trust the connection and allow traffic to flow freely. As mentioned above, the inspector uses a standard list of known certificate authorities to validate server certificates. However, it's also possible you have servers in your network that use certificates that can't be authenticated this way. Perhaps you have yourcertificate authority, or use self-signed certificates. Whatever the reason, you can use this section of the configuration page to upload additional certificates you want the inspector to trust.

Rules

The Rules tab allows you to specify explicit rules to Inspect or Ignore HTTPS traffic that crosses the NG Firewall. By default, many common HTTPS sites (Google, YouTube, Yahoo, and so on) are inspected, but not all HTTPS. This provides a safe default which provides HTTPS inspection on those sites without interfering with other HTTPS communications. It can easily be configured to inspect all HTTPS by enabling the "Inspect All Traffic" rule.

The Rules describes how rules work and how they are configured. SSL Inspector uses rules to determine if it should inspect or ignore traffic for the specific session.

In addition to all the common rule types, there are three that are unique to the SSL Inspector, and these can be very useful for ignoring traffic that you don't want to inspect or that isn't compatible with the SSL Inspector.

HTTPS: SNI Hostname

Most web browsers and many client applications include the destination hostname in the initial packet of an HTTPS session. The mechanism is called the Server Name Indication or the SNI extension to the TLS protocol. The main purpose is to allow a single web server to host multiple secure web sites. By analyzing the SNI hostname in the client request, the server can decide which SSL certificate to use for encrypting the session. This extension is necessary because the encryption must be established long before the server ever sees the HTTP request, and by then, it would be too late to use a different certificate.

Creating ignore rules based on the SNI hostname is an effective way to have the SSL Inspector ignore incompatible traffic. A prime example is the default rule for Microsoft Update. The Microsoft Update client checks the server certificate to ensure it was signed by a specific authority. Since it doesn't trust the Root Authority the SSL Inspector uses to generate certificates on-the-fly, Microsoft Update will fail with an error. The default rule allows this traffic to be detected and ignored, allowing Microsoft Update to work properly.

HTTPS: Certificate Subject and HTTPS: Certificate Issuer

These two rule conditions are useful when dealing with client applications that don't use SNI and aren't compatible with SSL Inspector. An excellent example is the Dropbox client utility, for which there is also a default rule. Like Microsoft Update, the Dropbox client will reject SSL certificates it doesn't explicitly trust.

Using either of these rule conditions, you can match traffic on any portion of the Subject or Issuer Distinguished Name (DN) included in the server certificate. In both cases, the information in the match string includes the standard information fields commonly stored within the SSL certificates, such as CN (common name), C (country), ST (state), L (locality), O (organization), and OU (organizational unit). Each of these is appended to the match string and separated by commas. Note that not all fields are required in all certificates, and some certificates may have others not listed. The order they occur in the match string is also not guaranteed.

The Subject DN generally includes information about the company to which the certificate was issued. Here is an example Certificate Subject:

CN=*.dropbox.com, O="Dropbox, Inc.", L=San Francisco, ST=California, C=US

The Issuer DN generally includes information about the company that issued and authenticated the certificate. Here is an example Certificate Issuer:

CN=Thawte SSL CA, O="Thawte, Inc.", C=US

Rule Actions

Inspect: Causes the traffic which matched the rule to be decrypted and passed along to other applications and service for further inspection, classification, and possible action.
Ignore: Causes the traffic which matched the rule to be ignored by the SSL Inspector.

Figure 6. SSL Inspector Rules

Reports

The Reports tab provides a view of all reports and events for all traffic handled by HTTPS Inspector.

Pre-defined report queries:


Report Entry	Description
SSL Inspector Summary	A summary of SSL Inspector actions.
Sessions Scanned	The amount of SSL sessions over time.
Sessions Inspected	The amount of inspected SSL sessions over time.
Top Inspected Sites	The number of inspected sessions grouped by site.
Top Ignored Sites	The number of ignored sessions grouped by site.
All Sessions	All sessions detected by SSL Inspector.
Inspected Sessions	Events where traffic was fully processed by the inspector, and all traffic was passed through all the other applications and services.
Ignored Sessions	Events where traffic was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
Blocked Sessions	Events where traffic was blocked because it did not contain a valid SSL request and the Block Invalid Traffic option was enabled.
Untrusted Sessions	Events where traffic was blocked because the server certificate could not be authenticated.
Abandoned Sessions	Events where traffic was blocked due to an underlying problems with the SSL session.

The tables queried to render these reports:

Database Schema

Status

The status of the session that generated the event.

INSPECTED means the session was fully processed by the inspector, and all traffic was passed through all the other applications and services.
IGNORED means the session was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
BLOCKED means the traffic was blocked because it did not contain a valid HTTPS request, and the Block Invalid Traffic option was enabled.
UNTRUSTED means the traffic was blocked because the server certificate could not be authenticated.
ABANDONED means the connection failed because an an underlying SSL connection problem. Usually that the client abandoned the connection because the certificate was not trusted.

Detail

Extra details about the session, with the exact content dependent on the event status.

For INSPECTED and UNTRUSTED sessions, this field will include the SNI hostname extracted from the initial message sent from the client to the server. If the SNI information is not available, the server IP address will be used instead.

For BLOCKED or IGNORED sessions, this field will contain the description of the rule that matched and was applied to the session.

For ABANDONED sessions, detail will usually record information about the error that caused inspection to fail. For SSL exceptions, this will include Client or Server to indicate the session endpoint for which traffic was being processed. It will also include Encrypt or Decrypt to indicate the state of traffic inspection when the exception occurred. If available, the SSL error message will also be included. The following table lists the most common error messages and detailed information about each one.

Table 1. SSL Exception Messages
SSL Exception Message	Description
unexpected_message	An inappropriate message was received. This alert is always fatal and should never be observed in communication between proper implementations.
bad_record_mac	This alert is returned if a record is received with an incorrect MAC. This alert also MUST be returned if an alert is sent because a `TLSCiphertext` decrypted in an invalid way: either it wasn't an even multiple of the block length, or its padding values, when checked, weren't correct. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decryption_failed	This alert was used in some earlier versions of TLS, and may have permitted certain attacks against the CBC mode [CBCATT]. It MUST NOT be sent by compliant implementations.
record_overflow	A TLSCiphertext record was received that had a length more than 2^14+2048 bytes, or a record decrypted to a `TLSCompressed` record with more than 2^14+1024 bytes. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decompression_failure	The decompression function received improper input (e.g., data that would expand to excessive length). This message is always fatal and should never be observed in communication between proper implementations.
handshake_failure	Reception of a `handshake_failure` alert message indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. This is a fatal error.
no_certificate	This alert was used in SSLv3 but not any version of TLS. It MUST NOT be sent by compliant implementations.
bad_certificate	A certificate was corrupt, contained signatures that did not verify correctly, etc.
unsupported_certificate	A certificate was of an unsupported type.
certificate_revoked	A certificate was revoked by its signer.
certificate_expired	A certificate has expired or is not currently valid.
certificate_unknown	Some other (unspecified) issue arose in processing the certificate, rendering it unacceptable.
illegal_parameter	A field in the handshake was out of range or inconsistent with other fields. This message is always fatal.
unknown_ca	A valid certificate chain or partial chain was received, but the certificate was not accepted because the CA certificate could not be located or couldn't be matched with a known, trusted CA. This message is always fatal.
access_denied	A valid certificate was received, but when access control was applied, the sender decided not to proceed with negotiation. This message is always fatal.
decode_error	A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decrypt_error	A handshake cryptographic operation failed, including being unable to correctly verify a signature or validate a Finished message. This message is always fatal.
export_restriction	This alert was used in some earlier versions of TLS. It MUST NOT be sent by compliant implementations.
protocol_version	The protocol version the client has attempted to negotiate is recognized but not supported. (For example, old protocol versions might be avoided for security reasons.) This message is always fatal.
insufficient_security	Returned instead of `handshake_failure` when a negotiation has failed specifically because the server requires ciphers more secure than those supported by the client. This message is always fatal.
internal_error	An internal error unrelated to the peer or the correctness of the protocol (such as a memory allocation failure) makes it impossible to continue. This message is always fatal.
user_canceled	This handshake is being canceled for some reason unrelated to a protocol failure. If the user cancels an operation after the handshake is complete, just closing the connection by sending a `close_notify` is more appropriate. This alert should be followed by a `close_notify`. This message is generally a warning.
no_renegotiation	Sent by the client in response to a hello request or by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation; when that is not appropriate, the recipient should respond with this alert. At that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate is where a server has spawned a process to satisfy a request; the process might receive security parameters (key length, authentication, etc.) at startup, and it might be difficult to communicate changes to these parameters after that point. This message is always a warning.
unsupported_extension	sent by clients that receive an extended server hello containing an extension that they did not put in the corresponding client hello. This message is always fatal.

SSL Inspector Reports

The Reports tab provides a view of all reports and events for all traffic handled by HTTPS Inspector.

Reports

Pre-defined report queries:


Report Entry	Description
SSL Inspector Summary	A summary of SSL Inspector actions.
Sessions Scanned	The amount of SSL sessions over time.
Sessions Inspected	The amount of inspected SSL sessions over time.
Top Inspected Sites	The number of inspected sessions grouped by site.
Top Ignored Sites	The number of ignored sessions grouped by site.
All Sessions	All sessions detected by SSL Inspector.
Inspected Sessions	Events where traffic was fully processed by the inspector, and all traffic was passed through all the other applications and services.
Ignored Sessions	Events where traffic was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
Blocked Sessions	Events where traffic was blocked because it did not contain a valid SSL request and the Block Invalid Traffic option was enabled.
Untrusted Sessions	Events where traffic was blocked because the server certificate could not be authenticated.
Abandoned Sessions	Events where traffic was blocked due to an underlying problems with the SSL session.

The tables queried to render these reports:

Database Schema

Status

The status of the session that generated the event.

INSPECTED means the session was fully processed by the inspector, and all traffic was passed through all the other applications and services.
IGNORED means the session was not or could not be inspected, so the traffic was completely ignored and not analyzed by any applications or services.
BLOCKED means the traffic was blocked because it did not contain a valid HTTPS request, and the Block Invalid Traffic option was enabled.
UNTRUSTED means the traffic was blocked because the server certificate could not be authenticated.
ABANDONED means the connection failed because an an underlying SSL connection problem. Usually, the client abandoned the connection because the certificate was not trusted.

Detail

Extra details about the session, with the exact content dependent on the event status.

For BLOCKED or IGNORED sessions, this field will contain the description of the rule that matched and was applied to the session.

For ABANDONED sessions, the detail will usually record information about the error that caused the inspection to fail. For SSL exceptions, this will include the Client or Server to indicate the session endpoint for which traffic was being processed. It will also include Encrypt or Decrypt to indicate the state of traffic inspection when the exception occurred. If available, the SSL error message will also be included. The following table lists the most common error messages and detailed information about each one.

Table 2. SSL Exception Messages
SSL Exception Messages	Description
unexpected_message	An inappropriate message was received. This alert is always fatal and should never be observed in communication between proper implementations.
bad_record_mac	This alert is returned if a record is received with an incorrect MAC. This alert also MUST be returned if an alert is sent because a `TLSCiphertext` decrypted in an invalid way: either it wasn't an even multiple of the block length, or its padding values, when checked, weren't correct. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decryption_failed	This alert was used in some earlier versions of TLS, and may have permitted certain attacks against the CBC mode [CBCATT]. It MUST NOT be sent by compliant implementations.
record_overflow	A `TLSCiphertext` record was received that had a length more than 2^14+2048 bytes, or a record decrypted to a `TLSCompressed` record with more than 2^14+1024 bytes. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decompression_failure	The decompression function received improper input (e.g., data that would expand to excessive length). This message is always fatal and should never be observed in communication between proper implementations.
handshake_failure	Reception of a `handshake_failure` alert message indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. This is a fatal error.
no_certificate	This alert was used in SSLv3 but not any version of TLS. It MUST NOT be sent by compliant implementations.
bad_certificate	A certificate was corrupt, contained signatures that did not verify correctly, etc.
unsupported_certificate	A certificate was of an unsupported type.
certificate_revoked	A certificate was revoked by its signer.
certificate_expired	A certificate has expired or is not currently valid.
certificate_unknown	Some other (unspecified) issue arose in processing the certificate, rendering it unacceptable.
illegal_parameter	A field in the handshake was out of range or inconsistent with other fields. This message is always fatal.
unknown_ca	A valid certificate chain or partial chain was received, but the certificate was not accepted because the CA certificate could not be located or couldn't be matched with a known, trusted CA. This message is always fatal.
access_denied	A valid certificate was received, but when access control was applied, the sender decided not to proceed with negotiation. This message is always fatal.
decode_error	A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message is always fatal and should never be observed in communication between proper implementations (except when messages were corrupted in the network).
decrypt_error	A handshake cryptographic operation failed, including being unable to correctly verify a signature or validate a Finished message. This message is always fatal.
export_restriction	This alert was used in some earlier versions of TLS. It MUST NOT be sent by compliant implementations.
protocol_version	The protocol version the client has attempted to negotiate is recognized but not supported. (For example, old protocol versions might be avoided for security reasons.) This message is always fatal.
insufficient_security	Returned instead of `handshake_failure` when a negotiation has failed specifically because the server requires ciphers more secure than those supported by the client. This message is always fatal.
internal_error	An internal error unrelated to the peer or the correctness of the protocol (such as a memory allocation failure) makes it impossible to continue. This message is always fatal.
user_canceled	This handshake is being canceled for some reason unrelated to a protocol failure. If the user cancels an operation after the handshake is complete, just closing the connection by sending a `close_notify` is more appropriate. This alert should be followed by a `close_notify`. This message is generally a warning.
no_renegotiation	Sent by the client in response to a hello request or by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation; when that is not appropriate, the recipient should respond with this alert. At that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate is where a server has spawned a process to satisfy a request; the process might receive security parameters (key length, authentication, etc.) at startup, and it might be difficult to communicate changes to these parameters after that point. This message is always a warning.
unsupported_extension	sent by clients that receive an extended server hello containing an extension that they did not put in the corresponding client hello. This message is always fatal.

Spam Blocker

Spam Blocker is an intelligent email filter that identifies and handles spam (unsolicited bulk email). It leverages technology from the SpamAssassin project and improves upon itby integrating a commercial spam engine. It can scan any email that is transported via SMTP.

Spam Blocker transparently scans email transported over SMTP on the way to your mail server (or outbound if configured so). It does not require reconfiguring your DNS MX records or the email server. Any SMTP traffic going through the NG Firewall server will be scanned.

Settings

This section reviews the different settings and configuration options available for Spam Blocker.

Status

This displays the current status and some statistics.

Email

These settings apply only to the SMTP protocol.

Scan SMTP: This enables or disables SMTP scanning.

Strength: If the Spam Score of a message is equal to or greater than this setting, your chosen action will be taken regarding the message. Higher values make Spam Blocker more sensitive to spam.
Action: The action was taken on the message if the Spam Score is high enough.

If set to Mark, "[Spam]..." will be prepended to the email subject line and delivered. If set to Pass, the message will be delivered as originally sent. Drop will inform the sending server the mail was successfully delivered, but NG Firewall will drop the mail so it is never delivered. Quarantine will send the mail to users' email quarantine for them to release or delete as they see fit. For more information, refer to Quarantine.
Drop Super Spam: If this option is enabled, any emails with a score greater than the Super spam score will be dropped.
Super Spam Score: The score emails must reach to be dropped as Super Spam.
Advanced SMTP Configuration

Important: The default values are the suggested values. Changing and customizing settings can easily and will likely cause Spam Blocker to perform less than optimally.
- Enable tarpitting: This option enables Tarpit. If enabled, when an SMTP session is first caught Spam Blocker will check if the client IP is on a Domain Name System BLocklist (DNSBL). If it is, the session is rejected before the remote server is even allowed to send the email. This increases the capacity of a given server by quite a bit and can also save bandwidth, but it can increase false positives if the remote email server has mistakenly been put on a blacklist. This setting will not increase spam scanning accuracy - it will actually decrease it as it will prevent valuable super-spam training data from reaching the spam engine. By enabling this feature, you are getting lower spam accuracy but increased email scanning capacity.
- Enable greylisting: This option enables greylisting. If enabled, each time a new sender tries to send mail to a specific receiver, it will receive a "421 Please try again" error. The second time, mail will be allowed. Greylisting will reduce spam because spammers often won't retry transmissions as they should, or the extra time delay will increase the chances of the spam engine properly identifying new spam waves. However, greylisting adds a delay to all legitimate email. This settings is not suggested for most sites because of the complications.
- Add email headers: When enabled, NG Firewall adds information about the Spam Score and the test run to get that score to the headers of the message.
- Close connection on scan failure: This option will close the connection if the scan fails, so the message will be resent and retested. If disabled, a scan failure will allow the email to be delivered without being scanned.
- Scan outbound (WAN) SMTP: If unchecked, outbound mail (mail in sessions going out a WAN interface) will not be scanned. If checked, outbound mail will be scanned just like incoming mail.
- Allow and ignore TLS sessions: This option controls the allowance of TLS sessions. If unchecked (the default), the TLS advertisement (if present) is removed from the server advertisements and TLS is not allowed on any scanned sessions. If checked, the TLS advertisement is allowed and if the client initialized TLS the message will pass through completely unscanned, even if it is spam.
- CPU Load Limit: If your CPU Load exceeds this number incoming connections are stopped until the load decreases. This is specified so that spam scanning can not monopolize the server resources.
- Concurrent Scan Limit: This is the maximum number of messages that can be scanned at the same time. This is specified so that spam scanning can not monopolize the server resources.
- Message Size Limit: This option allows you to change the maximum size of a message that will be scanned for spam. The default maximum size is 256KB. Spam will typically be much smaller, as spammers rely on the sheer number of messages sent.
  Note: This does not control the message size limit of messages passed through NG Firewall. This does not effect the maximum size of the message your server will accept, only the limit on the size of message that will be checked for spam.

Reporting

The Reports tab provides a view of all reports and events for all traffic handled by Spam Blocker.

Reports

Pre-defined report queries:


Report Entry	Description
Spam Blocker Summary	A summary of spam blocking actions for email activity.
Email Usage (all)	The amount of scanned, clean, and spam email over time.
Email Usage (scanned)	The amount of scanned email over time.
Email Usage (clean)	The amount of clean email over time.
Email Usage (spam)	The amount of spam email over time.
Spam Ratio	The ratio of spam (true) to ham (false)
Top Spam Recipients	The number of email addresses with spam.
Top Spam Sender Addresses	The number of IP addresses sending spam.
All Email Events	All emails are scanned by Spam Blocker.
All Spam Events	All emails are marked as Spam.
Quarantined Events	All emails marked as Spam and quarantined.
Tarpit Events	All email sessions that were tarpitted.

The tables queried to render these reports:

Spam Blocker Reports

The Reports tab provides a view of all reports and events for all traffic handled by Spam Blocker.

Reports

Pre-defined report queries:


Report Entry	Description
Spam Blocker Summary	A summary of spam blocking actions for email activity.
Email Usage (all)	The amount of scanned, clean, and spam email over time.
Email Usage (scanned)	The amount of scanned email over time.
Email Usage (clean)	The amount of clean email over time.
Email Usage (spam)	The amount of spam email over time.
Spam Ratio	The ratio of spam (true) to ham (false)
Top Spam Recipients	The number of email addresses with spam.
Top Spam Sender Addresses	The number of IP addresses sending spam.
All Email Events	All emails are scanned by Spam Blocker.
All Spam Events	All emails are marked as Spam.
Quarantined Events	All emails are marked as Spam and quarantined.
Tarpit Events	All email sessions that were tarpitted.

The tables queried to render these reports:

Web Filter

Web Filter monitors HTTP and HTTPS traffic on your network to filter and log web activities and block inappropriate content. Web Filter also appeals to customers who require an added level of protection or are subject to regulations, for example Web Filter helps libraries comply with the Children's Internet Protection Act). Need to block Pornography or Hate Speech on your network? Web Filter is your answer.

About Web Filter

Real-time classification and updates: When your users visit a site, NG Firewall sends the URL to the Webroot BrightCloud® to be categorized. When the data is returned, NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at Webroot and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention, plus you have the option of requesting recategorization of sites.
HTTPS Filtering: Web Filter has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.
Detailed categorization: Web Filter offers 79 categories and tens of billions of URLs. The Web Filter database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to flag websites related to nudity, but allow sites dealing with Sexual Education.
Advanced features: Force safe-search on search engines, filter and log user searches, restrict google domains, and more!

Traffic Flow

When scanning traffic, Web Filter evaluates the pass lists, block lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the request is received from the client and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of filtering and control over both resources that are requested, and content that is returned in response.

HTTP Request

When evaluating HTTP requests, Web Filter applies the configured rules and lists in the following order:

A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Filter as well as other applications.
The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
If 'Restrict Google applications' is enabled, the appropriate header is added to the request using the 'Allowed Domains' that have been configured.
If 'Pass if referrer matches any Pass Sites' is enabled, the referrer is checked against the Pass Sites list. If a match is found, the traffic is allowed.
If the Unblock option is enabled, the destination site and the source IP of the request are checked against the unblock list. If a match is found, the traffic is allowed.
If 'Block pages from IP only hosts' is enabled, the request will be evaluated and blocked if the destination is an IP address.
The destination site of the request is checked against the Block Sites list. If a match is found, the traffic is blocked.
The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.
The category determined in Step 1 is compared to the Categories list, and the traffic is allowed, flagged, or blocked based on the corresponding match. If the category could not be determined, the traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Filter applies the configured rules and lists in the following order:

The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
If the Unblock option is enabled, the site from which the response was received and the client IP are checked against the unblock list. If a match is found, the traffic is allowed.
The traffic details are passed to the Rules list. If a match is found, the traffic is allowed, flagged, or blocked based on the options configured in the rule that was matched.

Settings

This section reviews the different settings and configuration options available for Web Filter.

Status

This displays the current status and some statistics.

Search Terms

Search Terms filtering enables you to flag or block specific search terms your users perform on popular search sites including Google, Bing, Ask, Yahoo, and YouTube. For example, if someone searches Google and includes the word "suicide", or they search for "twerking" videos on YouTube you can have these activities flagged or blocked.

Under Search Filter you can add terms you want to be blocked or flagged. Search Filter terms use the Glob Matcher syntax.

In many cases you may have an existing list of search terms that you want to import. For example, you can find several banned words from Facebook, YouTube, Wordpress and other sources Full List of Bad Words and Top Swear Words Banned by Google Block Facebook, YouTube. The import feature enables you to import these lists from a comma separated or newline delimited file. You can also import in JSON format in case you are transferring a list from another NG Firewall.

Note: SSL Inspector must be installed and enabled to use search filtering.

Site Lookup

Site Lookup allows you to find the categorization of a URL. Clicking it brings up a dialog. In Site URL specify the URL to find and click Search to find the URL's categorization.

If you feel the current categorization is incorrect, check Suggest a different category, select a new category from the list, and click Suggest to submit the category change for consideration.

Note: This is only a suggestion - it will trigger the URL categorization provider to recategorize all categories for the URL.

Block Sites

Under Block Sites you can add individual domain names you want to be blocked or flagged - just enter the domain name (e.g. youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

Pass Sites

Pass Sites is used to pass content that would have otherwise been blocked. This can be useful for "unblocking" sites that you don't want blocked according to block settings. Any domains you add to the Passed Sites list will be allowed, even if blocked by category or by individual URL - just add the domain and save. Unchecking the pass option will allow the site to be blocked as if the entry was not present. This list uses URL Matcher syntax.

Pass Clients

If you add an IP address to this list, Web Filter will not block any traffic from that IP regardless of the blocked categories or sites. Just add the IP and save. Unchecking the pass option will have the block/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

If you have a few users that need to completely bypass Web Filter controls, consider using pass lists. If you have users that simply need different Web Filter settings, you should set up a separate policy using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.

Rules

The Rules tab allows you to specify rules to Block or Flag traffic that passes through Web Filter.

The Rules describes how rules work and how they are configured. Web Filter uses rules to determine to block or flag the specific session. Flagging a session marks it in the logs for reviewing in the event logs or reports, but has no direct effect on the network traffic.

Rule Actions

Flag: Allows the traffic which matched the rule to flow, and flags the traffic for easier viewing in the event log.
Block: Blocks the traffic which matched the rule.

Rule Types

In previous versions of Web Filter, there were dedicated lists for blocking certain file extensions or MIME types. This capability is still available using the more flexible filter rules. For blocking specific file extensions, you can create a rule with the condition HTTP: Response File Extension that has a comma separated list of the extensions to block in the Value field. For blocking MIME types, you would create a rule with the condition HTTP: Content Type that has a comma separated list of the content types to block in the Value field.

Below are tables that list the default file extensions and MIME types that were available in previous versions. Note that these lists are not exhaustive, but are included here as a reference, and to simplify creation of such rules via copy/paste of the values in the tables.


Extension	Category	Description
exe	executable	an executable file format
ocx	executable	an executable file format
dll	executable	an executable file format
cab	executable	an ActiveX executable file format
bin	executable	an executable file format
com	executable	an executable file format
jpg	image	an image file format
png	image	an image file format
gif	image	an image file format
jar	java	a Java file format
class	java	a Java file format
swf	flash	the flash file format
mp3	audio	an audio file format
wav	audio	an audio file format
wmf	audio	an audio file format
mpg	video	a video file format
mov	video	a video file format
avi	video	a video file format
hqx	archive	an archived file format
cpt	compression	a compressed file format


Content	Category	Description
application/octet-stream	unspecified data	byte stream
application/x-msdownload	Microsoft download	executable
application/exe	executable	executable
application/x-exe	executable	executable
application/dos-exe	DOS executable	executable
application/x-winexe	Windows executable	executable
application/msdos-windows	MS-DOS executable	executable
application/x-msdos-program	MS-DOS program	executable
application/x-oleobject	Microsoft OLE Object	executable
application/x-java-applet	Java Applet	executable
audio/mpegurl	MPEG audio URLs	audio
audio/x-mpegurl	MPEG audio URLs	audio
audio/mp3	MP3 audio	audio
audio/x-mp3	MP3 audio	audio
audio/mpeg	MPEG audio	audio
audio/mpg	MPEG audio	audio
audio/x-mpeg	MPEG audio	audio
audio/x-mpg	MPEG audio	audio
application/x-ogg	Ogg Vorbis	audio
audio/m4a	MPEG 4 audio	audio
audio/mp2	MP2 audio	audio
audio/mp1	MP1 audio	audio
application/ogg	Ogg Vorbis	audio
audio/wav	Microsoft WAV	audio
audio/x-wav	Microsoft WAV	audio
audio/x-pn-wav	Microsoft WAV	audio
audio/aac	Advanced Audio Coding	audio
audio/midi	MIDI audio	audio
audio/mpeg	MPEG audio	audio
audio/aiff	AIFF audio	audio
audio/x-aiff	AIFF audio	audio
audio/x-pn-aiff	AIFF audio	audio
audio/x-pn-windows-acm	Windows ACM	audio
audio/x-pn-windows-pcm	Windows PCM	audio
audio/basic	8-bit u-law PCM	audio
audio/x-pn-au	Sun audio	audio
audio/3gpp	3GPP	audio
audio/3gpp-encrypted	encrypted 3GPP	audio
audio/scpls	streaming mp3 playlists	audio
audio/x-scpls	streaming mp3 playlists	audio
application/smil	SMIL	audio
application/sdp	Streaming Download Project	audio
application/x-sdp	Streaming Download Project	audio
audio/amr	AMR codec	audio
audio/amr-encrypted	AMR encrypted codec	audio
audio/amr-wb	AMR-WB codec	audio
audio/amr-wb-encrypted	AMR-WB encrypted codec	audio
audio/x-rn-3gpp-amr	3GPP codec	audio
audio/x-rn-3gpp-amr-encrypted	3GPP-AMR encrypted codec	audio
audio/x-rn-3gpp-amr-wb	3gpp-AMR-WB codec	audio
audio/x-rn-3gpp-amr-wb-encrypted	3gpp-AMR_WB encrypted codec	audio
application/streamingmedia	Streaming Media	audio
video/mpeg	MPEG video	video
audio/x-ms-wma	Windows Media	video
video/quicktime	QuickTime	video
video/x-ms-asf	Microsoft ASF	video
video/x-msvideo	Microsoft AVI	video
video/x-sgi-mov	SGI movie	video
video/3gpp	3GPP video	video
video/3gpp-encrypted	3GPP encrypted video	video
video/3gpp2	3GPP2 video	video
audio/x-realaudio	RealAudio	audio
text/vnd.rn-realtext	RealText	text
audio/vnd.rn-realaudio	RealAudio	audio
audio/x-pn-realaudio	RealAudio plug-in	audio
image/vnd.rn-realpix	RealPix	image
application/vnd.rn-realmedia	RealMedia	video
application/vnd.rn-realmedia-vbr	RealMedia VBR	video
application/vnd.rn-realmedia-secure	secure RealMedia	video
application/vnd.rn-realaudio-secure	secure RealAudio	audio
audio/x-realaudio-secure	secure RealAudio	audio
video/vnd.rn-realvideo-secure	secure RealVideo	video
video/vnd.rn-realvideo	RealVideo	video
application/vnd.rn-realsystem-rmj	RealSystem media	video
application/vnd.rn-realsystem-rmx	RealSystem secure media	video
audio/rn-mpeg	MPEG audio	audio
application/x-shockwave-flash	Macromedia Shockwave	multimedia
application/x-director	Macromedia Shockwave	multimedia
application/x-authorware-bin	Macromedia Authorware binary	multimedia
application/x-authorware-map	Macromedia Authorware shocked file	multimedia
application/x-authorware-seg	Macromedia Authorware shocked packet	multimedia
application/futuresplash	Macromedia FutureSplash	multimedia
application/zip	ZIP	archive
application/x-lzh	LZH archive	archive
image/gif	Graphics Interchange Format	image
image/png	Portable Network Graphics	image
image/jpeg	JPEG	image
image/bmp	Microsoft BMP	image
image/tiff	Tagged Image File Format	image
image/x-freehand	Macromedia Freehand	image
image/x-cmu-raster	CMU Raster	image
image/x-rgb	RGB image	image
text/css	cascading style sheet	text
text/html	HTML	text
text/plain	plain text	text
text/richtext	rich text	text
text/tab-separated-values	tab separated values	text
text/xml	XML	text
text/xsl	XSL	text
text/x-sgml	SGML	text
text/x-vcard	vCard	text
application/mac-binhex40	Macintosh BinHex	archive
application/x-stuffit	Macintosh Stuffit archive	archive
application/macwriteii	MacWrite Document	document
application/applefile	Macintosh File	archive
application/mac-compactpro	Macintosh Compact Pro	archive
application/x-bzip2	block compressed	compressed
application/x-shar	shell archive	archive
application/x-gtar	gzipped tar archive	archive
application/x-gzip	gzip compressed	compressed
application/x-tar	4.3BSD tar archive	archive
application/x-ustar	POSIX tar archive	archive
application/x-cpio	old cpio archive	archive
application/x-bcpio	POSIX cpio archive	archive
application/x-sv4crc	System V cpio with CRC	archive
application/x-compress	UNIX compressed	compressed
application/x-sv4cpio	System V cpio	archive
application/x-sh	UNIX shell script	executable
application/x-csh	UNIX csh script	executable
application/x-tcl	Tcl script	executable
application/x-javascript	JavaScript	executable
application/x-excel	Microsoft Excel	document
application/mspowerpoint	Microsoft Powerpoint	document
application/msword	Microsoft Word	document
application/wordperfect5.1	Word Perfect	document
application/rtf	Rich Text Format	document
application/pdf	Adobe Acrobat	document
application/postscript	Postscript	document

Advanced

The Advanced section enables you to configure additional web filter options.

Secure Name Indication

Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in HTTPS Options.
Process HTTPS traffic by hostname in server certificate when SNI information not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server and the server name on the certificate will be used for categorization and filtering purposes.
Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available: If this option is enabled and neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in HTTPS Options.

Safe Browsing

Enforce safe search on popular search engines: When this option is enabled, safe search will be enforced on all searches using supported search engines: Google, Yahoo!, Bing, Ask.
Enforce restrict mode on YouTube: When this option is enabled, restrict mode will be enforced on all YouTube content.
Force searches through kid-friendly search engine: When this option is enabled, all searches in popular search engines will be redirected through kidzsearch.com. kidzsearch is a visual child-safe search engine and web portal powered by Google Custom Search with academic autocomplete that emphasizes safety for children.

Note: SSL Inspector must be installed and enabled to use all Safe browsing options.

Block Options

Block QUIC Sessions (UDP port 443): If enabled, Web Filter prevents browsers from using the QUIC protocol. QUIC is used by Chrome browser to access many types of Google applications and services. By allowing QUIC, NG Firewall has less visibility and control over this type of traffic.
Block pages from IP only hosts: When this option is enabled, users entering an IP address rather than domain name will be blocked.
Pass if referrer matches any Pass Sites. When this option is checked, if a site allowed via a Pass Site entry links to external content using a referrer, that external content will be passed regardless of Category settings. Please note that this setting requires SSL Inspector to work with HTTPS sites and many sites no longer use referrers.
Close connection for blocked HTTPS sessions without redirecting to block page. If enabled, secure sites blocked by Web Filter do not redirect the user to a denial page and close the connection without any notice to the user. This is useful when you are not using SSL Inspector and the server's root certificate authority is not installed on the client device.

Google Restrictions

Restrict Google applications: When this option is enabled, only domains listed in Domain are allowed to access Google applications such as Gmail. All others are blocked by Google. Multiple domains can be specified, separated by commas such as google.com,domain.com. This adds an X-GoogApps-Allowed-Domains header to web requests which then gets enforced on Google's servers. More information on this feature can be found here.

Note: SSL Inspector must be installed and enabled to restrict Google applications.

Custom Block Page

Custom block page URL: Set an external location where to redirect users when they are denied access to a web site by Web Filter. This is useful if you want to use your own server to process the denial in a different way than the built-in denial options.

The following query string variables are passed to the forward location so the receiving system can process the information.


Variable	Description
reason	The reason the user was denied access.
appname	The NG Firewall app that is responsible for the denial.
appid	The ID of the NG Firewall app that is responsible for the denial.
clientAddress	The IP Address of the denied device.
url	The denied URL that the user requested.

Unblock Options

Unblock: This section can be used to add a button to allow users to bypass restrictions on a case-by-case basis.

: If Unblock is set to None, no users will be allowed to bypass the block page. If Unblock is set to Temporary, users will be allowed to visit the site for one hour from the time it is unblocked. If Unblock is set to Permanent and Global, then users will be allowed to visit the site and unblocked sites will be added to the permanent global pass list so it will always be allowed in the future.; You also have the option of setting a password to Unblock; it can either be the existing Administrator password for the NG Firewall or you can set a new, separate password only for the Unblock feature.

Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.

Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Filter.

Pre-defined report queries:


Report Entry	Description
Web Filter Summary	A summary of web filter actions.
Web Usage	The amount of total, flagged, and blocked web requests over time.
Web Usage (scanned)	The amount of total, flagged, and blocked web requests over time.
Web Usage (flagged)	The amount of flagged, and blocked web requests over time.
Web Usage (blocked)	The amount of flagged, and blocked web requests over time.
Top Categories (by request)	The number of web requests grouped by category.
Top Categories (by size)	The sum of the size of requested web content grouped by category.
Top Flagged Categories	The number of flagged web requests grouped by category.
Top Blocked Categories	The number of blocked web requests grouped by category.
Top Sites (by request)	The number of web requests grouped by website.
Top Sites (by size)	The sum of the size of requested web content grouped by website.
Top Flagged Sites	The number of flagged web requests grouped by website.
Top Blocked Sites	The number of blocked web requests grouped by website.
Top Domains (by request)	The number of web requests grouped by domain.
Top Domains (by size)	The sum of the size of requested web content grouped by domain.
Top Flagged Domains	The number of flagged web requests grouped by domain.
Top Blocked Domains	The number of blocked web requests grouped by domain.
Top Domains Usage	The amount of web requests per top domain.
Top Hostnames (by requests)	The number of web requests grouped by hostname.
Top Hostnames (by size)	The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames	The number of flagged web request grouped by hostname.
Top Blocked Hostnames	The number of blocked web request grouped by hostname.
Top Clients (by requests)	The number of web requests grouped by client.
Top Clients (by size)	The sum of the size of requested web content grouped by client.
Top Flagged Clients	The number of flagged web request grouped by client.
Top Blocked Clients	The number of blocked web request grouped by client.
Top Usernames (by requests)	The number of web requests grouped by username.
Top Usernames (by size)	The sum of the size of requested web content grouped by username.
Top Flagged Usernames	The number of flagged web request grouped by username.
Top Blocked Usernames	The number of blocked web request grouped by username.
Top Content (by request)	The number of web requests grouped by category.
Top Content (by size)	The sum of the size of requested web content grouped by category.
Top Searches	The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches	The number of flagged search queries grouped by term.
Top Blocked Searches	The number of blocked search queries grouped by term.
All Web Events	Shows all scanned web requests.
Flagged Web Events	Shows all flagged web requests.
Blocked Web Events	Shows all blocked web requests.
All HTTP Events	Shows all scanned unencrypted HTTP requests.
All HTTPS Events	Shows all encrypted HTTPS requests.
Unblocked Web Events	Shows all unblocked web requests
All Search Events	Shows all search queries processed by Web Filter.
Flagged Search Events	Shows flagged search queries processed by Web Filter.
Blocked Search Events	Shows blocked search queries processed by Web Filter.

The tables queried to render these reports:

HTTPS Options

There are many ways to handle HTTPS. An overview of the various techniques is described here.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Filter processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.

Process HTTPS traffic by SNI (Server Name Indication) if present.
Process HTTPS traffic by hostname in server certificate when SNI information not present
Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

If the SNI-based categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on the SNI information is logged ("https://example.com/").

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, "wellsfargo.com" is seen as the SNI information. If SNI-based categorization is enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If Web Filter is configured to flag "Financial Services" , then "https://wellsfargo.com/welcome" will be flagged, unless "wellsfargo.com" is in the pass list or the client IP address is in the client IP pass list.

If No SNI information is present and Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled, then the hostname will be pulled from the certificate presented to the client.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

If no SNI or certificate information was available and Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available the session will be processed and categorized by IP address. If the IP-based processing and categorization of the web requests determines the session should be flagged, the session is reset and no more processing of this session will be done. If the IP-based processing and categorization determines the page should be passed (and/or flagged) then the session is allowed and the appropriate event based on its IP is logged ("https://1.2.3.4").

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS processing (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required. Read HTTPS.

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Filter Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Filter.

Reports

Pre-defined report queries:


Report Entry	Description
Web Filter Summary	A summary of web filter actions.
Web Usage	The amount of total, flagged, and blocked web requests over time.
Web Usage (scanned)	The amount of total, flagged, and blocked web requests over time.
Web Usage (flagged)	The amount of flagged, and blocked web requests over time.
Web Usage (blocked)	The amount of flagged, and blocked web requests over time.
Top Categories (by request)	The number of web requests grouped by category.
Top Categories (by size)	The sum of the size of requested web content grouped by category.
Top Flagged Categories	The number of flagged web requests grouped by category.
Top Blocked Categories	The number of blocked web requests grouped by category.
Top Sites (by request)	The number of web requests grouped by website.
Top Sites (by size)	The sum of the size of requested web content grouped by website.
Top Flagged Sites	The number of flagged web requests grouped by website.
Top Blocked Sites	The number of blocked web requests grouped by website.
Top Domains (by request)	The number of web requests grouped by domain.
Top Domains (by size)	The sum of the size of requested web content grouped by domain.
Top Flagged Domains	The number of flagged web requests grouped by domain.
Top Blocked Domains	The number of blocked web requests grouped by domain.
Top Domains Usage	The amount of web requests per top domain.
Top Hostnames (by requests)	The number of web requests grouped by hostname.
Top Hostnames (by size)	The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames	The number of flagged web request grouped by hostname.
Top Blocked Hostnames	The number of blocked web request grouped by hostname.
Top Clients (by requests)	The number of web requests grouped by client.
Top Clients (by size)	The sum of the size of requested web content grouped by client.
Top Flagged Clients	The number of flagged web request grouped by client.
Top Blocked Clients	The number of blocked web request grouped by client.
Top Usernames (by requests)	The number of web requests grouped by username.
Top Usernames (by size)	The sum of the size of requested web content grouped by username.
Top Flagged Usernames	The number of flagged web request grouped by username.
Top Blocked Usernames	The number of blocked web request grouped by username.
Top Content (by request)	The number of web requests grouped by category.
Top Content (by size)	The sum of the size of requested web content grouped by category.
Top Searches	The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches	The number of flagged search queries grouped by term.
Top Blocked Searches	The number of blocked search queries grouped by term.
All Web Events	Shows all scanned web requests.
Flagged Web Events	Shows all flagged web requests.
Blocked Web Events	Shows all blocked web requests.
All HTTP Events	Shows all scanned unencrypted HTTP requests.
All HTTPS Events	Shows all encrypted HTTPS requests.
Unblocked Web Events	Shows all unblocked web requests
All Search Events	Shows all search queries processed by Web Filter.
Flagged Search Events	Shows flagged search queries processed by Web Filter.
Blocked Search Events	Shows blocked search queries processed by Web Filter.

The tables queried to render these reports:

Web Monitor

Web Monitor monitors HTTP and HTTPS traffic on your network to log web activities and flag inappropriate content.

About Web Monitor

Real-time classification and updates: When your users visit a site, NG Firewall sends the URL to the Webroot BrightCloud® to be categorized. When the data is returned, NG Firewall keeps a temporary local cache of the site and category to speed up the process the next time the URL is requested. This data is then used to flag or allow users access to the site they have requested, all without any appreciable increase in load time. If a site is not categorized upon request, it is autocategorized by our partners at Webroot and put into a queue to be verified by a human. Because this is done dynamically, new sites and updated URLs are allowed or flagged according to your settings without additional intervention, plus you have the option of requesting recategorization of sites.
HTTPS Filtering: Web Monitor has multiple techniques to deal with HTTPS, SSL-encrypted HTTP. HTTPS traffic is encrypted so only some information is visible and this information is used to categorize the session. More information on how this is down below.
Detailed categorization: Web Monitor offers 79 categories and tens of billions of URLs. The Web Monitor database is over 100 times larger and more accurate. The abundance of categories means that you can narrow your scope - maybe you want to flag websites related to nudity, but allow sites dealing with Sexual Education.

Traffic Flow

When scanning traffic, Web Monitor evaluates the pass lists, flag lists, categories, and rules at two distinct points of the HTTP transaction. The first evaluation happens after the request is received from the client and before it is forwarded to the server. The second is after the response is received from the server and before it is passed back to the client. This allows a high degree of monitoring over both resources that are requested, and content that is returned in response.

HTTP Request

When evaluating HTTP requests, Web Monitor applies the configured rules and lists in the following order:

A lookup is performed to determine the category for the requested site. The category is attached to the session for use by Web Monitor as well as other applications.
The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
The destination site of the request is checked against the Pass Sites list. If a match is found, the traffic is allowed.
The destination site of the request is checked against the Flag Sites list. If a match is found, the traffic is flagged.
The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the rule that was matched.
The category determined in step #1 is compared to the Categories list, and the traffic is allowed and possibly flagged based on the corresponding match. If the category could not be determined, the traffic is allowed.

HTTP Response

When evaluating HTTP responses, Web Monitor applies the configured rules and lists in the following order:

The source IP of the request is checked against the Pass Clients list. If a match is found, the traffic is allowed.
The site from which the response was received is checked against the Pass Sites list. If a match is found, the traffic is allowed.
The traffic details are passed to the Rules list. If a match is found, the traffic is allowed and possibly flagged based on the options configured in the rule that was matched.

Settings

This section reviews the different settings and configuration options available for Web Monitor.

Status

This displays the current status and some statistics.

Flag Sites

Under Flag Sites you can add individual domain names you want to be flagged - just enter the domain name (e.g. youtube.com) and specify your chosen action. This list uses URL Matcher syntax.

Pass Sites

Pass Sites is used to pass content that would have otherwise been flagged. This can be useful for "unflagging" sites that you don't want flagged according to flag settings. Any domains you add to the Passed Sites list will be allowed, even if flagged by category or by individual URL - just add the domain and save. Unchecking the pass option will allow the site to be flagged as if the entry was not present. This list uses URL Matcher syntax.

Pass Clients

If you add an IP address to this list, Web Monitor will not flag any traffic from that IP regardless of the flagged categories or sites. Just add the IP and save. Unchecking the pass option will have the flag/pass lists affect the user as if they were not entered into the Passed Client IPs list. This list uses IP Matcher syntax.

: If you have a few users that need to completely bypass Web Monitor controls, consider using pass lists. If you have users that simply need different Web Monitor settings, you should set up a separate rack using Policy Manager. When using this feature, please remember that DHCP IPs can change, so you'll probably want to set up either a Static IP or a Static DHCP Lease for the machine in question.

Figure 20. Web Monitor Pass Clients

Rules

The Rules tab allows you to specify rules to Flag traffic that passes through Web Monitor.

The Rules describes how rules work and how they are configured. Web Monitor uses rules to determine when to flag specific sessions. Flagging a session marks it in the logs for reviewing in the event logs or reports, but has no direct effect on the network traffic.

Rule Actions

Flag: Allows the traffic which matched the rule to flow, and flags the traffic for easier viewing in the event log.

Rule Types

In previous versions of Web Monitor, there were dedicated lists for flagging certain file extensions or MIME types. This capability is still available using the more flexible filter rules. For flagging specific file extensions, you can create a rule with the condition Web Filter: Response File Extension that has a comma separated list of the extensions to flag in the Value field. For flagging MIME types, you would create a rule with the condition Web Filter: Response Content Type that has a comma separated list of the content types to flag in the Value field.


Extension	Category	Description
exe	executable	an executable file format
ocx	executable	an executable file format
dll	executable	an executable file format
cab	executable	an ActiveX executable file format
bin	executable	an executable file format
com	executable	an executable file format
jpg	image	an image file format
png	image	an image file format
gif	image	an image file format
jar	java	a Java file format
class	java	a Java file format
swf	flash	the flash file format
mp3	audio	an audio file format
wav	audio	an audio file format
wmf	audio	an audio file format
mpg	video	a video file format
mov	video	a video file format
avi	video	a video file format
hqx	archive	an archived file format
cpt	compression	a compressed file format


Content	Category	Description
application/octet-stream	unspecified data	byte stream
application/x-msdownload	Microsoft download	executable
application/exe	executable	executable
application/x-exe	executable	executable
application/dos-exe	DOS executable	executable
application/x-winexe	Windows executable	executable
application/msdos-windows	MS-DOS executable	executable
application/x-msdos-program	MS-DOS program	executable
application/x-oleobject	Microsoft OLE Object	executable
application/x-java-applet	Java Applet	executable
audio/mpegurl	MPEG audio URLs	audio
audio/x-mpegurl	MPEG audio URLs	audio
audio/mp3	MP3 audio	audio
audio/x-mp3	MP3 audio	audio
audio/mpeg	MPEG audio	audio
audio/mpg	MPEG audio	audio
audio/x-mpeg	MPEG audio	audio
audio/x-mpg	MPEG audio	audio
application/x-ogg	Ogg Vorbis	audio
audio/m4a	MPEG 4 audio	audio
audio/mp2	MP2 audio	audio
audio/mp1	MP1 audio	audio
application/ogg	Ogg Vorbis	audio
audio/wav	Microsoft WAV	audio
audio/x-wav	Microsoft WAV	audio
audio/x-pn-wav	Microsoft WAV	audio
audio/aac	Advanced Audio Coding	audio
audio/midi	MIDI audio	audio
audio/mpeg	MPEG audio	audio
audio/aiff	AIFF audio	audio
audio/x-aiff	AIFF audio	audio
audio/x-pn-aiff	AIFF audio	audio
audio/x-pn-windows-acm	Windows ACM	audio
audio/x-pn-windows-pcm	Windows PCM	audio
audio/basic	8-bit u-law PCM	audio
audio/x-pn-au	Sun audio	audio
audio/3gpp	3GPP	audio
audio/3gpp-encrypted	encrypted 3GPP	audio
audio/scpls	streaming mp3 playlists	audio
audio/x-scpls	streaming mp3 playlists	audio
application/smil	SMIL	audio
application/sdp	Streaming Download Project	audio
application/x-sdp	Streaming Download Project	audio
audio/amr	AMR codec	audio
audio/amr-encrypted	AMR encrypted codec	audio
audio/amr-wb	AMR-WB codec	audio
audio/amr-wb-encrypted	AMR-WB encrypted codec	audio
audio/x-rn-3gpp-amr	3GPP codec	audio
audio/x-rn-3gpp-amr-encrypted	3GPP-AMR encrypted codec	audio
audio/x-rn-3gpp-amr-wb	3gpp-AMR-WB codec	audio
audio/x-rn-3gpp-amr-wb-encrypted	3gpp-AMR_WB encrypted codec	audio
application/streamingmedia	Streaming Media	audio
video/mpeg	MPEG video	video
audio/x-ms-wma	Windows Media	video
video/quicktime	QuickTime	video
video/x-ms-asf	Microsoft ASF	video
video/x-msvideo	Microsoft AVI	video
video/x-sgi-mov	SGI movie	video
video/3gpp	3GPP video	video
video/3gpp-encrypted	3GPP encrypted video	video
video/3gpp2	3GPP2 video	video
audio/x-realaudio	RealAudio	audio
text/vnd.rn-realtext	RealText	text
audio/vnd.rn-realaudio	RealAudio	audio
audio/x-pn-realaudio	RealAudio plug-in	audio
image/vnd.rn-realpix	RealPix	image
application/vnd.rn-realmedia	RealMedia	video
application/vnd.rn-realmedia-vbr	RealMedia VBR	video
application/vnd.rn-realmedia-secure	secure RealMedia	video
application/vnd.rn-realaudio-secure	secure RealAudio	audio
audio/x-realaudio-secure	secure RealAudio	audio
video/vnd.rn-realvideo-secure	secure RealVideo	video
video/vnd.rn-realvideo	RealVideo	video
application/vnd.rn-realsystem-rmj	RealSystem media	video
application/vnd.rn-realsystem-rmx	RealSystem secure media	video
audio/rn-mpeg	MPEG audio	audio
application/x-shockwave-flash	Macromedia Shockwave	multimedia
application/x-director	Macromedia Shockwave	multimedia
application/x-authorware-bin	Macromedia Authorware binary	multimedia
application/x-authorware-map	Macromedia Authorware shocked file	multimedia
application/x-authorware-seg	Macromedia Authorware shocked packet	multimedia
application/futuresplash	Macromedia FutureSplash	multimedia
application/zip	ZIP	archive
application/x-lzh	LZH archive	archive
image/gif	Graphics Interchange Format	image
image/png	Portable Network Graphics	image
image/jpeg	JPEG	image
image/bmp	Microsoft BMP	image
image/tiff	Tagged Image File Format	image
image/x-freehand	Macromedia Freehand	image
image/x-cmu-raster	CMU Raster	image
image/x-rgb	RGB image	image
text/css	cascading style sheet	text
text/html	HTML	text
text/plain	plain text	text
text/richtext	rich text	text
text/tab-separated-values	tab separated values	text
text/xml	XML	text
text/xsl	XSL	text
text/x-sgml	SGML	text
text/x-vcard	vCard	text
application/mac-binhex40	Macintosh BinHex	archive
application/x-stuffit	Macintosh Stuffit archive	archive
application/macwriteii	MacWrite Document	document
application/applefile	Macintosh File	archive
application/mac-compactpro	Macintosh Compact Pro	archive
application/x-bzip2	block compressed	compressed
application/x-shar	shell archive	archive
application/x-gtar	gzipped tar archive	archive
application/x-gzip	gzip compressed	compressed
application/x-tar	4.3BSD tar archive	archive
application/x-ustar	POSIX tar archive	archive
application/x-cpio	old cpio archive	archive
application/x-bcpio	POSIX cpio archive	archive
application/x-sv4crc	System V cpio with CRC	archive
application/x-compress	UNIX compressed	compressed
application/x-sv4cpio	System V cpio	archive
application/x-sh	UNIX shell script	executable
application/x-csh	UNIX csh script	executable
application/x-tcl	Tcl script	executable
application/x-javascript	JavaScript	executable
application/x-excel	Microsoft Excel	document
application/mspowerpoint	Microsoft Powerpoint	document
application/msword	Microsoft Word	document
application/wordperfect5.1	Word Perfect	document
application/rtf	Rich Text Format	document
application/pdf	Adobe Acrobat	document
application/postscript	Postscript	documen

Advanced

The Advanced section allows you to configure additional Web Monitor options.

Process HTTPS traffic by SNI (Server Name Indication) if present: If this option is enabled, HTTPS traffic will be categorized using the "Server Name Indication" in the HTTPS data stream, if present. More details in HTTPS Options.
Process HTTPS traffic by hostname in server certificate when SNI information not present: If this option is enabled and SNI information is not present, the certificate is fetched from the HTTPS server and the server name on the certificate will be used for categorization and filtering purposes.
Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available: If this option is enabled and neither of the previous options worked, HTTPS traffic will be categorized using the IP address. More details in HTTPS Options.
Clear Category URL Cache: This option will clear the local cache of categorized sites and URLs. After clearing the cache all new web visits will be looked up fresh using the categorization service. The cache automatically cleans itself as entries become old or stale, so this is mostly for testing.

Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Monitor.

Pre-defined report queries:


Report Entry	Description
Web Monitor Summary	A summary of web monitor actions.
Web Usage	The amount of total and flagged web requests over time.
Web Usage (scanned)	The amount of total web requests over time.
Web Usage (flagged)	The amount of flagged web requests over time.
Top Categories (by request)	The number of web requests grouped by category.
Top Categories (by size)	The sum of the size of requested web content grouped by category.
Top Flagged Categories	The number of flagged web requests grouped by category.
Top Sites (by request)	The number of web requests grouped by website.
Top Sites (by size)	The sum of the size of requested web content grouped by website.
Top Flagged Sites	The number of flagged web requests grouped by website.
Top Domains (by request)	The number of web requests grouped by domain.
Top Domains (by size)	The sum of the size of requested web content grouped by domain.
Top Flagged Domains	The number of flagged web requests grouped by domain.
Top Domains Usage	The amount of web requests per top domain.
Top Hostnames (by requests)	The number of web requests grouped by hostname.
Top Hostnames (by size)	The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames	The number of flagged web request grouped by hostname.
Top Clients (by requests)	The number of web requests grouped by client.
Top Clients (by size)	The sum of the size of requested web content grouped by client.
Top Flagged Clients	The number of flagged web request grouped by client.
Top Usernames (by requests)	The number of web requests grouped by username.
Top Usernames (by size)	The sum of the size of requested web content grouped by username.
Top Flagged Usernames	The number of flagged web request grouped by username.
Top Content (by request)	The number of web requests grouped by category.
Top Content (by size)	The sum of the size of requested web content grouped by category.
Top Searches	The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches	The number of flagged search queries grouped by term.
All Web Events	Shows all scanned web requests.
Flagged Web Events	Shows all flagged web requests.
All HTTP Events	Shows all scanned unencrypted HTTP requests.
All HTTPS Events	Shows all encrypted HTTPS requests.
All Search Events	Shows all search queries processed by Web Monitor.
Flagged Search Events	Shows flagged search queries processed by Web Monitor.

The tables queried to render these reports:

HTTPS Options

There are many ways to handle HTTPS. An overview of the various techniques is described here.

If SSL Inspector is installed and inspects a session, then it is fully decrypted to HTTP before Web Monitor processes the session. In this case HTTPS is treated identically to HTTP. If SSL Inspector is not installed or the session is not inspected, there are still several techniques to handle encrypted HTTP sessions.

There are three HTTPS options.

Process HTTPS traffic by SNI (Server Name Indication) if present.
Process HTTPS traffic by hostname in server certificate when SNI information not present
Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available.

If Process HTTPS traffic by SNI (Server Name Indication) if present encrypted port-443 traffic will be scanned. Most modern browsers on modern OSs will send the hostname of the server in cleartext - this is called "Server Name Indication" or SNI. SNI is an optional cleartext field in the HTTPS request that shows the hostname of the server. If this option is enabled and the SNI information is present in the HTTPS request, this hostname will be used as the URL for this request and all categorization, flag lists, and pass lists, will be processed as if this were a regular HTTP request to that URL.

For example, if the user visits "https://wellsfargo.com/welcome" in the browser, it will see "wellsfargo.com" as the SNI information. If enabled, the request will be handled exactly like "http://wellsfargo.com" would be. If "Banking" is flagged it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list. If "wellsfargo.com" is flaggeed it will be flagged, unless "wellsfargo.com" is in the pass list or the client IP is in the client IP pass list.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. In this case if Process HTTPS traffic by hostname in server certificate when SNI information not present is enabled it will use the certificate information instead to categorize the session. It will download the certificate from the site and see that the certificate is "Issued To" "www.wellsfargo.com." It will use this information to check the category for "https://www.wellsfargo.com" and categorize the session.

For example, if the user visits "https://wellsfargo.com/welcome" in a non-SNI enabled browser, then there is no SNI information. If the the certificate information was missing for some reason then this session can only be identified by IP address. In this case if Process HTTPS traffic by server IP if both SNI and certificate hostname information are not available is enabled it will use the IP address instead. So it will process/categorize this web request as 'http://1.2.3.4' if 1.2.3.4 is the IP of wellsfargo.com. This will still often result in correct categorization for dedicated web servers, but does poorly when using generic cloud computing servers that offer a wide variety of websites.

Note: Neither HTTPS process (SNI, certificate, or IP-based categorization) can read the URI information as it is not sent in cleartext. As such the URI will not be used as part of the categorization and the URI is assumed to be "/" when evaluating pass rules. If scanning the URI is necessary then full SSL Inspection may be required.

To see the HTTPS categorization in action use the "All HTTPS Events" query in the event log.

Web Monitor Reports

The Reports tab provides a view of all reports and events for all traffic handled by Web Monitor.

Reports

Pre-defined report queries:


Report Entry	Description
Web Monitor Summary	A summary of web monitor actions.
Web Usage	The amount of total and flagged web requests over time.
Web Usage (scanned)	The amount of total web requests over time.
Web Usage (flagged)	The amount of flagged web requests over time.
Top Categories (by request)	The number of web requests grouped by category.
Top Categories (by size)	The sum of the size of requested web content grouped by category.
Top Flagged Categories	The number of flagged web requests grouped by category.
Top Sites (by request)	The number of web requests grouped by website.
Top Sites (by size)	The sum of the size of requested web content grouped by website.
Top Flagged Sites	The number of flagged web requests grouped by website.
Top Domains (by request)	The number of web requests grouped by domain.
Top Domains (by size)	The sum of the size of requested web content grouped by domain.
Top Flagged Domains	The number of flagged web requests grouped by domain.
Top Domains Usage	The amount of web requests per top domain.
Top Hostnames (by requests)	The number of web requests grouped by hostname.
Top Hostnames (by size)	The sum of the size of requested web content grouped by hostname.
Top Flagged Hostnames	The number of flagged web request grouped by hostname.
Top Clients (by requests)	The number of web requests grouped by client.
Top Clients (by size)	The sum of the size of requested web content grouped by client.
Top Flagged Clients	The number of flagged web request grouped by client.
Top Usernames (by requests)	The number of web requests grouped by username.
Top Usernames (by size)	The sum of the size of requested web content grouped by username.
Top Flagged Usernames	The number of flagged web request grouped by username.
Top Content (by request)	The number of web requests grouped by category.
Top Content (by size)	The sum of the size of requested web content grouped by category.
Top Searches	The number of non blocked, non-flagged search queries grouped by term.
Top Flagged Searches	The number of flagged search queries grouped by term.
All Web Events	Shows all scanned web requests.
Flagged Web Events	Shows all flagged web requests.
All HTTP Events	Shows all scanned unencrypted HTTP requests.
All HTTPS Events	Shows all encrypted HTTPS requests.
All Search Events	Shows all search queries processed by Web Monitor.
Flagged Search Events	Shows flagged search queries processed by Web Monitor.

The tables queried to render these reports:

ETM Micro Edge User Manual

NG Firewall Filter Apps

Ad Blocker

Settings

Status

Options

Ad Filters

Cookie Filters

Pass Lists

Reports

Related Topics

Ad Blocker Reports

Reports

Related Topics

Application Control

About Application Control

How It Works

Settings

Status

Applications

Rules

Anatomy of a Rule

Application Detail

Reports

Related Topics

Application Control Reports

Reports

Application Control Lite

Settings

Status

Signatures

Reporting

Reports

Related Topics

Application Control Lite Reports

Reports

SSL Inspector

About SSL Inspector

SSL Certificates

Client Configuration

Group Policy Distribution

Settings

Status

Configuration

Download Root Certificate

Enable SMTPS Traffic Processing

Enable HTTPS Traffic Processing

Block Invalid HTTPS Traffic

Client/Server Connection Protocols

Trust All Server Certificates

Upload Trusted Certificate

Rules

Reports

Status

Detail

SSL Inspector Reports

Reports

Status

Detail

Related Topics

Spam Blocker

Settings

Status

Email

Reporting

Reports

Spam Blocker Reports

Reports

Web Filter

About Web Filter

Traffic Flow

HTTP Request

HTTP Response

Settings

Status

Categories

Search Terms

Site Lookup

Block Sites

Pass Sites