Log Formats and Versioning

Suggest Edits

Zipped CSV log files are available for download from the S3 bucket. Unzipping and opening these files displays multiple columns of information extracted from your Umbrella logs. There are additional fields that are exposed in these logs that are not normally shown through Umbrella's reports. For more information on reporting, see Get Started with Reports.
Note: Logs are not always chronological and will not always be in the specific time bucket based on the timestamp of the log event.

File Name Format
Find Your Log Schema Version
Log File Fields
DNS Logs
Web Logs
Cloud Firewall Logs
Admin Audit Logs
IPS Logs
DLP Logs
Activity Search Logs
Top Destinations Logs
Top Categories Log
Top Identities Log
Estimate the Size of Your Logs

File Name Format

Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 bucket. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket. To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket (Amazon S3 > <bucketname> > dnslogs). The logs will appear in a GZIP format with the following file name format. The files will also be sorted into date-stamped folders.

DNS traffic
dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Firewall traffic
firewalllogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Admin Audit logs
auditlogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

IPS traffic
intrusionlogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Web traffic
proxylogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Subfolders

Logs are uploaded to S3 buckets in the appropriate subfolder with the following naming format.
<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz

Umbrella names a log subfolder () with one of the following folder names:

auditlogs
dnslogs
firewallogs
intrusionlogs
proxylogs

The segment of the log GZIP file name is a random string of four alphanumeric characters, which prevents duplicate file names from being overwritten.

Example: dnslogs/2019-01-01/2019-01-01-00-00-e4e1.csv.gz

Find Your Log Schema Version

Depending on the Umbrella subscription you have, there are different versions of the log schemas available. Once your system is configured to log to an Amazon S3 bucket you can view the log schema version in use.

Prerequisites

Configure logging to an Amazon S3 bucket. For more information, see Manage Your Logs.

Log Schema Versions

v1—For customers who have configured their own S3 bucket.
v2— The same as version 1, but adds two new fields: Most Granular Identity Type and Identity Types for DNS logs.
v3—The same as version 2, but adds the Blocked Categories field for DNS and Proxy logs.
v4—The same as version 3, but adds three new fields: all Identities, all Identity Types, and Request Method for Proxy logs.
v5—The same as version 4, but adds the following fields to Proxy logs: Certificate Errors, Destination Lists IDs, DLP Status, File Name, Rule ID, and Ruleset ID.
v6—The same as version 5, but adds the DLP file label field.
v7—The same as version 6, but adds fields to the Proxy, DLP, and Firewall logs.
- Proxy logs—Adds the Isolate Action, File Action, and Warn Status fields.
- DLP logs—Changes the Event Type field. The event type is either Real Time or SaaS API.
- Firewall logs—Adds the FQDNs and Destination List IDs fields.
v8—The same fields as version 7, but the v8 log format adds fields to the IPS and Web logs.

Procedure

Navigate to Admin > Log Management.

In the Amazon S3 area, view the Schema Version in use.

Log File Fields

Each type of Umbrella log contains various log fields. Not all field values are available in every log record. When a field does not have a value, Umbrella sets the field to the empty string ("").

DNS Logs

DNS logs show traffic that has reached our DNS resolvers.
Example:
"2015-01-16 17:48:41","ActiveDirectoryUserName", "ActiveDirectoryUserName,ADSite,Network", "10.10.1.100","172.16.132.133","Allowed","1 (A)", "NOERROR","domain-visited.com.", "Chat,Photo Sharing,Social Networking,Allow List"

The example entry is 224 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in DNS Log Record

<timestamp><most granular identity><identities><internal ip><external ip><action><query type><response code><domain><categories><most granular identity type><identity types><blocked categories>

Timestamp—When this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
Most Granular Identity—The first identity matched with this request in order of granularity.
Identities—All identities associated with this request.
Internal IP—The internal IP address that made the request.
External IP—The external IP address that made the request.
Action—Whether the request was allowed or blocked.
Query Type—The type of DNS request that was made. For more information, see Common DNS Request Types.
Response Code—The DNS return code for this request. For more information, see Common DNS return codes for any DNS service (and Umbrella).
Domain—The domain that was requested.
Categories—The security or content categories that the destination matches. For category definitions, see Understanding Security Categories and Understanding Content Categories.
Most Granular Identity Type—The first identity type matched with this request in order of granularity. Available in version 3 and above.
Identity Types—The type of identity that made the request. For example, Roaming Computer, Network, and so on. Available in version 3 and above.
Blocked Categories—The categories that resulted in the destination being blocked. Available in version 4 and above.

Web Logs

Web logs show traffic that has passed through the Umbrella secure web gateway (SWG).

Example:

"2017-10-02 23:52:53","TheComputerName","192.0.192.135","10.10.10.91", "3.4.5.6","","ALLOWED","http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","Search Engines","","","","","","Roaming Computer","","TheComputerName, ADSite,Network","Roaming Computer, Site, Network","GET","","","the.js","","","","isolated","downloaded_original_file","warn-session"

The example entry is 490 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in Web Log Record

Note: Not all fields listed are found in most or all requests. When a field does not have a value, Umbrella sets the field to the empty string ("") in the log.

Timestamp—The timestamp of the request transaction in UTC (2015-01-16 17:48:41).
Policy Identity Label—The identity that made the request.
Internal Client IP—The internal IP address of the computer making the request.
External Client IP—The egress IP address of the network where the request originated.
Destination IP—The destination IP address of the request.
Content Type—The type of web content, typically text/html.
Action—Whether the request was allowed or blocked.
URL—The URL requested.
Referer—The referring domain or URL.
User Agent—The browser agent that made the request.
Status Code—The HTTP status code; should always be 200 or 201.
Request Size (bytes)—Request size in bytes.
Response Size (bytes)—Response size in bytes.
Response Body Size (bytes)—Response body size in bytes.
SHA—SHA256—The hex digest of the response content.
Categories—The security categories for this request, such as Malware.
AV Detections—The detection name according to the antivirus engine used in file inspection.
PUAs—A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
AMP Disposition—The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
AMP Malware Name—If Malicious, the name of the malware according to AMP.
AMP Score—The score of the malware from AMP. This field returns blank unless the verdict is Unknown, in which the value will be 0.
Policy Identity Type—The first identity type that made the request. For example, Roaming Computer, Network, and so on.
Blocked Categories—The category that resulted in the destination being blocked. Available in version 4 and above.
Identities—All identities associated with this request.
Identity Types—The type of identities that were associated with the request. For example, Roaming Computer, Network, and so on. Available in version 5 and above.
Request Method—The request method, for example: GET, POST, HEAD, PUT, DELETE.
DLP Status—If the request was Blocked for DLP.
Certificate Errors—Any certificate or protocol errors in the request.
File Name—The name of the file.
Ruleset ID—The ID number assigned to the ruleset.
Rule ID—The ID number assigned to the rule.
Destination List IDs—The ID number assigned to a destination list.
Isolate Action—The remote browser isolation state associated with the request.
File Action—The action taken on a file in a remote browser isolation session.
Warn Status—The warn page state associated with the request.

Cloud Firewall Logs

Cloud Firewall logs show traffic that has been handled by network tunnels.

Example:

"2019-01-14 18:03:46","[211039844]","Passive Monitor", "CDFW Tunnel Device","OUTBOUND","1","84","172.17.3.4","","192.0.2.129", "","ams1.edc","12","ALLOW","google.com,apple.com","44,66"

The example entry is 188 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in Cloud Firewall Log Record

<timestamp><origin IDs><identities><identity type><direction><protocol><packet size><source IP><source port><destination IP><destination port><data center><rule ID><action><fqdns><destination list IDs>

Timestamp—The timestamp of the request transaction in UTC.
Origin IDs—The unique identity of the network tunnel.
Identities—The names of the network tunnel.
Identity Type—The type of identity that made the request. Should always be "CDFW Tunnel Device".
Direction—The direction of the packet. It is destined either towards the internet or to the customer's network.
Protocol—The actual protocol of the traffic. It could be TCP, UDP, ICMP.
Packet Size—The size of the packet that Umbrella CDFW received.
Source IP—The internal IP address of the user-generated traffic towards the CDFW. If the traffic goes through NAT before it comes to CDFW, it will be the NAT IP address.
Source Port—The internal port number of the user-generated traffic towards the CDFW.
Destination IP—The destination IP address of the user-generated traffic towards the CDFW.
Destination Port—The destination port number of the user-generated traffic towards the CDFW.
Data Center—The name of the Umbrella data center that processed the user-generated traffic.
Rule ID—The ID of the rule that processed the user traffic.
Action—The final verdict whether to allow or block the traffic based on the rule.
FQDNs—The fully qualified domain names (FQDNs) that match the request.
Destination List IDs—The destination list IDs that Umbrella applied in the rule.

Admin Audit Logs

Admin Audit logs show changes made by your administrative team in your organization's Umbrella settings.
Example:
"","2021-07-22 10:46:45","[email protected]","","logexportconfigurations", "update","209.165.200.227","version: 4","version: 5"

The example entry is 126 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in Admin Audit Log Record

<id><timestamp><email><user><type><action><logged in from><before><after>

ID—A unique identifier of the audit event.
Timestamp—The date and time when this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
Email—The email of the user that triggered the event.
User—The account name of the user who created the change.
Type—Where the change was made, such as settings or a policy.
Action—The type of change made, such as Create, update, or Delete.
Logged in from—The user's IP source.
Before—The policy or setting before the change was made.
After—The policy or setting after the change was made.

##IPS Logs
IPS logs show traffic, events, and possible threats detected by Umbrella's Intrusion Prevention System.
Example:

"2022-04-12 16:14:09","Firewall Tunnel Name","Network Tunnels","1","16606","SERVER-ORACLE BEA WebLogic Server Plug-ins Certificate overflow attempt","1323","HIGH","Attempted User Privilege Gain","cve-2009-1016","TCP","12345","123.123.123.123","33010","10.10.10.10","443","Would Block"

The example entry is 281 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in IPS Log Record

<timestamp><identities><identity types><generator id><signature id><signature message><signature list id><severity><attach classification><CVEs><IP protocol><session ID><source IP><source port><destination IP><destination port><action>

Timestamp—When this request was made in UTC.
Identities—All tunnel identities associated with this request.
Identity Types—The type of identity associated with this request.
Generator ID—Unique ID assigned to the part of the IPS which generated the event.
Signature ID—Used to uniquely identify signatures.
Signature Message—A brief description of the signature.
Signature List ID—Unique ID assigned to a Default or Custom Signature List.
Severity—The severity level of the rule, such as High, Medium, Low, and Very Low.
Attack Classification—The category of attack detected by a rule that is part of a more general type of attack class, such as trojan-activity, attempted-user, and unknown.
CVEs—A list of information about security vulnerabilities and exposures.
IP Protocol—The actual protocol of the traffic, such as TCP, UDP, ICMP.
Session ID—The unique identifier of a session, which is used to group the correlated events between various services.
Source IP—The IP of the computer making the request.
Source Port—The port the request was made on.
Destination IP—The destination IP requested.
Destination Port—The destination port the request was made on.
Action—The action performed when criteria meets a rule, such as block, warn, and would_block.

DLP Logs

DLP logs show information about DLP events where data identifiers were triggered and a violation occurred. DLP logs are available in all versions.
Note: A single DLP event can present in multiple rows of the logs when different data identifiers and file labels are triggered for the same content. Rows pertinent to the same content or event have the same Unique Event ID.

Example:
"2022-02-15 12:05:45","Real Time","f64dcc3f-50fa-410a-b8e1-589894276cee_17c81f85-34f7-4bc5-aa4c-155571f484f6","CRITICAL","Network1","","first.xlsx","Dropbox","http://google.com","BLOCK","rule-1","classification-2","classifier-2.1","text/html","48","abbd2352c3cfea8846871928bf99ca24dc3a6f162170926649381a6d968869ab", "Confidential"

The example entry is 312 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in DLP Log Record

<timestamp><event type><unique event id><severity><identity><owner><name><application><destination><action><rule><data classification><data identifier><content type><file size><SHA 256 hash><file label>

Timestamp—The timestamp of the request transaction in UTC.
Event Type—The type of event that matched a data identifier. "Real Time" denotes a proxy-based DLP event triggered by a Real Time rule and "SaaS API" denotes a DLP event triggered by any of the SaaS API rules.
Unique Event ID—The unique identifier for the event. There can be multiple violation matches in one event.
Severity—The severity of the rule (Low, Medium, High or Critical).
Identity—The source that triggered the violation.
Owner—The owner of the file.
Note: This column has limited availability. Contact Support at [email protected] for more information.
Name—The name of the file.
Application—The application of the request.
Destination—The domain of the request.
Action—If the violation was Blocked or Monitored.
Rule—The DLP rule name.
Data Classification—The data classification whose data identifier matched on the violation.
Data Identifier—The data identifier that matched on the request.
Content Type—The mime type of the file that matches the data identifier.
File Size—The size of the file.
SHA256 Hash—The hex digest of the response content.
File Label—The file name label that matched on the file properties.

Activity Search Logs

You can export the results of the Activity Search Report to a CSV format. For more information, see Export Report Data to CSV and Activity Search Report.

Fields in the Activity Search Log

Type—The type of request made, such as DNS.
Date—The date the request was made.
Time—The time the request was made, in UTC.
Action—Whether the request was Allowed or Blocked.
Errors—Any certificate or protocol errors in the request.
Ruleset ID—The ID number assigned to the ruleset.
Ruleset Name—The ruleset that was applied.
Rule ID—The ID number assigned to the rule.
Rule Name—The rule that was applied.
Destination List IDs—The ID number assigned to a destination list.
Signature List ID—The unique ID assigned to a Default or Custom Signature List.
IPS Signature—The threat detected in our IPS/IDS protection.
IPS SIgnature Severity—The severity of the IPS Signature.
IPS Signature CVE—Common vulnerabilities and exposures related to the IPS Signature.
Identities—All tunnel identities associated with this request.
Identity Types—The type of identity that were associated with the request. For example, Roaming Computers or Networks.
Policy or Ruleset Identity—The identity that made the request.
Policy or Ruleset Identity Type—The type of the identity that made the request.
Forwarding Method—The method used to forward the identity of the client to the proxy.
Internal IP—The internal IP address that made the request.
External IP—The external IP address that made the request.
Source IP—The IP of the computer making the request.
Destination IP—The destination IP requested.
Source Port—The port the request was made on.
Destination Port—The destination port the request was made on.
Destination—The domain of the request.
Hostname—The name of the host.
Categories—The content categories, if any, that matched against the destination IP address or port requested.
Integrations—Integration categories you set.
Blocked Categories—The category that resulted in the destination being blocked.
Application—The application associated with the request.
Application Category—The categories for any applications associated with the request.
Query Type—The type of DNS request that was made.
Content Type—The type of web content; typically text or html.
Protocol—The actual protocol of the traffic. For example, TCP, UDP, or ICMP.
Filename—The name of the file.
File Action (Remote Browser Isolation)—The action taken on a file during a Remote Browser Isolation session.
Total Size in Bytes—The total size in bytes.
Request Size—Request size in bytes.
Response Size—Response size in bytes.
Packet Size—Packet size in bytes.
Referrer—The referring domain or URL.
User Agent—The browser agent that made the request.
Status Code—The HTTP status code.
Direction—The direction of the packet. It is directed either towards the internet or to the customer's network.
Threats—Any threats associated with the request.
Threat Types—The types of threats associated with the request.
SHA256 Hash—The hex digest of the response content.
Cisco AMP Result—The malware detected by AMP.
Cisco AMP Disposition—What action was taken on the file download.
Cisco AMP Score—The risk score associated with the downloaded file. This field returns blank unless the verdict is Unknown, in which the value will be 0.
Antivirus Result—Threats detected by the antivirus.
Potentially Unwanted Applications—A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
Detected Response File Type—The file type of the response, as detected by the file type control that blocked the request based on factors such as URL or content type header.
Isolated State—Whether the Remote Browser Isolation state was isolated or not.
Data Loss Prevention State—Whether the DLP status was allowed or blocked.
Tenant Controls—Whether the request is Tenant Application Access Control protected.

Top Categories Log

You can export the results of the Top Categories Report to a CSV format. For more information, see Export Report Data to CSV and Top Categories Report.

Fields in the Top Categories Log

Category—A content category in which a request was made. See Manage Content Categories.
Count—The number of requests made for the category.

Top Destinations Log

You can export the results of the Top Destinations Report to a CSV format. For more information, see Export Report Data to CSV and Top Destinations Report.

Fields in the Top Destinations Log

Domain—The domain that was requested.
Query Count—The number of requests for the domain.
Categories—The content categories that matched against the destination requested. See Manage Content Categories.

Top Identities Log

You can export the results of the Top Identities Report to a CSV format. For more information, see Export Report Data to CSV and Top Identities Report.

Fields in the Top Identities Log

Identity—The identity making requests.
Query Count—The number of requests made by the identity.

Estimate the Size of Your Logs

The size of your S3 logs depends on the number of events that occur, which is dependent on the volume of your DNS traffic. The size of each log line varies based on a number of items—for example, the length of the domain name or the number of categories. Assuming each log line is 220 bytes, a million requests would be 220 MB.

In the Umbrella dashboard, navigate to Reporting > Activity Search.

Under Filters, run a report for the last 24 hours and then click the Export CSV icon.

Open the downloaded .csv file. The number of rows (minus one for the header) is the number of DNS queries per day; multiply that by 220 bytes to get the estimate for one day.

Stop Logging < Log Format and Versioning > Manage Authentication

Updated 2 months ago

Table of Contents