Guides
ProductDeveloperPartnerPersonal
ProductDeveloperPartnerPersonal

Log Formats and Versioning

Zipped CSV log files are available for download from either Cisco's managed S3 bucket or your own S3 bucket. Unzipping and opening these files displays multiple columns of information extracted from your Umbrella logs. There are additional fields that are exposed in these logs that are not normally shown through Umbrella's reports. For more information on reporting, see Get Started with Reports.
Note: Logs are not always chronological and will not always be in the specific time bucket based on the timestamp of the log event.

Table of Contents

File Name Format

Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 bucket. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket. To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket (Amazon S3 > <bucketname> > dnslogs). The logs will appear in a GZIP format with the following file name format. The files will also be sorted into date-stamped folders.

DNS traffic
dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Proxied traffic (the intelligent proxy)
proxylogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Admin Audit
auditlogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz

Subfolders

Logs are uploaded to S3 buckets in the appropriate subfolder with the following naming format.
<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz

Umbrella names a log subfolder () with one of the following folder names:

  • dnslogs
  • proxylogs
  • auditlogs

The segment of the log GZIP file name is a random string of four alphanumeric characters, which prevents duplicate file names from being overwritten.

Example: dnslogs/2019-01-01/2019-01-01-00-00-e4e1.csv.gz

Find Your Log Schema Version

Umbrella provides multiple versions of log schemas. The availability of various Umbrella log schemas depends on your Umbrella subscription and the type of S3 bucket that you configure. Once your system is configured to log to an Amazon S3 bucket you can view the log schema version in use.

Prerequisites

Log Schema Versions

  • v1—For customers who have configured their own S3 bucket before November 2017. This version has a single sub-folder in the bucket and contains only DNS traffic logs.
  • v2—For customers who have configured their own S3 bucket after November 2017, or are using a Cisco-managed bucket. This version is inclusive of everything in version 1, and adds two new log types: Proxy traffic logs and IP traffic logs. Each log type has its own sub-folder.
  • v3— The same as version 2, but adds two new fields: Most Granular Identity Type and Identity Types for DNS logs.
  • v4—The same as version 3, but adds the Blocked Categories field for DNS and Proxy logs.
  • v5—The same as version 4, but adds three new fields: all Identities, all Identity Types, and Request Method for Proxy logs.
  • v6—The same as version 5 with these additional fields to Proxy logs: Certificate Errors, Destination Lists IDs, DLP Status, File Name, Rule ID, and Ruleset ID.
  • v7—The same as version 6, but adds the DLP file label field.
  • v8—The same as version 7, but adds the Isolate Action, File Action, and Warn Status fields to the Proxy log.

👍

Version 1 Bucket Recreation

To upgrade from v1 to a higher version of the Umbrella log format, you must remove your existing S3 bucket, disable the integration, and then recreate a new bucket. For all other versions, you can upgrade from the Log Management screen of the Umbrella dashboard by clicking Upgrade.

Procedure

  1. Navigate to Admin > Log Management.
1261
  1. In the Amazon S3 area, view the Schema Version in use.
1077

Log File Fields

Each type of Umbrella log contains various log fields. Not all field values are available in every log record. When a field does not have a value, Umbrella sets the field to the empty string ("").

DNS Logs

DNS logs show traffic that has reached our DNS resolvers.
Example:

"2015-01-16 17:48:41","ActiveDirectoryUserName","ActiveDirectoryUserName,ADSite,Network", "10.10.1.100","24.123.132.133","Allowed","1 (A)","NOERROR","domain-visited.com.","Chat,Photo Sharing,Social Networking,Allow List","AD User","AD User,Site,Network",""

The example entry is 224 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in DNS Log Record

<timestamp><most granular identity><identities><internal ip><external ip><action><query type><response code><domain><categories><most granular identity type><identity types><blocked categories>

  • Timestamp—When this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
  • Most Granular Identity—The first identity matched with this request in order of granularity.
  • Identities—All identities associated with this request.
  • Internal IP—The internal IP address that made the request.
  • External IP—The external IP address that made the request.
  • Action—Whether the request was allowed or blocked.
  • Query Type—The type of DNS request that was made. For more information, see Common DNS Request Types.
  • Response Code—The DNS return code for this request. For more information, see Common DNS return codes for any DNS service (and Umbrella).
  • Domain—The domain that was requested.
  • Categories—The security or content categories that the destination matches. For category definitions, see Understanding Security Categories and Understanding Content Categories.
  • Most Granular Identity Type—The first identity type matched with this request in order of granularity. Available in version 3 and above.
  • Identity Types—The type of identity that made the request. For example, Roaming Computer, Network, and so on. Available in version 3 and above.
  • Blocked Categories—The categories that resulted in the destination being blocked. Available in version 4 and above.

Proxy Logs

Proxy logs show traffic that has passed through the Umbrella secure web gateway (SWG) or the Selective Proxy.
Example:

"2017-10-02 23:52:53","TheComputerName","192.192.192.135","1.1.1.91", "3.4.5.6","","ALLOWED","http://google.com/the.js","www.google.com","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200","562","1489","","","Search Engines","","","","","","Roaming Computer","","TheComputerName, ADSite,Network","Roaming Computer, Site, Network","GET","","","the.js","","","","isolated","downloaded_original_file","warn-session"

The example entry is 490 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Note: Umbrella only shows proxied URLs in the proxy folder of your S3 bucket. Umbrella logs all other traffic, including any DNS requests, to the dnslogs folder with the action Proxied.

Order of Fields in Proxy Log Record

<timestamp><policy identity label><internal client ip><external client ip><destination ip><content type><action><url><referer><user agent><status code><request size><response size><response body size><sha—sha256><categories><av detections><PUAs><AMP disposition><AMP malware name><AMP score><policy identity type><blocked categories><identities><identity types><request method><DLP status><certificate errors><file name><ruleset ID><rule ID><destination list IDs><isolate action><file action><warn status>

Note: Not all fields listed are found in most or all requests. When a field does not have a value, Umbrella sets the field to the empty string ("").

  • Timestamp—The timestamp of the request transaction in UTC (2015-01-16 17:48:41).
  • Policy Identity Label—The identity that made the request.
  • Internal Client IP—The internal IP address of the computer making the request.
  • External Client IP—The egress IP address of the network where the request originated.
  • Destination IP—The destination IP address of the request.
  • Content Type—The type of web content, typically text/html.
  • Action—Whether the request was allowed or blocked.
  • URL—The URL requested.
  • Referer—The referring domain or URL.
  • User Agent—The browser agent that made the request.
  • Status Code—The HTTP status code; should always be 200 or 201.
  • Request Size (bytes)—Request size in bytes.
  • Response Size (bytes)—Response size in bytes.
  • Response Body Size (bytes)—Response body size in bytes.
  • SHA—SHA256—The hex digest of the response content.
  • Categories—The security categories for this request, such as Malware.
  • AV Detections—The detection name according to the antivirus engine used in file inspection.
  • PUAs—A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
  • AMP Disposition—The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
  • AMP Malware Name—If Malicious, the name of the malware according to AMP.
  • AMP Score—The score of the malware from AMP. This field returns blank unless the verdict is Unknown, in which the value will be 0.
  • Policy Identity Type—The first identity type that made the request. For example, Roaming Computer, Network, and so on.
  • Blocked Categories—The category that resulted in the destination being blocked. Available in version 4 and above.
  • Identities—All identities associated with this request.
  • Identity Types—The type of identities that were associated with the request. For example, Roaming Computer, Network, and so on. Available in version 5 and above.
  • Request Method—The request method (GET, POST, HEAD, etc.)
  • DLP Status—If the request was Blocked for DLP.
  • Certificate Errors—Any certificate or protocol errors in the request.
  • File Name—The name of the file.
  • Ruleset ID—The ID number assigned to the ruleset by Umbrella.
  • Rule ID—The ID number assigned to the rule by Umbrella.
  • Destination List IDs—The ID number umbrella assigns to a destination list.
  • Isolate Action—The remote browser isolation state associated with the request.
  • File Action—The action taken on a file in a remote browser isolation session.
  • Warn Status—The warn page state associated with the request.

Admin Audit Logs

Admin Audit logs show changes made by your administrative team in your organization's Umbrella settings.
Example:
"","2021-07-22 10:46:45","[email protected]","", "logexportconfigurations", "update","209.165.200.227","version: 4","version: 5"

The example entry is 126 bytes. To estimate the size of your S3 Logs, see Estimate the Size of Your Logs.

Order of Fields in Admin Audit Log Record

<id><timestamp><email><user><type><action><logged in from><before><after>

  • ID—A unique identifier of the audit event.
  • Timestamp—The date and time when this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
  • Email—The email of the user that triggered the event.
  • User—The account name of the user who created the change.
  • Type—Where the change was made, such as settings or a policy.
  • Action—The type of change made, such as Create, update, or Delete.
  • Logged in from—The user's IP source.
  • Before—The policy or setting before the change was made.
  • After—The policy or setting after the change was made.

Estimate the Size of Your Logs

The size of your S3 logs depends on the number of events that occur, which is dependent on the volume of your DNS traffic. The size of each log line varies based on a number of items—for example, the length of the domain name or the number of categories. Assuming each log line is 220 bytes, a million requests would be 220 MB.

  1. In the Umbrella dashboard, navigate to Reporting > Activity Search.
1141
  1. Under Filters, run a report for the last 24 hours and then click the Export CSV icon.
563
  1. Open the downloaded .csv file. The number of rows (minus one for the header) is the number of DNS queries per day; multiply that by 220 bytes to get the estimate for one day.

Delete Logs < Log Format and Versioning > Manage Authentication

Updated about 1 month ago