After configuring Log Management with AWS S3, the logs you download will be gzipped CSVs. Unzipping the logs and opening the CSV files will show the columns of information extracted from your Umbrella logs. There are additional fields that are exposed in these logs that are not normally shown through the Reporting section of the dashboard. For more information on reporting, see Getting Started with Reports.
Logs are uploaded in ten-minute intervals from the Umbrella log queue to the S3 bucket. Within the first two hours after a completed configuration, you should receive your first log upload to your S3 bucket. To check to see if everything is working, the Last Sync time in the Umbrella dashboard should update and logs should begin to appear in your S3 bucket (Amazon S3 > bucket-name > dnslogs). The logs will appear in a GZIP format with the following file name format. The files will also be sorted into date-stamped folders.
Proxied traffic (the intelligent proxy in the Advanced Settings):
IP traffic generated from the IP Layer enforcement feature (a subfeature of the intelligent proxy):
Depending on the Umbrella subscription you have, and depending on the type of bucket you configure, there are different versions of the log formats. Currently, there are four versions:
- Version 1—for customers who have configured their own S3 bucket before November 2017. This version has a single sub-folder in the bucket and contains only DNS traffic logs.
- Version 2—for customers who have configured their own S3 bucket after November 2017, are using a Cisco-managed bucket, or are using Centralized Log Management for MSP/MSSP/Multi-org consoles. This version is inclusive of everything in version 1, and adds two new log types: Proxy traffic logs and IP traffic logs. Each log type has its own sub-folder.
- Version 3— the same as version 2, but adds two new fields: Most Granular Identity Type and Identity Types for DNS logs.
- Version 4—the same as version 3, but adds the Blocked Categories field for DNS and Proxy logs.
Version 1 Bucket Recreation
If you are on version 1, you will need to remove your existing S3 bucket, disable the integration, then create a new bucket from scratch. For all other versions, you can upgrade from the Log Management screen of the Umbrella dashboard by clicking the Upgrade button.
Logs are uploaded to S3 buckets in the appropriate subfolder with the following naming format.
<subfolder> will either be dnslogs, proxylogs, or iplogs, depending on the types of logs within. <xxxx> is a random string of four alphanumeric characters, which prevents duplicate file names from being overwritten.
DNS logs show traffic that has reached our DNS resolvers. A typical snippet of DNS logs will look like this:
"2015-01-16 17:48:41","ActiveDirectoryUserName", "ActiveDirectoryUserName,ADSite,Network", "10.10.1.100","126.96.36.199","Allowed","1 (A)", "NOERROR","domain-visited.com.", "Chat,Photo Sharing,Social Networking,Allow List"
- Timestamp—When this request was made in UTC. This is different than the Umbrella dashboard, which converts the time to your specified time zone.
- Policy Identity—The first identity that matched the request.
- Identities—All identities associated with this request.
- InternalIp—The internal IP address that made the request.
- ExternalIp—The external IP address that made the request.
- Action—Whether the request was allowed or blocked.
- QueryType—The type of DNS request that was made. For more information, see Common DNS Request Types.
- ResponseCode—The DNS return code for this request. For more information, see Common DNS return codes for any DNS service (and Umbrella).
- Domain—The domain that was requested.
- Categories—The security or content categories that the destination matches. For category definitions, see Understanding Security Categories and Understanding Content Categories.
- Policy Identity Type—The first identity type matched with this request. Available in version 3 and above.
- Identity Types—The type of identity that made the request. For example, Roaming Computer, Network, and so on. Available in version 3 and above.
- Blocked Categories—The categories that resulted in the destination being blocked. Available in version 4 and above.
Proxy logs show traffic that has passed through the Umbrella Secure Web Gateway or the Selective Proxy. A typical snippet of proxy logs will look like this:
"2017-10-02 23:52:53","TheComputerName","ActiveDirectoryUserName, ADSite,Network","188.8.131.52","184.108.40.206","","ALLOWED", "http://google.com/the.js","www.google.com", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36","200", "562","1489","","","","","","","","Networks"
Not all fields below are actually used in most or all requests and are included for future enhancement.
- Timestamp—The timestamp of the request transaction in UTC (2015-01-16 17:48:41).
- Identities—Which identities, in order of granularity, made the request through the intelligent proxy.
- Internal IP—The internal IP address of the computer making the request.
- External IP—The egress IP address of the network where the request originated.
- Destination IP—The destination IP address of the request.
- Content Type—The type of web content, typically text/html.
- Verdict—Whether the destination was blocked or allowed.
- URL—The URL requested.
- Referer—The referring domain or URL.
- userAgent—The browser agent that made the request.
- statusCode—The HTTP status code; should always be 200 or 201.
- requestSize (bytes)—Request size in bytes.
- responseSize (bytes)—Response size in bytes.
- responseBodySize (bytes)—Response body size in bytes.
- SHA—SHA256 hex digest of the response content.
- Categories—The security categories for this request, such as Malware.
- AVDetections—The detection name according to the antivirus engine used in file inspection.
- PUAs—A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
- AMP Disposition—The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
- AMP Malware Name—If Malicious, the name of the malware according to AMP.
- AMP Score—The score of the malware from AMP. This field is not currently used and will be blank.
* Identity Type—The type of identity that made the request. For example, Roaming Computer, Network, and so on.
- Blocked Categories—The category that resulted in the destination being blocked. Available in version 4 and above.
IP logs show traffic that has been handled by the IP Layer Enforcement feature. A typical snippet of IP logs will look like this:
"2017-10-02 19:58:12","TheComputerName","220.127.116.11", "55605","18.104.22.168","443","Unauthorized IP Tunnel Access"
- Timestamp—When this request was made in UTC.
- Identity—The first identity matched with this request in order of granularity.
- Source IP—The IP of the computer making the request.
- Source Port—The port the request was made on.
- Destination IP—The destination IP requested.
- Destination Port—The destination port the request was made on.
- Categories—Which security categories, if any, matched against the destination IP address/port requested.
For more information on the IP Layer Enforcement feature, see Add IP Layer Enforcement.
Cloud Firewall logs show traffic that has been handled by network tunnels. A typical snippet of Cloud Firewall logs will look like this:
"2019-01-14 18:03:46","","Passive Monitor", "CDFW Tunnel Device","OUTBOUND","1","84","172.17.3.4","","22.214.171.124", "","ams1.edc","12","ALLOW"
- Timestamp—The timestamp of the request transaction in UTC.
- originId—The unique identity of the network tunnel.
- Identity—The name of the network tunnel.
- Identity Type—The type of identity that made the request. Should always be "CDFW Tunnel Device".
- Direction—The direction of the packet. It is destined either towards the internet or to the customer's network.
- ipProtocol—The actual protocol of the traffic. It could be TCP, UDP, ICMP.
- packetSize—The size of the packet that Umbrella CDFW received.
- sourceIp—The internal IP address of the user-generated traffic towards the CDFW. If the traffic goes through NAT before it comes to CDFW, it will be the NAT IP address.
- sourcePort—The internal port number of the user-generated traffic towards the CDFW.
- destinationIp—The destination IP address of the user-generated traffic towards the CDFW.
- destinationPort—The destination port number of the user-generated traffic towards the CDFW.
- dataCenter—The name of the Umbrella Data Center that processed the user-generated traffic.
- ruleId—The ID of the rule that processed the user traffic.
- verdict—The final verdict whether to allow or block the traffic based on the rule.
Updated 2 months ago