Mehmet Ergene

Threat Hunting and Detection Using Web Proxy Logs

Web proxy logs are a valuable source of telemetry for identifying suspicious behaviors and uncovering potential threats in your environment. These logs typically include details such as Duration, HTTP Status, Bytes In/Out, Protocol, HTTP Method, HTTP Version, URL Category, Hostname, Path, Query, MIME Type, File Name, and User-Agent.

This post outlines how each of these fields can be leveraged for threat hunting and detection use cases.

Duration

This information shows how long the transaction has taken. Malware can communicate with the C2 server over the HTTP(S) protocol. When this is the case, it asks for commands periodically. This period doesn't have to be a constant value like every 10 minutes. Malware can also use jitter to make random-looking requests. Also, keeping the connection open can also be used by malware. In any case, it needs to either ask for commands very often or keep the connection open.

Technique

Aggregate total connection duration per SourceIP-DestinationIP over a 12/24-hour period.

What to look for

Higher values may indicate beaconing. Keep in mind that not all beacons are malicious. That's why we are hunting.
Note: If you apply the same method to your public websites, you can detect web scraping or customer data scraping.

HTTP Status

Users visit websites, post something, sometimes upload some data, or download a file. In normal conditions, these transactions have an HTTP 200 result. When it comes to malware, it is possible to use HTTP error codes as a C2 channel. Also, most malware use DGA(domain generation algorithm) in order to keep the connection persistent if one of the domains is blocked. In such a case, the malware keeps getting HTTP errors and tries the next domain.

Technique

  1. Count HTTP status codes per SourceIP or SourceIP-DestinationIP over a specific time period.
  2. List URLs having only HTTP Errors.

What to look for

  1. Higher values of an uncommon HTTP Status Code may indicate C2 activity.
  2. Higher values of HTTP errors for a website can indicate failed C2 activity.

Bytes In

In normal conditions, when a user visits a website, downloads a file, etc., each transaction is made with a different data size. On the other hand, malware visits the same page(URL) every time. This makes the downloaded content has the same size unless the attacker starts interacting with the victim machine.

Technique

  1. Count the BytesIn per Source-Destination pair over 12/24 hours. You have the best chance when the attackers sleep as there is no interaction.
  2. Compute the ratio of count(BytesIn) per Source-Destination pair. This is for comparing the attacker interaction versus idle status.

What to look for

  1. Higher values may indicate beaconing. C2 servers reply with the same data, making Bytes In value the same.
  2. Higher values of ratio may indicate C2 beaconing.

Bytes Out

A normal user activity consists mostly of downloading data. Uploaded data is usually small unless there is a file/data upload to a website.

Technique

  1. Compute the total BytesOut per Source-Destination pair over 12/24 hours.
  2. Compute the ratio of count(BytesOut) per Source-Destination pair over 12/24 hours.

What to look for

  1. Higher values may indicate data exfiltration.
  2. Higher values of ratio may indicate beaconing.

HTTP Method

In normal circumstances, a user's web traffic contains a large amount of HTTP GET, a small amount of HTTP POST methods. Other HTTP methods, such as HTTP PUT, are expected to be seen less.

Technique

Compute the ratio of the POST or PUT over GET per Source-Destination over 4/8/12/24 hours.

What to look for

Higher values of ratio may indicate beaconing or exfiltration.

URL Hostname

Usually, a user visits websites that are in the top 1M list. In some cases, an unpopular website can be visited by lots of users as well (think about 3rd parties having business with the company).

Technique

  1. Compare with top 1M domains and calculate the visit count.
  2. Compute the visit count per Hostname.

What to look for

  1. Hit count <5 and Hostname is not in the top 1M may indicate malicious payload delivery.
  2. Small number of hit count may indicate malicious payload delivery.

URL Path

C2 beacons usually use the same URL path for C2 communication.

Technique

Compute the count per Source-Destination-URLPath pair.

What to look for

Higher values may indicate beaconing.

URL Query

URL query information is seen when you search for an item on a website. Malware does the same when asking the C2 server if there is anything to run on the victim machine or sending its ID. The query can be encoded/encrypted as well.

Technique

  1. Compute the count per Source-Destination-URLQuery.
  2. Compute the length of URLQuery.
  3. Look for base64 encoded strings in URLQuery.

What to look for

  1. Higher values may indicate beaconing.
  2. Higher values may indicate encoded data, a sign of exfiltration or beaconing.
  3. Encoded strings may indicate beaconing or exfiltration.

Mime(Content) Type

Unfortunately, most web proxies fail to determine the exact type of content.

Technique

List mime type per Source-Destination pair.

What to look for

Uncommon mime types may indicate a malicious file.

User Agent

Normally, all applications have their own user agent information. Malware can try to mimic a legitimate application user agent but sometimes fail to do that with a small typo.

Technique

Calculate count within the environment(long tail analysis).

What to look for

Lower values may indicate a malicious binary existence.

URL Category

In most environments, there are commonly blocked web categories like Hacking, Pornography, Dynamic DNS, etc. Uncategorized websites are a pain and sometimes this category has to be allowed for the sake of business continuity.

Technique

Query for Uncategorized, Dynamic DNS, and other suspicious categories. Compute the distinct count of SourceAddress by URLHostname.

What to look for

Small dcount values may indicate abnormal/suspicious/malicious activity. If an uncategorized URL is visited by many users, it is less likely that the URL is malicious.

HTTP Version

There are five HTTP versions — HTTP/0.9, HTTP/1.0, HTTP/1.1, HTTP/2.0, and HTTP/3.0. The current version is 1.1 and the future ones are/will be 2.0/3.0

Technique

Analyze HTTP versions.

What to look for

HTTP/0.9 and HTTP/1.0 are old. This may be an indication of malicious activity.

Protocol

Web proxies are able to determine the protocol by analyzing the traffic.

Technique

Compare ports with their standard protocols.

What to look for

Common Protocol-Uncommon Port or Common Port-Uncommon Protocol may indicate malicious traffic.

File Name

It's not always possible to log reliably the names of the files that are downloaded from the internet. If it's logged properly, file names can be used for hunting. Some malware droppers download randomly named files.

Technique

  1. Entropy analysis on filenames.
  2. Lenght of the filename

What to look for

  1. High entorpy may indicate malicious payload delivery.
  2. Short file name may indicate malicious payload delivery(1.bat, 3.exe, etc.).

Conclusion

Web proxy logs offer a rich dataset for identifying adversary activities. By applying these techniques, defenders can proactively surface indicators of compromise and improve detection coverage.

A downloadable PDF cheat sheet summarizing these techniques is available on my GitHub repository. Feel free to explore!
Share this post

Latest from our blog