While this analysis was done on alerts triggered on a McAfee IPS, the below applies to any IPS device, there might be minor differences in the way some tasks need to be done like how the logs are extracted and which format (CSV, PDF, etc) can be exported, regardless, the concept is the same!

What’s Going On?

For one customer, the IPS report showed a large number of alerts triggered for TCP probe signatures, we needed to analyze those in order to find out why these packets exist in the network.

What Are These Signatures?

NMAP network scanner offers a lot of options when it gets to scan techniques, specifically, XMAS, NULL & FIN scans are used to get around some stateful firewall restrictions and are supposed to be more stealthy than the traditional SYN scan, of course that was in the past… Now these scans light up the IPS logs of any modern device like a christmas tree.

Nmap scan techniques include the TCP Null, Fin & Xmas scans

What Is Common Between Those Alert Signatures?

All these signatures trigger alerts when abnormal TCP traffic is detected, by abnormal it means traffic that shouldn’t exist as per the TCP protocol standards, In my opinion, all these signatures should all be consolidated under one signature for unusual TCP scan traffic or OS fingerprinting & stateful firewall circumvention as they are very similar in nature, For some alerts, even two signatures are triggered for the same packet!

True or False Positive?

Just to be clear, these signatures never trigger a false positive alert, under normal circumstances there should never be a TCP packet with a SYN/FIN flag combination so these alerts are always for true positive scans because the signature pattern is very unique to anomalies in the TCP protocol standard… Now we need to find out what else the source IP of the scans is doing and whether this machine is authorized to carry out scans in the network, there are two possibilities here:

-A vulnerability scan is being run from this machine but the IPS administrator is not aware so no coordination between two entities.

-A machine is already compromised and scanning the rest of the network or someone is experimenting with NMAP 🙂

Signatures To Be Checked

NMAP: XMAS Probe

A TCP packet is detected with FIN/PSH/URG set, for example in the packet capture below taken by the IPS

Alert triggered by IPS engine when a packet with FIN/PSH/URG flags are detected

NMAP: XMAS With SYN Probe

A TCP packet is detected with FIN/SYN/PSH/URG set, for example in the packet capture below taken by the IPS, same at the previously mentioned XMAS Probe scan but here there is an additional SYN flag.

Alert triggered by IPS engine when a packet with FIN/SYN/PSH/URG flags are detected

TCP: Illegal FIN Probe

This signature is triggered when the IPS detects a TCP packet with unusual TCP flags combination, in particular SYN/FIN flags set at the same time since the SYN packet is supposed to start a three-way handshake while the FIN packet should end the TCP session so they shouldn’t co-exist in the same packet.

Alert is triggered when a TCP packet with an unusual flag along with a FIN packet

SCAN: NULL Probe

This alert is triggered when the IPS engine detects a TCP packet with no flags set, same for the other signatures, a packet like this should never exist in a normal TCP session, after all what is the purpose of a TCP packet without even the ACK flag set?

A TCP packet with no flags set at all is indeed very abnormal

SCAN: SYN FIN Based Probes

As the name implies as well, a packet with both SYN/FIN flags set, other flags may be set as well…

Triggered by a packet with both SYN & FIN flags set

How to investigate these alerts?

In order to investigate most if not all of the alerts detected on the IPS, we need to gather some statistics about the attack in order to arrive at meaningful conclusions, we need to least find out the top attackers’ IP address, top destinations, ports, etc…

Gathering some statistics

The first step is to identify and determine the host(s) triggering the largest number of the alerts, in our case, the host(s) scanning the network, for this we will need to:

-Obviously, filter the Attack Logs for attack of interest.

-Generate a report for this attack only or download the filtered attack logs in CSV format for a large period of time to be able to make better conclusions, as always, the more statistics the better!

Saving the attack log for a large period of time in CSV format is a quick and easy way to gather statistics

After the attack log is downloaded in CSV format, there is some basic excel work or whatever spreadsheet application is being used, first some extra blank columns need to be deleted and then we create something that is very useful to generate statistics, a pivot table… We will insert one using the data in the sheet to show which source IP’s trigger each signature most.

In order to create a clean PivotTable, we need to get rid of those blank rows in the CSV

Click on Insert > PivotTable to create the PivotTable in another sheet

After the PivotTable has been created, we can clearly see the top offending IPs, now we are ready to proceed with the next step.

The PivotTable shows clearly the top offenders’ IP addresses

Diving Deeper Into The Cause

Up to this point, we have enough to move on and alert whoever we needs to be alerted to find out the purpose of this scan and investigate the source IPs if necessary, but it would be better and more professional if we can provide more insight into the top offending IPs.

Packet Capture, Why Not?

A packet capture is recommended for a number of reasons, consider at least : Isn’t there a possibility that this top offending IP address is also doing other malicious activity that did not necessarily trigger an IPS signature?

Also, the Wireshark conversations window provides more insight to the attack pattern which confirms more the behaviour observed by the offending IPs.

I like to also capture all traffic to and from the offending IP addresses but one important thing to note is to set a limit to the file capture size of number of packets to be captured, this is enforced by most if not all the IPS devices to not fill the device’s storage.

Also for McAfee IPS, I needed to create one rule for each direction of traffic and two rules for each protocol (TCP, UDP & ICMP), so we have two rules for each direction and each protocol, six capture rules in total.

Preparing for a packet capture on McAfee IPS and creating some extra rules to capture also non-TCP traffic, why not?

After the packet capture is taken and downloaded, open up Wireshark and go to the Statistics menu > Conversations, it’s amazing how much insight can be revealed here.

In the conversations window, we can clearly see the scan pattern, same host is scanning other hosts at different ports

Conclusion

In our case, all the triggered alerts were sourced from two hosts in the network which turned out to be legitimate hosts probing the network, we ended up ignoring those alerts to reduce noise in the logs.

Final Words

The aforementioned attacks have a lot in common but in order to investigate them we needed to follow the same sequence that should be used to investigate most of the alerts that are triggered in huge numbers on the IPS, mainly collecting more statistics to arrive at better conclusions and to correlate between multiple triggered events, taking packet captures is highly essential as Wireshark offers a lot of useful tools like the conversations window that can also be used to further understand the root cause and nature of the alerts/signatures under investigation.

McAfee IPS : Analysis of XMAS Probe, Illegal FIN Probe, NULL Probe & SYN FIN Based Probe Signatures

A Quick Note