What and How much to Collect concentrated on giving a starting point for log collection and usage. Here, I am going to talk about Log Filtering for Security Investigations and Analysis purpose. If you are collecting logs from a troubleshooting perspective, you can collect without filtering, but for Security analysis, we would need less of haystack so that we can have a better chance of finding the needle.
In organizations, where several thousands of devices are sending Syslog, managing them is really a nightmare. In such scenarios, more focus is required on Log filtering. Log Filtering for Security Infrastructure use is one of the most ignored aspects of Log Management but it is the most important aspect for a cleaner and efficient log management and Security Event Analysis.
Log Filtering can be done at the Source or at the Syslog Server Location.
- Filtering at Source is the best approach when it comes to Log Filtering. This provides for better control of your infrastructure as you know what is being logged and what is not. Indirectly, it also aids in less utilization of System Resources. But the downside is, this is the most difficult to implement. Network devices have very little capability of Source filtering. Firewalls have a lot of options when it comes to Logging, but Routers and Switches offer very less. UNIX offers basic filtering natively and with third party tools like rsyslog, syslog-ng offer granular filtering capabilities. As all would know, there is no native Syslog capability in Windows and it would typically need a third party client to fill in for this lack of capability. This works in favor of filtering because; generally Third Party tools give filtering options.
- Filtering at Destination is the most practical and easiest to implement when it comes to Log Filtering. Here, Log Management Solutions or SIEM Tools take care of filtering what is needed and what is not. However, your Source Devices will be generating tons of logs and will be using System resources as well as Bandwidth.
Based on your organizations Architecture, it will help to decide on what is easier to implement and manage. Both have their pros and cons and its up to the Security Teams to decide. Once the filtering approach is decided, it is time to move on to the “Filtering” itself. This is the hard part. Before you start filtering, start understanding the device families present in the Environment. Every Vendor logs events differently and every event log means differently as well. So, care should be taken that we don’t end up filtering logs for one version and in reality we have another.
Example Log Filtering for Cisco ASA:
Let’s take an example of a Cisco ASA Firewall. The Cisco Adaptive Security Appliance (CISCO ASA in short) Operating System generates several logs. Out of the several thousands of messages, the most important events from a Security perspective are the following events given in the table. These events are built in protection defaults for Cisco ASA appliances. Apart from the following events, we can pick and choose Traffic Based events for analysis purpose. If interested to receive the full list, please comment below and I will put up that list too.
| %PIX|ASA-2-106016 | Deny IP spoof from (IP_address) to IP_address on interface interface_name. – Spoofing Detection |
| %PIX|ASA-2-106017 | Deny IP due to Land Attack from IP_address to IP_address – Spoofing Detection |
| %PIX|ASA-2-106020 | Deny IP teardrop fragment (size = number, offset = number) from IP_address to IP_address – Teardrop Attack |
| %PIX|ASA-1-106021 | Deny protocol reverse path check from source_address to dest_address on interface interface_name – Protocol Reverse Attack |
| %PIX|ASA-1-106022 | Deny protocol connection spoof from source_address to dest_address on interface interface_name – Spoofing Detection |
| %PIX|ASA-3-109010 | Auth from inside_address/inside_port to outside_address/outside_port failed (too many pending auths) on interface interface_name. (floodguard enable) |
| %PIX|ASA-4-109017 | User at IP_address exceeded auth proxy connection limit (max) – DOS Attack Possible |
| %PIX|ASA-4-109022 | exceeded HTTPS proxy process limit – DOS Attack Possible |
| %PIX|ASA-5-111001 | Begin configuration: IP_address writing to device – Privilege Use |
| %PIX|ASA-5-111003 | IP_address Erase configuration – Privilege Use |
| %PIX|ASA-6-113006 | User user locked out on exceeding number successive failed authentication attempts – Brute Force |
| %PIX|ASA-3-201002 | Too many TCP connections on {static|xlate} global_address! econns nconns – DOS Attack Possible Behavior |
| %PIX|ASA-3-201004 | Too many UDP connections on {static|xlate} global_address! udp connections limit – DOS Attack Possible Behavior |
| %PIX|ASA-4-209003 | Fragment database limit of number exceeded: src = source_address, dest = dest_address, proto = protocol, id = number – DOS Attack Possible Behavior |
| %PIX|ASA-4-209004 | Invalid IP fragment, size = bytes exceeds maximum size = bytes: src = source_address, dest = dest_address, proto = protocol, id = number – Possible Intrusion Event |
| %PIX|ASA-4-209005 | Discard IP fragment set with more than number elements: src = Too many elements are in a fragment set – Possible Intrusion Event |
| %PIX|ASA-3-210011 | Connection limit exceeded cnt/limit for dir packet from sip/sport to dip/dport on interface if_name – DOS Attack Possible Behavior |
| %PIX|ASA-4-405001 | Received ARP {request | response} collision from IP_address/MAC_address on interface interface_name to IP_address/MAC_address on interface interface_name – ARP Poisoning |
| %PIX|ASA-4-405002 | Received mac mismatch collision from IP_address/MAC_address for authenticated host – ARP Spoofing |
| %PIX|ASA-2-410002 | Dropped num DNS responses with mis-matched id in the past sec second(s): from src_ifc:sip/sport to dest_ifc:dip/dport – Possible DNS Attack Detected |
| %PIX|ASA-4-412002 | Detected bridge table full while inserting MAC MAC_address on interface interface. Number of entries = num – Possible L2 Attack Detected |
| %ASA-4-424001 | Packet denied protocol_string intf_in:src_ip/src_port intf_out:dst_ip/dst_port. [Ingress|Egress] interface is in a backup state – Possible Intrusion |
| %ASA-4-424002 | Connection to the backup interface is denied: protocol_string intf:src_ip/src_port intf:dst_ip/dst_port – Possible Intrusion |
| %PIX|ASA-6-605004 | Login denied from source-address/source-port to interface:destination/service for user “username” – Possible Intrusion |
| %PIX|ASA-5-304001 | user@source_address Accessed {JAVA URL|URL} dest_address: url |
If you carefully note the structure of the Event message itself you will notice that almost all of the Syslog Messages I have chosen have facilities that range from 1 to 5. These Event IDs can be used for analysis of different scenarios. Here, I have listed down some pointers on how to select these Events of Interest. Some approaches that we can use are:
- Focus on the Utility of an Event rather than the data it generates. The description of an Event ID will be misleading many a times, and hence looking at the Generated Data from the Event is important.
- Focus on what events will help you track back the actual Attacker/User/Service responsible for triggering the Event in the first place. This event independently or in tandem with another should be able to help you
- Look for what Event Collection is more suited to your environment in terms of Security Monitoring. For Example, if you have an Internet Firewall, Spoofing events don’t make sense to collect, whereas internal firewalls would need spoofing events.
- Look for Optimal Event ID Selection for auditing so that you get the right amount of data with One Event ID rather than many. This may be difficult in some cases with some devices, but still where ever possible this should be employed.
- Finally, see if your SIEM tools can parse these event properly for you to analyze
Now let us look at Juniper NetScreen Firewall Events using the same approach as we used for Cisco ASA.
| Critical (00031) Message | 〈string〉 detected an IP conflict (IP 〈IP address〉, MAC %m) on interface 〈string〉 |
| Notification (00031)Message | ARP detected IP conflict: IP address 〈ip〉 changed from interface 〈if_old〉 to interface 〈if_new〉 |
| Notification (00051) Message | Static ARP entry added to interface 〈string〉 with IP 〈IP address〉 and MAC %m |
| Notification (00052)Message | Static ARP entry deleted from interface 〈string〉 with IP address 〈IPaddress〉 and MAC address %m |
| Emergency Message | Ping of Death! From 〈src_ip〉 to 〈dst_ip〉, proto 1 (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Emergency Message | SYN flood! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto TCP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Emergency Message | Teardrop attack! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto { TCP | UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number2〉 times. |
| Alert Message | Address sweep! From 〈src_ip〉 to 〈dst_ip〉, proto 1 (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Alert Message | ICMP flood! From 〈src_ip〉 to 〈dst_ip〉, proto 1 (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Alert Message | IP spoofing! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto {TCP | UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Alert Message | Land attack! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto TCP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Alert Message | Port scan! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto { TCP| UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number2〉 times. |
| Alert Message | Source Route IP option! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto { TCP | UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number2〉 times. |
| Alert Message | UDP flood! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto UDP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Alert Message | WinNuke attack! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:139, proto TCP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Critical Message – | Bad IP option! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto{ TCP | UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉).Occurred 〈number2〉 times. |
| Critical Message | EXE file blocked! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto { TCP | UDP | 〈number1〉 } (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number2〉 times. |
| Critical Message | FIN but no ACK bit! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉, proto TCP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
| Critical Message | SYN and FIN bits! From 〈src_ip〉:〈src_port〉 to 〈dst_ip〉:〈dst_port〉,proto TCP (zone 〈zone_name〉, int 〈interface_name〉). Occurred 〈number〉 times. |
If you see in the case of Juniper, Alert and Critical messages are sent to Syslog and these events are not always identifiable using Event IDs. From a Syslog facility all these would be still at Level 5. These events are also default Security Messages generated from Juniper. In addition to these messages, we would be collecting the Traffic Logs (Event ID 000257) from Juniper that helps in Security analysis for correlation.
This same approach can be used for Windows, Unix, Security Application Devices etc. For more details of specific Device type filtering, please request in the comment section and I will post the filtering for the same.
If you effectively identify the Events of Interest for various devices and filter out the chaff, you would be able to harness the power of Syslog for your Security Investigation purposes. SIEM is getting bigger by the day in terms of data usage. The world is moving towards big data collection. But the key thing to note is, in spite of data growth, the quality of SIEM tools in log analysis has not been going North. Quality does not essentially come with Quantity. Filtering will always be needed no matter how much logs we collect.
Remember “Logs don’t lie”
Again its very important Topic which usually missed due less attention of Senior executive. I am glad that you brought this topic.
I would like add few things.
1. Device filtering is also important like if you are not collecting logs for troubleshooting purpose see the options of removing Switches.
2. you can see option of remove duplicate logs for active active devices generating same kind logs, Active directory replication logs are same on all DC. you might wanna collect from one or two source.
3. Windows logs again consumes lot of Space see if you can filter logs out of six default like DNS server( ( if you have are not using AD DNS),application logs.
1. Device Filtering is important Somesh, but Switch Logs are really important because of several scenarios involving L2 attacks. Switches don’t log that much information at Level 5. So we would really not face an issue on Log Volume.
2. In case of Network devices, HA Pairs don’t have any duplicate issues if configured correctly. However, for DC Replication logs are classified as System Health logs and not security. Hence we would not need them. However, every DC Security log is needed to counter the various scenarios involving “weak link compromises”
3. Windows filtering is a really important topic and requires a lot of analysis and time. I have done some filtering for Windows based on certain analysis. I can share with you through mail the same.
Let me know
You really make it seem so easy together with your presentation but I in finding this matter to be really something which I feel I might never understand. It sort of feels too complex and extremely extensive for me. I’m having a look forward for your next publish, I will attempt to get the hold of it!
It is actually easy once you understand the intent of doing this. Let me know what you would need a better understanding of and I will be glad to help
Hello i discovered your website at some other blog and i think the information here is something very good and i have never been on this site before. I Will come back more and see your updates.
Thanks for visiting. Please follow for regular updates!!!
This information are very helpful.
Do you have the same for Windows, Unix and Database like MySQL or Oracle?
For Oracle, I have a post in the website here.
For Windows and Unix, it all boils down to what is important from your organisation’s policy.
Did you post the Traffic Based events? Great post, very helpful, btw.
If interested to receive the full list, please comment below and I will put up that list too.
I am interested, do you have anything else like on Checkpoint and/or routers. Your blog is an excellent resource for information about SEIMS. I appreciate you taking the time to share it and it is appreciated.
Yeah I would like to see for the Unix and windows or like cisco ASA, AV, etc. tks
Hi,
I’m analyzing Cisco ASA Inbound and Outbound traffic log message. Could you please provide a complete list for Inbound and Outbound traffic events
Regards,
Shalendra
I would like to have the critical syslogs messages for juniper srx, palo alto, fortigate and checkpoint firewall as given for cisco asa
Oh wow… This is quite a bit of challenge and involves a lot of manual effort. One of the easiest ways to do this is look at the Syslog Message Library available in the help files of these products and look for major categories. For example, UTM devices like Palo Alto, Checkpoint and Fortigate, it will be beneficial to get the Web filtering with URL. As far as traffic logs are concerned, Connection Accepted is more than enough. Denied connections need not be logged. And so on and so forth.
I am sorry that a readymade list is not available for you.
Plz email me the full list. [email protected]
Please email or post the full lists, and any you might have from above comments, thanks in advance
please send me a full listing of important ASA Security Events.
please send me a full listing of important ASA Security Events.
hi,
can I have full list of events for db and ips/ids to kamil.s.bernas(at)gmail dot com ?
Hi,
Can you please mention for what keyword we should look into to detect top 10 web attack i.e cross site scripting, A1 to A10.?
Thank You.
There is no specific keyword per say. It is entirely dependent on the Application log source you collect from. Depending on the business logic and the XSS method, you can create a log correlation logic
Can you please send me a full listing of important ASA security events? Thank you.
Can you please send me a full listing of important Firewall’s (ASA, Juniper, Checkpoint, PaloAlto) security events ? Thank you
As mentioned previously on the comments here, the events are up to the teams to decide on what they want to monitor for.
I have started filtering out noisy events generated by cisco ASA. I want to know whether we can filter build inbound/outbound ICMP connection and Teardown ICMP connection.
You could filter those messages if you are doing IDS/IPS
Please send me a full listing of important ASA Security Events. Good to know information!
Please send me the full list for ASA.