Category Archives: Security Investigation Series

Articles in this category are Lecture series that talk about various Security Investigation Process/Methodology for the various types of Security Incidents and Events identified.

Security Investigation Series

Episode 3 – Security Investigation Series – Should I press the panic button?

Gallery 3 Comments

 

Scenario:

  • A User has a machine installed with an Application Client and connects to the Web Application Server which in turn connects to the Database servers on the back-end.
  • The Client machine has been reported to be being slow.
  • Some basic SIEM based analysis of Network logs for the client machine shows some Network connections resembling a Port Scan from the client machine in its own subnet
  • The machine is also doing some random internet connections to Public IP Addresses.
  • On checking the Anti-Virus software on the machine it shows as being outdated.
  • Suspicion is that the machine is infected with some kind of a malware.
  • Since this client machine has connections to the Critical App and Database server, this event is critical

Should I press the panic button?

Before we press the panic button (raise an Official Security Incident which has SLA tagged to it), there are several questions we need answers for. This is something I learnt from one of my mentors (he blogs here). What I learnt was that before beginning a Security Incident investigation and press the panic button, we need to sit down and collect as much as corroborating data as possible. In order to do that we can follow a series of logical steps:

  1. Collect the Public IP Addresses it connects to and find out more about them. Sometimes a simple Google search can help to determine the authenticity and reputation of the website/domain or public IP.
  2. Establish Ground Zero for the infection. This can be determined by historical log data. Some SIEM tools do this natively for you. Else, you can use a white board or pencil/paper to draw this visualization yourself.
  3. Gather as much as possible infected machine IPs and sort them geographically, Network Zone based and if possible VLAN based as well.
  4. Other Transitive Infections that could be identified.
  5. Analyze one of the infected machines to gather Malware analysis data

Next Steps – Remediation, Control etc

  • Quarantine all the machines infected.
  • In case of the server, ensure that all instances (if any) of the infected file are cleaned.
  • In case of a database, if we exactly know the infection source and possible file names, try purging them from the database contents.
  • Submit the samples of the Malware to the Security Vendor and ask them to start working on a Signature or definition.
  • Perform a Reverse Engineering of the Malware to identify specific Network connections/Behaviors so that they can go into our prevention systems in the Network. (Read Network IPS, Host IPS, Proxy etc). Reverse Engineering skill set  is a good skill to have because for Zero day scenarios, vendors are slow to react. If a Reverse Engineer is available, the job of getting a control measure put in place (although a crude one) will be quicker.
  • Document the characteristics of the malware, the response taken and the findings during the course of this entire incident.
And Finally, if you did all of the above after following the Security Investigation Series blog, say thanks!!!
Security Investigation Series

Episode 2 – Security Investigation Series – Reverse Protocol Attack

Gallery 3 Comments
Preface: The image in the blog is a capture from the Cisco Firewall Syslog Message guide explaining an Event regarding a Reverse Protocol Attack. Cisco tells you that the packet is dropped and that the firewall defended you from a Spoofing attack. Such events are dicey in nature because you don’t know what to do, how to react and how to respond to these events in Security Operations.As with every Investigation, everything begins with a “Tip”The Tip:

  • The Event itself – In this case Cisco PIX/ASA Firewall message
  • The IP source from which the packet originated
  • The timestamp at which the event occurred
  • The Firewall from which the alert originated

Investigation Methodology:

We need to identify “who” is the person performing this attack. Since this is a firewall, all we can get is a IP address from where this originated. But from an investigation perspective, we need to identify the “real machine”doing this. If it is an internal segment or VLAN in the enterprise network we can do a few things like below:

  • Tracing the MAC address from which the IP originated. For this we would typically need Switch Logs and/or DHCP logs.
  • Identify the IP and MAC pairings logs during the said time period. Watch out for MAC address Spoofing scenarios as well.
  • Based on these two data sources we will be able to identify the physical machines which are possible candidates for this attack
  • In MAC Spoofing scenarios, looking at the physical machine will help. There is a Microsoft API that helps you gather this real MAC information. I have seen some tools that provide this information as well.
In case of Internet facing Firewall, this spoofing detection is going to be a tough ask. We may not have a capability to identify the source performing the spoof. So if the Internet Firewall throws this alert, its safe to ignore as Cisco recommends, whereas if the Internal Firewall throws this alert, it needs to be investigated to identify insider attack scenarios involving spoofing.
Detection:
Detection of this event is the simplest in SIEM terms. The event ID is unique for Cisco devices and hence they can be triggered directly. These alerts can be filtered for internal firewalls only unless and until the Internet firewall is overwhelmed with these alerts. In case of enormous amount of alerts from Internet firewall, we may have to look at potential DoS scenarios. Several threshold based rules can be created in SIEM to look for such attack patterns
Prevention:
In case of Internet facing Firewall, uRPF filtering definitely helps. There are several different modes for enabling uRPF. Before enabling we need to understand whether our network is routed symmetrically or not. In case of symmetric routing, the uRPF filtering is straight forward and we can go for Strict Mode, but in case of asymmetric routing, the filtering is not easy and hence we may configure uRPF in Loose Mode or Feasible Mode. There are several articles from Cisco on uRPF that can serve as good references.
Conclusion:
In essence, every event means different things in different scenarios. Placement of the device in the network and its relevance gives a new meaning to the event triggered. Investigation of every “event type” is mandatory in Security Operations. Results of such analysis should be fed back into a Knowledge Base that will help the enterprise react to several different alerts in the right way.
Update based on the comments received:
For scenarios where both IP and MAC spoofing is involved, there is no easy way of identifying the real machine. The closest in case of Internal Network would be to look at the Switch Network to identify the source segment of the MAC thereby narrowing done on the investigation. But even after narrowing down, you will have to go down to the Asset Enumeration level to exactly identify the spoofing host. This enumeration should be done in tandem with packet captures in the switched network. One of the other things to look at in packet captures would be to look for the presence of a Locally administered address bit. This will be set in case of a spoof. Smart hackers know this and can circumvent it by a custom driver code, but this is just another pointer of what we can look for.
Security Investigation Series

Episode 1 – Security Investigation Series – Torrents

Gallery 4 Comments

Preface: I would like to share a methodology I used to track down an individual who was involved in downloading illegal/copyrighted movies resulting in a legal lawsuit for the company. In this post, I would try to simplify the entire thought process into simpler steps so that the reader can get an understanding on how Security Investigation can be done. Several organizations block torrent use in the Network, but several organizations don’t have such a policy in place. This investigation methodology is for people who are part of organizations of the second type 🙂

The Tip: Every Security investigation starts with a tip. In this particular case, there were three tips to work off

1. Public IP Address Registered to the Company
2. File Name of the Torrent Download
3. Time Stamps of the Download

More and More Information – Logs

Based on this above information it will be very difficult to identify who was the exact person who downloaded the Torrent. In order to do this Security Investigation, we would need logs from several devices in the IT Infrastructure. The places from where we would collect these logs can range from Network Devices to Hosts. It really depends on the level of logging enabled in the enterprise. If there are no logs, identifying this type of incident would be impossible. More and More information will help in the investigation process.

What and How much Logs to Collect is a section where I talk about typical devices in IT Infrastructure and common ways to collect, filter, analyze and store logs. For this particular investigation, the following logs were of great use:

1. Firewall Logs
2. IDS/IPS Logs
3. Authentication Server Logs
4. VPN Logs
5. DHCP Logs (if any)

Apart from the logs, we would need Corporate Network Awareness, Good Investigation skills and Patience.

Methodology:

1. Based on the Public IP (given in Tip 1), we can determine the Firewall from which the Traffic has been detected. This is critical in the case of Global enterprise wide Networks because, there are several firewalls deployed World over with several different Public IP addresses. Zeroing on a specific Firewall will help reduce the amount of logs to look for.

2. The next thing is to look for the above Identified Firewall logs during the Time Stamps of the event(given in Tip 3) to narrow it down to a manageable subset (this would still be Gigs of logs in case of high volume firewalls).

3. Now is the time to filter the Firewall Logs for P2P Traffic pattern. If an IDS/IPS is available in the Network and is connected to the said firewall, it makes the job easier to filter on all P2P Activity based on P2P Detect Signatures. If no IDS/IPS is available then we will have to follow a manual and time-consuming analysis for P2P.
P2P Analysis can be done in two ways. One is to look for Filter for UDP connections and see Torrent Client Port Activity. The other way is to look for Network Devices logs for Application Protocol detection capability to identify P2P activity (Symantec has a Detailed Article on Traffic Analysis of P2P here). Once the analysis for P2P is done, we can narrow down on the P2P Client set. This set is the list of Client machines that were involved in doing P2P activity during the said Timeframe (1 hour before and after the timestamp in Tip 3. This is done in order to ensure that we capture as much P2P traffic events as possible to be able to determine the exact client involved in this activity. Also, in highly utilized Network or in High Log volume environment, latency is an issue therefore having a buffer time frame is always a good option).

4. Once we have the subset of logs from the above Step, its time to identify the list of Source IP Addresses that are P2P Clients. This is present in both the IDS/IPS logs as well as Firewall Log. Doing a Union of these two data sources to narrow down the list of IP Addresses should be easy.

5. With the IP Addresses in hand, it is time to identify the hostname these IP addresses were assigned to during the specified time frame. DHCP Logs (In case of dynamic environments) can give this information. In DHCP Logs, we would get Hostname, IP and MAC Address of all the machines that have been allocated an IP within the said time frame. In case of Static IP Addresses, we would already have the information about the Hostname based on the IP addresses assigned.

6. At this juncture, we have the List of IP Addresses, the Associated Hostnames and MAC Addresses of the suspected clients. Based on the Authentication Server Logs, we can identify the Usernames associated with the above information set. Authentication Server Logs in this case are Windows (AD Logs). By filtering on the Logon events for the IP/Hostname we can identify the users.

Note: In case of VPN Connections, the VPN Connection entry has the user name information as well because VPN has a backend Authentication Component to it.

7. Once the Users are identified, we need to proceed with the Forensic Analysis of the individual machines, either remotely or physically. The clue to identifying the exact Person who did this, would be to search for the specific Torrent Filename’s (given in Tip 2) presence in the Torrent Application directory (Chances are they are not deleted by the user) or in the entire file system. The Movie file may also be present if we are lucky. In case of Movie sharing between users, its highly likely you would see Torrent + Data on one machine and Data on all other machines.

8. Once the person is identified, the appropriate teams should be notified as per the Security Investigation procedures defined for the Organization.

Note: The entire Log Collection and Analysis can be done quite easily if a SIEM Tool is available in the Enterprise. Else, it will be a painstaking job involving few hours of Analyst Time.

Conclusion:
A Security Investigation is possible only when we have data in the right format. Right tools as well as Right mindset is also important in the Investigation process. As mentioned earlier in the post, there is no right or wrong way in Security Investigation, only a way that works best for the Current Situation and Current IT Infrastructure Setup. I would greatly welcome suggestions and additions to this post so that I can make this series more and more useful.