Category Archives: Security Investigation Series

Articles in this category are Lecture series that talk about various Security Investigation Process/Methodology for the various types of Security Incidents and Events identified.

Part 5 – Incident Recovery


Incidents can’t be avoided entirely, however the damage can be greatly minimized by a mature Incident Detection and Response function. In the CSIRT Series, we have been looking in detail at the various functions that make up a good IR process framework. Incident Containment and Incident Recovery are complimentary processes. While  containment is aimed at stopping the spread of a breach, Recovery is all about getting back on feet by reversing to a “Known good state”. The “known good state” in our opinion is very ambiguous in its meaning. It may apply to a single machine, or an entire network. However, in our opinion, Recovery process or getting back to a “Known good state” is a combination of three sub-steps:

  1. Pre-Recovery – Forensics Evidence Collection in our opinion is a Pre-Recovery step.  This is a critical process and is important for collecting and maintaining evidence that may be required to pursue future legal actions.
  2. Recovery from Backup – Ensure that systems or networks are returned to the pre-breach state.
  3. Post-Recovery –  As a post-recovery step, Remediation of the threat vector is crucial. A process to ensure that the infection or threat vector is a non-issue.

Let us look at each of these sub-steps in detail

Pre-Recovery – In cases which need legal course of action, it is important that we clearly document how all evidence has been collected, preserved and handled so that it is admissible in court. This is called Forensic Evidence Collection. It is key to note that legal requirements vary from region to region, jurisdiction to jurisdiction and a forensic person should be aware of that. It is recommended to have some of the team members obtain computer forensics training and certification to be able to handle the entire process end to end. However, it is not un-common to get professional third party help for conducting Forensic evidence collection and investigations during an Incident. Forensics is a standalone field in itself and to detail all the process steps her would be impractical. Hence, we have tried to give a succinct summary of what forensics entails:

    1. Determine legal issues regarding the incident that may cause an impact
    2. Determine technology and processes within the scope of the forensic analysis
    3. Identify evidence from the infected machine or person. The evidence can be electronic or physical.
      • Document and Collect the identified Evidence following the chain of custody
      • Perform Forensics Investigation and analysis.

Once the incident forensic process has been initiated, it is possible that the incident may need to be reclassified based on the results. Based on this, the entire recovery and remediation process attains a different color. For Example: A malicious code incident was originally triaged and classified as a medium security incident. The forensic analysis reveals that the malicious code has installed hidden back-door processes that can now be traced to additional systems that were not originally identified as being affected. The incident should be reclassified from a medium security incident to a high security incident.

Recovery from Backup – If the incident fits the criteria of high severity and or high impact, the CSIRT team should determine if IT business continuity, disaster recovery, and or backup restoration procedures should be initiated. The reason this is limited to high severity and high impact incidents is nothing but practical consideration. The goal of the Recovery phase is to safely put the impacted systems back into production. To complete the recovery process the following three steps have to be followed:

  • Validation  of the recovered systems  – Involves asking the user base if the system is operating properly or comparing that the ports and services of the system are consistent using profiling tools
  • Restoring Operations – Involves placing the system into full production, allowing it to interact fully with other devices on the network
  • Monitoring – Involves checking systems for back-doors or any other issues which may have escaped previous detection. If possible, host-based and network based monitoring should be used to compare that the attacker did not leave any back-doors on the system

Ultimately, when services are restored, the system should have an effective defence against future attacks of the same nature. Any access methods which may have been used to conduct such an attack should be corrected. When restoring services, systems or data from archived backups, consideration should be taken based on the type of attack, the data affected and most importantly the timeline in which the attack initially took place. This information should have been discovered and documented as part of the forensic analysis. This step in the recovery process is critical so that vulnerabilities, malware or corrupted data is not re-introduced into the operating environment. Depending on the severity of the incident, it may be required to do a full system rebuild in order to re-establish the integrity of the system.

Post-Recovery – Once the recovery is completed, Incident remediation steps should be followed. Most of the times, the Threat Vector will be a System vulnerability or Network vulnerability. For such vectors, available patches or system updates should be applied. System hardening techniques may also need to be applied and core deployment images may need to be updated to prevent the introduction of the weakness elsewhere in the organization. In the case of Non-Vulnerability related vector, the root cause should be identified and appropriate fixes have to be implemented.


It is important to have a well defined and smooth functioning recovery capability in the CSIRT team.  Without recovery capabilities, the probability of a security incident or issue recurring persists.
Go back or Continue to Part 6 – Continuous Improvement

Part 6 – Continuous Improvement


One of the things we do in the Incident Recovery phase is to  determine the root cause of the incident and to identify appropriate remediation steps. This typically follows the Root Cause Analysis workflow which many of you are aware of.  Once the remediation is done, it is important to document the “lessons learnt”.

Why is it important?

Lessons learnt are an important aspect of a CSIRT organization. “A stationary object gathers more moss”. This is the philosophy of a CSIRT organization – Continuous evolution and improvement. This is typically a 15 to 30 minute exercise every CSIRT member who handled the incident should go through. In this exercise, the following key items should be discussed:

  1. What process, technology or people worked?
  2. What did not work? Why?
  3. Response and Resolution effectiveness? Why?
  4. Any recurring issues or themes?

Once answers for all these questions have been penned down and discussed, a detailed action plan needs to be devised on how to improve the CSIRT function. The Action plan can be categorized under two major groups:

  • Control Improvements  – This section should describe any changes or improvements that should be put in place to better detect future incidents of this type and/or prevent similar incidents. Some of the examples are
    1. Policy Changes, typically related to organization wide policies related to user, IT systems etc.
    2. Monitoring System changes, typically these are configuration changes that will be made in SIEM, perimeter or endpoint defences to improve better detection and efficient reporting
    3. Architectural Changes, typically are long term major changes in the way the systems are built.
  • Process Improvements – This section should describe any improvements that could be made to the actual response process itself. Some of the examples are:
    1. Improving the Incident handling cheat sheet with additional details
    2. Improving the communications plan to get speedier response
    3. Escalation matrix improvements
    4. Process automation
    5. Staff training and awareness

Rinse and Repeat!!!

As you can see from above, the goal is not to do this exercise as a one time activity. Instead this is a repetitive process. However, this may not be possible for every single incident that is detected and worked by the CSIRT. Hence, this is where practicality dictates that this process should be done in a way that is scalable. Keeping that in mind, below is a recommended approach for doing this:

  1. Perform “Lessons Learnt” exercise for all Major and High Severity incidents
  2. Perform “Lessons Learnt” exercise for all repeat incident category (refer to Incident Classification for more details)
  3. Perform “Lessons Learnt” exercise on a monthly or quarterly basis for CSIRT processes.


Lessons learnt are an important part of continued learning and quest for functional perfection. CSIRT is no different and it should also be improved on a regular basis. These improvements should be aimed at efficiently detecting and responding to cyber incidents in a timely fashion.

With this post, the CSIRT Series comes to a conclusion. Please feel free to post your comments on the section below.


Episode 6 – ShellShock Investigation Part 2

Over the past few days, we have been researching about Shellshock and found a number of attack vectors that makes the vulnerability much easier to exploit. This article may not contain the entire list of attack vectors but we are doing our best in updating this list. Kindly leave us a comment if you have faced any attack vectors apart from what we are discussing here.


The most commonly used vectors that can lead to exploiting this vulnerability are

  1. CGI
  2. SSH and
  3. DHCP

Apart from these commonly known attack vectors, we were able to find a list of other attack vectors, which aid in exploiting this vulnerability. We will discuss mainly the above-mentioned vectors and plunge into other less common vectors.

Shellshock is a vulnerability that allows hackers to inject commands to a computer’s operating system over the network. The vulnerability potentially affects most versions of the Linux and Unix operating systems, Mac OS X, Servers, Wi-Fi routers, Firewalls and even appliances, which invoke bash shell.

For a system to be vulnerable to Shellshock, the following 3 conditions must be met:

  1. It must set an environment variable whose value is attacker-controlled and must begin with “() } “
  2. It must invoke bash shell

3.The system must be running on a vulnerable version of bash.

Having a basic idea of Shellshock, let us begin our discussion first on the common attack vectors that are being employed.

1.Common Gateway Interface (CGI) which is an interface between a web server and executable that produce dynamic content and has been identified as the major attack vector.

Let us start our discussion with a typical HTTP request looks like the following:

GET /path?query-param-name=query-param-value HTTP/1.1
Custom: custom-header-value

The CGI specification maps all parts to environment variables. With Apache httpd, the magic string “() {” can appear in these places:

* Host (“”, as REMOTE_HOST)
* Header value (“custom-header-value”, as HTTP_CUSTOM in this example)
* Server protocol (“HTTP/1.1”, as SERVER_PROTOCOL)

The user name embedded in an Authorization header could be a vector as well, but the corresponding REMOTE_USER variable is only set if the user name corresponds to a known account according to the authentication configuration, and a configuration which accepts the magic string appears somewhat unlikely.

In addition, with other CGI implementations, the request method (“GET”), path (“/path”) and query string (“query-param-name=query-param-value”) may be vectors, and it is conceivable for “query-param-value” as well, and perhaps even “query-param-name”.

  1. Secure Shell (SSH) vector arises from the ForceCommand functionality, which allows a SSH server to be configured to restrict user actions. An authenticated malicious user could send a crafted communication that would trigger the BASH vulnerability, effectively allowing the attacker to break out of these restrictions and execute arbitrary commands. Since SSH is often used to tunnel and facilitate other services, applications that depend on this functionality may also be affected.
  1. DHCP clients can manipulate environment variables using data taken from DHCP server. If the DHCP client machine is running BASH, then the vulnerability will be triggered when it connects to a malicious DHCP server. This will often occur automatically, silently and with no user input. To make matters worse, DHCP clients have more privileges than CGI scripts. This affects the default DHCP clients found on most Linux flavors, but OSX is unaffected, as it uses a different implementation

We can start a DHCP server on the network and set the random string value to () { ;}; echo “This is a test”. We can replace the string to “This is a test” to any command that we want the client machine to execute.

Additional information can be grabbed from:


As we have discussed the most common attack vectors of Shellshock vulnerability, we will also have an overview of the less common vectors that are being used to exploit the vulnerability.

MAIL SERVICES: Majority of the UNIX mail services are exposed to Shellshock vulnerability during their mail handling procedures. Exim, Postfix, qmail and Procmail are the mail services that can be used to exploit the vulnerability.

EXIM sets a number of attacker controlled environment variables when invoking the pipe transport. Whether or not it uses the shell invoke it is determined by the use_shell op­tion. Even if the use_shell is not set, the program invoked is often a shell script. Therefore, many Exim configurations are likely to be vulnerable.

Postfix does not set any attacker-controller environment variables, so Postfix is not typically vulnerable.

qmail can be used as an attack vector to exploit the bash vulnerability which can be used to execute arbitrary commands as any valid user with a .qmail containing a program delivery. The conditions that need to be met for exploiting the vulnerability using qmail are:

1) “Shellshock”-vulnerable bash

2) /bin/sh symlinked to bash

3) Email delivery via qmail to a valid user with a .qmail file containing ANY program delivery (the actual program being delivered to is irrelevant)

OPEN VPN can be configured to call out to a number of user-supplied helper programs. Open VPN does not itself use the shell invoke them, but the programs themselves are usually shell scripts. It sets a number of environment variables few of which, such as fields parsed out of the X.509 certificate, are possibly attacker-controlled. Servers can cause clients (but not vice versa) to set $foreign_option_* to arbitrary values. Many Open VPN configurations are likely to be vulnerable.