Category Archives: What you need to know?

Random Thoughts, Security Learning, Security Tools, What you need to know?

Big Data – What you need to know?

Gallery 2 Comments

Big Data is the buzzword in IT Circle nowadays. The major reason for this is the exploding “Netizen” base. Today Everything is happening Online and Online Data is estimated in zettabytes. The wealth of information one can carve from Online data is undeniably attractive for several organizations for marketing and sales. Organizations like Google, Yahoo, Facebook, Amazon etc process several Petabytes of data on a daily basis. Many more organizations are moving towards being able to collect, store and make sense of data in the Internet to further their interests.That is where “Big Data” has caught the imagination of people around the world. But What is Big Data and How can I jump into this bandwagon. Fret not, for in the blog post, you are going to find all about it.  The structure of this blog will be typical of a What you need to know? series posted at Infosecnirvana.com. So lets get started!!!

What is Data?
Data is anything that provides value in a structured or unstructured format. It is the lowest level of abstraction in Computing terms because after this, it is binary digits only. Data is typically stored in File Systems

Introducing File Systems
File Systems are the basis of storing and accessing data from a hardware device. It is nothing but an abstraction layer of software/firmware that gives you the capability to store data in a structured format, remember the structure and when queried, help retrieve it as quickly as possible. There are 2 major and common types of File Systems – Disk Based (local access) and Network Based (remote access). To give a simple example, FAT is a Windows Disk based File System wheres NFS is a Network based File System.

Even though both the file systems continue to dominate IT space, more and more relevance is given to Network based File Systems for obvious reasons like Distributed Data storage, redundancy, fault tolerance capabilities etc. This is the basis of “Big Data Tools and Technologies”.

Introducing DFS
Distributed File Systems are Network based File Systems that allow data to be shared across multiple machines across multiple networks. This makes it possible for multiple users on multiple machines to share files and storage resources. The client machines don’t have direct access to the Storage disk itself (as in a Disk based file system), but are able to interact with the Data using a File System protocol. One classic example of DFS is Microsoft SMB where All Windows machines are SMB Clients and access a common SMB Share on the File Server. But SMB suffers from issues pertaining to scalability and fault tolerance. This is where systems like Google File System – GFS (Google uses this in their search engine) and Hadoop Distributed File System – HDFS (Yahoo and others) come into prominence. What these File Systems do is provide a mechanism to effectively manage big data collection, storage and processing across multiple machine nodes.

Introducing HDFS:

Hadoop Distributed File Systems or shortly HDFS is similar to the other DFS file systems talked above, however it is significantly different as well. HDFS can be deployed on Commodity Hardware, is Highly Fault Tolerant and is very capable of handling large data sets. Originally HDFS was developed as part of the Apache NUTCH Project for an alternate Search Engine akin to Google. Some of the most prominent software players for HDFS are “Apache Hadoop”, “Greenplum”, Cloudera etc.

In this post, we will be looking at Log Collection and Management using the Hadoop Platform.

APACHE Hadoop: The Apache Hadoop architecture in a Nutshell consists of the following components:

  • HDFS is a Master Slave Architecture
  • Master Server is called a NameNode
  • Slave Servers are called DataNodes
  • Underlying Data Replication across Nodes
  • Interface Language – Java

Installing Hadoop: Installation of Apache Hadoop is not a very easy task, but at the same time it is not too complex either. Understanding of the Hardware Requirements, Operating System Requirements and Java Programming Language can help you install Apache Hadoop without any issues. Installing Hadoop can be either a Single Node Installation or a Cluster Installation. For this post, we will look at only Single Node Installation steps:

  1. Install Oracle Java on your machine – Ubuntu
  2. Install OpenSSH Server
  3. Create a Hadoop Group and Hadoop User and set Key Based Login for SSH
  4. Download the Latest Distribution of Hadoop from http://www.apache.org/dyn/closer.cgi
  5. Installation is just extracting the Hadoop files into a folder and editing some property files
  6. Provide the location for the JAVA home in the following file location- hadoop/conf/hadoop-env.sh
  7. Create a working folder in Hadoop User Home Directory /home//tmp
  8. Add the relevant details about the host and the home directory following configuration elements in /hadoop/conf/core-site.xml 
    conf/core-site.xml —>

hadoop.tmp.dir
/home//tmp
A base for other temporary directories.

fs.default.name
hdfs://localhost:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.
  9. Then we need to edit the hadoop/conf/mapred-site.xml using a text editor and add the following configuration values (like core-site.xml) 
    conf/mapred-site.xml —>

mapred.job.tracker
localhost:54311
The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.
  10. Open hadoop/conf/hdfs-site.xml using a text editor and add the following configurations: 
    conf/hdfs-site.xml —>

dfs.replication
1
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
  11. Before running the Hadoop Installation, the most important step is to format the NameNode or the Master Server. This is critical because, Without the NameNode, the DataNodes will not be setup. In a Single Node Installation, NameNode and DataNodes will reside on the same host, where as in Cluster Installation, NameNodes and DataNodes will reside on different hosts. In order to format the NameNode using Hadoop commands, Run the following command – /hadoop/bin/hadoop namenode -format
  12. In order to start the Hadoop Instance, from hadoop/bin run ./start-dfs.sh and Running the commands will start up Hadoop and when you query the Java Process, you should be able to see the following components of Hadoop Running: 
    NameNode
DataNode
SecondaryNameNode
JobTracker
TaskTracker
  13. If you have successfully completed till this, then you now have a Hadoop Single Node Instance running on your machine.

Getting Data in/out of Hadoop:

Once the installation is completed, the next thing we need to worry about is getting data in and out of Hadoop File System. Typically in order to get the data into the system, we need a API interface into HDFS. This typically is a JAVA or HTTP API. Tools like FluentD, Flume etc help in getting data in and out of Hadoop. Both the tools have plugins for receiving HTTP data, Streaming data and Syslog Data as well.

MapReduce: Hadoop and Big data discussions are incomplete without talking about MapReduce. MapReduce is a software policy framework that maps Input data based on a map file and outputs data in key value pairs. These are two different jobs when it comes to actual processing. One is the Map Task that splits the data into smaller chunks and there is the Reduce Job that generates a Key Value combination for each of the smaller data chunks. This framework is the powerhouse for Hadoop because, this is built with parallelism in mind. Map Tasks and Reduce Tasks can both be run parallel on several machines without compromising on speed, cpu and memory resources. The NameNode is the central master that tracks the Maps and the Jobs where as the DataNodes are just providing processing resource.

Finally, Using Hadoop: Now that we know what drives Hadoop and how to get Hadoop installed, the easiest thing would be to start using them. Several examples for MapReduce jobs using Java are available to aid in learning. There are several related projects running to make the Hadoop Ecosystem more scalable and mature. Some of them are:

  • HBase, a Bigtable-like structured storage system for Hadoop HDFS
  • Apache Pig is a high-level data-flow language and execution framework for parallel computation. It is built on top of Hadoop Core.
  • Hive a data warehouse infrastructure which allows sql-like adhoc querying of data (in any format) stored in Hadoop
  • ZooKeeper is a high-performance coordination service for distributed applications.
  • Hama, a Google’s Pregel-like distributed computing framework based on BSP (Bulk Synchronous Parallel) computing techniques for massive scientific computations.
  • Mahout, scalable Machine Learning algorithms using Hadoop

Conclusion: Hope this post helped you in understanding the basic concepts of Big Data and also to setup a Hadoop Single Node Installation to play with. Please do post your thoughts on how Big Data is playing a major role in your organisations.

Malware Reversing, Security Learning, Security Tools, What you need to know?

Reverse Engineering Malware – What you need to know?

Gallery Leave a comment

Every now and then, a nasty piece of Malware raises its ugly head and wreck havoc on the Enterprise Infrastructure. It is often necessary to analyze the Malware and understand its working so that

  • The impact of the Malware on IT Systems can be ascertained AND
  • The nature of preventative controls that can be put in place so that this threat does not spread further.

In such scenarios, Reverse Engineering of the Malware becomes a requirement. Reverse engineering of a malware or an unknown piece of binary file is the process of analyzing and understanding its characteristics, behavior. There are several approaches that several different people use. But in this Blog post, the goal is to give a quick little guide for Malware Reversing so that anyone with an inclination to pick it can do so very easily. This in my mind is an essential tool in the hands of a Security Analyst. The basic skills needed to do this are listed below:

  • Some programming skills, or at the very least be able to understand and read source code
  • Logical Mindset capable of analyzing and interpreting the Vectors used by the Malware Code
  • Lots and Lots of Patience and Interest

Let us now get into the basics. We need to understand that Malware programs have several stages before they infect and compromise a machine. Typically a malware performs the following:

  • Get itself persistent by adding its executable path in the registry, autorun, etc – Exploit
  • Comfortably elevate itself to another process, so that it cannot be easily found – Masquerade
  • Deter the process of it’s analysis by rapidly changing its code signature – Polymorphism
  • Make connections to remote servers sometimes to update itself or also to report back to its Master – Callback
  • Perform the necessary tasks on the affected system – Data Exfiltration or Zombie

All these Malicious Programs, have one goal or the other, but eventually they end up handing over control of your machine to strangers and potentially bad guys as well. Some Intentions of Malware are listed below:

  • Steal sensitive information / Key-logging / identity theft / usernames and password / banking information / Company patents / source code / etc (including personal data that may have been part of the system)
  • Access private networks
  • Perform DDoS Attacks
  • Spamming
  • Browser hijacking, ad-wares to perform fraud
  • Ransom-ware: deny access to the users’ own data and demand money to give access back – in other words extortion
  • Data exfiltration

Reverse Engineering Methodology:
This effort involves determining not only what the malware can do specifically, but also establishing how to identify the presence of such programs on affected systems. There might be so many (right) ways to do this, but for something quick we shall follow the steps laid out here.

Why Quick?? Because in an Enterprise sometimes we might not have all the time to really perform in-depth analysis, as time is a major factor when responding to incidents of this kind.

The Setup:
In order to perform an Effective Malware Analysis, we need to have a Tool Kit and an Environment for Analysis. Some of the key things to take care while setting up the environment are:

  • The Environment should be isolated with no connections to the Enterprise Data Sensitive network.
  • The environment should have its own  Proxy service so that it does not have scope to spread. The Proxy can be a sink hole where it just logs the connections made.
  • Set-up 2 sand boxes, 1 physical and 1 VM, as some malware programs only work on  on a physical box as they are VM-aware.
  • Make sure these sandboxes are standard images, with bare minimum corporate patching done. This should theoretically be equivalent to the weakest link in the organization.
  • Install all the required tools listed below to do certain type of analysis.
  • Tools required: strings, ida pro, pmdump, volatility framework, upx, packerid, pescanner, pe explorer, md5hash, ollydbg, deep freeze, winalysis, lp

The Analysis: The analysis of Malware is usually a two-phased approach – Behavior Analysis and Code Analysis. These two analysis methods yield so much information that detection and response becomes easy.

  • Behavioral analysis: Observing malware interactions with its environment like network connections, files dropped, evasive measures taken etc. This can be identified by installing the Malware, “getting infected” as you may call it.
    • Once infected, you can capture the network packets, to look at potential domains and IP addresses the software tries to connect. This will help in perimeter filtering and Endpoint ‘Firewall’ing
    • If the Malware drops some files using C2, then that also can be observed as a part of the getting infected process. This will help in gather SHA and MD5 values for the dropped files and banning them in Endpoint solutions from execution
  • Code analysis: Examining the code that comprises the program to infer what exactly the malware is capable of doing when executed. This does not help in response schemes, but is very important from a Forensics Purpose. Code Analysis can help in determining the extent of loss, the extent of vulnerability in the system that is being exploited etc.
  • Code Analysis can be done as follows:
    • Firstly Identify if either the unknown file is protected, obfuscated, encrypted (armoring) and/or packed (the original code is compressed, encrypted or both). To do this, we can use packerid or pe explorer.This technique is applied in an attempt to evade signature based malware detection, and to deter the efforts of static analysis. Identifying the packer specifically, can exactly tell you what you are missing in terms of detection using Perimeter tools.
    • Then with basic analysis like enumerating exports, imports, function use, syscalls, winapi, mutex, dll dependencies, strings and some grepping, using the winalysis or other similar tools that you might be comfortable with, you can come up with several theories about the file. These theories will give an understanding of the various attack vectors employed by the file. This can help lock down a system to these kinds of Malware attempts.
    • Drilling down further into the specific attack functions and looking at the code itself can help understand the vulnerability being exploited. This is very useful for Developers in fixing the holes in the software. This will help in a sort of retroactive patching methodology.
Post Analysis Steps: 
  1. Once the analysis is done on the Behavior and the Code aspects of the Malware, you have lots of data about at hand. Documenting the Analysis is very key because, future variants may use the same Attack Vector, Same Exploit Code etc to gain access to a machine/application.
  2. Use the documentation prepared as above to compare against subsequent analysis. This will save a great deal of time in detecting and responding to future threats posed.
  3. Snapshot of the VM also can be retained for future reference.
  4. Destroy the Analysis VM and start over again!!!
Practical Example
There will be a follow up post to this with a Hands-On Tutorial of how its done!!!! Keep Following this blog and Happy Reversing!!!!
Additional Resources:

 

Website to get malware samples for analysis:
http://oc.gtisc.gatech.edu:8080/

Websites to assist you in malware analysis:
REMnux (Linux distribution for malware analysis) – http://zeltser.com/remnux/
ISEC Labs Anubis Tool – http://anubis.iseclab.org/
GFI Sandbox – http://www.gfi.com/malware-analysis-tool
Hex to Binary/ASCII – http://home.paulschou.net/tools/xlate/
Hex to ASCII – http://www.dolcevie.com/js/converter.html
Jsunpack – http://jsunpack.jeek.org/

 

Random Thoughts, Security Learning, What you need to know?

APT – What you need to know?

Gallery 2 Comments

APT – Advanced Persistent Threat is the latest buzz word in the industry. Everyone who is in the Security Industry, professionals and business alike want to get into the bandwagon that is called APT. Security product vendors are all gearing to cater to “APT” and all their current product lines or future releases address APT in some form of the other. Now, the fever has spread to the IT Management as well and now they want their Security teams to detect and prevent APT. Even though the InfoSec public has caught up with it, how much thought have we put into understanding the magnitude of the problem at hand? Is it enough to just jump on to something without understanding it fully or do we need a more educated and intelligent decision making?

Let us find out more in this post!!!!

As always, I would like to define APT to start with. This is key because once the definitions are clear, all we would need is to align our thinking to that definition. Then, I will list down what flaws we have in our current approach towards security. Finally, I will try to list down as many possible solutions to the problem at hand.

Defining APT:
Simply put, APT is a Security Threat to the Enterprise (even End User for that matter) that is Advanced in execution that traditional security filters are not able to catch outright and is persistent enough that it keeps moving from one compromised target to another evading detection. 

Is it a technology of the future? – No, it is not. APT is nothing but a threat we are not trained to see. One of the main reasons why APT has been so successful in many organizations is the fact that we have an outdated security strategy. For example, we are keen on tracking a Data Exfiltration from a compromised machine. How do we do it today?

  • To start of with, we look for Data Loss Prevention Solutions and see which vendor is the market leader
  • Then we implement DLP solutions with basic policies for generic data loss (PDF, WORD DOC, XLS, Source Codes, Credit Card Numbers, PAN, PII etc)
  • We fine tune the DLP policies for our enterprise specifically and implement detection and prevention capabilities
  • We log the data from DLP solutions to SIEM and alert when something of interest happens.
  • In addition or In replacement, IDS/IPS rules will be implemented to identify data loss traffic based on REGEX file names etc.
  • In some cases we would also look at Traffic going to Blacklisted Domains and IP.
I am sure all of them or majority of the organizations do this to identify Data Exfiltrations. But  can all those organizations say that they are safe against APT? The answer is a SAD NO. The reason being, Known (Policy or Signature of What is Bad) is a drop, Unknown (Where APT works) is an Ocean. The threat landscape has evolved to exploit the Unknown, but we have not evolved to detect and respond to it. What is the solution for this problem?
There are several solutions being proposed by several people in the industry.  In my opinion one of the most important solutions is to do behavior profiling and Anomaly Detection.
Now What is Behavior profiling?
Behavior Profiling – Every network, every segment of the network has a behavior profile that is deemed normal. Today how many of us know what our Network Segments look like in terms of Connections they accept, they deny, Traffic flowing within the segment, what are the most used protocols, what are not used, What size of packets flow, what outbound and inbound communications happen, Access in and out, Who is supposed to and Who is not etc etc.. I seriously doubt it. We are more concerned about getting the system up, providing the service it is deemed to provide. We seldom think about the Security Profile the segment has. Once we profile, we can identify several Anomalies.

Let us now take the same example of Data Exfiltration and see how Behavior profiling would help:

  1. We would have complete details about where sensitive data is residing, the VLAN, the Server, the Folder, The file, The DB tables etc.
  2. To the Sensitive Machine/Network/Data, We would know who has access to and Who does not?
  3. We would also track who has a copy of that data – what is the machine, where is it residing (desktop, laptop, mobile) etc.
  4. The data usage by which team, which individuals etc are also profiled and that would give us the subset of people handling that sensitive data
  5. Any theft of that data would be through one of the above actors/entities.
  6. Tracking each of their machines activity over time would give us a Normal behavior profile.
  7. Digital Markers on such sensitive data can also be placed by the corporations to track data use/flow
  8. We can also track periodicity of data access, time of access, track the data changes etc through Digital Markers
  9. Any deviations from Normal behavior is a potential Data Exfiltration action and needs to be investigated
  10. Behavior profiles thus created can be used in addition to Signature based detection

This requires intimate co-ordination with various teams and also requires great understanding of what your Network does, what it is supposed to do. This while being the most logical is the most challenging to implement and thus the most rewarding as well. Behavior profiling is being used in the Intelligence Community for a long time, but the Technology community is still to embrace this. Enterprise data is becoming critical and with threats like APT, our fundamentals are being questioned.

This approach can help after the fact but from preventing the occurrence a Long term solution is needed. From a long term perspective the only solution is building Networks and Applications (OS as well as Apps) from ground up to treat security as a embedded character and not an add on feature.

What are your thoughts on APT? How do you think we should change our Security thought process, technology and all to combat it? Sound on below!!!