Category Archives: Security Tools

Random Thoughts, Security Learning, Security Tools, What you need to know?

Big Data – What you need to know?

Gallery 2 Comments

Big Data is the buzzword in IT Circle nowadays. The major reason for this is the exploding “Netizen” base. Today Everything is happening Online and Online Data is estimated in zettabytes. The wealth of information one can carve from Online data is undeniably attractive for several organizations for marketing and sales. Organizations like Google, Yahoo, Facebook, Amazon etc process several Petabytes of data on a daily basis. Many more organizations are moving towards being able to collect, store and make sense of data in the Internet to further their interests.That is where “Big Data” has caught the imagination of people around the world. But What is Big Data and How can I jump into this bandwagon. Fret not, for in the blog post, you are going to find all about it.  The structure of this blog will be typical of a What you need to know? series posted at Infosecnirvana.com. So lets get started!!!

What is Data?
Data is anything that provides value in a structured or unstructured format. It is the lowest level of abstraction in Computing terms because after this, it is binary digits only. Data is typically stored in File Systems

Introducing File Systems
File Systems are the basis of storing and accessing data from a hardware device. It is nothing but an abstraction layer of software/firmware that gives you the capability to store data in a structured format, remember the structure and when queried, help retrieve it as quickly as possible. There are 2 major and common types of File Systems – Disk Based (local access) and Network Based (remote access). To give a simple example, FAT is a Windows Disk based File System wheres NFS is a Network based File System.

Even though both the file systems continue to dominate IT space, more and more relevance is given to Network based File Systems for obvious reasons like Distributed Data storage, redundancy, fault tolerance capabilities etc. This is the basis of “Big Data Tools and Technologies”.

Introducing DFS
Distributed File Systems are Network based File Systems that allow data to be shared across multiple machines across multiple networks. This makes it possible for multiple users on multiple machines to share files and storage resources. The client machines don’t have direct access to the Storage disk itself (as in a Disk based file system), but are able to interact with the Data using a File System protocol. One classic example of DFS is Microsoft SMB where All Windows machines are SMB Clients and access a common SMB Share on the File Server. But SMB suffers from issues pertaining to scalability and fault tolerance. This is where systems like Google File System – GFS (Google uses this in their search engine) and Hadoop Distributed File System – HDFS (Yahoo and others) come into prominence. What these File Systems do is provide a mechanism to effectively manage big data collection, storage and processing across multiple machine nodes.

Introducing HDFS:

Hadoop Distributed File Systems or shortly HDFS is similar to the other DFS file systems talked above, however it is significantly different as well. HDFS can be deployed on Commodity Hardware, is Highly Fault Tolerant and is very capable of handling large data sets. Originally HDFS was developed as part of the Apache NUTCH Project for an alternate Search Engine akin to Google. Some of the most prominent software players for HDFS are “Apache Hadoop”, “Greenplum”, Cloudera etc.

In this post, we will be looking at Log Collection and Management using the Hadoop Platform.

APACHE Hadoop: The Apache Hadoop architecture in a Nutshell consists of the following components:

  • HDFS is a Master Slave Architecture
  • Master Server is called a NameNode
  • Slave Servers are called DataNodes
  • Underlying Data Replication across Nodes
  • Interface Language – Java

Installing Hadoop: Installation of Apache Hadoop is not a very easy task, but at the same time it is not too complex either. Understanding of the Hardware Requirements, Operating System Requirements and Java Programming Language can help you install Apache Hadoop without any issues. Installing Hadoop can be either a Single Node Installation or a Cluster Installation. For this post, we will look at only Single Node Installation steps:

  1. Install Oracle Java on your machine – Ubuntu
  2. Install OpenSSH Server
  3. Create a Hadoop Group and Hadoop User and set Key Based Login for SSH
  4. Download the Latest Distribution of Hadoop from http://www.apache.org/dyn/closer.cgi
  5. Installation is just extracting the Hadoop files into a folder and editing some property files
  6. Provide the location for the JAVA home in the following file location- hadoop/conf/hadoop-env.sh
  7. Create a working folder in Hadoop User Home Directory /home//tmp
  8. Add the relevant details about the host and the home directory following configuration elements in /hadoop/conf/core-site.xml 
    conf/core-site.xml —>

hadoop.tmp.dir
/home//tmp
A base for other temporary directories.

fs.default.name
hdfs://localhost:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri’s scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri’s authority is used to
determine the host, port, etc. for a filesystem.
  9. Then we need to edit the hadoop/conf/mapred-site.xml using a text editor and add the following configuration values (like core-site.xml) 
    conf/mapred-site.xml —>

mapred.job.tracker
localhost:54311
The host and port that the MapReduce job tracker runs
at. If “local”, then jobs are run in-process as a single map
and reduce task.
  10. Open hadoop/conf/hdfs-site.xml using a text editor and add the following configurations: 
    conf/hdfs-site.xml —>

dfs.replication
1
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
  11. Before running the Hadoop Installation, the most important step is to format the NameNode or the Master Server. This is critical because, Without the NameNode, the DataNodes will not be setup. In a Single Node Installation, NameNode and DataNodes will reside on the same host, where as in Cluster Installation, NameNodes and DataNodes will reside on different hosts. In order to format the NameNode using Hadoop commands, Run the following command – /hadoop/bin/hadoop namenode -format
  12. In order to start the Hadoop Instance, from hadoop/bin run ./start-dfs.sh and Running the commands will start up Hadoop and when you query the Java Process, you should be able to see the following components of Hadoop Running: 
    NameNode
DataNode
SecondaryNameNode
JobTracker
TaskTracker
  13. If you have successfully completed till this, then you now have a Hadoop Single Node Instance running on your machine.

Getting Data in/out of Hadoop:

Once the installation is completed, the next thing we need to worry about is getting data in and out of Hadoop File System. Typically in order to get the data into the system, we need a API interface into HDFS. This typically is a JAVA or HTTP API. Tools like FluentD, Flume etc help in getting data in and out of Hadoop. Both the tools have plugins for receiving HTTP data, Streaming data and Syslog Data as well.

MapReduce: Hadoop and Big data discussions are incomplete without talking about MapReduce. MapReduce is a software policy framework that maps Input data based on a map file and outputs data in key value pairs. These are two different jobs when it comes to actual processing. One is the Map Task that splits the data into smaller chunks and there is the Reduce Job that generates a Key Value combination for each of the smaller data chunks. This framework is the powerhouse for Hadoop because, this is built with parallelism in mind. Map Tasks and Reduce Tasks can both be run parallel on several machines without compromising on speed, cpu and memory resources. The NameNode is the central master that tracks the Maps and the Jobs where as the DataNodes are just providing processing resource.

Finally, Using Hadoop: Now that we know what drives Hadoop and how to get Hadoop installed, the easiest thing would be to start using them. Several examples for MapReduce jobs using Java are available to aid in learning. There are several related projects running to make the Hadoop Ecosystem more scalable and mature. Some of them are:

  • HBase, a Bigtable-like structured storage system for Hadoop HDFS
  • Apache Pig is a high-level data-flow language and execution framework for parallel computation. It is built on top of Hadoop Core.
  • Hive a data warehouse infrastructure which allows sql-like adhoc querying of data (in any format) stored in Hadoop
  • ZooKeeper is a high-performance coordination service for distributed applications.
  • Hama, a Google’s Pregel-like distributed computing framework based on BSP (Bulk Synchronous Parallel) computing techniques for massive scientific computations.
  • Mahout, scalable Machine Learning algorithms using Hadoop

Conclusion: Hope this post helped you in understanding the basic concepts of Big Data and also to setup a Hadoop Single Node Installation to play with. Please do post your thoughts on how Big Data is playing a major role in your organisations.

Head 2 Head, Installation & Configuration, Security Learning, Security Tools, SIEM

ArcSight CORR 6.0 – Install and Migration

Gallery 12 Comments

ArcSight (now HP) Enterprise Security Manager (ESM) is the premiere security event manager that analyzes and correlates every other event in order to support the Security Team or analysts in every aspect of security event monitoring, from compliance and risk management to security intelligence and operations. There have been several versions of ArcSight ESM released over a period in time. Their latest version is ArcSight CORR 6.0. At InfoSecNirvana.com we have got a copy of the latest version and we will be writing a multi-part post on how to Install, Migrate from Older versions to 6.0 and some basic walk around.

In this Part 1 post, we shall cover about the installation of ArcSight CORR (Correlation Optimized Retention and Retrieval), a proprietary data storage and retrieval framework that receives and processes events at high rates, and performs high-speed searches; the latest ArcSight ESM by HP. With the ArcSight CORR, Oracle database is now eliminated.

CORR components:

  • ArcSight Manager
  • CORR Engine
  • ArcSight Console
  • ArcSight Web
  • Management Console
  • Smart Connectors

Requirements:

System: This completely depends on the EPS that you expect to receive. InfoSecNirvana has been working on getting a PoC for this and the below configuration was used:
A VMWare box with 8 cores, 32GB Ram, 256GB SSD HDD, 2TB WD 7200 RPM SATA HDD (Note: for production, there might be/recommend a higher configuration. Check with ArcSight manuals on the same)

OS: Red Hat Enterprise Linux Server release 6.2 x64, installed with xfsprogs-3.1.1-6.el6.x86_64 rpm; this is required to convert some of the ext4 file systems to xfs filesystems. XFS Partition is the most apt format for us to fully utilize the performance enhancements coming with CORR. Typically, I would recommend /opt/ to be formatted with XFS and maximum storage can be allocated to this partition. This is crucial because, the very first step of installation would verify whether the entire /opt/ directory is in XFS. When using VMWare with LVM, we faced some issues during the installation and ArcSight Support could not help us with this. However, when raw devices were mounted as /OPT/ we did not face any issues.

Storage: Please allocate the required storage (calculate based on Number of Devices, Events per second, Average Event Size and Retention period). Remember, CORR is like an ESM with a built in Logger. You can still use a Logger for long term retention if that is what you prefer so that ESM will be lean and mean.

Permissions: The installation has to be done using a Non-Root account. This account can be a service account named”arcsight”. This account should have RWX permissions on the /opt/ directory. Make sure this is satisfied.

Misc: /TMP/ partition should have at least 3GB space. /home/arcsight also should have a minimum of 5GB free space. This is crucial again because, the INSTALL DIR log files are written in these location and if sufficient space is not allocated the installation fails.

The CORR package: Get the CORR installation package and the license from HP ArcSight. This can be obtained from your sales representative with HP/ArcSight.

CORR Installation:
The installation is pretty straightforward and is just a series of clicks. I have given most of the screenshots below just as a reference. Obviously, if you have already installed ArcSight Software, you would not even need this. Once done, you would be able to install the Console to access CORR and play around.



Once the installation is completed, we would want to test the following before we call the install as complete:

  1. Validate the Log Files in the Manager Install Logs and find out if there are any warnings and errors. Generally, this is a best practice to ensure valid installation.
  2. Install the Console and try to connect to ESM, with the default user name and password (mentioned in the install guide). First time when you connect, A certificate import of the Manager happens. If you use a self-signed certificate make sure you note down the parameters used to create cause this will help in future migrations, troubleshooting or recovery.
  3. After connecting to the console, you are ready to go.

Migrations from Existing Installs – Migrating from earlier versions to this CORR instance is tricky, because you are migrating from a DB back end to a NON-DB back end. I will be posting a followup of this post in PART 2 that will detail the migration procedure from 4.X to 5.X.

Stay Tuned to InfoSecNirvana.com for more!!!

Malware Reversing, Security Learning, Security Tools, What you need to know?

Reverse Engineering Malware – What you need to know?

Gallery Leave a comment

Every now and then, a nasty piece of Malware raises its ugly head and wreck havoc on the Enterprise Infrastructure. It is often necessary to analyze the Malware and understand its working so that

  • The impact of the Malware on IT Systems can be ascertained AND
  • The nature of preventative controls that can be put in place so that this threat does not spread further.

In such scenarios, Reverse Engineering of the Malware becomes a requirement. Reverse engineering of a malware or an unknown piece of binary file is the process of analyzing and understanding its characteristics, behavior. There are several approaches that several different people use. But in this Blog post, the goal is to give a quick little guide for Malware Reversing so that anyone with an inclination to pick it can do so very easily. This in my mind is an essential tool in the hands of a Security Analyst. The basic skills needed to do this are listed below:

  • Some programming skills, or at the very least be able to understand and read source code
  • Logical Mindset capable of analyzing and interpreting the Vectors used by the Malware Code
  • Lots and Lots of Patience and Interest

Let us now get into the basics. We need to understand that Malware programs have several stages before they infect and compromise a machine. Typically a malware performs the following:

  • Get itself persistent by adding its executable path in the registry, autorun, etc – Exploit
  • Comfortably elevate itself to another process, so that it cannot be easily found – Masquerade
  • Deter the process of it’s analysis by rapidly changing its code signature – Polymorphism
  • Make connections to remote servers sometimes to update itself or also to report back to its Master – Callback
  • Perform the necessary tasks on the affected system – Data Exfiltration or Zombie

All these Malicious Programs, have one goal or the other, but eventually they end up handing over control of your machine to strangers and potentially bad guys as well. Some Intentions of Malware are listed below:

  • Steal sensitive information / Key-logging / identity theft / usernames and password / banking information / Company patents / source code / etc (including personal data that may have been part of the system)
  • Access private networks
  • Perform DDoS Attacks
  • Spamming
  • Browser hijacking, ad-wares to perform fraud
  • Ransom-ware: deny access to the users’ own data and demand money to give access back – in other words extortion
  • Data exfiltration

Reverse Engineering Methodology:
This effort involves determining not only what the malware can do specifically, but also establishing how to identify the presence of such programs on affected systems. There might be so many (right) ways to do this, but for something quick we shall follow the steps laid out here.

Why Quick?? Because in an Enterprise sometimes we might not have all the time to really perform in-depth analysis, as time is a major factor when responding to incidents of this kind.

The Setup:
In order to perform an Effective Malware Analysis, we need to have a Tool Kit and an Environment for Analysis. Some of the key things to take care while setting up the environment are:

  • The Environment should be isolated with no connections to the Enterprise Data Sensitive network.
  • The environment should have its own  Proxy service so that it does not have scope to spread. The Proxy can be a sink hole where it just logs the connections made.
  • Set-up 2 sand boxes, 1 physical and 1 VM, as some malware programs only work on  on a physical box as they are VM-aware.
  • Make sure these sandboxes are standard images, with bare minimum corporate patching done. This should theoretically be equivalent to the weakest link in the organization.
  • Install all the required tools listed below to do certain type of analysis.
  • Tools required: strings, ida pro, pmdump, volatility framework, upx, packerid, pescanner, pe explorer, md5hash, ollydbg, deep freeze, winalysis, lp

The Analysis: The analysis of Malware is usually a two-phased approach – Behavior Analysis and Code Analysis. These two analysis methods yield so much information that detection and response becomes easy.

  • Behavioral analysis: Observing malware interactions with its environment like network connections, files dropped, evasive measures taken etc. This can be identified by installing the Malware, “getting infected” as you may call it.
    • Once infected, you can capture the network packets, to look at potential domains and IP addresses the software tries to connect. This will help in perimeter filtering and Endpoint ‘Firewall’ing
    • If the Malware drops some files using C2, then that also can be observed as a part of the getting infected process. This will help in gather SHA and MD5 values for the dropped files and banning them in Endpoint solutions from execution
  • Code analysis: Examining the code that comprises the program to infer what exactly the malware is capable of doing when executed. This does not help in response schemes, but is very important from a Forensics Purpose. Code Analysis can help in determining the extent of loss, the extent of vulnerability in the system that is being exploited etc.
  • Code Analysis can be done as follows:
    • Firstly Identify if either the unknown file is protected, obfuscated, encrypted (armoring) and/or packed (the original code is compressed, encrypted or both). To do this, we can use packerid or pe explorer.This technique is applied in an attempt to evade signature based malware detection, and to deter the efforts of static analysis. Identifying the packer specifically, can exactly tell you what you are missing in terms of detection using Perimeter tools.
    • Then with basic analysis like enumerating exports, imports, function use, syscalls, winapi, mutex, dll dependencies, strings and some grepping, using the winalysis or other similar tools that you might be comfortable with, you can come up with several theories about the file. These theories will give an understanding of the various attack vectors employed by the file. This can help lock down a system to these kinds of Malware attempts.
    • Drilling down further into the specific attack functions and looking at the code itself can help understand the vulnerability being exploited. This is very useful for Developers in fixing the holes in the software. This will help in a sort of retroactive patching methodology.
Post Analysis Steps: 
  1. Once the analysis is done on the Behavior and the Code aspects of the Malware, you have lots of data about at hand. Documenting the Analysis is very key because, future variants may use the same Attack Vector, Same Exploit Code etc to gain access to a machine/application.
  2. Use the documentation prepared as above to compare against subsequent analysis. This will save a great deal of time in detecting and responding to future threats posed.
  3. Snapshot of the VM also can be retained for future reference.
  4. Destroy the Analysis VM and start over again!!!
Practical Example
There will be a follow up post to this with a Hands-On Tutorial of how its done!!!! Keep Following this blog and Happy Reversing!!!!
Additional Resources:

 

Website to get malware samples for analysis:
http://oc.gtisc.gatech.edu:8080/

Websites to assist you in malware analysis:
REMnux (Linux distribution for malware analysis) – http://zeltser.com/remnux/
ISEC Labs Anubis Tool – http://anubis.iseclab.org/
GFI Sandbox – http://www.gfi.com/malware-analysis-tool
Hex to Binary/ASCII – http://home.paulschou.net/tools/xlate/
Hex to ASCII – http://www.dolcevie.com/js/converter.html
Jsunpack – http://jsunpack.jeek.org/