Category Archives: Random Thoughts

Big Data – What you need to know?

Big Data is the buzzword in IT Circle nowadays. The major reason for this is the exploding “Netizen” base. Today Everything is happening Online and Online Data is estimated in zettabytes. The wealth of information one can carve from Online data is undeniably attractive for several organizations for marketing and sales. Organizations like Google, Yahoo, Facebook, Amazon etc process several Petabytes of data on a daily basis. Many more organizations are moving towards being able to collect, store and make sense of data in the Internet to further their interests.That is where “Big Data” has caught the imagination of people around the world. But What is Big Data and How can I jump into this bandwagon. Fret not, for in the blog post, you are going to find all about it.  The structure of this blog will be typical of a What you need to know? series posted at So lets get started!!!

What is Data?
Data is anything that provides value in a structured or unstructured format. It is the lowest level of abstraction in Computing terms because after this, it is binary digits only. Data is typically stored in File Systems

Introducing File Systems
File Systems are the basis of storing and accessing data from a hardware device. It is nothing but an abstraction layer of software/firmware that gives you the capability to store data in a structured format, remember the structure and when queried, help retrieve it as quickly as possible. There are 2 major and common types of File Systems – Disk Based (local access) and Network Based (remote access). To give a simple example, FAT is a Windows Disk based File System wheres NFS is a Network based File System.

Even though both the file systems continue to dominate IT space, more and more relevance is given to Network based File Systems for obvious reasons like Distributed Data storage, redundancy, fault tolerance capabilities etc. This is the basis of “Big Data Tools and Technologies”.

Introducing DFS
Distributed File Systems are Network based File Systems that allow data to be shared across multiple machines across multiple networks. This makes it possible for multiple users on multiple machines to share files and storage resources. The client machines don’t have direct access to the Storage disk itself (as in a Disk based file system), but are able to interact with the Data using a File System protocol. One classic example of DFS is Microsoft SMB where All Windows machines are SMB Clients and access a common SMB Share on the File Server. But SMB suffers from issues pertaining to scalability and fault tolerance. This is where systems like Google File System – GFS (Google uses this in their search engine) and Hadoop Distributed File System – HDFS (Yahoo and others) come into prominence. What these File Systems do is provide a mechanism to effectively manage big data collection, storage and processing across multiple machine nodes.

Introducing HDFS:

Hadoop Distributed File Systems or shortly HDFS is similar to the other DFS file systems talked above, however it is significantly different as well. HDFS can be deployed on Commodity Hardware, is Highly Fault Tolerant and is very capable of handling large data sets. Originally HDFS was developed as part of the Apache NUTCH Project for an alternate Search Engine akin to Google. Some of the most prominent software players for HDFS are “Apache Hadoop”, “Greenplum”, Cloudera etc.

In this post, we will be looking at Log Collection and Management using the Hadoop Platform.

APACHE Hadoop: The Apache Hadoop architecture in a Nutshell consists of the following components:

  • HDFS is a Master Slave Architecture
  • Master Server is called a NameNode
  • Slave Servers are called DataNodes
  • Underlying Data Replication across Nodes
  • Interface Language – Java

Installing Hadoop: Installation of Apache Hadoop is not a very easy task, but at the same time it is not too complex either. Understanding of the Hardware Requirements, Operating System Requirements and Java Programming Language can help you install Apache Hadoop without any issues. Installing Hadoop can be either a Single Node Installation or a Cluster Installation. For this post, we will look at only Single Node Installation steps:

  1. Install Oracle Java on your machine – Ubuntu
  2. Install OpenSSH Server
  3. Create a Hadoop Group and Hadoop User and set Key Based Login for SSH
  4. Download the Latest Distribution of Hadoop from
  5. Installation is just extracting the Hadoop files into a folder and editing some property files
  6. Provide the location for the JAVA home in the following file location- hadoop/conf/
  7. Create a working folder in Hadoop User Home Directory /home//tmp
  8. Add the relevant details about the host and the home directory following configuration elements in /hadoop/conf/core-site.xml
    conf/core-site.xml —>
    A base for other temporary directories.
    The name of the default file system. A URI whose
    scheme and authority determine the FileSystem implementation. The
    uri’s scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class. The uri’s authority is used to
    determine the host, port, etc. for a filesystem.
  9. Then we need to edit the hadoop/conf/mapred-site.xml using a text editor and add the following configuration values (like core-site.xml)
    conf/mapred-site.xml —>
    The host and port that the MapReduce job tracker runs
    at. If “local”, then jobs are run in-process as a single map
    and reduce task.
  10. Open hadoop/conf/hdfs-site.xml using a text editor and add the following configurations:
    conf/hdfs-site.xml —>
    Default block replication.
    The actual number of replications can be specified when the file is created.
    The default is used if replication is not specified in create time.
  11. Before running the Hadoop Installation, the most important step is to format the NameNode or the Master Server. This is critical because, Without the NameNode, the DataNodes will not be setup. In a Single Node Installation, NameNode and DataNodes will reside on the same host, where as in Cluster Installation, NameNodes and DataNodes will reside on different hosts. In order to format the NameNode using Hadoop commands, Run the following command – /hadoop/bin/hadoop namenode -format
  12. In order to start the Hadoop Instance, from hadoop/bin run ./ and Running the commands will start up Hadoop and when you query the Java Process, you should be able to see the following components of Hadoop Running:
  13. If you have successfully completed till this, then you now have a Hadoop Single Node Instance running on your machine.

Getting Data in/out of Hadoop:

Once the installation is completed, the next thing we need to worry about is getting data in and out of Hadoop File System. Typically in order to get the data into the system, we need a API interface into HDFS. This typically is a JAVA or HTTP API. Tools like FluentD, Flume etc help in getting data in and out of Hadoop. Both the tools have plugins for receiving HTTP data, Streaming data and Syslog Data as well.

MapReduce: Hadoop and Big data discussions are incomplete without talking about MapReduce. MapReduce is a software policy framework that maps Input data based on a map file and outputs data in key value pairs. These are two different jobs when it comes to actual processing. One is the Map Task that splits the data into smaller chunks and there is the Reduce Job that generates a Key Value combination for each of the smaller data chunks. This framework is the powerhouse for Hadoop because, this is built with parallelism in mind. Map Tasks and Reduce Tasks can both be run parallel on several machines without compromising on speed, cpu and memory resources. The NameNode is the central master that tracks the Maps and the Jobs where as the DataNodes are just providing processing resource.

Finally, Using Hadoop: Now that we know what drives Hadoop and how to get Hadoop installed, the easiest thing would be to start using them. Several examples for MapReduce jobs using Java are available to aid in learning. There are several related projects running to make the Hadoop Ecosystem more scalable and mature. Some of them are:

  • HBase, a Bigtable-like structured storage system for Hadoop HDFS
  • Apache Pig is a high-level data-flow language and execution framework for parallel computation. It is built on top of Hadoop Core.
  • Hive a data warehouse infrastructure which allows sql-like adhoc querying of data (in any format) stored in Hadoop
  • ZooKeeper is a high-performance coordination service for distributed applications.
  • Hama, a Google’s Pregel-like distributed computing framework based on BSP (Bulk Synchronous Parallel) computing techniques for massive scientific computations.
  • Mahout, scalable Machine Learning algorithms using Hadoop

Conclusion: Hope this post helped you in understanding the basic concepts of Big Data and also to setup a Hadoop Single Node Installation to play with. Please do post your thoughts on how Big Data is playing a major role in your organisations.

SIEM – The Good, The Bad and The Ugly – Part 2

In Part 1 of the post, we discussed about the several shortcomings of SIEM that has risen over the years. These problems need to be addressed if we need to progress further in our maturity with SIEM technology. Let me start with the main capability required for a SIEM to function – Log Management. This is where majority of the problems are.

Log Management: The problems plaguing Log Management in a Client/Server model are way too many to comprehend. Centralized log Management is the solution, however we don’t have a sound Log Management solution today that addresses these needs. Let us see what problems are there in Log Management and how we could solve this. Log Management is broadly divided into four parts:

  1. Log Collection
  2. Log Categorization
  3. Log Storage
  4. Log Management (Making it easily searchable across large sets)

Log Collection – Client-less and Standardized: In my view, ideal solution would be that, the source devices should not have any clients installed on them, should not need special formatting, should not be taxed in terms of processing. Standard method of data collection should be set as the norm. From a higher order, an RFC or a Standard should be floated that standardizes all the Log Data from every IT device. This standardization would help in two things – One to improve the Overall Logging and Auditing capabilities of devices (Client Less – Out Of Box) in a standard format and the other is to Improve the Security Consciousness of the Application development teams. Think in terms of Logging being a part of standard protocol suite. What this would do is help interoperability between devices, be it Log Sources, Log Collectors, Log Managers etc along with bringing together all the fragmented parts of the existing log management under one umbrella. For example, Every IDS vendor today supports SNORT formatting. Similarly, all SIEM vendors should support a standard log Collection and processing so that interoperability, migration between vendors etc becomes more easy. I know CEF is one of the standards, but I am not sure everyone adopts that today.

Log Categorization – Security and Non-Security: When large volumes of Log data comes in, there is a need to separate the list of Security events from normal Non-Security events. The Log Standard also should specify categorization of Events clearly as Security and Non-Security. For every device class, the Security Events should be listed out and only those logs should be collected by SIEM for correlation and incident management. Many organizations collect way too much Log data (up to 100K) but effectively use only 10% of it for Security purposes. That means the remaining 80-90% is Non-Security related Logs. And this junk is the one eating up Terra bytes of space. There is a huge disparity between the Log Management tier and the SIEM tier. Data Collection is always more and more, however SIEM processing is more focussed. This is where the categorization helps so that SIEM receives only Security events to process.

Log Storage and Log Management – Streamlined and consistent data sets: The moment we start collecting standard data and properly categorizing them, the storage, indexing and retrieval becomes easier. This space we are good at and should continue to improve. Things like using Big Data Storage technologies instead of relying on Oracle or SQL or MySQL as the backend limits the capabilities when handling big data sets. Storage has to be streamlined and new indexing and searching capabilities should be thrown into the future development of Log Management tools and solutions.

Security Incident and Event Management (SIEM) – Once the Log Management problems are sorted out, the SIEM problems become easier to solve. One of the major pain points in using SIEM was the client-server architecture. When Log Collection becomes Client-less, the SIEM solutions need not focus on building Log Collection clients and instead focus their energies on better correlation and intelligence data mining. Some sweeping changes that can be brought into SIEM world are as below:

  • SIEM should be an inference engine, a correlation engine and a Data mining engine only. This will bring more value in the intelligence piece of Log Data Mining rather than just parsing and doing some basic alerting defined. Remember, as I always say, Security is an Intelligence Function and not an Operations Functions
  • SIEM should be able to focus only on Security events for Alerting, reporting and investigation. This is where the Log Standardization and Categorization plays a big role. If SIEM were to process only Security Data, we would never be hitting more than 5K in most enterprises.
  • SIEM should be a fast and agile product and not rely on backend DB queries, reports and stats usually driven using Oracle or SQL. This is something that some vendors are starting to explore. I know Novell e-Sentinel has this capability for a long time, HP is now trying to do similar thing with CORR. This is in my opinion the right way forward
  • The SIEM should also become a more Active tool instead of being a Passive tool as it is today. What I mean by this is, a SIEM should be able to respond to threats in a comprehensive way. It should be able to alert, do basic ITIL Service Management Integration out of the box and also if need be, execute boxed responses for alerts. This is helpful because, many a times the rules written in SIEM are basic and over a period of time become repetitive. Such repeatable alert responses can be automated into a Workflow and the system should be capable of becoming self-sufficient.
  • SIEM should get better at Large Data Set Mining. Today, SIEM Management consoles are “code heavy” in the front end and “CPU heavy” on the backend. This kinda reduces the efficiency of SIEM technologies in perform real-time correlation. A radical change is needed in the way SIEM applications are developed to make the application lightning fast.
  • SIEM should also provide flexibility in terms of customization of Incident Detection and Response templates, writing visualization rules (where you are able to chart Attack Vectors, Vulnerable points, Network Maps etc), trending Historical Data in a way where the system automatically detects a pattern of similar issues in the past and so on and so forth. In short, add “Intelligence as well as Learning Capabilities” to SIEM.

Again, I am not sure I have covered everything in terms of possible improvements to existing problems, but at least grazed on a few.

What else do you think can help improve the SIEM capabilities? Comment on below.

SIEM – The Good, The Bad and The Ugly – Part 1

SIEM Technology – The Good, The Bad and The Ugly

SIEM is one of those technologies most of the organizations adopt in the wake of Security Log Analysis/Incident/Event Reporting requirements. If you already know what SIEM technology and want to get into the domain, these are the things to know (SIEM – What you need to know). If you don’t know what SIEM is, read it nevertheless!!! This blog post is to talk about SIEM technology by analyzing it critically (even though I am a big fan of SIEM, I believe that maturity comes from review and feedback). Almost a decade ago, SIEM started gaining traction and has come a long way since. Now, I think is a good time to review the technology from a critical view point. So here is my blog on The Good, The Bad and The Ugly!!! This will be a 2 part post, with Part 1 concentrating on Introducing SIEM and then highlighting what it has and has not achieved. Part 2 will concentrate on a proposal/vision on how SIEM should move ahead in the coming years

SIEM is data driven. Data in the form of logs from IT Infrastructure is the key driver for SIEM tools to perform their so called “magic”. Logs have been around in IT for a long time. Logs have been one of the main tools to troubleshoot programs/operating systems etc since long. Gradually, Security gained importance and because of an established logging platform available across IT landscape, Security Events also slowly started to trickle into Logs. With time, along came several compliance and Audit requirements that were driving the Security Log Management domain. Then gradually there arose a need to analyze Log Data and based on the analysis, perform an action. This is where SIM tools gained prominence. This later started to get focussed on Security related incidents and diverged as SIEM. If you look at the pro genesis of SIEM, it has all to do with Data. That is why in today’s world, where Data is exploding in the Internet, it is of utmost importance to understand a technology as SIEM and improve it with time.

What SIEM has accomplished?
For more than a Decade, SIEM has done a lot of things for IT folks. When there was no capability to analyze lines and lines of Log files, SIEM was our savior. SIEM gave us the following capabilities right off the bat:

  1. Process Log Streams from Various Products and standardize them into a single Application data set.
  2. Provide capabilities to work with several thousands of events per second and still give what we need in terms of searching and querying Log data
  3. Provide capabilities to co-relate data from different entities so that we can trace the progeny of an issue
  4. Provide nice Alerts/Reports/Dashboards/Summary for the IT Log data
  5. Finally, a Incident and Event Management Workflow to make it operational.

Several vendors of SIEM (SIM/SEM also used interchangeably but SIEM is becoming standard) exist and google searches will give you more than 20 in number. The SIEM market today has grown into a Multi-Billion dollar market and companies, people etc are all embracing the change.

SIEM Shortcomings: 
While SIEM is a lucrative segment to be in, the problem is that the technology is not mature and has some gaping holes. The technology instead of solving a problem for good, fixed some and introduced several other collateral issues. Let us look at some of them below:

  • Log Management as a technology, as a solution was never mature. We never had good enterprise wide Log Management technologies and tools around before SIEM arrived.
  • Several Log Management issues still exist. These are around Big Data Sets, Standard Log Format Specifications, Integration of Log Sources, Standardization of Applications logging with respect to Security etc. Instead of focussing on fixing these issues, we jumped into SIEM solutions (Log Management + Event Management).
  • SIEM came packaged with Log Management solutions as well, but they were not as efficient as they should be. SIEM came packaged with Event Management Solutions as well, but what is good Event Management, when Log Management is not efficient.
    • Sample this, Windows Logs are resident files in a proprietary format. All Network devices send Syslog messages using the same RFC, but content is varied. Database Audit logs are a mix of Table Data and File Audit Data. When we have a variety of such logs from vendors, there is no way we can effectively perform Log Management and subsequently Event Management
  • One of the best and easiest solution for Log Management was that SIEM vendors packaged a client that can collect and normalize the data into its proprietary format. Then the processed data was sent to a Central Manager where all Event Management capabilities existed.
  • The problem with the above approach is, different data sources need different processing and hence a different client for every data source. Though this seems to be a simple solution at the outset, it adds a layer of complexity in terms of managing the Clients themselves. Imagine this problem for a huge enterprise and you know what a pain point this is for SIEM solutions.
  • Client management is a decentralized approach and hence a failure. Monitoring the health of the client is one of the management headaches one has to bear with. Patching them, updating properties, remote management etc are all points of failure, Not to mention keeping them up and running with constant care and feed like a new born.
  • Since the log standardization in SIEM is in proprietary format, migrating from one system to another, one vendor to another is a pain point. This would require client re-installation and data re-processing. This is a problem where you are stuck with a product for life. Inter-operability between systems has been always a problem for Vendors in IT space. This while protects their business, limits the capability of the end user to get what he wants. The solution cannot be more and more new products, new projects to replace existing SIEM solutions etc. It has to be more robust than that.
  • Searching data across TBs (terabytes) of data is the most important problem every organization faces. How do our SIEM solutions solve this? By using some sort of Databasing and Indexing. All the databases today (Read Oracle/SQL/MYSQL/PGSQL) are all limited in terms of handling such randomly formatted, high volume feeds, thereby rendering long term searches, trend analysis etc a slow, frustrating and time consuming job.
  • Client Server Models implemented by SIEM does not scale for BIG DATA!!! Let me tell you how:
    1. Most of the SIEM solutions I have worked with have 3 layers of architecture – Data Collector Layer (Event Collectors), Data Storage Layer (Event Indexers/Storage) and the Data Processing Layer (Event Management/Administration/Web Console/Server).
    2. In the above architecture, Data Collection and Data Storage is High Volume ranging up to 100K events per second. However, for Data Processing Layer or the Manager Layer, there is a limit of how much it can process (typically in 1/10 – 1/20 of collected data)
    3. If the effective use of Log data is only going to be 10-20%, what about the rest?
    4. People say aggregation and filtering is done to consolidate the data to be within the 10-20% range. Filtering and Aggregation have their own pros and cons but the end result is what you collect, is not used entirely.
  • Managing SIEM solutions (from architecting, implementing, integrating, customization, event management, content development, maintenance etc) is not a simple task and usually requires huge investments in people and training. The vendors make money with this I know, but honestly, being a User, you know that “If it is complicated, adoption will be difficult”
  • Most SIEM solutions are not integrated with ITIL process of Incident and Event Management (A rather standard form for IT framework used across the industry) thereby limiting deployments that should be a seamless transition.

I have to be honest about the fact that the above list is not comprehensive and there are several points you as readers would like to point out as far as the Positives and Negatives of SIEM. Please comment on and I will update the post with your views and comments. Part 2 will be discussing about the various options for SIEM to learn and improve based on Industry feedback and User feedback.