In Part 1 of the post, we discussed about the several shortcomings of SIEM that has risen over the years. These problems need to be addressed if we need to progress further in our maturity with SIEM technology. Let me start with the main capability required for a SIEM to function – Log Management. This is where majority of the problems are.

Log Management: The problems plaguing Log Management in a Client/Server model are way too many to comprehend. Centralized log Management is the solution, however we don’t have a sound Log Management solution today that addresses these needs. Let us see what problems are there in Log Management and how we could solve this. Log Management is broadly divided into four parts:

1. Log Collection
2. Log Categorization
3. Log Storage
4. Log Management (Making it easily searchable across large sets)

Log Collection – Client-less and Standardized: In my view, ideal solution would be that, the source devices should not have any clients installed on them, should not need special formatting, should not be taxed in terms of processing. Standard method of data collection should be set as the norm. From a higher order, an RFC or a Standard should be floated that standardizes all the Log Data from every IT device. This standardization would help in two things – One to improve the Overall Logging and Auditing capabilities of devices (Client Less – Out Of Box) in a standard format and the other is to Improve the Security Consciousness of the Application development teams. Think in terms of Logging being a part of standard protocol suite. What this would do is help interoperability between devices, be it Log Sources, Log Collectors, Log Managers etc along with bringing together all the fragmented parts of the existing log management under one umbrella. For example, Every IDS vendor today supports SNORT formatting. Similarly, all SIEM vendors should support a standard log Collection and processing so that interoperability, migration between vendors etc becomes more easy. I know CEF is one of the standards, but I am not sure everyone adopts that today.

Log Categorization – Security and Non-Security: When large volumes of Log data comes in, there is a need to separate the list of Security events from normal Non-Security events. The Log Standard also should specify categorization of Events clearly as Security and Non-Security. For every device class, the Security Events should be listed out and only those logs should be collected by SIEM for correlation and incident management. Many organizations collect way too much Log data (up to 100K) but effectively use only 10% of it for Security purposes. That means the remaining 80-90% is Non-Security related Logs. And this junk is the one eating up Terra bytes of space. There is a huge disparity between the Log Management tier and the SIEM tier. Data Collection is always more and more, however SIEM processing is more focussed. This is where the categorization helps so that SIEM receives only Security events to process.

Log Storage and Log Management – Streamlined and consistent data sets: The moment we start collecting standard data and properly categorizing them, the storage, indexing and retrieval becomes easier. This space we are good at and should continue to improve. Things like using Big Data Storage technologies instead of relying on Oracle or SQL or MySQL as the backend limits the capabilities when handling big data sets. Storage has to be streamlined and new indexing and searching capabilities should be thrown into the future development of Log Management tools and solutions.

Security Incident and Event Management (SIEM) – Once the Log Management problems are sorted out, the SIEM problems become easier to solve. One of the major pain points in using SIEM was the client-server architecture. When Log Collection becomes Client-less, the SIEM solutions need not focus on building Log Collection clients and instead focus their energies on better correlation and intelligence data mining. Some sweeping changes that can be brought into SIEM world are as below:

• SIEM should be an inference engine, a correlation engine and a Data mining engine only. This will bring more value in the intelligence piece of Log Data Mining rather than just parsing and doing some basic alerting defined. Remember, as I always say, Security is an Intelligence Function and not an Operations Functions
• SIEM should be able to focus only on Security events for Alerting, reporting and investigation. This is where the Log Standardization and Categorization plays a big role. If SIEM were to process only Security Data, we would never be hitting more than 5K in most enterprises.
• SIEM should be a fast and agile product and not rely on backend DB queries, reports and stats usually driven using Oracle or SQL. This is something that some vendors are starting to explore. I know Novell e-Sentinel has this capability for a long time, HP is now trying to do similar thing with CORR. This is in my opinion the right way forward
• The SIEM should also become a more Active tool instead of being a Passive tool as it is today. What I mean by this is, a SIEM should be able to respond to threats in a comprehensive way. It should be able to alert, do basic ITIL Service Management Integration out of the box and also if need be, execute boxed responses for alerts. This is helpful because, many a times the rules written in SIEM are basic and over a period of time become repetitive. Such repeatable alert responses can be automated into a Workflow and the system should be capable of becoming self-sufficient.
• SIEM should get better at Large Data Set Mining. Today, SIEM Management consoles are “code heavy” in the front end and “CPU heavy” on the backend. This kinda reduces the efficiency of SIEM technologies in perform real-time correlation. A radical change is needed in the way SIEM applications are developed to make the application lightning fast.
• SIEM should also provide flexibility in terms of customization of Incident Detection and Response templates, writing visualization rules (where you are able to chart Attack Vectors, Vulnerable points, Network Maps etc), trending Historical Data in a way where the system automatically detects a pattern of similar issues in the past and so on and so forth. In short, add “Intelligence as well as Learning Capabilities” to SIEM.

Again, I am not sure I have covered everything in terms of possible improvements to existing problems, but at least grazed on a few.

What else do you think can help improve the SIEM capabilities? Comment on below.