Category Archives: Log Management

Splunk Enterprise – What you need to know?


splunk-logo
SIEM posts have grown in number at Infosecnirvana, but the requests to write about more products keep coming in. One of the oft asked about product is Splunk Enterprise. We have posted on HP ArcSightIBM QRadar and McAfee Nitro SIEM. However, readers have been asking us repeatedly to write on Splunk.
So here it is finally after being in the works for a long time
Introduction: 
In 2003, One of the most interesting products rolled out and vowed to simplify Log management once and for all (and it did!!!) –Splunk. Their motto was simple – Throw logs at me and I will provide a web based console to search through it intuitively. Interestingly they are one of the few companies that have not been acquired, in spite of being a very innovative product. So let’s see what makes Splunk tick.
Architecture:
As always, a product is as good as its architecture. It has to be solid both internally as well as externally (meaning solution deployment, integration, ease of use, compatibility etc.).
  • Internal Architecture: Under the hood Splunk has two main services – The Splunk Daemon that is written in C++ used for data collection, indexing, search etc. and the The Splunk Web Services that is a web application written using a combination of Python, AJAX, XML, XSLT etc . which provides the super intuitive graphical UI. Splunk also provides API access using REST and it can integrate with any web framework needed. Splunk is one of the few products that still use C++ and Python instead of the clunky Java and its cousins. This provides the edge to Splunk when processing large data volumes thrown at it.
  • Data Architecture: Splunk is a unique search engine like “data architecture”. In fact, some of the early development was based on the same concept of the path breaking GFS (Google file system) which provided a lot of direction and research into flat file storage, indexing and free text search capabilities with unmatched speed when compared to a relational DB. Splunk went on to master the distributed file system architecture and built their own proprietary data store which powers Splunk Enterprise today.
  • Deployment Architecture: The deployment of Splunk is based on true Big Data Architecture – Slave and Master, where the Slaves are the Search Indexers and the Master is a search head. Of course you can have both the nodes in the same Physical server, but in a true distributed architecture, you need a master and a slave. Read more at Big Data – What you need to know? to understand better on what Big Data is and how to try your hand at it.
  • Typical Setup: Lets look at a typical architecture deployment of Splunk in distributed mode.

Splunk_img4As you can see, there are three distinct components of this architecture and they are as follows:

  1. Log collectors or Splunk Log Forwarders are installed closer to the source and forward all the logs to Splunk Indexers. This is similar to the Log Collectors in SIEM. They are not great, but are decent enough to get the job done.
  2. The Splunk indexers typically run only the Splunk Daemon service, that receives the data and indexes it based on a pre-defined Syntax (this is akin to parsers but lot more simpler and faster to process). This is then sent to the Splunk data store. Each data store has a set of indexes based on the amount of logs received. The data store can then be configured for retention, hot or cold or warm standby etc. etc.  In big data terminology, these are the slave nodes.
  3. These indexers then use a process called as “Summarizer” or in big data terms – “Map reduce” to create a summary index of all the indexes available.
  4. Splunk Search head, which serves as the single console to search across all data stores has the “summary index” to know which Indexer (slave) node to query and what index to query. Now this is where the scalable search power of Splunk comes from. This is the master node in big data world.

What’s good about Splunk? 

  • Search, Search & Search: Splunk is arguably the best search engine for logs out there. We have started looking at ELK, Hadoop and other big data search engines but for the moment, Splunk rules the roost. The Splunk Search Processing Language (SPL) is the reason behind this power. The search can be done historically (on indexed data) or in real time (data before indexing) and this is as good as Log search can get. None of the SIEM products can come close to the search power of Splunk. In other words, Splunk is to search Log Data and SIEM is to search Event Data.
  • Fully customizable as far as searching capabilities is concerned, Splunk lets us add scripts to search queries, provides field extraction capabilities for custom logs, provides API, SDK and Web framework support to achieve all that you would need for Log management, Investigations, Reporting and alerting.
  • Web Interface: Even though UI is a subjective benefit, Splunk has one of the most pleasing interfaces we have seen for log management tools. It really is super easy and intuitive to use. It has great visualization capabilities, dashboards, app widgets and what not. It really puts the cool factor in a rather dull log analysis experience.
  • No Parsing: Basically, Splunk is an “All you can eat” for logs. Splunk follows a “store now, parse later” approach which takes care of receiving any logs thrown at it without any parsing or support issues. If it is a known log type, the indexes are added and updated appropriately. If it is not a known type, still the logs are stored and indexed to be searchable for later. You can then use Field Extractions and build custom field parsings. This is one of the killer differentiators compared to traditional SIEM products as Splunk is a lot more forgiving and agnostic in log collection and storage and does not require specialized connectors or collectors to do the job. This makes it a great log management product.
  • Splunk Apps help in building on top of the Search head to provide parsing, visualizations, reporting, metrics, saved searching and alerting and even SIEM-like capabilities. This, in my opinion is the power of Splunk compared to the other products in the market. They have an App Store for Splunk Apps. Cool isn’t it? These apps not only are written by product vendors, but also by User community.
  • Scalability: Splunk is a true big data architecture. It can scale with addition of Indexers and search heads. Ratio of Search Heads to Indexers is at a good 1:6. This means that if you have 1 search head, you can have 6 search indexers. This is very attractive when compared to other SIEM solutions in the market when it comes to scaling at the log management layer.

What’s bad?

  • Not a SIEM: Splunk is not your traditional SIEM. Let me clarify further. SIEM has several things in it that assists in performing security event management, monitoring, operations and workflow. In short the keyword for SIEM is “Operational Security Management”. Now the question is – Can Splunk be an SIEM? The simple answer is YES, however the real answer lies in how much customisation and how much product expertise  you have in store to make it a SIEM product.
  • Poor Correlation: Splunk does not do any correlation as it is not designed to do that. However, it can be used to correlate events using the Splunk search language. You can do manual correlation using piped searches, lookup tables, scripted searches etc. but again you need to be familiar with the language. You can also automate it by scheduled and real time search triggers. However, nothing is out of the box. Anton blogs about Splunk Correlation being far superior to ArcSight (which btw is the best correlation engine we have worked with) but honestly, we don’t have real life implementation experience to justify that.
  • SIEM App: Splunk has an enterprise SIEM app that aids in SIEM-like functions. But it is definitely not a replacement killer for SIEM product. It is very basic and and does not do much out of the box.
  • No Aggregation: The logs being sent to Splunk are received as is and sent to the data store. It is not aggregated. This while is a good thing for log collection and search performance, it is not good for underlying storage sizing. SIEM solutions have this capability but Splunk does not. This in turn affects the scalability aspect.
  • Poor Compression: Many SIEM products have a compression ratio of 10:1. However for Splunk, we have consistently seen the ratio to be around 4:1. This while good for smaller log volumes, is very poor for larger volumes. The main reason for this is that the Indexes take a lot of storage compared to the raw logs. While they aid in greater search capabilities, they increase underlying storage and maintenance cost.
  • Scalability: Even though, Scalability is one of the benefits of using Splunk for Log management, there is a downside to it too. Add to it the lack of aggregation, compression etc. and you can see how it impacts Scale. For example, Every indexer can handle only 100 – 150 GB/day on a good server hardware. In spite of what people might say about Splunk sizing and performance tuning, from years of personal use and experience, we can safely say that for standard enterprise hardware, this limit is as good as it gets. So assume you are looking at 1 TB/day. You would need 8 indexer servers and 2 search head servers for Splunk. However, if you were to take ArcSight or QRadar, you could do the same on two appliances with compression enabled (10:1 ratio of compression). This from a management perspective leads to larger foot print for Splunk than other SIEM products.
  • Price: Contrary to popular belief, Splunk can get very expensive very fast. For all the reasons mentioned above, Splunk can get very expensive compared to other SIEM vendors to do large data collection as well as SIEM functionality. In a word – Be Cautious!!!

Conclusion: In our opinion, Splunk is one of the most innovative log management tools out there.  But as a SIEM, to use in day to day security management, monitoring, ticketing etc. it has a lot of catching up to do. The ideal scenario will be to use Splunk in the log management layer and use any market leading SIEM in the correlation, workflow and operational management layer. We have seen several successful implementations where Splunk serves as the log management tool and ArcSight or QRadar serves as the Correlation engine. Best of both worlds!!!

Until next time – Ciao!!!

PS: Please feel free to add on to the list of  What’s good and bad? based on your experience with Splunk and we will be happy to update our posts appropriately.

Clash of the Titans – ArcSight vs QRadar


ArcSight vs QRadar
Continuing with the SIEM posts we have done at Infosecnirvana, this post is a Head to head comparison of the two Industry leading SIEM products in the market – HP ArcSight and IBM QRadar.
Both the products have consistently been in the Gartner Leaders Quadrant. Both HP and IBM took over niche SIEM players and have made themselves relevant in the SIEM market.
We have worked on both the products and feel that this comparison is a good way to start the discussion rolling on features of both the products and how they approach the problem of Security Information & Event Management.
Okay, Let’s get started!!!
ArcSight vs QRadar
Subject ArcSight QRadar
Product Birth Year 2000, ArcSight SIEM came into the market and incidentally this was the only product they have worked on. In 2011 HP bought them Year 2004-2005, Q1 Labs entered into the SIEM market modifying their NBAD platform (QFLOW) and in 2012, IBM bought them.
Logging Format CEF – Common Event Format LEEF – Log Event Extended Format
Underlying DB Oracle till 2012, then combination of MySQL, PSQL etc. Proprietary based on Ariel Data store and probably Ariel Query Language (AQL)
Vendor Support ArcSight supports more than 400 vendors with their CEF certification program QRadar supports more than 250 vendors with their LEEF certification program
Portfolio Log Correlation – HP ArcSight ESM

Log Management – HP ArcSight Logger

Identity Correlation – HP Identity View

Intelligence Feeds – HPRepSM

Threat Detection – HP ArcSight Threat Detector

Response and Action – HP ArcSight TRM

Log Correlation – IBM QRadar Console

Log Management – IBM QRadar Log Manager

Network Forensics – IBM QRadar NBAD (using QFlow)

Intelligence Feeds – IBM X-Force

Vulnerability Management – IBM QRadar VM (with dedicated Scanner)

Response and Action –  IBM QRadar Incident Forensics for Response only

Identity monitoring ArcSight has a separate feature called IdentityView (separate license) to provide the identity perspective of events occurring in ArcSight. It integrates with Identity solutions (AD, Oracle) to keep track of user activity regardless of the account being used. It assigns risk scores to users based on their activity, and can graphically represent this activity and compare it to others with similar roles. QRadar does not have the capability similar to Identity View, however, it does integrate with Identity solution to provide user information in the offenses created.
Network Behavioral Analysis ArcSight does not natively collect flow data however, it can obtain Netflow data from other devices such as routers, etc. The Netflow data provides visibility only up to layer 4 (no application visibility) QRadar due to its origin as a NBAD product has powerful Network Behavioral Analysis (NBAD) capability through its  QFlow appliance (Network Flows data including Layer 7 flows, Jflow, Netflow, IPFIX, SFlow, and Packeteer’s Flow Data Records can be collected and processed). This would allow us to review application and network flows and assess it for anomalous traffic, persistent threats etc.
Vulnerability Management ArcSight can integrate with Vulnerability scanners and gather Scan reports for correlating vulnerability information with the security events collected.  However, it is more of a data aggregator in the case of VM tools. QRadar has a Vulnerability Management product (QVM). This has all the features comparable to ArcSight, however, IBM has upped the ante in this space by including a Scanner in the product that can actively scan hosts if enabled with QVM license. This provides security analysts to gather real time information if they choose to from the same SIEM console.
Dynamic Risk Management ArcSight does not have any risk management capabilities. However, it can integrate with commercial risk management products to provide basic correlation QRadar has a Risk Manager (QRM) product that collects Network configuration information and provides a risk modeling capability to assist in understanding the extent of impact of a configuration change in the network. This is akin to Skybox, Algosec or RedSeal and perform in similar capacity
Log Collection Agent Less – Using Connector Appliance. Logger Appliance can also serve as Log receiversAgent Based – Software Install on Servers for all types of log collection Agent Less – Any QRadar Appliance, Console, All-in-One Combo boxes, Event Collector etc. can collect Logs remotelyAgent Based – Connector software available for Windows. For others, Agentless is the only option.Flow Collection – By default any appliance can collect flow data, however, dedicated Flow Collectors are an option in QRadar.
Log Management Separate Log Management Software, Appliance which is different from the ESM appliance. They have a Express version which combines both but in general HP Logger fills the space of a dedicated Log Management appliance Same software, same appliance can behave as all in one SIEM + Log Manager or dedicated Log Manager or SIEM depending on License added. There is no distinct product differentiation as in ArcSight family.
Event Transmission Events from the source are sent in clear text to the SmartConnectors, however, all further upstream communication happens encrypted. Compression and Aggregation can also be employed in the ArcSight ecosystem from the connectors onwards. Events from the source are sent in clear text, however, communication between QRadar Appliances happen using encrypted SSH tunnels. However, compression happens on Appliance at event storage level and does not happen in event transit.
Handling EPS bursts ArcSight uses large buffers to cache events in case of an EPS burst. Once the buffer is filled, the Queue starts to fill. Once the queue overflows, events get dropped. But the burst EPS can be sustained for longer periods of time compared to QRadar. In QRadar, Each event type has a memory buffer, once the EPS exceed the licensed level and the buffer is filled, all new events are queued and processed on a best effort basis. However, this burst EPS is not sustainable for longer periods of time as with ArcSight. So even though it can take burst EPS during times of attack, it is not sustainable.
Filtering ArcSight provides the ability to filter or modify events at the collection and logging level to eliminate the events that are not of security value. This can be as close to event source as possible using SmartConnectors QRadar provides capability to filter using Routing rules. However, for field based filtering (where only one field from the log needs to be omitted during parsing) can’t be done in QRadar.
Aggregation Log Aggregation can be done based on any field combination. This is really useful when it comes to toning down on the high volume logs of network firewalls and proxies etc. Log Aggregation or Coalescing in QRadar terminology happens at the event collection layer based on the source IP and user only and not on customizable field combinations
Data obfuscation ArcSight allows for obfuscating any field at the log collection level using SmartConnectors. This is very powerful when monitoring confidential data in logs. QRadar does provide Obfuscation abilities using a custom Regex Based, Key Based Obfuscation config. This will allow for encrypting a field, based on the Regex Match when event is processed.
Custom Log Collection Require development of customized configuration files. However, ArcSight Flex Connector SDK is  a very powerful tool to build custom connectors and parsers. Also, the ArcSight community shares knowledge about custom connectors and hence more help available in case you want to develop on your own. QRadar has two parts of custom log collection capability. For supported logs or generic logs, it can update/develop parsers using the “Extract Custom Property” feature. However, if a new log source is to be integrated, then it is through customized configuration files which is much harder to create, test and maintain. Also, help to develop on your own is scarce so Professional services is mandatory.
Scalability ArcSight is really scalable such that it can support multi-tier Correlation Engines, multi-tier Loggers, Connectors etc. and also have effective peering. QRadar scales very well horizontally at the Log Collection layer, however at the Correlation layer it does not scale as well as ArcSight. This is a challenge in large and distributed environments.
High Availability One of the long standing issues of ArcSight is HA. It does not have a true HA capability. It supports failover routing at the Collection layer but does not have any thing at the correlation layer. QRadar has the most simple to setup HA configuration ever. This allows sync of two Appliances in true HA style.
Multi-Tenancy ArcSight has always been the SIEM product of choice for MSSP vendors. The main reason being the ability of the product to delineate events based on customers so that monitoring can be efficiently  performed in a MSSP environment. It maps IP addresses to customer names and network zones to avoid overlap. QRadar did not have the feature until recently (I think v7.2 and above) and was one of the reasons it had very poor Multi-Tenancy support. However, the new feature with “Domain” based categorization provides ability to support MSSP environments. Maturity is yet to be achieved but it’s a step in the right direction.
Out-of-the-box use cases ArcSight’s out-of-the-box use cases are very light compared to and only include limited Multi-Device/Event correlation use cases. QRadar comes with a comprehensive set of basic out-of-the-box use cases for various threat types  such as malware, recon, dos, authentication and access control, etc. Also, several of these use cases are Multi-Device/Event types.
Customizable dashboards and reports ArcSight reporting system includes over 350 standard report templates that address common compliance and risk requirements. The report design system is similar to what you would find in a BI solution, though not as complex. Support for charts and graphs is available, and templates can be customized through Velocity. Reports can be scheduled and distributed automatically by e-mail. QRadar provides over 2000 report templates relevant to specific roles, devices, compliance regulations and vertical industries. Only basic report customization is available. However, if advanced report customization is required, QRadar reporting seems limited. However, majority of the customers using QRadar are happy with the out-of-the box reports.
Case management ArcSight has a built-in case management system that allows the association of events to cases, limited workflow, and the ability to launch investigation tools (anything that can run from a command-line) directly from the console. Cases can contain analyst notes and customizable fields. QRadar  provides a rudimentary case management capability through its Offense Management. Offense Management provides basic features such as open, close, assign, and add notes. Additional events cannot be added to Offenses. This is in stark contrast to ArcSight which has full blown case management system built in.
User portal ArcSight requires a java client to provide most of its functionality, but also provides a web interface primarily for business users. Provides all functionalities for security event monitoring and threat content development through web based GUI
User licenses Individual console licenses should be purchased for each user to perform investigation/monitoring Additional user licenses are not required to be purchased
Pricing Pricing is based on number of log sources and total log size per day Pricing is based on EPS. Linear incremental cost for scaling the solution is based on tier based EPS licensing.
Updates:
This section is for posting differences based on reader feedback. So readers, feel free to add on.

Pattern Discovery

ArcSight has something called a Threat Detector tool. It basically runs a set of search queries on real time data and provides patterns detected. If interesting monitoring patterns are detected, they can quickly be converted to Use Cases. This is basically useful if you want to create new use cases and you don’t know where to start

QRadar does not have anything similar to Pattern discovery.

Compliance

ArcSight has compliance packages that can be purchased to aid in providing compliance specific alerting, reporting etc. However, these are priced separately.

QRadar has more than 2000 reports grouped based on Compliance requirement which should mostly satisfy compliance needs

I think the list can still be improved based on your feedback.  Please feel free to add them in the  comments section below and the feedback will be incorporated.
Until next time – Ciao!!!

Evaluating SIEM – Version 2.0


We are Infosecnirvana have posted Evaluating SIEM – What you need to know?. Readers had a lot of good comments on this post and one of the readers – Frank Bijkersma who blogs here came up with an idea to expand the content in the post to make it a Whitepaper which will serve as a good post. So here you go – Evaluating SIEM – Version 2.0.

Evaluating SIEM - Version 2.0

So here it is. Please feel free to read the guide and let us know your thoughts on it.