We, at Infosecnirvana have published several posts on SIEM. SIEM as a product has created a unique place for itself in the IT Defense in Depth Strategy and has helped several organizations to effectively detect and respond to security threats as well as rapidly achieve compliance needs. Such an important product in the Security space also has a steep price attached to it. The price is not only in dollar terms, but also in ongoing human effort to manage, maintain and generate value out of it. So it becomes paramount that a right choice is made when it comes to SIEM. This blog post aims to give a set of product evaluation criteria or set of questions customers should ask in the evaluation of SIEM. This guide is using a vendor agnostic approach.
Introduction:
Before selecting a SIEM vendor we need to make sure that the product meets certain selection requirements. Often times, IT organizations have only a vague idea of what is required from a SIEM. They don’t have a solid understanding of the various parameters to be considered in selecting SIEM products. Some of the key requirements to be considered are as follows:
- Company & Product
- Architecture
- Installation & Configuration
- Event Collection
- Event Storage
- User Interface & User Experience
- Certifications
Company & Product:
A SIEM product or any other software product for that matter is as good as the company that develops it. This is key because, a company that is stable and has a long term road map focusses better on product development and building expertise. Hence, evaluating the company also becomes important when buying the product. Some of the key items to look for are:
- Industry Focus, Market presence, Years of experience in the field
- Financial Performance – Subjective Measurements over the past years
- Marketplace opinions and reviews about the product and the company as a whole.
- Do they have customer references for both the product as well as the company?
- Analyst Reports for the last few years – Gartner, Forrester etcetera.
- Know the Executive Leadership of the Company. Is the leadership team strong? Is it Trustworthy?
- Do they have a track record of successful product launches, revisions, development etcetera.
- Licensing and Pricing models
- How strong is their product road map?
- $$$$ spent on R&D for new products VERSUS Development on incremental growth of core product.
- Is the vision of the product group forward thinking? Are they innovative?
- How is the Company’s product support and services group? Is it a dedicated team in house or is it outsourced?
- What are the product support & professional services options available? How focused is the management team in providing Support services? Is support available globally?
- Is there expertise across the vendors partners (VAR, MSSP, Consulting Organizations) to support both basic and advanced consulting needs? How mature is the partnership or alliance relationship?
Architecture:
One of the key components of a product is its architecture maturity. The product should be capable of catering to IT Infrastructure needs that vary from industry to industry, from enterprise to enterprise. Some of the key questions related to Architecture are listed below:
- How flexible is the product deployment architecture? Can it be run as an Appliance, a Software standalone, a virtual appliance/machine or a SaaS?
- Can the architecture be deployed in a way where individual data storage capability is available per business unit/location?
- Does the architecture allow for full data replication for HA purposes? Is HA a built-in function or additional equipment is required?
- Does the architecture allow for interoperability with Network Management devices, System Management devices etcetera.
- Does the architecture support scalability? Is it modular enough to expand based on growth needs, storage needs and performance needs?
- Does the product support granular Role based Access control for the underlying hardware & application software?
- Does it meet the organization’s policy and standards compliance requirements
- Does the product have a secure data transmission between Event Collection, Event Storage and Event Correlation layers? Does it use encryption? If so, how strong?
Installation & Configuration:
- Can the Installation and Initial Configuration be handled by technical staff with minimal training?
- Ease of set-up, Maturity of Product Documentation and Support to facilitate this effort?
- Ease of post install maintenance, patching, routine tuning?
- Ease of patch management of the product including the underlying data architecture.
- How does the product or solution facilitate asset tracking?
- From a log collection perspective, who are the supported Vendors, what Products and Versions are supported for integration?
- How varied and comprehensive is the Data export feature (extract logs, alerts, raw data etcetera.)? Does it support CSV, PDF, HTML, Raw text etcetera?
- Data Workflow Integration (Bidirectional access to information via external workflow tools?)
- Email interface for report distribution, ease of customization of the email templates
- Interface to 3rd party applications (ticketing/workflow application, existing business logging solutions etcetera.)
Event Collection:
- Does the product have support for both Agent based Collection and Agent-less Collection?
- For Agent systems, does the solution support Windows, Unix and Linux Platforms, File readers, XML readers, Structured and unstructured data etcetera.
- For Agent-less, does the solution support Syslog, SNMP, SQL, ODBC/JDBC, and API collection
- Is the Agent management function centralized or is it standalone?
- Does it have any limitations in Input and Output Events Per Sec (EPS)?
- Does it offer the following capabilities to ensure reliability and flexibility?
- Aggregation – Can the Agent aggregate similar information based on custom defined grouping values defined by the System Administrator to cater to the changing Event Collection requirements?
- Bandwidth Throttling – Can the Agent prioritize forwarding of events based on defined values such as event priority? Can it send events at a specific bandwidth rate
- Filtering – Can the agent provide Include as well as Exclude criteria for filtering?
- Caching – Can the agent cache all the events in the event that the Log Store goes down? When forwarding the cache after failure does it intelligently throttle the events?
- Fail-over Capabilities – Can the agent send Log events to a different alternate data store when the principal data store is down? Can it do multiple destination forwarding?
- Transport Integrity – Can the agent encrypt the log transport to ensure confidentiality? What compression and encryption mechanisms are used?
- Health Monitoring – Can the agent send health messages and statistics?
- For Microsoft Windows Event collection can the agent map the GUID/SUID to local registry/names/references for each event ID in the SYSTEM, SECURITY and all APPLICATION logs on the system?
- Are Agents that rely on Event Source Vendor API’s to connect and collect information approved and/or certified by that event source Vendor?
- Can the Agent follow dynamically changing folders and file names? For example in order to support event sources like IIS Web Logs or custom applications that create a log per “site/application” per logging interval?
- Does the Agent support Database Administrator Logging (From both SQL and System / File Based Sources) for Oracle, MSSQL, MySQL and DB2?
- Agent parsing and mapping customization. Can the agent’s parsing be modified to assist with custom log messages? Can the normalization or categorization schema be updated to support custom log messages? How is system default functionality affected if these are modified?
- Can the Agent act as a NTP source for Source Event Logs or otherwise help in time synchronization for source event logs?
- Time difference adjustment feature (to allow the logging system to cope with devices having inconsistent times)
Event Storage:
- Does the product allow storage of data locally, remotely in a SAN or NAS?
- Is the data storage capable of compression? If so what is the rate of compression?
- Is the storage architecture dependent on standard database or does it use proprietary architecture? If proprietary, does it have all the capabilities to meet storage security requirements?
- Is Data Archival flexible? If so, is it built-in? What options does the product have?
User Interface & User Experience:
- Is the interface user friendly or technical? Is the interface a standalone client console or a web console?
- How is the performance of the User Interface?
- Performance when searching for various data elements (IP addresses, usernames, event types, etcetera)
- Performance when generating reports, query results, data extracts?
- Will the product function to support the needs of the Tier based SOC Analysts, Incident Handlers, Responders?
- Can access to data in the system be restricted according to access rights (i.e. business units can see “only their data”)?
- Can the console present just the events a particular analyst is assigned to handle?
- Does the interface allow easy access to actionable data? Does the interface organization require a steep learning curve? Can the analyst drive deeper analysis or via tools with a single action (right click and select)?
- Is the data presented in a manner that makes sense to the analyst?
- Can analyst understand correlation actions?
- Can analyst easily change correlation actions?
- Could the product act as an incident management tool, accepting case notes, and related information on an incident?
- Does the Solution provide graphical Business reporting using visual aids, graphics, dashboards, template based documents etcetera?
- How mature is the reporting capability? Matches or exceeds requirements?
- Ease of development of new reports, customization of existing reports, tuning of generated reports, scheduling reports etcetera.
- How easy is the accessibility to internal, centralized log sources, in normalized and raw form in case of reporting needs?
- How customizable is the reporting query?
- Does it have compliance packages to aid in compliance reporting?
- Does the product allow users to perform Advanced Analysis – Statistical, Visual, Mathematical, empirical?
- Does the product support the most difficult use-cases for correlation? (Multi Vendor, Multi-Event, Custom Application and Custom field correlation)
- Can the system support “live”, “custom”, or “dynamic” threat feeds for live correlation and alerting? Threat Intelligence Feeds such as IP’s, Subnets, Domain, Files, Patterns, etcetera.
- Does the concept of a “Hot-list” or comparison list exist? Automated Hot-list Trigger. A Hot-list can be a watch list or any other static form of data that can be used as a reference point.
- Hot-list Updates are manual or automated?Customizable real-time alerting based on specified criteria
- Distributed search across multiple data stores
- Functionality to initiate certain actions based on real-time alert (sending email/text message, executing script etcetera)
- Is the software designed to track user input to understand how users interact with the system? Objective measures of feature use (misuse)? How is this information mined? Are there any Privilege user management content to audit and track privilege access?
Certifications & Training:
- Training options for the product – Classroom? On-line? Mixed?
- Certification path and criteria
- Continuous training options available for new product releases, feature releases etcetera?
Conclusion: Phew!!! That is a long list of things to consider for SIEM evaluation and I still feel that there are many items I am missing from this list. As mentioned at the beginning, this post is a guide to perform SIEM evaluation and should be a great starting point in terms of a check list creation, Tender floatation, proposal requests and the like. Please feel free to add in the comments section or send me an message if you feel any more items need to be added.
Until next time!!! Ciao
Hi Vinod how are you ?
Concur the above aspects are important and should be part of evaluation process. In addition would like to put the following for consideration:
An effective and efficient SIEM should have metrics such as:
1 are all high risks security events being collated from the touch points (source of events and are being logged) , correlated and present potential security incidents.
2. This means risk assessment has to be carried to identify the risks and how SIEM is more detective control can play effective role
3 to ensure events reaches SIEM , the considerstion on when to use agentless and agent as well should the events be sent from end device directly to SIEM or through Syslog or directly
4 Consider ations should be given on the Security Operations Centre processes and how SIEM conplements and supports operations.
Finally to have effective SIEM, processes and people also needs to considered holistically. Many SIEM implementations encountered issues due to focusing on one aspect.
Brgds
Ram
Hi Ram, I am fine and hopefully you are doing good too.
I completely agree with your views as without solid implementation and operations SIEM seldom generate value
It’s a really nice list, but as you said at the end it is likely to still be lacking some stuff. I’ve put my brain dump of items of which I think they should be added, changed, reordered on the checklist below. it mentions certain SIEM products, but that’s mainly for illustration as to why I think it should be added only.
0. Before starting a SIEM selection tool it is fundamental to know why you are in the market for one as it heavily influences the weight you have to give to certain features. There are four reasons to have a SIEM; Security, Compliance, Operational or Development debugging.
If you only need compliance you can go for a “simple” product such as an ArcSight logger or Sysloghost with limited reporting needs, if you want to use the SIEM as part of your SOC offerings you’ll need a solution that offers realtime processing, threat intelligence options etc.
1. Company and product; I have some additions here
a) How well the company interacts with 3th parties, do they have ecosystem for 3th party offerings; do they offer a SDK for development.
Example; Splunk has a thriving ecosystem, ArcSight not so much.
b) User community; Is there a lively community surrounding the product with official and non-official sources. Splunk is open and has a lot of forums across the web, but the discussions can often be low in value. ArcSight has a closed community that is somewhat active but the content on it is much much better
c) Track record of support given by the company. Some SIEM offerings have dreadful support which are widely complained about.
d) In bullet 7, “do they have a track record” it should also mention big architecture changes, as the change from Oracle to CORR for ArcSight did mean quite a lot of changes that have not been properly communicated.
e) Licensing and Price really needs to have it’s own section as there is so much to say about it; licence for usage, licence for users, licence for devices, multi or single licences, perpetual licence or limited, licence violation policy, licensing terms, costs for products in certain ranges, MSSP licences, costs for support contracts, obligatory support contracts. So much that is relevant in choosing a product, as it always comes down to the costs of the product.
2. Architecture
a) integration with existing SSO solutions
b) User management can be addressed in this more explicitly, but should really have it’s own section.
3. Installation and Configuration
a) Types of context/sources updates and ease of applying/sourcing them.
b) Easy of troubleshooting; is this built in, do you need to do manual checks, is tooling (3th pary maybe even) available for this.
c) Maintenance of Databases. Every Database needs a DBA to care for it at some time.
4. Event Collection
a) Agent placement, where should the Agent be places, local, on a separate machine, directly on the SIEM itself. How flexible is the SIEM software with this.
b) Local management of Agents, or centralised through SIEM product, ease of updating all the Agents when needed.
c) with item 6, “does it offer the following…” you have put emphasis on “reliability and flexibility” but proceeded to also name security and integrity items. I think this could be renamed to agent abilities and can also have the following added, EPS throttling, batching, compression of data, encryption of data, intervals of sending, data integrity checks, device monitoring (does the connected device still exist?), ability to retroactively get events from the source to fill loggaps.
d) Data enrichment options (such as networking, zones, geolocation, threat intelligence,
5. User Interface & User experience.
Though it has a lot of good items I think they section title is wrong, as most items are about reporting. Notifications do seem to miss overall. Depending on your reason for a SIEM this may change quite a bit. There are companies that actively use SIEM, and companies that just want to run a preconfigured report to show to the compliance auditor.
think for a v2 of this checklist the following structure should be used to address everything needed and with it’s own specifics:
00. Reasons for implementing SIEM.
01. Companies and products
02. Licence and Reporting
03. Architectural design
04. Installation and Config
05. Day to day operation
06. Event Collection
07. Event Storage
08. User Management
09. Rules and Notifications
10. Reporting
I will be more then happy to brainstorm with you about extending and updating this checklist in a solid document. Please contact me directly if you are interested.
Hi Frank,
Great points to add on. I would definitely be interested in preparing a detailed document and releasing it as a 2.0 version document. Let me know and we can work on it.
Hi Misnomer, I am assuming you are an admin on the wordpress blog, so you should be able to get my e-mail from the comment manager.
Hello folks! Very nice article and the discussion. I think a huge improvement would be (in v2 maybe?) for some of the sections to include definitions of metrics that could assist in comparing different vendors and products. For example, such capabilities as time needed to develop a new parser or agent for new log source.
Actually Gintautas, I am working on a Version 2 and will be posting it shortly. One of the readers of the blog is co-ordinating on this.
CorreLog checks every item … take a look at CorreLog