A SIEM system (Security Information and Event Management) is often used by security operations centers (SOCs) for real-time detection of suspicious activity and security events.
While some teams choose to adopt a purpose-built SIEM, others rely on the same DevOps tools they are already using for tasks like troubleshooting and operational log data analysis.
That’s why we’re here to explore whether the ELK Stack (Elasticsearch, Logstash, and Kibana) can be sufficiently used to replace SIEM tools.
While the ELK Stack is not a SIEM itself, it can be used to build one. However, alternative approaches like a modular security data lake are less costly, and can augment existing real-time monitoring and observability solutions you already have in place.
Let’s learn more.
Understanding SIEM tools
SIEM systems have become an important part of a cybersecurity tech stack, because of their ability to effectively monitor user activity and track any anomalies in data. In addition, SIEMs can help SOC analysts gain visibility into who is accessing your systems, when, and by what means. They work by gathering event files and log data across various systems, and normalizing the data into a standard format so it can be effectively analyzed.
While SIEMs are great for real-time threat detection and prevention, advanced persistent threats (APTs) are increasingly common. APTs linger in your system and engage in activity that appears to be completely normal at the surface level. Over time, an APT may escalate its privileges and exfiltrate sensitive data. When an SOC is focused on detecting real-time threats, an APT may remain unnoticed. However, by analyzing long-term log files, security teams can more effectively engage in proactive threat hunting to track down these bad actors.
Now that we know more about a SIEM, let’s look at the basic capabilities of the ELK Stack.
ELK Stack Overview
The ELK Stack, also known as the Elastic Stack, comprises three open-source software tools: Elasticsearch, Logstash, and Kibana. When integrated, these tools create a solution for centralizing log data aggregation, management, and querying, applicable to both on-premises and cloud-based IT environments.
Want to learn more about Elasticsearch, Logstash, and Kibana? Check out the Ultimate Guide to ELK Logging Analysis.
Can you use the ELK Stack as a SIEM?
The ELK stack itself is not a SIEM, but can be used to build one. The ELK stack shares many common features with a SIEM on its surface.
For example, you can collect and query log data from a variety of sources. However, it’s important to note that you must configure it appropriately for real-time security threat detection and carefully monitor its outputs, which may be a significant challenge for under-resourced teams.
Another important aspect of a SIEM is the log processing pipeline in Logstash. SIEMs typically normalize data coming from different sources — meaning that data is broken down into meaningful field names, mapped to the right field types, and enriched with more information (for example, geographical information). Without data normalization, it’s incredibly difficult to analyze your log data in Kibana. Configuring Logstash to process various log file types is complex, and requires dedicated expertise.
With that said, many teams choose to use open-source tools as they architect their own SIEM solution. The pros? Open-source tools like the ELK Stack are cost-effective to get started with, and can be customized in a number of ways. There’s also a vast community supporting the open source solution. But, the downsides of ELK such as the management complexity, resource intensity, hidden cost centers (e.g. the high costs of log ingestion and retention), and expertise required may deter many from using ELK as a SIEM.
Retention can be an issue, in particular. Managing Elasticsearch clusters is costly and complex. If you don’t retain enough historical log data, it can be nearly impossible to determine whether a security event is anomalous or normal.
As one Quora user put it, “While you certainly can run security analysis on ELK, SIEMs typically provide additional out-of-the-box workflow features for Security Analysts that are vital when handling threats.”
For example, if you’re using ELK, you’ll need dedicated expertise to be able to normalize various log data formats, query data, create dashboards and reports, and correlate data with rules so that you receive alerts when an incident occurs. SIEMs also include dedicated incident management workflows, which are not provided in the ELK Stack.
Alternatives to ELK Stack for SIEM
There are a wealth of SIEM tools available today from companies including Splunk, QRadar, Elastic (the proprietary software company behind the open-source ELK Stack), and more. These tools are effective for short-term security workloads, such as real-time threat detection and alerting.
Some SOCs find shortcomings with SIEMs, due to the sheer volume of cloud telemetry data.
The most common SIEM challenges:
- Cost and Maintenance: The upfront costs, along with the costs of setting up, fine-tuning and maintaining a SIEM can add up. This includes the cost of data retention and processing. Security analysts often need to manually tag and continually update these systems in order to keep signals accurate, which can take a lot of time for companies facing a security talent shortage.
- Data retention: Many SIEMs only retain data for 30 days due to cost issues, which limits the ability to investigate advanced persistent threats.
- Alert fatigue: SIEMs throw a lot of false positives, which makes it hard for security analysts to distinguish between signal and noise within their real-time alerts.
Consider a Modular Security Data Lake (with or without XDR)
Another option is to embrace a modular security data lake option. Many teams choose to augment Splunk and other SIEMs with a solution that’s built for historical data analysis. Data lake solutions like ChaosSearch, for example, leverage low-cost cloud object storage like Amazon S3 to avoid the high costs of ingestion and data retention.
A modular security data lake combined with an XDR (Extended Detection and Response) solution is another option to consider. Security attacks today are increasingly sophisticated and rarely exploit a single endpoint. An XDR can move beyond the limits of a SIEM by providing comprehensive monitoring of the entire attack surface. Having this broader visibility means that an XDR can identify more patterns in your data to detect potential threats. The goal is to help security teams correlate seemingly disconnected events, to take immediate action and mitigate cybersecurity threats.
Organizations should carefully assess their specific requirements and resources before choosing ELK as a SIEM or consider alternative SIEM solutions like the ones presented above. For example, you might ask yourself these questions before building your own SIEM with the ELK stack:
- Do you have the resources to appropriately configure your ELK Stack for security workflows? For example, can you connect your ELK-powered SIEM with the right incident management tools for your team?
- Do you have enough storage capacity to handle the indexing and retention complexity of the ELK Stack?
- Is your organization prepared for the fluctuating costs associated with this storage?
- Are you able to get the historical data you need from your SIEM solution to conduct threat hunting?
If your answer to these questions is “no,” a modular security data lake may be a more cost-effective solution, augmenting long-term workflows like threat hunting, while leaving real-time workflows to a purpose-built SIEM.
