On Friday, March 12th, ChaosSearch announced that its ChaosSearch Data Platform won Gold for two product categories in the 2021 Cybersecurity Excellence Awards: Best Security Analytics Solution and Best Security Log Analysis solution.
Of course, we are thrilled to be recognized for our leadership and innovation in security analytics, but beyond that, these awards help to highlight ChaosSearch’s advantages in an area of growing importance to Security Operations (SecOps) teams.
Taking two golds in one of the cybersecurity industry’s most prominent award events validates something our customers have known since our inception a few years ago: The ChaosSearch Data Platform is an ideal foundation of an enterprise’s overall cybersecurity architecture.
We thought we’d take this opportunity to provide a little more background on what led the Cybersecurity Excellence Award judges to name us winners of these two key categories, and what the broader implications are for SecOps teams looking to improve their overall security posture by improving their analytical capabilities.
Read the Solutions Brief: Scalable Log Analytics for Security Operations and Threat Hunting
What Role Does Analytics Play in Cybersecurity?
At its heart, cybersecurity is an analytical field. SecOps teams are in a state of continual analysis—analyzing new threats in the global landscape, analyzing specific vulnerabilities in their environment, analyzing various alerts to determine whether benign or malicious, etc. A platform that centralizes the important data (inputs) and allows for analysis, queries, visualization, and reports (outputs) is crucial to a well-functioning, high-performance SecOps team.
Log data specifically is the lifeblood of the cyberdefense operation.
Log data is where every transaction and every event across an organization’s entire IT landscape is captured and stored. Specifically, log data can provide details on extreme traffic, unauthorized access, suspicious changes, and many other pointers used to identify potential threats. As these logs are consolidated and stored, they become a “single source of truth”, giving SecOps teams the ability to scan history to see what has happened and take the pulse of the current situation to assess threats and what may happen.
To make use of log data, SecOps teams require a centralized log management and analytics system to organize and index data and make it searchable. In a recent report, Gartner examined the role of a centralized log management platform for security. The diagram below highlights some of the common uses of log analytics in an organization’s security operation.
What Are the Most Important Factors for a Log Analytics Platform for Security?
In conducting the various analyses that allow them to keep their corporate assets safe, SecOps teams always want more data, rather than less. Provided they have the means to quickly and efficiently analyze it, having access to more data, from more sources, including long-term historical data allows them to more effectively find and block threats, stop hacks in their tracks, mitigate the damage of threats, and investigate breaches that did occur to find and fix the root cause problems.
Given the thirst for more data, the most important quality of a centralized log analytics platform is its ability to scale. This is where most solutions fall short today (more on that later). As SecOps teams evaluate centralized analytics platforms, scalability should be assessed in a few dimensions:
- Scaling-Up: This refers to the amount of data that can be ingested and processed each day. As data comes flooding in from multiple sources, can the platform easily be scaled up to ingest the data without creating undo hassles like creating multiple repositories and sharding databases?
- Scaling-Out: This refers to the ability to scale the total capacity of the repository where the data is stored.
- Data Retention: This is related to both points above, but is worth emphasizing in the context of cybersecurity. Many of the analytical functions performed by SecOps teams today require access to long-term historical data.
There is an important underlying factor that impacts a system’s ability to scale up that is worth highlighting here: The ability of the system to ingest new data sources easily, without significant planning or disruption.
As we survey SecOps teams today, we find that while they all use analytics platforms in their overall operation, their use is suboptimal. In most cases, the analytics platform in use is limited in scale, forcing SecOps teams to ingest less data from less sources, and retain data for shorter periods of time. A common by-product of the inherent scale limitations is to create multiple islands of storage. While this approach achieves more overall capacity, it comes with the problems of managing a sprawling environment of disparate clusters, leading to significant cost, complexity, and wasted time of the team.
What Are the Biggest Advantages of the (2x Gold Winning) ChaosSearch Data Platform Over Other Log Management Solutions?
You might have guessed the answer already: The core advantage of the ChaosSearch Data Platform – and the key to the two Golds -- is its ability to massively scale, both up and out. While that answer might have been obvious, the way we do it is completely unique.
The ChaosSearch Data Platform does what the legacy platforms cannot—it allows customers to collect and analyze ALL of their data without tradeoffs, with virtually unlimited scalability, and with massive cost savings. Our customers typically see 50-80% annual cost savings compared to their legacy solution.
The ChaosSearch approach solves the two primary inhibitors of the legacy solutions that prevent them from scaling efficiently. First, legacy solutions have complex data pipelines which entail data parsing or transformations on ingest. This introduces a significant bottleneck and reduces the maximum data volume that can be processed daily. Secondly, they are “closed systems”, which means that once data is ingested, they take custody of the data and are responsible for not just the data itself but the underlying layers of the infrastructure stack. This impacts both the daily workload and the overall storage capacity of the system, which in turn, limits the customer’s data retention period.
Instead of ingesting, manipulating, and managing the data as a custodian, ChaosSearch simply connects to and indexes all of the data within the customer’s cloud storage environment- such as Amazon AWS or Google GCP. This is a fundamentally different approach. As new data floods in daily, ChaosSearch continually indexes it in real-time without any performance hit to the data ingest speeds. Since we never take custody of the underlying data, we avoid complex data pipelines and avoid the need for any data parsing or data transformation. ChaosSearch indexes data as-is, while auto-detecting native schemas.
Once indexed, the ChaosSearch platform allows customers to conduct search and analytics on the data using existing tools in use today, like Kibana, leveraging the open APIs of these tools. This unique approach allows us to enable all of the required security analytics use cases that customers need, while leveraging the performance, scale, and economics of the public cloud. It’s this approach that is key to customers like Armor selecting ChaosSearch to run all of their security analytics.
“With our daily log and event volume exceeding tens of terabytes a day and growing, it became clear that we needed to find a new log analysis solution which could scale with Armor’s growing business,” said Josh Bosquez, Chief Technology Officer at Armor, a cybersecurity software provider with over 1500 customers worldwide. “We explored several options, and found that ChaosSearch could deliver the reliability, scale and expanded retention we needed.”
With ChaosSearch, SecOps teams can switch from legacy systems that are limited in performance and scale, and adopt a security data lake approach that allows them to easily manage massive volumes of security event data from multiple sources, with near-infinite scalability.
By allowing customers to centralize more data, from a wider range of sources, and retain data online for 12 months or longer, ChaosSearch enables significant improvements in several security use cases that require log analysis including monitoring and alerting, forensic investigations, threat hunting, and several others.
With this unique approach, we are helping companies realize the vision and the promise of the data lake. This is really the key that allows us to deliver unlimited scalability, industry-leading resiliency, and massive time and cost savings.
What is a Security Data Lake?
Given our approach, many customers and analysts suggest that ChaosSearch helps them achieve a security data lake. A security data lake is a central repository of all security-related data designed to support cost-effective high-volume data storage and expanded data access for the SecOps team, leading to increased data utilization, better analyses, and ultimately a superior overall security posture.
As the diagram shows, data comes streaming into the centralized security data lake from a large number of sources, and the goal is to make all of that data accessible to the SecOps teams when and how they need it, by making the entire data lake searchable and ready for analytics.
Thus, the core attributes of an effective security data lake include:
- Easy and frictionless ingest for ALL data
- Massive Scalability:
- Scale Up: TBs/Day
- Scale Out: Unlimited Capacity & Long-term data retention
- Scale Economically
- Simplified Management
- Simplified Access, Search, Query and Analysis
Indeed, this is exactly what ChaosSearch enables for SecOps teams. The diagram below brings it together to show how the ChaosSearch Data Platform connects to a customer’s existing cloud storage—in this case either Amazon AWS or Google GCP – and allows for various analytical use cases on all of the data within the security data lake.
How Does ChaosSearch Help an Organization Improve its Overall Security Posture?
So, now that we’ve demonstrated what’s different about the ChaosSearch Data Platform, and how our approach delivers superior scalability and economics compared to all other alternatives, let’s bring it back to how customers today leverage the ChaosSearch data platform to improve their overall cybersecurity operations. After all, it is how we help the SecOps team that won us the two Gold Medals!
Generally, we’ve shown that SecOps teams need more access to more data, in order to do their jobs effectively. And we’ve shown how ChaosSearch has clear advantages over alternatives, by enabling a massively scalable security data lake approach. Where the rubber meets the road is how SecOps teams use the data in various security use cases that require access to data at scale. Below are a few prominent examples of how our customers use the ChaosSearch Data Platform today to improve their overall cyber defenses and threat readiness posture.
- Improved ability to find advanced persistent threats (APTs): The persistent nature of this type of attack makes it necessary to look back further in time to identify the pattern of attack commonly used by APTs. The probing and infiltration stage can take months, and that mandates the need to review data collected over a long period of time.
- Support for threat hunting: As threat actors become more clever and harder to spot, SecOps teams benefit from having more data to analyze and drive pattern recognition. As threats attack different aspects of infrastructure at different times, having both more data and data that reflects a longer time frame improves the analytics or machine learning/AI that supports threat hunting. Importantly, the median dwell time for an incursion is 56 days, based on the latest research from FireEye. Unfortunately, if the log data covers only a 14-day period, any incursion that happened weeks before won’t be easily identified. Increasing the retention period to go beyond average dwell time improves the odds of finding incursions that occurred weeks or months ago.
- DDoS Attack: The ChaosSearch Platform’s scale and resilience also enable customers to quickly identify and neutralize threats as they are in progress, including DDoS attacks. ChaosSearch integrates with popular content delivery networks (CDN) and security services like Fastly, Cloudflare, AWS CloudFront, Carbon Black, Auth0, and Okta, helping customers understand application usage, traffic patterns, location of origin, and when and where their website or application has been compromised. ChaosSearch’s built-in alerting enables them to set thresholds to tag and automate response to threats in near real-time.
Stephen Salinas, Engineering Lead at HubSpot, commented, “ChaosSearch now serves as one of our team’s primary monitoring tools for identifying DDoS attacks and protecting our customers from them. The additional data retention also serves to help our security team audit issues over past months to better identify bad actors.”
The ChaosSearch Data Platform Forms the Foundation of an Enterprise Cybersecurity Architecture
The significance of the Cybersecurity Excellence awards is that it helps highlight the crucial role of a scalable, centralized analytics platform in an organization’s overall security operation.
Given its unique “data lake” approach, ChaosSearch delivers unmatched scalability, both in terms of the daily data ingest rate and the long-term data retention period, giving SecOps teams more access to more data—exactly what they need to conduct various mission-critical operations such as forensic investigations, threat hunting, and DDoS attack mitigation. With the growing recognition of the need for superior analytics in cybersecurity, ChaosSearch is an ideal solution to deploy at the foundation of an enterprise’s overall security architecture.
