The market for enterprise observability solutions is growing in 2022, as organizations search for more effective ways to maintain security and oversight of increasingly complex and distributed IT systems.
Traditional observability solutions like Splunk, Datadog and New Relic are still widely used by enterprises to analyze logs, metrics, and traces from their IT environments. But as enterprises generate increasing volumes of log data, two things tend to happen:
- Log pipelines that move logs from the source application to the observability solution become increasingly time-consuming to manage, and
- The daily cost of ingesting logs into the observability solution explodes out of control.
The challenge for companies generating massive log data volumes is to make IT observability more cost-efficient (better economics around data movement and storage) and easier to manage (more functional log pipelines) without imposing data retention limits that restrict log analytics use cases (e.g. long-term trend analysis, root cause analysis, etc.) and waste valuable opportunities to generate insights.
And solving these challenges is exactly where ChaosSearch + Cribl for Observability can help.
In this blog, you’ll discover how Cribl is gaining traction in the observability space with its unique approach. Keep reading to learn the top five benefits of using ChaosSearch with Cribl for observability.
What is Cribl Stream?
Before launching Cribl, founders Dritan Bitincka, Ledion Bitincka, and Clint Sharp spent a combined 20+ years working at Splunk.
Then, in 2017, the trio noticed a shift in the observability landscape: customers were beginning to experience unprecedented log data growth, and many were struggling to manage complex observability pipelines or generate value from observability tools while ingesting terabytes of logs per day. Recognizing the opportunity to address these challenges through innovation prompted Cribl’s founders to leave Splunk in 2017 and launch their new company just 15 months later, in June 2018.
Cribl’s flagship product is Cribl Stream, a vendor-agnostic observability pipeline tool that allows enterprises to:
- Consolidate logs, metrics and traces from any source,
- Clean, transform, normalize, and enrich the data, and
- Route the data to multiple destinations (e.g. analytics tools, data lakes, cloud object storage, etc.).
Cribl’s founders knew that enterprise data engineers were spending too much time building and managing data pipelines, so they purpose-built Cribl Stream to put customers back in control of their data and make it easy to route data at scale from any source to any destination.
This value proposition has resonated strongly with both customers and investors. Cribl’s fast-growing customer base includes 10 of the Fortune 50 companies, along with globally recognized firms like Accenture, Autodesk, TransUnion, and Vodafone. On the investment side, Cribl has generated over $400 million in funding, including a $150 million Series D round in May 2022 that valued the company at $2.5 billion.
Cribl’s Revolutionary Approach to Observability
As momentum increases behind enterprise observability solutions, we’re seeing a growing number of software companies describing themselves as observability companies. But as we know, observability comes from the three pillars - logs, metrics, and traces - and while many software companies are trying to cover all three, it’s a difficult task to achieve this without costing a fortune.
In fact, Cribl’s perspective has been that observability is a journey that’s different for each company, with no one-size-fits-all solution or silver-bullet application that can address everyone’s needs at the same time.
That’s why Cribl takes a different approach: allow the customer to collect, transform/filter, and route observability data to where it’s needed. Customers still use their existing observability tools to analyze the data, but with better control over data flows, lower storage costs, and a hugely simplified approach to managing the observability pipeline.
Without Cribl for observability, enterprise data engineers would configure data pipelines to dump logs into Splunk, which gets very expensive as daily log ingest scales up. But with Cribl, enterprises can collect and filter log data from all sources, send to Splunk only what needs to be analyzed now, and route the remaining logs into low-cost cloud object storage.
The end result is that customers can retain more of their data for longer without ballooning cloud infrastructure costs, enabling log use cases like security breach forensics, root cause analysis, and measuring long-term application performance trends.
Three Benefits of Logging with ChaosSearch + Cribl for Observability
Now that we’ve shared details about Cribl Stream for observability, check out these three benefits of building out your stack with Cribl Stream + ChaosSearch.
1. Create a Consolidated View of Observability Data Sources
Cribl enables customers to route their observability data from any source to any destination. This often means routing metrics to one analysis tool, traces to a second tool, and logs to a third tool. Sometimes it means sending security data to one destination and operational data to another. Dispersing data in this way creates unwanted data silos that reduce data visibility and hide insights.
This is where ChaosSearch comes in: Our data lake platform was built to read and index log data directly in your Amazon S3, with no data movement and no data ingest fees. Our patented Chaos Index® creates a full fidelity representation of your data with up to 20x compression, enabling full searchability with minimal resource costs.
With Cribl Stream + ChaosSearch, organizations can route observability data into cost-effective Amazon S3 cloud storage and index the data for full searchability using ChaosSearch. This results in a detailed and consolidated view of observability data that can support security and cloud operations use cases.
2. Replay Logs with Cribl Stream Replay - But Keep Instant Visibility
The high cost of storing and retaining data in traditional observability tools like Splunk often leads data engineers to set data retention limits. Logs are routed to Splunk, retained for as little as seven days, then discarded, never to be analyzed again. If data engineers ever want to look back at historic logs, nothing is saved.
Cribl Stream helps enterprises side-step this issue with its Replay feature, which enables customers to recall older observability data from low-cost storage and ingest it into observability tools for analysis as needed. For example, when a security incident is detected, Cribl customers can load older security logs from Amazon S3 into Splunk (or another observability tool) to support the investigation.
But this process takes time, and during this time there’s no visibility of the data! As a result, the enterprise security team may be left “flying blind” during what could be an ongoing security breach with consequences mounting by the second.
To achieve fast and thorough root cause analysis of a security incident, SecOps teams need instantaneous visibility and access to the relevant data. By routing observability data into Amazon S3 with Cribl Stream and indexing with ChaosSearch, security teams gain instantaneous visibility across all logs to enhance incident response and other security operations use cases.
3. Keep the Same Data Analytics Tools
Using ChaosSearch + Cribl for Observability means that customers can keep using the same analytics tools they’re familiar with, but without the constraints that exist today.
Traditional observability solutions cost too much because they were never intended for the massive scale of data that enterprises now generate. Cribl Stream helps reduce those costs by enabling its customers to manage the observability pipeline at scale and route observability data into low-cost Amazon S3 cloud object storage instead of high-cost Splunk.
Once the data lands in S3, ChaosSearch indexes the logs and makes them fully visible, accessible, and searchable with no data movement or duplication and no practical limits on data retention.
Get Started with ChaosSearch + Cribl for Observability
When adopted together, Cribl Stream + ChaosSearch (and AWS) provide a long-awaited solution for cost-effective observability at scale.
Amazon S3 delivers the most cost-effective option for long-term data storage in the cloud, Cribl Stream simplifies the process of collecting, transforming, and routing logs into Amazon S3, and ChaosSearch enables you to index, query, and visualize observability data in Amazon S3 with no data movement, no duplicate storage, and without costly data egress fees. Simultaneously utilizing Amazon S3, Cribl Stream and ChaosSearch makes for the optimal observability stack that is not only cost effective and efficient but also provides crucial insights for your organization without data movement.
Ready to learn more?
Start a Free Trial of ChaosSearch
Additional Resources
Read the Blog: The Right Time to Right-Size Your Observability Process
Listen to the Podcast: Trends and Emerging Technologies in Data Analytics
Check out the Whitepaper: DevOps Forensic Files: Using Log Analytics to Increase Efficiency
