Customer Story

Replacing Elasticsearch With CHAOSSEARCH for Log Analytics

A large SaaS marketing company uses CHAOSSEARCH to analyze 100s of TBs of log data stored on Amazon S3. The log analytics platform is strategic in helping the company deliver a high-quality, uninterrupted experience to the thousands of businesses that rely on the SaaS vendor’s solution.

The Challenge: Terabytes of new Cloudflare logs per day, managed in Elasticsearch

With a core focus on business improvement and customer success, the organization used Cloudflare for web performance acceleration, DDoS protection, data loss prevention, and bot mitigation. While Cloudflare has a powerful range of features, advanced users like this SaaS vendor perform additional analysis on Cloudflare logs to derive value.

The customer uses the Cloudflare Logpush feature to output terabytes of log data daily to Amazon S3. From there, scripts they developed prepare and load the log data into a self-managed, 50+ node Elasticsearch cluster running on AWS. In addition, they use Kibana to query and visualize this log data to help them detect, stop, and prevent DDoS attacks.

Some of the challenges they faced using Elasticsearch for log analytics included:

  • High cost: tens of thousands of dollars per month in Elasticsearch hosting and operations costs that reduce the bottom line and lifetime value (LTV) that is so critical to every SaaS vendor.
  • Low data retention: they could only afford to retain one week of log data.
  • Instability: the wrong type of query or too many concurrent queries often caused their Elasticsearch cluster to fail or become unstable.
  • Unanswerable questions: Elasticsearch queries that aggregated data could be impossible to answer due to prohibitively slow performance.

Due to the importance and continued cost and operational pain of the self-managed Elasticsearch cluster, the SaaS vendor launched a strategic initiative to find a more scalable and economical way to put their Cloudflare log data to work.

The Solution: CHAOSSEARCH — Analyze Cloudflare logs in-place, on Amazon S3

The customer chose CHAOSSEARCH as their Elasticsearch replacement. Unlike Elasticsearch, CHAOSSEARCH analyzes data in-place on Amazon S3. There is no need to reformat log data or load it into an external database for analysis.

Scaling, high availability, and security are all expensive and difficult challenges for people managing an Elasticsearch cluster. But Amazon S3 provides those benefits automatically and economically — orders of magnitude less expensively than Elasticsearch. CHAOSSEARCH enhances S3 with a powerful indexing and search engine, which is compatible with the Elasticsearch API and includes a Kibana front end.

The Results: 10x more data retention at half the cost of Elasticsearch

The CHAOSSEARCH impact was noticeable immediately. During their initial proof of concept, CHAOSSEARCH impressed the customer with:

  • Immediate time to value: CHAOSSEARCH is a fully managed cloud service that analyzed their Cloudflare logs in place on S3. There was no database to set up, data reformatting, or duplication required.
  • Greater data retention: CHAOSSEARCH was able to search 10x more log data — over two months worth — faster than their Elasticsearch cluster could search one week of data.
  • New insights: CHAOSSEARCH handled their aggregate query requirements, which helped them identify and block DDoS attack IP addresses faster and completely.
  • Cost savings: Compared to the hosting, licensing, and administrative costs of operating a 90-node Elasticsearch cluster, CHAOSSEARCH cost the customer less than half as much to run.

Get started with CHAOSSEARCH “No-Migration On-boarding” Experience

If your log data is in Amazon S3 buckets, you can activate CHAOSSEARCH against it as-is. There is no new external database to install. No data to migrate. No ETL to set up. Just index and search.

To begin your free trial of CHAOSSEARCH, visit chaossearch.io today.

CHAOSSEARCH was able to search 10x more log data — over two months worth — faster than their Elasticsearch cluster could search one week of data.