Customer Story

Overcoming the Growing Challenges Around Data Management

Klaviyo: Leading data and marketing platform

Headquartered in Boston, Klaviyo, Inc. is a fast-growing SaaS eCommerce marketing company that obsesses over helping brands grow faster. Quality of service and uptime are critically important to Klaviyo and their customers, who depend on the Klaviyo service to access, store, analyze, and use transactional and behavioral data to power highly-targeted email and advertising campaigns.

To ensure service quality and continuously improve the user experience, Klaviyo collects and analyzes enormous volumes of log data. With customers more than doubling year over year, Klaviyo turned to CHAOSSEARCH to overcome the continuously growing challenges around data management.

The Challenge: Cost of analyzing 75 TB of log data in Athena

The Klaviyo system generates Amazon Elastic Load Balancing (ELB) log data, which they had been storing on Amazon S3 and analyzing with Amazon Athena. By the beginning of 2019, they were analyzing over 75 TB of log data in Amazon S3, which was becoming prohibitively expensive with Athena. The high query costs were causing them to:

  • Reduce the amount of data they retained
  • Spend more engineering time tuning storage and queries to reduce Athena costs
  • Limit the number and type of queries they ran

As a result, they launched a strategic initiative to select and implement a log management alternative to Athena, which would make it affordable to retain and analyze 100s of TBs of log data.

The Solution: CHAOSSEARCH — Fastest time to value with search-in-place analytics on Amazon S3

After a three-month evaluation process, Klaviyo selected CHAOSSEARCH as their new log analytics platform for the following reasons:

  • Immediate time to value: CHAOSSEARCH, a fully managed cloud service, analyzed Klaviyo’s ELB log data in place on Amazon S3. There was no database to set up, data reformatting, or duplication required.
  • Powerful search and analytics: CHAOSSEARCH proved to be a true “search platform” for all of Klaviyo’s current and future data in Amazon S3.
  • Cost savings: CHAOSSEARCH was the most cost-effective solution that Klaviyo evaluated.

Klaviyo’s use cases included search and analysis of a subset of their Amazon ELB logs and their application logs. After seeing the power of the CHAOSSEARCH platform, Klaviyo determined that they could completely replace their Athena implementation with CHAOSSEARCH.

The Results: From 75 TB to 100s of TB of log data, at a fraction of the cost

After a few months of use, Klaviyo made CHAOSSEARCH its search platform coupled with Amazon S3 for operational and unstructured data. This made data operations significantly easier, by establishing Amazon S3 as the central data repository, with CHAOSSEARCH as the standard analytics platform analyzing Amazon S3 data in-place. With this strategy, Klaviyo’s Amazon S3 environment grew rapidly as developers realized the value, and began to use both Amazon S3 and CHAOSSEARCH for new use cases and more frequency.

Get started with the CHAOSSEARCH “No-Migration On-boarding” Experience

If your log data is in Amazon S3 buckets, you can activate CHAOSSEARCH against it as-is. There is no new external database to install. No data to migrate. No ETL to set up. Just index and search.

To begin your free trial of CHAOSSEARCH, visit chaossearch.io today.

CHAOSSEARCH provided the managed search capabilities we needed while allowing a longer data retention period. Using the auto schema detection and data discovery features, we were able to more easily drill into our data and query it to extract insight.

Andrew Kenney, VP Engineering