As high-scale Financial Technology (FinTech) companies build on their data-driven capabilities and establish product-market fit, they often experience rapid growth that creates challenges for IT and DevOps teams.
These are challenges that we’ve touched on in the past - they include issues such as explosive data growth, complicated analytics pipelines that require ongoing maintenance, siloed data that stifles inquiry and collaboration, and difficulty maintaining oversight and cybersecurity in an increasingly complex cloud environment.
ChaosSearch is an ideal solution for growing FinTech companies seeking to modernize their log analytics and enhance capabilities that range from threat hunting and security operations to business intelligence, observability, CloudOps management, application troubleshooting, and optimizing DevOps efficiency.
In this week’s blog, we’re excited to share our newest FinTech success story with global investment firm Sixth Street. You’ll discover the log analytics challenges faced by DevOps teams at Sixth Street and how ChaosSearch is helping the firm save valuable time and money with a scalable and efficient approach to analyzing log data.
Who is Sixth Street?
Sixth Street is a global investment firm with 400+ team members and more than $60 billion in assets under management.
The firm operates as a unified team of investment and control-side professionals, building businesses, investing for growth, acquiring assets, providing direct lending/financing, identifying value in public markets, and purchasing royalty streams on behalf of institutional investors and their beneficiaries across the globe.
Since its inception, Sixth Street has strategically invested in some of the world’s leading companies and most recognized brands, including Airbnb, Spotify, and European soccer teams FC Barcelona and Real Madrid. The firm also invests in other sectors, including real estate, energy and renewables, public markets, and agriculture.
Why Sixth Street Needed a Scalable, Centralized Logging Solution
As an investment firm with a global footprint, Sixth Street relies on a variety of in-house developed and off-the-shelf applications to connect with customers and support its daily operations. This includes nine software investment platforms operated by Sixth Street, all of which allow stakeholders to purchase and manage investments across multiple sectors, locations, and asset classes.
The challenge for Sixth Street was to develop and implement a cohesive and cloud-centric approach to application logging, monitoring, and lifecycle management while minimizing the operational burden on infrastructure and DevOps teams.
Sixth Street developed a strategy for consolidating application logs from business units across three regions in a central log archive account for consumption. Then needed a reliable process to query, index, and analyze log data.
ChaosSearch developed a bespoke solution for Sixth Street, combining the ChaosSearch cloud data platform with a customized logging architecture designed to streamline the ingesting and analyzing log data process. Here’s how it works:
- Sixth Street uses the Amazon Cloudwatch observability platform to collect logs from throughout the cloud environment, including logs from SQL servers, Windows Events, AWS Systems Manager Agent (SSM), and VPC Flow logs.
- Next, Sixth Street uses Amazon Kinesis Firehose to stream log data into cloud object storage (Amazon S3). To enhance the quality of data entering S3 buckets, ChaosSearch engineers wrote customized AWS Lambda functions to enrich log data with valuable contextual information before it would be ingested into object storage.
- As the enriched log files land in Amazon S3 buckets, event notifications are created and sent to an AWS SQS queue. ChaosSearch monitors SQS and automatically indexes the referenced objects in near real-time. As a result, newly ingested logs are available to be queried just 1-2 minutes after being written to S3.
- The indexed data, compressed by up to 95% to maximize storage efficiency, is stored in a separate S3 bucket owned and controlled by Sixth Street.
With help from ChaosSearch, Sixth Street can now enrich logs with contextual data before they land in Amazon S3, cost-effectively index and store log data, consume log data by creating customized views in the ChaosSearch data refinery, and visualize log data or create dashboards with embedded OpenDistro Kibana.
Sixth Street is also using ChaosSearch to monitor and alert on aggregated log data, enhancing the overall observability of cloud applications and accelerating their response to application errors or cybersecurity issues that trigger an alert.
Read: Building a Cost-Effective Full Observability Solution Around Open APIs and CNCF Projects
Measuring the Impact of ChaosSearch for Log Analytics at Sixth Street
Moving away from traditional logging solutions like ELK stack and adopting ChaosSearch has allowed Sixth Street to implement a cohesive and cost-effective log analytics solution while minimizing the management burden on IT and DevOps teams.
ChaosSearch combines data lake philosophy (e.g. easy data ingest, schema on read, loosely coupled storage and compute) with built-in analytics functionality, allowing Sixth Street to index, query, and analyze log data, at scale and in near real-time, with no excess data movement, no duplication, and no limits on data retention.
After adopting ChaosSearch for log analytics, Sixth Street reported saving five hours of labor for each engineer on its DevOps team in addition to other labor costs. Time previously spent on maintaining log analytics infrastructure can now be allocated to revenue-generating projects.
Plus, thanks to Chaos Index® data compression and cost-efficient storage, Sixth Street has unlocked the full potential of its log data, including new use cases like long-term trend analysis, advanced persistent threat (APT) detection, and root cause analysis.
Streamline Your Log Analysis for Best-of-Breed Observability with ChaosSearch
Adopting ChaosSearch has allowed Sixth Street to streamline and simplify its log analytics program, efficiently monitor and alert on aggregate log data, and unlock new use cases for log data - all while saving on logging costs and reducing management overhead.
Sixth Street is our most recent success story - but it’s not our first. We helped a European FinTech company extend log data retention and reduce logging costs by $3.5 million annually. We’ve also helped Equifax modernize its log analytics program with a single pane of access for up to 50 TB of daily ingested log data.
With these wins under our belt, we’re excited to continue showing organizations what’s possible with ChaosSearch. Will your business be next?
Click the link below to launch our free trial experience and see for yourself how ChaosSearch can upgrade and future-proof your log analytics program.
Additional Resources
Read the Blog: 10 Essential Cloud DevOps Tools for AWS
Watch the Webinar: Advanced Analytics - Data Architecture Best Practices for Advanced Analytics
Check out the Whitepaper: Save Your Sanity: Achieving the Security Data Lake
