As public cloud and multi-cloud adoption skyrockets, many organizations are looking to implement compatible services. These services increase the utility of cloud infrastructure by tapping into the underlying building blocks (otherwise known as primitives) of the cloud. That’s where the idea of a “supercloud” comes into play. SiliconAngle defines a supercloud as “an architecture that taps the underlying services and primitives of hyperscale clouds to deliver additional value above and beyond what’s available from public cloud providers.”
These supercloud software and services plug into existing environments like Amazon Web Services or Google Cloud Platform to help organizations add features such as analytic capabilities, single sign on, security and more. Let’s learn more about the idea of a supercloud and identify where a supercloud could benefit your organization the most.
Many organizations require custom capabilities to build upon and scale the functionality of existing public cloud services. As the SiliconAngle article above mentions, at the 2021 AWS Re:Invent event, companies like Goldman Sachs, Nasdaq, Dish Network and United Airlines spoke about building superclouds. These are custom clouds built for use cases to manage data, IT automation, wireless networks, travel and more.
Typically, these organizations are not perceived as tech companies, yet are building and marketing their custom solutions on top of AWS. These solutions can be leveraged by similar companies that operate within their vertical markets, and are often made available as free, open source tools.
Superclouds can be built upon a single public cloud platform, or within multi-cloud environments. The idea is that a supercloud’s components add value to the public cloud platform, or make multiple clouds function better than any individual cloud platform could alone. Supercloud vendors include companies that easily integrate with the top cloud platforms, such as ChaosSearch, Okta, Datadog, Hashicorp, MongoDB and more.
Learn More: Elasticsearch Replacement for Log Analytics at Scale
A supercloud is an abstraction layer that can hide the underlying complexity of various clouds, but at the same time is capable of leveraging these capabilities to add value on top of native cloud primitives.
Depending on the use case of the supercloud service, an organization can experience benefits such as:
Let’s look at leveraging a supercloud approach for analytics as one example of how this concept plays out.
Many organizations want to maximize the value of their existing data lake resources. For example, most teams already use low-cost cloud object storage like AWS S3 for storing cloud-native application production data or enterprise application data. However, existing capabilities within public cloud providers do not allow for easy access to analytical insights within these object storage data lakes.
Read: Optimize Your AWS Data Lake with Data Enrichment and Smart Pipelines
Typically, organizations get around these obstacles with teams of data scientists or data engineers. These teams build data analytics pipelines using a process called extract, transform and load (ETL). ETL pipeline creation can slow the time to insights, and can serve as a bottleneck for analysts looking to interrogate their data in multiple ways.
Supercloud services like ChaosSearch allow organizations to unlock the value from their existing AWS and GCP cloud object storage resources. ChaosSearch works by turning object storage into a hot, analytical data lake or self-service data analytics engine. That way, users can directly query data in place, without having to rely on a data engineer to move the data or make copies for analysis.
As a result, analysts and business users can get insights quickly, and ask different questions about their data based on the insights they’d like to discover. Considering that data science talent is in short supply, technology like ChaosSearch frees up these specialists to focus on more strategic tasks.
Read: Talent Shortage 2022: Stretching Your Lean DevSecOps Team
As the ecosystem around public cloud platforms evolves, other terminology such as data mesh and data lakehouse have emerged. Some of these terms have overlapping definitions with supercloud services for analytics, and with one another.
In short, data mesh and data lakehouse solutions and approaches are intended to prevent organizations from building data pipelines and slowing down analytical processes. In reality, these approaches can be combined to help organizations extract value from their existing cloud resources, as described above with the AWS S3 example.
Regardless of the terminology you use, look for a solution that easily integrates into your existing public cloud infrastructure. The goal is to deliver value above and beyond what’s currently available within an existing cloud platform – while making it easier for teams to get what they need out of these powerful resources.
Read the Blog: AWS vs GCP: Top Cloud Services Logs to Watch and Why
Listen to the Podcast: Differentiate or Drown: Managing Modern-Day Data
Check out the eBook: Beyond Observability: The Hidden Value of Log Analytics