3 Effective Tips for Cloud-Native Compliance

The ephemeral nature of the cloud has made compliance and security a greater challenge for organizations. The volume of data that companies must collect and retain from their cloud services, depending on their industry, is ballooning fast. According to ESG, 71% of companies believe their observability data (logs, metrics and traces) is growing at a concerning rate. Even so, outcomes are getting worse, not better. Six out of 10 teams are unable to prevent issues before customers are impacted.

All of this data comes at a cost. Organizations need to retain log data for security and compliance purposes. This data provides an audit trail for security operations (SecOps), as well as customer data protection. However, most organizations don’t retain the volume of log data they need to investigate security risks like advanced persistent threats.

In this article we’ll cover some of the challenges of cloud-native compliance requirements, and how to solve common security and compliance retention issues for log data.

Challenges of Compliance in a Cloud-Native World

Overall, cloud-native infrastructure is largely ephemeral, meaning that resources are temporarily created and destroyed on demand. That’s a big challenge for compliance, considering that applications often use complex, distributed architectures. For example, with microservices-based applications, each distinct service must be observed on its own, yet combined with other microservices to understand the overall application state.

Another challenge is that cloud providers like AWS or Google Cloud Platform have a shared responsibility model for security and compliance. It’s sometimes hard to know where the provider’s responsibility ends and the customer’s begins. For example, AWS says that it is responsible for “security of the cloud,” while customers are responsible for “security in the cloud.” As a part of their compliance responsibilities, AWS customers must manage their own security and compliance controls, including the protection of their customers’ data. Demonstrating compliance starts with a strong data retention policy, which we’ll cover more later.

A diagram of cloud security and compliance responsibilities in the AWS Shared Responsibility Model

One other challenge is that log data is important for security and compliance, but most companies don’t keep it for that long. DevSecOps techniques like audit logging, fault tolerance and threat hunting depend on organizations using log data to discover potential security incidents or issues in their applications and infrastructure. However, modern cloud-native applications and infrastructure produce a lot of logs!

Due to the ballooning costs of increased log data, many companies compromise on the volume of easily-accessible data they store in their analytics platforms and reduce their data retention windows to 30 days or less, depending on the perceived importance of that data. Typical observability platforms charge a premium for data ingestion, retention and data movement (case in point, the rehydration process in Datadog).

Limiting your data retention can create unintended issues with security and compliance, including root cause and forensic analysis, threat hunting, and the ability to report on what’s happened in your environment for compliance purposes.

Security Monitoring for Cloud-Native Applications and Infrastructure

Fortunately, modern observability techniques can alleviate some of these security and compliance challenges. For example, a security data lake approach provides continuous monitoring at lower cost.

Using a security data lake, organizations can centralize all of their security telemetry, making it easier to correlate events across complex, distributed systems. Centralizing your logs in cloud object storage like Amazon S3 or GCP provides a single source of truth and holistic view into your applications and infrastructure. As a result, your security team can rely on this data for better threat detection and faster incident response.

The fragmentation of the SecOps stack means companies typically have multiple silos of data that make it hard to create a single view across all systems. As mentioned above, with the emergence of microservices architectures, monitoring your attack surface is increasingly more complex and expensive. Centralizing all your logs delivers full visibility across your entire infrastructure and tools to tackle internal and external threats.

What’s more, the low cost of cloud object storage can offset some of the exorbitant retention charges incurred from observability tools. Many teams choose to use an XDR, SOAR or SIEM for critical day-to-day security controls. Then, they centralize log data on cloud object storage and apply a cost-effective log analytics solution to that data. This approach reduces storage and maintenance costs, without limits on ingest or retention.

Read: Why Midsized SecOps Teams Should Consider Security Log Analytics Instead of Security and Information Event Management (SIEM)

3 Tips to Meet Compliance within Cloud-Native Environments

1. Check Security Components (Required vs. Optional)

Ensuring compliance in a cloud-native world means fully understanding regulations. Required compliance mandates depend on industry or region but they may include:

• HIPAA for healthcare
• PCI for organizations that accept credit card payments, like retailers
• GDPR for organizations that do business in the European Union
• State-by-state data privacy regulations in the U.S., such as CCPA

Added security and compliance controls might include:

• ISO 27000 series for information security, privacy, and cybersecurity
• SOC 2 for protection of customer data
• And more.

For many companies, compliance is a business imperative, particularly SaaS companies who sell to the enterprise.

2. Collect App and Infrastructure Data Continuously

Continuously monitor your cloud environment establishing a best-of-breed, cloud-native observability approach to monitor logs, metrics and traces. Some telemetry data – like metrics and traces – can be retained for shorter periods of time, and therefore be used as an effective real-time alert mechanism within tools such as Splunk, DataDog, New Relic or Dynatrace.

You can offset the cost of log retention by using the security data lake approach described in the section above.

3. Institute a Solid Data Retention Policy for Your Organization

An effective data retention policy ensures that data is:

• Available and accessible when needed for internal business processes, such as analytics, reporting, or financial auditing.
• Stored in the most cost-effective format based on its intended purpose and how frequently it will be accessed.
• Retained for the appropriate time frame when required by law.
• Deleted or destroyed when it is no longer needed by the organization, or as required by law.

A part of this process involves creating a compliance needs assessment.

Example Template for event log data

Finally, adopt tools that provide cost-effective data retention and remove unnecessary retention limits. Technologies like ChaosSearch can support your cloud data retention strategy. ChaosSearch uses a proprietary indexing technology to greatly reduce the amount of storage required in S3 for a full, searchable representation of your data, and eliminates the need for any additional data movement for analysis.

ChaosSearch also has its own data retention features that can be used to augment your existing S3 retention policies and automation. The ability to index, search, and analyze log data at scale means you can retain log data for longer periods, while fully realizing its value through applications like security monitoring and cloud log analysis.

Conclusion

To wrap up, the complexity of cloud-native environments adds up to lots of telemetry data.  For many teams, that leads to ballooning costs. At the same time, continuous monitoring is essential for both security and compliance purposes. The ability to centrally analyze log data is critical, with data retention policies in place supporting security and compliance goals.

Fortunately, with a best-of-breed approach, teams can lower the cost of observability, while retaining the data they need to meet their cloud-native security requirements.

Want to learn more about how to use log data to detect security threats?