Amazon Security Lake

ChaosSearch revolutionizes the AWS Security Lake ecosystem with real-time data indexing and analytics, transforming the way security data is processed and analyzed.

• Real-Time Data Indexing and Analytics: ChaosSearch ingests and indexes data in real-time, adapting to changing data schemas and enhancing threat detection and response.
• Integration with Analytics Tools via Open APIs: ChaosSearch's Open APIs allow for seamless integration with familiar analytics tools like OpenSearch Dashboard and Superset, simplifying data access and analysis across different teams.
• AI-Powered Data Exploration: ChaosSearch's AI Assistant (early access) enables intuitive data exploration, interpreting natural language queries to provide relevant insights, and democratizing data access. To see our AI Assistant in action and request Early Access, simply submit a request here!
• Proactive Threat Hunting: Offering broad visibility and deep analytics, ChaosSearch enables swift identification and mitigation of suspicious activities.
• Swift Incident Response: With immediate access to indexed security data, ChaosSearch accelerates incident response, shortening the time from detection to resolution.
• Enhanced Collaboration and Cost Efficiency: Leveraging existing cloud storage and open APIs, ChaosSearch's AI Assistant facilitates efficient teamwork and provides cost-effective analytics.

ChaosSearch's capabilities empower organizations to utilize their security data more efficiently and effectively, significantly bolstering their security posture within the AWS Security Lake ecosystem.

More about this integration

Amazon Security Lake is a powerful data gathering platform that centralizes security data from AWS and third-party sources into a data lake stored in your AWS S3 account. Security Lake stores the data in Apache Parquet format and the standard open-source schema called the Open Cybersecurity Schema Framework (OCSF).

The ChaosSearch Data integration for Amazon Security Lake cost-effectively ingests and activates that important security object store for investigative analytics. ChaosSearch offers multi-model data access to users via open APIs such as Elasticsearch and SQL, or via the Kibana and Superset UIs included natively. Monitor, alert, and threat hunt using all your Security Lake data—without retention limits—to face today’s complex security environments and persistent threats. Combine the valuable Security Lake data with your application or infrastructure logs to improve your organization-wide observability and security posture.

With Amazon Security Lake, important AWS applications and key third-party source partners have united to home their security information into one reliable, secure, and accessible location, with common formats and an OCSF schema. Security Lake is a new way to confront the persistent threats to global business. ChaosSearch brings a new approach to increase your ability to monitor and analyze that security content at scale, while reducing operational costs of text search and relational analytics.

Setup Steps (existing customers)

If you already have ChaosSearch set up and indexing your log and event data, you can add Amazon Security Lake S3 buckets for indexing and analysis. (If you are a new ChaosSearch customer, go to the next section.)

1. Configure Amazon Security Lake for your AWS account using the Getting Started steps in the AWS user guide. This will create a new IAM role with two policies.

2. Copy those policies and attach them to the existing ChaosSearch role ARN following the ChaosSearch help topic Attach the Policy to the ChaosSearch IAM Role. NOTE: If your existing ChaosSearch read-only policy has permissions to read and get all buckets on the account that Security Lake has created, and you want ChaosSearch to be able to read/index all Security Lake S3 data, you do not need to make any changes to existing permissions. This change allows ChaosSearch to see the new Security Lake bucket(s).

3. On the ChaosSearch Storage tab, create a live object group to index the Parquet files in the security lake S3 bucket. ChaosSearch has built-in support for the Parquet format.

   
4. To enable live indexing, specify the security lake SQS queue ARN that Security Lake creates for its S3 bucket, and create the group.
   
   

5. Create a refinery view to enable searching and visualization for the indexed data created for the object group.

6. Use ChaosSearch Search Analytics, SQL Analytics, or APIs to search your Security Lake data, to configure alerts on conditions, and to create dashboards to review trends and security analytics.

Setup Steps (new customers)

If you are new to ChaosSearch and want to start using ChaosSearch to index Security Lake S3 files, follow these steps.

1. Contact ChaosSearch to bring up a dedicated environment for you. Your ChaosSearch TAM will provide you with a Customer-UID.
2. Configure Amazon Security Lake for your AWS account using the Getting Started steps in the AWS user guide. This will create a new IAM role with two policies.
3. Log in to the AWS Console using your customer account, go to IAM-Roles page, and add the following policy for ChaosSearch using the process outlined in the ChaosSearch help topic Attach the Policy to the ChaosSearch IAM Role:
   
   

4. In AWS S3, in the same account as the role, create a new bucket aws-security-data-lake-cs-<CUSTOMER-UID> (match the name used in the policy above). This is the read-write bucket for the ChaosSearch configuration and indexed data.

5. Connect your AWS resources to the ChaosSearch environment following the ChaosSearch help topic Add Role ARN to ChaosSearch AWS Credentials.

6. Import the Security Lake bucket(s) into ChaosSearch using a command similar to the following. NOTE: The bucket endpoints currently require an AWS V4 signature for authentication.

   
7. Complete with steps 3 to 6 of the "Setup Steps (Existing Customers)" section earlier to create object groups and views, and to run searches and visualizations on your Security Lake data.

Sample Analytics

The following sample images show an example of the ChaosSearch Discover results for VPC Flow Logs, which you can use to hunt for potential threats on network traffic, and a VPC Flow Dashboard, which you can use to monitor network traffic, with the underlying VPC Flow Log data coming from Amazon Security Lake.

Troubleshooting Steps

Common setup issues typically include errors or changes in the AWS role/policy definition and ARN information, or configuration of the object group(s) that ingest and index the data from the Security Lake S3 bucket (such as filtering/prefix changes or unexpected format errors). See the ChaosSearch documentation for the required IAM policies and configuration best practices. ChaosSearch Customer Success works with you to identify and correct any setup issues, and to refine RBAC policies for access to ChaosSearch object groups and views. Reach out to your ChaosSearch Customer Success representative via the dedicated Slack channel created for each customer or via email.

Support Process

ChaosSearch Customer Success supports customers via dedicated Slack channels and email. Customer Success works with you to resolve configuration and querying issues, and can also help to tune settings to maximize economy and performance based on ingest volume and querying activity.

Maintenance Process

ChaosSearch offers a regular cadence of software updates for enhancements and fixes. Customer Success works with you to schedule upgrades, which can include enhancements to adapt for changes such as OCSF schema updates as well as evolutions to the Security Lake implementation.