Only about four months to go before nearly 60,000 people descend on Las Vegas for the eighth annual AWS re:Invent user conference. However, that doesn’t mean that Amazon is going to embargo new service announcements until then. The recent AWS Summit in NYC looked more like a mini-re:Invent with nearly 12,000 people in attendance. Werner Vogels kicked things off with the keynote talking about the growth and scale of the AWS cloud services and continued by announcing a bevy of new software and services. One of the key points that Werner shared during his keynote is how people are using Amazon S3 as the home base for all their data. It was a message shared also by Chris Dyl from Epic who showed how all of their data goes into Amazon S3 where other services can interact with it. It’s a message we share constantly at CHAOSSEARCH since we help so many companies with search and analytics on S3 while avoiding having to actually move the data into another system. With so many new services and features announced, some big and some small, I want to share my personal top 3 AWS announcements from the summit.
The most significant announcement of the event, in my opinion, was AWS EventBridge. The basic premise of what AWS EventBridge can provide comes from the event-driven alerting and notification features of CloudWatch Events and takes it to the next level. EventBridge allows for tighter integration between various SaaS services you are currently consuming today such as PagerDuty or Zendesk. Now you can have events from those services get published to an event bus that various Amazon services can subscribe to and take action. Imagine a world where you can have security events from your SaaS monitoring system trigger events that allow multiple AWS services to consume and route to take various actions. I would check out Jared Short’s fantastic blog post to learn more about why this is such an impressive new service.
The next feature I was very excited to see announced was the AWS Toolkit for Visual Studio Code. While I’ve been a steadfast VIM user for nearly two decades, it can be challenging to get it to become as robust of an IDE as a tool like Visual Studio Code. Over the past couple years I’ve used VS Code whenever I’ve needed to dive into some coding work, and this new plugin for VS Code would have been a HUGE help a few weeks ago when I was building a Lambda function to process some CloudFront logs into a useful format. It makes building and debugging Lambda functions almost comically easy, and it integrates directly into the IDE. It took me only minutes to get it set up in my environment, and I love how it used the same authentication model as my AWS command-line tools do.
The third and final announcement that I was most excited to see was Amazon’s support for the Fluentbit log forwarder. One of the common issues we hear from customers who are adopting services like Docker or Kubernetes is that when the container or pods go away, so do the logs. A frequently used solution to this problem is deploying Fluentd in your environment and routing logs into Elasticsearch. Many of these customers soon learn, as your log data volume and retention limits grow on Elasticsearch it becomes cost-prohibitive. Our lead SRE Patrick Flaherty did a whole series on pushing logs to S3, and shared the excitement of quickly getting logs to Amazon S3 for query and analysis. That’s why I was extremely excited to see them show off the Fluentbit plugin for AWS container images which can route your logs directly to Amazon CloudWatch and Amazon Kinesis Data Firehose as both services can shove data directly into your Amazon S3 buckets. With a 5 minute setup with the CHAOSSEARCH service, you can then index all the log data in your Amazon S3 account and start running search and analytics on all this data without ever having to provision an Elasticsearch cluster. With CHAOSSEARCH service, all the data and indexes live within your Amazon S3 account which provides you the ability to no longer worry about data retention limits since the cost of data storage on Amazon S3 is extremely low.
The AWS NYC Summit was a fantastic event. We loved attending to have the opportunity to chat with many different companies at various stages of their growth on AWS. If you would like to learn more about how we can enable search and analytics on your Amazon S3 data, or even start a free trial to see it in action with your own data, reach out to us today.
