If your business is among the more than one million organizations that use Amazon Web Services (AWS) to host applications and data, there is a good chance that you struggle to monitor AWS. After all, although AWS makes it easy to deploy cloud services, collecting and analyzing data about those services in an efficient, centralized way can be a real challenge.
That’s the topic that ChaosSearch CTO Thomas Hazel sat down to discuss in Monitor and Analyze Your AWS Services Directly on S3, the second in our three-part webinar series. Accompanied by Corey Quinn, Chief Cloud Economist at Duckbill Group (and chief snark master at Last Week in AWS), the pair talked about why AWS monitoring can be much harder than it may appear, and how to avoid turning your company’s AWS monitoring stack into a mess of Rube Goldberg proportions (the Pulitzer Prize winning cartoonist was famous for his satirical depictions of the machine age.)
Keep reading for a recap of the main takeaways from their discussion about why AWS monitoring can be such a mess, and how to build an AWS monitoring strategy that actually makes sense.
In a public cloud like AWS, the cloud vendor manages most cloud services for you. You may therefore be tempted to believe that monitoring those services yourself is not as important as it would be when you are dealing with a self-hosted, self-managed environment.
But that would be a big mistake. Even the best-managed cloud services can go awry. Whether they fail because of a problem on the cloud vendor’s end (like temporary downtime) or yours (like improper capacity planning that causes performance degradation), you need to monitor services so you can stay ahead of performance issues.
Not only that, but monitoring AWS is also crucial for cost reasons. As Corey (who has built a business around helping other businesses understand their AWS bills) well knows, AWS pricing schedules are the stuff of mind-boggling complexity. If you don’t carefully monitor your AWS services in order to determine where you are overspending, you risk overpaying significantly for them.
Unfortunately, collecting and analyzing data about AWS services often turns out to be much harder than organizations anticipate.
You may be thinking: “Really? Doesn’t AWS provide tools, like CloudWatch, that make it easy to monitor AWS services?”
The answer is that, while it’s true that AWS offers several native monitoring tools and services to help with metrics collection — including CloudWatch, CloudTrail, and Kinesis Firehose — those solutions fall far short of offering an efficient, centralized way to keep track of what is happening within your AWS environment, for several reasons:
These challenges explain why many businesses end up with an AWS monitoring stack that could have been designed by Rube Goldberg. They depend on a litany of different services, connected together in complex ways, to collect, analyze and alert on their monitoring data.
They get bloated cloud bills to boot, because, as noted above, the more often you move your AWS data, and the more AWS services you depend on to ingest, process and store it, the more you’re going to be paying to AWS.
At first glance, you may think that complex AWS monitoring stacks are bad news only for the engineers who have to manage them. And you may not have a lot of sympathy for those engineers. After all, figuring out how to manage complex technology stacks is their job.
The reality, however, is that an overly complex AWS monitoring tool does more than make engineers’ jobs a bit more difficult. It severely undercuts their ability to optimize the performance of the workloads that their business hosts on AWS. It’s much more difficult to perform effective capacity planning, create the right autoscaling policies, and find and fix performance problems when your monitoring stack is unnecessarily fragmented and complicated.
In turn, customers suffer when applications fail to respond as quickly as they would like, or data becomes unavailable.
And ultimately, the business suffers. Inefficient AWS monitoring means higher costs and lower ROI on cloud services. That’s quite bad in the long term, especially given that most businesses are likely to become only more dependent on the AWS cloud over time. Without the monitoring visibility necessary to optimize your AWS performance and minimize your costs, you are quite unlikely to reap the business benefits of cloud computing.
What if there were a better way to handle AWS monitoring? What if, instead of relying on a tangled web of disparate AWS monitoring tools and services, you could ingest all of your cloud data from all of your services into a central location within AWS, and analyze it right there?
Good news! You can. With a solution like ChaosSearch, you can easily transform S3, the native AWS object storage service, into a data lake and analytics database for AWS monitoring.
Part 1A: Data Lake Challenges: Or, Why Your Data Lake Isn’t Working Out
Part 1B: Data Lake Opportunities: Rethinking Data Analytics Optimization
Part 2: AWS Monitoring Challenges: How to Approach AWS Management