If your business is among the more than one million organizations that use Amazon Web Services (AWS) to host applications and data, there is a good chance that you struggle to monitor AWS. After all, although AWS makes it easy to deploy cloud services, collecting and analyzing data about those services in an efficient, centralized way can be a real challenge.
That’s the topic that ChaosSearch CTO Thomas Hazel sat down to discuss in Monitor and Analyze Your AWS Services Directly on S3, the second in our three-part webinar series. Accompanied by Corey Quinn, Chief Cloud Economist at Duckbill Group (and chief snark master at Last Week in AWS), the pair talked about why AWS monitoring can be much harder than it may appear, and how to avoid turning your company’s AWS monitoring stack into a mess of Rube Goldberg proportions (the Pulitzer Prize winning cartoonist was famous for his satirical depictions of the machine age.)
Keep reading for a recap of the main takeaways from their discussion about why AWS monitoring can be such a mess, and how to build an AWS monitoring strategy that actually makes sense.
Why Monitor AWS?
In a public cloud like AWS, the cloud vendor manages most cloud services for you. You may therefore be tempted to believe that monitoring those services yourself is not as important as it would be when you are dealing with a self-hosted, self-managed environment.
But that would be a big mistake. Even the best-managed cloud services can go awry. Whether they fail because of a problem on the cloud vendor’s end (like temporary downtime) or yours (like improper capacity planning that causes performance degradation), you need to monitor services so you can stay ahead of performance issues.
Not only that, but monitoring AWS is also crucial for cost reasons. As Corey (who has built a business around helping other businesses understand their AWS bills) well knows, AWS pricing schedules are the stuff of mind-boggling complexity. If you don’t carefully monitor your AWS services in order to determine where you are overspending, you risk overpaying significantly for them.
The AWS Monitoring Headache
Unfortunately, collecting and analyzing data about AWS services often turns out to be much harder than organizations anticipate.
You may be thinking: “Really? Doesn’t AWS provide tools, like CloudWatch, that make it easy to monitor AWS services?”
The answer is that, while it’s true that AWS offers several native monitoring tools and services to help with metrics collection — including CloudWatch, CloudTrail, and Kinesis Firehose — those solutions fall far short of offering an efficient, centralized way to keep track of what is happening within your AWS environment, for several reasons:
- Fragmentation: AWS doesn’t offer a centralized monitoring tool that can collect and analyze all data from all AWS services in a single place. Depending on which services you need to monitor, you will likely half to juggle multiple monitoring tools — like CloudWatch and Kinesis — at the same time to get metrics and logs from your services into a place where you can monitor them.
- Limited analytics: AWS’s native monitoring tools offer little in the way of analytics. Sure, CloudWatch will show you some graphs and dashboards. But if you require deep visibility into your AWS services, you won’t get very far with native AWS tooling alone.
- Data ingestion: Perhaps the biggest limitation of all — and also the easiest one to overlook — is how hard it is to get monitoring data into AWS’s native tools. You often need to pass it through Kinesis, then into Lambda functions, then possibly into an S3 bucket, before you can do anything with it. Not only is all this data movement difficult to manage, but it can also increase your cloud costs, because AWS will charge you for all of these services.
These challenges explain why many businesses end up with an AWS monitoring stack that could have been designed by Rube Goldberg. They depend on a litany of different services, connected together in complex ways, to collect, analyze and alert on their monitoring data.
They get bloated cloud bills to boot, because, as noted above, the more often you move your AWS data, and the more AWS services you depend on to ingest, process and store it, the more you’re going to be paying to AWS.
Who Suffers from Poor AWS Monitoring?
At first glance, you may think that complex AWS monitoring stacks are bad news only for the engineers who have to manage them. And you may not have a lot of sympathy for those engineers. After all, figuring out how to manage complex technology stacks is their job.
The reality, however, is that an overly complex AWS monitoring tool does more than make engineers’ jobs a bit more difficult. It severely undercuts their ability to optimize the performance of the workloads that their business hosts on AWS. It’s much more difficult to perform effective capacity planning, create the right autoscaling policies, and find and fix performance problems when your monitoring stack is unnecessarily fragmented and complicated.
In turn, customers suffer when applications fail to respond as quickly as they would like, or data becomes unavailable.
And ultimately, the business suffers. Inefficient AWS monitoring means higher costs and lower ROI on cloud services. That’s quite bad in the long term, especially given that most businesses are likely to become only more dependent on the AWS cloud over time. Without the monitoring visibility necessary to optimize your AWS performance and minimize your costs, you are quite unlikely to reap the business benefits of cloud computing.
A Better Approach to AWS Monitoring and Analytics
What if there were a better way to handle AWS monitoring? What if, instead of relying on a tangled web of disparate AWS monitoring tools and services, you could ingest all of your cloud data from all of your services into a central location within AWS, and analyze it right there?
Good news! You can. With a solution like ChaosSearch, you can easily transform S3, the native AWS object storage service, into a data lake and analytics database for AWS monitoring.
Read the Series
Part 1A: Data Lake Challenges: Or, Why Your Data Lake Isn’t Working Out
Part 1B: Data Lake Opportunities: Rethinking Data Analytics Optimization
Part 2: AWS Monitoring Challenges: How to Approach AWS Management
