Monitor Like a DevOps Pro: Build A Log Aggregation System in AWS
Modern cloud environments are increasingly complex distributed systems with numerous software components. The challenge of maintaining moving parts and tracking changes in your AWS systems continues to grow but there are solutions. Some natural questions are:
- How can you understand, at a high level, what is happening in your cloud?
- Can you track usage trends over time?
- Can you debug any issues that might arise?
- Can you search through logs without combing through files on many disks?
The answer to each is yes you can! A sophisticated tool called a log aggregation system gathers operational information and logs from across your entire cloud. The log aggregation system is an advanced DevOps technique that enables you to quickly search your logs and graph any trends arising from structured logs.
In this Lab, you will create a distributed, scalable log aggregation system within AWS running on Amazon OpenSearch Service. This Log Aggregation System will ingest as many of your CloudWatch log stream events as you want, events generated from AWS EC2 Instances, Lambda functions, Databases, and anything else you want to submit log events from.
Please note, this lab involves creating an Amazon OpenSearch Service domain, which can take up to twenty minutes to finish setting up. Please make sure you have enough time available before starting this lab.
Learning Objectives
Upon completion of this Lab, you will be able to:
- Compare and contrast the log analysis capabilities in CloudWatch Logs and Elastic Stack, particularly OpenSearch and Kibana
- Subscribe the Amazon OpenSearch Service to CloudWatch and automatically stream log events to OpenSearch
- Search and discover log events using Kibana
- Create Kibana visualizations and dashboards to monitor the state of your cloud
Intended Audience
- Candidates for the AWS Certified DevOps Engineer Professional certification
- DevOps Engineers
- Cloud Architects
Prerequisites
Familarity with the following will be beneficial but is not required:
- Amazon OpenSearch Service (OSS)
- Amazon CloudWatch
- AWS Lambda
The following content can be used to fulfill the prerequisites:
- How to Implement & Enable Logging Across AWS Services (Part 1 of 2)
- How to Implement & Enable Logging Across AWS Services (Part 2 of 2)
- Understanding AWS Lambda to Run & Scale Your Code
Updates
February 5th, 2024 - Updated the instructions and screenshots to reflect the latest UI
May 27th, 2023 - Updated Node.js version
May 4th, 2023 - Updated the instructions and screenshots to reflect the latest UI
March 1st, 2023 - Updated AWS Lambda screenshots to match the latest user interface changes
February 10th, 2023 - Minor changes to improve the lab
September 28th, 2022 - Resolved an issue regarding OpenSearch Domain creation
May 10th, 2022 - Updated OpenSearch Domain configuration to reduce provisioning time
April 29th, 2022 - Updated the instructions and screenshots to reflect the latest UI
April 14th, 2022 - Added a check for the OpenSearch Domain lab step
December 20th, 2021 - Updated screenshots to reflect the latest Kibana interface experience
December 14th, 2021 - Updated screenshots to reflect the latest interface experience
December 1st, 2021 - Update Amazon OpenSearch Service domain creation instructions to reflect latest user interface changes
September 10th, 2021 - Updated lab to reflect the renaming of ElasticSearch to OpenSearch by Amazon
April 28th, 2021 - Updated the Amazon Elastic Search domain version to the latest available
March 10th, 2021 - Updated AWS Lambda screenshots and instructions to reflect the latest user-interface changes
November 11th, 2020 - Fixed an issue preventing the creation of the ElasticSearch domain
October 1st, 2020 - Updated some instructions to make them clearer, updated some screenshots to match new UI
May 27th, 2020 - Updated a lab step's instructions to account for long Elasticsearch deployment times
October 4th, 2019 - Because of CloudFormation changes, we have updated instructions and screenshots.
May 9th, 2019 - Removed unnecessary permissions from the Lambda DynamoDB role and improved the explanation of how Lambda functions map to CloudWatch log streams
January 11th, 2019 - Fixed an issue that caused the streaming of CloudWatch Logs to ElasticSearch to fail
January 10th, 2019 - Added a validation Lab Step to check the work you perform in the Lab
June 5, 2018 - Complete update (easier to follow instructions and screenshots, update to Elastic Stack version 6)
Environment before
Environment after
Nothing gets me more excited than the AWS Cloud platform! Teaching cloud skills has become a passion of mine. I have been a software and AWS cloud consultant for several years. I hold all 5 possible AWS Certifications: Developer Associate, SysOps Administrator Associate, Solutions Architect Associate, Solutions Architect Professional, and DevOps Engineer Professional. I live in Austin, Texas, USA, and work as development lead at my consulting firm, Tuple Labs.