- November 30, 2021
- By Nick Whitt
- 3 min read
Application Logging with Watchtower and CloudWatch Logs
Centralized logging for cloud-based architectures. Watchtower enables log collection and storage in AWS CloudWatch Logs for ephemeral compute resources.
CENTRALIZED LOGGING
Introduction
In short
- Cloud architectures need centralized logging because compute resources are ephemeral, and Watchtower connects Python logging to AWS CloudWatch Logs.
- Watchtower uses the CloudWatchLogHandler to send events directly to CloudWatch without external aggregator services or additional memory overhead.
- Lambda functions automatically log to CloudWatch, but Watchtower adds the ability to organize events by application logic across multiple log groups.
Centralized logging systems are vital in cloud-based architectures due to their inherent ephemeral nature. Whether using auto-scaling instances or serverless computing resources, event streams must persist beyond the temporary lifecycle of the resource. Many third-party solutions exist to solve this problem, including Fuentd, Splunk, and Papertrail; but we’re going to focus on Watchtower. Watchtower is a tool to enable log collection and storage in AWS CloudWatch Logs.
PYTHON LOGGING
Loggers
Python’s native Loggers allow for a variety of Handlers to support dispatching (aka shipping) events to a specific destination. Typically, these destinations are within the compute environment, such as a local file or a memory buffer, but they could be on a remote server using Syslog. With the Logging basicConfig factory, we can easily generate a root logger:
$ python
>>> import logging
>>> logging.basicConfig(format='%(asctime)s %(message)s', handlers=[logging.StreamHandler()])
>>> logging.error('foo')
2020-11-05 11:47:44,390 foo
Many aggregator services, such as Fluent Bit and Logstash, work by “watching” system files and forwarding events to an external document store. While this makes the migration from traditional to cloud-based servers easier, it requires additional memory resources and configuration beyond those of your application.
WATCHTOWER INTEGRATION
CloudWatch Logs
AWS CloudWatch is directly integrated with many AWS services to provide real-time monitors of system metrics. An additional CloudWatch Agent can be installed on EC2 instances to provide log aggregator services as described above. This solves the problem of data persistence, but still requires a lot of external configuration to ensure proper logging streams and filters exist.
Enter Watchtower, “a lightweight adapter between the Python logging system and CloudWatch Logs.” With the CloudWatchLogHandler() Handler, Watchtower will not just forward your events, but directly send them to CloudWatch Logs using the boto3 library you are likely already using.
$ python
>>> import logging, watchtower
>>> logging.basicConfig(format='%(asctime)s %(message)s', handlers=[watchtower.CloudWatchLogHandler()])
>>> logging.error('foo')
2020-11-05 11:47:44,390 foo
$ aws logs get-log-events --log-group-name watchtower --log-stream-name 20201105
{
"nextForwardToken": "f/31961209122447488583055879464742346735121166569214640130",
"events": [
{
"ingestionTime": 1604681353115,
"timestamp": 1604681264390,
"message": "foo"
}
],
"nextBackwardToken": "b/31961209122358285602261756944988674324553373268216709120"
}
To integrate Watchtower with the popular Django framework, you can define a new handler for your Loggers to use. This allows you to continue using Loggers exactly as you have, without the need to manage additional applications and/or configurations.
LOGGING = {
...
"handlers": {
...
"watchtower": {
"level": "INFO",
"class": "watchtower.CloudWatchLogHandler",
"log_group": "AppName",
"stream_name": "StreamName",
},
}
...
}
SERVERLESS LOGGING
Lambda Functions
Lambda functions will automatically store events from stdout or stderr in CloudWatch Logs. This can be invaluable for function-specific details such as billed duration, memory usage, and X-Ray tracing. However, it does not easily integrate itself with Python’s logging.Handler objects. All events are reported to a single log group (by function), with an individual log stream for each function instance. Watchtower will not prevent any of this, but rather adds to this the ability to group events by application or module logic, per Handler definition.
With the following dictionary-based configuration, the root Logger will use both a console and watchtower Handler, ensuring all events are sent to both log groups: the default via stdout, and an application-specific stream defined below.
version: 1
formatters:
simple:
format: "%(asctime)s %(message)s"
handlers:
console:
(): logging.StreamHandler
level: DEBUG
formatter: simple
stream: sys.stdout
watchtower:
(): watchtower.CloudWatchLogHandler
formatter: simple
level: INFO
log_group: "AppName"
stream_name: "StreamName"
loggers:
root:
handlers: [console, watchtower]
KEEP READING
Related Resources
Case Study
AI Listing Creation for a Real Estate Client
An Amazon Bedrock platform and a Chrome extension cut listing creation from up to a full day to under 10 minutes.
Read the real estate case study
Case Study
Modernizing a Bulk Liquids Shipping Platform on AWS
A Python and Django rebuild on AWS added Tableau self-service reporting and vessel tracking, with considerable cost savings.
Read the shipping platform case study
Article
Getting Started with AWS CDK
Start building infrastructure as code with AWS CDK best practices
Read the article
