Containers and Real-Time Resource Monitoring

Are you monitoring your containers’ resources in real time? If not,
then you’re probably not monitoring as effectively as possible. In a
fast-moving, dynamic microservices environment, monitoring data that is
even seconds old may no longer be actionable. To prevent disruptions,
you need real-time monitoring. In this post, I explain why real-time
monitoring of container resources is important, and which types of
container metrics you should focus on monitoring in real time. And just
to be clear up front, this isn’t a post endorsing any particular
monitoring vendor’s toolset. While there are now plenty of
container-ready monitoring platforms out there that can support
real-time monitoring, I think it’s better to understand the underlying
essentials of container monitoring, rather than focusing on the feature
set of a particular product. If you know what to monitor in real time in
order to keep your containerized infrastructure running healthily,
you’ll be well-positioned to choose the best toolset to meet your
real-time monitoring needs.

The Challenges of Real-Time Container Monitoring

Before discussing how to perform real-time monitoring for containers, it’s worth pointing out the special challenges that arise from monitoring containers in real time.

The most obvious is that, in a containerized environment, components disappear
all the time by design. In a legacy environment, you focused on
monitoring servers and apps that were relatively static. But containers
spin up and down constantly. As a result, there is a lot more monitoring
in a containerized environment. By extension, there is a lot more noise.
Separating meaningful data from noise is therefore more difficult,
especially when you need instant monitoring insights and can’t waste
time identifying noise. Real-time monitoring can also be harder in a
containerized environment because of the way Docker abstract containers
away from the host. When you’re dealing with containers, you can’t
simply run monitoring commands like top or ps from the host and get an
accurate picture of what’s happening inside the containers. Since
logging into containers to peer inside in real time for monitoring
purposes is not feasible at scale, the answer to this challenge is to
use agents or another clever type of monitoring solution that provides
real-time visibility into containers and the services they support.

What Can You Monitor?

Let’s now take a look at which real-time container metrics you can
monitor. Taking Docker as the most obvious example (although much of the
following applies to other container systems, including the Linux-native
LXD), we can break real-time container metrics into four basic
categories:

Memory

Docker can monitor the total memory used by an individual container,
along with the amount of cache and swap memory, and the resident set
size, or RSS, which represents memory used by processes and not cached
or stored on disk, such as anonymous memory maps and stacks.

Both RSS and cache memory can be broken down into active and inactive
memory. Minor (duplication or allocation) and major (full read from
disk) page faults are also included in Docker’s memory statistics.

CPU

Docker monitors both user CPU time (CPU use by the processes
themselves), and system CPU time (system calls by processes). If CPU
throttling (limiting the time available for a given container) is being
enforced, the throttling count and time for the container will also be
reported.

I/O

For I/O, Docker monitors both the number of I/O operations and the
volume of I/O in bytes. In both cases, it counts synchronous /
asynchronous and read / write separately. Docker also provides a count
of sectors (512-byte) read and written (reads/writes are counted
together), and a count of operations currently in the queue.

Network Resources

Docker also reports overall network metrics for individual containers,
including packet count, traffic volume in bytes, dropped packets, and
transmit and receive errors.

And more… Other metrics to consider are those involving storage
(and storage-related performance metrics), as well as the total number
of containers in use. In addition to container-specific metrics, it is,
of course, important to monitor such traditional factors as overall
system performance, traffic, patterns of user behavior, and application
performance, all of which may directly or indirectly impact container
activity.

The Best Ways to Monitor

Methods of monitoring and monitoring services are of course important as
well. Docker’s native monitoring tools have a bare-bones interface, but
many of the services which are built on or incorporate those tools have
considerably greater capabilities, which may include non-Docker resource
monitoring, dashboards, analytics at both the container and aggregate
levels, and an API for alerts and other automated responses. Many of
these tools are easily integrated with
Rancher, and can be used to monitor (and
analyze) Rancher-specific resources, as well as those common to
containers in general.

Why Is Container Monitoring Important?

Why is it important to monitor metrics such as these? Not surprisingly,
the main reasons for monitoring containers closely parallel the main
reasons for monitoring other applications: performance, error detection,
and detection of anomalous behavior. In the case of containers,
monitoring may help you detect problems at the system, container, and
application levels. This doesn’t mean, by the way, that the approach you
take to container monitoring is identical to the one you use in
traditional environments. As noted above, container monitoring presents
particular challenges. But the benefits of container monitoring are the
same in either case.

Real-Time Container Monitoring and Performance Optimization

Perhaps the most obvious metrics for monitoring container performance
are those involving CPU and memory use. Is a specific container (or more
typically, many or most instances of a container which compose a
specific microservice) taking up too much CPU time, or too much memory?
If so, then you have an opportunity to optimize performance by finding
and fixing the problem. The following are some specific strategies you
can adopt to address performance issues that you can identify through
real-time monitoring.

Throttling CPU

You may be able to solve some problems with excessive CPU use simply by
enforcing CPU throttling. In other cases, however, such performance
issues may be an indication of problems in design (at either the overall
application or microservices level), or coding errors. Such
performance-related problems may also show up in I/O or even network
metrics.

Throttling can serve a function similar to that of traditional
load-balancing, but it is important when confronted with CPU-related
performance problems not to simply throttle and assume that will solve
the problem. If a crucial service is using excessive CPU time,
throttling it may simply degrade performance in other ways.

When faced with chronic CPU or memory problems or similar performance
issues, it is important to look for bottlenecks at the design level, and
application errors which may result in inefficient or incorrect use of
memory, CPU services, or other resources.

Provisioning Resources

Performance problems may also result from inadequate provisioning of
resources at the system level. You may need to provision more memory,
more storage, more CPU access, or switch to a cloud service contract
which gives you higher priority in accessing resources.

But Provisioning isn’t a Cure-All As is the case with throttling,
however, it is important not to simply provision more resources and hope
that it will solve performance problems. You should first look at
application architecture, microservice design, and possible functional
problems at the coding level. You can’t fix design problems or bugs by
throwing resources at them. You may be able to overcome the obvious and
immediate inefficiencies that way, but other effects of the basic
problem may continue undetected, resulting in even greater trouble at
some point.

Container Monitoring: Bugs and Anomalous Behavior

Performance problems aren’t the only thing that real-time monitoring can
help you find and address. The following are other types of issues (ones
related to cost-optimization, security and user experience) that you
should also keep in mind when performing real-time container monitoring.

Under-utilized Resources

A container that uses resources at a lower-than-expected level may be as
serious an indication of trouble as overuse of resources. A credit-card
authorization microservice which makes almost no use of I/O or network
resources, for example, could be a sign of major problems—either with
the authorization microservice itself, with one or more of the
microservices which are supposed to use it, or with some other part of
the application which may be only indirectly involved with credit
authorization.

Suspicious Traffic

Container monitoring may uncover other forms of anomalous behavior as
well. If containers are accessing (or simply requesting) resources which
they would not ordinarily use, or if they show an unusual pattern of I/O
or network traffic, it may indicate security problems.

Unmet needs

Anomalous container behavior may also be an indication of less alarming
(but still important) problems, such as unexpected patterns of user
activity. If users are (for legitimate reasons) accessing specific
services at a much greater level than originally anticipated, for
example, you may need to look at overall architecture, at patterns of
deployment, or at the possibility of adding new services to meet
currently unmet (or under-met) user needs.

So, while individual, here-one-millisecond-gone-the-next containers may
not be persistent, everything else about your container ecosystem
(infrastructure, stored data, user interactions, resource availability)
does have an ongoing life, one which is strongly impacted by container
behavior, and which may in turn have a major impact on your
application’s performance, and on your organization’s bottom line.
Real-time container monitoring isn’t just important. It’s a necessity.