How to collect Docker metrics with cgroups

Posted by : Thibault Laurens | Docker Containers, Engineering, Docker

blog-docker-stats.png

This blog post is a retrospective on how Outlyer built a specific part of its Docker monitoring solution: the container's metrics collection.

Phase 1: quick and hacky easy: cAdvisor

Outlyer's very first implementation to get metrics from Docker containers was based on Google's cAdvisor.

At that time, Docker was getting increasingly popular and some of our customers started to ask for a better solution to visualize the data from their Docker containers inside Outlyer. We needed to deliver a solution in a short amount of time that would be reliable and easy to setup for our users.

At first, cAdvisor appeared to tick all of the boxes.

cadvisor.png

Getting Docker statistics from your containers with cAdvisor is very easy. The only thing to do is deploy and run a cAdvisor container on all your Docker hosts.

Out of the box it will collect, process and even display your container metrics through a basic web interface. You can watch your containers' performance in real-time.

The best part for us is that it would expose all the collected metrics through a REST API.

Our solution was then as simple as packaging a container running cAdvisor along with some code that would collect the data from the cAdvisor API at regular intervals. We simply forward that data to our graphite endpoint. Problem solved!

Well no, not really.

While this solution seemed effective at first, we quickly realized that it was far from ideal. Some of our customers started to report that our container would use way too much memory on their system, or even worse; cAdvisor would start crashing after a while. We decided to change how we would pull cAdvisor data. One solution was to use the Prometheus plugin format since cAdvisor also exposes container statistics as Prometheus metrics. Unfortunately that didn't fix the problem and to top it all off, we started to find that cAdvisor was very hard to debug. We also knew it would be difficult to customize later on.

Being unable to control what kind of container metrics we would grab was not a good solution for the long term. It was time to look elsewhere.

Phase 2: Down the Cgroups rabbit hole

Between cAdvisor and Cgroups we briefly considered using the Docker stats api. However, the constant changes and lack of support in older Docker versions meant that our prototype didn't even make it to production

Under the hood, Docker uses a feature from the Linux kernel called Control groups, or Cgroups, to allocate and track system resources for containers (like cpu, memory and block IO). Cgroups use a pseudo-filesystem to expose these metrics, making them available to collect.

So how do we do that? Let me explain.

  1. Locating the Cgroups pseudo-filesystem

One thing you will notice if you run our Docker container on your hosts is that we ask you to mount the /proc directory of the host inside our container. The reason for this is that one of the first things we need to do is look into /proc/mounts to locate the Cgroups pseudo-filesystem mountpoint (generally you will find it under /sys/fs/cgroup, but some operating systems do differ).

bash-4.3# grep cgroup /rootfs/proc/mounts
cpuset /rootfs/sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
cpu /rootfs/sys/fs/cgroup/cpu cgroup ro,nosuid,nodev,noexec,relatime,cpu 0 0
cpuacct /rootfs/sys/fs/cgroup/cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpuacct 0 0
blkio /rootfs/sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
memory /rootfs/sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
  1. Locating the containers' pseudo-files containing statistics

Next, for a given container process id, we read /proc/<pid>/cgroup. Here, we find a path relative to the Cgroup mountpoint for each Cgroup subsystem: cpu, memory, block IO, etc.

For example: /memory/docker/<container_id>.

Putting it all together, to get container metrics related to say memory, we will look under/sys/fs/cgroup/memory/docker/<container_id>. In there we find some files ready to be parsed containing the metrics we are interested in.

As you see, there are quite a few:

bash-4.3# ls /rootfs/sys/fs/cgroup/memory/docker/ea203878f6759a0d41a6532dc840446a5f91020bbbbc1f14fe32a7a56473f54b/
cgroup.clone_children               memory.kmem.slabinfo                memory.memsw.failcnt                memory.stat
cgroup.event_control                memory.kmem.tcp.failcnt             memory.memsw.limit_in_bytes         memory.swappiness
cgroup.procs                        memory.kmem.tcp.limit_in_bytes      memory.memsw.max_usage_in_bytes     memory.usage_in_bytes
memory.failcnt                      memory.kmem.tcp.max_usage_in_bytes  memory.memsw.usage_in_bytes         memory.use_hierarchy
memory.force_empty                  memory.kmem.tcp.usage_in_bytes      memory.move_charge_at_immigrate     notify_on_release
memory.kmem.failcnt                 memory.kmem.usage_in_bytes          memory.oom_control                  tasks
memory.kmem.limit_in_bytes          memory.limit_in_bytes               memory.pressure_level
memory.kmem.max_usage_in_bytes      memory.max_usage_in_bytes           memory.soft_limit_in_bytes
  1. Looking elsewhere for network statistics

For the metrics related to the configured network interfaces, we read directly from/proc/<pid>/net/dev as shown below:

bash-4.3# cat /rootfs/proc/11303/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
gretap0:     0       0    0    0    0     0          0         0        0       0    0    0    0     0       0         0
    lo:    430       4    0    0    0     0          0         0      430       4    0    0    0     0       0          0
  sit0:      0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip_vti0:     0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  gre0:      0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip6tnl0:     0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  eth0: 120087    1175    0    0    0     0          0         0  3993998     747    0    0    0     0       0          0
 tunl0:      0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip6_vti0:    0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
ip6gre0:     0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
  1. Bonus points: Load Average

It's a little bit more tricky for our container to get the load average for all the other containers running on the host. The way we do it: we use netlink, a socket based interface in Linux, to ask the kernel directly. I won’t be going into too much detail about that here as this will be covered entirely in a future blog post. One thing to keep in mind though, as stated in our documentation you have to run the container with --net host if you would like load averages for your containers.

Ok, that's pretty much it now.

As we have seen, messing around with Cgroups, pseudo-files and system call is not as straight forward as spinning up a cAdvisor container. But the advantages are quite significant:

  • it removes any third party dependency (unexpected crash, high memory footprint: no more).
  • we get full control over the metrics we collect and how we collect them. The list can be found here.
  • this solution is not limited to Docker only and it will make it easy to support a new container technology in the future.

The end result:

dashboard.png

Conclusion

We can summarize Docker metric collection in two statements:

  1. Out-of-the-box solutions are easy to setup but you have to take into account their limitations. It is probably not going to be good enough for monitoring a production environment.

  2. Cgroups on the other hand, seems like the way to go. But building a solution around the lower levels will give you a headache. And that’s where Outlyer comes in.

Deploy our agent directly on your hosts, or within a container.

We also provide template files so you can deploy our container directly in your kubernetes or Docker Swarm cluster

You can also run our plugins directly against your container to get custom metrics. But that might be for another post.

hostview-1.png

Go Beyond Cloud Monitoring

Scale your infrastructure monitoring for Cloud, SaaS, Microservices and IoT-ready deployments with Outlyer. We’ve designed a solution for the agile organization. Start now and see your operational metrics in minutes.

Start Your 14 Day Trial Now