Containers Resource Accounting with cgroups

When building containers for HPC it is more than advisable to measure how much memory, CPU time and other resources such container needs. We had to dive deep into detailed resources accounting on Linux to get some actual numbers.

Measuring resource utilization for programs in general can be a tricky endeavour especially with respect to applicable methodology. If we look at the memory usage, it may vary during the program execution - which might be interesting for figuring out where the maximums are. However in the end the important number is the global maximum of used memory because that is what is needed to run such program. Containers are no different with respect to memory usage.

Another resource worth analyzing is the number of simultaneously running threads. That gets a bit more complicated with containers as there might be many programs running in the container with each program utilizing multiple threads. Again the development of the number of threads during the container execution might be interesting but make sure there is no parallelism bottleneck, we need to know the global maximum again.

Last but not least is the information about time. How long did the container take to run, how much CPU time was spent on actual user computations and how much was the system overhead - those are crucial numbers as people want to know how fast they get the results. But again during container execution both the system and user time grows at some (typically non-linear) pace and the values might help with some optimizations.

With Docker, it is rather easy to obtain the cgroup of particular container. We run the docker image and get the long id:

docker run --name unique-name image arguments ...
LONGID=$(docker ps --no-trunc -f name=unique-name --quiet)
SCOPE=/sys/fs/cgroup/system.slice/docker-$LONGID.scope

And all the accounting information for the image being run can be accessed at the $SCOPE path.

With Apptainer/Singularity we need to wrap it in systemd-run to ensure it gets its own cgroup:

systemd-run --scope --user sh apptainer-script.sh
CPID=$!
CGROUP=$(sed -e 's#^[^/]*/##' /proc/$CPID/cgroup)
SCOPE=/sys/fs/cgroup/$CGROUP

And again, all the accounting information is available under the $SCOPE path. Now we can have a look at what we can find there.

There are several interesting files available:

• cgroup.threads contains the list of all active threads and we can get their number just by counting the number of lines,
• memory.peak contains the maximum memory allocated to this cgroup ever,
• memory.current is the current memory utilization in bytes,
• cpu.stat contains the information about both user and system time spent on this cgroup, however the units are the kernel scheduler time slices.

In order to convert the scheduler time slices into something tangible (say seconds) we need to know the kernel configuration variable CONFIG_HZ. The easiest way on current systems is to look at bundled configuration in the boot directory and at the same information provided by the proc filesystem. At least one of those should always be present on current systems:

CONFIG_HZ=$(zgrep ^CONFIG_HZ= /proc/config.gz /boot/config-`uname -r` 2>/dev/null|sed 's/^.*=//')

Now with all the variables in place, the only thing needed is to run a separate process that will - at regular intervals - output all the collected statistics into a simple text file. And when the $CPID process or the given docker image terminates, terminate as well.

All the tools used for these measurements will be released under an open source licence once the related research gets published. So stay tuned if you want to try it yourself.

Hope you found our little venture into cgroup-based resource accounting interesting and see ya next time!