Debug Containerd in Production

Photo by John Barkiple on Unsplash

Sometimes reading logs won’t be enough to understand the problem. There are many ways to debug the error such as reading trace logs, putting more logs in the code, or using a debug library. I believe that the easiest way to do it is to run the program in a code editor.

In this article, I will demonstrate how you can debug Containerd with VSCode in a remote production environment.

You need Golang in the Containerd running machine. You can find the installation here.

Steps for Debugging

For short, we will clone the Containerd repository, configure the Virtual Studio Code, and do some configuration in the Linux system.

If your user doesn’t have SUDO privileges, you must adjust your Linux system. So, I have to add sudo to each command to get SUDO privileges or run sudo su - command to become the root user. To automate this process, you can addsudo su - command to $HOME/.profile. This way whenever you connect to the machine, you will become the root user.

1 — Clone repository

There are a couple of ways to bind your project to a remote machine for example some code editors (IDEs) can copy the project to the remote machine. Here, we will clone the repository to the remote machine.

1.1 — Add your remote machine

Remote — SSH extension allows us to add remote machines in the VSCode. This will simplify our work here, and you can easily switch environments with this features.

1.2 — Add the repository

Remote — SSH features can clone the repository in the remote machine.

• Click the button shown in the screenshot to connect.

• Select Clone Repositoryand type containerd/containerd to clone containerd from Github.

Note: VSCode would ask you to install some packages and extensions. Make sure you don’t skip any of these.

2 — Launch.json

This file is beneficial since it allows you to configure debugging details.

2.1 — Create a launch.json file

launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Launch file",
      "type": "go",
      "debugAdapter": "legacy",
      "request": "launch",
      "program": "${file}"
    }
  ]
}

This is what the launch.json should look like. ${file} which will allow us to execute the actively opened file. You can define a static file. Also, you can add additional configurations like mode which is up to you. You can find more details about debugging here.

Configuration is done. Time to disable containerd service and start debugging.

3 — Disable service

Since we are using a production environment, we need to stop running the service. This will cause a service shortage but it is acceptable since we are working on a problematic node. However, you might want to consider draining the node.

$ systemctl stop containerd
$ systemctl status containderd
● containerd.service - containerd container runtime
  Loaded: loaded (/etc/systemd/system/containerd.service; enabled; vendor preset: enabled)
  Active: inactive (dead) since Tue 2021-12-07 22:47:44 +03; 13h ago
  Docs: https://containerd.io
  Process: 1072 ExecStartPre=/sbin/modprobe overlay (code=exited, status=1/FAILURE)
  Process: 1109 ExecStart=/usr/local/bin/containerd (code=exited, status=0/SUCCESS)
  Main PID: 1109 (code=exited, status=0/SUCCESS)
$ cat /etc/systemd/system/containerd.service
.
.
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
.
.

4 — Start the program in debug mode.

For containerd project, we need to run the cmd/containerd/main.go which starts the program. Like in this screenshot, when you are in the correct file you can hit the button shown with the arrow.

The program is running on the remote machine but you are working in your local system. Whenever you put a breaking point you will see that program will stop when it reaches the point. Congratulations!

Note: If you face this btrfs/ioctl.h: No such file or directory error, you nedd to install this libbtrfs-dev package , install the asked package.

$ apt-get update
$ apt-get install libbtrfs-dev

In summary, since we are working in a production environment, we didn’t have to deal with all the installation processes for the containerd to run we just put the actual binary aside and put our code in the line. You can use this method for other projects to debug in production.