[Operations] Possible docker issues after recent security patch

Support
#1

Dear bridge validators,
please, check that your oracles are functioning correctly. To do that - go to your instance and run:

sudo docker ps

You should see all the containers up and running, like:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                NAMES
be429b1987d5        bridge_bridge_affirmation     "npm run watcher:aff…"   20 hours ago        Up 20 hours                                              bridge_bridge_affirmation_1
516718c6754d        bridge_bridge_senderhome      "npm run sender:home…"   20 hours ago        Up 20 hours                                              bridge_bridge_senderhome_1
9a4f166705f1        redis:4                       "docker-entrypoint.s…"   20 hours ago        Up 20 hours         6379/tcp                             bridge_redis_1
061a5a82547f        bridge_bridge_request         "npm run watcher:sig…"   20 hours ago        Up 20 hours                                              bridge_bridge_request_1
9175f0566606        rabbitmq:3                    "docker-entrypoint.s…"   20 hours ago        Up 20 hours         4369/tcp, 5671-5672/tcp, 25672/tcp   bridge_rabbit_1
fb0b574e6fcd        bridge_bridge_senderforeign   "npm run sender:fore…"   20 hours ago        Up 20 hours                                              bridge_bridge_senderforeign_1
236314b5797c        bridge_bridge_collected       "npm run watcher:col…"   20 hours ago        Up 20 hours                                              bridge_bridge_collected_1

If the output is empty in your case - please, proceed with the information below to fix the validator issue.

TL;DR

sudo -i
apt install containerd
if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi;
service poabridge rebuild
exit

Issue description
There was a security vulnerability in a sudo package (see the related issue).
At the same time the new version of containerd (1.2.10-2) released.

You might have decided, that vulnerability discovery is a good point to update all your system packages to keep node up-to-date. So, if you deviated a little from the instructions on the sudo package update and updated all the system packages instead of the sudo only, you might have ended up with a 1.2.10-2 containerd version installed.

As mentioned at Docker’s Github issue #813 there was a major issue at containerd, that led to all containers being broken. When the system daemon tried to restart the service, an error “No status provided on response: unknown” appeared, which can be observed by the system logs.

Solution
The initial solution was to downgrade the version of containerd package, but the fix was already released. Now, you can safely upgrade to 1.2.10-3 version of containerd to fix the issue. Do the apt install conatinerd to install latest version of containerd containing the hot-fix patch.

However, it might happen that even after the patch some of Oracle’s containers still will not work properly (see the related issue). The reason for that might be in a major downtime duration. The bridge may not have enough time to process all the events, so it will fail due to timeout.
Please, restart the service using sudo service poabridge restart command and monitor your containers using sudo docker ps command. Keep an eye on status command for a few minutes. If some of the containers keep restarting over and over - do the following:

  1. Open the .env configuration file - it should be located on the node, at /home/poadocker/bridge folder. Depending on your Oracle version it may be located either at root of this directory or at the oracle subfolder.
  2. Add either MAX_PROCESSING_TIME or ORACLE_MAX_PROCESSING_TIME variable. Set it value equal to 60000. You can use the following command to do this automatically:
if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi;
  1. Rebuild your Oracle using sudo service poabridge rebuild command
  2. Check that the issue is fixed using sudo docker ps command. Monitor it for several minutes. All the container should be working stably now.

Thanks for reading this and keeping your environments safe and sound!

2 Likes