Dear bridge validators,
please, check that your oracles are functioning correctly. To do that - go to your instance and run:
sudo docker ps
You should see all the containers up and running, like:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
be429b1987d5 bridge_bridge_affirmation "npm run watcher:aff…" 20 hours ago Up 20 hours bridge_bridge_affirmation_1
516718c6754d bridge_bridge_senderhome "npm run sender:home…" 20 hours ago Up 20 hours bridge_bridge_senderhome_1
9a4f166705f1 redis:4 "docker-entrypoint.s…" 20 hours ago Up 20 hours 6379/tcp bridge_redis_1
061a5a82547f bridge_bridge_request "npm run watcher:sig…" 20 hours ago Up 20 hours bridge_bridge_request_1
9175f0566606 rabbitmq:3 "docker-entrypoint.s…" 20 hours ago Up 20 hours 4369/tcp, 5671-5672/tcp, 25672/tcp bridge_rabbit_1
fb0b574e6fcd bridge_bridge_senderforeign "npm run sender:fore…" 20 hours ago Up 20 hours bridge_bridge_senderforeign_1
236314b5797c bridge_bridge_collected "npm run watcher:col…" 20 hours ago Up 20 hours bridge_bridge_collected_1
If the output is empty in your case - please, proceed with the information below to fix the validator issue.
TL;DR
sudo -i
apt install containerd
if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi;
service poabridge rebuild
exit
Issue description
There was a security vulnerability in a sudo package (see the related issue).
At the same time the new version of containerd (1.2.10-2) released.
You might have decided, that vulnerability discovery is a good point to update all your system packages to keep node up-to-date. So, if you deviated a little from the instructions on the sudo package update and updated all the system packages instead of the sudo only, you might have ended up with a 1.2.10-2 containerd version installed.
As mentioned at Docker’s Github issue #813 there was a major issue at containerd, that led to all containers being broken. When the system daemon tried to restart the service, an error “No status provided on response: unknown” appeared, which can be observed by the system logs.
Solution
The initial solution was to downgrade the version of containerd package, but the fix was already released. Now, you can safely upgrade to 1.2.10-3 version of containerd to fix the issue. Do the apt install conatinerd to install latest version of containerd containing the hot-fix patch.
However, it might happen that even after the patch some of Oracle’s containers still will not work properly (see the related issue). The reason for that might be in a major downtime duration. The bridge may not have enough time to process all the events, so it will fail due to timeout.
Please, restart the service using sudo service poabridge restart command and monitor your containers using sudo docker ps command. Keep an eye on status command for a few minutes. If some of the containers keep restarting over and over - do the following:
- Open the
.envconfiguration file - it should be located on the node, at/home/poadocker/bridgefolder. Depending on your Oracle version it may be located either at root of this directory or at theoraclesubfolder. - Add either
MAX_PROCESSING_TIMEorORACLE_MAX_PROCESSING_TIMEvariable. Set it value equal to 60000. You can use the following command to do this automatically:
if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi;
- Rebuild your Oracle using
sudo service poabridge rebuildcommand - Check that the issue is fixed using
sudo docker pscommand. Monitor it for several minutes. All the container should be working stably now.
Thanks for reading this and keeping your environments safe and sound!