Dear bridge validators,
please, check that your oracles are functioning correctly. To do that - go to your instance and run:
sudo docker ps
You should see all the containers up and running, like:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES be429b1987d5 bridge_bridge_affirmation "npm run watcher:aff…" 20 hours ago Up 20 hours bridge_bridge_affirmation_1 516718c6754d bridge_bridge_senderhome "npm run sender:home…" 20 hours ago Up 20 hours bridge_bridge_senderhome_1 9a4f166705f1 redis:4 "docker-entrypoint.s…" 20 hours ago Up 20 hours 6379/tcp bridge_redis_1 061a5a82547f bridge_bridge_request "npm run watcher:sig…" 20 hours ago Up 20 hours bridge_bridge_request_1 9175f0566606 rabbitmq:3 "docker-entrypoint.s…" 20 hours ago Up 20 hours 4369/tcp, 5671-5672/tcp, 25672/tcp bridge_rabbit_1 fb0b574e6fcd bridge_bridge_senderforeign "npm run sender:fore…" 20 hours ago Up 20 hours bridge_bridge_senderforeign_1 236314b5797c bridge_bridge_collected "npm run watcher:col…" 20 hours ago Up 20 hours bridge_bridge_collected_1
If the output is empty in your case - please, proceed with the information below to fix the validator issue.
sudo -i apt install containerd if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi; service poabridge rebuild exit
There was a security vulnerability in a
sudo package (see the related issue).
At the same time the new version of
containerd (1.2.10-2) released.
You might have decided, that vulnerability discovery is a good point to update all your system packages to keep node up-to-date. So, if you deviated a little from the instructions on the
sudo package update and updated all the system packages instead of the
sudo only, you might have ended up with a 1.2.10-2
containerd version installed.
As mentioned at Docker’s Github issue #813 there was a major issue at
containerd, that led to all containers being broken. When the system daemon tried to restart the service, an error “No status provided on response: unknown” appeared, which can be observed by the system logs.
The initial solution was to downgrade the version of
containerd package, but the fix was already released. Now, you can safely upgrade to
1.2.10-3 version of
containerd to fix the issue. Do the
apt install conatinerd to install latest version of
containerd containing the hot-fix patch.
However, it might happen that even after the patch some of Oracle’s containers still will not work properly (see the related issue). The reason for that might be in a major downtime duration. The bridge may not have enough time to process all the events, so it will fail due to timeout.
Please, restart the service using
sudo service poabridge restart command and monitor your containers using
sudo docker ps command. Keep an eye on status command for a few minutes. If some of the containers keep restarting over and over - do the following:
- Open the
.envconfiguration file - it should be located on the node, at
/home/poadocker/bridgefolder. Depending on your Oracle version it may be located either at root of this directory or at the
- Add either
ORACLE_MAX_PROCESSING_TIMEvariable. Set it value equal to 60000. You can use the following command to do this automatically:
if [ ! -f /home/poadocker/bridge/.env ]; then echo "ORACLE_MAX_PROCESSING_TIME=60000" >> /home/poadocker/tokenbridge/oracle/.env; else echo "MAX_PROCESSING_TIME=60000" >> /home/poadocker/bridge/.env; fi;
- Rebuild your Oracle using
sudo service poabridge rebuildcommand
- Check that the issue is fixed using
sudo docker pscommand. Monitor it for several minutes. All the container should be working stably now.
Thanks for reading this and keeping your environments safe and sound!