Hi. Recently we have discovered an issue with the disk space on one of our bridge validators. The detailed investigation led us to the root of the issue which was the YAML parsing features in our deployments scripts. YAML parsed the
644 mode of the file not as an octal number (which is the Linux file security permissions), but as decimal.
644 in decimal were converted to
1204 in octal and applied to the config files. As the result, logrotate were not able parse the docker logs config file and the logrotation were not performed.
(TL;DR) To fix the issue, please:
- SSH to your instance(s) of bridge validator(s) and execute:
sudo chmod 0644 /etc/logrotate.d/docker-logs /etc/rsyslog.d/35-remote-logging.conf /etc/rsyslog.d/30-docker.conf
This should fix the issue, logs will be rotated soon.
- (Optional) However, if you want, you can forcibly rotate your logs to free up space immediately:
sudo logrotate -vf /etc/logrotate.d/docker-logs
Issue in GitHub: https://github.com/poanetwork/tokenbridge/pull/51/