xDai Stable Chain xDai Stable Chain BlockScout BlockScout TokenBridge TokenBridge

How do I fix a stuck token / coin transfer?

When testing the NATIVE_TO_ERC bridge between Volta (Energy Web Foundation Testnet) and Kovan, one of the transfers was not processed correctly.

The coins were successfully transferred to the Volta bridge, but the Kovan tokens were never minted.

I did successful transfers before and after this incident.

I also can’t find any difference between the failed and the successful transfer, so I assume the bridge watcher just didn’t pick up the event.

The bridge-monitor tools seem to only work for ERC_TO_ERC bridges, so is there a way to either

  • “Reset” the bridge, so that past transfers are checked again and reprocessed if necessary?

or

  • Refund/Release the coins of this transfer on Volta?

Hi!

First of all I have verified the contracts in the both Volta and Kovan chains. Now the source code of contracts is associated with exact bytecode. If you look at the list of calls after that it is easy to see how exactly user interact to the contracts and contracts interact to each other.

Then let’s look at the bridge operations just after the transaction you refer:

  1. 0x11b286fcde98b591109ddaac5da2d5612b2e413068283714f98ac780bcb52629 - the validator 0x054a3EbE6599e9e7299fEc5abBFa6c697BCec2EE sends its confirmation to relay the assets to another side of the bridge.
  2. 0xbe2567727de7784426906040e0cfe9c030fe70206b8dd31f731eb1c4d58014fb - the validator 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 sends the confirmation.
  3. 0x0feeb1a80fbe6fa17f2cdeeca33c64f5f8e8f7df84178359c982103c1946c20f - the validator 0x8BEf9E7aeb7Ef2EC7680C299F7DD8BdA27490eb7 sends its confirmation. But the call is reverted since enough number of signatures already collected (the first two confirmations).
  4. The validator 0x064Ad23A6688d5d4127a1f0BEbBf1D2E249E11a2 did not send its confirmation at all. Either it handled the situation that enough signatures already collected or it was turned off.

So, the last transaction applied to change the state of the contract is 0xbe2567727de7784426906040e0cfe9c030fe70206b8dd31f731eb1c4d58014fb. The logs tab in this transaction shows to events raised by the contract. The second event is SignaturesCollected() and it contains the address of the validator responsible for transferring the collected signatures to the contract on the foreign side of the bridge.


Another point that is important to notice is the timestamp of the block which this transaction was included into.
image
This timestamp is needed to try to find the corresponding transaction sent by the validator 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 on the foreign side.

Let’s open the Blockscout for Kovan chain and look at transactions sent to the bridge contract in that chain.

  1. Block #11558540 was produced on June-13-2019 09:03:04 PM +3 UTC.
  2. Block #11571274 - on June-14-2019 11:12:16 AM +3 UTC.
  3. Block #11571438 - on June-14-2019 11:23:12 AM +3 UTC
  4. Block #11571628 - on June-14-2019 11:35:52 AM +3 UTC.

It is expected that 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 should send the transaction with signatures somewhere in this time frame. But as we can see there is no such transaction.

The reason why the validator did not send the transaction cannot be analysed without the logs from the node where the corresponding token bridge oracle is run. Most probably it is due to fact that the originator of some transactions to transfer tokens to VT chain was 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 and it is the bridge validator account too. So, the nonce in the Kovan chain for this account was increased by these manual operations to send the tokens whereas the corresponding oracle instance doesn’t know about this and tries to operate with old nonce. Anyway please provide the logs which could help identify the root cause. Our statistic of running xDai bridge shows that the validators never missed the blocks so it is quite impossible that this issue happened on your system.

In order to fix the situation you need to stop the token bridge oracle, reset the block observed by the validator in the oracle database and start the oracle. For this you need to login to the node where the validator with the address 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 is run and follow the instruction in the deployment component README file. It is necessary to reset the block for the collected-signatures watcher. Since we want that the watcher handles the CollectedSignature() event one more time the target block should be 771653.
Here is the list of the commands that needs to be performed on the node. They are applicable only for the case if you used the deployment playbooks to deploy the token bridge oracles on the nodes. If you used another approach please let me know and I will prepare another instruction:

$ sudo service poabridge stop
$ sudo su poadocker
$ cd ~/bridge
$ docker-compose stop bridge_affirmation bridge_request bridge_collected
$ docker-compose exec bridge_senderhome bash ./reset-lastBlock.sh collected-signatures 771653
$ exit
$ sudo service poabridge start

Monitor the new transactions on Kovan chain after the oracle restart. Few new transactions should appear from the 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 address.

If it is no transaction which transfer 42 VT to 0x58D29CeB1A7D5192DD3A840D77D88779c8BF5246 (yes, the originator of the transaction to relay the assets is the same private key as the bridge validator) like this:


Prepare the gist with logs from that system and specify the link to the gist in the reply to this message.

2 Likes

It should not be true. Please provide steps how you tried to run the monitor and logs as so I can assist you to configure the monitor.

1 Like

Thanks a lot! Resetting the block number did solve the problem immediately. As you said, I was using the account as the validator as well as for sending test transactions. I’ll avoid using the validator accounts for anything else in the future, which should be a good practice anyway.

I was running the validator in debug logging mode, so the logs are nearly useless, but some grepping showed some error messages related to the nonce, as you said :+1:

Found the issue in the code. I had to configure ERC20_ADDRESS in the .env file for checkWorker3.js to work.

This is not in the .env.example file for version 2.0.0 or 2.2.0. See https://github.com/poanetwork/bridge-monitor/blob/eaf2a65685e64594013349abd8bc35904a7540be/stuckTransfers.js#L78

@saschagoebel as it is described in the documentation the worker checkWorker3.js is not applicable for v2 contracts. It should be used only with old contracts. E.g. they were deployed for the POA20 Bridge.

# check stuck transfers called by transfer, not transferAndCall
# only applicable for bridge-rust-v1-native-to-erc
node checkWorker3.js

So, please do not use this worker for your version of the bridge.

:+1: Using the tools coming with the bridge itself seems to work better anyway :wink: