POA Forum

Xdai -> Dai transactions take a highly variable amount of time

Althea has been working on automated bridging of user deposits over to Xdai and the deposit side of the bridge (Dai -> Xdai) is fast and pretty reliable. But withdraws (Xdai -> Dai) are highly variable, sometimes they are completed in minutes, other times not for hours.

Is there any way we can host some infrastructure or participate to ensure that withdraws from Althea devices are processed in a timely and reliable manner?

1 Like

Hi! Could you share blockscout-links to specific transactions as so I can take look at them to investigate particular reason of delays?

How do you specify gas price on Dai to xDai transition
Which oracle do you use?
Do you adjust oracle’s price?

Transactions from Ethereum chain to the Xdai chain are generally fast enough.

Double publishing a send did cause my DAI to be stuck in limbo for about a day in this case.

https://etherscan.io/tx/0xc71672958a0ccebca9905ca955bada74fc6596c29bf6cc2a89ba3f9b47b258e9
https://etherscan.io/tx/0x8ff51d3a434f19ce4390165557f4d0b777926be7c0cf4d7f43c628d925840a11

But I’m able to avoid this with better code on my end that doesn’t double publish. So it’s not an immediate issue.

Specifically Xdai -> DAI transactions have a very high variance with no cause I can identify on my end.

Here I have two Xdai -> DAI transactions, same address, same day, one took less than 5 minutes the other took more than an hour.

less than 5 minutes Xdai -> DAI
https://etherscan.io/tx/0x9b6cf4668184a8148ca6677058da4dbba3332887db180d608ab89dae269f2eef
More than an hour Xdai -> DAI
https://etherscan.io/tx/0x5b0b29bbe1604d5e9bad3bf9af2a615eaffdd946b2569dbad50a4c6937936b1b

1 Like

Thanks for information. I will take a look and provide results.

2 Likes

Sorry for delay with answer!

Below is results of my investigation.

The transfer that stuck for 4 hours was originated by the transaction at August-08-2019 10:03:25 PM +3 UTC on the side of the XDai Chain.

The bridge validators caught the event UserRequestForSignature and signed it in few next blocks (on the xDai chain) at August-08-2019 10:03:35 PM +3 UTC:

As soon as all 3 signatures were collected, the event CollectedSignatures was raised and the corresponding validator immidately (at 10:03:44 +3 UTC) reacted on it by sending the transaction with signatures to unlock tokens to the Ethereum Mainnet: logs from the validator (time in UTC time zone).

But the transaction was not included in next block. It was verified much later and included almost 4 hours after sending (at August-09-2019 01:02:04 AM +3 UTC).

I paid attention on the gas price of the transaction assuming that it causes delay in the verification. The price of gas in the transaction was 1.1 Gwei.

But if we analyze other transactions that was included in the blocks on the Ethereum Mainnet near the period of time when the validator sent the signatures we see that lots of transaction with the price 1.1 Gwei was included in the blocks:

block date and time tx count max median min txs <= 1.1 gwei
8311738 August-08-2019 07:00:54 PM (UTC) 73 23.0 2.0 1.1 8
8311739 August-08-2019 07:01:01 PM (UTC) 72 60.0 5.0 1.1 15
8311740 August-08-2019 07:01:08 PM (UTC) 40 95.238095238 20.0 5.0 0
8311741 August-08-2019 07:01:09 PM (UTC) 187 50.0 8.0 1.0 34
8311742 August-08-2019 07:01:37 PM (UTC) 106 60.0 10.0 1.1 20
8311743 August-08-2019 07:01:54 PM (UTC) 39 23.715510754 2.0 1.0 16
8311744 August-08-2019 07:02:02 PM (UTC) 152 60.0 10.0 1.0 12
8311745 August-08-2019 07:02:20 PM (UTC) 27 20.0 1.1 1.0 17
8311746 August-08-2019 07:02:22 PM (UTC) 27 20.574662824 2.0 1.1 11
8311747 August-08-2019 07:02:33 PM (UTC) 67 50.0 10.0 3.0 0
8311748 August-08-2019 07:02:38 PM (UTC) 27 30.0 1.45 1.099999905 9
8311749 August-08-2019 07:02:42 PM (UTC) 35 41.0 10.0 3.0 0
8311750 August-08-2019 07:02:51 PM (UTC) 169 50.0 9.6 1.1 6
8311751 August-08-2019 07:03:19 PM (UTC) 86 50.1 9.213182383 1.000000002 14
8311752 August-08-2019 07:03:34 PM (UTC) 46 50.1 10.0 3.0 0
8311753 August-08-2019 07:03:56 PM (UTC) 193 60.0 10.0 1.1 13
8311754 August-08-2019 07:04:27 PM (UTC) 70 50.0 8.0 1.1 20
8311755 August-08-2019 07:04:29 PM (UTC) 146 60.0 10.0 1.1 8
8311756 August-08-2019 07:05:02 PM (UTC) 127 60.0 8.0 1.0 21
8311757 August-08-2019 07:05:20 PM (UTC) 123 60.0 8.0 1.1 10
8311758 August-08-2019 07:05:48 PM (UTC) 20 12.0 1.1 1.01 12
8311759 August-08-2019 07:06:12 PM (UTC) 175 60.0 10.0 1.1 3
8311760 August-08-2019 07:06:54 PM (UTC) 192 60.0 10.0 1.1 20
8311761 August-08-2019 07:07:14 PM (UTC) 272 60.0 10.0 1.0 76
8311762 August-08-2019 07:07:54 PM (UTC) 49 32.0 3.0 1.1 15
8311763 August-08-2019 07:07:59 PM (UTC) 153 50.0 10.0 1.0 25
8311764 August-08-2019 07:08:23 PM (UTC) 174 41.0 20.0 3.0 0
8311765 August-08-2019 07:08:51 PM (UTC) 64 20.0 1.1 1.1 33
8311766 August-08-2019 07:08:53 PM (UTC) 54 50.0 8.0 1.1 14
8311767 August-08-2019 07:09:05 PM (UTC) 78 50.0 3.0 1.1 31

So, relatively low gas price of the transaction is not the exact reason why the transaction was in the txpool for a long period of time.

I would say that it is sum of factors: large txpool and low gas price used in the transaction.

The proof that the issue is not on the bridge validators side can be got after analysis of the bridge operations for another transfer you mentioned.

Originated:

Signatures at August-08-2019 09:48:10 PM +3 UTC:

The signatures was relayed at August-08-2019 09:48:18 PM +3 UTC and included in the chain at August-08-2019 09:50:15 PM +3 UTC. The gas price was 1.1 Gwei.

The gas price for transaction included in the blocks issued near that time had no big difference with the previous table:

block date and time tx count max median min txs <= 1.1 gwei
8311679 August-08-2019 06:46:28 PM (UTC) 131 60.0 4.0 1.0 29
8311680 August-08-2019 06:46:35 PM (UTC) 98 60.0 9.6 1.0 7
8311681 August-08-2019 06:46:56 PM (UTC) 0 0.0 0.0 0.0 0
8311682 August-08-2019 06:46:58 PM (UTC) 64 60.0 10.0 3.0 0
8311683 August-08-2019 06:47:25 PM (UTC) 119 60.0 8.0 1.0 30
8311684 August-08-2019 06:47:26 PM (UTC) 67 40.0 10.0 1.0 16
8311685 August-08-2019 06:47:45 PM (UTC) 35 41.0 8.0 1.0 5
8311686 August-08-2019 06:47:47 PM (UTC) 15 33.333333333 1.1 1.0 8
8311687 August-08-2019 06:47:51 PM (UTC) 58 50.0 8.0 1.0 14
8311688 August-08-2019 06:48:06 PM (UTC) 37 40.0 4.0 1.0 15
8311689 August-08-2019 06:48:09 PM (UTC) 164 45.0 2.1 1.0 70
8311690 August-08-2019 06:48:32 PM (UTC) 9 20.0 10.0 4.0 0
8311691 August-08-2019 06:48:37 PM (UTC) 31 42.0 10.0 3.0 0
8311692 August-08-2019 06:48:47 PM (UTC) 0 0.0 0.0 0.0 0
8311693 August-08-2019 06:48:49 PM (UTC) 94 60.0 10.0 1.1 1
8311694 August-08-2019 06:49:07 PM (UTC) 34 32.0 10.0 1.1 2
8311695 August-08-2019 06:49:12 PM (UTC) 1 10.0 10.0 10.0 0
8311696 August-08-2019 06:49:32 PM (UTC) 122 60.0 9.6 1.0 13
8311697 August-08-2019 06:49:41 PM (UTC) 47 50.0 10.0 3.0 0
8311698 August-08-2019 06:49:46 PM (UTC) 62 60.0 10.0 3.0 0
8311699 August-08-2019 06:50:15 PM (UTC) 132 50.0 8.0 1.0 20
8311700 August-08-2019 06:50:19 PM (UTC) 21 30.0 1.1 1.0 11
8311701 August-08-2019 06:50:20 PM (UTC) 22 10.0 1.1 1.0 16
8311702 August-08-2019 06:50:24 PM (UTC) 21 41.0 8.0 1.1 3
8311703 August-08-2019 06:50:55 PM (UTC) 208 99.999997952 10.0 3.0 0
1 Like

@ttk2 in order to better understand how you can handle such situations in the future could you share the typical scenario for the transfers like you presented in the issue? Is your system behaves like a custodian for the tokens or the transfers are fully decentralized?

@akolotov sorry for the delay I had my email notifications setup incorrectly.

Our system is not custodial, every user has an embedded device in their home (a router) where they deposit Eth, this then gets bridged to xdai where it is spent in micropayments to buy bandwidth.

At some point people who have earned more xdai than they have spent on bandwidth withdraw it to Eth that they can exchange for cash.

In this case we where just testing our code and generating many deposits and withdraws of arbitrary sizes.

1 Like

Why not to go Fiat <> xDai directly?

Exchanges are too few and unreliable. It also provides us the option of moving sidechains in the future easily.

So, the piece of logic to withdraw xDai is part of the software in the embedded device, isn’t it? What actions should user perform to withdraw xDai?

They just hit the ‘withdraw’ button. It looks like a web wallet but it’s really just a frontend for the router.

The router then determines if the user is in a deployment that’s working with xdai directly or depositing/withdrawing ETH. If it’s the latter is bridges the xdai, converts it using uniswap, then sends eth to the deposit address. For most users this means they can just paste their exchange address into the box and hit go.

In the same way they can send straight ETH from an exchange to the router and it will perform this process in reverse with absolutely no user interaction. (there is a minimum transfer amount of $2, so a dust transfer won’t trigger the bridge)

Thanks for the answer.

Here is my proposal:

As you probably saw in my previous analysis that time when any withdraw transaction appears in the Ethereum Mainnet depends on two factors: the gas price proposed by the gas price oracle and the current txpool state. The gas price depends on the transactions included in the previous blocks. The gas price oracle just says us (I will intentionally simplify it) N-of-M transactions mined recently had the gas price not lesser than X, where X is the suggested gas price. The gas price oracle does not operate with the state of txpool so it does not know if it still has any sense to suggest this gas price since the pool could be flooded by transaction that will provide higher fee for the validators. It means that we either need to relay on a smarter gas price oracle to support constant velocity of the bridge transactions verification or have a mechanism to push the bridge transaction if they were jammed due to the high traffic issues.

The variant with the smarter gas price oracle is still not good enough because we need constantly monitor what it returns and analyse how returned data impact the bridge transaction velocity: if data still cause the transactions jamming we need to change the oracle. But do we have enough oracles to choose?

The functionality to push transaction could be easily implemented since does not require the bridge validators to participate. In order to push the transaction you need to just make sure that a transaction that logged CollectedSignatures has no corresponding transaction in the Ethereum Mainnet (no transaction passing a specific tx hash towards the bridge contract). If so, you need collect signatures for the bridge contract (as it is performed by collected-signatures-watcher), choose an adequate gas price, construct a transaction to invoke the executeSignature method and send it to Ethereum Mainnet. The bridge contract on this chain is built as so it work with any sender - the proof of data authenticity is the signatures provided as part of executeSignature call.

I see that this could be implemented as part of the router software: as soon as the user requested withdrawal the router is observing life of bridge operations linked with the withdrawal. If the withdrawal is stuck, the router pushes it in similar way as I described above. In fact, you need to adapt our collected-signatures-watcher and foreign-sender - most of the code already exist there.

1 Like

We actually had a solve a very similar issue (our routers transact in eth currently), if you select a gas price 1% higher than the median gas price provided by full node endpoints you’ll be fine.

If you select a higher value than the median you’re by definition ahead of most transactions. This is a good strategy for dealing with txspam, which usually has a fixed gas price. In theory a wide array of transactions with varied fees could exist that’s well above the capacity of the blockchain but that happens far less often than a rouge script.

This of course is a bit of a tragedy of the commons, if everyone did this gas prices would spiral to infinity. Someone has to either have a more complicated system or keep track manually.

Agreed, it seems that breaking out that component and running one ourselves is a viable solution.

Do you have any link to results of such kind of testing? I am very curious to see a comparison of such behavior and other strategies to choose the gas price.

Is this statement true only for transactions in the past. My understanding is that the median value shows us a retrospective of the gas price.

So, you can contact to us here and we could provide any assistance.

Sadly this is anecdotal right now. Our data gathering doesn’t have an easy way to graph time to block inclusion. Just the failure rate, which dropped by an order of magnitude.