Can't publish transactions from an xdai 'full node'

**ttk2** · January 25, 2019, 4:58pm

This link provides instructions for setting up a local Parity client running on the xdai chain using the chain definition files.

Overall I would describe this as very easy, but when I go to submit a transaction using a hash I get the following message.

Jan 24 21:49:45 fedora-s-1vcpu-1gb-nyc1-01-dai-infra-01 parity[993]: 2019-01-24 21:49:45 UTC Closing the block failed with error Error(Engine(FailedSystemCall("please ensure the contract and method you\'re calling exist! failed to decode empty bytes. if you\'re using jsonrpc this is likely due to jsonrpc returning `0x` in case contract or method don\'t exist")), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } }). This is likely an error in chain specificiations or on-chain consensus smart contracts.

which seems to say that I need to be a validator to submit transactions… combined with strange behaviour from the official (presumably validating) nodes RPC endpoints effectively makes it impossible for althea.org light client devices to operate using xDai

**igorbarinov** · January 25, 2019, 12:32am

a. You don’t need to be a validator to submit a tx to the network.
b. The message you are getting is a known warning (not an error)
Here is the issue for it:

c. We are testing the solution to the issue on the testnet. If you are curious about it you can read more information here Bugfix test on Sokol testnet (step 1)
d. The warning doesn’t affect tx you sent to the network. Likely it was not executed because of the different reason

**ttk2** · January 25, 2019, 4:26pm

The transaction is accepted as valid by the full node, which responds with a txid, but the txid never shows up on https://blockscout.com/poa/dai and the full node, not previously printing this error message begins doing so.

I’ll try and get a longer stack trace and see if I can’t find something more specific about why transactions aren’t being propagated.

**ttk2** · January 25, 2019, 5:01pm

grep 0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc /var/log/parity.log
2019-01-25 14:26:31 UTC jsonrpc-eventloop-0 TRACE own_tx  Importing transaction: PendingTransaction { transaction: SignedTransaction { transaction: UnverifiedTransaction { unsigned: Transaction { nonce: 54, gas_price: 250000000000, gas: 21000, action: Call(0x72d9e579f691d62aa7e0703840db6dd2fa9fae21), value: 302765744400000000, data: [] }, v: 236, r: 87698742786621454063727467889542601869030536055776995617625870797135893878243, s: 15768194683199897964444217664501426774764034905144141017617127538235788840004, hash: 0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc }, sender: 0x9401dcf086e223763604f9cb826814ffff48ec70, public: Some(0x8b28169f589659658a0f43c59de1794964687e9cdd8d81f7acf0329a8f6566b3a983997d9a326ef2618977fe5ab92d48e2d5d0f44bb80e95725a0ba77d264749) }, condition: None }
2019-01-25 14:26:31 UTC jsonrpc-eventloop-0 DEBUG own_tx  Imported to the pool (hash 0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc)
2019-01-25 14:26:31 UTC jsonrpc-eventloop-0 DEBUG txqueue  [0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc] Added to the pool.
2019-01-25 14:26:31 UTC jsonrpc-eventloop-0 DEBUG txqueue  [0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc] Sender: 0x9401…ec70, nonce: 54, gasPrice: 250000000000, gas: 21000, value: 302765744400000000, dataLen: 0))
2019-01-25 14:26:31 UTC jsonrpc-eventloop-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":"0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc","id":1}.
2019-01-25 14:26:32 UTC jsonrpc-eventloop-0 TRACE mio::poll  registering with poller2019-01-25 14:26:32 UTC jsonrpc-eventloop-0 TRACE rpc  Request: {"id":1,"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc"]}.
2019-01-25 14:26:32 UTC jsonrpc-eventloop-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":null,"blockNumber":null,"chainId":"0x64","condition":null,"creates":null,"from":"0x9401dcf086e223763604f9cb826814ffff48ec70","gas":"0x5208","gasPrice":"0x3a35294400","hash":"0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc","input":"0x","nonce":"0x36","publicKey":"0x8b28169f589659658a0f43c59de1794964687e9cdd8d81f7acf0329a8f6566b3a983997d9a326ef2618977fe5ab92d48e2d5d0f44bb80e95725a0ba77d264749","r":"0xc1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3","raw":"0xf86d36853a352944008252089472d9e579f691d62aa7e0703840db6dd2fa9fae21880433a3d7251e64008081eca0c1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3a022dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","s":"0x22dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","standardV":"0x1","to":"0x72d9e579f691d62aa7e0703840db6dd2fa9fae21","transactionIndex":null,"v":"0xec","value":"0x433a3d7251e6400"},"id":1}.
2019-01-25 14:26:37 UTC jsonrpc-eventloop-0 TRACE rpc  Request: {"id":1,"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc"]}.
2019-01-25 14:26:37 UTC jsonrpc-eventloop-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":null,"blockNumber":null,"chainId":"0x64","condition":null,"creates":null,"from":"0x9401dcf086e223763604f9cb826814ffff48ec70","gas":"0x5208","gasPrice":"0x3a35294400","hash":"0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc","input":"0x","nonce":"0x36","publicKey":"0x8b28169f589659658a0f43c59de1794964687e9cdd8d81f7acf0329a8f6566b3a983997d9a326ef2618977fe5ab92d48e2d5d0f44bb80e95725a0ba77d264749","r":"0xc1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3","raw":"0xf86d36853a352944008252089472d9e579f691d62aa7e0703840db6dd2fa9fae21880433a3d7251e64008081eca0c1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3a022dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","s":"0x22dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","standardV":"0x1","to":"0x72d9e579f691d62aa7e0703840db6dd2fa9fae21","transactionIndex":null,"v":"0xec","value":"0x433a3d7251e6400"},"id":1}.
2019-01-25 14:26:42 UTC jsonrpc-eventloop-0 TRACE rpc  Request: {"id":1,"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc"]}.
2019-01-25 14:26:42 UTC jsonrpc-eventloop-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":null,"blockNumber":null,"chainId":"0x64","condition":null,"creates":null,"from":"0x9401dcf086e223763604f9cb826814ffff48ec70","gas":"0x5208","gasPrice":"0x3a35294400","hash":"0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc","input":"0x","nonce":"0x36","publicKey":"0x8b28169f589659658a0f43c59de1794964687e9cdd8d81f7acf0329a8f6566b3a983997d9a326ef2618977fe5ab92d48e2d5d0f44bb80e95725a0ba77d264749","r":"0xc1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3","raw":"0xf86d36853a352944008252089472d9e579f691d62aa7e0703840db6dd2fa9fae21880433a3d7251e64008081eca0c1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3a022dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","s":"0x22dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","standardV":"0x1","to":"0x72d9e579f691d62aa7e0703840db6dd2fa9fae21","transactionIndex":null,"v":"0xec","value":"0x433a3d7251e6400"},"id":1}.
2019-01-25 14:26:47 UTC jsonrpc-eventloop-0 TRACE rpc  Request: {"id":1,"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc"]}.
2019-01-25 14:26:47 UTC jsonrpc-eventloop-0 DEBUG rpc  Response: {"jsonrpc":"2.0","result":{"blockHash":null,"blockNumber":null,"chainId":"0x64","condition":null,"creates":null,"from":"0x9401dcf086e223763604f9cb826814ffff48ec70","gas":"0x5208","gasPrice":"0x3a35294400","hash":"0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc","input":"0x","nonce":"0x36","publicKey":"0x8b28169f589659658a0f43c59de1794964687e9cdd8d81f7acf0329a8f6566b3a983997d9a326ef2618977fe5ab92d48e2d5d0f44bb80e95725a0ba77d264749","r":"0xc1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3","raw":"0xf86d36853a352944008252089472d9e579f691d62aa7e0703840db6dd2fa9fae21880433a3d7251e64008081eca0c1e3bab98453d67e43983fd52209607c4abeae269f742d7569ebcabea225d5e3a022dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","s":"0x22dc7b5f0aa773095c50060d24787699e3900482ca2b99fb50a6e8d02d1d4c44","standardV":"0x1","to":"0x72d9e579f691d62aa7e0703840db6dd2fa9fae21","transactionIndex":null,"v":"0xec","value":"0x433a3d7251e6400"},"id":1}.

Trace level logging hasn’t revealed anything to me, the txid is valid and gets added to my nodes mempool but never verified. Are there any special conditions to transaction generation I should know about?

**igorbarinov** · January 25, 2019, 2:38pm

Could you please proved the tx and txid?

**ttk2** · January 25, 2019, 5:01pm

@igorbarinov I did post some logs form parity but they got spammed

**igorbarinov** · January 25, 2019, 2:46pm

thx, I marked it as a not spam

**andrewcravenho** · January 25, 2019, 3:23pm

I took a look at TX 0xf956402f91231e415c3529f551d1c2a199b81c0e22bcd5b9a7fb9ae31ddaa3f8 and can see that this transaction was replaced by TX 0x68173f2a6579bb2fc9107db6611d458166ba60cf0dd222d5119b90ebcbe0b058

#1370 is a pull request that we’ve been working on for BlockScout which will display when a transaction is dropped/replaced by another transaction. Currently dropped/replaced transactions look like they’re still in a pending state on BlockScout. This should go live later today.

**ttk2** · January 25, 2019, 5:01pm

The source address here is

0x9401dcf086e223763604f9cb826814ffff48ec70

and the txid is

0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc

which shows up as ‘page not found’ on block scout.

The transactions you linked seem to be for contract creation from a different address?

**andrewcravenho** · January 25, 2019, 3:44pm

Sorry, I was confused by a link that was sent to me regarding another issue.


nonce	54
gas_price	250000000000
gas	21000
action	Call(0x72d9e579f691d62aa7e0703840db6dd2fa9fae21)
value	302765744400000000
data
hash	0x1f3e2d3b90912e54ea2b28768899492c40703d8bcde1c411953704687cda40bc

Why did you use such a high nonce? This is the only issue I see with this transaction.

I’m not seeing anything picked up on my node for this transaction. I’m not sure why it wasn’t broadcasted.

**ttk2** · January 25, 2019, 5:01pm

This is a change that we made after running on prod eth, when you would go more than one payment interval without a transaction being placed into a block the get_nonce call would give you the value for a mempool transaction and that would cause issues, since we don’t want to replace the transaction but instead have both get into the same block and total to the payment we’re trying to send. So when we try and send a transaction we increment the nonce locally even if the blockchain doesn’t agree yet.

**andrewcravenho** · January 25, 2019, 3:56pm

Won’t you need to fill in the nonce gap in order for that transaction to broadcast?

**ttk2** · January 25, 2019, 5:01pm

Let me try that real quickly, I’ve seen conflicting info on that. It’s never really been an issue with Eth because we’ve never had more than one transaction out of order at a time, which resolves itself.

**andrewcravenho** · January 25, 2019, 4:00pm

I’ve experienced this previously. When the gap is filled in all the higher nonce transactions in the mempool are completed as well.

**igorbarinov** · January 25, 2019, 4:35pm

**igorbarinov** · January 25, 2019, 7:05pm

any news after your test?

**ttk2** · March 8, 2019, 12:26pm

It was an internal state problem based on xdai’s speed.

We had based our transaction generation time off of the stated transaction fee which was so low it lead to us generating a huge amount of transactions.

Some way or another one of those transactions was lost and after that we where stuck with an incorrect nonce.

I changed the logic to both override it’s local nonce with the full nodes stated value and to reduce the speed of transactions even with zero transaction fees.