Invalid utf-8 in some RPC responses from https://dai.poa.network/

#1

Hi, I’m a developer for althea.org. We have a simple light client implementation in Rust that runs on wifi routers that pay each other for bandwidth in a decentralized network.

I’ve tested and successfully used this implementation with Geth, Parity, and Infura, Eth RPC implementations in various combinations of load balancing, cdn, and ssl setups.

But when attempting to get the balance of non-zero-balance addresses using eth_getBalance dai.poa.network returns invalid utf-8.

[2019-01-24T13:42:05Z TRACE web3::jsonrpc::client] web3 response headers 
ClientResponse HTTP/1.1 200 OK
  headers:
    "date": "Thu, 24 Jan 2019 13:42:05 GMT"
    "content-type": "application/json; charset=utf-8"
    "transfer-encoding": "chunked"
    "connection": "keep-alive"
    "set-cookie": "__cfduid=de0a7d348367e79791b441850834950241548337325; expires=Fri, 24-Jan-20 13:42:05 GMT; path=/; domain=.poa.network; HttpOnly; Secure"
    "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\""
    "server": "cloudflare"
    "cf-ray": "49e2ebdb8ad39224-EWR"
    "content-encoding": "br"

[2019-01-24T13:42:05Z TRACE actix_web::server::h1decoder] Chunked read, remaining=59
[2019-01-24T13:42:05Z TRACE actix_web::server::h1decoder] End of chunked stream
[2019-01-24T13:42:05Z TRACE web3::jsonrpc::client] Raw web3 encoded bytes [33, 212, 0, 0, 32, 214, 206, 221, 50, 222, 44, 29, 122, 229, 183, 241, 125, 202, 38, 147, 18, 112, 200, 1, 243, 15, 24, 97, 0, 216, 106, 193, 242, 62, 121, 104, 235, 158, 61, 160, 40, 137, 136, 240, 96, 242, 182, 73, 49, 233, 94, 109, 53, 133, 94, 14, 205, 63, 12]
[2019-01-24T13:42:05Z TRACE web3::jsonrpc::client] Raw web3 response "!�\u{0}\u{0} ���2�,\u{1d}z��}�&�\u{12}p�\u{1}�\u{f}\u{18}a\u{0}�j��>yh�=�(���`�I1�^m5�^\u{e}�?\u{c}"
[2019-01-24T13:42:05Z WARN  web3::jsonrpc::client] Failed to deserialize JSON from slice
[2019-01-24T13:42:05Z WARN  rita::rita_common::oracle] Balance request failed with Error("expected value", line: 1, column: 1)

Here’s the equivalent curl command

curl -v -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBalance","params":["0x9401dcf086e223763604f9cb826814ffff48ec70", "latest"],"id":1}' https://dai.poa.network
Note: Unnecessary use of -X or --request, POST is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700:10::6817:72a8...
* TCP_NODELAY set
*   Trying 104.23.115.168...
* TCP_NODELAY set
* Connected to dai.poa.network (104.23.115.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3858 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [115 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=ssl371114.cloudflaressl.com
*  start date: Jan 15 00:00:00 2019 GMT
*  expire date: Jul 24 23:59:59 2019 GMT
*  subjectAltName: host "dai.poa.network" matched cert's "*.poa.network"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55bfe775c270)
} [5 bytes data]
> POST / HTTP/2
> Host: dai.poa.network
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 116
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
< HTTP/2 200 
< date: Wed, 23 Jan 2019 23:04:51 GMT
< content-type: application/json; charset=utf-8
< content-length: 54
< set-cookie: __cfduid=d94b42adf1c8349009068da4f5e10af101548284691; expires=Thu, 23-Jan-20 23:04:51 GMT; path=/; domain=.poa.network; HttpOnly; Secure
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 49dde6d70946c172-IAD
< 
{ [54 bytes data]

100   170  100    54  100   116    169    363 --:--:-- --:--:-- --:--:--   532
* Connection #0 to host dai.poa.network left intact
{"jsonrpc":"2.0","result":"0x6f05b59d3b20000","id":1}

And a comparison between verbose curl executions between dai.poa.network and our own Parity based Eth-full node cluster.

Since curl parses the response just fine I’m unsure if we’re seeing a different response based on headers or if there’s some low level encoding correction going on here. Regardless this issue makes the POA RPC unusable for us.

1 Like
#2

Hey Justin, just wanted to check, you said that this is somehow possibly Cloudflare related?

#3

That’s the conclusion I came to, if dai.poa.network is backed by Parity 2.3.0 exactly like our own cluster. So it’s clearly not a RPC server level issue.

The only other obvious place to look is Cloudflare which has a history for doing strange stuff that works great in browsers but causes problems in scripts.

See most of the OpenWRT infrastructure which just dies if you try to use links behind cloudflare.

#4

Hi ttk2,
Thanks for the detailed issue.

I don’t see the problem on my side when I query the endpoint with curl on my MacOS Mojave 10.14.2.

curl -X POST --data ‘{“jsonrpc”:“2.0”,“method”:“eth_getBalance”,“params”:[“0x032d15c4372ad18e4e25ab51e3cc1519bf2796c3”, “latest”],“id”:1}’ https://dai.poa.network
{“jsonrpc”:“2.0”,“result”:“0x2c62949fbe4e800”,“id”:1}

Can you try it on the node where you run your light client?

Cheers

#6

@igorbarinov I think I’ve got this figured out, apparently http/2 support for the Rust client software is not quite finished.

The primary issue is that other CDN’s allow a downgrade but Cloudflare goes forward with http/2 and we get this junk message, which is not a utf-8 server response but the start of an attempted http/2 connection negotiation which is being forced on.

I’ll work on fixing this in the library side.

1 Like
#7

I thought the same looking into headers difff.
We can create one more endpoint without http/2 if it helps.

#8

Eh setting up my own load balanced node cluster was quite easy, it seems my real remaining issue is that my transactions are somehow considered malformed by the network and are accepted by my own nodes but not placed into blocks. See the Parity logs I posted in the other thread. You may have to go to the moderating queue to get them as automod spammed it.

1 Like
listed #10
#11

What did you use for your load-balancing? Interested in your setup.

1 Like
#12

It’s a pretty simple dns load balancing solution, nothing fancy.

1 Like