a864dd2b-2b49-4eee-b5b2-627b3c084d43

I get this error when trying to access it externally

curl -X POST https://sonic-node-production.up.railway.app -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber"}'
-------
{"status":"error","code":502,"message":"Application failed to respond","request_id":"lkW-NHaVTzaLfux9H3rXOA_3500563535"}%

And a 403 when trying to access it from within the local network

When I test with my local docker container, it all seems to work correctly

Happy to provide more details if needed

is it listening on :: ?

Right now it's listening on

INFO [12-12|20:28:09.013] HTTP server started                      endpoint=[fd12:e16:9f4f:0:4000:a:9ded:4c28]:18545 auth=false prefix= cors= vhosts=localhost

Which is the resolved address for

--http --http.addr=lavish-liberation.railway.internal --ws --ws.addr=lavish-liberation.railway.internal

you don't want to listen on its IPv6 address

please have it listen on ::

I've just created a new one listening on ::

INFO [12-13|07:38:38.806] HTTP server started                      endpoint=[::]:18545 auth=false prefix= cors= vhosts=localhost

It has the exact same issue

hey now it's responding

now what host is it talking about

Thast's a good question

I initially had that issue, so I tried resolving to the internal network address

Not sure how relevant it is, but I'm setting the following env variables

RUN_CMD_ARGS=--http --http.addr=0.0.0.0 --ws --ws.addr=0.0.0.0

is there anything in the deploy logs that could help?

Nope, I see nothing in the logs when I get that error

If I look at the http logs, it shows a 403

so what even is sonic node?

It's an RPC node for a blockchain

by having it on railway it increases the speed of our operation by 5-6x (when it works <:kek:885396556520493056> )

I don't think our public proxy supports RPC

it's strictly http

But the RPC works over http, it's just a post request

does it? try the TCP proxy instead

on it

Tried both http and https, got error on both

curl -v -X POST https://junction.proxy.rlwy.net:29067 -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host junction.proxy.rlwy.net:29067 was resolved.
* IPv6: (none)
* IPv4: 35.214.222.137
*   Trying 35.214.222.137:29067...
* Connected to junction.proxy.rlwy.net (35.214.222.137) port 29067
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection
curl: (35) Recv failure: Connection reset by peer
❯ curl -v -X POST http://junction.proxy.rlwy.net:29067 -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber"}'
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host junction.proxy.rlwy.net:29067 was resolved.
* IPv6: (none)
* IPv4: 35.214.222.137
*   Trying 35.214.222.137:29067...
* Connected to junction.proxy.rlwy.net (35.214.222.137) port 29067
> POST / HTTP/1.1
> Host: junction.proxy.rlwy.net:29067
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 51
>
* upload completely sent off: 51 bytes
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

the TCP proxy is pointing to the correct port right?

As far as I understand I send the RPC requests over HTTP POST or websocket with the payload as part of the body

yes

hmm well I'm not sure what to tell you, I'm not familiar with the software

Not sure what else to try myself, I have it running locally without issues, I set it up on a VPS last night to test and it also runs just fine

Exact same setup as on Railway

what would you typically see when you open the domain in the browser?

nothing

The only way to see if it's working is by sending a post request with a simple method

This would be the simplest one

curl -X POST http://localhost:18545 -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber"}'

This is what the result looks like

❯ curl -X POST http://localhost:18545 -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber"}'
{"jsonrpc":"2.0","id":0,"result":"0x583f7"}

maybe it was never designed to work behind a proxy

Maybe

It's one source software, I'm happy to review the source if needed

What would be a pointer that it's not designed to work behind a proxy?

In case you're curious, this is the source https://github.com/Fantom-foundation/Sonic/tree/v2.0.1

find out what triggers "invalid host specified" to be returned, and don't return that error lol

<:kek:885396556520493056>

Alright, thank you for your time

i know we for sure pass through the Host value unmodified

Knowing that it's not a configuration error is a good start

I'll post my findings here

sounds good, wish I could have helped more!

@Brody Found a solution, turns out I had to enable a vhosts flag https://github.com/ethereum/go-ethereum/issues/16526#issuecomment-992480651

It's working now <:pog:925010410518749194>

Another question on the topic, is there any way to gracefully shutdown a deployment? When making changes and re-deploying, it seems to just kill the container which corrupts the databse and makes it so we have to wipe the volume and re-index the entire blockchain, which is ~2h

It's not the end of the world since we can just keep two deployments open at all times and use one of them as a failover

but not idea 😄

old deployments are always gracefully shut down.

when the system goes to remove a deployment it is sent sigterm and then waits 3 seconds before force killing, is that 3 seconds not enough?

I don't really see it gracfully shutting down

the application would need to properly handle sigterm

you are using a volume right? and making sure data is actually being written to said volume

Looks like it's an implementation issue, when shutting down through a ctrl + c on the console it does gracefully shutdown, but I can't seem to be able to do the same on Docker

yeah we don't send ctrl + c haha, sigterm is a standardized way to tell an app to shut down

I know, I just expected both to send the same signal

Feel free to close the ticket btw, thanks for your help!

this is what it'd look like

yeah it's probably just listening in the wrong signal, sounds like an easy pr

yeah ctrl + c is sigint

looks like an easy PR indeed

I don't know much go but that looks like it should listen to SIGTERMs too

I know go and it looks like that will do what you need to

as long as 3 seconds is enough, but if it's not you can always up that time via railway

It should be more than enough, takes like under 100ms locally

Just in case, what's the parameter to up that in Railway?

perfect

RAILWAY_DEPLOYMENT_DRAINING_SECONDS

Thanks!

Tweaking this worked, skill issue on my end as per usual

happy to hear that

!s