grpc + http issue
0xmukesh
PROOP

7 months ago

hey there! i'm facing a weird issue. i'll try my best to explain it. i've an application that's meant to run in the background 24/7 and processes some information. it opens a grpc request, processes that data, sleeps for a while, and does this in a loop. while processing the information, it also sends a few http requests to external servers to fetch some data.

i've tried running the code locally and it works fine — the grpc request is opened properly, the http requests are sent out properly, and overall the data is being processed as expected.

when i tried deploying this service on railway, the grpc request was opened properly, but the http requests weren't being sent out. i've attached a screenshot of a function that sends out http requests while processing the data.

i had added a few logs earlier (removed them for the sake of brevity), and from the logs, it looks like it doesn't get past the getTokenInfos function. it's a simple function that sends out a post request. i tried logging the response, but it wasn't visible in the railway logs. am i doing something wrong? i'm unable to figure out what's the exact issue because the logs don't tell anything i.e. no error logs.

$20 Bounty

41 Replies

0xmukesh
PROOP

7 months ago

1391765672190808300


0xmukesh
PROOP

7 months ago

1391765713345053000


0xmukesh
PROOP

7 months ago

1391765824964006100


0xmukesh
PROOP

7 months ago

bb48a25d-c29e-446e-9af8-60edf17779ba


0xmukesh
PROOP

7 months ago

that's the project id. you can try re-deploying the latest deployment and notice that it acts the records to positions table but not ixns table, which might be because of the above mentioned issue


noahd
EMPLOYEE

7 months ago

You said that it sleeps after the grpc and http request. Are you waiting until the HTTP request is completed before sleeping?


noahd
EMPLOYEE

7 months ago

Which library are you using for grcp and http? node-http?


0xmukesh
PROOP

7 months ago

yes, im doing it


0xmukesh
PROOP

7 months ago


noahd
EMPLOYEE

7 months ago

So to clarify the grcp goes through but the http is what has 0 response hmmm


noahd
EMPLOYEE

7 months ago

Are you reaching out to a server that has strict cors or access policy?


0xmukesh
PROOP

7 months ago

yup, correct. the server doesn't have any strict cors or access policy afaik. ive tried to run a simple curl request via sshing into the service to confirm that the sending a request manually doesn't have any issues


noahd
EMPLOYEE

7 months ago

And you are using node-http or axios


0xmukesh
PROOP

7 months ago

im using ky library for sending http requests which uses fetch api under the hood


noahd
EMPLOYEE

7 months ago

And the catch block is never being triggered correct?


noahd
EMPLOYEE

7 months ago

On first start could you make a similar/dummy request with node-fetch? Would be interested if the ky library has some issues <:Hmmmmm:891565000068587551>


noahd
EMPLOYEE

7 months ago

The first thing that comes to my mind is await failing because of sleeps. But as you said thats not an issue


0xmukesh
PROOP

7 months ago

yea, it's quite weird and unexpected behavior. ive faced it again with another service which is a script which just fetches a bunch of rows from a db and processes them in parallel. during processing, it sends requests to a few external api services (https://public-api.birdeye.so and https://mainnet.helius-rpc.com). it's a metal instance. it was just stuck for a while ~10 mins. there was no request timeout error in the logs as well (ive set the timeout to be 10s during experimentation). are those apis blocklisted or something?


noahd
EMPLOYEE

7 months ago

if when you ssh or connect to the container you are able to make a request to those endpoints they arent locked. I wonder if during processing something is changing which is just odd?


noahd
EMPLOYEE

7 months ago

yeah let me google around a bit


noahd
EMPLOYEE

7 months ago

what node version are you using?



0xmukesh
PROOP

7 months ago

via ssh, it seems fine. that's what left me confused. the script is quite simple - just fetches some records from db, getting some metadata from external api services and updating the data in db.


noahd
EMPLOYEE

7 months ago

I'm still a bit sketchy about the ky thing. Are you able to temporarily swap to using axios/plain fetch and see?


0xmukesh
PROOP

7 months ago

ill give it a shot but im not quite sure if that would have any effect because the code works fine locally.


noahd
EMPLOYEE

7 months ago

Yeah I'm not sure that would fix it either but its something to rule out. If that doesnt work I think its on railway core itself. I also dont know if this would do anything but I'm curious if enabling static IPs would do anything.
Generally confusing and I'm sorry. Wish I had a better answer on this but it seems like you are doing everything right on your end.


noahd
EMPLOYEE

7 months ago

Unless, you are using docker for deployments right?


noahd
EMPLOYEE

7 months ago

Well nevermind that wouldnt be an issue


0xmukesh
PROOP

7 months ago

ah, fixed the issue. had to change the nodejs version. railway was using v18


0xmukesh
PROOP

7 months ago

just had to change it to the current lts


0xmukesh
PROOP

7 months ago

v20.18.1


0xmukesh
PROOP

7 months ago

nonetheless, thank you!


noahd
EMPLOYEE

7 months ago

Oh wonderful yeah I had asked about that lol


noahd
EMPLOYEE

7 months ago

Hope I was able to help resolve this!


noahd
EMPLOYEE

7 months ago

Even if it was just poking at what it wasnt 😂


0xmukesh
PROOP

7 months ago

oh, i totally missed that message!


noahd
EMPLOYEE

7 months ago

Oh no worries lol


noahd
EMPLOYEE

7 months ago

LTS is the best


0xmukesh
PROOP

7 months ago

yup, it is! i was just trying to check which things where different from my local environment and the railway deployment and final went down till node.js version and fixed it


noahd
EMPLOYEE

7 months ago

Crazy that was the issue but good to know for the future. Will ref this thread if I see that issue again.
https://station.railway.com/questions/grpc-http-issue-168f8606


noahd
EMPLOYEE

7 months ago

Feel free to close the thread <:picassoface:1281722225593946123>


Loading...