grpc + http issue
0xmukesh
PROOP

a year ago

hey there! i'm facing a weird issue. i'll try my best to explain it. i've an application that's meant to run in the background 24/7 and processes some information. it opens a grpc request, processes that data, sleeps for a while, and does this in a loop. while processing the information, it also sends a few http requests to external servers to fetch some data.

i've tried running the code locally and it works fine — the grpc request is opened properly, the http requests are sent out properly, and overall the data is being processed as expected.

when i tried deploying this service on railway, the grpc request was opened properly, but the http requests weren't being sent out. i've attached a screenshot of a function that sends out http requests while processing the data.

i had added a few logs earlier (removed them for the sake of brevity), and from the logs, it looks like it doesn't get past the getTokenInfos function. it's a simple function that sends out a post request. i tried logging the response, but it wasn't visible in the railway logs. am i doing something wrong? i'm unable to figure out what's the exact issue because the logs don't tell anything i.e. no error logs.

$20 Bounty

41 Replies

0xmukesh
PROOP

a year ago

1391765672190808234


0xmukesh
PROOP

a year ago

1391765713345052822


0xmukesh
PROOP

a year ago

1391765824964006018


0xmukesh
PROOP

a year ago

bb48a25d-c29e-446e-9af8-60edf17779ba


0xmukesh
PROOP

a year ago

that's the project id. you can try re-deploying the latest deployment and notice that it acts the records to positions table but not ixns table, which might be because of the above mentioned issue


a year ago

You said that it sleeps after the grpc and http request. Are you waiting until the HTTP request is completed before sleeping?


a year ago

Which library are you using for grcp and http? node-http?


0xmukesh
PROOP

a year ago

yes, im doing it


0xmukesh
PROOP

a year ago


a year ago

So to clarify the grcp goes through but the http is what has 0 response hmmm


a year ago

Are you reaching out to a server that has strict cors or access policy?


0xmukesh
PROOP

a year ago

yup, correct. the server doesn't have any strict cors or access policy afaik. ive tried to run a simple curl request via sshing into the service to confirm that the sending a request manually doesn't have any issues


a year ago

And you are using node-http or axios


0xmukesh
PROOP

a year ago

im using ky library for sending http requests which uses fetch api under the hood


a year ago

And the catch block is never being triggered correct?


a year ago

On first start could you make a similar/dummy request with node-fetch? Would be interested if the ky library has some issues <:Hmmmmm:891565000068587551>


a year ago

The first thing that comes to my mind is await failing because of sleeps. But as you said thats not an issue


0xmukesh
PROOP

a year ago

yea, it's quite weird and unexpected behavior. ive faced it again with another service which is a script which just fetches a bunch of rows from a db and processes them in parallel. during processing, it sends requests to a few external api services (https://public-api.birdeye.so and https://mainnet.helius-rpc.com). it's a metal instance. it was just stuck for a while ~10 mins. there was no request timeout error in the logs as well (ive set the timeout to be 10s during experimentation). are those apis blocklisted or something?


a year ago

if when you ssh or connect to the container you are able to make a request to those endpoints they arent locked. I wonder if during processing something is changing which is just odd?


a year ago

yeah let me google around a bit


a year ago

what node version are you using?



0xmukesh
PROOP

a year ago

via ssh, it seems fine. that's what left me confused. the script is quite simple - just fetches some records from db, getting some metadata from external api services and updating the data in db.


a year ago

I'm still a bit sketchy about the ky thing. Are you able to temporarily swap to using axios/plain fetch and see?


0xmukesh
PROOP

a year ago

ill give it a shot but im not quite sure if that would have any effect because the code works fine locally.


a year ago

Yeah I'm not sure that would fix it either but its something to rule out. If that doesnt work I think its on railway core itself. I also dont know if this would do anything but I'm curious if enabling static IPs would do anything.

Generally confusing and I'm sorry. Wish I had a better answer on this but it seems like you are doing everything right on your end.


a year ago

Unless, you are using docker for deployments right?


a year ago

Well nevermind that wouldnt be an issue


0xmukesh
PROOP

a year ago

ah, fixed the issue. had to change the nodejs version. railway was using v18


0xmukesh
PROOP

a year ago

just had to change it to the current lts


0xmukesh
PROOP

a year ago

v20.18.1


0xmukesh
PROOP

a year ago

nonetheless, thank you!


a year ago

Oh wonderful yeah I had asked about that lol


a year ago

Hope I was able to help resolve this!


a year ago

Even if it was just poking at what it wasnt 😂


0xmukesh
PROOP

a year ago

oh, i totally missed that message!


a year ago

Oh no worries lol


a year ago

LTS is the best


0xmukesh
PROOP

a year ago

yup, it is! i was just trying to check which things where different from my local environment and the railway deployment and final went down till node.js version and fixed it


a year ago

Crazy that was the issue but good to know for the future. Will ref this thread if I see that issue again.

https://station.railway.com/questions/grpc-http-issue-168f8606


a year ago

Feel free to close the thread <:picassoface:1281722225593946123>


Welcome!

Sign in to your Railway account to join the conversation.

Loading...