Is this happening with the private network endpoint?

Nope, only using public endpoints because it is between different projects so private network cannot be used here.

Gotcha- unfortunately we don't control the public network endpoints since that is upstream from our vendor (GCP) we do plan to move this to become fully Railway managed but the responses you are getting are within tolerances.

I would consolidate the projects down and use the private network. It'll both be more reliable AND faster

Thank you for this suggestion. I will do it. Have a good day.

Hey, do you happen to have the request id of the network request that hung?

(We are trying to run an investigation to figure out what may have happened.)

Sorry but I don't have request ids but I have discovered that there are still timeouts

I have uptime kuma in the same project and I switched to use private network instead of public network and it looks like... some requests are "blocked" or timeouted

*the target service just works fine at the same time from browser perspective.

I'm attaching screenshots from uptime kuma and logs from service:

2024-12-06T06:57:49+01:00 [MONITOR] WARN: Monitor #5 'service': Pending: timeout of 48000ms exceeded | type HTTP

2024-12-06T06:59:37+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:01:25+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:04:14+01:00 [MONITOR] WARN: Monitor #5 'service': Pending: timeout of 48000ms exceeded

2024-12-06T07:07:02+01:00 [MONITOR] WARN: Monitor #5 'service': Pending: timeout of 48000ms exceeded

2024-12-06T07:08:50+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:10:38+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:12:26+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:14:14+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:16:02+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:17:50+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

2024-12-06T07:19:38+01:00 [MONITOR] WARN: Monitor #5 'service': Failing: timeout of 48000ms exceeded

Sorry for the potentially deaf line of questioning:

This was immediately after you moved to private network? If the issue is present in the private network, it's unlikely to be related (two different systems).

However, in the process of investigating this, we found that HTTP logs were dropping requests. Next time you see this, can you go to HTTP logs and give us a request_id? This way we can see the whole lifecycle of the packet.