Project ID: aabf52ba-51a4-493c-a24b-f2354c48f9bd
Deployment ID: f21a6a59-54c0-4b4b-aacc-81db87431cfe

As I was writing this, I received another EHOSTUNREACH error to the same IP address. I jinxed it, I guess!

the EHOSTUNREACH error could be due to a blip in GCP's networking, but the DNS error for the internal redis domain is unfortunately all too common, yet it's fixable, let me ask some questions about that -
nixpacks or Dockerfile?
do you have a 3 second sleep before starting your app?

I'm using nixpacks (railway up) for deploying the code, and I don't have a sleep in my start command currently but I can certainly add one

yes a 3 second sleep is definitely needed as the private network's DNS resolver is not available for the first ~3 seconds upon start

Perfect, I'll set that up now. Nothing to be done about the EHOSTUNREACH error I assume - I started noticing that pop up for the first time a week or so ago, which I think was when that GCP outage happened; I feel like it's gotten more common, but that's probably just hearsay :)

Well… nothing to be done other than adding some more robust error handling to my little hobby project! Ha

yep unfortunately railway is at the mercy of gcp until they finish their move to bare metal

Understood. Thanks very much for the quck response!

no problem, let me know if that 3 second sleep works

and make sure that you're using a readiness style health check so that railway doesn't switch over traffic while your service is sleeping for those 3 seconds

Three second sleep worked great @Brody , thanks for the tip. I've had one more weird error pop up today that's not making sense - could also be a GCP networking issue? I've now started getting an ETIMEDOUT error on my Redis connection. I'm making a call to redis at least once per minute, so I'm not seeing any reason on the code side why suddenly it's timing out.

Error Log

Again, will implement some error handling to try re-opening the connection when I run into this, but I'm curious why it might be happening to begin with

and you're getting that error from the redis database that you are connecting to via the private network?

Correct, yeah

In this case, it was stable for 10 minutes before that error, so 9-10 successful per-minute checks

is redis running into any resource limitations? disk, mem, etc

I'd be shocked if so, it has a whole one key in it currently XD

are you using ioredis?

I'm using node-redis currently

Can definitely give ioredis a try though if that's a better option!

it's got its own issues, are you using an older version of node-redis?

4.6.14, current version

Then I'm currently stumped, would it be too much to ask you to switch to dragonfly just for a test instead?

I've been getting internal dns errors to local MySQL the last 2 days, they disappear as quickly as they appear (usually overnight uk tine)
Sometimes error is silent - loss of front end but nothing in logs until a restart then the errors appear but resolve after minutes.

please share the specific error

loss of front end but nothing in logs until a restart

without errors there's not too much we can help with

I also had similiar errors, I can copy paste my erros logs here if needed

yes please

MySQL DB error :

1244665248208785408

MySQL DB Error :

1244665526748315771

is the database accessible locally

Any API Error at the same time :
Exception in thread "Timer-0" java.net.NoRouteToHostException: No route to host

at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)

at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:50)

at io.ktor.network.sockets.SocketImpl$connect$1.invokeSuspend(SocketImpl.kt)

at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)

at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)

at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)

at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)

at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

I changed the server and it's now accessible locally

I went from USA to EUW

oh you're pro?

Yes

fixed your badges

i tried a couple of times but it never changed

are you getting this error intermittently or only during the start of your app

always at the start of the app if i use the local url for the db, intermittently with the public URL

You have to wait a few seconds for local DNS to come up sometimes

can you try connecting through the private network and add a 3 second sleep before starting your app

I will switch back to the US West (Oregon, USA) to retry

Same error without the 3 second sleep in US West (Oregon, USA), the build with the sleep time is still deploying

Okey it worked with the 3 second sleep

perfect

It didn't need the 3 second sleep in EUW tho

you need the 3 second sleep everywhere, even if you don't immediately see an issue

I removed the auto request retry with the NoRouteToHostException so I can have the alert and report back to you if I have that error again

sounds good

The error is still occuring with API calls :
Exception in thread "Timer-2" java.net.NoRouteToHostException: No route to host

at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)

at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:50)

at io.ktor.network.sockets.SocketImpl$connect$1.invokeSuspend(SocketImpl.kt)

at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)

at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)

at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)

at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)

at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

what domain are you getting that error from

https://www.furious-squad.com/
but before I use the private link, both the db communication and the api call were throwing errors

you get errors when you try to call that domain?

I never had that specific error before

please answer the question to the best of your ability

No, except that one

I'm sorry but that doesn't answer the question

I didn't understand your question then

I keep having this error once or twice per hour

HttpClient: REQUEST https://mprez.furious-squad.com/api/v2/project/?query= failed with exception: java.net.NoRouteToHostException: No route to host

Exception in thread "Timer-4" java.net.NoRouteToHostException: No route to host

at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)

at io.ktor.network.sockets.SocketImpl.connect$ktor_network(SocketImpl.kt:50)

at io.ktor.network.sockets.SocketImpl$connect$1.invokeSuspend(SocketImpl.kt)

at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)

at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)

at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)

at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:100)

at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)

at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)