Can you go more in depth on what you mean when you say it can't pick it up?

ah gotcha, let me see if I can capture the logs and some pics

Does your base environment use reference variables?

ah yess, we have an environment in one service that sets SCOUT_DATABASE_URL=${{Postgres.DATABASE_URL}}?schema=scout_objs

most of the time this works, but occasionally, we noticed our app just not connecting to it on some rainy day, so we switch to SCOUT_DATABASE_URL=${{Postgres.DATABASE_PUBLIC_URL}}?schema=scout_objs

Does the reference resolve? or is this just purely an application level issue?

the DATABASE_URL resolves to postgresql://postgres:...@postgres.railway.internal:5432/railway and the application usually picks it up and does its thing

but on rare occasions, our nextjs server no longer wants to connect to it

so I would have to change it to the public URL for it to pick it up

although I am testing that environment again with the internal URL and it's working fine again <:monkaS:819507963915796490>

so it's a really intermittent heisenbug

Do you have the errors from the times when it won't connect?

let me see if I can capture one on my work machine

oh at least the old deployments still have it

this is all I have I think 💀

Haha, I think you know that error is extraordinarily vague and wouldn't be helpful here.

going to pick a rainy day and see if I can reproduce it with some network traces

i'll ask my colleague to add some sidecars

Sounds good, I'll be here when you have more information!

https://github.com/monotykamary/railway-network-sidecar
I've added a sidecar on my colleague's project to debug whether there was intermittent connection issues on railway

what I found was it had nothing to do with railway

and everything to do with alpine's musl <:facepalm:580292052525383690>

even after all these years it's still flaky

but thankfully nothing on railway's side

sorry for the trouble <:chemat:823803293310386216>

Oh, tell me more about how you came to that conclusion, and what about Alpine's musl was causing issues?

ah, I noticed that after several days the network sidecar ran to no issues; resolved the internal DNS just fine and gave perfect reliability

so I looked into what was so different with our webapp container (since it was the only one having network issues)

turns out it was using alpine and I remembered faintly this was a common problem during the before coredns k8s era

switching out alpine for debian-slim solved it immediately

likely a dns over tcp issue (I think)?

Well, now I am here wondering if this was Alpine, or the recent issue with IPv4 private networking that we found that would affect a very small subset of hosts (and have since fixed).

oh it might have cascaded to alpine in that sense

Yeah, any request made over IPv4 would time out, while IPv6 worked fine.

If only Prima gave a better error message, you could have been the first user to find the IPv4 issue, haha.

Either way, I'm happy you are in a good place now, and that was a very nice solution with your Zig sidecar.

likewise, thanks for the support as always

Happy to help!