Service Crash
slavcetrajkovski
PROOP

4 months ago

I just recently pushed a new small change and the deployment crashed. I reverted the changes and again the deployment crashed. I am reading the logs and it seems to be a problem with the connection with my database. This is weird since I did not apply any big changes to my code, and the database is the same. Everything worked correctly about 30 minutes ago.

Are some railway services down or?

$40 Bounty

15 Replies

Railway
BOT

4 months ago

Hey there! We've found the following might help you get unblocked faster:

If you find the answer from one of these, please let us know by solving the thread!


4 months ago

Hello,

We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.


slavcetrajkovski
PROOP

4 months ago

Well as I mentioned I have not made any changes, why would my application crash suddenly?


Status changed to Awaiting User Response Railway 4 months ago


brody

Hello,We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.

slavcetrajkovski
PROOP

4 months ago

Well as I mentioned I have not made any changes, why would my application crash suddenly?


Status changed to Awaiting Railway Response Railway 4 months ago


brody

Hello,We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.

slavcetrajkovski
PROOP

4 months ago

Even when I am redeploying older versions the service crashes, so it is not functional at all


brody

Hello,We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.

slavcetrajkovski
PROOP

4 months ago

just checked brody, I get an infinite loop on my database connection

Attachments


brody

Hello,We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.

slavcetrajkovski
PROOP

4 months ago

well the infinite loading stoped i can access the database, but my deployment is still crashing. i have made no changes to the database nor to the backend service. i am trying older versions and it still crashing. I have the correct credentials i checked but the connection still fails


brody

Hello,We have an ongoing incident, but it does not relate to deployment workloads or network availability in any way.

slavcetrajkovski
PROOP

4 months ago

can somebody look into this???


4 months ago

Hmm,
Can you link that database? I just pinged the postgres one you had and it returned healthy. All loads in the UI as well.


Status changed to Awaiting User Response Railway 4 months ago


slavcetrajkovski
PROOP

3 months ago

I had to delete the database to make it work, which is not what I want to do

I have them same problem now. How do I link the database?


Status changed to Awaiting Railway Response Railway 3 months ago


noahd

Hmm, Can you link that database? I just pinged the postgres one you had and it returned healthy. All loads in the UI as well.

slavcetrajkovski
PROOP

3 months ago

I literally don't understand, why it happens, I am also trying to connect to the database from my local project and I can't do it I triple checked the credentials and I have the correct ones.


slavcetrajkovski
PROOP

3 months ago

I works with the public url but not with the internal configuration


3 months ago

This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.

Status changed to Open brody 3 months ago


bytekeim
PRO

3 months ago

hy,

if you want an immediate solution , since your internal networking suddenly broke, temporarily use the public URL to get your service back online:

1 - in your backend service variables, change from DATABASE_URL (private network) to DATABASE_PUBLIC_URL (TCP proxy)

2- if you're manually constructing the connection string, use the TCP proxy host instead of postgres.railway.internal

3-redeploy

the public URL will get you online immediately while we diagnose the private networking. we should search for the root cause , it may be an Alpine Linux issue - If using Alpine-based container, add this variable:

   ENABLE_ALPINE_PRIVATE_NETWORKING=true

or dns timing - railway's private dns needs ~3 seconds. update your start command:

   sleep 3 && your-start-command

the last thing, builder issue - check service settings → Builder is set to Nixpacks (not legacy builder)

lmk if it works, if not tell me which framework/language you're using, and if you are using an Alpine-based Docker image? and I can help troubleshoot further


bytekeim

hy,if you want an immediate solution , since your internal networking suddenly broke, temporarily use the public URL to get your service back online:1 - in your backend service variables, change from DATABASE_URL (private network) to DATABASE_PUBLIC_URL (TCP proxy)2- if you're manually constructing the connection string, use the TCP proxy host instead of postgres.railway.internal3-redeploythe public URL will get you online immediately while we diagnose the private networking. we should search for the root cause , it may be an Alpine Linux issue - If using Alpine-based container, add this variable:ENABLE_ALPINE_PRIVATE_NETWORKING=trueor dns timing - railway's private dns needs ~3 seconds. update your start command:sleep 3 && your-start-commandthe last thing, builder issue - check service settings → Builder is set to Nixpacks (not legacy builder)lmk if it works, if not tell me which framework/language you're using, and if you are using an Alpine-based Docker image? and I can help troubleshoot further

slavcetrajkovski
PROOP

3 months ago

Yes I already switched to the public connection string as you mentioned. Buy my app was experiencing really slow queries. I am using Spring Boot with Next.js, and I am using the standard configuration that Railway provides for the Docker containers.

But funny enough I just now tried again to revert back to the internal connection of the database, and now it works, but I don’t know why and how.


bytekeim
PRO

3 months ago

hy again!

glad it's working again! here's what i tjink is happening and how to prevent it:

it broke then fixed itself because of DNS propagation delay. when services restart, the internal DNS (postgres.railway.internal) takes time to propagate. Your app was trying to connect before DNS resolved, causing failures. After 30 minutes, dns propagated and it started working.

and the public URL was slow because Public connection routes through TCP proxy = added latency. Internal networking is direct = much faster.

to prevent this from happening again:

add a startup delay to your Docker start command in Railway:

sleep 5 && java -jar your-app.jar

this ensures DNS is ready before Spring Boot attempts connections.

optionnaly you can also optimize your connection pool: add to application.properties:

spring.datasource.hikari.maximum-pool-size=5
spring.datasource.hikari.connection-timeout=30000

stay on internal networking (DATABASE_URL) for best performance and the sleep command should prevent future issues


Loading...