a month ago
Hello Support,
Following the incident that just happen today I have noticed my application has chnage as a login field is now missing.
Project Name - DobreTech LMS
Environment - Production
When I go to https://dobrelms.com/login there should be a first field which is a drop down for institution and users are to select before login but it it is now missing.
The login was supposed to be same as Dev https://dev.hippo-h.com/login
All was working fine before the incident and this lowers my confidence of Railway.
4 Replies
a month ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 1 month ago
a month ago
After reviewing the issue, this does not appear to be a random UI change from the application itself, but rather a possible deployment/configuration inconsistency caused after the incident.
Based on the behavior described, the most likely causes are:
the production deployment rolled back or started serving an older cached build
environment variables related to institutions/authentication were lost or not injected correctly
frontend static assets/build cache became inconsistent after the platform incident
the application connected to a different database/environment unexpectedly
a failed deployment partially updated the application
CDN/cache invalidation issue causing outdated frontend files to be served
Since the Dev environment still displays the institution dropdown correctly, the production environment is clearly not behaving the same way anymore.
Recommended checks from Railway/platform side:
verify whether Production is running the correct/latest deployment image
verify deployment rollback history after the incident
verify environment variables consistency between Dev and Production
verify no stale build cache or CDN cache is being served
verify if the application was attached to a different database instance after the outage
inspect logs during the incident for failed deployments or runtime restarts
From the application side, comparing the exact build hash/version running in Dev vs Production would also help identify whether Production is serving an outdated build.
This issue started immediately after the Railway incident and was not present before, which strongly suggests the platform instability may have caused deployment or runtime inconsistency.
Please investigate the deployment/runtime state for this project and confirm whether any rollback, cache corruption, failed release, or infrastructure issue occurred during the incident.
marcelo2212
After reviewing the issue, this does not appear to be a random UI change from the application itself, but rather a possible deployment/configuration inconsistency caused after the incident. Based on the behavior described, the most likely causes are: the production deployment rolled back or started serving an older cached build environment variables related to institutions/authentication were lost or not injected correctly frontend static assets/build cache became inconsistent after the platform incident the application connected to a different database/environment unexpectedly a failed deployment partially updated the application CDN/cache invalidation issue causing outdated frontend files to be served Since the Dev environment still displays the institution dropdown correctly, the production environment is clearly not behaving the same way anymore. Recommended checks from Railway/platform side: verify whether Production is running the correct/latest deployment image verify deployment rollback history after the incident verify environment variables consistency between Dev and Production verify no stale build cache or CDN cache is being served verify if the application was attached to a different database instance after the outage inspect logs during the incident for failed deployments or runtime restarts From the application side, comparing the exact build hash/version running in Dev vs Production would also help identify whether Production is serving an outdated build. This issue started immediately after the Railway incident and was not present before, which strongly suggests the platform instability may have caused deployment or runtime inconsistency. Please investigate the deployment/runtime state for this project and confirm whether any rollback, cache corruption, failed release, or infrastructure issue occurred during the incident.
a month ago
Thanks Marcelo,
I have checked and the latest deployment is from 2 weeks ago
No new logs since May 19, 2026, 9:54:52 PM CDT
Got the below for database deployment logs
2026-05-19 22:22:04.132 UTC [3] LOG: database system is shut down
2026-05-19 22:22:04.082 UTC [3] LOG: received fast shutdown request
2026-05-19 22:22:04.082 UTC [82] LOG: shutting down
2026-05-19 22:22:04.082 UTC [87] ERROR: canceling statement due to user request
2026-05-19 22:22:04.086 UTC [82] LOG: checkpoint starting: shutdown immediate
2026-05-19 22:22:04.086 UTC [3] LOG: aborting any active transactions
2026-05-19 22:22:04.086 UTC [204942] FATAL: terminating connection due to administrator command
2026-05-19 22:22:04.087 UTC [3] LOG: background worker "logical replication launcher" (PID 87) exited with exit code 1
Railway AI gave me this
The Postgres image (postgres-ssl:18) has a corrupted init process. This is a Railway infrastructure issue from the incident. You need to redeploy Postgres to get a fresh container:
Very frustrating right now
a month ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Apologies for this canned message but in an effort to help all our customers get back up and running, we are sending this bulk message. As you may know, we had a major interruption to our services yesterday. We've published a post-mortem if you'd like more information on the incident. It describes what happened and what we are doing to prevent it in the future. We are deeply sorry for the impact that it has had on you.
It is taking some time to bring everything back up, but we are working on it as fast as we can. In general, a redeployment should fix most service issues. Due to the volume of customers redeploying right now, builds and deploys may take longer than normal to process.
You can track recovery status here: https://status.railway.com/incident/KVZ1Z8GY
If you are still having other issues that might be related to the incident you can read more here: https://station.railway.com/community/road-to-recovery-post-gcp-outage-builds-d362e48c
Feel free to respond if your question has not been addressed.
a month ago
This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!
Status changed to Solved Railway • about 1 month ago