App and Website Not Loading for Users
prodcontroller
PROOP

22 days ago

As of this morning our app is no longer loading for users and our website is not as well.

"The web server reported a bad gateway error"

prodcontrol-production.up.railway.app

Screenshot_2026-05-22_at_10.07.41_AM.png

Attachments

Solved

5 Replies

22 days ago

Looking now, can you drop a link to your project page on Railway- will help me check it out a bit faster



22 days ago

Hey! I think I found the issue.

Looks like this has been ongoing long before the most recent incident; I can see in your metrics the service has been 500ing for quite awhile.

I triggered a redeploy and that should in theory bring you back into a good state. Will monitor until the deploy is done.


22 days ago

Hey, post re-deploy this is back up.

Let me know if you need help with anything else at all.


Status changed to Solved mykal 22 days ago


mykal

Hey, post re-deploy this is back up. Let me know if you need help with anything else at all.

prodcontroller
PROOP

22 days ago

Thank you Mykal!

"This issue was laying dormant in the code since those migrations were written in April. It presented itself today due to a combination of how Railway runs containers and how our migration history is tracked:

How it stayed hidden:

The server only attempts to execute a migration file if its filename is not present in the _migrations database table.

When the migrations were originally pushed, they might have been skipped in production because the base tables already existed or they were manually marked as applied in the PostgreSQL database.

What triggered it today:

Container Restart: Hosting platforms like Railway routinely recycle, migrate, or restart application containers for maintenance, resource reallocation, or if the process hits memory thresholds.

Migration Evaluation: During today's container boot sequence, the NodeJS process ran its startup database hook (initializeDatabase()). For some reason (such as a database connection reconnect, a schema synchronization check, a minor database restore, or a temporary replica sync lag), the migration engine attempted to evaluate or run migration 094 or 098 against the PostgreSQL database.

Syntax Exception: Because PostgreSQL does not understand SQLite-specific functions like randomblob or INSERT OR IGNORE, the query failed immediately. This database crash on startup prevented the server from listening on its port, putting it into an infinite boot-loop."


Status changed to Awaiting Railway Response Railway 22 days ago


Status changed to Solved Railway 22 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...