Railway Domains Changed Unexpectedly | Buggy project 'Members' and Github Connection
Anonymous
FREEOP

a month ago

I am experiencing a critical issue with a project containing 3 services. All services show as 'online' in the dashboard, but their custom domains are returning 404s.

It appears the underlying Railway .up.railway.app domains changed unexpectedly without our input. Our DNS was previously configured to point to the original targets, which are now dead.

Here is the domain mismatch we are seeing:

  • app.89paint.com
    • Old/Current DNS Target:lc6sa51e.up.railway.app
    • New Target in Dashboard:acceptable-intuition-production.up.railway.app
  • admin.89paint.com
    • Old/Current DNS Target:1ec9cf34.up.railway.app
    • New Target in Dashboard:admin-portal-production-d207.up.railway.app
  • portal.89paint.com
    • Old/Current DNS Target:spsaeedb.up.railway.app
    • New Target in Dashboard:89-paint-client-portal-production.up.railway.app

Before I ask our IT admin to update the DNS records, can you help me understand why these underlying Railway domains changed in the first place? I want to ensure the project state is stable before we reroute production traffic.

Secondary Issue: Membership & Deployment Bug

We believe the project state might be glitched, which could be related to the domain changes.

  • In 'Project Settings' > 'Members', no members were listed, even though my peer and I were currently viewing the project.
    • My supervisor tried to re-promote/invite me. When I click the new invite link in my email, it says I am already a member and does nothing. I logged out to re-accept the invitation to the project, and now I have permissions
  • The repo says 'Github repo not found' under 'Branch connected to production' for each service.

Could you please look into the project state to see what caused the domain swap and help us restore our deployment permissions?

Also, this thread requires me to select a 'Service'. The only option I see is N/A.

Project ID: bfc24f18-5da9-4d61-a95c-1cf4fded9e12

$10 Bounty

3 Replies

Status changed to Open Railway about 1 month ago


Anonymous
FREEOP

a month ago

Project ID: bfc24f18-5da9-4d61-a95c-1cf4fded9e12


Anonymous
FREEOP

a month ago

hello? anything?


pokematpok
FREE

18 days ago

Hey there,

I noticed this has been sitting without a response for over 20 days, so I wanted to share my perspective on what might be happening, though I should note upfront that I'm not Railway staff and can't provide an official solution or detailed fix.

That said, you are right... this doesn't look like a typical DNS issue. The fact that all three Railway ingress domains changed simultaneously is the key indicator.

These domains are Railway-generated and typically remain stable unless there's been some kind of project-level reprovisioning, service recreation, environment migration, or internal re-association event.

The secondary symptoms you're experiencing strongly support this theory.

This pattern suggests the deployments themselves are probably healthy, but the project metadata and integrations may have become partially desynced.

Regarding the GitHub issue: There was indeed a recent GitHub incident involving app installation token authentication failures (manifesting as "Bad credentials" and "repository not authorized" errors). This could definitely explain the GitHub-related symptoms, missing repos and broken deployment connections. However, I'm not certain whether that alone would trigger Railway to regenerate all three ingress domains. My best guess is the GitHub outage may have contributed to or triggered a separate reprovisioning/reassociation event on Railway's side.

Why you won't find a public explanation: Railway generally doesn't publish detailed changelogs for low-level infrastructure or routing migrations unless there's a broader platform-wide incident. Because of this, nobody outside Railway staff can tell you the exact trigger with certainty. I checked the public changelog, Discord, and GitHub issues/discussions but didn't find anything matching your situation exactly.

Before updating DNS, verify project stability:

  • Test the new Railway URLs directly, you should confirm each app loads and functions correctly
  • Check the Domains section for each service, verify custom domains are still properly attached
  • Re-authorize the GitHub integration if repos are still showing as missing
  • Verify project members/roles are now visible and correct

Then update your DNS CNAMEs to point to the new Railway targets

If the new Railway domains work as expected, updating DNS should restore production traffic normally.

One final observation: The old domains were short random IDs, while the new ones follow Railway's newer human-readable naming convention. This suggests these services were re-issued new ingress endpoints during some kind of project-level migration or reprovisioning event, rather than being manually edited.

Some links that may help you:

https://www.githubstatus.com/incidents/k5z4d1v1tqmt & https://status.railway.com/incident/RMR32LTK - GitHub's and Railway's incident reports about errors with GitHub app installation token authentication, currently an active incident, but there may have been similar in the past

https://station.railway.com/questions/bad-credentials-error-despite-valid-gi-82f45582 - a very similar thread concerning the GitHub issue

Since your issue has been ongoing for 20+ days and involves project-level changes, you'll ultimately need Railway support to investigate the root cause, but hopefully this context helps you understand what might have happened and verify stability before updating DNS.

Best of luck getting everything back online!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...