Service ID: 8f89edd8-3299-4873-a176-3e5f9d9b6054 (Cashem-App, production)

alexbeckman83

PROOP

2 months ago

Issue: A stale service domain cashem-app-production-f726.up.railway.app is stuck in the service's networking config and cannot be removed. Setting it to null via the API/Agent does not delete the key — it persists as a null value, which the edge interprets as a broken domain and refuses to route any traffic.

Impact: Total outage. The container is healthy and serving 200s on port 5000 internally (confirmed in deploy logs), but the edge returns 503 "Offline" / 500 on all public domains (cashemapp.com, www.cashemapp.com, the generated domain). Production site fully down.

Request: Please manually delete the -f726 domain key from the service config (full deletion, not null), so the edge can route to the healthy container.

$20 Bounty

1 Replies

Status changed to Awaiting Railway Response Railway • about 2 months ago

Railway

BOT

2 months ago

This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.

Status changed to Open Railway • about 2 months ago

ayalaa12

FREETop 5% Contributor

13 days ago

The null behavior is the key clue: that is an update of a field, not deletion of the domain object. Railway now exposes domain deletion directly, including for Railway-provided service domains.

I would use the current CLI to remove only the stale *.up.railway.app object and leave the service, deployment, and custom domains untouched:

railway domain list --service Cashem-App --environment production --project <PROJECT_ID> --json

railway domain status cashem-app-production-f726.up.railway.app \
  --service Cashem-App --environment production --project <PROJECT_ID>

railway domain delete cashem-app-production-f726.up.railway.app \
  --service Cashem-App --environment production --project <PROJECT_ID>

railway domain list --service Cashem-App --environment production --project <PROJECT_ID> --json

Use the exact domain name or domain ID returned by domain list. Deliberately omit --yes on the delete command so the CLI shows the target and asks for confirmation. Confirm that it is the Railway-provided ...-f726.up.railway.app entry, not cashemapp.com or www.cashemapp.com.

The current CLI documentation says railway domain delete <DOMAIN_OR_ID> deletes a custom or service domain, while domain update changes settings such as the target port or Railway-domain name. That is why setting the value to null leaves a broken object behind instead of removing it: https://docs.railway.com/cli/domain

Once the final domain list no longer contains the stale object, test one custom hostname and inspect its response headers. If the stale object is absent from the list but the edge still returns 503/500, send Railway the UTC timestamp of that test, the service/environment IDs, the redacted before/after domain list output, and any Railway request ID from the response. That would distinguish a remaining edge/control-plane cache entry from an application problem and gives staff a precise object and trace to purge.

I would not delete/recreate the service or redeploy the healthy container for this; the evidence already points to the domain control-plane object, and those actions add risk without addressing it.

Welcome!