Update — this has escalated from a dashboard display bug to an actual API authorization failure:

Both affected services (constructiq-api-staging / 76d69a9a-a9f5-4bd3-9544-ec01133b4240 and constructiq-worker-staging / 62ffbf9d-e3c0-48bc-80ad-1f5579572d3b) now return "Not Authorized" when queried via the public GraphQL API (service(id: ...) query), where before they were returning valid data (I had successfully queried both moments earlier in the same session).

This started immediately after creating an unrelated third service (constructiq-clamav-staging, a plain Docker-image service, clamav/clamav:stable) in the same project via railway add. I did not touch the two affected services directly — this was a side effect of an unrelated action elsewhere in the project.

I confirmed this is isolated specifically to these two services, not a broader auth/session issue:

• Still logged in correctly (railway whoami confirms the same account)
• Every other service in the project queries fine via the exact same service(id) query: Redis, thriving-growth, @constructiq/api, @constructiq/web, ConstructIQ, and the newly-created constructiq-clamav-staging
• Only constructiq-api-staging and constructiq-worker-staging — the two services I renamed via the API earlier in this thread — are affected

Given the pattern (rename → broken dashboard navigation → variables not shown in dashboard despite being present via API → now full API access denied), it looks like these two service records may be in a corrupted or inconsistent internal state. I've stopped making further changes to them to avoid making it worse. Would appreciate someone from the team taking a look directly.

Hi — I traced this through our audit log. The two services were deleted, but not by the rename. Sequence in agile-laughter / staging:

• 19:30:19Z — you created constructiq-clamav-staging
• 19:30:22Z — constructiq-worker-staging (62ffbf9d…) removed by system
• 19:30:23Z — constructiq-api-staging (76d69a9a…) removed by system

Committing the clamav change triggered it. When staged changes are committed, we clean up service records in the project that have no deployed instance in any environment and aren't referenced by any staged changes. Both of those matched — they'd been created but never committed, so no instance ever existed for them. That cleanup is intended; firing it off an unrelated commit with no notification isn't, and I'm sorry it caught you out.

The missing instance also explains what you saw beforehand. A service record with no instance still renders on the canvas, appears in Ctrl+K, and serves service(id), variableUpsert, and deployment triggers — but there's nothing for the dashboard to open, so clicks did nothing. That predates your rename: serviceUpdate only writes name and icon, and the dashboard navigates by service ID, not name, so no name-derived cache is involved. The Not Authorized is the same cause from the API side — service(id) returns that for any ID that doesn't exist, including a randomly invented UUID.

Since they never had an instance they were removed outright rather than tombstoned, so there's nothing to restore. Looks like you've already recreated them as constructiq-api and constructiq-worker. Both latest deploys failed, but that's a separate build issue — happy to look if you share what you're hitting.

To avoid a repeat: don't leave services staged and uncommitted. You have staged changes in both environments right now — thriving-growth and ConstructIQ in staging, @constructiq/api and @constructiq/web in production. Discarding either set cleans up those records the same way.

Thanks for tracing that — the cleanup-on-commit explanation makes total sense, and I appreciate the detail. Good to know for the future; I'll make sure not to leave services staged-and-uncommitted going forward.

On the build issue you offered to look at: it's actually not a build problem, it's a Redis authentication problem, and it's now happened twice in a row in this same project (agile-laughter / staging), including on a brand-new Redis instance created from scratch. Details:

1. Original Redis instance (serviceId 0d2da226-b495-44d5-a145-016d5846c2ae) started returning WRONGPASS invalid username-password pair or user is disabled to constructiq-api / constructiq-worker, despite:

• The REDIS_URL resolved via variable reference on the consuming services matching, byte-for-byte (SHA256 hash comparison), the REDIS_URL Redis's own service variables reported.
• Restarting the Redis service twice (confirmed via updatedAt timestamp changes) — no change in behavior.
• Testing with railway connect Redis --environment staging --ssh, which uses Railway's own internally-managed credential injection (never exposed to me/the client) — this ALSO failed with the identical WRONGPASS error.

1. Since that looked unrecoverable, I deleted that Redis service (serviceDelete, succeeded) and created a brand-new one via railway add --database redis (serviceId eba3df5b-21e4-4afb-a1e3-0cad1635535a). It deployed successfully (deployment 067410ba-d3a2-4a80-981d-bc23d197cb51, status SUCCESS).
2. Re-pointed REDIS_URL references on constructiq-api and constructiq-worker to the new Redis service, confirmed (again via SHA256 hash) that the resolved value matches the new Redis's own stored REDIS_URL exactly.
3. Tried railway connect Redis --environment staging --ssh again on this brand-new instance — same failure: AUTH failed: WRONGPASS invalid username-password pair or user is disabled. NOAUTH Authentication required.

So this is now confirmed on two separate, freshly-created Redis instances in the same project/environment, using Railway's own credential injection (not anything on my end). Could you check server-side whether redis-server is actually enforcing the REDIS_PASSWORD value the control plane believes is current for Redis template deployments in this project? Happy to provide any deployment/service IDs you need.

Good news: your new Redis isn't broken. I verified server side that the password it enforces is exactly the one stored in its service variables, and our logs show a client successfully authenticating and writing to it at 00:58 UTC, right in the middle of your testing window. So at least one of your connect attempts actually worked.

Here's what I think tripped you up:

1. The WRONGPASS errors in constructiq-api's logs are all from 00:40 to 00:43 UTC, against the old Redis, before the new one existed. After you re-pointed REDIS_URL, the api was never successfully redeployed (the 05:09 deploy failed at build), so the running container still has the old URL baked in. Variable changes only apply on a redeploy, a restart isn't enough.
2. constructiq-worker did redeploy with the new URL, but it's crash looping before it ever reaches Redis: its DATABASE_URL, DATABASE_ADMIN_URL, and S3 variables are empty strings in staging.
3. For railway connect, the CLI pulls the current password fresh each time, so a failure there usually means a stale value sneaking in locally (an exported REDIS_URL in your shell, or a saved connection from the old instance). Worth ruling out before retrying.

So: fill in the worker's missing variables, fix the api build and redeploy it, and both should connect fine.

Thanks, this is really helpful — appreciate you digging through the audit log.

Quick correction on my end: you're right that the new Redis wasn't actually broken. I re-tested railway connect --ssh against it and got the same WRONGPASS I'd seen before, which is what led me to (wrongly) conclude it was a second instance of the same bug and report it as such. Given your server-side evidence (successful auth+write at 00:58 UTC), that was very likely a stale/local issue on my end with the CLI session, not your platform — sorry for the noise, and please disregard that part of the earlier report.

On the two action items:

1. Worker's missing variables — fixed. DATABASE_URL and the S3_* vars are now set as variable references pointing at constructiq-api's values (which were already correct), confirmed non-empty. Still missing DATABASE_ADMIN_URL entirely (a distinct Postgres role our app uses for one background job) — that one's on us to provision, not something Railway needs to look at.
2. API build/deploy — this turned out to be unrelated to Redis entirely. Root cause: our API sends a startup test email over SMTP with no connection timeout configured, and it turns out Railway blocks outbound SMTP (tested both 465 and 587 against smtp.resend.com — both hang until TCP timeout). With no timeout on our side, that hung sendMail() call blocked our whole app's boot sequence forever, so it never reached app.listen() — no error in the logs, no crash, just an app that silently never finished starting. We've since moved to Resend's HTTP API instead of SMTP and added timeouts everywhere. Just flagging this in case the SMTP-egress block is worth documenting somewhere for other users hitting the same silent-hang symptom.

Still finishing verification on our side (health check timing), will update if anything else comes up. Thanks again for the help.

Glad everything came together. On the SMTP point, that is already documented - SMTP is only available on Pro and above, and Hobby plans need to use HTTP-based email APIs like Resend. Details are at docs.railway.com/networking/outbound-networking.