Sealed service variable configured but unavailable to runtime deployment
minh-trieulam8888
PROOP

12 days ago

A service-scoped sealed variable is reported through boolean metadata as configured, nonempty, and sealed.

However, new deployments do not receive a usable value at runtime. The application fails closed before opening its HTTP port with this redacted application error:

worker key self-test PRIVATE_UNAVAILABLE

The corresponding public-key variable is configured and nonempty. The failure is not caused by a public-key mismatch, application code, or the database.

The previous active deployment has been removed as containment. The service currently has no active deployment, while the database and other services remain unchanged.

Expected behavior:

According to the Railway Variables documentation, a sealed service variable should be unavailable through the UI/API/CLI while still being injected into the service runtime.

Security restriction:

Please do not request the secret value, length, hash, fingerprint, prefix, suffix, raw environment dump, or unredacted logs.

Could Railway staff please confirm:

  1. Whether sealed variables are expected to be injected into runtime deployments.
  2. Whether a conflicting empty override or stale deployment snapshot can prevent injection.
  3. How to verify variable scope and deployment revision binding without revealing the value.
  4. Whether this could be a sealed-variable injection defect.
  5. Whether the safe remediation is to create a new sealed value directly inside Railway.
$10 Bounty

7 Replies

Railway
BOT

12 days ago

Sealed variables are injected into runtime deployments per the variables documentation - their values are hidden in the UI/API/CLI but still provided to builds and deployments. However, the private-key variable on this service is not actually marked as sealed. Additionally, there are staged (unapplied) changes to both the private-key and public-key variables that have not been deployed yet. Any previous deployment would have run with the old values, and the new values will only take effect once the staged changes are applied. You can review and deploy them from the service's canvas (the pending changes bar).


Status changed to Awaiting User Response Railway 12 days ago


minh-trieulam8888
PROOP

12 days ago

Additional redacted diagnosis:

  • The private and public variables are configured.
  • The private variable is sealed.
  • There are no pending variable changes.
  • A new deployment still fails healthcheck with the application code PRIVATE_UNAVAILABLE.
  • The previous deployment remains active.
  • No secret value or derived characteristic has been accessed.

Reapplying staged changes is not applicable because pending_changes=0.

We need a provider-managed remediation that can either:

  1. repair the sealed-variable runtime binding; or
  2. create and inject a replacement Ed25519 private/public keypair directly within Railway without revealing or exporting the private key.

Please confirm whether Railway provides such a provider-side operation. Do not request the secret value, raw environment, key length, hash, fingerprint, prefix or suffix.


Status changed to Awaiting Railway Response Railway 12 days ago


Railway
BOT

12 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway 12 days ago


minh-trieulam8888

Additional redacted diagnosis: - The private and public variables are configured. - The private variable is sealed. - There are no pending variable changes. - A new deployment still fails healthcheck with the application code PRIVATE_UNAVAILABLE. - The previous deployment remains active. - No secret value or derived characteristic has been accessed. Reapplying staged changes is not applicable because pending_changes=0. We need a provider-managed remediation that can either: 1. repair the sealed-variable runtime binding; or 2. create and inject a replacement Ed25519 private/public keypair directly within Railway without revealing or exporting the private key. Please confirm whether Railway provides such a provider-side operation. Do not request the secret value, raw environment, key length, hash, fingerprint, prefix or suffix.

Have you tried with a dummy key via unsealed variable?


minh-trieulam8888
PROOP

11 days ago

Not yet. The application requires a structurally valid Ed25519 keypair, so an arbitrary fake value would only produce an encoding or key-mismatch failure and would not conclusively test sealed-variable runtime injection.

A non-sensitive synthetic Ed25519 keypair could potentially be used for a disposable diagnostic deployment, provided that:

  • it is explicitly treated as non-secret test data;
  • the existing active deployment remains untouched;
  • no production or customer traffic reaches the diagnostic deployment;
  • both the PKCS8 private variable and matching SPKI public variable are staged together;
  • the test only reports a redacted self-test code;
  • the diagnostic deployment is removed afterward.

Would Railway staff recommend this as a valid way to distinguish a sealed-variable injection problem from an application-level key-loading problem?


minh-trieulam8888
PROOP

11 days ago

Additional differential test result:

  • The same service image and application code deploy successfully when the private variable is configured but unsealed.
  • After sealing the same private variable and applying the staged revision through the Railway UI, the newly created deployment fails during the network/health process.
  • Railway reports all expected service variables in the candidate revision.
  • The application still fails closed with the redacted code PRIVATE_UNAVAILABLE.
  • The corresponding public variable is configured.
  • No secret value or derived characteristic was accessed or exposed.
  • The previous healthy deployment remains available, so there was no service interruption.

This rules out:

  1. The variable being absent from the deployment revision.
  2. The application reading the wrong variable name.
  3. Pending staged changes not being applied.
  4. A general application-image failure.

The strongest remaining explanation is that the sealed variable is not being injected into the runtime candidate, while the unsealed form is injected successfully.

Please escalate this as a possible sealed-variable runtime injection defect. We need a provider-side remediation that preserves sealing and does not require revealing, exporting or unsealing the secret.

Please do not request the value, length, hash, fingerprint, prefix, suffix, raw environment or unredacted logs.


minh-trieulam8888
PROOP

10 days ago

Final redacted diagnosis and provider investigation request:

Project: central-orchestrator-staging

Environment: staging

Service: control-plane-worker-b

Failed sealed candidate: 8378836d-1596-455c-b941-884fc38995c2

Previous healthy candidate: 9a9f7a03-c167-458b-817a-c290a4a18404

build=PASS

container_deploy=PASS

self_test=PRIVATE_UNAVAILABLE

http_server_started=false

readiness=FAIL

deployment=FAILED

external_acceptance=NOT_RUN

secret_exposure_detected=false

The private variable is configured and sealed, and the sealing revision was applied. The same image and matching keypair passed startup before sealing; the sealed revision fails before the HTTP server starts. Railway documentation states sealed values are provided to deployments.

Please perform a provider-side investigation of the sealed-variable runtime binding for deployment 8378836d-1596-455c-b941-884fc38995c2 and advise a documented remediation that preserves sealing. We will not redeploy, rotate, unseal, or change Variables while this is investigated.

Do not request or disclose the secret value, raw environment, raw logs, key length, hash, fingerprint, prefix, or suffix.


minh-trieulam8888
PROOP

10 days ago

Workspace support tier is now Pro. The private Technical Help form detected this as a duplicate and directed us to continue on this existing thread.

Please treat this thread as a Pro technical-support escalation for the provider-side sealed-variable runtime binding failure documented above. The selected service is central-orchestrator-staging / control-plane-worker-b / staging.

All non-disclosure constraints remain mandatory: no secret value, raw environment, raw logs, length, hash, fingerprint, prefix, suffix, unseal, rotation, redeploy, or Variable changes without a documented provider-coordinated remediation.


Welcome!

Sign in to your Railway account to join the conversation.

Loading...