ariableUpsert mutation persistently fails with 504/"Problem processing request" across all services
comunidadfeliz-interno
PROOP

a month ago

Hi team,

We're seeing persistent failures on the variableUpsert GraphQL mutation

across our entire project — affects newly created services AND existing

services. Started 2026-06-02 and still ongoing on 2026-06-03.

Account / Project:

  • Project ID: e3068361-49f0-4544-8bec-a83db3854e30
  • Environment: production
  • Environment ID: c9eb5bdf-839b-4575-98af-4591ca360afd

Symptom:

  • variableUpsert(input: { projectId, environmentId, serviceId?, name, value, skipDeploys: true })

    returns HTTP 504 (Cloudflare gateway timeout, HTML body) on most calls.

  • After multiple retries, some eventually land (the mutation succeeded

    server-side even though HTTP returned 504). Others continue to fail or

    return GraphQL error: { "message": "Problem processing request" }.

  • Same failure mode on project-level shared variable writes (no serviceId)

    AND on per-service writes (with serviceId).

  • Same failure mode on existing services AND on newly created services.

  • variableDelete calls in the same project work fine, no 504s. Only

    variableUpsert and sharedVariableConfigure exhibit the timeout.

Reproduction:

  1. POST graphql/v2 with variableUpsert mutation

  2. Connection times out at ~60s with HTTP 504

  3. Retry → same result, until eventually it lands or hits the GraphQL

    "Problem processing request" error

Sample traceIds from "Problem processing request" responses:

  • 6137817602381378991
  • 8287597822565014605

Status page reports "fully operational" but the history shows

"Elevated Deploy Times in US East" incidents on June 1-2.

We have automated deploys that depend on this mutation. A significant

fraction of deploys end up in a half-built state because the variable

writes never succeed.

Questions:

  1. Is the variableUpsert pipeline degraded for our project? What's queued

    on your end?

  2. Is variableCollectionUpsert a more reliable alternative?

  3. Can you investigate the traceIds above?

Solved

5 Replies

Status changed to Awaiting Railway Response Railway about 1 month ago


comunidadfeliz-interno
PROOP

a month ago

Hello team, we're still waiting for an answer for this. We can't deploy our services since it's impossible to update env variables


dizzydes90
EMPLOYEE

a month ago

This isn't project-specific queueing or a rate limit. The variableUpsert resolver is expensive server-side in projects with many cross-service variable references (chained ${{...}} references across your 13 services), and right now its p95 latency sits well above the 60s Cloudflare gateway timeout. That's why you see 504s with an HTML body, why some calls land server-side even though the HTTP response timed out, and why variableDelete (a cheap resolver) is unaffected. We can confirm elevated latency on that resolver across your project's recent calls. Does this make sense given your level of variables?

The recommended fix - and the answer to your question 2 - is to batch your writes with variableCollectionUpsert instead of issuing individual variableUpsert calls. The CLI's railway variable set issues one variableUpsert per variable under the hood, so 15 sequential expensive mutations per service become a single request:


  variableCollectionUpsert(

    input: {

      ...

      variables: { KEY_A: "valA", KEY_B: "valB", POSTGRES_URI: "..." }```

Pass `skipDeploys: true` semantics by setting variables before triggering your deploy. This is the same change another team with an identical PR-preview workflow adopted, and it cleared their timeouts. Switching your CI from railway variable set loops to direct GraphQL variableCollectionUpsert (one call per service) is the path we'd recommend.

On your questions 1 and 3: the pipeline isn't degraded specifically for your project and nothing is stuck in a queue - it's resolver cost that scales with reference complexity. The traceIds you shared are client-side IDs that don't map into our server-side tracing, but we located your project's mutations directly and confirmed the latency pattern above.

Status changed to Awaiting User Response Railway about 1 month ago


jose-lico
PRO

a month ago

I am also experiencing this issue for days, now that my project has grown to ~15 services, I only get 504s


Status changed to Awaiting Railway Response Railway about 1 month ago


comunidadfeliz-interno
PROOP

a month ago

This issue is still present in all my variable upserts, even when its only one variable.

Additionally, this is the fourth thay of general service degradation, variable upserting, service deployment, and all api requests are failing constantly.


dizzydes90
EMPLOYEE

a month ago

Ah, in that case I thnk its related to something we're looking at with 504s in GraphQL, I have escalated to our eng team who are looking into it.


Status changed to Awaiting User Response Railway about 1 month ago


Railway
BOT

22 days ago

This thread has been marked as solved automatically due to a lack of recent activity. Please re-open this thread or create a new one if you require further assistance. Thank you!

Status changed to Solved Railway 22 days ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...