a month ago
I have had several commits fail to build over the last few days as I have been making updates to my project via an AI agent. The build hangs for 20+ minutes before Railway times out. Seems to be at the same point, during the Vite build.
Normally, there are notices when Railway is having issues, but no notices have been seen in my account, so I have to believe there is something with my project or the commits that have been submitted?
27 Replies
a month ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • about 2 months ago
a month ago
To distinguish a code regression from a Railway-specific build issue, compare the last successful commit and first failing commit using the same Node version, package manager and lockfile in an isolated local checkout.
Could you share the exact build command, Node/Vite versions, and the final build-log lines before the stall, with secrets removed? Also, does that same commit build locally?>
A timeout alone doesn’t establish an out-of-memory problem. Those details will help narrow down whether this involves a build plugin, dependency change or the build environment.>
AI-assisted; I haven’t reproduced your project yet.>
> To distinguish a code regression from a Railway-specific build issue, compare the last successful commit and first failing commit using the same Node version, package manager and lockfile in an isolated local checkout. > > Could you share the exact build command, Node/Vite versions, and the final build-log lines before the stall, with secrets removed? Also, does that same commit build locally?> > > A timeout alone doesn’t establish an out-of-memory problem. Those details will help narrow down whether this involves a build plugin, dependency change or the build environment.> > > AI-assisted; I haven’t reproduced your project yet.>
a month ago
Build command: bun install && bun run build (where build = tsc --noEmit && vite build)
Versions in our build environment, from your own build logs:
Bun: 1.1.43
Vite: 7.3.6
Node compat: 22.6.0 (Vite's own startup warning — this is Bun's Node-compat shim version, not a separate Node install)
One real discrepancy worth you checking: our committed bun.lock pins [email protected], but your build logs show [email protected] actually running. That's not something we changed — same lockfile, same commit, across every deploy including the one that succeeded. Worth confirming whether your install step is resolving against the lockfile or against the registry fresh each time.
Final log lines before the stall, every time, byte-identical:
transforming...
node_modules/zod/v4/core/regexes.js (74:0): A comment "..." contains an annotation that Rollup cannot interpret due to the position of the comment. The comment will be removed to avoid issues.
node_modules/zod/v4/core/util.js (400:0): A comment "..." contains an annotation that Rollup cannot interpret due to the position of the comment. The comment will be removed to avoid issues.
Then nothing — no further output, no error, until your platform times it out.
Does it build locally? Yes, every time, in ~10-13 seconds, with the committed lockfile and reasonable memory headroom.
One more thing we found and already fixed on our end (pushed as c8bce73): our production build was a single ~2.8MB/799KB-gzip JS chunk with everything bundled together — Vite's own build output was flagging this. We split it into separate vendor chunks (mapbox-gl alone is ~1.8MB). That halves the size of the largest single Rollup/Terser minification pass. We can't prove that's your root cause, but if it's memory pressure on your builder host during that minify step, this should meaningfully help, and it's a legitimate improvement either way.
timatosystems
Build command: bun install && bun run build (where build = tsc --noEmit && vite build) Versions in our build environment, from your own build logs: Bun: 1.1.43 Vite: 7.3.6 Node compat: 22.6.0 (Vite's own startup warning — this is Bun's Node-compat shim version, not a separate Node install) One real discrepancy worth you checking: our committed bun.lock pins [email protected], but your build logs show [email protected] actually running. That's not something we changed — same lockfile, same commit, across every deploy including the one that succeeded. Worth confirming whether your install step is resolving against the lockfile or against the registry fresh each time. Final log lines before the stall, every time, byte-identical: transforming... node_modules/zod/v4/core/regexes.js (74:0): A comment "..." contains an annotation that Rollup cannot interpret due to the position of the comment. The comment will be removed to avoid issues. node_modules/zod/v4/core/util.js (400:0): A comment "..." contains an annotation that Rollup cannot interpret due to the position of the comment. The comment will be removed to avoid issues. Then nothing — no further output, no error, until your platform times it out. Does it build locally? Yes, every time, in ~10-13 seconds, with the committed lockfile and reasonable memory headroom. One more thing we found and already fixed on our end (pushed as c8bce73): our production build was a single ~2.8MB/799KB-gzip JS chunk with everything bundled together — Vite's own build output was flagging this. We split it into separate vendor chunks (mapbox-gl alone is ~1.8MB). That halves the size of the largest single Rollup/Terser minification pass. We can't prove that's your root cause, but if it's memory pressure on your builder host during that minify step, this should meaningfully help, and it's a legitimate improvement either way.
a month ago
Thanks, that narrows it down quite a bit.
The Vite version mismatch is the first thing I’d investigate. If the committed lockfile resolves Vite 7.3.2 but Railway is actually running 7.3.6 on the same commit, I’d try making the install strictly immutable/frozen and confirm the build uses exactly the locked dependency graph.
I’d also test the same commit on Railway with minification temporarily disabled. Since the build consistently stops after transformation and local builds complete in ~10–13 seconds, that would help distinguish a dependency/install issue from a Railway resource or minifier stall.
The zod annotation warnings themselves don’t look like the root cause, since they are warnings and the build continues past module transformation locally.
If a frozen install still gives Vite 7.3.6, that would be especially interesting because it would point more strongly to the Railway/Bun build environment rather than your application code.
Thanks, that narrows it down quite a bit. The Vite version mismatch is the first thing I’d investigate. If the committed lockfile resolves Vite 7.3.2 but Railway is actually running 7.3.6 on the same commit, I’d try making the install strictly immutable/frozen and confirm the build uses exactly the locked dependency graph. I’d also test the same commit on Railway with minification temporarily disabled. Since the build consistently stops after transformation and local builds complete in ~10–13 seconds, that would help distinguish a dependency/install issue from a Railway resource or minifier stall. The zod annotation warnings themselves don’t look like the root cause, since they are warnings and the build continues past module transformation locally. If a frozen install still gives Vite 7.3.6, that would be especially interesting because it would point more strongly to the Railway/Bun build environment rather than your application code.
a month ago
Ran both tests you suggested. Results below.
- Frozen install — Vite still resolves to 7.3.6
I set the build command to bun install --frozen-lockfile && tsc --noEmit && vite build (previously it was running bun install without the flag, pulled from our Railway service's build command directly — our railway.json/nixpacks.toml in the repo turned out not to be read by Railway at all for this service, so the flag had never actually been applied before).
With --frozen-lockfile, Vite still resolves to 7.3.6, even though our bun.lock pins @template/web's vite to 7.3.2 exactly. So this isn't an unfrozen/stale install — the frozen install itself resolves to the wrong version.
I pushed it further: I noticed your builder is running Bun 1.1.43 (an old Nix-pinned version), and confirmed locally that Bun 1.1.43 can't parse our bun.lock's format at all — it throws Unknown lockfile version and silently falls back to ignoring it. That looked like a strong candidate explanation for the version drift, so I pinned Bun to a newer version via packageManager in package.json and redeployed. Railway picked it up and used Bun 1.3.0 this time (confirmed in build logs). Vite still resolved to 7.3.6 even with the newer Bun that parses our lockfile correctly.
So: the vite 7.3.2-vs-7.3.6 discrepancy is reproducible with frozen installs on two different Bun versions (1.1.43 and 1.3.0). It's not a stale-install issue and not a Bun-can't-read-our-lockfile issue — something in your build install step is resolving/hoisting vite to 7.3.6 regardless.
- Minification disabled — build still hangs, at the same point, before minify even starts
Set vite build --minify false and redeployed. The build still hung, and critically, it hung at the exact same spot as every prior failed build: right after Vite emits two warnings about node_modules/zod/v4/core/regexes.js and util.js ("comment Rollup cannot interpret"), during the module transform phase — well before rendering/minification ever starts. So the minifier isn't the bottleneck; the build never gets that far.
Summary across all variants tested
I tried six combinations (Bun 1.1.43 / 1.3.0 × frozen / default install × minified / unminified). Every single one hangs at the identical point — immediately after those two zod warnings, zero further log output until your platform timeout kills it. Nothing I changed moved that line. That consistency across every variable you asked us to isolate points at something in the build container itself stalling during module transform on this dependency graph (we have ~2,065 modules, including mapbox-gl, AI SDK packages, etc.) — not our Vite/Bun version pinning or our minifier config.
Given that, a couple of things I'd ask on your end:
Can you confirm whether this service's build container has a memory or CPU ceiling that could cause a silent hang (rather than a visible OOM kill) during module transform under load?
Any chance you can see what's actually happening on your infra at the point builds stall — the logs on our end just stop with no error.
Happy to run anything else you want isolated — let me know what's next.
Thanks, Tim
a month ago
One clarification first: I’m an independent community participant, not Railway staff, so I don’t have access to Railway’s internal builder logs or infrastructure.
Thanks for running those tests. Since updating Bun and disabling minification didn’t resolve the hang, I’d like to check one remaining ambiguity before suggesting more dependency changes: which Vite executable the failing build actually uses.
The 7.3.2 vs 7.3.6 difference could be related to workspace dependency resolution or the working directory. It doesn’t necessarily explain the hang, but it’s worth checking.
Could you add the following diagnostics inside the existing build script for @template/web, immediately before its current build command? Please keep Railway’s working directory and existing workspace invocation unchanged so we inspect the same environment as the failing build.
echo "=== working directory ==="
pwd
echo "=== Bun version ==="
bun --version
echo "=== local bin directory ==="
bun pm bin || true
echo "=== Vite executable ==="
command -v vite || true
if command -v vite >/dev/null 2>&1; then
readlink -f "$(command -v vite)" || true
fi
vite --version || true
echo "=== resolved Vite package ==="
bun -e 'try {
const p = require.resolve("vite/package.json");
console.log("Package path:", p);
console.log("Version:", require(p).version);
} catch (e) {
console.error(e.message);
}' || true
echo "=== Vite dependency chain ==="
bun why vite || true
echo "=== existing build follows ==="Then run your existing build command unchanged.
Please share that diagnostic output along with the final build logs. This should let us compare the working directory, executable location, resolved package version, and dependency chain.
If those point to an unexpected Vite installation, we can investigate the workspace resolution before changing anything else. If they match the expected installation, I’d put the version discrepancy aside and focus next on identifying which plugin or transform operation stops progressing, alongside checking memory usage during the build.
The two Zod warnings alone don’t identify which module or operation is hanging—they may simply be the last visible output before it stalls.
One clarification first: I’m an independent community participant, not Railway staff, so I don’t have access to Railway’s internal builder logs or infrastructure. Thanks for running those tests. Since updating Bun and disabling minification didn’t resolve the hang, I’d like to check one remaining ambiguity before suggesting more dependency changes: which Vite executable the failing build actually uses. The 7.3.2 vs 7.3.6 difference could be related to workspace dependency resolution or the working directory. It doesn’t necessarily explain the hang, but it’s worth checking. Could you add the following diagnostics inside the existing build script for @template/web, immediately before its current build command? Please keep Railway’s working directory and existing workspace invocation unchanged so we inspect the same environment as the failing build. ```sh echo "=== working directory ===" pwd echo "=== Bun version ===" bun --version echo "=== local bin directory ===" bun pm bin || true echo "=== Vite executable ===" command -v vite || true if command -v vite >/dev/null 2>&1; then readlink -f "$(command -v vite)" || true fi vite --version || true echo "=== resolved Vite package ===" bun -e 'try { const p = require.resolve("vite/package.json"); console.log("Package path:", p); console.log("Version:", require(p).version); } catch (e) { console.error(e.message); }' || true echo "=== Vite dependency chain ===" bun why vite || true echo "=== existing build follows ===" ``` Then run your existing build command unchanged. Please share that diagnostic output along with the final build logs. This should let us compare the working directory, executable location, resolved package version, and dependency chain. If those point to an unexpected Vite installation, we can investigate the workspace resolution before changing anything else. If they match the expected installation, I’d put the version discrepancy aside and focus next on identifying which plugin or transform operation stops progressing, alongside checking memory usage during the build. The two Zod warnings alone don’t identify which module or operation is hanging—they may simply be the last visible output before it stalls.
a month ago
What we found
Diagnostic data (your requested block, run right before the build):
pwd: /app
bun --version: 1.3.0
bun pm bin: /app/node_modules/.bin
vite executable: /app/node_modules/.bin/vite → ../vite/bin/vite.js
vite --version: vite/7.3.7 linux-x64 node-v24.3.0
require.resolve('vite/package.json'): /app/node_modules/vite/package.json → version 7.3.7
bun why vite errored (Lockfile not found) — turned out to be our own command, should've been bun pm why, not critical
The build then hung at the identical spot as every prior attempt — right after the two Rollup warnings about node_modules/zod/v4/core/regexes.js and node_modules/zod/v4/core/util.js, mid-"transforming," zero further output.
New finding that reframes the version mismatch: I reproduced locally with the exact same lockfile and exact same Bun version (1.3.0) — bun install --frozen-lockfile resolves [email protected] correctly every single time, matching the lockfile's pinned sha512 hash exactly. So this isn't a lockfile or Bun-version issue on our end at all — the identical inputs produce the correct version locally and the wrong version on your infra.
Likely cause: your BuildKit cache mount. Your build logs show the install step uses --mount=type=cache,id=s/d12dea82-8ef3-4110-bd98-e7ea1915ed84-/root/bun,target=/root/.bun — a persistent cache across builds. If that cache is holding a stale/mismatched vite package from before, or from whatever last resolved a non-pinned version, that could explain Bun resolving 7.3.7 despite --frozen-lockfile pinning 7.3.2. It would also explain the hang: a cached install that doesn't match what the lockfile/metadata expect could easily behave unpredictably mid-transform on specific files.
(Side note, unrelated to the hang: your dashboard's buildCommand setting doesn't appear to take effect for this service — I changed it four separate times across four fresh deploys and every one still ran a different, fixed command. Had to inject the diagnostic by editing the actual npm script in the repo instead to get this data. Worth a look separately.)
timatosystems
What we found Diagnostic data (your requested block, run right before the build): pwd: /app bun --version: 1.3.0 bun pm bin: /app/node_modules/.bin vite executable: /app/node_modules/.bin/vite → ../vite/bin/vite.js vite --version: vite/7.3.7 linux-x64 node-v24.3.0 require.resolve('vite/package.json'): /app/node_modules/vite/package.json → version 7.3.7 bun why vite errored (Lockfile not found) — turned out to be our own command, should've been bun pm why, not critical The build then hung at the identical spot as every prior attempt — right after the two Rollup warnings about node_modules/zod/v4/core/regexes.js and node_modules/zod/v4/core/util.js, mid-"transforming," zero further output. New finding that reframes the version mismatch: I reproduced locally with the exact same lockfile and exact same Bun version (1.3.0) — bun install --frozen-lockfile resolves [email protected] correctly every single time, matching the lockfile's pinned sha512 hash exactly. So this isn't a lockfile or Bun-version issue on our end at all — the identical inputs produce the correct version locally and the wrong version on your infra. Likely cause: your BuildKit cache mount. Your build logs show the install step uses --mount=type=cache,id=s/d12dea82-8ef3-4110-bd98-e7ea1915ed84-/root/bun,target=/root/.bun — a persistent cache across builds. If that cache is holding a stale/mismatched vite package from before, or from whatever last resolved a non-pinned version, that could explain Bun resolving 7.3.7 despite --frozen-lockfile pinning 7.3.2. It would also explain the hang: a cached install that doesn't match what the lockfile/metadata expect could easily behave unpredictably mid-transform on specific files. (Side note, unrelated to the hang: your dashboard's buildCommand setting doesn't appear to take effect for this service — I changed it four separate times across four fresh deploys and every one still ran a different, fixed command. Had to inject the diagnostic by editing the actual npm script in the repo instead to get this data. Worth a look separately.)
a month ago
Thanks — that’s useful evidence. The executable and package-resolution checks both confirm that the failing build is using Vite 7.3.7.
Your local reproduction makes the discrepancy worth investigating, but I’d keep the cache explanation as a hypothesis for now. The presence of a persistent cache mount doesn’t by itself prove that Bun installed an incorrect package, or that the version difference causes the hang.
The most useful next step would be a controlled clean-install comparison. For the same commit, could you capture:
- The exact remote install command and its working directory.
-
- The lockfile’s SHA-256 checksum immediately before installation, compared with your local copy.
-
- The resolved Vite version immediately after installation, then again immediately before the build.
That last comparison would help establish whether the install itself produces 7.3.7 or whether something changes it afterward.
Then repeat with a fresh dependency installation, without reusing the existing Bun cache mount or node_modules. A fresh deployment alone may still reuse those caches.
If the clean installation produces 7.3.2, that would implicate reused cache/install state. If the build still hangs with 7.3.2, we should treat the transform stall as a separate problem. The two Zod warnings could simply be the last visible output, rather than the operation that is hanging.
For the buildCommand issue, is the command also defined in railway.toml/railway.json, a Dockerfile, or Railpack/Nixpacks configuration? Checking those alongside the effective build plan would help distinguish a configuration override from a dashboard issue.
If the remote install demonstrably receives identical inputs but still produces a different version in a clean environment, that would make a much stronger infrastructure bug report.
Thanks — that’s useful evidence. The executable and package-resolution checks both confirm that the failing build is using Vite 7.3.7. Your local reproduction makes the discrepancy worth investigating, but I’d keep the cache explanation as a hypothesis for now. The presence of a persistent cache mount doesn’t by itself prove that Bun installed an incorrect package, or that the version difference causes the hang. The most useful next step would be a controlled clean-install comparison. For the same commit, could you capture: - The exact remote install command and its working directory. - - The lockfile’s SHA-256 checksum immediately before installation, compared with your local copy. - - The resolved Vite version immediately after installation, then again immediately before the build. That last comparison would help establish whether the install itself produces 7.3.7 or whether something changes it afterward. Then repeat with a fresh dependency installation, without reusing the existing Bun cache mount or node_modules. A fresh deployment alone may still reuse those caches. If the clean installation produces 7.3.2, that would implicate reused cache/install state. If the build still hangs with 7.3.2, we should treat the transform stall as a separate problem. The two Zod warnings could simply be the last visible output, rather than the operation that is hanging. For the buildCommand issue, is the command also defined in railway.toml/railway.json, a Dockerfile, or Railpack/Nixpacks configuration? Checking those alongside the effective build plan would help distinguish a configuration override from a dashboard issue. If the remote install demonstrably receives identical inputs but still produces a different version in a clean environment, that would make a much stronger infrastructure bug report.
a month ago
Good news / bad news. The clean-install comparison wasn't quite how I ran it (explain below), but I found something more direct:
I confirmed packages/web/railway.json's build.buildCommand field is the one actually governing builds — not the dashboard setting. I proved it with a one-line marker test (echo MARKER && ...) committed to that file; it showed up in the build log immediately, while four separate dashboard-set buildCommand changes across four earlier deploys never did. So to answer your question directly: yes, the command is also defined in packages/web/railway.json, and that's the one Railway is actually using, regardless of what the dashboard/API shows as saved.
While poking at that file I found the actual cause of the version drift: this service's build context is scoped to packages/web (via its Root Directory setting), and there's no bun.lock reachable anywhere in that scoped context — I confirmed this with a filesystem-wide search during a build. So bun install --frozen-lockfile has nothing to freeze against, and Vite resolves fresh against the ^7.3.1 caret range in package.json instead of the lockfile's pinned 7.3.2. That explains the drift (7.3.2 → 7.3.6 → 7.3.7) as new Vite patches shipped — nothing to do with your cache mounts, I think I was off-base there.
Fix: I pinned vite to an exact 7.3.2 in package.json (no caret), confirmed locally it resolves and builds cleanly in 16s, then deployed it for real.
The build still hung at the identical spot — right after the same two zod/v4 Rollup warnings — even with 7.3.2 confirmed in the logs. So per what you said: we should treat this as the separate problem now. Version mismatch is resolved; the transform stall is not.
Given that, I think the next useful step on your end is checking build memory/CPU usage for this service around the hang point, since the warnings are probably just the last visible output rather than the actual cause, same as you guessed. Anything I can pull from my side to help narrow that down?
timatosystems
Good news / bad news. The clean-install comparison wasn't quite how I ran it (explain below), but I found something more direct: I confirmed packages/web/railway.json's build.buildCommand field is the one actually governing builds — not the dashboard setting. I proved it with a one-line marker test (echo MARKER && ...) committed to that file; it showed up in the build log immediately, while four separate dashboard-set buildCommand changes across four earlier deploys never did. So to answer your question directly: yes, the command is also defined in packages/web/railway.json, and that's the one Railway is actually using, regardless of what the dashboard/API shows as saved. While poking at that file I found the actual cause of the version drift: this service's build context is scoped to packages/web (via its Root Directory setting), and there's no bun.lock reachable anywhere in that scoped context — I confirmed this with a filesystem-wide search during a build. So bun install --frozen-lockfile has nothing to freeze against, and Vite resolves fresh against the ^7.3.1 caret range in package.json instead of the lockfile's pinned 7.3.2. That explains the drift (7.3.2 → 7.3.6 → 7.3.7) as new Vite patches shipped — nothing to do with your cache mounts, I think I was off-base there. Fix: I pinned vite to an exact 7.3.2 in package.json (no caret), confirmed locally it resolves and builds cleanly in 16s, then deployed it for real. The build still hung at the identical spot — right after the same two zod/v4 Rollup warnings — even with 7.3.2 confirmed in the logs. So per what you said: we should treat this as the separate problem now. Version mismatch is resolved; the transform stall is not. Given that, I think the next useful step on your end is checking build memory/CPU usage for this service around the hang point, since the warnings are probably just the last visible output rather than the actual cause, same as you guessed. Anything I can pull from my side to help narrow that down?
a month ago
Thanks — that separates the two issues nicely. The railway.json marker explains which build command is effective, and reproducing the hang with 7.3.2 means the Vite version discrepancy is not required for the stall.
The missing lockfile in the scoped build context is also an important reproducibility issue. Pinning Vite fixes that direct dependency, but it doesn’t lock the rest of the dependency tree, so the local and remote installations may still differ.
I can’t access service-level CPU or memory telemetry myself, but yes — there are useful things you can collect from your side:
-
The full build log from installation onward, including the eventual timeout or exit message, plus the current build command.
-
Your vite.config and the scripts section of packages/web/package.json, with secrets removed. Please also include any custom Vite/Rollup plugins.
-
One build with Vite’s --debug flag added to the existing invocation. It may not identify the exact stalled transform, but it should expose the effective configuration and plugin list.
For resource evidence, if the build container exposes cgroup v2 metrics, these files are useful:
/sys/fs/cgroup/memory.current
/sys/fs/cgroup/memory.peak
/sys/fs/cgroup/memory.max
/sys/fs/cgroup/memory.events
/sys/fs/cgroup/cpu.stat
They need to be sampled periodically while the build is running, not only before it starts. A timestamped sample every 10–15 seconds would let us compare memory usage and CPU-counter changes during the apparent stall. If those files aren’t available, Railway support may need to provide the builder telemetry.
I’d avoid changing more dependency versions for now. With the configuration and resource samples, we can decide whether to investigate memory pressure, ongoing CPU-heavy work, or a plugin/transform waiting without progress.
Separately, restoring a lockfile-backed install in the actual build context would make the local-versus-remote comparison much stronger.
Thanks — that separates the two issues nicely. The railway.json marker explains which build command is effective, and reproducing the hang with 7.3.2 means the Vite version discrepancy is not required for the stall. The missing lockfile in the scoped build context is also an important reproducibility issue. Pinning Vite fixes that direct dependency, but it doesn’t lock the rest of the dependency tree, so the local and remote installations may still differ. I can’t access service-level CPU or memory telemetry myself, but yes — there are useful things you can collect from your side: 1. The full build log from installation onward, including the eventual timeout or exit message, plus the current build command. 2. Your vite.config and the scripts section of packages/web/package.json, with secrets removed. Please also include any custom Vite/Rollup plugins. 3. One build with Vite’s --debug flag added to the existing invocation. It may not identify the exact stalled transform, but it should expose the effective configuration and plugin list. For resource evidence, if the build container exposes cgroup v2 metrics, these files are useful: /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.events /sys/fs/cgroup/cpu.stat They need to be sampled periodically while the build is running, not only before it starts. A timestamped sample every 10–15 seconds would let us compare memory usage and CPU-counter changes during the apparent stall. If those files aren’t available, Railway support may need to provide the builder telemetry. I’d avoid changing more dependency versions for now. With the configuration and resource samples, we can decide whether to investigate memory pressure, ongoing CPU-heavy work, or a plugin/transform waiting without progress. Separately, restoring a lockfile-backed install in the actual build context would make the local-versus-remote comparison much stronger.
a month ago
Thanks for the detailed questions — we built a one-off instrumented build command specifically to answer them cleanly, ran it on a throwaway deploy so it couldn't ship to production if it failed, and got a clean, complete capture of the hang on the first try. Full raw build log is attached (diag-v3-full-buildlog.txt, install through the timeout). Summary of what it shows below.
Repro status: 100% reproducible on Railway's build infra, 0% reproducible locally (same lockfile, same Bun version, same source tree, dozens of local builds, every one finishes in ~13–16s). This strongly points at something specific to Railway's build container/sandboxing rather than our code or config in isolation — though we don't rule out an interaction between the two.
What we ran: our normal build (tsc --noEmit && vite build) wrapped in a diagnostic script that:
Runs vite build --debug instead of plain vite build
Samples /sys/fs/cgroup/memory.current, memory.peak, memory.max, memory.events, cpu.stat, cpu.max, plus free -m and a filtered ps aux (vite/rollup/esbuild/node) every 10 seconds in the background during the build
Wraps the build in timeout --kill-after=10 300 so a hang hard-fails at exactly 300s instead of hanging indefinitely, and only exits 0 if dist/index.html actually exists afterward — so this could not accidentally ship a broken build to prod even if something went wrong
Deploy ID for this run: 897f97ab-7aff-472b-b8a7-d209c504f1e8. It failed as designed (exit 124), production was never touched.
Timeline (all UTC, 2026-10-06):
18:03:14 — tsc --noEmit finishes clean, exit 0
18:03:16 — vite build --debug starts, "building client environment for production..."
18:03:19 — the only build output we ever see: two Rollup warnings about comment placement in node_modules/zod/v4/core/regexes.js and node_modules/zod/v4/core/util.js ("comment cannot be interpreted due to the position of the comment")
Immediately after those warnings, the process stops producing any further output and never recovers. No further log lines, no progress, no crash — it just goes silent.
18:08:14 (300s later) — our timeout --kill-after=10 fires and SIGKILLs the process tree. Final ps sample confirms it was still alive and resource-consuming right up to the kill: node .../vite build --debug (pid 153) and its esbuild --service=0.27.7 --ping child (pid 196) both end up (zombied by the kill) rather than exiting on their own.
Memory (cgroup v2 memory.current / memory.peak, sampled every 10s):
time mem.current mem.peak
18:03:13.6 0.70 GB 1.42 GB
18:03:24.0 1.99 GB 2.00 GB
18:03:34.1 4.55 GB 4.55 GB
18:03:44.1 8.68 GB 8.69 GB
18:03:54.1 12.14 GB 12.14 GB
18:04:04 → 18:08:14 (final)
plateaus 11.8–12.0 GB
peak locks at 12.71 GB
Memory ramps roughly linearly from ~0.7GB to ~12GB in the ~40 seconds right as/after those zod warnings print, then plateaus and holds flat for the remaining 4+ minutes until we kill it — it never grows further, but also never completes, never frees, and memory.events shows zero oom/oom_kill the whole time. That plateau-not-crash pattern reads to us like something stuck in a loop or a blocked wait, not a runaway allocator heading for OOM. Machine had plenty of headroom either way: free -m shows ~58GB total RAM, ~34GB available at the moment of the kill, so this isn't memory pressure from the host.
CPU (cgroup v2 cpu.stat, usec):
time usage user sys sys % of total
18:03:13.6 9.4s 5.2s 4.3s 45%
18:03:54.1 64.1s 53.4s 10.8s 17% (lowest)
18:05:04.5 145.6s 74.7s 70.9s 49%
18:08:14.2 (final). 335.5s 103.5s 232.0s 69%
timatosystems
Thanks for the detailed questions — we built a one-off instrumented build command specifically to answer them cleanly, ran it on a throwaway deploy so it couldn't ship to production if it failed, and got a clean, complete capture of the hang on the first try. Full raw build log is attached (diag-v3-full-buildlog.txt, install through the timeout). Summary of what it shows below. Repro status: 100% reproducible on Railway's build infra, 0% reproducible locally (same lockfile, same Bun version, same source tree, dozens of local builds, every one finishes in ~13–16s). This strongly points at something specific to Railway's build container/sandboxing rather than our code or config in isolation — though we don't rule out an interaction between the two. What we ran: our normal build (tsc --noEmit && vite build) wrapped in a diagnostic script that: Runs vite build --debug instead of plain vite build Samples /sys/fs/cgroup/memory.current, memory.peak, memory.max, memory.events, cpu.stat, cpu.max, plus free -m and a filtered ps aux (vite/rollup/esbuild/node) every 10 seconds in the background during the build Wraps the build in timeout --kill-after=10 300 so a hang hard-fails at exactly 300s instead of hanging indefinitely, and only exits 0 if dist/index.html actually exists afterward — so this could not accidentally ship a broken build to prod even if something went wrong Deploy ID for this run: 897f97ab-7aff-472b-b8a7-d209c504f1e8. It failed as designed (exit 124), production was never touched. Timeline (all UTC, 2026-10-06): 18:03:14 — tsc --noEmit finishes clean, exit 0 18:03:16 — vite build --debug starts, "building client environment for production..." 18:03:19 — the only build output we ever see: two Rollup warnings about comment placement in node_modules/zod/v4/core/regexes.js and node_modules/zod/v4/core/util.js ("comment cannot be interpreted due to the position of the comment") Immediately after those warnings, the process stops producing any further output and never recovers. No further log lines, no progress, no crash — it just goes silent. 18:08:14 (300s later) — our timeout --kill-after=10 fires and SIGKILLs the process tree. Final ps sample confirms it was still alive and resource-consuming right up to the kill: node .../vite build --debug (pid 153) and its esbuild --service=0.27.7 --ping child (pid 196) both end up <defunct> (zombied by the kill) rather than exiting on their own. Memory (cgroup v2 memory.current / memory.peak, sampled every 10s): time mem.current mem.peak 18:03:13.6 0.70 GB 1.42 GB 18:03:24.0 1.99 GB 2.00 GB 18:03:34.1 4.55 GB 4.55 GB 18:03:44.1 8.68 GB 8.69 GB 18:03:54.1 12.14 GB 12.14 GB 18:04:04 → 18:08:14 (final) plateaus 11.8–12.0 GB peak locks at 12.71 GB Memory ramps roughly linearly from ~0.7GB to ~12GB in the ~40 seconds right as/after those zod warnings print, then plateaus and holds flat for the remaining 4+ minutes until we kill it — it never grows further, but also never completes, never frees, and memory.events shows zero oom/oom_kill the whole time. That plateau-not-crash pattern reads to us like something stuck in a loop or a blocked wait, not a runaway allocator heading for OOM. Machine had plenty of headroom either way: free -m shows ~58GB total RAM, ~34GB available at the moment of the kill, so this isn't memory pressure from the host. CPU (cgroup v2 cpu.stat, usec): time usage user sys sys % of total 18:03:13.6 9.4s 5.2s 4.3s 45% 18:03:54.1 64.1s 53.4s 10.8s 17% (lowest) 18:05:04.5 145.6s 74.7s 70.9s 49% 18:08:14.2 (final). 335.5s 103.5s 232.0s 69%
a month ago
Sorry, hit a character limit:
User-time growth nearly flatlines after the initial ramp (74.7s → 103.5s over the last ~4 minutes) while system-time keeps climbing hard in the same window (70.9s → 232.0s). The process spends more and more of its time in the kernel rather than doing user-space work as the hang drags on — consistent with the memory plateau: something is waiting/polling (syscalls, I/O, or a lock/futex) rather than actively computing or actively leaking.
Process tree at the moment of the kill (from our final ps sample, 18:08:14):
root 153 107%CPU 20.5%MEM Sl 5:14 node /app/node_modules/.bin/vite build --debug
root 196 0.3%CPU 0.0%MEM Sl 0:01 .../esbuild/node_modules/@esbuild/linux-x64/bin/esbuild --service=0.27.7 --ping
Worth flagging: across the samples, the esbuild service child's CPU% decayed toward ~0% early and stayed there, while the parent node/vite process stayed pinned above 100% CPU for the entire ~5 minutes. That reads like esbuild went idle (waiting on something) while vite's Node process kept spinning — maybe polling or busy-waiting on a response from that esbuild service process that never arrives. If you've seen this pattern before in your build sandboxing/virtualization layer interacting with Node's worker_threads or an IPC pipe/socket to a child service process, that would be a very useful lead for us.
cgroup v2 availability on your build containers, since you'd asked: memory.current, memory.peak, memory.max, memory.events, memory.stat, cpu.stat, cpu.max, and /proc/meminfo are all present and readable in the build container — no missing files, cgroup v2 confirmed. (For contrast, the same check run on our local machine is missing memory.current/peak/max/events — only memory.stat, cpu.stat, /proc/meminfo exist locally. Not claiming that's causal, just flagging the environment difference since you asked us to compare.)
Build machine specs observed: nproc = 32, free -m total ≈ 58GB RAM / 32GB swap, cpu.max = max 100000 (unlimited).
Config you asked for (unchanged since our last message, re-verified before sending):
packages/web/vite.config.ts — attached / pasted below
packages/web/package.json scripts — "build": "tsc --noEmit && vite build", nothing unusual
Custom Vite plugins: two, both attached
honoDevPlugin — dev-server-only (configureServer hook), does nothing during a production build, included for completeness
runableAnalyticsPlugin — does run during build, via transformIndexHtml, and uses jsdom to inject a tag into index.html. We haven't isolated this as the cause yet (next on our list to test — temporarily stub it out and see if the hang still reproduces), but flagging it now since it's the one plugin that touches the HTML output during the actual build step rather than just dev serving.
What we're asking: does the memory-plateau + rising-system-time + esbuild-idle-while-node-spins pattern match anything you've seen on your end — a known interaction between your build container/sandbox and Node's worker threads, or an esbuild service process losing its ping/pong with the parent? Happy to run further instrumented one-off builds on our end if there's a specific thing you'd like us to check next; we've got a safe, fail-closed way to do that now (this diagnostic run never touched production — we keep a known-good pre-built bypass on standby and only run instrumented builds as disposable one-off deploys).
Thanks again for staying on this with us.
timatosystems
Sorry, hit a character limit: User-time growth nearly flatlines after the initial ramp (74.7s → 103.5s over the last ~4 minutes) while system-time keeps climbing hard in the same window (70.9s → 232.0s). The process spends more and more of its time in the kernel rather than doing user-space work as the hang drags on — consistent with the memory plateau: something is waiting/polling (syscalls, I/O, or a lock/futex) rather than actively computing or actively leaking. Process tree at the moment of the kill (from our final ps sample, 18:08:14): root 153 107%CPU 20.5%MEM Sl 5:14 node /app/node_modules/.bin/vite build --debug root 196 0.3%CPU 0.0%MEM Sl 0:01 .../esbuild/node_modules/@esbuild/linux-x64/bin/esbuild --service=0.27.7 --ping Worth flagging: across the samples, the esbuild service child's CPU% decayed toward ~0% early and stayed there, while the parent node/vite process stayed pinned above 100% CPU for the entire ~5 minutes. That reads like esbuild went idle (waiting on something) while vite's Node process kept spinning — maybe polling or busy-waiting on a response from that esbuild service process that never arrives. If you've seen this pattern before in your build sandboxing/virtualization layer interacting with Node's worker_threads or an IPC pipe/socket to a child service process, that would be a very useful lead for us. cgroup v2 availability on your build containers, since you'd asked: memory.current, memory.peak, memory.max, memory.events, memory.stat, cpu.stat, cpu.max, and /proc/meminfo are all present and readable in the build container — no missing files, cgroup v2 confirmed. (For contrast, the same check run on our local machine is missing memory.current/peak/max/events — only memory.stat, cpu.stat, /proc/meminfo exist locally. Not claiming that's causal, just flagging the environment difference since you asked us to compare.) Build machine specs observed: nproc = 32, free -m total ≈ 58GB RAM / 32GB swap, cpu.max = max 100000 (unlimited). Config you asked for (unchanged since our last message, re-verified before sending): packages/web/vite.config.ts — attached / pasted below packages/web/package.json scripts — "build": "tsc --noEmit && vite build", nothing unusual Custom Vite plugins: two, both attached honoDevPlugin — dev-server-only (configureServer hook), does nothing during a production build, included for completeness runableAnalyticsPlugin — does run during build, via transformIndexHtml, and uses jsdom to inject a <script> tag into index.html. We haven't isolated this as the cause yet (next on our list to test — temporarily stub it out and see if the hang still reproduces), but flagging it now since it's the one plugin that touches the HTML output during the actual build step rather than just dev serving. What we're asking: does the memory-plateau + rising-system-time + esbuild-idle-while-node-spins pattern match anything you've seen on your end — a known interaction between your build container/sandbox and Node's worker threads, or an esbuild service process losing its ping/pong with the parent? Happy to run further instrumented one-off builds on our end if there's a specific thing you'd like us to check next; we've got a safe, fail-closed way to do that now (this diagnostic run never touched production — we keep a known-good pre-built bypass on standby and only run instrumented builds as disposable one-off deploys). Thanks again for staying on this with us.
a month ago
Here is the Build log
a month ago
Thanks — I’ve now reviewed the full attachment, and there’s an important runtime detail that changes which test I’d prioritize.
The key diagnostic lines are:
- node: command not found
-
- The build runs via bun run vite build --debug
-
- PATH includes /tmp/bun-node-b0a6feca5
-
- NODE and npm_node_execpath point to /tmp/bun-node-b0a6feca5
This strongly suggests Vite is running through Bun’s Node fallback rather than a separately installed Node.js runtime. The “node” process name and node-v24.3.0 version string alone don’t establish which runtime is executing it.
Before changing plugins, I’d test that directly:
- In the current environment, capture the runtime identity:
bun run node -e 'console.log(JSON.stringify({
execPath: process.execPath,
node: process.versions.node,
bun: process.versions.bun
}, null, 2))'If possible, also run readlink -f /proc//exe against the actual Vite process while it is running.
- On a disposable deployment, explicitly install a supported real Node.js version and invoke Vite with that executable:
/absolute/path/to/node ./node_modules/vite/bin/vite.js buildUse the verified Node binary’s actual path. Keep the source, installed dependency tree, Vite version, plugins, resource sampler, and timeout unchanged. Bun can remain the package manager; this test changes only the runtime executing Vite.
Please capture the local build’s runtime identity too. Matching Bun package-manager versions doesn’t necessarily mean local and remote Vite builds use the same JavaScript runtime.
The full log also answers my earlier memory-limit question: memory.max is “max”, all recorded memory.events counters remain zero, and cpu.max is “max 100000”. There’s no recorded cgroup OOM or CPU throttling here, so I wouldn’t prioritize raising limits based on this capture.
If the explicit Node build succeeds, we’ll have isolated a runtime-dependent difference, though not yet the underlying bug. If it still hangs, the analytics-plugin removal test is the next useful comparison.
The idle esbuild child and rising system CPU time are useful observations, but they don’t by themselves establish an IPC deadlock or a Railway sandbox fault.
Thanks — I’ve now reviewed the full attachment, and there’s an important runtime detail that changes which test I’d prioritize. The key diagnostic lines are: - node: command not found - - The build runs via bun run vite build --debug - - PATH includes /tmp/bun-node-b0a6feca5 - - NODE and npm_node_execpath point to /tmp/bun-node-b0a6feca5 This strongly suggests Vite is running through Bun’s Node fallback rather than a separately installed Node.js runtime. The “node” process name and node-v24.3.0 version string alone don’t establish which runtime is executing it. Before changing plugins, I’d test that directly: 1. In the current environment, capture the runtime identity: ```sh bun run node -e 'console.log(JSON.stringify({ execPath: process.execPath, node: process.versions.node, bun: process.versions.bun }, null, 2))' ``` If possible, also run readlink -f /proc/<vite-pid>/exe against the actual Vite process while it is running. 2. On a disposable deployment, explicitly install a supported real Node.js version and invoke Vite with that executable: ```sh /absolute/path/to/node ./node_modules/vite/bin/vite.js build ``` Use the verified Node binary’s actual path. Keep the source, installed dependency tree, Vite version, plugins, resource sampler, and timeout unchanged. Bun can remain the package manager; this test changes only the runtime executing Vite. Please capture the local build’s runtime identity too. Matching Bun package-manager versions doesn’t necessarily mean local and remote Vite builds use the same JavaScript runtime. The full log also answers my earlier memory-limit question: memory.max is “max”, all recorded memory.events counters remain zero, and cpu.max is “max 100000”. There’s no recorded cgroup OOM or CPU throttling here, so I wouldn’t prioritize raising limits based on this capture. If the explicit Node build succeeds, we’ll have isolated a runtime-dependent difference, though not yet the underlying bug. If it still hangs, the analytics-plugin removal test is the next useful comparison. The idle esbuild child and rising system CPU time are useful observations, but they don’t by themselves establish an IPC deadlock or a Railway sandbox fault.
a month ago
Ran your 3 tests, but only in a local sandbox with the same repo checkout — not actual Railway infra. Flagging that gap explicitly since it matters for #2.
-
Runtime identity (bun run node -e '...'): execPath: "/usr/bin/node", node: "26.3.0". process.versions.bun came back undefined in that shim context (bun key not printed).
-
readlink -f /proc//exe on the live Vite process, captured mid-build twice:
Via the real command (bun run vite build): spawned Vite process → exe=/usr/bin/node (real binary). /proc//environ shows npm_execpath=/usr/lib/node_modules/bun/bin/bun.exe, NODE=/usr/bin/node, npm_node_execpath=/usr/bin/node.
This is the critical difference from your logs: yours show NODE/npm_node_execpath pointing at /tmp/bun-node-b0a6feca5..., mine resolves straight to /usr/bin/node with no /tmp/bun-node-* on PATH at all. My sandbox doesn't reproduce the condition you flagged, so a clean run here does not rule out a bug in Bun's /tmp/bun-node-* fallback specific to Railway's base image. Treat everything below as supporting evidence, not a direct answer to that question.
- Explicit-Node build (/usr/bin/node ./node_modules/vite/bin/vite.js build), source/deps/Vite version/plugins unchanged:
Ran 3x back to back: every run clean in ~10-12s (11.95s, 10.26s, ~11s), full dist/ output, same mapbox-*.js chunk warning you'd expect (>500kB, raw ~1.84MB / gzip ~510KB).
Confirmed mid-build the process was really under /usr/bin/node (pid captured, exe=/usr/bin/node, cmd /usr/bin/node ./node_modules/vite/bin/vite.js build), not a shim.
Also reran the normal path (bun run vite build) 3x for comparison: same result, no hang/error, ~11s each, identical dist output, process tree = bash wrapper → node .../vite.js build → esbuild --service=0.27.5 --ping child.
Vite resolves to 7.3.2 consistently here (not drifting), vs the 7.3.6/7.3.7 drift you saw earlier — may just mean the exact-version pin from before is holding in this checkout; flagging as another local-vs-Railway difference worth noting.
- Resource limits: agreed, not chasing that — your memory.max/memory.events/cpu.max numbers already rule out OOM/throttle.
Where I'm stuck: I can't actually run #3 "on a disposable deployment" from my side — that needs Railway's own infra where /tmp/bun-node-b0a6feca5 resolution actually happens, which my sandbox doesn't reproduce. Can you (or point me to the right Railway CLI/API flow to) spin up that throwaway deploy with the build command swapped to the explicit-Node invocation, so we get the real answer instead of a sandbox proxy? Happy to push a branch with that build command change if useful — just say which.
If the explicit-Node build still hangs for real on Railway, we'll go straight to pulling the analytics plugin next, per your suggestion.
timatosystems
Ran your 3 tests, but only in a local sandbox with the same repo checkout — not actual Railway infra. Flagging that gap explicitly since it matters for #2. 1) Runtime identity (bun run node -e '...'): execPath: "/usr/bin/node", node: "26.3.0". process.versions.bun came back undefined in that shim context (bun key not printed). 2) readlink -f /proc/<pid>/exe on the live Vite process, captured mid-build twice: Via the real command (bun run vite build): spawned Vite process → exe=/usr/bin/node (real binary). /proc/<pid>/environ shows npm_execpath=/usr/lib/node_modules/bun/bin/bun.exe, NODE=/usr/bin/node, npm_node_execpath=/usr/bin/node. This is the critical difference from your logs: yours show NODE/npm_node_execpath pointing at /tmp/bun-node-b0a6feca5..., mine resolves straight to /usr/bin/node with no /tmp/bun-node-* on PATH at all. My sandbox doesn't reproduce the condition you flagged, so a clean run here does not rule out a bug in Bun's /tmp/bun-node-* fallback specific to Railway's base image. Treat everything below as supporting evidence, not a direct answer to that question. 3) Explicit-Node build (/usr/bin/node ./node_modules/vite/bin/vite.js build), source/deps/Vite version/plugins unchanged: Ran 3x back to back: every run clean in ~10-12s (11.95s, 10.26s, ~11s), full dist/ output, same mapbox-*.js chunk warning you'd expect (>500kB, raw ~1.84MB / gzip ~510KB). Confirmed mid-build the process was really under /usr/bin/node (pid captured, exe=/usr/bin/node, cmd /usr/bin/node ./node_modules/vite/bin/vite.js build), not a shim. Also reran the normal path (bun run vite build) 3x for comparison: same result, no hang/error, ~11s each, identical dist output, process tree = bash wrapper → node .../vite.js build → esbuild --service=0.27.5 --ping child. Vite resolves to 7.3.2 consistently here (not drifting), vs the 7.3.6/7.3.7 drift you saw earlier — may just mean the exact-version pin from before is holding in this checkout; flagging as another local-vs-Railway difference worth noting. 4) Resource limits: agreed, not chasing that — your memory.max/memory.events/cpu.max numbers already rule out OOM/throttle. Where I'm stuck: I can't actually run #3 "on a disposable deployment" from my side — that needs Railway's own infra where /tmp/bun-node-b0a6feca5 resolution actually happens, which my sandbox doesn't reproduce. Can you (or point me to the right Railway CLI/API flow to) spin up that throwaway deploy with the build command swapped to the explicit-Node invocation, so we get the real answer instead of a sandbox proxy? Happy to push a branch with that build command change if useful — just say which. If the explicit-Node build still hangs for real on Railway, we'll go straight to pulling the analytics plugin next, per your suggestion.
a month ago
Thanks for explicitly flagging that these were local tests. They confirm that both local invocation paths use real Node, but they don’t yet test the fallback-runtime condition seen in the Railway log.
One additional difference: your local esbuild service reports 0.27.5, whereas the Railway capture reports 0.27.7. So matching Vite 7.3.2 alone does not establish an identical installed dependency tree. The earlier Vite drift is already resolved; I wouldn’t reopen that issue.
I also need to clarify the access boundary: I’m a community participant, not Railway staff, and I don’t have access to your Railway project. I can help prepare the test, but I can’t create a deployment in your account.
Yes, a diagnostic branch would be useful. The next test should run in a separate Railway service connected to that branch, with the same relevant build settings and Root Directory, without changing the production service.
There are two necessary parts:
-
Explicitly provision a supported real Node.js runtime in the build environment. Changing the command to “node ...” alone won’t work if Node is still absent, as it was in the captured build.
-
In the configuration file actually governing that diagnostic service, invoke Vite directly with the verified Node executable, keeping the source, plugins, dependency installation and timeout unchanged.
Before Vite starts, log only the runtime identity, executable path, and resolved Vite/esbuild versions. Please don’t dump the full environment again.
Could you share the current railway.json and any nixpacks.toml, with secrets removed? Your log identifies Nixpacks as the active builder. With those files, we can make the Node setup and build-command change concrete rather than give you a command that assumes Node is installed.
Also, have the credentials exposed in the previous attachment been revoked and the attachment removed?
If a real-Node build succeeds on Railway, that gives us a strong runtime-dependent result. If it still hangs, we can test removal of the analytics plugin next, changing one variable at a time.
One small correction on resources: the recorded counters show no OOM or CPU throttling in the sampled cgroup. That makes those explanations less likely, but doesn’t categorically rule out every form of memory pressure.
Thanks for explicitly flagging that these were local tests. They confirm that both local invocation paths use real Node, but they don’t yet test the fallback-runtime condition seen in the Railway log. One additional difference: your local esbuild service reports 0.27.5, whereas the Railway capture reports 0.27.7. So matching Vite 7.3.2 alone does not establish an identical installed dependency tree. The earlier Vite drift is already resolved; I wouldn’t reopen that issue. I also need to clarify the access boundary: I’m a community participant, not Railway staff, and I don’t have access to your Railway project. I can help prepare the test, but I can’t create a deployment in your account. Yes, a diagnostic branch would be useful. The next test should run in a separate Railway service connected to that branch, with the same relevant build settings and Root Directory, without changing the production service. There are two necessary parts: 1. Explicitly provision a supported real Node.js runtime in the build environment. Changing the command to “node ...” alone won’t work if Node is still absent, as it was in the captured build. 2. In the configuration file actually governing that diagnostic service, invoke Vite directly with the verified Node executable, keeping the source, plugins, dependency installation and timeout unchanged. Before Vite starts, log only the runtime identity, executable path, and resolved Vite/esbuild versions. Please don’t dump the full environment again. Could you share the current railway.json and any nixpacks.toml, with secrets removed? Your log identifies Nixpacks as the active builder. With those files, we can make the Node setup and build-command change concrete rather than give you a command that assumes Node is installed. Also, have the credentials exposed in the previous attachment been revoked and the attachment removed? If a real-Node build succeeds on Railway, that gives us a strong runtime-dependent result. If it still hangs, we can test removal of the analytics plugin next, changing one variable at a time. One small correction on resources: the recorded counters show no OOM or CPU throttling in the sampled cgroup. That makes those explanations less likely, but doesn’t categorically rule out every form of memory pressure.
a month ago
Credentials question: I went back through everything attached or pasted in this thread. No actual secret values appear anywhere — the only env-var-shaped thing in the whole history is a Docker build-log lint warning from an unrelated, months-old session that named BETTER_AUTH_SECRET / DATABASE_AUTH_TOKEN as flagged ARG/ENV usage (just the names, not values, and it predates this investigation). Every diagnostic script since then (v2, v3, and this real-Node one) only logs versions, paths, exit codes, and cgroup stats — never a printenv or full env dump. Nothing to revoke.
Diagnostic branch: diag/real-node-build-test, Root Directory packages/web (same as production), separate service, nothing touching prod.
nixpacks.toml ([phases.setup]):
text
nixPkgs = ["bun", "nodejs_22"]
Everything else (install phase, start command) unchanged from main.
railway.json (build.buildCommand):
text
bun install --frozen-lockfile && bash diag/debug-build-realnode.sh
Install step and the 300s hard timeout are unchanged. Only the build invocation changed: Vite is called directly through the resolved real node binary (timeout --kill-after=10 300 "$NODE_BIN" ./node_modules/vite/bin/vite.js build), not bun run vite build.
Result — real Node runtime confirmed, hang still happens:
text
node resolved from PATH: /root/.nix-profile/bin/node
node exe (readlink -f): /nix/store/cwxxy2is6hb4dcqsx8rhnvwic3hfzpcd-nodejs-22.19.0/bin/node
node --version: v22.19.0
vite: 7.3.2
esbuild: 0.27.7
tsc exit: 0
...
vite build exit: 124 (HANG CONFIRMED even with a real Node runtime)
This is a real, Nix-provisioned Node 22.19.0 — not Bun's /tmp/bun-node-* shim. And esbuild now resolves to 0.27.7, matching your earlier capture exactly, so the dependency-tree-drift question is also closed: identical runtime, identical esbuild, build still hangs at the same tsc: 0 → transform phase. Rules out both the fallback-runtime theory and the dependency-drift theory.
One possibly-irrelevant data point: the last build-time output before the hang was Rollup's "comment annotation" warning on two files in zod/v4/core (regexes.js, util.js) — no module-transformed count ever printed after that. Could be coincidental ordering, could be where it's stuck. Flagging it since I can't tell from stdout alone.
Per your earlier suggestion, next test on my side is removing the analytics plugin and re-running this same script, one variable at a time.
timatosystems
Credentials question: I went back through everything attached or pasted in this thread. No actual secret values appear anywhere — the only env-var-shaped thing in the whole history is a Docker build-log lint warning from an unrelated, months-old session that named BETTER_AUTH_SECRET / DATABASE_AUTH_TOKEN as flagged ARG/ENV usage (just the names, not values, and it predates this investigation). Every diagnostic script since then (v2, v3, and this real-Node one) only logs versions, paths, exit codes, and cgroup stats — never a printenv or full env dump. Nothing to revoke. Diagnostic branch: diag/real-node-build-test, Root Directory packages/web (same as production), separate service, nothing touching prod. nixpacks.toml ([phases.setup]): text nixPkgs = ["bun", "nodejs_22"] Everything else (install phase, start command) unchanged from main. railway.json (build.buildCommand): text bun install --frozen-lockfile && bash diag/debug-build-realnode.sh Install step and the 300s hard timeout are unchanged. Only the build invocation changed: Vite is called directly through the resolved real node binary (timeout --kill-after=10 300 "$NODE_BIN" ./node_modules/vite/bin/vite.js build), not bun run vite build. Result — real Node runtime confirmed, hang still happens: text node resolved from PATH: /root/.nix-profile/bin/node node exe (readlink -f): /nix/store/cwxxy2is6hb4dcqsx8rhnvwic3hfzpcd-nodejs-22.19.0/bin/node node --version: v22.19.0 vite: 7.3.2 esbuild: 0.27.7 tsc exit: 0 ... vite build exit: 124 (HANG CONFIRMED even with a real Node runtime) This is a real, Nix-provisioned Node 22.19.0 — not Bun's /tmp/bun-node-* shim. And esbuild now resolves to 0.27.7, matching your earlier capture exactly, so the dependency-tree-drift question is also closed: identical runtime, identical esbuild, build still hangs at the same tsc: 0 → transform phase. Rules out both the fallback-runtime theory and the dependency-drift theory. One possibly-irrelevant data point: the last build-time output before the hang was Rollup's "comment annotation" warning on two files in zod/v4/core (regexes.js, util.js) — no module-transformed count ever printed after that. Could be coincidental ordering, could be where it's stuck. Flagging it since I can't tell from stdout alone. Per your earlier suggestion, next test on my side is removing the analytics plugin and re-running this same script, one variable at a time.
a month ago
Full tally across every real-Node attempt on Railway's actual builder this session:
Commit Analytics plugin Result
a6a861e present (original, with jsdom) hang
dae18fc fully removed success
f28d64b present (jsdom-free rewrite) hang
e682f00 same as f28d64b, rebuilt hang
cd7c193 fully removed, file deleted hang
Two runs with the plugin fully absent gave opposite results. That rules the plugin out as the deciding factor — dae18fc succeeding was most likely luck, not a fix. 4 hangs out of 5 real-Node attempts, no code variable reliably correlates. This points away from "something in our build graph" and toward intermittent resource contention or stalling on Railway's shared builder itself — something only visible from their side (CPU steal, I/O wait, network stall fetching a registry mid-build), not from our cgroup snapshots.
How can we get Railway itself their own infra-side metrics for these specific build IDs during the hang windows?
a month ago
Thanks — these Railway-side tests narrow things down. The hang also occurs under real Node, so Bun’s fallback runtime isn’t required to trigger it. Removing the analytics plugin is not a reliable fix either, given the mixed results.
I agree that asking Railway staff to investigate is now appropriate. I’d keep shared-builder contention as a hypothesis rather than a confirmed cause, though. Also, matching Vite and esbuild versions doesn’t establish that the entire installed dependency tree is identical.
For an actionable escalation, could you provide:
- Deployment/build IDs or links for all five attempts, identifying the successful one.
-
- Start times and approximate hang windows in UTC.
-
- The corresponding build logs, with any sensitive values redacted.
-
- The diagnostic service’s region, builder configuration, and commit SHA for each attempt.
Railway team: could you correlate those attempts with any available builder-side telemetry, particularly host memory pressure, CPU contention, disk/I/O stalls, or worker errors around the hang windows? The captured cgroup counters showed no OOM events or CPU throttling, but they don’t establish what was happening on the underlying host.
If you run further comparisons, the most useful would be repeated builds of one unchanged commit with the same configuration, recording the IDs and outcomes. That would give us a cleaner comparison than additional plugin changes.
I’m a community participant, so I can help interpret the evidence, but I can’t access those internal metrics. The next useful step is a staff-side comparison of the successful and stalled builds.
Thanks — these Railway-side tests narrow things down. The hang also occurs under real Node, so Bun’s fallback runtime isn’t required to trigger it. Removing the analytics plugin is not a reliable fix either, given the mixed results. I agree that asking Railway staff to investigate is now appropriate. I’d keep shared-builder contention as a hypothesis rather than a confirmed cause, though. Also, matching Vite and esbuild versions doesn’t establish that the entire installed dependency tree is identical. For an actionable escalation, could you provide: - Deployment/build IDs or links for all five attempts, identifying the successful one. - - Start times and approximate hang windows in UTC. - - The corresponding build logs, with any sensitive values redacted. - - The diagnostic service’s region, builder configuration, and commit SHA for each attempt. Railway team: could you correlate those attempts with any available builder-side telemetry, particularly host memory pressure, CPU contention, disk/I/O stalls, or worker errors around the hang windows? The captured cgroup counters showed no OOM events or CPU throttling, but they don’t establish what was happening on the underlying host. If you run further comparisons, the most useful would be repeated builds of one unchanged commit with the same configuration, recording the IDs and outcomes. That would give us a cleaner comparison than additional plugin changes. I’m a community participant, so I can help interpret the evidence, but I can’t access those internal metrics. The next useful step is a staff-side comparison of the successful and stalled builds.
a month ago
Repo / branch / Root Directory: timatosystems/faithshield, branch diag/real-node-build-test, Root Directory packages/web — same as production.
Diagnostic service identifier: I don't have dashboard/API access to pull the deployment UUIDs directly, but the build logs' Docker cache-mount line carries what's almost certainly the service ID: --mount=type=cache,id=s/638379d7-b931-4de0-abb1-d346900b5941-node_modules/cache,.... Can you confirm that's the service ID on your side and pull the five actual deployment IDs from it? That'll be more reliable than me reconstructing them from timestamps.
Builder: production-builderv3-us-west1-nd5v (Metal builder, us-west1) — confirmed in the log for the one run where that header line was visible (the success). I didn't capture that line on the other four pastes, but it's the same service, so it's very likely the same builder/pool each time — worth confirming on your end rather than assumed.
All five attempts, in order, same diagnostic service, nixpacks v1.41.0:
Commit Analytics plugin state Build start (UTC) Total build-step duration Result
a6a861e present, original (jsdom) 2026-10-07T20:03:16.994Z not captured in paste; hard-timed-out at the 300s mark + 10s kill-after → ends ~20:08:27–20:08:37 hang, exit 124
dae18fc fully removed not captured (sequenced between the two above — window ~20:08–20:26) 4m 45s success — dist/index.html present
f28d64b present, jsdom-free rewrite 2026-10-07T20:26:58.108Z 5m 4s → ends ~20:32:02 hang, exit 124
e682f00 same code as f28d64b, empty commit to force rebuild 2026-10-07T21:14:07.780Z 5m 3s → ends ~21:19:11 hang, exit 124
cd7c193 fully removed, file deleted 2026-10-07T21:23:55.933Z 5m 20s → ends ~21:29:16 hang, exit 124
That e682f00 row is actually the "repeated build of one unchanged commit" you asked for — I couldn't hit redeploy on f28d64b directly (the service never reached Deploy stage, so no redeploy button existed), so I pushed an empty commit on identical code instead. Same result both times: hang, same spot, nearly identical duration (5m4s vs 5m3s).
The detail I think matters most, and hadn't fully registered until I laid these side by side: the one success wasn't fast. Locally this build (tsc + vite build) finishes in ~10–15s. On Railway, the successful run still took 4m45s — barely under the 4 hung runs' 5m0s–5m20s. That doesn't look like "fixed" vs. "broken." It looks like every attempt runs severely slow regardless of outcome, with 4 of 5 additionally crossing the 300s cutoff before finishing. That's a stronger argument for builder-side contention than anything I had before — a deterministic code-level deadlock wouldn't produce a "success" that's still 28x slower than local.
Full stdout for all 5, redacted — there's nothing sensitive in any of them (consistent with my earlier answer on the credentials question: these scripts only ever log versions, paths, exit codes, never env vars):
All five hit the identical point — tsc exit: 0 → vite build → transforming... → Rollup's two @PURE comment warnings on node_modules/zod/v4/core/regexes.js and util.js → nothing further until timeout. Real Node confirmed every time (/root/.nix-profile/bin/node → /nix/store/.../nodejs-22.19.0/bin/node, v22.19.0), esbuild consistently 0.27.7 (matches your capture exactly).
Conclusion: this rules out both the Bun-fallback-runtime theory and the esbuild-drift theory, and now also the analytics-plugin theory — two "fully removed" runs gave opposite results (dae18fc success, cd7c193 hang), so the one success was most likely builder-side luck, not a fix. I'm not going to keep editing app code chasing a variable that isn't the cause.
What we need from your side: correlate these five attempts (service 638379d7-b931-4de0-abb1-d346900b5941, builder production-builderv3-us-west1-nd5v, 2026-10-07, timestamps above) against host-level telemetry during those windows — CPU steal, memory pressure, disk/I/O stalls, network stalls on registry fetches, worker errors. Our cgroup counters showed no OOM or throttling, but that's container-internal and won't show host-level contention.
Happy to run more repeated builds of one unchanged commit if that's useful data while you look — just say how many.
timatosystems
Repo / branch / Root Directory: timatosystems/faithshield, branch diag/real-node-build-test, Root Directory packages/web — same as production. Diagnostic service identifier: I don't have dashboard/API access to pull the deployment UUIDs directly, but the build logs' Docker cache-mount line carries what's almost certainly the service ID: --mount=type=cache,id=s/638379d7-b931-4de0-abb1-d346900b5941-node_modules/cache,.... Can you confirm that's the service ID on your side and pull the five actual deployment IDs from it? That'll be more reliable than me reconstructing them from timestamps. Builder: production-builderv3-us-west1-nd5v (Metal builder, us-west1) — confirmed in the log for the one run where that header line was visible (the success). I didn't capture that line on the other four pastes, but it's the same service, so it's very likely the same builder/pool each time — worth confirming on your end rather than assumed. All five attempts, in order, same diagnostic service, nixpacks v1.41.0: Commit Analytics plugin state Build start (UTC) Total build-step duration Result a6a861e present, original (jsdom) 2026-10-07T20:03:16.994Z not captured in paste; hard-timed-out at the 300s mark + 10s kill-after → ends ~20:08:27–20:08:37 hang, exit 124 dae18fc fully removed not captured (sequenced between the two above — window ~20:08–20:26) 4m 45s success — dist/index.html present f28d64b present, jsdom-free rewrite 2026-10-07T20:26:58.108Z 5m 4s → ends ~20:32:02 hang, exit 124 e682f00 same code as f28d64b, empty commit to force rebuild 2026-10-07T21:14:07.780Z 5m 3s → ends ~21:19:11 hang, exit 124 cd7c193 fully removed, file deleted 2026-10-07T21:23:55.933Z 5m 20s → ends ~21:29:16 hang, exit 124 That e682f00 row is actually the "repeated build of one unchanged commit" you asked for — I couldn't hit redeploy on f28d64b directly (the service never reached Deploy stage, so no redeploy button existed), so I pushed an empty commit on identical code instead. Same result both times: hang, same spot, nearly identical duration (5m4s vs 5m3s). The detail I think matters most, and hadn't fully registered until I laid these side by side: the one success wasn't fast. Locally this build (tsc + vite build) finishes in ~10–15s. On Railway, the successful run still took 4m45s — barely under the 4 hung runs' 5m0s–5m20s. That doesn't look like "fixed" vs. "broken." It looks like every attempt runs severely slow regardless of outcome, with 4 of 5 additionally crossing the 300s cutoff before finishing. That's a stronger argument for builder-side contention than anything I had before — a deterministic code-level deadlock wouldn't produce a "success" that's still 28x slower than local. Full stdout for all 5, redacted — there's nothing sensitive in any of them (consistent with my earlier answer on the credentials question: these scripts only ever log versions, paths, exit codes, never env vars): All five hit the identical point — tsc exit: 0 → vite build → transforming... → Rollup's two @__PURE__ comment warnings on node_modules/zod/v4/core/regexes.js and util.js → nothing further until timeout. Real Node confirmed every time (/root/.nix-profile/bin/node → /nix/store/.../nodejs-22.19.0/bin/node, v22.19.0), esbuild consistently 0.27.7 (matches your capture exactly). Conclusion: this rules out both the Bun-fallback-runtime theory and the esbuild-drift theory, and now also the analytics-plugin theory — two "fully removed" runs gave opposite results (dae18fc success, cd7c193 hang), so the one success was most likely builder-side luck, not a fix. I'm not going to keep editing app code chasing a variable that isn't the cause. What we need from your side: correlate these five attempts (service 638379d7-b931-4de0-abb1-d346900b5941, builder production-builderv3-us-west1-nd5v, 2026-10-07, timestamps above) against host-level telemetry during those windows — CPU steal, memory pressure, disk/I/O stalls, network stalls on registry fetches, worker errors. Our cgroup counters showed no OOM or throttling, but that's container-internal and won't show host-level contention. Happy to run more repeated builds of one unchanged commit if that's useful data while you look — just say how many.
a month ago
Thanks — the timeline is useful, and I agree that further speculative application-code changes aren’t justified right now.
First, an important access clarification: I’m an independent community participant, not Railway staff. I cannot confirm that UUID, retrieve your deployment IDs, or inspect host telemetry. My previous request for infrastructure correlation was addressed to Railway staff reading this thread.
The cache-mount UUID is a useful lead, but I can’t verify that it is the service ID. Likewise, the builder hostname is confirmed for the successful attempt only; using the same service doesn’t establish that every attempt ran on the same host.
The 4m45s successful run is worth investigating, with one important distinction: is that the duration of the entire build step—including installation, diagnostics and TypeScript—or specifically the Vite invocation? Since the 300s timeout wraps Vite, we need comparable timings before concluding that the successful Vite build narrowly beat that cutoff. Exit 124 confirms that the timeout was reached, not whether the process was deadlocked or simply progressing very slowly.
The unchanged-source retry is useful evidence. The results establish that the failure can occur with real Node and without the analytics plugin. They don’t yet identify host contention as the cause, although it remains a reasonable hypothesis.
Railway staff: could you help identify the diagnostic service and deployments from the repository, branch, cache identifier and UTC windows above, then compare the successful and timed-out attempts against available builder telemetry?
If you don’t have dashboard/API access yourself, please ask whoever manages the Railway project to supply the deployment links and raise this through the support channel available to that account, linking this thread.
I wouldn’t request more builds just to accumulate failures at this stage. The existing five attempts are a useful escalation package; the missing pieces are confirmed deployment identifiers, comparable phase timings, and Railway-side investigation
Thanks — the timeline is useful, and I agree that further speculative application-code changes aren’t justified right now. First, an important access clarification: I’m an independent community participant, not Railway staff. I cannot confirm that UUID, retrieve your deployment IDs, or inspect host telemetry. My previous request for infrastructure correlation was addressed to Railway staff reading this thread. The cache-mount UUID is a useful lead, but I can’t verify that it is the service ID. Likewise, the builder hostname is confirmed for the successful attempt only; using the same service doesn’t establish that every attempt ran on the same host. The 4m45s successful run is worth investigating, with one important distinction: is that the duration of the entire build step—including installation, diagnostics and TypeScript—or specifically the Vite invocation? Since the 300s timeout wraps Vite, we need comparable timings before concluding that the successful Vite build narrowly beat that cutoff. Exit 124 confirms that the timeout was reached, not whether the process was deadlocked or simply progressing very slowly. The unchanged-source retry is useful evidence. The results establish that the failure can occur with real Node and without the analytics plugin. They don’t yet identify host contention as the cause, although it remains a reasonable hypothesis. **Railway staff:** could you help identify the diagnostic service and deployments from the repository, branch, cache identifier and UTC windows above, then compare the successful and timed-out attempts against available builder telemetry? If you don’t have dashboard/API access yourself, please ask whoever manages the Railway project to supply the deployment links and raise this through the support channel available to that account, linking this thread. I wouldn’t request more builds just to accumulate failures at this stage. The existing five attempts are a useful escalation package; the missing pieces are confirmed deployment identifiers, comparable phase timings, and Railway-side investigation
a month ago
Ran the updated script (commit b043006) on Railway's builder just now. Got a clean hang on exit 124, and this time the phase timing actually tells us something.
tsc phase: 0.298s (05:58:33.934Z → 05:58:34.232Z) — confirmed not the problem.
vite build phase: started 05:58:34.233Z. Config loaded fine (~714ms). Then DEBUG=vite:* logged exactly four module resolutions, all within 1.5ms of each other, at 05:58:35.26x Z:
vite:resolve 0.25ms tailwindcss -> null
vite:resolve 0.47ms tw-animate-css -> null
vite:resolve 1.49ms tailwindcss -> /app/node_modules/tailwindcss/index.css
vite:resolve 1.55ms tw-animate-css -> /app/node_modules/tw-animate-css/dist/tw-animate.css
And then nothing. No further vite:resolve, vite:transform, vite:load — total silence in Vite's own trace output for 4m59s, while the heartbeat kept ticking every 10s right up to the 300s hard timeout (vite build end: 06:03:34.451Z, exit 124 — exactly 300.218s after vite build start, so it's the timeout firing, not Vite exiting on its own).
So this isn't "crawling slowly through a big module graph" — it goes dead silent immediately after Tailwind v4's content-scan plugin kicks in (@tailwindcss/vite:scan, enforce: 'pre', confirmed present in the resolved plugin list in this log). That plugin walks the project filesystem to find class names for zero-config detection, and that's exactly where the trace stops. Locally this same step is near-instant (full local build is 18.84s end to end with the same DEBUG=vite:* overhead). On Railway's builder it just stops producing any Vite-level output and never comes back, with no error.
That reads to me like a stall below the Node/Vite layer entirely — a filesystem walk (readdir/stat) that doesn't return on whatever Railway's builder has mounted at /app for this service, not anything in our module graph or app code. The env dump this run also confirmed the service ID I'd guessed earlier: RAILWAY_SERVICE_ID: 638379d7-b931-4de0-abb1-d346900b5941, commit b0430063dd4ffa8ad41e5e2115f52665fdb8027c, builder region us-west1.
Appreciate the correction last time on access boundaries — makes sense, and I get that pulling deployment IDs or host-side I/O traces isn't something you can do from where you sit. I'm filing a ticket with Railway support directly with this exact timing breakdown and the service/commit IDs so they can correlate it against their own host telemetry for that build. Will post back here if they find something, in case it's useful to anyone else hitting a silent hang on this builder.
a month ago
So how does one get an actual Railway support person to jump in on this? The system won't let me submit a ticket because I already have this thread going...
a month ago
Your existing Central Station thread is already the right Railway support channel; you shouldn’t need to open a duplicate ticket. I’m still only a community participant and can’t bring a staff member into the thread myself.
Railway’s support policy says staff responses aren’t guaranteed on Trial/Free/Hobby plans. Pro accounts receive direct help through Central Station, usually within 72 hours. If you’re on Pro and the interface blocks a separate private request because this thread exists, I’d ask for staff review here, noting your plan and that the issue now needs builder-side investigation.
The new timing is valuable:
tsccompleted in 0.298s, while the Vite process remained silent until its 300-second timeout. The Tailwind scan is a useful place to investigate, but the trace alone doesn’t prove which filesystem call or plugin operation stalled.>
Railway team: could someone review the build for service
638379d7-b931-4de0-abb1-d346900b5941, commitb0430063dd4ffa8ad41e5e2115f52665fdb8027c, inus-west1on 2026-10-08 from approximately 05:58:34 to 06:03:34 UTC? The remaining question is what the builder/process was doing during that interval, which we can’t determine from the customer-visible Vite logs.>
Your existing Central Station thread is already the right Railway support channel; you shouldn’t need to open a duplicate ticket. I’m still only a community participant and can’t bring a staff member into the thread myself. > > Railway’s support policy says staff responses aren’t guaranteed on Trial/Free/Hobby plans. Pro accounts receive direct help through Central Station, usually within 72 hours. If you’re on Pro and the interface blocks a separate private request because this thread exists, I’d ask for staff review **here**, noting your plan and that the issue now needs builder-side investigation. > > The new timing is valuable: `tsc` completed in 0.298s, while the Vite process remained silent until its 300-second timeout. The Tailwind scan is a useful place to investigate, but the trace alone doesn’t prove which filesystem call or plugin operation stalled.> > > **Railway team:** could someone review the build for service `638379d7-b931-4de0-abb1-d346900b5941`, commit `b0430063dd4ffa8ad41e5e2115f52665fdb8027c`, in `us-west1` on 2026-10-08 from approximately 05:58:34 to 06:03:34 UTC? The remaining question is what the builder/process was doing during that interval, which we can’t determine from the customer-visible Vite logs.>
a month ago
Your staff review "here" link isn't clickable, FYI. Appreciate the help
a month ago
Still having these issues? I'm assuming you've sent over the same errors you're encountering to the AI agent you used that has access to your codebase to search for solutions?
jack
Still having these issues? I'm assuming you've sent over the same errors you're encountering to the AI agent you used that has access to your codebase to search for solutions?
a month ago
Yes, still having these issues. I think given the sheer amount of back and forth testing and re-testing we have already done, you guys are due to answer the questions we have asked on this thread. I also upgraded my account to get support from you, not an AI agent.
a month ago
Sorry, jack, I thought you were with Railway. My apologies. I'm frustrated that Railway isn't responding and won't let me submit a ticket since I have this post already. It's incredibly frustrating since the issue seems definitively on their side.