My Railway service hermes-agent is crashing during startup.
noneyabiznz
PROOP

6 days ago

Error: s6-applyuidgid: fatal: unable to exec hermes: No such file or directory

The service uses the Docker image nousresearch/hermes-agent and has a persistent volume mounted at /opt/data.

I need help recovering service access and verifying/backing up the persistent volume BEFORE any further deployment or modification.

Please do not delete, detach, replace, or reset the volume.

Can you help me inspect the failed container's executable path and recover the service while preserving all stored files?

$10 Bounty

4 Replies

Railway
BOT

6 days ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway • 6 days ago


Try selecting your service, press Cmd/Ctrl + K, then deploy source image.


mxiand
FREE

12 hours ago

Please preserve the /opt/data volume before redeploying. That is the official Hermes data directory; the application and virtual environment live separately under /opt/hermes. This launch error does not itself establish that your stored data is damaged.

There is a related historical PATH issue: NousResearch/hermes-agent#18673, closed June 1, reports that the hermes executable was not discoverable in some Docker execution contexts. The current Dockerfile explicitly exports /opt/hermes/bin and /opt/hermes/.venv/bin on PATH. This is a useful lead, not proof that your deployed image has that exact defect: ENOENT can also mean a missing launcher interpreter or broken symlink.

Before changing the service:

  1. Record the currently deployed image tag/digest, Start Command and any user-defined PATH override. Do not publish environment dumps or .env contents.
  2. Use Railway's volume-file tooling to inspect/download the existing volume without running Hermes: select the correct project/environment and volume, then railway volume files --volume <volume-id> list / and railway volume files --volume <volume-id> download / ./hermes-volume-backup. Keep that backup encrypted and private; it can contain API keys and session data. If access is unavailable while crashed, ask Railway for a supported recovery/export path before any deployment change.
  3. With the same image digest, inspect the launcher and its interpreter separately from the production volume. If the launcher exists but is absent from PATH, correct the deployment's PATH override or use a release verified to include the upstream fix after preservation. If the launcher/interpreter is missing from the image, PATH alone will not fix it.

Keep the image's supported entrypoint/s6 startup flow. Railway Start Command overrides can replace Docker ENTRYPOINT, so simply pasting an absolute hermes command into that field may create a different startup problem. Do not run a setup/reset command, reinstall into /opt/data, or start a second gateway against it.

Can you share only the image tag/digest, Start Command and whether a custom PATH is set? No secrets or backup files are needed.

Sources:

https://github.com/NousResearch/hermes-agent/issues/18673

https://github.com/NousResearch/hermes-agent/blob/main/Dockerfile

https://hermes-agent.nousresearch.com/docs/user-guide/docker

https://docs.railway.com/cli/volume

https://docs.railway.com/deployments/start-command

AI-assisted, checked against the linked sources. I have not accessed or tested your deployed container.


sundayabasi684-byte
FREE

4 hours ago

The s6-applyuidgid failure indicates the supervisor cannot locate the hermes binary on the execution path when dropping privileges. This is typically triggered by a custom Start Command override that bypasses the image's built-in entrypoint dispatch script, or a mismatched PATH in the container environment.

To recover service access safely without risking your persistent volume at /opt/data:

Check your service settings and clear any custom Start Command override so the container uses the image's default entrypoint-dispatch script.

Ensure no custom PATH override is stripping /opt/hermes/bin or /opt/hermes/.venv/bin from the execution environment.

Trigger a clean redeploy of the exact image tag/digest to let s6 initialize the process cleanly against your existing mounted volume.

AI-assisted; checked against Railway documentation and settings.


syndexco
FREE

an hour ago

AI-assisted, source-checked. One backup-first caution: current upstream Hermes startup is not read-only. Its stage2 hook changes ownership/permissions in the state directory and syncs skills before normal startup. Clearing Start Command and redeploying should not be treated as risk-free before your requested backup is verified. This is a preservation warning, not a confirmed diagnosis of your image.

Source: https://github.com/NousResearch/hermes-agent/blob/72df5aa60e46890d2c11438c60f2d1a212660585/docker/stage2-hook.sh


Welcome!

Sign in to your Railway account to join the conversation.

Loading...