restore accidentally hard-deleted production Omar project
effacermonexistence
HOBBYOP

a month ago

Urgent recovery request.

Project: omar-benchmark-replay

Project ID: 9b9c438a-fb27-4665-989c-ed4ef895956d

Workspace ID: c599acaa-1c56-46ed-8a0b-26c58a857957

Environment: production (24c5e3ba-41c9-4bf7-8b6c-a75b4177d2c6)

Railway emailed “Scheduled Project Deletion” at 2026-08-21 08:10:33 UTC and explicitly promised a 48-hour grace period. However, the authenticated includeDeleted query now shows deletedAt=2026-08-21T08:25:27.816Z—only about 15 minutes later. The project page is 404 and projectScheduleDeleteCancel returns “Project not found” (traceId 2351271485176028462). This production deletion was accidental and must be reversed.

Please urgently restore the full project from Railway’s soft-delete/backend recovery, including both services and configuration:

  • app service 4f5440b7-b06e-43e4-a92b-f43cb31e1c4f
  • worker service fbe50cd2-bfa6-4cf3-ae48-d7afbcbbb8a8
  • volume c585c422-b11c-4968-b8d7-cef19cddb354
  • volume instance 94123815-c621-4d42-b211-1e5ff7ff1226 mounted at /data
  • custom domains, GitHub source bindings, and deployment settings

The volume had about 512 MB of durable production data. Please do not create an empty replacement volume; restore/preserve the original volume contents and attachment. A separate “Scheduled Volume Deletion” email arrived at 08:16:20 UTC.

Canonical source is effacermonexistence/omar-benchmark-replay-live-source. The authenticated workspace owner is available here. This is time-critical because omaragi.com is currently down.

Solved

4 Replies

effacermonexistence
HOBBYOP

a month ago

Additional recovery evidence: the delete audit attributes Project.deleted and Volume.deleted to the owner account via Railway CLI 5.41.2, so the dashboard 48-hour self-service path is no longer available. Please escalate immediately to the storage team and preserve/check the original volume's underlying storage before garbage collection. Also preserve the Hobby retained deployment images/records for the last successful ec43bb9295c4620eb8d6c818ab9f614bca352ad2 deployments: app deployment 2c37caec-93ac-4010-be6e-cf4b790ceee6 and worker deployment 24fb2326-3294-4658-b352-1086df9c5992. Please do not create an empty replacement volume; we need the original volume contents reattached if recoverable.


Railway
BOT

a month ago

The project was deleted today at 08:25 UTC via railway delete (CLI). The 48-hour restore window applies only to deletions made from the dashboard, not from the CLI, which deletes immediately. That is why the project is no longer restorable despite the short time since deletion. We are not able to recover deleted projects, services, volumes, or their data once the deletion has taken an immediate path, and this does not change based on how the request is framed.


Status changed to Awaiting User Response Railway • about 2 months ago


Railway

The project was deleted today at 08:25 UTC via `railway delete` (CLI). The 48-hour restore window applies only to deletions made from the dashboard, not from the CLI, which deletes immediately. That is why the project is no longer restorable despite the short time since deletion. We are not able to recover deleted projects, services, volumes, or their data once the deletion has taken an immediate path, and this does not change based on how the request is framed.

effacermonexistence
HOBBYOP

a month ago

I still need human Railway staff help. The automated answer does not address the time-critical storage-team check documented in prior Railway incidents: whether the original us-west2 underlying volume object and retained deployment records still exist before garbage collection. Please escalate this thread to a human/storage engineer now. Do not create an empty replacement. If full project restore is impossible, please preserve and reattach the original volume contents to a replacement project, and preserve the last successful app/worker deployment snapshots, settings, and variables while Hobby's removed-image retention window is still open.


Status changed to Awaiting Railway Response Railway • about 2 months ago


dizzydes90
EMPLOYEE

a month ago

The deletion record for this project shows the scheduled deletion was created from your account and then explicitly completed immediately from your account about fifteen minutes later, which is what ended the 48 hour window early. The volume and the services had also been deleted individually from your account before that. We do not restore projects, services, volumes, or their data after a user-initiated deletion. The platform-level tooling we maintain internally is reserved for catastrophic platform-side recovery, not for reversing user actions. We will not be reattaching previous volume contents to a replacement project or preserving prior deployment images, settings, or variables.


Status changed to Awaiting User Response dizzydes90 • about 2 months ago


Status changed to Solved dizzydes90 • about 2 months ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...