a month ago
Hi Railway team,
My workspace has been restricted with the reason: "Network abuse from service builds". All my deployments are down as a result, including production.
I am not aware of anything on my side that would qualify as network abuse, and I would like to understand what was detected so I can fix it. The account tied to this thread is the affected one. Could you please tell me which service and which build(s) triggered the detection, the approximate time window, and what kind of traffic was flagged?
I am happy to remove, reconfigure or migrate whatever caused it. If you would rather not discuss the specifics in a public thread, just tell me where to send them.
Thanks,
Adrien
9 Replies
a month ago
We're not able to share the specifics of why a workspace restriction was applied. There is no action on your side that will lift this restriction, and we're unable to provide access to export data from a restricted workspace.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
We're not able to share the specifics of why a workspace restriction was applied. There is no action on your side that will lift this restriction, and we're unable to provide access to export data from a restricted workspace.
a month ago
What can I do ?
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We understand this is a difficult situation. Unfortunately there is no path available to restore access to the workspace or its data, and we're not able to share further details about the restriction.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
We understand this is a difficult situation. Unfortunately there is no path available to restore access to the workspace or its data, and we're not able to share further details about the restriction.
a month ago
Ok but what can I do ?
How can we slove that ?
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We're not able to lift this restriction from your side or provide further details about what triggered it, and there is no self-serve path to restore access to the workspace or its data.
Status changed to Awaiting User Response Railway • about 1 month ago
a month ago
Hello,
Our workspace was restricted with the reason "Network abuse from service builds".
We want to fix the actual cause rather than guess, so this message has two parts:
what we found and already fixed on our side, and the data we need from you to
confirm we fixed the right thing.
WHAT WE FOUND — MOST LIKELY CAUSE
Our own security audit found that our application exposed a Socket.IO namespace
(/terminal) with no authentication on the public port. It opened a shell inside
the container. The feature was dormant and none of our clients used it, but a
third party could have used it to originate traffic from our containers. We are
not claiming this is what you detected — but it is the finding that would most
plausibly look like network abuse, and we are reporting it rather than hiding it.
Fixed on 27 Aug: the Socket.IO server is no longer instantiated unless explicitly
enabled, and then requires an admin JWT. /socket.io now returns 404.
WHAT WE FOUND — SECONDARY CONTRIBUTORS
Build volume. On 27 Aug our main service was deployed 29 times in one day. This
was not an attack: an automated coding assistant was pushing commits to main in
a loop, and every push triggered a fresh Docker build. Those builds are heavy —
apt-get for ~25 Chromium system libraries, then npm ci on a dependency tree
that installs to 1.4 GB. Our build logs show [5/7] RUN npm ci executing rather
than being served from cache, so each of the 29 builds appears to have
re-downloaded everything. Plausibly tens of GB of build traffic in 24 hours.
A dead service. A second service (evolution-api) pointed at an upstream Docker
image that had been deleted. It failed 5 deployments that day, 4 within the same
second — a retry burst against an image that no longer exists.
WHAT WE HAVE ALREADY DONE
-
Removed the unauthenticated public shell (27 Aug). Socket.IO is off by default
and admin-JWT gated when enabled; /socket.io returns 404.
-
Blocked our agents' outbound network tools for unauthenticated visitors.
-
Capped the deployments our application can trigger through your API: minimum
90 seconds apart, 12 per day per client project. Previously unbounded.
-
Removed a double trigger — we called serviceInstanceDeployV2 after a
variableUpsert that already caused a redeploy.
-
Stopped the push loop that caused 29 builds in a day. Commits are now batched.
-
Reduced build weight: the Dockerfile is now a two-stage build that prunes dev
dependencies after compiling. Measured image-to-image, 4.81 GB -> 3.69 GB. We
used
npm prune --omit=devrather than a secondnpm ci --omit=devspecifically so the build does not download anything twice.
-
Deleted the dead
evolution-apiservice so it stops retrying.
WHAT WE NEED FROM YOU
We have two very different plausible causes — an exposed shell originating egress
from a running container, versus a spike of ordinary build traffic. They point to
opposite fixes, and we cannot tell them apart without your telemetry. Could you
share:
-
the source IP, container ID or build ID the flagged traffic came from;
-
the service and project involved;
-
the destinations observed — IPs or domains, and ports;
-
the exact time window, start and end, in UTC;
-
the volume measured — rate, connection count, bytes transferred;
-
whether this was traffic during a BUILD or from a RUNNING service. The stated
reason says "service builds", but we would like certainty on this point.
Our application does make legitimate outbound requests to third-party APIs
(Google, LinkedIn, email providers) on behalf of its users, so if the flag is
about application egress we need to know in order to adjust it properly rather
than guess.
ONE CLARIFICATION ON WORKSPACE IDS
The UI shows the restriction on workspace 4cfd84ca-cf01-41a7-946b-f216905e9fb3.
However, the deployments actually being blocked are in project crm
(7fda3be6-be8e-419b-ae44-e3aabe6bd217), which belongs to workspace
782d3a97-ca10-4e67-b075-082b0c3cf110 ("checkeasy's Projects"). Please check
whether both workspaces are restricted — we may be looking at the wrong one.
IMPACT
A production application serving users runs in this workspace and can no longer
be deployed or restarted. We would appreciate a review whenever you can, and we
are happy to provide any further detail or logs you need.
Thanks,
Adrien
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We appreciate the thorough audit and the steps you have taken, but we are not able to share the telemetry, service identifiers, or traffic details you requested, and the restriction is not something that can be lifted based on changes made on your side.
Status changed to Awaiting User Response Railway • about 1 month ago
Railway
We appreciate the thorough audit and the steps you have taken, but we are not able to share the telemetry, service identifiers, or traffic details you requested, and the restriction is not something that can be lifted based on changes made on your side.
a month ago
Please you need to help me
Status changed to Awaiting Railway Response Railway • about 1 month ago
a month ago
We genuinely wish we could offer a path forward here, but there is nothing we or you can do to change the status of this workspace, and we are not in a position to share more about why it was applied.
Status changed to Awaiting User Response Railway • about 1 month ago
Status changed to Solved checkeasy • about 1 month ago