Workspace restricted - "Network abuse from service builds" - request for review
checkeasy
PROOP

a month ago

Hi Railway team,

My workspace has been restricted with the reason: "Network abuse from service builds". All my deployments are down as a result, including production.

I am not aware of anything on my side that would qualify as network abuse, and I would like to understand what was detected so I can fix it. The account tied to this thread is the affected one. Could you please tell me which service and which build(s) triggered the detection, the approximate time window, and what kind of traffic was flagged?

I am happy to remove, reconfigure or migrate whatever caused it. If you would rather not discuss the specifics in a public thread, just tell me where to send them.

Thanks,

Adrien

Solved

9 Replies

Railway
BOT

a month ago

We're not able to share the specifics of why a workspace restriction was applied. There is no action on your side that will lift this restriction, and we're unable to provide access to export data from a restricted workspace.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway

We're not able to share the specifics of why a workspace restriction was applied. There is no action on your side that will lift this restriction, and we're unable to provide access to export data from a restricted workspace.

checkeasy
PROOP

a month ago

What can I do ?


Status changed to Awaiting Railway Response Railway • about 1 month ago


Railway
BOT

a month ago

We understand this is a difficult situation. Unfortunately there is no path available to restore access to the workspace or its data, and we're not able to share further details about the restriction.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway

We understand this is a difficult situation. Unfortunately there is no path available to restore access to the workspace or its data, and we're not able to share further details about the restriction.

checkeasy
PROOP

a month ago

Ok but what can I do ?

How can we slove that ?


Status changed to Awaiting Railway Response Railway • about 1 month ago


Railway
BOT

a month ago

We're not able to lift this restriction from your side or provide further details about what triggered it, and there is no self-serve path to restore access to the workspace or its data.


Status changed to Awaiting User Response Railway • about 1 month ago


checkeasy
PROOP

a month ago

Hello,

Our workspace was restricted with the reason "Network abuse from service builds".

We want to fix the actual cause rather than guess, so this message has two parts:

what we found and already fixed on our side, and the data we need from you to

confirm we fixed the right thing.

WHAT WE FOUND — MOST LIKELY CAUSE

Our own security audit found that our application exposed a Socket.IO namespace

(/terminal) with no authentication on the public port. It opened a shell inside

the container. The feature was dormant and none of our clients used it, but a

third party could have used it to originate traffic from our containers. We are

not claiming this is what you detected — but it is the finding that would most

plausibly look like network abuse, and we are reporting it rather than hiding it.

Fixed on 27 Aug: the Socket.IO server is no longer instantiated unless explicitly

enabled, and then requires an admin JWT. /socket.io now returns 404.

WHAT WE FOUND — SECONDARY CONTRIBUTORS

Build volume. On 27 Aug our main service was deployed 29 times in one day. This

was not an attack: an automated coding assistant was pushing commits to main in

a loop, and every push triggered a fresh Docker build. Those builds are heavy —

apt-get for ~25 Chromium system libraries, then npm ci on a dependency tree

that installs to 1.4 GB. Our build logs show [5/7] RUN npm ci executing rather

than being served from cache, so each of the 29 builds appears to have

re-downloaded everything. Plausibly tens of GB of build traffic in 24 hours.

A dead service. A second service (evolution-api) pointed at an upstream Docker

image that had been deleted. It failed 5 deployments that day, 4 within the same

second — a retry burst against an image that no longer exists.

WHAT WE HAVE ALREADY DONE

  1. Removed the unauthenticated public shell (27 Aug). Socket.IO is off by default

    and admin-JWT gated when enabled; /socket.io returns 404.

  2. Blocked our agents' outbound network tools for unauthenticated visitors.

  3. Capped the deployments our application can trigger through your API: minimum

    90 seconds apart, 12 per day per client project. Previously unbounded.

  4. Removed a double trigger — we called serviceInstanceDeployV2 after a

    variableUpsert that already caused a redeploy.

  5. Stopped the push loop that caused 29 builds in a day. Commits are now batched.

  6. Reduced build weight: the Dockerfile is now a two-stage build that prunes dev

    dependencies after compiling. Measured image-to-image, 4.81 GB -> 3.69 GB. We

    used npm prune --omit=dev rather than a second npm ci --omit=dev

    specifically so the build does not download anything twice.

  7. Deleted the dead evolution-api service so it stops retrying.

WHAT WE NEED FROM YOU

We have two very different plausible causes — an exposed shell originating egress

from a running container, versus a spike of ordinary build traffic. They point to

opposite fixes, and we cannot tell them apart without your telemetry. Could you

share:

  • the source IP, container ID or build ID the flagged traffic came from;

  • the service and project involved;

  • the destinations observed — IPs or domains, and ports;

  • the exact time window, start and end, in UTC;

  • the volume measured — rate, connection count, bytes transferred;

  • whether this was traffic during a BUILD or from a RUNNING service. The stated

    reason says "service builds", but we would like certainty on this point.

Our application does make legitimate outbound requests to third-party APIs

(Google, LinkedIn, email providers) on behalf of its users, so if the flag is

about application egress we need to know in order to adjust it properly rather

than guess.

ONE CLARIFICATION ON WORKSPACE IDS

The UI shows the restriction on workspace 4cfd84ca-cf01-41a7-946b-f216905e9fb3.

However, the deployments actually being blocked are in project crm

(7fda3be6-be8e-419b-ae44-e3aabe6bd217), which belongs to workspace

782d3a97-ca10-4e67-b075-082b0c3cf110 ("checkeasy's Projects"). Please check

whether both workspaces are restricted — we may be looking at the wrong one.

IMPACT

A production application serving users runs in this workspace and can no longer

be deployed or restarted. We would appreciate a review whenever you can, and we

are happy to provide any further detail or logs you need.

Thanks,

Adrien


Status changed to Awaiting Railway Response Railway • about 1 month ago


Railway
BOT

a month ago

We appreciate the thorough audit and the steps you have taken, but we are not able to share the telemetry, service identifiers, or traffic details you requested, and the restriction is not something that can be lifted based on changes made on your side.


Status changed to Awaiting User Response Railway • about 1 month ago


Railway

We appreciate the thorough audit and the steps you have taken, but we are not able to share the telemetry, service identifiers, or traffic details you requested, and the restriction is not something that can be lifted based on changes made on your side.

checkeasy
PROOP

a month ago

Please you need to help me


Status changed to Awaiting Railway Response Railway • about 1 month ago


Railway
BOT

a month ago

We genuinely wish we could offer a path forward here, but there is nothing we or you can do to change the status of this workspace, and we are not in a position to share more about why it was applied.


Status changed to Awaiting User Response Railway • about 1 month ago


Status changed to Solved checkeasy • about 1 month ago


Welcome!

Sign in to your Railway account to join the conversation.

Loading...