How to do secure bucket uploads with high concurrency
enesakyuz
PROOP

an hour ago

In my app, users upload a bunch of files and we have a bunch of users in the other hosted system. I need to find a way to do a malicious upload scanner in the process of uploading. I kind of did a clamAV setup but the concurrency of it is not enough.

What is the Railway native way of handling secure uploads to bucket?

$20 Bounty

4 Replies

Railway
BOT

an hour ago

This thread has been opened as a bounty so the community can help solve it.

Status changed to Open Railway about 1 hour ago


The "native" way would be to use the S3 SDK to create a presigned URL to upload files directly into the bucket. But for that, the scanning would need to be on the client side. Otherwise, you can also retrieve the file in your backend after it's uploaded and scan it from there.


0x5b62656e5d

The "native" way would be to use the S3 SDK to create a presigned URL to upload files directly into the bucket. But for that, the scanning would need to be on the client side. Otherwise, you can also retrieve the file in your backend after it's uploaded and scan it from there.

enesakyuz
PROOP

an hour ago

Did the latter since I do not want to trust the client for this. The current situation is that we use presigned URLs with multipart uploads to the bucket, then a worker picks up and scans from the quarantine bucket and if it is healthy, copies to the main bucket. But this system is not super scalable, max I could do was 16 processes * replicas but Railway cannot have replicas with volumes attached so I get forced to deploy a bunch of clamavs. Basically the issue is the limitation on concurrency. Is there something like AWS GuardDuty type of thing we can use?


nadielo
FREE

an hour ago

I wouldn’t put ClamAV behind a volume-backed service for this. That’s what’s killing your ability to scale horizontally.

Your current quarantine → scan → promote flow is actually the right pattern. I’d keep the files in Railway Buckets and make the scanners completely stateless:

"Client -> presigned upload -> quarantine bucket -> queue -> stateless scanner replicas -> clean bucket"

Railway Buckets are S3-compatible and designed for direct presigned uploads, so you don’t need a volume for the scan workers at all.

Run ClamAV inside each scanner replica and use only ephemeral disk for the temporary file being scanned. Once scanning is finished, delete the temp file. Since there’s no Railway Volume attached, you can scale the scanner service horizontally instead of deploying separate ClamAV services manually. Railway only blocks replicas when a Volume is attached.

I’d also put jobs into a queue and let every scanner claim work independently. Then you can have:

"scanner x 10 replicas"

"scanner concurrency 8-16 each"

without any shared filesystem.

I don’t see a Railway-native equivalent of AWS GuardDuty Malware Protection that automatically scans bucket objects for you. Railway’s own upload guidance says to treat uploads as untrusted and scan/validate them after upload, so the scanner is still something you run yourself or get from an external security provider.

One important detail: Railway Buckets are currently accessed over the public network, even from Railway services, so downloading quarantine objects into your scanner service can count as service egress even though bucket egress itself is free.

So the main fix here is: remove the volume from ClamAV, make the scanners stateless, queue the scan jobs, and horizontally scale that service. That gets rid of the replica limitation while keeping the client completely untrusted.


nadielo

I wouldn’t put ClamAV behind a volume-backed service for this. That’s what’s killing your ability to scale horizontally. Your current quarantine → scan → promote flow is actually the right pattern. I’d keep the files in Railway Buckets and make the scanners completely stateless: "Client -> presigned upload -> quarantine bucket -> queue -> stateless scanner replicas -> clean bucket" Railway Buckets are S3-compatible and designed for direct presigned uploads, so you don’t need a volume for the scan workers at all. Run ClamAV inside each scanner replica and use only ephemeral disk for the temporary file being scanned. Once scanning is finished, delete the temp file. Since there’s no Railway Volume attached, you can scale the scanner service horizontally instead of deploying separate ClamAV services manually. Railway only blocks replicas when a Volume is attached. I’d also put jobs into a queue and let every scanner claim work independently. Then you can have: "scanner x 10 replicas" "scanner concurrency 8-16 each" without any shared filesystem. I don’t see a Railway-native equivalent of AWS GuardDuty Malware Protection that automatically scans bucket objects for you. Railway’s own upload guidance says to treat uploads as untrusted and scan/validate them after upload, so the scanner is still something you run yourself or get from an external security provider. One important detail: Railway Buckets are currently accessed over the public network, even from Railway services, so downloading quarantine objects into your scanner service can count as service egress even though bucket egress itself is free. So the main fix here is: remove the volume from ClamAV, make the scanners stateless, queue the scan jobs, and horizontally scale that service. That gets rid of the replica limitation while keeping the client completely untrusted.

enesakyuz
PROOP

an hour ago

Thank you for the detailed explanation, will try this out!


Welcome!

Sign in to your Railway account to join the conversation.

Loading...