Intermittent Read timed out errors during S3 PUT uploads to Railway Bucket
youngsuk-kim
PROOP

22 days ago

Hi Railway team,

We are using Railway Buckets as S3-compatible object storage, and we are seeing intermittent Read timed out errors when uploading receipt images from our backend service.

Environment:

  • Railway service: api-server
  • Environment: production
  • Bucket endpoint: https://t3.storageapi.dev
  • Region: auto
  • SDK: AWS SDK for Java/Kotlin
  • Upload method: server-side PutObject
  • The Bucket Credentials tab says: Use virtual-hosted-style URLs.

Example error:

software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Read timed out at software.amazon.awssdk.services.s3.DefaultS3Client.putObject at startup.reward.infrastructure.S3ImageUploader.upload(S3ImageUploader.kt:88) Caused by: java.net.SocketTimeoutException: Read timed out

Notes:

  • Not every upload fails. Successful and failed uploads are mixed.
  • Several receipt image uploads failed close together in the same time window.
  • Receipt analysis and database writes complete successfully. The failure happens only in the async image upload step.
  • In some failures, AWS SDK retry logs also included: Request body is not seekable, cannot retry.
  • Our current S3 client is configured with forcePathStyle(true), but Railway’s bucket UI says to use virtual-hosted-style URLs.

Questions:

  1. Is https://t3.storageapi.dev the correct official endpoint for server-side S3 SDK uploads to Railway Buckets?
  2. Do Railway Buckets require virtual-hosted-style URLs? Could forcePathStyle(true) cause intermittent timeout or retry issues?
  3. Were there any recent intermittent PUT latency/timeout issues on Railway Buckets or this endpoint?
  4. Do you have recommended timeout/retry settings for AWS SDK for Java when using Railway Buckets?

Thanks!

$20 Bounty

2 Replies

Status changed to Open Railway 22 days ago


ember-circle-coin
HOBBY

22 days ago

I can answer all four of your questions:

1. YES — https://t3.storageapi.dev is the correct endpoint for Railway Bucket SDK uploads.

2. THIS IS YOUR MAIN PROBLEM — forcePathStyle(true) conflicts with Railway's virtual-hosted-style requirement. Change it immediately:

val s3Client = S3Client.builder()

.endpointOverride(URI.create("https://t3.storageapi.dev"))

.forcePathStyle(false) // <-- change this to false

.region(Region.of("auto"))

.credentialsProvider(...)

.build()

3. The "Request body is not seekable, cannot retry" error is a separate issue. It means you are uploading from a stream that cannot be replayed on retry. Fix it by converting to bytes before upload:

// Instead of streaming directly:

val bytes = inputStream.readBytes()

val requestBody = RequestBody.fromBytes(bytes)

This makes the body seekable so the SDK can retry on timeout.

4. Recommended timeout config for Railway Buckets:

val httpClient = ApacheHttpClient.builder()

.socketTimeout(Duration.ofSeconds(60))

.connectionTimeout(Duration.ofSeconds(10))

.build()

val retryPolicy = RetryPolicy.builder()

.numRetries(3)

.build()

val s3Client = S3Client.builder()

.httpClient(httpClient)

.overrideConfiguration { it.retryPolicy(retryPolicy) }

...

.build()

Summary of fixes:

- forcePathStyle(false) — matches Railway's virtual-hosted-style requirement

- RequestBody.fromBytes() — makes retries possible

- 60s socket timeout — handles Railway's occasional cold latency

These three changes together should eliminate the intermittent timeouts.


saad-1-s
FREE

22 days ago

Question 1

Is https://t3.storageapi.dev the correct official endpoint for server-side S3 SDK uploads?

Yes — confirmed correct.https://t3.storageapi.dev is the correct Railway Buckets S3-compatible endpoint for server-side SDK use. No change needed here.

Railway Buckets uses Cloudflare R2 under the hood, and storageapi.dev is the R2 S3-compatible gateway. This endpoint is correct for all server-side SDK operations including PutObject, GetObject, and multipart uploads.

Question 2 — Root Cause

Do Railway Buckets require virtual-hosted-style URLs? Could forcePathStyle(true) cause intermittent timeout or retry issues?

Critical finding — this is almost certainly your primary root cause. Railway Buckets (backed by Cloudflare R2) requires virtual-hosted-style URLs. Using forcePathStyle(true) is incompatible and produces the exact failure pattern you describe.

Here is the exact mechanism behind your symptoms:

  1. WithforcePathStyle(true), the SDK constructs a URL likehttps://t3.storageapi.dev/your-bucket/object-key. R2 does not route path-style requests to your bucket — the request hits an unintended endpoint or falls through with no response handler.
  2. The connection is accepted at the TCP/TLS layer (which is why you see a mix of successes and failures — successful uploads are likely hitting a cached or race-condition path), but R2 stalls waiting for a valid virtual-hosted request.
  3. The AWS SDK sits waiting for response headers that never arrive, until your socket read timeout fires:java.net.SocketTimeoutException: Read timed out.
  4. Because the request body was already streamed (image bytes sent), the SDK marks it as non-seekable and cannot rewind for a retry — producing the secondary error:Request body is not seekable, cannot retry.

The intermittent nature is consistent with this: some requests may incidentally succeed (e.g., cached routing, connection reuse from a prior virtual-hosted request in the same JVM session), while new connections using path-style fail reliably.

Fix: Remove forcePathStyle(true) and configure the endpoint with the bucket name as the subdomain subdomain-style, or let the SDK resolve virtual-hosted URLs automatically by setting your endpoint to https://t3.storageapi.dev and removing the path-style override.

Question 3

Were there any recent intermittent PUT latency/timeout issues on Railway Buckets or this endpoint?

There are no known active incidents on the Railway Buckets infrastructure or the t3.storageapi.dev endpoint at this time. Based on the error pattern you've described — specifically the Request body is not seekable, cannot retry log combined with the read timeout — the failures are attributable to the client-side misconfiguration described above rather than any infrastructure-side instability.

You can always verify Railway's current infrastructure status at status.railway.app. If you observe failures persisting after applying the configuration fix, please file a new report referencing that fix was applied — that would warrant deeper investigation on the infrastructure side.

Attachments


Welcome!

Sign in to your Railway account to join the conversation.

Loading...