2 months ago
Hi Railway support,
Following the recent Railway outage affecting our infrastructure, our Directus instance is no longer able to access or write files to Railway Object Storage / S3.
Current stack affected:
Directus
Railway Object Storage (S3)
Redis
PostgreSQL
Observed behaviour:
Existing uploaded images no longer display
Even Directus admin/logo assets are broken
Existing objects are still physically present in the bucket
Re-uploading or regenerating assets fails
Directus cannot save new files
Errors received:
Accessing existing assets:
{
"errors": [
{
"message": "You don't have permission to access this.",
"extensions": {
"code": "FORBIDDEN"
}
}
]
}
Uploading/regenerating files:
{
"errors": [
{
"message": "Service "files" is unavailable. Couldn't save file d5ee5574-3127-4e4b-81de-6a27dca56ef7.jpg.",
"extensions": {
"service": "files",
"reason": "Couldn't save file d5ee5574-3127-4e4b-81de-6a27dca56ef7.jpg",
"code": "SERVICE_UNAVAILABLE"
}
}
]
}
Troubleshooting already attempted:
Re-entered all S3/Object Storage environment variables
Redeployed Directus
Restarted services
Verified bucket objects still exist
Attempted asset regeneration
Attempted re-uploading assets
Issue persists and appears related to Directus being unable to communicate properly with Railway Object Storage after the outage.
Could you please investigate whether:
Object Storage endpoints changed
Internal networking/private DNS changed
Credentials or bucket access were impacted
There is an ongoing issue affecting S3 compatibility or object access
Thank you.
Project ID: 7908863c-e65c-4fed-a1f3-2a9b92110b43
2 Replies
2 months ago
From the symptoms, this looks less like missing bucket data and more like Directus losing effective access to Railway Object Storage after the outage.
A few things stand out:
- Existing objects are still present in the bucket
- Both old and new assets fail
- Directus admin/logo assets are also broken
- Uploads now return
SERVICE_UNAVAILABLE - Existing asset requests return
FORBIDDEN
That combination usually points to one of these:
- S3 credentials/token mismatch after the outage
- Internal endpoint/private networking changes
- Bucket policy/access regression
- S3 endpoint resolution/DNS issue between services
- Directus storage adapter failing authentication silently
A few things worth checking additionally:
STORAGE_LOCATIONS=s3
STORAGE_S3_DRIVER=s3
STORAGE_S3_KEY=...
STORAGE_S3_SECRET=...
STORAGE_S3_BUCKET=...
STORAGE_S3_REGION=auto
STORAGE_S3_ENDPOINT=...
STORAGE_S3_FORCE_PATH_STYLE=trueAlso verify the endpoint still resolves correctly from inside the Directus container:
nslookup <your-s3-endpoint>
curl <your-s3-endpoint>Another important thing:
after the outage, some users experienced stale internal networking/private DNS resolution until services were fully recreated rather than only redeployed.
Potential workaround that may help:
- Create a fresh Object Storage instance
- Reattach credentials
- Recreate Directus service variables from scratch
- Fully recreate the Directus deployment instead of simple redeploy
Since the bucket contents still exist, this strongly suggests the issue is access/authentication/connectivity related rather than data loss.
Given multiple services were affected after the outage, Railway support should probably verify:
- S3-compatible endpoint integrity
- Internal DNS resolution
- Bucket IAM/access propagation
- Object Storage auth/token consistency
- Cross-service networking between Directus and Object Storage
Project ID:
7908863c-e65c-4fed-a1f3-2a9b92110b43
a month ago
Solved with Railway support but thanks for your suggestion.
Status changed to Solved brody • about 1 month ago