20 days ago
Hi, my project is currently impacted by the recent service disruption.
I'm running Directus with a Postgres database in EU West.
Current situation:
- Postgres shows as "Completed" but intermittently not serving connections
- Directus crashes repeatedly with KnexTimeoutError (timeout acquiring connection)
- I adjusted DB pool variables to reduce connections
- After redeploy, the deployment is now stuck in "Queued" for over 10 minutes
Because of this, Directus is not responding at all ("Application failed to respond").
Can you confirm:
- If EU West is still experiencing degraded deployment or database connectivity?
- If deployments are currently stuck in queue due to capacity?
- If my Postgres volume (postgres-volume) is safe and unaffected?
This is a production project and I currently cannot access my CMS.
Thanks.
Pinned Solution
18 days ago
That's sad to hear. Atleast we're figured something.
With no STORAGE_* variables configured, Directus falls back to its default local filesystem storage (./uploads). The database metadata survived in Postgres, so /files/<id> can still return 200, but the old binary file that /assets/<id> needs is no longer present on disk.
The new upload returning 200 is the key confirmation: current permissions and asset serving are working. The old file returning 403 is most likely Directus failing to serve an object that only exists as metadata now.
The DB did its job. Wishing you the best with restoring.
14 Replies
20 days ago
Your project was directly impacted by the service disruption that affected EU West and has since been resolved. Your Postgres service is currently crash-looping (unable to start its main process), and your Directus deployment is stuck in an initializing state, both consistent with lingering effects of the outage. Your Postgres volume is being mounted successfully, so your data appears safe. We recommend redeploying your Postgres service now that the incident is resolved, and once Postgres is healthy, redeploy Directus as well. If either service remains stuck after a fresh redeploy, please let us know.
Status changed to Awaiting User Response Railway • 20 days ago
19 days ago
Following up here since my new thread was closed as duplicate.
After the outage recovery, my Directus instance is partially working:
- /items endpoints work publicly
- but ALL /assets/:id return 403
Error:
"You don't have permission to access collection "directus_files" or it does not exist."
This affects all images, PDFs, and file access across the site.
Important:
- This issue started AFTER the Railway outage and recovery
- I did not change anything in infrastructure
- I only attempted a backup via API during recovery
Even after reviewing Public Policy permissions, assets remain blocked.
Could the outage or recovery process have affected Directus system collections or public access state?
This is currently breaking the entire frontend.
Would appreciate urgent guidance.
Status changed to Awaiting Railway Response Railway • 19 days ago
Status changed to Open Railway • 19 days ago
19 days ago
This is currently breaking the entire frontend (all images and PDFs are inaccessible publicly).
Would really appreciate prioritizing this, as content endpoints work but the site is visually unusable.
Railway
Your project was directly impacted by the [service disruption](https://status.railway.com/incident/I23M92U0) that affected EU West and has since been resolved. Your Postgres service is currently crash-looping (unable to start its main process), and your Directus deployment is stuck in an initializing state, both consistent with lingering effects of the outage. Your Postgres volume is being mounted successfully, so your data appears safe. We recommend redeploying your Postgres service now that the incident is resolved, and once Postgres is healthy, redeploy Directus as well. If either service remains stuck after a fresh redeploy, please let us know.
19 days ago
Following up here with additional diagnostics:
- directus_files Read is set to "All Access"
- No filters are applied
- Field permissions are fully enabled
- /items endpoints work correctly in incognito
- ONLY /assets endpoints return 403
Error:
"You don't have permission to access collection 'directus_files' or it does not exist."
This suggests the issue is not basic permission configuration.
Given this started immediately after the outage and recovery, could this indicate an inconsistent Directus state affecting system collections like directus_files or public access resolution?
Would a full redeploy of Directus (after confirming Postgres stability) be the recommended next step?
This is currently breaking a live production site, so I would appreciate guidance on the safest fix.
androikzone
Following up here with additional diagnostics: - directus_files Read is set to "All Access" - No filters are applied - Field permissions are fully enabled - /items endpoints work correctly in incognito - ONLY /assets endpoints return 403 Error: "You don't have permission to access collection 'directus_files' or it does not exist." This suggests the issue is not basic permission configuration. Given this started immediately after the outage and recovery, could this indicate an inconsistent Directus state affecting system collections like directus_files or public access resolution? Would a full redeploy of Directus (after confirming Postgres stability) be the recommended next step? This is currently breaking a live production site, so I would appreciate guidance on the safest fix.
19 days ago
Update after testing:
I performed a clean redeploy of the Directus service while Postgres was online and stable.
Directus deployed successfully, but /assets/:id still returns:
{"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]}
So the issue persists after redeploy.
Content endpoints still work publicly; only assets remain blocked.
androikzone
Update after testing: I performed a clean redeploy of the Directus service while Postgres was online and stable. Directus deployed successfully, but /assets/:id still returns: {"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]} So the issue persists after redeploy. Content endpoints still work publicly; only assets remain blocked.
19 days ago
Additional confirmation:
- directus_files collection exists and is readable in admin
- issue only affects public /assets endpoint
- reproducible consistently in incognito
Happy to provide logs if needed.
18 days ago
I would like to help (if that's will be helpful).
A few checks that usually narrow this down quickly:
- Test the same file through the files API:
GET /files/<file-id>
GET /items/directus_files/<file-id>Do both return 403 publicly too, or this is still only to all /assets/?
If you use a permission filter on directus_files, temporarily test with a single known file and no filter. Then add the folder/type filter back after confirming assets work.
Check Directus logs for the exact 403. If the error mentions storage/S3/local filesystem, then it may be storage access. Any additional logs here could be helpful.
18 days ago
Try to test these without auth, since the issue is public access.
The Public policy permission row for System Collections -> directus_files, especially:
read permission level ,
fields ,
any filter
whether it is attached to the Public role/policy
ve-jo
Try to test these without auth, since the issue is public access. The Public policy permission row for System Collections -> directus_files, especially: read permission level , fields , any filter whether it is attached to the Public role/policy
18 days ago
Thanks for jumping in @ve-jo I tested without auth in incognito with the same file ID:
/assets/f300c194-ae9e-4dbe-9094-257a87d1baf2
→ 403 FORBIDDEN
{"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]}
/files/f300c194-ae9e-4dbe-9094-257a87d1baf2
→ 200 OK, returns file metadata:
storage: "local"
filename_disk: "f300c194-ae9e-4dbe-9094-257a87d1baf2.pdf"
filename_download: "Ramírez March_Dialogue I_español.pdf"
type: "application/pdf"
filesize: "331892"
/items/directus_files/f300c194-ae9e-4dbe-9094-257a87d1baf2
→ 403 FORBIDDEN
{"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]}
Public policy:
- directus_files Read is enabled
- no filter is applied
- fields are enabled
- I also created a clean new test public policy with directus_files Read enabled, no filters, and it still returns 403
Logs:
- Calling /assets only shows a clean GET /assets/... 403
- No visible filesystem/S3/local storage error appears in Railway logs
Storage:
- There is no S3/external storage configured
- /files metadata shows storage: "local"
- I do not see a mounted volume for Directus uploads in Railway
So current state seems to be:
- file metadata exists and is publicly readable via /files/:id
- /assets/:id is blocked
- /items/directus_files/:id is blocked
- storage appears to be local container storage
androikzone
Thanks for jumping in @ve-jo I tested without auth in incognito with the same file ID: /assets/f300c194-ae9e-4dbe-9094-257a87d1baf2 → 403 FORBIDDEN {"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]} /files/f300c194-ae9e-4dbe-9094-257a87d1baf2 → 200 OK, returns file metadata: storage: "local" filename_disk: "f300c194-ae9e-4dbe-9094-257a87d1baf2.pdf" filename_download: "Ramírez March_Dialogue I_español.pdf" type: "application/pdf" filesize: "331892" /items/directus_files/f300c194-ae9e-4dbe-9094-257a87d1baf2 → 403 FORBIDDEN {"errors":[{"message":"You don't have permission to access this.","extensions":{"code":"FORBIDDEN"}}]} Public policy: - directus_files Read is enabled - no filter is applied - fields are enabled - I also created a clean new test public policy with directus_files Read enabled, no filters, and it still returns 403 Logs: - Calling /assets only shows a clean GET /assets/... 403 - No visible filesystem/S3/local storage error appears in Railway logs Storage: - There is no S3/external storage configured - /files metadata shows storage: "local" - I do not see a mounted volume for Directus uploads in Railway So current state seems to be: - file metadata exists and is publicly readable via /files/:id - /assets/:id is blocked - /items/directus_files/:id is blocked - storage appears to be local container storage
18 days ago
The fact that /files/<id> returns 200 means Directus can read the file metadata from Postgres. But /assets/<id> has to fetch the actual binary from the configured storage adapter.
I would test this before changing more permissions:
- Upload a brand new test PDF now.
- Open its
/assets/<new-file-id>URL in incognito. - If the new file works but old files return 403, then permissions are probably fine and the old local upload files were lost from storage.
- If the new file also returns 403, then it is still an active Directus config/permission issue.
Also, /items/directus_files/<id> returning 403 may be important. directus_files is a system collection and the normal file metadata API is /files/<id>, which is already returning 200 for you.
18 days ago
Could you please also look into you're variables?
STORAGE_LOCATIONS
STORAGE_LOCAL_DRIVER
STORAGE_LOCAL_ROOT
STORAGE_LOCAL_BUCKET
STORAGE_LOCAL_REGION
STORAGE_LOCAL_ENDPOINT
STORAGE_LOCAL_FORCE_PATH_STYLEIf STORAGE_LOCAL_DRIVER is missing or equals local, then Directus is using filesystem storage and old files may indeed have been stored on ephemeral container disk.
If STORAGE_LOCAL_DRIVER=s3, then the next check is whether the Railway Bucket service is still connected and whether the object for this file exists in the bucket:
filename_disk: f300c194-ae9e-4dbe-9094-257a87d1baf2.pdf
storage location: localve-jo
Could you please also look into you're variables? ``` STORAGE_LOCATIONS STORAGE_LOCAL_DRIVER STORAGE_LOCAL_ROOT STORAGE_LOCAL_BUCKET STORAGE_LOCAL_REGION STORAGE_LOCAL_ENDPOINT STORAGE_LOCAL_FORCE_PATH_STYLE ``` If STORAGE_LOCAL_DRIVER is missing or equals local, then Directus is using filesystem storage and old files may indeed have been stored on ephemeral container disk. If STORAGE_LOCAL_DRIVER=s3, then the next check is whether the Railway Bucket service is still connected and whether the object for this file exists in the bucket: ``` filename_disk: f300c194-ae9e-4dbe-9094-257a87d1baf2.pdf storage location: local ```
18 days ago
Thanks, I checked the variables and there are no STORAGE_* variables configured.
I also uploaded a brand new test file now:
/assets/af9ca12d-7bea-4992-b4a0-b591ae9188d2
→ 200 OK, loads correctly
Old file:
/assets/f300c194-ae9e-4dbe-9094-257a87d1baf2
→ 403 FORBIDDEN
So it looks like the current Directus config/permissions are working for newly uploaded files, but the old local files were likely lost from ephemeral container storage after the outage/redeploy;(
I’ll move forward with restoring/re-uploading the assets and configuring persistent/external storage.
androikzone
Thanks, I checked the variables and there are no STORAGE_* variables configured. I also uploaded a brand new test file now: /assets/af9ca12d-7bea-4992-b4a0-b591ae9188d2 → 200 OK, loads correctly Old file: /assets/f300c194-ae9e-4dbe-9094-257a87d1baf2 → 403 FORBIDDEN So it looks like the current Directus config/permissions are working for newly uploaded files, but the old local files were likely lost from ephemeral container storage after the outage/redeploy;( I’ll move forward with restoring/re-uploading the assets and configuring persistent/external storage.
18 days ago
That's sad to hear. Atleast we're figured something.
With no STORAGE_* variables configured, Directus falls back to its default local filesystem storage (./uploads). The database metadata survived in Postgres, so /files/<id> can still return 200, but the old binary file that /assets/<id> needs is no longer present on disk.
The new upload returning 200 is the key confirmation: current permissions and asset serving are working. The old file returning 403 is most likely Directus failing to serve an object that only exists as metadata now.
The DB did its job. Wishing you the best with restoring.
ve-jo
That's sad to hear. Atleast we're figured something. With no `STORAGE_*` variables configured, Directus falls back to its default local filesystem storage (`./uploads`). The database metadata survived in Postgres, so `/files/<id>` can still return 200, but the old binary file that `/assets/<id>` needs is no longer present on disk. The new upload returning 200 is the key confirmation: current permissions and asset serving are working. The old file returning 403 is most likely Directus failing to serve an object that only exists as metadata now. The DB did its job. Wishing you the best with restoring.
18 days ago
Thanks @ve-jo , really appreciate you walking through this with me.
That makes total sense now. The new upload working vs old ones returning 403 made it clear the issue is the missing underlying files, not permissions or config.
So yes, looks like local container storage got wiped after the outage/redeploy, while Postgres kept the metadata.
Thanks again for helping narrow it down 🙏 I’ll move forward with restoring the assets and setting up proper persistent storage so this doesn’t happen again.