for option 1, how not small of a rewrite would it be? you are using a storage engine that ties into postgres and uses local storage (a volume when on railway)
does this storage engine provide an S3 layer? that way you could use cloudflare R2 with a custom domain and then you'd have a no egress bucket and files would be cached

I guess I could do that. Now that I think about it, mysql has a record of the file name, to get started, I could modify the the db records adding prefix or suffixes, and have the records point to other server with cloudflare. This is rather a minimal rewrite. May be not even a rewrite. This would save me a lot of network bandwidth as well I think. Let me think about this a bit but thanks Brody.

I dont know if you remember but you helped me get my media drive up and running for videos and streaming stuff. You even did an example with fastapi

yes I do!

I forget the fine details of it though, so if the storage engine supports S3 + MySQL as a backend, it should be relatively simple, store the files in a cloudflare R2 bucket, and using a custom domain to get the files cached

Yes, that works. I'll check that out. I event have the files on an other server too with a custom domain. So, I could use that server for videos. I just need to integrate the admin panel to save the video to the other server. If integrating admin does not work out, I know admin that I use supports S3 so, I'll what you suggested now. Thanks.
When you initially helped me, my files werenot surviving between deploys since I used the wrong mount point. Now I use /app/media.. Also static files did not have both slashes etc.. You made an example for it. Thanks for that

always happy to help!

Thanks Brody 🙂

no problem! 🙂

let me know what you end up doing!

I'll post it here when I make a move. I am reading the R2 option looks promising. I use SQL Alchemy Admin with FastAPI which has S3 Storage option, so it should be compatible with what you suggested ( Cloudflare R2 )

https://aminalaee.dev/sqladmin/workingwithfiles/

perfect, it does indeed have an S3 storage backend!

Unfortunately, I could not set this up after spending a full day. R2 docs are not that good. I could not find any examples doing this with fastapi_storage. I found one example for regular S3. I will check that out and if that does not work, then I have to find another solution for CDN.

cloudflare R2 is AWS S3 compatible

so yes you'd want to use the S3 backend

I managed to get file info and download a file from the bucket. I was working on it with an example FastAPI project. If I can manage everything, including PresignedUrl etc… , I will post the github example here. Let's see how it's gonna workout.

awesome, I look forward to that!

Here are the examples…

1. This one shows FastAPI, SqlAdmin, CloudFlare R2 integration. boto3 wrapped with another class called S3Storage for easier integration to admin.

https://github.com/yavuzakyazici/s3_example/tree/main

Here is the second example with regular fastapi upload/download file without any admin integration. This one is done with boto3 library..

https://github.com/yavuzakyazici/cfs3fastapi

you work fast!

I'm trying 🙂 Now, I will look into generating PresignedUrl for each video with 20 seconds expiration. This way, I can keep the bucket access private and cache CDN content. When I manage to work it, I'll add that to examples.

So I updated boto3 version with PresignedUrl method.
If you don't want your bucket contents public, you might just want your client app display it or whatever the reason you might have. You can just return a PresignedUrl expires in seconds. Once it is consumed by your client app thru API, it expires 🙂 Nice isn't it ?

https://github.com/yavuzakyazici/cfs3fastapi

awesome work!

looks like this is turning out to be exactly what you want?

Yes it did. Thanks to your suggestion 🙂 I was not expecting it to be this convenient. One thing I could not do was adding CNAME record to my custom domain at A2 hosting. I have to get suport. Adding custom domain to CloudFlare was very very easy though.

yeah A2 likely wouldnt support root level cnames, best to use cloudflare for your nameservers

I'll ask and see what they say 🙂

please note you can not use A or ALIAS types despite what they may say, that will break your site

OK. Good to know thanks. I may not even need to get a custom domain for api. CDN was done and helps caching. Having a custom domain for the api not a must I think

yeah thats up to you, but switching to cloudflare's nameservers is incredibly easy, they even copy over your dns settings for you!

Yes, I noticed it was very easy and It took like a minute to update.

do you have different root domains for the cdn and what you would use for the api?

yes.

they were bot subdomains. CloudFlare worked. But I did it from cloudflare settings. I did not need to touch A2 Panel.

To change the api railway instructs to add a CNAME but A2 does not let me the way it is suggested. So, did not work. But I'll ask support ands see what they say

you tried in the style of api.domain.com? any dns provider should support cnames on a subdomain

api.domain.com? yes exactly..

can you try again and show me what a2 says?

oki.. I'll try and take screenshots as well

It says dns zone is invalid

have you tried just "api" for the name?

I did. It does not let me. If I reverse the values. It accepts it but It adds stuff

like this

have you tried "api" as the name, and then the record "domain.up.railway.app." (with a period at the end)

let me try now

so where do I enter just api

Suggested domain at railway is yynudahm.up.railway.app

please see my message

replace domain with my domain right?

It does not accept it

correct, please show me what you tried?

I tried this

please try what I suggested

when I type api, it completes it to what you saw

may I ask what you removed the image?

Too many images.

I can put it back in

I think discord can handle a few more images lol

here

A2 too complicated, move nameservers to cloudflare, there's no reason to stay with A2 anyway right?

No…

I'll look into it. Let me talk to support first.

may I ask why you are inclined to stay with A2?

I'm not inclined.

I paid the services up front for 2 yrs

ah I see

It was before I had anything here

long before 🙂

gotcha

Let me see what happens. I am happy the way things are if it does not work out. I might move dns to cloudflare if I can

sounds good

It turns out, DNS was hosted at CloudFlare. I managed to change the CNAME there but now I get too many redirects error to reach the subdomain

Thanks. Fixed it right away! 🙂 Works now.

no problem!