2 months ago
I'm experiencing a recurring issue where my Supabase service role key stops working intermittently, causing my FastAPI backend to fail
authentication with the error: "User not allowed".
Project Details:
- Project name: honest-endurance
- Service: Python/FastAPI backend
- Backend URL: https://lead-tracking-production-c64b.up.railway.app
Symptoms:
1. Health endpoint returns: {"status":"unhealthy","supabase_db":"ok","supabase_auth":"error: User not allowed"}
2. The same SUPABASE_SERVICE_ROLE_KEY works when tested locally
3. Deleting and re-pasting the environment variable temporarily fixes the issue
4. The problem recurs after some time (hours to days)
What I've Tried:
- Deleting and re-adding the SUPABASE_SERVICE_ROLE_KEY variable (temporary fix)
- Adding NO_CACHE=1 environment variable
- Adding .railway-clear-cache file to the repo
- Creating an entirely new Railway project (this project is the new one - previous project motivated-creativity had the same issue)
Environment Variable Details:
- The key is a JWT string (~200 characters)
- I'm careful to paste without trailing spaces or newlines
- The key has not been rotated on Supabase's side
Questions:
1. Is there any caching of environment variables that could cause this?
2. Could there be character encoding issues when storing/retrieving long JWT strings?
3. Is there a way to see the raw stored value of an environment variable for debugging?
4. Are there any known issues with environment variable persistence?
Expected Behavior:
Environment variables should remain stable and unchanged between deployments.
Thank you for your help.
3 Replies
2 months ago
Sorry, the project name is actually: blissful-courtesy
2 months ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open Railway • about 2 months ago
2 months ago
The fact that you mentioned "The problem recurs after some time (hours to days)", suggests that the issue is likely not related to Railway's environment variables or caching. It's most likley a FastAPI/Python client lifecycle issue, meaning you have something wrong in your code that's causing this issue.
If you're initializing the supabase client globally (at the top level of your file), it could possibly cause such scenario. So I would recommend you check if that's the case. If it is the case, move the supabase client initialization inside the request lifecycle.
darseen
The fact that you mentioned "The problem recurs after some time (hours to days)", suggests that the issue is likely not related to Railway's environment variables or caching. It's most likley a FastAPI/Python client lifecycle issue, meaning you have something wrong in your code that's causing this issue.If you're initializing the supabase client globally (at the top level of your file), it could possibly cause such scenario. So I would recommend you check if that's the case. If it is the case, move the supabase client initialization inside the request lifecycle.
2 months ago
Thank you, darseen. I wanted to update you. I've been bashing my head against the wall trying to fix this with LLM help. Here is the summary of what we did to temporarily fix it. Let's see if it holds.
Summary of what I did:
The Problem:
Pipeline's health check was returning "unhealthy" with supabase_auth: error: User not allowed after users logged in. The service_role key
was correct, but Auth Admin API calls kept failing with 403.
Root Cause:
supabase-py stores session state in the client object. When sign_in_with_password() runs, it saves the user's JWT in the client. Subsequent
requests send that JWT instead of the service_role key, causing "User not allowed" errors on admin operations.
Failed Attempt:
Tried separate cached clients with ClientOptions(auto_refresh_token=False, persist_session=False) - sessions still leaked.
Working Fix:
Changed database.py to create fresh clients for every auth operation:
- get_supabase_admin() → Fresh client for auth.admin.* calls
- get_supabase_auth() → Fresh client for sign_in_with_password()
- get_supabase_db() → Cached client for table queries only (never auth)