Solid Queue Service not processing bin/jobs
michaelwehrley
HOBBYOP

2 months ago

I have a Rails 8 app with a primary db, and I set up a solid queue db. My jobs get queued in my solid queue db, but they don't get processed. I then followed the setup guide here https://docs.railway.com/guides/rails#option-1-separate-database-recommended-for-high-volume . However, my empty service (i.e., Worker Service) can't seem to connect to my solid queue postgres db (the db that holds the jobs to being processed) when it gets to the command bin/jobs . In the setup docs, it sounds like a REDIS_URL is needed. So I did something similar and set up my DATABASE_URL to match the url of my solid queue db. I know from printing out logs that this DATABASE_URL is writing correctly: postgresql://postgres:foo@postgres-bar.railway.internal:5432/railway
My error is "connection to server on socket "/run/postgresql/.s.PGSQL.5432" failed: No such file or directory". Any thoughts, b/c I can't seem to get these jobs working wink emoji . Thanks!

Solved$10 Bounty

6 Replies

ilyassbreth
FREE

2 months ago

the socket error means your worker service can't find the database url. rails defaults to looking for a local postgres socket when DATABASE_URL isn't set

in your worker service on railway:

  1. make sure you have this env var set:

DATABASE_URL=${{Postgres.DATABASE_URL}}

(where "Postgres" is whatever you named your solid queue database)

  1. if both services are in the same railway project, use the private url instead:

DATABASE_URL=${{Postgres.DATABASE_PRIVATE_URL}}
  1. important: after setting the env var, do a FULL REDEPLOY (not just restart). railway sometimes caches env vars and needs a fresh deploy to pick them up.

check your database.yml has this for production:

yaml

production:
  <<: *default
  url: <%= ENV["DATABASE_URL"] %>

this is from railway's official rails guide: https://docs.railway.com/guides/rails

the socket error happens because without DATABASE_URL, the pg gem looks for /run/postgresql/.s.PGSQL.5432 which doesn't exist in railway containers. setting DATABASE_URL fixes it

i hope this help you slightly_smiling_face emoji


michaelwehrley
HOBBYOP

2 months ago

This is why I am so confused b/c my worker service does have DATABASE_URL set and it is pointing to ${{MY_QUEUE.DATABASE_URL}} . I don't think this matters, but for my project DATABASE_URL are private and DATABASE_PUBLIC_URL are the public urls. I also tried the public URL, with no success. As for my main rails app, database.yml is correct and is pointing to my postgres db url. It's almost as if my queue database isn't considered a database, but it does have tables and rows of data within it. Also, I have redeployed multiple times. thinking emoji


ilyassbreth
FREE

2 months ago

okay this is helpful info. need to dig deeper to find the exact issue;check these things in your worker service:

  1. what is RAILS_ENV set to? it needs to be production

  2. in the railway logs for your worker service, can you see the actual resolved DATABASE_URL value? (railway should show resolved variables in deploy logs)

  3. what does your config/database.yml production section look like? especially if there's a queue: section

  4. is MY_QUEUE definitely the correct name of your postgres service in railway? try clicking on your queue postgres service and check what it's actually called

based on the railway docs, when using separate database (option 1), the worker service needs:

DATABASE_URL pointing to the queue database (you have this)

database.yml should have the primary db connection BUT since this is a worker-only service running bin/jobs, it might only need the queue connection

in your worker service , try to temporarily add these variables to help debug:

RAILS_ENV=production
DATABASE_URL=${{MY_QUEUE.DATABASE_PRIVATE_URL}}

(use PRIVATE_URL since it's in the same project)

then check the deploy logs , does it show the full postgres connection string or is it still trying the socket?

the socket error usually means the DATABASE_URL env var literally isn't there at runtime, even if you see it set in railway dashboard

hope this help you slightly_smiling_face emoji


michaelwehrley
HOBBYOP

2 months ago

Yep, RAILS_ENV is production; If I putsDATABASE_URL in bin/jobs it points to my MY_QUEUE.DATABASE_URL. My production app has config/database.yml as

production:
  primary: &primary_production
    <<: *default
    url: <%= ENV["DATABASE_URL"] %>
  cache:
    <<: *primary_production
    url: <%= ENV["DATABASE_CACHE_URL"] %>
    migrations_paths: db/cache_migrate
  queue:
    <<: *primary_production
    url: <%= ENV["DATABASE_QUEUE_URL"] %>
    migrations_paths: db/queue_migrate
  cable:
    <<: *primary_production
    url: <%= ENV["DATABASE_CABLE_URL"] %>
    migrations_paths: db/cable_migrate

Where the DATABASE_QUEUE_URL also points to the MY_QUEUE.DATABASE_URL . Attached is my setup diagram.

Attachments


ilyassbreth
FREE

2 months ago

looking at your setup i think the problem is your worker service has DATABASE_URL pointing to the queue db, but your database.yml is looking for DATABASE_QUEUE_URL for the queue connection!

when bin/jobs runs, solid queue needs to connect to the database specified in your config/solid_queue.yml. if that config says database: queue, then rails looks for the queue: section in database.yml, which needs DATABASE_QUEUE_URL.

the fix is that in your worker service, you need:

DATABASE_URL=${{Queue.DATABASE_URL}}
DATABASE_QUEUE_URL=${{Queue.DATABASE_URL}}

both pointing to the same queue database. the worker service doesn't need access to your primary postgres db, cache, or cable dbs , only the queue db

why this is becayse even though the worker only uses the queue db, rails still tries to establish the primary connection first (using DATABASE_URL), then it needs DATABASE_QUEUE_URL for the actual queue connection that solid queue uses

alternatively, you could simplify your worker's database.yml to only have a primary connection (no multi-db), but setting both env vars is easier and doesn't require changing code.

try adding DATABASE_QUEUE_URL=${{Queue.DATABASE_URL}} to your worker service and redeploy

hope this help uslightly_smiling_face emoji


michaelwehrley
HOBBYOP

2 months ago

Okay, thank you for the help! I definitely did not understand that when bin/jobs runs, it would reference the key queue defined in my config/database.yml file and try to access that ENV, which in this case was DATABASE_QUEUE_URL. Therefore, in the end, I don't think I even need to define DATABASE_URL, just DATABASE_QUEUE_URL and I pointed it to ${{Queue.DATABASE_URL}}.


Status changed to Open brody about 2 months ago


Status changed to Solved brody about 2 months ago


Loading...