6 months ago
Hey railway team, so currently we have a Postgres table which processes pending jobs using a Railway cron every 5 minutes.
We are aiming for realtime processing with a limit on spawned containers and bound on runaway cost. Our proposed design is:
->API inserts pending row into Postgres (a job)
-> upon insert, a railway worker service picks up and runs backend operations
->worker processes jobs with app-level concurrency limit
Questions:
1. Is this the recommended Railway pattern for queue workers? Should we consider redis/rabbitMQ?
2. Are worker service replicas fixed unless we explicitly configure scaling? Should we consider serverless?
3. How do we hard-cap max replicas/cost so infinite containers don't spawn when we have ghost/orphaned containers?
2 Replies
Status changed to Open Railway • 7 months ago
6 months ago
Postgres is totally fine for this.
You don't really need the overhead of RabbitMQ or Redis yet unless you're hitting massive scale like thousands of jobs a second.
On Railway, worker services are "always-on" containers, not serverless functions. Replicas won't scale unless you manually enable Autoscaling in the service settings. but keep in mind it’s based on CPU/RAM usage, not your Postgres row count.
Also you can set a hard "Max Replicas" cap in the UI or just set project-wide budget alerts in the billing settings - that’s the safest way to prevent a runaway bill from ghost containers or loops.
Hope that helps :)
a month ago
AI-assisted answer, checked against Railway’s current documentation. A bounded, long-running worker consuming your existing Postgres queue is a reasonable starting point; inserting a row does not itself launch a new Railway container. Redis/RabbitMQ are optional architectural choices, not required simply to process jobs sooner.
For your three questions:
-
Keep a fixed number of worker replicas and an explicit concurrency limit per worker. Claim jobs atomically, and make processing idempotent with bounded retries and a lease/timeout so crashed workers do not strand jobs. PostgreSQL documents SKIP LOCKED as useful for multiple consumers of a queue-like table: https://www.postgresql.org/docs/current/sql-select.html
-
Railway’s documented horizontal scaling is manual replica configuration; vertical autoscaling changes CPU/memory use within limits, not the number of queue consumers. One replica with concurrency C limits that worker to C simultaneous jobs; R replicas allow up to R × C if each enforces that limit. https://docs.railway.com/deployments/scaling
-
An email budget alert does NOT stop workloads. Configure the workspace Compute Usage Hard limit if you accept ALL workloads being taken offline at the threshold. Separately set per-replica CPU/memory limits under Settings → Deploy → Replica Limits; those limit resources, not replica count. https://docs.railway.com/pricing/cost-control
For a continuously polling Postgres worker, I would leave Serverless off: ongoing DB traffic can prevent sleeping, and a queued row is not itself an incoming request to wake a sleeping worker. https://docs.railway.com/deployments/serverless
Avoid application code that creates services or scales replicas for each job. Process jobs inside the bounded worker pool instead. These are design recommendations; I have not changed or tested your production service.