12 days ago
I want to deploy Airflow components. Right now, I am using Docker Compose with the Celery Executor, so the worker runs 24x7. In Kubernetes (Minikube), I tested the Kubernetes Executor, which spins up a pod for a task and terminates it once the job finishes.
Since I am currently trying to keep infrastructure costs low, I do not want to maintain a full Kubernetes cluster and run the complete setup all the time. Is it possible to dynamically scale up a service based on schedules or workload from another application running in the same project? Alternatively, is there a way to temporarily spin up larger workers only during heavy job processing?
My workload is event driven, where data processing starts only when new data arrives at the source.
1 Replies
10 days ago
Yes, but I would model this as Railway services plus a queue, not as Airflow KubernetesExecutor.
Railway will not spin up one isolated pod per Airflow task like KubernetesExecutor does. The low-cost pattern is:
- Keep a tiny always-on control plane: Airflow scheduler/webserver, or your event-ingest app if the workload is purely event driven.
- Put Redis/Postgres in the same project and write incoming source events to a durable queue/table.
- Deploy workers as separate Railway services from the same image/repo, with a different start command. Example: one normal Celery worker service, and a second heavy batch-worker service whose command claims work from the queue, processes it, then exits.
- For schedule-based jobs that can exit, use Railway Cron Jobs. The process must finish and exit cleanly; cron is UTC/schedule based, and it is not an event-triggered pod system.
- For event bursts, have the controller call Railway CLI/API to scale the worker service up while queue depth is high, then scale it back down. Example shape: scale
airflow-workerfrom 0/1 replicas to 3 during a burst, then back down when the queue drains. Keep the controller itself running; do not scale the only process that can trigger work to zero. - For rare heavy jobs, use a separate service with higher resources and
restartPolicyType = NEVERorON_FAILURE, then trigger a redeploy/restart when work arrives. Make the worker idempotent because redeploy/restart is coarser than Kubernetes per-task pods.
So the answer is: yes for dynamic worker capacity, but no for true KubernetesExecutor semantics. On Railway, use a durable queue + separate worker services + cron/CLI/API scaling/redeploy hooks. If you need per-task pod isolation, per-task images, or per-task CPU/RAM specs, that is still a Kubernetes feature.
Useful docs:
- Cron jobs: https://docs.railway.com/reference/cron-jobs
- Workers/queues: https://docs.railway.com/guides/cron-workers-queues
- Replica scaling: https://docs.railway.com/cli/scale
- Deployment management: https://docs.railway.com/guides/manage-deployments