I want to deploy Airflow components. Right now, I am using Docker Compose with the Celery Executor, so the worker runs 24x7. In Kubernetes (Minikube), I tested the Kubernetes Executor, which spins up a pod for a task and terminates it once the job finishes.

Since I am currently building a startup, I do not want to maintain a full Kubernetes cluster and run the complete setup all the time. Is it possible to dynamically scale up a service based on schedules or workload from another application running in the same project? Alternatively, is there a way to temporarily spin up larger workers only during heavy job processing?

My workload is event driven, where data processing starts only when new data arrives at the source.