More than 10 concurrent Dagster jobs failing
sereenaa
PROOP

4 months ago

hi I am hosting dagster on the pro Railway plan and running into issues when having more than 10 concurrent runs. the variable I am setting is DAGSTERMAXCONCURRENT_RUNS=15, setting it to 10 works fine.

this is the error i am getting:

pyo3_runtime.PanicException: could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

i was wondering if this is a Railway limitation or a setting i need to configure to increase the number of threads

the memory and CPU looks fine on the metrics tab, i am not exceeding the plan limits

Project ID: 5ee74426-d012-4973-be29-4829cc6c9954

<@71946189913726976> FYI

0 Replies

brody
EMPLOYEE

4 months ago

This is likely the PID limit, unfortunately, we can only increase that on the Enterprise plan


sereenaa
PROOP

4 months ago

gotcha. am i able to request an upgrade via this channel?


brody
EMPLOYEE

4 months ago

Enterprise starts at $2k /m and if that's something you are interested in, I can get you the link to book a call with sales.


Loading...