More than 10 concurrent Dagster jobs failing
sereenaa
PROOP

9 months ago

hi I am hosting dagster on the pro Railway plan and running into issues when having more than 10 concurrent runs. the variable I am setting is DAGSTER_MAX_CONCURRENT_RUNS=15, setting it to 10 works fine.

this is the error i am getting:

pyo3_runtime.PanicException: could not spawn threads: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

i was wondering if this is a Railway limitation or a setting i need to configure to increase the number of threads

the memory and CPU looks fine on the metrics tab, i am not exceeding the plan limits

Project ID: 5ee74426-d012-4973-be29-4829cc6c9954

@Antonidas FYI

image.png

Attachments

4 Replies

9 months ago

This is likely the PID limit, unfortunately, we can only increase that on the Enterprise plan


sereenaa
PROOP

9 months ago

gotcha. am i able to request an upgrade via this channel?


9 months ago

Enterprise starts at $5k /m and if that's something you are interested in, I can get you the link to book a call with sales.


louisdeconinck
PRO

16 days ago

@sez Would you mind sharing your Dagster template on how you got it running on Railway?


Welcome!

Sign in to your Railway account to join the conversation.

Loading...