ThreadPoolBuildError
sereenaa
PROOP

10 months ago

{ kind: IOError(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }) }

I have a dagster instance deployed on Railway and using multithreading in Python. I also have dbt assets materializing and I am running out of resources, leading to the deployed instance crashing. I'm wondering if this is due to a resource limit on our Pro Railway plan? I've currently got max 8 workers for the multithreading job and 10 concurrent asset materializations allowed.

24 Replies

sereenaa
PROOP

10 months ago

5ee74426-d012-4973-be29-4829cc6c9954


brody
EMPLOYEE

10 months ago

can you try lowering the workers and jobs?


antonyip
PRO

10 months ago

We fixed this by restarting the container every 4 hours. Our software works by starting threads and processes in the container and we were hitting NPROC_LIMIT on the container. Raising the limit was not possible.

RLIMIT_NPROC=(soft:99999 hard:99999)

We suspect that this was unlimited before and now set to 100k


antonyip
PRO

10 months ago

Currently issue is this:

1357868831262904300


brody
EMPLOYEE

10 months ago

Where are you trying to migrate it to?


antonyip
PRO

10 months ago

from useast to useast metal


brody
EMPLOYEE

10 months ago

May I ask why you are trying to revert the Postgres version?


antonyip
PRO

10 months ago

the thing was on :latest


antonyip
PRO

10 months ago

i suspected the latest was 17


antonyip
PRO

10 months ago

so wanted to go back to the correct version


antonyip
PRO

10 months ago

think its saved now


brody
EMPLOYEE

10 months ago

latest is 16 for our images


antonyip
PRO

10 months ago

well, its working now, was in abit of panic when the db was down


antonyip
PRO

10 months ago

now i can migrate it again?


brody
EMPLOYEE

10 months ago

its possible the target region does not have the available space for that migration


antonyip
PRO

10 months ago

hm fair enough


antonyip
PRO

10 months ago

anyway to confirm this? I should just wait then?


brody
EMPLOYEE

10 months ago

Yes you would want to wait until Metal Volumes are GA for Pro users, currently it is only GA for Hobby users


brody
EMPLOYEE

10 months ago

hello, it is possible we just fixed this, could you try again?


antonyip
PRO

10 months ago

Yup sure


antonyip
PRO

10 months ago

nope, still same issue


brody
EMPLOYEE

10 months ago

thank you for trying


sereenaa
PROOP

10 months ago

hello there, we are continuing to get this error

alembic.util.exc.CommandError: Version table 'alembic_version' has more than one head present; please use get_current_heads()


brody
EMPLOYEE

10 months ago

I'm really sorry but I'm not too sure how much we can help you with that as it's unlikely to be related to the Railway platform


Loading...