[JVMs involved] Deploying Vespa on Railway, struggling with some limits even on a pro plan.
alexbalandi
PROOP

a year ago

I am trying to deploy https://docs.vespa.ai/ on railway as part of a bigger support bot https://docs.onyx.app/introduction

I use image vespaengine/vespa:8.277.17 with moutned volume at /opt/vespa/var with limits of 50 gb for volume and 32gb/vcpu.

Now, I'm quite sure that because vespa itself is essentially a cluster, it runs into a problem with constraints:

1739796351.347337    e5113ac9d486    5    configserver    vespa-start-configserver    warning    Wanted 409600 as limit for max user processes but cannot exceed current hard limit: 99999

runserver(configserver) running with pid: 21

1739796351.441423    e5113ac9d486    22    -    vespa-start-services    info    Too low vm.max_map_count [65530] - trying to increase it to 262144

1739796351.441523    e5113ac9d486    22    -    vespa-start-services    warning    Could not increase vm.max_map_count - current value [65530] too low, should be at least 262144

I thought switching to pro plan would help, but nah, same thing as hobby. Can't seem to change those in Dockerfile as well.

Would really appreciate any help/workaround for this. The rest of the stack is deployed and only Vespa seems to keep having trouble with railway :/

PrpjectID: 627cc3d0-e52b-4d9b-a144-4bb452f86cd5

Solved

4 Replies

alexbalandi
PROOP

a year ago

later down the line i get error messages like these:

[2025-02-17 12:59:18.533] ERROR   container        Container.com.yahoo.jdisc.core.StandaloneMain    JDisc exiting: Throwable caught: \nexception=\njava.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached\n\tat java.base/java.lang.Thread.start0(Native Method)\n\tat java.base/java.lang.Thread.start(Thread.java:809)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353)\n\tat com.yahoo.container.jdisc.component.Deconstructor.deconstruct(Deconstructor.java:67)\n\tat com.yahoo.container.di.Container.deconstructComponentsAndBundles(Container.java:189)\n\tat com.yahoo.container.di.Container.deconstructFailedGraph(Container.java:185)\n\tat com.yahoo.container.di.Container.waitForNextGraphGeneration(Container.java:92)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.waitForNextGraphGeneration(HandlersConfigurerDi.java:155)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.(HandlersConfigurerDi.java:78)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.(HandlersConfigurerDi.java:64)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.createConfigurer(ConfiguredApplication.java:425)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:174)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:157)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:33)\n

alexbalandi
PROOP

a year ago

[JVMs involved] Deploying Vespa on Railway, struggling with some limits even on a pro plan.


a year ago

the 99999 ulimit is not configurable, and if you need more than 32gb of memory you would need to be on enterprise


a year ago

!s


Status changed to Solved brody about 1 year ago


Loading...