a year ago
I am trying to deploy https://docs.vespa.ai/ on railway as part of a bigger support bot https://docs.onyx.app/introduction
I use image vespaengine/vespa:8.277.17 with moutned volume at /opt/vespa/var with limits of 50 gb for volume and 32gb/vcpu.
Now, I'm quite sure that because vespa itself is essentially a cluster, it runs into a problem with constraints:
1739796351.347337 e5113ac9d486 5 configserver vespa-start-configserver warning Wanted 409600 as limit for max user processes but cannot exceed current hard limit: 99999
runserver(configserver) running with pid: 21
1739796351.441423 e5113ac9d486 22 - vespa-start-services info Too low vm.max_map_count [65530] - trying to increase it to 262144
1739796351.441523 e5113ac9d486 22 - vespa-start-services warning Could not increase vm.max_map_count - current value [65530] too low, should be at least 262144I thought switching to pro plan would help, but nah, same thing as hobby. Can't seem to change those in Dockerfile as well.
Would really appreciate any help/workaround for this. The rest of the stack is deployed and only Vespa seems to keep having trouble with railway :/
PrpjectID: 627cc3d0-e52b-4d9b-a144-4bb452f86cd5
4 Replies
later down the line i get error messages like these:
[2025-02-17 12:59:18.533] ERROR container Container.com.yahoo.jdisc.core.StandaloneMain JDisc exiting: Throwable caught: \nexception=\njava.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached\n\tat java.base/java.lang.Thread.start0(Native Method)\n\tat java.base/java.lang.Thread.start(Thread.java:809)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1353)\n\tat com.yahoo.container.jdisc.component.Deconstructor.deconstruct(Deconstructor.java:67)\n\tat com.yahoo.container.di.Container.deconstructComponentsAndBundles(Container.java:189)\n\tat com.yahoo.container.di.Container.deconstructFailedGraph(Container.java:185)\n\tat com.yahoo.container.di.Container.waitForNextGraphGeneration(Container.java:92)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.waitForNextGraphGeneration(HandlersConfigurerDi.java:155)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.(HandlersConfigurerDi.java:78)\n\tat com.yahoo.container.core.config.HandlersConfigurerDi.(HandlersConfigurerDi.java:64)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.createConfigurer(ConfiguredApplication.java:425)\n\tat com.yahoo.container.jdisc.ConfiguredApplication.start(ConfiguredApplication.java:174)\n\tat com.yahoo.jdisc.core.ApplicationLoader.start(ApplicationLoader.java:157)\n\tat com.yahoo.jdisc.core.StandaloneMain.run(StandaloneMain.java:41)\n\tat com.yahoo.jdisc.core.StandaloneMain.main(StandaloneMain.java:33)\n[JVMs involved] Deploying Vespa on Railway, struggling with some limits even on a pro plan.
a year ago
the 99999 ulimit is not configurable, and if you need more than 32gb of memory you would need to be on enterprise
a year ago
!s
Status changed to Solved brody • about 1 year ago