a year ago
Hi there,
I am trying to run some CPU Based inference, but most of the frameworks interrogate the host to discover and optimized for hardware. I am guessing that you docker run, doesn't have the appropriate seccomp?
Has anyone been able to get any CPU based inference to work and can you share some details?
7 Replies
a year ago
Hey there! We've found the following might help you get unblocked faster:
If you find the answer from one of these, please let us know by solving the thread!
a year ago
LLM hardware acceleration is in beta. It's flagged on only for the following template deploys:
https://x.com/Railway/status/1961084101273759816
Please let us know what your usecase is and we will try and make sure it works!
Status changed to Awaiting User Response Railway • about 1 year ago
a year ago
hi there, we not looking for GPU's support in this regard, which would be great btw, but none of the containers for vllm for example works. They all cough up that it can't detect the platform. Even the vllm gpt-oss template container crokes. I have also tried the intel/vllm container.
Status changed to Awaiting Railway Response Railway • about 1 year ago
a year ago
Hmm. I'm not talking about GPUs here either. I'm talking like AVX processor flag acceleration.
Status changed to Awaiting User Response Railway • about 1 year ago
a year ago
They all cough up that it can't detect the platform
Can you show me this error please?
a year ago
This thread has been marked as public for community involvement, as it does not contain any sensitive or personal information. Any further activity in this thread will be visible to everyone.
Status changed to Open jake • about 1 year ago
a year ago
I've made the thread public so we can escalate it to the template authors. But, we need the error you're talking about here please