GPT OSS is a popular open-source reasoning model, as are Deepseek, Qwen, and Llama. The distilled versions make it feasible to self-host these models on Railway.

Railway is an optimal place to deploy high-scale models and access them via API, but needs a template to make it easier to get started.

This bounty will be paid out when:

• A high-quality GPT OSS model template is presented

• All potential template feedback has been incorporated

• All requirements are met and tested

• Follows all template best practices where applicable

Template requirements:

• vLLM inference server with specific model

• Request batching / caching through Redis

• Multiple model sizes are supported

• API layer - FastAPI with OpenAI compatible endpoints

• Volume-backed storage for all databases using correct mount paths

• Service dependencies should be correctly configured using proper startup order and health checks

• Environment variables correctly configured for Railway domains using private networking where applicable

A few resources to get you started:

• Template Best Practices

• Learn more about Railway Templates

• Railway Documentation

• GPT OSS information

• vLLM documentation