2 months ago
Hi,
We're moving from Opensearch to Grafana. I tried using the new version of Locomotive (Brody's one) to push the logs from Railway to Loki but I get the following error :
all services must exist within the environment set by the LOCOMOTIVE_ENVIRONMENT_ID variable
The problem is that my service IS in the environment (I used ${{RAILWAY_ENVIRONMENT_ID}} and ${{"My Service Name".RAILWAY_SERVICE_ID}})
I also double checked my API key and tested it against the GraphQL API but everything is alright. The same API key is still working using the old Locomotive version (FerretCode's one) on another enrivonment. I also tried with another, newly created, API key but it still doesn't work. My API key is an account-scoped one.
Thanks for any help :)
Pinned Solution
14 Replies
2 months ago
This thread has been opened as a bounty so the community can help solve it.
Status changed to Open Railway • 2 months ago
2 months ago
Here is my full ENV :
LOCOMOTIVE_ENABLE_DEPLOY_LOGS="true"
LOCOMOTIVE_ENABLE_HTTP_LOGS="false"
LOCOMOTIVE_ENVIRONMENT_ID="${{RAILWAY_ENVIRONMENT_ID}}"
LOCOMOTIVE_RAILWAY_API_KEY="<REDACTED>"
LOCOMOTIVE_REPORT_STATUS_EVERY="1m"
LOCOMOTIVE_SERVICE_IDS="${{\"My Service Name\".RAILWAY_SERVICE_ID}}"
LOCOMOTIVE_WEBHOOK_MODE="loki"
LOCOMOTIVE_WEBHOOK_URL="https://REDACTED/loki/api/v1/push"
2 months ago
Remove the double quotes, so the variable can correctly resolve to the underlying UUID.
Correct Syntax:
LOCOMOTIVE_SERVICE_IDS=${{ "My Service Name".RAILWAY_SERVICE_ID }}
Redeploy after change
2 months ago
The backslashes are juste added by Railway ENV Raw Editor but they are not the problem here. Here is the result of the locomotive tests against the API
Attachments
2 months ago
you need to strip all wrapping double quotes from the variable definitions in the Railway configuration so that the raw UUIDs are passed cleanly to the GraphQL API. Also check if any of your services are stopped?
2 months ago
There is no double quote at all in the created variable as I said these one are juste added by Railway Raw Editor 😄
Attachments
2 months ago
Here is a recap from the GraphiQL playground :
Request :
query($projectId: String!, $envId: String!, $serviceId: String!) {
project(id: $projectId) {
environments {
edges {
node {
name id
}
}
}
}
environment(id: $envId) {
serviceInstances {
edges {
node {
serviceName serviceId
}
}
}
}
service(id: $serviceId) {
name id
}
}Variables :
{
"projectId": "9300db95-9243-4966-a400-9a41e4021d2f",
"envId": "c53c0be7-47c1-465c-a125-8557b315f084",
"serviceId": "a3b96539-5c67-42bb-a4e4-88b23c14c68d"
}
Response :
{
"data": {
"project": {
"environments": {
"edges": [
{
"node": {
"name": "update-2.24.0",
"id": "032b9627-5959-4dbc-85b7-e3fc38093458"
}
},
{
"node": {
"name": "production",
"id": "a272f6fe-f008-4c8b-b3d4-3a526bba7221"
}
},
{
"node": {
"name": "update",
"id": "c53c0be7-47c1-465c-a125-8557b315f084"
}
}
]
}
},
"environment": {
"serviceInstances": {
"edges": [
{
"node": {
"serviceName": "Redis-dR2I",
"serviceId": "a68e472a-8cf2-4ef4-a666-deccfa1a67c3"
}
},
{
"node": {
"serviceName": "Postgres-rfeI",
"serviceId": "a5af2297-2c47-4d6f-9af2-613b82b81083"
}
},
{
"node": {
"serviceName": "Twenty Worker-95GQ",
"serviceId": "37fa4e94-c576-44b1-8404-8e2885736325"
}
},
{
"node": {
"serviceName": "Twenty Server",
"serviceId": "a3b96539-5c67-42bb-a4e4-88b23c14c68d"
}
},
{
"node": {
"serviceName": "Locomotive-10f7",
"serviceId": "998e8352-8fe1-4fce-b038-5b6a5328b9ca"
}
}
]
}
},
"service": {
"name": "Twenty Server",
"id": "a3b96539-5c67-42bb-a4e4-88b23c14c68d"
}
}
}2 months ago
Thanks for sharing these details.
Your playground response shows that Railway's API returns a completely different ID for "Meety Server" within the context of that specific environment's serviceInstances.
2 months ago
No it doesn't. a09a061b-5cb7-4298-adab-8b623e17ce35 doesn't even appear anywhere.
Anyway I found the problem it was a bit stupid 😅
2 of the services were stopped. Still weird that locomotive can't find stopped services. I think it should just ignore stopped services but not crash. I will open an issue about this on Locomotive github.
2 months ago
Locomotive's startup validation should ideally handle or ignore stopped/inactive services gracefully rather than crashing when a configured service is not currently running.
2 months ago
Issues are disabled on Locomotive repo I hope Brody will see this post since I can't manage to contact him on Discord either :/
2 months ago
- I will make it so that a service's state does not matter for the existence check.
- I will enable issues on the repo. I didn't even know they were off to begin with.
Status changed to Awaiting User Response Railway • 2 months ago
brody
1. I will make it so that a service's state does not matter for the existence check. 2. I will enable issues on the repo. I didn't even know they were off to begin with.
2 months ago
Status changed to Open brody • 2 months ago
Status changed to Awaiting User Response Railway • 2 months ago
Status changed to Solved brody • 2 months ago