a month ago
Hey, I'm currently battling elevated DPM in Grafana, and pinpointed the reason to the fact that we have a few services deployed in multiple regions, each exporting metrics every 60s with the same OTEL_RESOURCE_ATTRIBUTES. We use standard OTEL libraries that are configured entirely through environment variables across 6 different services.
The challenge now is that we need to ensure metrics from multi-instance services are exported with unique instance IDs. So I went looking for that and found RAILWAYREPLICAID and RAILWAYREGION. I thought I could just set OTELSERVICE_ATTRIBUTES to e.g.
service.name=filecoin-app,service.namespace=app,deployment.environment=production,service.instance.id=${{RAILWAY_REPLICA_ID}},cloud.region=${{RAILWAY_REGION}}but unfortunately those substitutions are empty. I assume that those two env vars are only available at run / build-time and not available for substitution within other env vars, and this is probably expected behavior?
But then, the question remains - how can I solve this problem? š¤
8 Replies
a month ago
That is expected behavior, those variables are dynamically injected at runtime.
any chance someone had this problem before and you're maybe aware of some workaround? not really sure what the best approach isā¦ with our node services i could relatively easily mess with the OTEL sdk and configure these values at runtime, but with Deno OTEL as far as I'm aware, the only way to set these attributes is through the environmentā¦ šµāš«
a month ago
Then set them through the environment at runtime?
ended up simply setting the start command to append the runtime vars:
sh -c 'OTEL_RESOURCE_ATTRIBUTES="${OTEL_RESOURCE_ATTRIBUTES},service.instance.id=${RAILWAY_REPLICA_ID},cloud.region=${RAILWAY_REPLICA_REGION}" node --require ./build/telemetry.cjs build'works perfectly š thanks @Brody, no idea why i didn't think of that š
a month ago
No problem!
Status changed to Solved brody ā¢ about 1 month ago