13 days ago
Hello Railway Support team
My name is Marco from GrantFox. We are an OSS platform that connects GitHub projects and organizations with contributors and developers. We currently have around 3,000 users and growing. Our application relies heavily on GitHub integrations (including webhooks) and we need consistent low response times from the backend.
We have mapped our technical debt and implemented several code and query optimizations. In general, response times are good and other metrics (error rate, CPU, memory) look healthy. However, yesterday and early today we detected significant spikes in the P99.
Attached is the Response Time graph from our monitoring dashboard that clearly shows the issue:
Most of the time, p50, p90, p95, and p99 remain low and stable.
There are two noticeable spikes in P99:
- One around midday yesterday (reaching ~13 seconds).
- A much larger spike early this morning (~6:00–7:00 AM), where P99 exceeded 25 seconds and P95 also rose significantly.
The median (p50) stays flat and low throughout the entire period.
No correlated spikes appear in CPU, memory, or error rate.
What we have already ruled out:
- Issues in the application code (we have already optimized and mapped technical debt).
- Elevated request error rate.
- CPU or memory limitations.
- Serverless / sleep mode (we have it disabled because we need the service always on with predictable low latency).
Our current suspicion: Since application code and resource metrics look fine, we believe these tail-latency spikes may be related to Railway infrastructure.
Request: Could you please help us with the following?
1- Review our project/service for any legacy or outdated configurations that could be causing intermittent high P99 latency.
2- Check if there are any known issues or behaviors in Railway’s infrastructure that could explain these occasional P99 spikes (networking, load balancer, nodes, etc.).
3- Provide specific configuration recommendations for services that require consistent low latency + always-on (connection pooling, deployment settings, region, replicas, etc.).
4- Share any technical insights from your side on what might be causing this pattern (stable p50 + occasional high P99).
We are happy to provide:
- Deployment IDs from the time of the spikes
- Additional logs
- Temporary project access if needed
- More screenshots or monitoring data
Thank you very much in advance! We really value Railway and want to keep the platform as smooth as possible for our users.
Best regards,
Marco
2 Replies
13 days ago
This thread has been opened as a public bounty so the community can help solve it. The thread and any further activity are now visible to everyone.
Status changed to Open Railway • 13 days ago
13 days ago
Hi, Marco!
Do you log request/response with timecodes? I suppose it can help to analyze it as long as problem can be on different levels, for example: db connection pool, github loading or any heavy process or your side
Can you check logs of every step or provide it if possible (or add is there is no logs on every step) to detect what exactly slows down the system
13 days ago
Hi aquamarissa!
Yeah sure, we have a good logging system, health system and many stuff and when this happen I review the DB metrics, logs and so on but all of them were looking good so I think it's a railway issue