Increase API rate limit?
swimerize
PROOP

6 months ago

I am using terraform-community-providers/railway to manage my railway infrastructure. Everything was going smoothly, until I hit the API rate limit near the finish line ("returned error 429 Too Many Requests: error code: 1015"). My infrastructure is not large (1 load balancer, 1 redis cache, 1 webapp, and about 10 cron jobs, a handful of individual environment variable terraform definitions, and a single collection of 80 environment variables).

Something odd is going on with the rate limiting implementation on Railway's side. When I first started setting up the Terraform configuration, I was creating and updating resources without any issues. Without making too many additional changes, I suddenly ran into the rate limit late last night.

I have waited varying lengths of time (multiple hours) and the issue persists. The changes I have made in that time do not explain the sudden exhaustion of API call allowance.

I've resorted to many things: creating a new token, trying to create a new project from scratch (to rule-out any initial sync'ing operations that may be consuming API calls). Whereas before I initially hit the rate limit I was able to do this without problems, this morning I immediately ran into the issue (well past the 1-hour rate limit cool-down period that would be required).

I have tried slow-rolling the creation of the infrastructure by using Terraform to create an empty project, but I am still running into the same threshold restrictions error on this simple operation.

Can someone from Railway engineering validate that the rate limiting is actually working as documented (1000 API requests per hour) - it seems to either be going off of a different timeline or not clearing its state? Is there any way to increase the threshold on a project-by-project basis? 1000 requests per hour seems very low considering how large some of the projects can get.

We have a requirement to manage all of our infrastructure through Terraform, but we are happy to evaluate alternatives to Terraform (IaC) that Railway would support there.

-Kyle

Solved

14 Replies

Railway
BOT

6 months ago

Hello!

We've escalated your issue to our engineering team.

We aim to provide an update within 1 business day.

Please reply to this thread if you have any questions!

Status changed to Awaiting User Response Railway 6 months ago


jake
EMPLOYEE

6 months ago

Hey Kyle,

Difficult to know what rate limit you're running into without error logs. We can configure a higher rate limit on Enterprise

If you could please include the actual HTTP response, we can go from there


swimerize
PROOP

6 months ago

That is going to be difficult considering this is hidden behind the terraform layer. The terraform logs state:
```

2025-06-13T13:16:53.179-0400 [ERROR] provider.terraform-provider-railway_v0.5.1: Response contains error diagnostic: diagnostic_detail="Una
ble to create environment, got error: returned error 429 Too Many Requests: error code: 1015" diagnostic_severity=ERROR diagnostic_summary=
"Client Error" tf_proto_version=6.3 tf_req_id=d1a07ec8-fa10-f0b1-6504-565bd66381ec @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0
/tfprotov6/internal/diag/diagnostics.go:55
tf_resource_type=railway_environment tf_rpc=ApplyResourceChange @module=sdk.proto tf_provider_ad
dr=registry.terraform.io/terraform-community-providers/railway timestamp=2025-06-13T13:16:53.179-0400
```
But this isn't giving you much more information. Capturing the wire traffic is also not going to be of much use due to the HTTPS endpoint. Is there anything on your end you can see that gives any insights as to how many requests are coming from my account to help validate if there is an issue on your end?


Status changed to Awaiting Railway Response Railway 6 months ago


sarahkb125
EMPLOYEE

6 months ago

We only expose the http logs. Could you please adjust to log the actual response and push that to the logs?


Status changed to Awaiting User Response Railway 6 months ago


swimerize
PROOP

6 months ago

OK. I modified and rebuilt the provider to provide the request and response. Let me know if you need anything else:

```
"------- REQUEST -------"

| URL: https://backboard.railway.app/graphql/v2

| Method: POST

| Headers:

| Content-Type: [application/json]

| Body:

| {"query":"\nquery getProject ($id: String!) {\n\tproject(id: $id) {\n\t\t... Project\n\t}\n}\nfragment Project on Project {\n\tid\n\tname\n\tdescription\n\tisPublic\n\tprDeploys\n\tteam {\n\t\tid\n\t}\n\tenvironments {\n\t\tedges {\n\t\t\tnode {\n\t\t\t\tid\n\t\t\t\tname\n\t\t\t\tcreatedAt\n\t\t\t}\n\t\t}\n\t}\n}\n","variables":{"id":"80f92ab8-d76e-46f1-bef8-dc6d00718ff6"},"operationName":"getProject"}

| -----------------------------------

| ------- RESPONSE -------

| Status: 429 Too Many Requests

| Headers:

| Server: [cloudflare]

| Alt-Svc: [h3=":443"; ma=86400]

| Content-Length: [16]

| X-Frame-Options: [SAMEORIGIN]

| Referrer-Policy: [same-origin]

| Cache-Control: [private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0]

| Expires: [Thu, 01 Jan 1970 00:00:01 GMT]

| Vary: [Accept-Encoding]

| Cf-Ray: [94f3eb09ead68f75-BOS]

| Date: [Fri, 13 Jun 2025 19:15:41 GMT]

| Content-Type: [text/plain; charset=UTF-8]

| Connection: [keep-alive]

| Retry-After: [19766]

| Body:

| error code: 1015

| ------------------------------------

```


Status changed to Awaiting Railway Response Railway 6 months ago


swimerize

That is going to be difficult considering this is hidden behind the terraform layer. The terraform logs state:```2025-06-13T13:16:53.179-0400 [ERROR] provider.terraform-provider-railway_v0.5.1: Response contains error diagnostic: diagnostic_detail="Unable to create environment, got error: returned error 429 Too Many Requests: error code: 1015" diagnostic_severity=ERROR diagnostic_summary="Client Error" tf_proto_version=6.3 tf_req_id=d1a07ec8-fa10-f0b1-6504-565bd66381ec @caller=github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov6/internal/diag/diagnostics.go:55 tf_resource_type=railway_environment tf_rpc=ApplyResourceChange @module=sdk.proto tf_provider_addr=registry.terraform.io/terraform-community-providers/railway timestamp=2025-06-13T13:16:53.179-0400```But this isn't giving you much more information. Capturing the wire traffic is also not going to be of much use due to the HTTPS endpoint. Is there anything on your end you can see that gives any insights as to how many requests are coming from my account to help validate if there is an issue on your end?

jake
EMPLOYEE

6 months ago

There's a secondary rate limit at 50 RPS for Pro by default. Likely what you're hitting here

You're likely running into that. Like I said, we can bump it up on an Enterprise plan if that's something you're interested in.


Status changed to Awaiting User Response Railway 6 months ago


swimerize
PROOP

6 months ago

This is a single request for the operation we are attempting to perform. There are no other requests as part of this operation, so something else is going on.

Sadly the Enterprise Plan is not something we can afford at the moment.


Status changed to Awaiting Railway Response Railway 6 months ago


sarahkb125
EMPLOYEE

6 months ago

Hi there - I've pushed this to some of our team to see what we can come up with as you are hitting the rate limit.


Status changed to Awaiting User Response Railway 6 months ago


jake
EMPLOYEE

6 months ago

Unfortunately this is usually an issue with the Terraform provider. It tends to massively balloon the number of requests required.

Could you create an issue on the Terraform provider with:

- Your TF
- The amount of requests the provider is making (might have to run it in verbose mode)

It's very likely that, due to the provider overfetching, this rate limit gets hit


swimerize
PROOP

6 months ago

I agree that the terraform provider implementation could be a possible problem, however, in the course of troubleshooting this issue I have eliminated that possibility.

To recap where we are at the moment, I have reduced the operations that the terraform provider is performing to a single operation (see above network logs). I performed the single operation above after a few hours of inactivity. It seems that the rate limiter in Railway is not functioning correctly.

As a side note I had been using this terraform provider for a couple of days setting up and updating multiple services without any issues. The moment this problem first occurred, I have been unable to resume using the terraform provider (even when limiting the test to a single network call hours after the cool-down period). This issue persisted even when trying to create a new project from scratch with a fresh terraform state.

Let me know if there is any other information I can provide that will help you zero in on the issue. Thanks again for you help!


Status changed to Awaiting Railway Response Railway 6 months ago


sarahkb125
EMPLOYEE

6 months ago

Hi there - we appreciate your concern. Our rate limits are as we've mentioned in this thread - 1000 req per hour, and 50 req per second. We'll do some investigation, but otherwise we would suggest throttling your requests.


Status changed to Awaiting User Response Railway 6 months ago


Status changed to Solved sarahkb125 6 months ago


sarahkb125

Hi there - we appreciate your concern. Our rate limits are as we've mentioned in this thread - 1000 req per hour, and 50 req per second. We'll do some investigation, but otherwise we would suggest throttling your requests.

swimerize
PROOP

6 months ago

Please see my above comments. The entirety of the problem manifests itself when doing a single request.


Status changed to Awaiting Railway Response Railway 6 months ago


sarahkb125
EMPLOYEE

6 months ago

Ok so what might be happening: you were hitting Cloudflare's rate limits, and they marked your IP as suspicious. Usually this clears out after a day or two. I would suggest to do nothing, then try again on Monday. Let us know then


Status changed to Awaiting User Response Railway 6 months ago


Railway
BOT

6 months ago

✅ The internal ticket Rate limit hit early has been marked as completed.


swimerize
PROOP

6 months ago

Good news is that it seems as if waiting a couple of days did the trick. Out of an abundance of caution, I'll execute each command moving forward manually. I'll report back if I get blocked again. Thank you again for your help!


Status changed to Awaiting Railway Response Railway 6 months ago


Status changed to Solved sarahkb125 6 months ago


Loading...