At $10/GB/month, a service sitting at ~2.62 GB of RAM continuously costs roughly $26/month, which matches the $26.70 memory line item on your most recent invoice (Jul 5 - Aug 5, $28.26 total after the $5 Hobby included-usage discount). Memory is by far the dominant cost; vCPU, disk, and egress are each under $0.20. The RAM figure shown in service metrics is what gets billed.

Reducing that figure is a MySQL tuning question (your temptable_max_ram at 1 GB, the allocator, thread/table caches, and how the container reports memory are all levers), which the community can help with. You can set up usage alerts and limits and see per-service costs at railway.com/workspace/usage.

Thanks for the clarification. We have continued investigating and now have an important observation.

Before restarting MySQL, memory usage had been sitting almost continuously at approximately 2.62 GB, despite the database itself containing only about 18.6 MB of user data and having very low CPU activity.

We restarted MySQL as a diagnostic test. Immediately afterward, memory dropped to approximately 327 MB. Without any meaningful increase in database size, we then observed the following progression:

327 MB → 333 MB → 425 MB → 483 MB → 545 MB

The important behavior is that memory increased following activity but did not appear to return to the previous baseline afterward. Another redeployment later reset memory again to approximately 232 MB.

We also verified the following MySQL settings:

• innodb_buffer_pool_size = 128 MB
• innodb_dedicated_server = OFF
• performance_schema = OFF
• temptable_max_ram = 1 GB
• tmp_table_size = 16 MB
• table_open_cache = 4000
• thread_cache_size = 9
• Database size ≈ 18.6 MB

Therefore, the 2.62 GB steady-state memory consumption does not appear to be explained by the InnoDB buffer pool or by the size of the database.

Before the restart, we also observed unusually high cumulative connection/thread counters over the long-running MySQL process:

• Connections = 23,836
• Threads_created = 16,007
• Aborted_clients = 21,389
• Max_used_connections = 118
• Open_tables = 1,875
• Created_tmp_tables = 54,133
• Created_tmp_disk_tables = 0

Could you please advise how we can determine which MySQL memory component is responsible for the steadily retained RSS?

In particular, we would like to distinguish between:

allocator memory retention/fragmentation, per-connection/thread memory, table/thread caches, TempTable memory, or another MySQL/container memory component.

Our main concern is not simply limiting MySQL memory. We would first like to understand why a fresh MySQL process starts around 230–330 MB and progressively grows toward multi-GB usage without returning to its previous baseline, because that retained memory is directly increasing our Railway bill.

If there are Railway-side metrics or container-level diagnostics that can break down the billed memory/RSS further, we would appreciate your help reviewing them

The RAM figure in service metrics is what gets billed, at $10/GB/month. We don't expose per-process memory breakdowns or container-level cgroup stats beyond the aggregate memory, CPU, disk, and network metrics shown in the dashboard. Diagnosing which MySQL memory subsystem is retaining the RSS (allocator fragmentation, temptable_max_ram, thread/table caches, etc.) is MySQL-side tuning that the community can help with on the open bounty.

the 2.62 gb railway metrics is not coming from innodb,thread allocations or database tables . my best guess is glipbc heap fragmentation combined with temptable memory retentioninside the mysqld process over its 199 day uptime check it with a simple restart

the trigger=with over 54000 tempory tables created during this uptime , mysql repeatedly allocated large contigious chunks of memory (up to 1gb)using standard glibc malloc standard glibc does not return memory to theOS when high water marked allocations are freeded it holds onto the heap arenas for future requests combined with 16000 completedc threads and 21 000 aborted connnections constantly allocating and freeiing snall connection buffers in parallel the heap became severely fragmented

Thanks — your glibc fragmentation / retained heap theory is looking increasingly plausible.

We restarted MySQL and RSS dropped from ~2.6 GB to ~300 MB. Since then it has climbed again to ~654 MB. Comparing the post-restart baseline (~328 MB) with the current state shows that ~319 MB of the ~326 MB increase is anonymous memory inside mysqld, while thread count stayed essentially unchanged (67 → 68) and AnonHugePages stayed almost flat (~256 → 258 MB).

We also confirmed that innodb_buffer_pool_size is fixed at 128 MB, so this growth is not InnoDB buffer pool expansion. We reduced temptable_max_ram to 256 MB, but RSS continued to grow.

So the remaining question is whether this anonymous growth is specifically retained/fragmented glibc arenas or another live allocation inside mysqld. We’re now focusing on the anonymous mappings/arenas to distinguish between the two.

We confirmed that most of the mysqld RSS growth is anonymous memory, and /proc/1/smaps shows many repeated ~64 MB arena-like anonymous mappings consistent with glibc malloc arenas. The main [heap] mapping is only ~5 MB.

Can you confirm whether Railway’s mysql:9 image uses glibc malloc for mysqld, and whether setting MALLOC_ARENA_MAX (for example 2 or 4) as a Railway service environment variable is supported/recommended?

Also, is there a way inside the Railway container to verify the current glibc arena count and how much memory is allocated vs free/retained inside those arenas before we change anything?

The MySQL service is deployed directly from the official mysql Docker image on Docker Hub, so the allocator and runtime internals are whatever that upstream image ships. Any environment variable you add to the service in Railway is passed to the container at runtime, so setting MALLOC_ARENA_MAX there will make it available to the process. Whether mysqld picks it up and how to inspect arena state are MySQL-side questions the community can help with on the open bounty.

Thanks. Our main goal here is to reduce the long-term RSS footprint and therefore the Railway memory cost.

We are seeing mysqld RSS increase after workload and not return close to its previous baseline, even when the workload has finished. Most of that growth appears in anonymous mappings rather than the main heap.

From Railway's perspective, are there any recommended settings or container-level approaches to make unused mysqld memory return to the OS more aggressively?

In particular, we are considering:

• limiting glibc arenas with MALLOC_ARENA_MAX
• reducing MySQL thread/cache-related memory
• tuning InnoDB buffer pool size
• reducing retained memory after temporary workload spikes

Our priority is not maximum MySQL performance; it is keeping the steady-state RAM footprint low and preventing RSS from continuously ratcheting upward over time.

If Railway billing is based on container RSS/memory usage, we would especially like to know whether there is a recommended configuration for small MySQL services to minimize retained memory.