Hey there! We've found the following might help you get unblocked faster:

• 🧵 Logs activity of who setting the usage limit

• 🧵 PostgreSQL Service Crash Loop - "No space left on device" with 99% Free Storage

• 📚 Use Cases

If you find the answer from one of these, please let us know by solving the thread!

hey there, unfortunately we’re unable to change this limit.

Best,

Nico

Is there alternative or suggestion to host such requirements within Railway infrastructure? e.g. dedicated bucket instead of volume, dedicated otel host etc.

I will direct your thread to our community for some help with your setup

Hey nazeb,

This is a classic and tricky problem when dealing with high-volume telemetry data. You've correctly identified that the issue isn't disk space but the filesystem's inode limit, which is a hard constraint you can't get around by simply adding more storage.

Since the inode limit on Railway volumes can't be changed (as confirmed by staff), the solution lies in changing your data architecture. Instead of writing a massive number of small files to the volume, you need a different approach to handle the data stream.

Here are three solutions, ranging from a quick fix to a more robust, long-term architecture.

Option 1: Batch Your File Writes (The Quick Fix)

The simplest solution is to stop creating so many files. Instead of writing each log or trace individually, you can configure your OpenTelemetry Collector to batch data and write to fewer, larger files.

• How it works: The Collector buffers data in memory for a short period (e.g., a few seconds or until a certain size is reached) and then flushes it to a single file in one go.

• Implementation:

  1. Adjust the configuration of the fileexporter in your OTEL Collector to manage batching and file rotation based on size, not time.

  2. A more advanced version of this is to write the data to an intermediary buffer, like a single SQLite database file. This consolidates all writes into one file, completely bypassing the inode limit for new data.

• Pros: Quickest to implement; likely just a configuration change.

• Cons: You're still managing raw data in files, which is difficult to query and analyze effectively.

Option 2: Offload to an External Object Store (S3-Compatible)

You mentioned a "dedicated bucket," and that's a great instinct. While Railway volumes aren't suited for this, you can use an external S3-compatible object storage service, which is designed to handle a virtually infinite number of objects (files).

• How it works: Your OTEL Collector sends data directly to an external storage provider's API instead of writing it to the local /data volume.

• Implementation:

  1. Sign up for a low-cost S3-compatible service like Cloudflare R2 or Backblaze B2.

  2. Configure your OTEL Collector to use an S3-compatible exporter (like the s3exporter).

  3. The Collector will stream data directly to your new bucket. No files are ever written to your Railway volume, using zero local inodes.

• Pros: Relatively easy to set up, cost-effective, and highly scalable for storage.

• Cons: Retrieving and analyzing the data can still be cumbersome compared to a proper database.

Option 3: Deploy a Specialized Observability Backend (Best Practice)

This is the industry-standard and most robust solution. OpenTelemetry is designed to send data to specialized databases that are optimized for ingesting, storing, and querying logs, metrics, and traces.

• How it works: You deploy another service (or services) on Railway that runs a dedicated observability backend. Your OTEL Collector then forwards all data to this service over the internal network.

• Implementation:

  • For Logs: Deploy a Loki service. Configure your OTEL Collector with the lokiexporter.

  • For Metrics: Deploy a time-series database (TSDB) like VictoriaMetrics or Prometheus. Use the prometheusremotewriteexporter.

  • For Traces: Deploy a tracing backend like Grafana Tempo or Jaeger.

• Pros: The most powerful and scalable solution. It solves the inode problem permanently and provides you with powerful tools (like Grafana) to query, visualize, and set alerts on your data.

• Cons: Requires setting up and managing additional services.

Recommendation:

• If you need a fix right now, try Option 1.

• If you want a scalable storage solution without the complexity of a database, Option 2 is excellent.

• If you want a professional, long-term solution that lets you actually use your telemetry data effectively, Option 3 is the way to go.

Hope this helps you get unblocked!

Hi tjayfl, thanks for the insightful suggestion. It's helpful to confirm that this is a common approach for these tricky problems. Ideally, we'd like to use Railway to host our entire infrastructure and avoid managing buckets separately (separate billing, resource access management etc.). However, due to this current infrastructure limitation (which prevents full Railway hosting), we'll proceed with Option 2 while the service is still new. Thanks!

Hey, Railway actually has native buckets that can be enabled via a priority flag https://railway.com/account/feature-flags. This feature is fairly new and is actively being worked on. You may give it a try if your application isn't meant for production.