Allow SSH access to containers via CLI

selfagencyHOBBY

4 years ago

I have a project that works in my local environment but not once pushed to Railway. If I could login to the container like I can locally, I could determine what the differences are that are causing the problem. Add a function like the "connect" command in the CLI to connect to the actual container.

Completed

0 Threads mention this feature

61 Replies

AnonymousTRIAL

4 years ago

does this feature exist in heroku or dokku ? cloudfoundary uses buildpacks too and I am guessing you can't ssh into the container..don't know


Anonymous

4 years ago

Islam El'ewady: on Heroku there's a "console" option that lets you drop into a shell inside a container to do live debugging. Unsure about CLI access, but it exists on their web interface.
Behaviour like this for Railway would be a huge bonus.


AnonymousTRIAL

4 years ago

Islam El'ewady: on Heroku there's a "console" option that lets you drop into a shell inside a container to do live debugging. Unsure about CLI access, but it exists on their web interface.
Behaviour like this for Railway would be a huge bonus.

Erisa A: agree on dokku i can do dokku enter app and i am in for a treat


4 years ago

Full Disclosure: We're pretty staunchly opposed to allowing people to SSH in because it affects the immutability property of the infrastructure
Example:
You go in and change a setting while SSH'd to fix an issue
You create a deployment
That deploy is now broken
Happy to dig into what exactly you'd want out of SSH. If it's file retrieval, we can look into allowing you to inspect the filesystem of the container itself (as read only)

Status changed to Under Review jake over 3 years ago


Anonymous

4 years ago

Full Disclosure: We're pretty staunchly opposed to allowing people to SSH in because it affects the immutability property of the infrastructure
Example:
You go in and change a setting while SSH'd to fix an issue
You create a deployment
That deploy is now broken
Happy to dig into what exactly you'd want out of SSH. If it's file retrieval, we can look into allowing you to inspect the filesystem of the container itself (as read only)

Jake Cooper: For me personally it would be about shell access for debugging rather than editing: if something goes wrong in a deployment then I can connect to a shell in the container and debug the problem there and then, before deploying the solution properly afterwards.
An alternative would be the ability to pull the image down locally and investigate there. Of course if your application is built with Dockerfile this is easy by building locally but in the case of buildpacks I found it quite hard to debug issues that occurred in the buildpack environment and not locally.


Anonymous

3 years ago

You can now basically do this by using Tailscale. It's not a replacement for it being built into Railway, but it’s a viable alternative. https://tailscale.com/blog/tailscale-ssh/


2 years ago

We think that there are enough legit use cases for this. Going to plan on how this might make sense for us.

Status changed to Planned angelo about 2 years ago


Anonymous

2 years ago

It's significant. I deploy a code server on Railway. If the code server backend crash, I will easily lose my code!


eyadabdallaHOBBY

a year ago

Any updates on this? Is there a way to inspect and retrieve files from the server?


a year ago

there is not as of yet


toshvelagaHOBBY

a year ago

yup this feature would be hella useful, haven't used tailscale yet but looking into that as an alternative


charliePRO

a year ago

One use case I'd really like SSH for is to simply understand the "Volume" file structure. My issue is I'm using Google Cloud so need to have the "credentials.json" somewhere on my production system but I can't commit it to the GitHub repo due to sensitive keys so it seems a Volume would solve the problem however I'm unsure exactly how that relates to my current project (e.g. a Django project) and this is the first time using a Volume so it feels a bit of stabbing in the dark


philosofonususPRO

a year ago

I believe this is super important feature, hard to debug big codebases without it


matejmPRO

a year ago

Agree, this is not a feature we would want to use every day, but will massively help to debug certain issues.


a year ago

could you give some examples of what this helps debug that you couldn't any other way?


n0rigHOBBY

a year ago

I just ran into a need for this on my wordpress site. Since it does plugin updates, I cannot edit the filesystem directory to delete a plugin I think has an issue.


a year ago

You would ideally want to delete that plugin locally and then push the changes.

Allowing SSH access is going to cause a lot of unforeseen issues and support burden.


philosofonususPRO

a year ago

The first case - I have a large heavy project which uses a lot of CPU and RAM. There is a memory leak. The memory leak doesn't seem to occur in the first 2-8 hours there is no sign of memory leak. After this period it is starting to spin up. I have tested it on localhost and can't reproduce it. So one of the viable options to test it is to run some debug commands to understand the issue better.
The second case - the project I described earlier has CLI to control operations and get/set/delete data. I don't want to make it public due to potential security issues, but still want to control it


franzwollangPRO

a year ago

Would very much appreciate this feature as well for debugging purposes. Perhaps it could be implemented in a limited way so that the immutability Jake (https://help.railway.app/feedback/allow-ssh-access-to-containers-via-cli-e4985f6f#p-4) referred to is not broken. Perhaps by only allowing certain commands?


a year ago

What kind of debugging purposes? why not build the image and debug locally?


ricardorothHOBBY

a year ago

I am currently deploying a Chatwoot image and part of the set up process involves Rails console commands. How can I execute them?


a year ago

It does? My Chatwoot template has 890 deploys and no one has yet to ask me about SSH access.


ricardorothHOBBY

a year ago

It does? My Chatwoot template has 890 deploys and no one has yet to ask me about SSH access.

Thanks brody, after further research, looks like there is an API alternative to the Rails console. Im looking at https://www.chatwoot.com/hc/user-guide/articles/1677497472-how-to-use-agent-bots
I´ll try that way.


philosofonususPRO

a year ago

The first case - I have a large heavy project which uses a lot of CPU and RAM. There is a memory leak. The memory leak doesn't seem to occur in the first 2-8 hours there is no sign of memory leak. After this period it is starting to spin up. I have tested it on localhost and can't reproduce it. So one of the viable options to test it is to run some debug commands to understand the issue better.
The second case - the project I described earlier has CLI to control operations and get/set/delete data. I don't want to make it public due to potential security issues, but still want to control it

brody What do you think? Is it enough to make ssh access viable? A lot of hostings provide this feature


franzwollangPRO

a year ago

Hey Brody, the use case is that I'm deploying an ancient PHP application that saves some state and configuration by creating certain folders and files. I set it up to run itself inside a volume for persistence. When, not if, there are problems, it would be helpful to be able to access those folders and files (including certain log output). I could in theory expose this access via the web server itself, but that seems like a step too far. As an alternative to live shell access, perhaps consider allowing read access to volumes at a minimum?


a year ago

philosofonusus, I'm not sure, if you had ssh access how exactly would you go about debugging that, that couldn't be done locally in a docker image?

franzwollang, this seems more along the lines of wanting a file browser for volumes and I think that's totally valid, but for logs you should be sending them to stdout/stderr.


jeremyccranePRO

9 months ago

Access to volumes via cli or browser would be hugely helpful for debugging. Often, deploying to a railway container is not the same as deploying locally. I can't remotely check my volume to make sure the proper file structure is being followed. Works great locally, fails in production.


9 months ago

Yes this would be handy. I'm trying to resolve some Rails issue but can't figure out how to actually get into the config 🫠


shivamgupta42HOBBY

9 months ago

What a disappointment that I read through 3 years of messages and still file access is not offered to users. I am new to railway and I have a use-case in which i needed to retrieve some important files from my container


yujongleeHOBBY

9 months ago

No SSH support is deal-breaker for all Elixir apps.


winreyPRO

8 months ago

Enter full of hope, leave disappointed.


magedsaeedHOBBY

8 months ago

Allowing SSH access for debugging is an essential feature, I guess. Hope to see this soon.


8 months ago

Please share how and what you would use SSH to debug?


zaidaldabbaghPRO

8 months ago

Agree, for example at times, when investigating failed builds, I like to SSH into my instance to ensure that my CMS code and scaffolding has installed correctly, e.g. in SilverStripe CMS, certain module assets are exposed to the webroot, as the 'vendor' url prefix is blocked from web-access by default. Another use case is importing MySQL dumps, with Render.com for example, I follow these steps to import my DB dump ...

# Secure copy db dump to MySQL Service container on Render
scp mysql-dump.sql srv-xyz@ssh.oregon.render.com:~

# SSH into MySQL Service container
ssh srv-xyz@ssh.oregon.render.com

# Import dump into newly created db
mysql -h mysql-host -u mysql --password=$MYSQL_PASSWORD mydb < mysql-dump.sql

magedsaeedHOBBY

8 months ago

Please share how and what you would use SSH to debug?

For example, I want to access log files that are not streamed to stdout. I want also to check services status via service/systemctl/supervisord etc. I may also access my app shell for some debugging (django shell). I do understand that the docker container will change its state since its last deployment from GitHub but I accept that.


qodeboyHOBBY

7 months ago

Gosh, this must be a core feature from day one. You're a k8s wrapper, act as so.

  1. Migrate/rollback migration.

  2. Inspect filesystem and logs.

  3. Run one-time jobs like imports from 3rd party service.

  4. Inspect and requeue failed jobs.

  5. million of other use-cases.

This and inability to deploy tags (lol good luck managing auto-deploying production envs) is a deal breaker for me and other folks who care about what and how they deliver to production.


jbirkeHOBBY

5 months ago

Just learned that connecting to my rails server and jumping into a console is not possible. Still determining if this is a dealbreaker but it very well might be. Being able to jump into a console, run various queries, validate assumptions, run some manual scripts, etc. feels like a necessity for many production applications. Hope to see this supported because I'm really enjoying the platform, but will likely need to switch because of this.


sarzixonPRO

5 months ago

Any updates on this topic? Is Railway team planning to implement this feature?


kvrushifaTRIAL

5 months ago

how is this still not a thing


levilansingTRIAL

4 months ago

Use case: I deployed a custom database docker container (forked from railway's postgres one), and I need to perform some first-time administration to create users for other services. railway connect does not allow me to connect because it claims there is no supported database found. I guess my only option is to temporarily expose the DB publicly (ew) for this one-time task or build a one-off docker container with a script? I tried deploying my own ssh server, but the TCP proxy doesn't seem to work for that use case as other support posts have confirmed.

I also agree with the other use cases here for debugging a deployment issue. SSH access is critical when services aren't deploying or behaving correctly and you need to investigate, especially in non-prod environments, but also in prod emergencies.


4 months ago

Previously we didn't want to do this because we were worried about people modifying the instance and having changes wiped on redeploy, but I think we've gotta/want to now

It's really important for the following reasons:

  • Languages like Ruby/Elixir have a repl that you can access that's critical for debugging

  • If you botch a migration for something like Postgres, you need access to run the pg_resetwal

  • Testing DNS and other container level actions

We'll do this next quarter for sure


brwajumaaHOBBY

4 months ago

Previously we didn't want to do this because we were worried about people modifying the instance and having changes wiped on redeploy, but I think we've gotta/want to now

It's really important for the following reasons:

  • Languages like Ruby/Elixir have a repl that you can access that's critical for debugging

  • If you botch a migration for something like Postgres, you need access to run the pg_resetwal

  • Testing DNS and other container level actions

We'll do this next quarter for sure

Finally...


jake

Previously we didn't want to do this because we were worried about people modifying the instance and having changes wiped on redeploy, but I think we've gotta/want to nowIt's really important for the following reasons:Languages like Ruby/Elixir have a repl that you can access that's critical for debuggingIf you botch a migration for something like Postgres, you need access to run the pg_resetwalTesting DNS and other container level actionsWe'll do this next quarter for sure

jimkringPRO

3 months ago

Thank you! There's stuff that simply can't be done/fixed/debugged without shell access to the running container. I'm evaluating moving a docker compose system over to railway and it's been very painful for me to debug why things aren't working quite right. Of course, once I figure it out, I'll change my configs and container builds, but I have to be able to debug...


sdavid14HOBBY

3 months ago

consider this a +1 for this feature. very hard to impossible to do certain types of debugging without it.


p4ulcristianPRO

3 months ago

Yes, please this feature is a must have. Next quarter sounds fascinating. (Clojure also uses repl)


danielroigTRIAL

2 months ago

I really love Railway but I always consider other SaaS for that reason. +1 for this feature


mcassanoPRO

2 months ago

Use case: I run python scripts to do large scale migrations. I get through his now by using Tailscale to attach my local dev machine to the Railway network but it is orders of magnitude slower than when running the migrations against my dev set up. I assume this is because my migrations leverage the Django ORM and it's just a ton of traffic that goes back and forth over the Tailscale wire. If I could ssh in to a Railway service then I would make an "ssh terminal" service in Railway, ssh into it and run my migrations on machines all local to each other.


matfish2PRO

2 months ago

Use case: Running ad-hoc commands from shell in NestJS.
If I understand correctly, current shell is not connecting to Railway's network, just pulls env vars locally, which forces me to use public DB URL, or call it from the Admin Frontend of my app.
This is super basic functionality, which I use every day at work. Very limiting, to the point I'm considering a different provider.


182exeTRIAL

2 months ago

praying for this (i literally just wanna put the logs on my 5 inch crt)


jake

Previously we didn't want to do this because we were worried about people modifying the instance and having changes wiped on redeploy, but I think we've gotta/want to nowIt's really important for the following reasons:Languages like Ruby/Elixir have a repl that you can access that's critical for debuggingIf you botch a migration for something like Postgres, you need access to run the pg_resetwalTesting DNS and other container level actionsWe'll do this next quarter for sure

winreyPRO

2 months ago

We have already migrated everything to a self-managed K8s cluster due to this issue, even though it has increased our costs fivefold compared to your service. We’ll consider returning once you provide support for it.


Status changed to In Progress brody about 2 months ago


a month ago

A big +1 for this.

It doesn't necessarily have to be via the CLI (although that would be cool), but any way to ssh into your app server to run admin console commands and the like would be excellent. Either via the CLI, a standard ssh from you terminal, or (ideally) an SSH session that you can run straight from your browser window. You could have this under the app server's settings tab, for example.

Something similar to the SSH window you can access via an App Service's dev tools you have in Azure would be great.


jake

Previously we didn't want to do this because we were worried about people modifying the instance and having changes wiped on redeploy, but I think we've gotta/want to nowIt's really important for the following reasons:Languages like Ruby/Elixir have a repl that you can access that's critical for debuggingIf you botch a migration for something like Postgres, you need access to run the pg_resetwalTesting DNS and other container level actionsWe'll do this next quarter for sure

a month ago

Fair enough; however, people can do this on virtually any platform anyway - whether that's their own managed on-prem servers or another cloud platform. I think most people know what they're getting into and can be trusted with admin tools like this (and if they don't: well, you only have to make that mistake once to learn your lesson )


danielhardej

Fair enough; however, people can do this on virtually any platform anyway - whether that's their own managed on-prem servers or another cloud platform. I think most people know what they're getting into and can be trusted with admin tools like this (and if they don't: well, you only have to make that mistake once to learn your lesson )

a month ago

For some insight on why we have been hesitant to do this, SSH has a large potential to do a lot of damage that would otherwise be slightly harder to achieve, or even spark more questions like I created a file in the shell redeployed and now it's gone, and sure that's on the user, but that user would open a support request asking to recover data or any other various common questions that we will have to sift through and turn down, thus increasing our support load, but in the end, there are too many needed usecases for us to punt on this any further.


danielhardej

A big +1 for this.It doesn't necessarily have to be via the CLI (although that would be cool), but any way to ssh into your app server to run admin console commands and the like would be excellent. Either via the CLI, a standard ssh from you terminal, or (ideally) an SSH session that you can run straight from your browser window. You could have this under the app server's settings tab, for example.Something similar to the SSH window you can access via an App Service's dev tools you have in Azure would be great.

a month ago

IIRC for the V0 release of this feature, it will only be integrated into the CLI.

Technically speaking, It won't actually be an SSH connection, but an authenticated QUIC stream that we just pass bytes back in forth in for the bash / sh shell.


burikellaPRO

a month ago

Is there any ETA on this?
BTW, is there any place where I can check the feature plans/status myself?


burikella

Is there any ETA on this?BTW, is there any place where I can check the feature plans/status myself?

a month ago

> Is there any ETA on this?

The current ETA is the end of this Q, but that's only an ETA.

> Is there any place where I can check the feature plans/status myself?

Not at the moment, but we are actively working on a public road map! we will announce its availability in one of our upcoming weekly changelogs.


a month ago

Update for everyone here- we have an internal demo, you should hear something from us in the coming weeks.


Status changed to In Progress david 26 days ago


eusoubrozaPRO

23 days ago

that'd be pretty useful


19 days ago

Hey @angelo and/or @brody - will SSH be available for the Empty Service resource too?


19 days ago

It won't, you'd need an actively running service to shell into.


19 days ago

Sorry, I should have been more specific about the use-case: So say you create a new Empty Service in your environment, connect a Docker image to the service, deploy and have it running, would you then be able to SSH into that service?

EDIT: being even more specific, use the Empty Service to run a GitHub Actions self-hosted runner by pulling the Actions runner application from GHCR as the source image for the service's settings, then SSH into the service to install additional dependencies/tools/configurations required for the workflows. Would that be possible?


19 days ago

Yes, anything that has /bin/bash or /bin/sh available will work.

Of course you likely wouldn't want to install anything since it's an ephemeral filesystem, you'd lose anything installed every deploy.


parmstarEMPLOYEE

16 days ago

Hello!

SSH HAS LAUNCHED!

https://blog.railway.com/p/ssh


Status changed to Completed parmstar 16 days ago


Allow SSH access to containers via CLI - Railway Help Station