To be clear, this suddenly started happening on multiple of my projects, that have had no changes, and deployed properly as recently as yesterday. Simply clicking 'redeploy' is enough to get the error to happen. So I've changed nothing and this new error is popping up. Very confused.

Railway projects suddenly failing with new "Cannot find type definition file for 'mime'" Error

are you commiting your node_modules folder to github?

Nope!

I'm guessing something happened with Railway's nixpacks version, that has resulted in a brand new error on previously working code.

And as a result multiple projects that worked in a previous infra are now all crashing down. Shame.

have you tried looking for solutions to this error? it wouldnt be isolated to railway

I'm even trying to 'redeploy' instances that were previously working and they also crash with this new error.

Including ones from several months ago.

redeploy will redeploy, you might want a rollback instead

Yes, rollback will work, but that doesn't solve forwards.

I was just confirming that it's railway infra related.

while i agree with you when you say that a nixpacks version bump introduced something different, this would not be an error that you could only get on railway

saying that its railway infra related is not a correct sentence

I'm not deeply familiar with nixpacks, but basically multiple versions, multiple branches, multiple databases, ALL now fail on a redeploy or new deploy. Why would old (previously working) projects all suddenly fail in the same way.

Perhaps it's more correct to say the way railway deploys nixpacks is the issue? I may be wrong to even be looking at nixpacks, hence asking for help.

databases dont use nixpacks, lets keep the scope of this to your service please

I meant applications pointing at different databases.

have you tried looking for solutions to this error?

Just trying to get across that it's a sudden issue across a whole bunch of different applications with different setups, applications versions etc, which to me points to infra (railway, nixpacks, something else, not sure), but I have no idea which part of the infra to look at.

The 'error' is in a library my program made no change or has no specific reference to. And yes I've searched for solutions to it, but have found nothing useful.

what is the nixpacks version of the last working deployment?

I'm not sure how to find that, or even control it. As I'm only using nixpacks as it was recommended for railway. I'm not even sure I understand the concept of a 'nixpacks version'.

Is there a way to tell the nixpacks version on a running deployment?

look at the build table in the build logs

The most recent successful build was over 30 days ago, so I cannot see build logs. Any other way to see it? And on Railway is there even a way to force an older version of nixpacks, or is that inaccessible?

For reference, my nixpacks.toml looks like this:
```[phases.setup]
nixPkgs = ['nodejs', 'yarn', 'postgresql_16']

[phases.install]
cmds=['yarn install']```

Perhaps the nodejs version got bumped too far and has created this Typescript error in my projects? If so, I could perhaps lock the nodejs version (if that's even possible).

respectfully, with a nixpacks.toml like that im suprised it has ever worked

do you use yarn or npm locally?

Heh, beginning to realise this…

Yarn locally…

do you have a yarn lock file?

Yes.

do you have an npm lock file?

No, since I use yarn, I removed that and don't commit it either.

So I'm guessing I need to have a more 'locked down' nixpacks file?

do you use yarn v1?

is your yarn lock file in sync with your package.json?

Yeah, I keep yarn lock in sync with package.json with yarn install, and committing it all together.

nope, the opposite actually

why do you need postgresql_16 ?

or.. do you need it?

Because I use postgresql_16 in a script in my application that backs up my database to S3.

Since the DB on Railway are now postgres16, I needed that running on the server.

indeed

replace your nixpacks.toml with this

[phases.setup]
  nixPkgs = ['...', 'postgresql_16']

Note that I originally got that nixpacks recommended format from here (https://docs.medusajs.com/deployments/server/deploying-on-railway), my project is a Medusa app.

i know

Ok, I will try that now.

but i dont know why they would add that

Heh, fair enough. For interest, does that mean Railway will pick what to install? (nixpacks dummy, remember)..

nixpacks will detect this is a node app that uses yarn and install and run the applicable stuff

Ok, so do I also remove this from my nixpacks.toml?
[phases.install] cmds=['yarn install']

yes

Thank you, rebuilding and the nixpacks header shows this now:

╔══════════════════════════════ Nixpacks v1.21.2 ══════════════════════════════╗ ║ setup │ nodejs_18, yarn-1_x, postgresql_16 ║ ║──────────────────────────────────────────────────────────────────────────────║ ║ install │ yarn install --check-cache ║ ║──────────────────────────────────────────────────────────────────────────────║ ║ build │ cross-env npm run clean && cross-env medusa-admin build && tsc ║ ║ │ -p tsconfig.server.json ║ ║──────────────────────────────────────────────────────────────────────────────║ ║ start │ medusa migrations run && medusa start ║ ╚══════════════════════════════════════════════════════════════════════════════╝

looks better

Locally my yarn version is 3.5.0 if that matters

it does

and my node version is v20.3.0

set engines.node to 20

locally? or in railway?

it would be done on your local project and you push the changes to github

I see, it's done through package.json, changing that now.

does create-medusa-app use yarn or npm by default? yarn

Ooof, not sure, that was almost a year ago I ran that. I think medusa gave you the option, I seem to remember manually removing the npm stuff.

looks like its yarn lock file only

Ok, that package.json worked to force it to node 20 instead of 18. Learning! (thank you btw, was scouring the internet for hours with no luck, this help is far more directed).

that would be outlined here

So now it's building, but hitting that same 'mime' error now.

use yarn v1 locally and sync your lock file again

Is there a reason to be moving backwards with yarn? Is it because railway doesn't properly support yarn v2?

railway uses yarn v1

Ok, wondering if that will create issues with medusa, which I believe uses yarn v2.

they use v1

so delete your lock file and then yarn install to make a new one with v1

Gotcha, now my yarn says version 1.22.19

Doing that added this to my package.json, do I want the version to be added that specifically?

"packageManager": "yarn@1.22.19"

And it also added yarnPath: .yarn/releases/yarn-1.22.19.cjs to my .yarnrc.yml file…

dont do any of that please

railway uses yarn v1 already, all you need to do is use yarn v1 locally and have a v1 lock file in your project

I think when I rolled back to yarn v1 with yarn set version classic it added those two lines…so I will remove them.

Okay, so that's all re-built, still getting that mime error, any further ideas from the railway side to investigate and figure out when this error actually creeped in? That is, back to the original issue of old deploys working on the same codebase where new deploys are now failing.

now it comes down to some issue with your project / deps

what solutions have you tried to fix this?

Well that's where I'm confused, it deployed fully fine a couple months ago, it's just when I redeploy (same code, config, etc) where it starts to fail. The language around that error appears to be a typescript thing, so I'm guessing the typescript version changed in what gets deployed on railway in the past couple months (I had stepped away from the project for a bit). And so my next deploy after that gap in time just now fails. Is there any way to force Railway to deploy with nixpacks versions from a month ago? Or am I thinking about this incorrectly?

I wanted to add in the type definitions for mime as a test, but as far as I can tell, it's typed right inside the library. So there's nothing to add. I added the latest mime-types anyways, but that didn't work.

Interestingly, this is what I get from yarn why mime - I have nothing about mime in my package.json though.

├─ @types/mime@npm:4.0.0 │ └─ mime@npm:4.0.1 (via npm:*) │ └─ send@npm:0.18.0 └─ mime@npm:1.6.0 (via npm:1.6.0)

please can we take the blame away from railway, i understand how it looks but this is not an issue with the platform

Not blaming, just trying to see that it may be something to do with infra (more likely that I haven't controlled something and it's simply manifesting with an infra change)

Is there any way to force Railway to deploy with nixpacks versions from a month ago?
yes

Or am I thinking about this incorrectly?
yes

I simply want to investigate that as it may help me find the bug.

pinning this on railway is only going to cloud your judgment

It's more that if I can find an old infra it works on, I can step forward and may be able to discover the source of the issue or better debug. But in parallel I'm also trying to figure out the bug with this 'mime' library and typing.

Arg, somehow yarn has 're-updated' itself to v3.5.0. Not sure how that happened.

thats not ideal

Looks like the moment I removed this line from the .yarnrc.yml it reverted to the v3.5.0

so that's interesting.

remove yarn v3 from your computer

Yeah, that's what I just figured is a good next step.

Ok, ran this command corepack prepare yarn@1.22.19 --activate and that worked, will reinstall and redeploy now with yarn v1 (as apparently I never did)

So this is very weird… I ran yarn why mime :

```~/sites/medusa-app/backend$ yarn why mime
yarn why v1.22.19
[1/4] Why do we have the module "mime"…?
[2/4] Initialising dependency graph…
[3/4] Finding dependency…
[4/4] Calculating file sizes…
=> Found "mime@1.6.0"
info Reasons this module exists

• "express#send" depends on it

• Hoisted from "express#send#mime"
  info Disk size without dependencies: "76KB"
  info Disk size with unique dependencies: "76KB"
  info Disk size with transitive dependencies: "76KB"
  info Number of shared dependencies: 0
  => Found "@types/mime#mime@4.0.1"
  info This module exists because "@types#express#@types#serve-static#@types#mime" depends on it.
  info Disk size without dependencies: "168KB"
  info Disk size with unique dependencies: "168KB"
  info Disk size with transitive dependencies: "168KB"
  info Number of shared dependencies: 0```

And then checking express, it's the latest version:

~/sites/medusa-app/backend$ yarn list express yarn list v1.22.19 warning Filtering by arguments is deprecated. Please use the pattern option instead. ├─ @medusajs/admin@7.1.12 │ └─ express@4.18.2 └─ express@4.19.2

So what doesn't make sense is how latest version of express seems to be using a very OLD version of mime (1.6.0) but a very NEW version of its types (4.0.1).

My package.json sets express as "express": "^4.17.2",

So, PERHAPS the bug is in express…digging in deeper but does this at all make sense to you?

have you deleted your node_modules folder and then ran all the same commands railway does to install deps, build and run your project right?

I did a yarn cache clean, but you're right, should have nuked node_modules for a fresh start.

Done, didnt' change anything.

and then ran all the same commands railway does to install deps, build and run your project right?

Innnnteresting. Getting the same local error now on yarn build. Only since getting properly on yarn v1. This should help debug without having to deploy!

More interesting, I just used medusa-create-app to make a fresh medusa install, and the response to yarn why mime is different… NO idea why this would be with my package.json, which refers to the SAME versions of express…

```=> Found "mime@1.6.0"
info Reasons this module exists

• "express#send" depends on it

• Hoisted from "express#send#mime"
  info Disk size without dependencies: "76KB"
  info Disk size with unique dependencies: "76KB"
  info Disk size with transitive dependencies: "76KB"
  info Number of shared dependencies: 0```

very odd

the joys of node dependencies

Yep, using this to dig further… thank you for your help to date. Recognise this is no longer (that is, never was) a 'railway' thing but nevertheless has been extremely useful.

So damn weird…

This is what the express and send versions are in the FRESH install:

~/sites/medusa-new-test$ yarn list express warning Filtering by arguments is deprecated. Please use the pattern option instead. ├─ @medusajs/admin@7.1.12 │ └─ express@4.18.2 └─ express@4.19.2 ~/sites/medusa-new-test$ yarn list send warning Filtering by arguments is deprecated. Please use the pattern option instead. └─ send@0.18.0

And in my install, after removeing the node_modules folder and the yarn.lock file:
~/sites/medusa-app/backend$ yarn list express warning Filtering by arguments is deprecated. Please use the pattern option instead. ├─ @medusajs/admin@7.1.12 │ └─ express@4.18.2 └─ express@4.19.2 ~/sites/medusa-app/backend$ yarn list send warning Filtering by arguments is deprecated. Please use the pattern option instead. └─ send@0.18.0

Banging head on wall at this point. Same package.json, same actual versions, but somehow this mime v4 in my project seems to be screwing everything up:
=> Found "@types/mime#mime@4.0.1" info This module exists because "@types#express#@types#serve-static#@types#mime" depends on it.

Going deeper with yarn why @types/mime:

My project:
```~/sites/medusa-app/backend$ yarn why @types/mime
yarn why v1.22.19
[1/4] Why do we have the module "@types/mime"…?
[2/4] Initialising dependency graph…
[3/4] Finding dependency…
[4/4] Calculating file sizes…
=> Found "@types/mime@4.0.0"
info Reasons this module exists

• "@types#express#@types#serve-static" depends on it

• Hoisted from "@types#express#@types#serve-static#@types#mime"
  info Disk size without dependencies: "196KB"
  info Disk size with unique dependencies: "272KB"
  info Disk size with transitive dependencies: "272KB"
  info Number of shared dependencies: 1
  => Found "@types/send#@types/mime@1.3.5"
  info This module exists because "@types#express#@types#express-serve-static-core#@types#send" depends on it.
  info Disk size without dependencies: "24KB"
  info Disk size with unique dependencies: "24KB"
  info Disk size with transitive dependencies: "24KB"
  info Number of shared dependencies: 0

And the new test project:

~/sites/medusa-new-test$ yarn why @types/mime
yarn why v1.22.19
[1/4] Why do we have the module "@types/mime"…?
[2/4] Initialising dependency graph…
[3/4] Finding dependency…
[4/4] Calculating file sizes…
=> Found "@types/mime@1.3.5"
info Reasons this module exists

• "@types#express#@types#serve-static" depends on it

• Hoisted from "@types#express#@types#serve-static#@types#mime"

• Hoisted from "@types#express#@types#express-serve-static-core#@types#send#@types#mime"
  info Disk size without dependencies: "24KB"
  info Disk size with unique dependencies: "24KB"
  info Disk size with transitive dependencies: "24KB"
  info Number of shared dependencies: 0```

welp time to start over with a new medusa project

Hah, no way… There must be some way to rectify this. My package.json isn't that much different from default. Is there anything to wipe aside from node_modules and the yarn cache that could be messing w this?

i honestly wouldnt know

WOW! Adding this to my package.json got past that failing build step:

"resolutions": { "@types/mime": "1.3.5" }

Not a great fix, forcing a dependency like this, but at least helps dig deeper…

thats node for you 🙂

One last question, does this seem right in the nixpacks like this with --check-cache? Shouldn't I be wiping the cache in Railway?

╔════════════════ Nixpacks v1.21.2 ═══════════════╗ ║ setup │ nodejs_20, yarn-1_x, postgresql_16 ║ ║─────────────────────────────────────────────────║ ║ install │ yarn install --check-cache ║ ║─────────────────────────────────────────────────║ ║ build │ yarn run build ║ ║─────────────────────────────────────────────────║ ║ start │ yarn run start ║ ╚═════════════════════════════════════════════════╝

that is odd

its doing that because you have a yarnrc.yml file

This is what's in my yarnrc.yml file, is this needed or should i nuke that file?

nodeLinker: node-modules

yeah get rid of it

Great, so several wins:

• I undersand nixpacks better and have a healthier use of it on Railway

• You've nicely updated Medusa's documentation to avoid future mishaps like mine

• I "sorta" understand that I have a node dependencies issue and have a short-term hack around it.

Wins and beers around!!

and at this point your build has gone out and you are back up?

Nope, got a bit of a pathway to get there…

note nixpacks looks like this now:

║ install │ yarn install --frozen-lockfile ║

thats what id expect it to be

what other issues are in the way?

Well now there seems to be an issue with medusa-admin not properly building before I can fully get to my prior state (says that date-fns package isn't available even though it's in the lockfile). I expect I'll get there though, and at that point, Medusa's routes suddenly stopped being registered for some of my medusa plugins and therefore the server application failed to start. Everything I've been blocked on today was simply to try to upgrade Medusa to the latest version to see if I just happened to catch a bug that only manifest in my project at that version. Have done all the necessary database migrations, etc.

All of this is very strange, as again there was no change to my codebase, environment variables, anything (hence my looking at infrastructure). All seems to be quite the tangle. Frustrating as it was all just chugging away working fine from code back before Christmas until last month and then just suddenly started falling apart. Natural return of order to chaos I suppose.

Hmmm, have just figured out one more 'Railway config' thing I don't understand. I had my nicely working from before Railway instance hooked up to github, and THAT one is failing during build, however I just made a 'fresh' one and it's working. Pointing at the same github branch though. Is there any history / memory / cache at all the Railway github instances hold onto between code deploys? Not sure why the build step would fail on one and succeed on another otherwise..

railway does do build caching on a per service bases

(it's been a service for a while so perhaps it's an old 'version' of the service too, it's been up since the fall)

Any way to 'clear' the cache?

NIXPACKS_NO_CACHE=1

In the environment variables I assume… Thank you!

in the service variables

Nice. Trying that to see if I can bring my service back to life. If that doesn't work, I'll just use the 'new' one I just made (that works) and then configure it to use SSL (through DNS) etc. to match my 'old' one then retire it. Not my fave method but it'll work.

configure it with SSH (through DNS) etc
huh?

Yep it worked! So that dependency thing in my admin build must have been from a cache issue. Is there a good reason to actually keep caches on in Railway? Seems unfortunate if they can mess w dependencies in this way.

Basically, I meant using the Networking to set a custom domain, and using DNS to get it to work (with SSL). Sorry spoke unclearly.

Is there a good reason to actually keep caches on in Railway?
it would normally speed up builds

whats ssh have to do with railway?

It's not directly, it's more that I use Railway to set redirects to https:// then then I have to set up that domain DNS to the proper Railway server, AND get the SSL (https) certificate. All just tasks that come off of the change in Railway.

odd because railway doesnt offer ssh

Awwww, shoot, everytime I said SSH up there I actually meant SSL, and now I understand your confusion…Oops! Edited my messages for clarity.

haha quite confused

Just talking about this really… and how it requires DNS configuration (completely ourside and not related to Railway) in order to function properly.

btw you can move your domain over to a new railway service without touching any dns stuff

but i assume you already did all that, so this is info for a next time

Even better, I got the original one working (by nuking the cache through that service variable) so didn't need to migrate at all.

oh perfect

Hah, get this absolute node craziness. Just on a whim, I decided to remove the resolution for mime now that it was all working, and guess what? It works both locally and on Railway. Makes no sense at all, but perhaps it 'kicked' it out of a weird dependency funk by just getting it to work once…

I mean it does sort of make sense as there's really no logical reason it should have been versioning mime in that inconsistent way…

(aka node magic)

this is indeed node magic

Further magic. Have switched back to the default NIXPACKS_NO_CACHE=false and lo-and-behold it all still works… Seems this was a 'spider-web-clearing' exercise…

Ok, full circle and in summary, I now have ALL environments [local, test (on railway), staging (on railway) and prod (on railway)] restored and properly running through a combination of:

• upgrading medusa

• cleaning up the nixpacks as per your suggestions

• clearing the nixpacks cache in Railway with the NIXPACKS_NO_CACHE service variable

• doing necessary database migrations (unrelated to this issue, but had to be done)

Of interest, to fix prod I did the least amount of manual stuff. That environment never saw the 'resolution' used to fix the 'mime' issue, I simply upgraded from November codebase to now. Then it presented the 'mime' issue, so I added the NIXPACKS_NO_CACHE, redeployed and the issue went away and everything deployed perfectly. So that's of interest for anyone presented with a similar future issue, try a pass of that (you can turn it back to default afterwards) for these weird issues presenting on Railway (or any deployment environment) when using Nixpacks. At this point, I'm actually unsure I needed to upgrade Medusa at all, perhaps the nixpacks cache fix on the old codebase would have been enough, but I can't know at this point as I fixed too fast.

I'm pretty sure NIXPACKS_NO_CACHE=false still disables cache, I don't think nixpacks checks if that value is a Boolean, I think it only checks if the value is set to anything and if it's set to anything it disables build caching

and I know that at the very least it doesn't clear cache, it just won't use cache for the builds it set on