rust deploy uses 10x more ram than local dev

hey guys, i have a rust api that is using 4-8gb of ram suddenly,

now for rust that's unheard of considering it used to chill at 200mb +/-

when running cargo run --release --package x
locally i get out to a max of 1.5gb ram (this includes compiling)

yet on railway it ticks out to 8gb at peak (this includes compiling)

takes roughly 26 minutes to compile, build, deploy on railway

There's a total of 950 +/- crate dependencies

projectId = 7c6a1d9f-d3d2-4808-96b1-10c8df175503

No deploy that happened in that weird spike, nor am i able to reproduce the low memory usage anymore

179 Replies

brody
EMPLOYEE

2 years ago

dockerfile or nixpacks?


nixpacks


// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = 'cargo run --package xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

brody
EMPLOYEE

2 years ago

you should not do cargo run as the start command, you should be building a release and then running the resulting binary


i had a cargo run --release


but i just changed it a few min ago


brody
EMPLOYEE

2 years ago

let's see your new nixpacks.toml file please


brody
EMPLOYEE

2 years ago

additionally, @salvage your input here would be greatly appreciated


// nixpacks.toml
providers = ['rust']

[phases.build]
cmds = ['echo building xylex image', 'cargo build --release --package xylex']

[phases.test]
cmds = ["cargo test --package xylex"]
dependsOn = ["build"]

[start]
cmd = './target/release/xylex'

[variables]
NIXPACKS_INSTALL_CACHE_DIRS = './nixpacks_cache'
NIXPACKS_BUILD_CACHE_DIRS = './nixpacks_cache'

brody
EMPLOYEE

2 years ago

yeah that looks appropriate now


yeah well i used to do it like this in the start but had tremendous amount of issues as you might remember


it would just brick and never deploy unless done via the cli


brody
EMPLOYEE

2 years ago

I do remember, did you also have those variables set? I don't see any purpose to them and can see them causing issues


yes


no


i hadn't set them yet


this is just me trying to get it working in a normal way


brody
EMPLOYEE

2 years ago

for good measure, remove them please


the nixpacks_cache ?


brody
EMPLOYEE

2 years ago

and then let me know if you run into further issues


brody
EMPLOYEE

2 years ago

yes, both those variables you have there


okay, gotcha 2 min


the target could be up to 20gb in some weird c folder hence i deferred it to the cache folder


brody
EMPLOYEE

2 years ago

I'm assuming the default cache stuff nixpacks does is sufficient


compiling rn


same issue as before


on railway and on nixpacks locally


local

1224469832414531600


brody
EMPLOYEE

2 years ago

just freezes there?


yes


milo
MODERATOR

2 years ago

god


i love railway


milo
MODERATOR

2 years ago

the rust provider in nixpacks honestly is shit


but rust


has been very painful to work with on railway


milo
MODERATOR

2 years ago

yeah i've moved to cargo-chef


milo
MODERATOR

2 years ago

in a dockerfile


i was reading about that yesterday


brody
EMPLOYEE

2 years ago

haha milo beat me to it


milo
MODERATOR

2 years ago

much better then nixpacks


brody
EMPLOYEE

2 years ago

agree with Milo here


milo
MODERATOR

2 years ago

also this issue looks unrelated to nixpacks


right okay, so you advise me to do the same


im not sure, it stalls local and on railway


milo
MODERATOR

2 years ago

nixpacks or


yes


milo
MODERATOR

2 years ago

okay nevermind


milo
MODERATOR

2 years ago

this is a nixpacks issue 😂



milo
MODERATOR

2 years ago

im not sure why it is making it stall that bad though


brody
EMPLOYEE

2 years ago

yes, I'm not a rust dev but have heard great things about cargo chef and when nixpacks is painful (almost always) dockerfiles are the way to go


we still chilling here


milo
MODERATOR

2 years ago

from yours truly


yeah it was the exact same issue months ago


somehow the cli fixed it


milo
MODERATOR

2 years ago

just use cargo-chef, see if that helps


omw


brody
EMPLOYEE

2 years ago

yep cargo chef in a dockerfile for sure


milo
MODERATOR

2 years ago

ping me if this happens again (or get brody to lol)



yeah as funnily enough we went to rust cus of low resource usage


whahaha


milo
MODERATOR

2 years ago

no you dont need to install it


milo
MODERATOR

2 years ago


you just make the recipe.json ?


i am


milo
MODERATOR

2 years ago

you do all that in the dockerfile



ah i see


milo
MODERATOR

2 years ago

jsut look at the example


milo
MODERATOR

2 years ago

this bit

1224470945012387800


should be able to copy most of that


milo
MODERATOR

2 years ago

yup



milo
MODERATOR

2 years ago

delete ur nixpacks.toml


milo
MODERATOR

2 years ago

make sure the Dockerfile is named Dockerfile


yes wait im showing u something




milo
MODERATOR

2 years ago

thats just nixpacks doing extra stuff


milo
MODERATOR

2 years ago

dont worry about that


milo
MODERATOR

2 years ago

please delete ur nixpacks,toml


brody
EMPLOYEE

2 years ago

you somehow needed up with two rust providers


brody
EMPLOYEE

2 years ago

but yeah, Dockerfile time


well i think it's a bug in NixPacks when cargo workspaces are enabled


as xylex_wss is not the main bin


it starts building a random workspace and the core bin


milo
MODERATOR

2 years ago

nixpacks and rust dont mix unfortunately


going to docker now ig


milo
MODERATOR

2 years ago

yup


fucking hate docker time consuming


milo
MODERATOR

2 years ago

not really


milo
MODERATOR

2 years ago

this will be easy


brody
EMPLOYEE

2 years ago

yeah you won't need any kind of complex Dockerfile


okay so i changed these from app to xylex

1224473010250186800


anything besides that ?


my cargo is called xylex


milo
MODERATOR

2 years ago

what is the binary u wanna


milo
MODERATOR

2 years ago

okay


milo
MODERATOR

2 years ago

cool


milo
MODERATOR

2 years ago

should be good


railway autodetects this yes


milo
MODERATOR

2 years ago

yes



milo
MODERATOR

2 years ago

let us know how it goes


brody
EMPLOYEE

2 years ago

and please cancel those old deployments


whahah yes i will



i am guessing one of these should've been app still


brody
EMPLOYEE

2 years ago

I assume that means you need to install openssl


bros too experienced


milo
MODERATOR

2 years ago

RUN apt update -y && apt install openssl libssl-dev -y


milo
MODERATOR

2 years ago

put that in the line before the ENTRYPOINT instruction


brody
EMPLOYEE

2 years ago

perfect


man idk where milo came from but they needa give him a raise


bro's got shortcuts to paste solutions


brody
EMPLOYEE

2 years ago

Milo is the rust professional around here


what a king


brody
EMPLOYEE

2 years ago

fr


milo
MODERATOR

2 years ago

just from my personal rust app dockerfile haha


are you in the railway dev team milo


brody
EMPLOYEE

2 years ago

he's not, just a community mod


i see, so it is personal projects/companies that u used rust in


milo
MODERATOR

2 years ago

yeah, some enterprise things as well


milo
MODERATOR

2 years ago

rust is really good and scalable in my experience


yes we switched in january it is amazing


milo
MODERATOR

2 years ago

haha



i am unsure if the ssl install was the right call


certificate verify failed"


milo
MODERATOR

2 years ago


milo
MODERATOR

2 years ago

what dependencies are u using


brody
EMPLOYEE

2 years ago

might need ca-certificates too? just a guess


shouldn't matter it deploys fine on ubuntu server


milo
MODERATOR

2 years ago

can you send ur cargo.toml


yes


brody
EMPLOYEE

2 years ago

I mean this is not the same at all though


milo
MODERATOR

2 years ago

are you calling reqwest directly


alright brace for impact guys


// cargo.toml 
[package]
name = "xylex"
version = "0.1.0"
edition = "2021"


[dependencies]
polars = {version = "0.35.0", features = ["lazy", "rolling_window"] }
reqwest = { version = "0.11.23", features = ["json", "blocking"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
supabase_rs = "0.2.2"
thiserror = "1"
rocket = "0.5.0"
serde_json = "1.0.111"
dotenv = "0.15.0"
mail-send = "0.4.6"
mail-builder = "0.3.1"
rsi = "0.1.2"
csv = "1.1"
arrow = "*"
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
clokwerk = "0.4.0"
image = "0.24.8"
rusttype = "0.9.3"
rand = "0.8.5"
qrcode = "0.13"
async-trait = "0.1.77"
futures = "0.3.3"
imageproc = "0.23.0"
tokio-tungstenite = { version = "0.21.0", features = ["native-tls"] }
serde_yaml = "0.9.31"
lazy_static = "1.4.0"
trading_sessions = "0.1.2"
md-5 = "0.10.6"
hex-literal = "0.4.1"
fibonacci_retracement = "0.1.0"
futures-util = "0.3.30"
futures-channel = "0.3.30"
url = "2.5.0"
sha256 = "1.5.0"
strum = "0.26"
strum_macros = "0.26"
sled = "0.34.7"

xylex_spread_tracker = { path = "./xylex_spread_tracker" }
xylex_card_generator = { path = "./xylex_card_generator" }
xylex_hash_router = { path = "./xylex_hash_router" }
xylex_scheduler = { path = "./xylex_scheduler" }
xylex_discord = { path = "./xylex_discord"}

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true


[profile.release]
opt-level = 3

[workspace]
members = [
    "xylex_wss",
    "xylex_card_generator",
    "xylex_spread_tracker",
    "xylex_scheduler",
    "xylex_hash_router",
    "xylex_discord"
]
[package]
name = "xylex_card_generator"
version = "0.1.0"
edition = "2021"

[dependencies]
image = "0.25.0"
imageproc = "0.24.0"
reqwest = "0.12.2"
rusttype = "0.9.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"
[package]
name = "xylex_discord"
version = "0.1.0"
edition = "2021"

[dependencies]
serenity = { version = "0.12", default-features = false, features = ["client", "gateway", "rustls_backend", "model", "collector"] }
tokio = { version = "1.35.1", features = ["full", "macros", "rt-multi-thread"] }
chrono = { version = "0.4", features = ["serde"] }
serde  = { version = "1.0.196", features = ["derive"] }
thiserror = "1"
serde_json = "1.0.111"
dotenv = "0.15.0"
serde_yaml = "0.9.31"
reqwest = "0.12.2"

[dev-dependencies]
anyhow = "1"

[profile.dev]
opt-level = 0
incremental = true

[profile.release]
opt-level = 3
[package]
name = "xylex_hash_router"
version = "0.1.0"
edition = "2021"

[dependencies]
dotenv = "0.15.0"
serde_json = "1.0.115"
supabase_rs = "0.2.2"
[package]
name = "xylex_spread_tracker"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = "1.10.4"
reqwest = "0.12.2"
serde = "1.0.197"
serde_derive = "1.0.197"
serde_json = "1.0.115"
serde_yaml = "0.9.34"
supabase_rs = "0.2.2"
tokio = "1.36.0"
[package]
name = "xylex_wss"
version = "0.1.0"
edition = "2021"

[dependencies]
amqprs = "1.5.3"
serde = "1.0.197"
serde_json = "1.0.114"
serde_yaml = "0.9.34"
tokio =  "1.36.0"

brody
EMPLOYEE

2 years ago

you can attach files lmao


oh


well this works too


milo
MODERATOR

2 years ago

add rustls-tls as a feature of reqwest


milo
MODERATOR

2 years ago

also, half these depenendices are obsolete and can be replaced by something better, not needed at all, or stdlib


i mean bro


im sure


but i'm not sure if refactoring 155k lines of rust is something i wanna do tonight


milo
MODERATOR

2 years ago

i mean, you don't need strum-macros if you have strum, lazy-static is just stupid these days, dotenv should use dotenvy and thats just to name a few


milo
MODERATOR

2 years ago

anyways


milo
MODERATOR

2 years ago

do this please


yes its deploying


milo
MODERATOR

2 years ago

okay cool


brody
EMPLOYEE

2 years ago

damn Milo knows his stuff


yeah but what's the difference between dotenv and dotenvy


milo
MODERATOR

2 years ago

dotenvy is maintained and isn't.. bad



i see ill make a linear ticket ig


milo
MODERATOR

2 years ago

same api, literally just changing crate name


milo
MODERATOR

2 years ago

anyways!


same issue lmfao


milo
MODERATOR

2 years ago

screenshot of error



"certificate verify failed", file


milo
MODERATOR

2 years ago

well then i dont know why


milo
MODERATOR

2 years ago

show me the line of code


milo
MODERATOR

2 years ago

that is causing this erorr


every single one that does an api request


brody
EMPLOYEE

2 years ago

maybe?


milo
MODERATOR

2 years ago

show me please


pub async fn discord_logs(
    log_message: &str
) -> Result<(), Box> {
    let client: Client = Client::new();
    let webhook_url: String = env::var("DISCORD_WEBHOOK_URL").unwrap();
    let current_time: String = Utc::now().to_string();

    client.post(&webhook_url)
        .body(json!({
            "content": format!(
                "**xylex-daemon @ {} | **{}",
                current_time, log_message
            )

        }).to_string())
        .header(
            "Content-Type",
             "application/json"
        )
        .send()
        .await?;

    Ok(())
}

all this shouldn't matter as it works fine on any other deploy it's just railway can't build it unless it's 4-8gb ram


milo
MODERATOR

2 years ago

well thats norrmal about high ram amounts


milo
MODERATOR

2 years ago

yeah no clue, you aren't directly telling it to use openssl and the features make it use rustls so im lost


milo
MODERATOR

2 years ago

sorry


ig so but only 10-20k lines r being deployed on railway and idle usage used to be 200mb or so


milo
MODERATOR

2 years ago

sorry


milo
MODERATOR

2 years ago

idk


thanks for the help though guys, i appreciate your time have a good night


Loading...