2 months ago
-
Project ID: 07be2947-a1d2-4bc9-863c-1f4a5cd30ec2
Service:
- ClickHouse
- Service ID: 90a5f867-9167-45b2-9792-8e70a02f815e
Volume:
- Volume name: clickhouse-volume
- Volume ID: d14d391d-567a-4fb1-8432-d1e701539c2b
- Volume instance ID: e6612480-a0d7-4a51-8879-be2f5370a9fa
What happened:
- The ClickHouse service was redeployed/restarted.
- After restart, the application began failing with UNKNOWN_TABLE for app tables like via_events.
- ClickHouse itself is running, but it appears to have started against a volume state that contains only system.* tables.
What we verified from inside the running ClickHouse container:
- ClickHouse starts successfully and answers simple queries like SELECT 1.
- Application queries now fail with:
- Code: 60
- UNKNOWN_TABLE
- via_events
- ClickHouse startup log shows:
- Loading metadata from /var/lib/clickhouse/
- Parsed metadata of 22 tables in 1 databases
- later Parsed metadata of 0 tables in 1 databases
- The 22 loaded tables are ClickHouse system.* tables only.
- The live mounted volume filesystem at /var/lib/clickhouse is a real ext4 mount, but currently only shows about 86-88 MB used and contains only system.* table metadata/parts.
- We do not see our app-table metadata, app-table parts, or detached parts on the mounted filesystem.
- SHOW TABLES FROM default returns no user tables.
- The default database exists, but its user tables are gone / not attached.
- Railway UI/API still reports the ClickHouse volume as using about 1.06 GB, which does not match what is visible on the mounted filesystem.
Important note:
- We did not run DROP TABLE on app tables.
- We only cleared ClickHouse internal system.*_log tables earlier, which are documented as safe to truncate.
- The actual data loss symptom appeared after the redeploy/restart, not from an app-table DDL statement.
4 Replies
if possible, maybe revert to any backups you guys have of this volume, no backups on my side.
a month ago
Hello windcrestai,
check the backups tab in the railway dashboard under your clickhouse service , if you see any, restore the most recent one before the issue happened
if the backups tab is empty contact railway staff directly with your volume id and volume instance id and point out that railway ui shows 1.06 gb used but only 86,88 mb is visible on the mounted filesystem only they can investigate at infrastructure level , don't touch or redeploy the volume until they respond
Hope this help you :)
domehane
Hello **windcrestai,** check the backups tab in the railway dashboard under your clickhouse service , if you see any, restore the most recent one before the issue happened if the backups tab is empty contact railway staff directly with your volume id and volume instance id and point out that railway ui shows 1.06 gb used but only 86,88 mb is visible on the mounted filesystem only they can investigate at infrastructure level , don't touch or redeploy the volume until they respond Hope this help you :)
a month ago
This is not something we can investigate on our end. Sorry, we would need to see greater than 1 of 1 reports.