Hangs on startup if Postgres doesn't start quickly enough #887
Description
Expected Behavior
Ganymede should handle the slow database startup and become available.
Current Behavior
Ganymede hangs on startup and stays that way.
Logs show Postgres wasn't ready:
2025-09-20 19:47:06,736 INFO spawned: 'api' with pid 96
2025-09-20 19:47:06,737 INFO spawned: 'worker' with pid 97
2025-09-20T19:47:06+02:00 INF starting worker | commit=6aca321 tag=v4.7.0 build_time=2025-09-13_13:37:15
2025-09-20T19:47:06+02:00 fatal error running auto migration | error=querying server version pq: the database system is starting up
2025-09-20T19:47:06+02:00 INF starting server | commit=6aca321 tag=v4.7.0 build_time=2025-09-13_13:35:41
2025-09-20 19:47:06,752 WARN exited: worker (exit status 1; not expected)
2025-09-20T19:47:06+02:00 fatal error running auto migration | error=querying server version pq: the database system is starting up
2025-09-20 19:47:06,827 WARN exited: api (exit status 1; not expected)
2025-09-20 19:47:09,840 INFO spawned: 'api' with pid 122
2025-09-20 19:47:09,844 INFO spawned: 'worker' with pid 123
2025-09-20T19:47:09+02:00 INF starting server | commit=6aca321 tag=v4.7.0 build_time=2025-09-13_13:35:41
2025-09-20T19:47:09+02:00 INF starting worker | commit=6aca321 tag=v4.7.0 build_time=2025-09-13_13:37:15
2025-09-20T19:47:09+02:00 fatal error running auto migration | error=querying server version pq: the database system is starting up
2025-09-20 19:47:09,931 WARN exited: worker (exit status 1; not expected)
2025-09-20 19:47:09,932 INFO gave up: worker entered FATAL state, too many start retries too quickly
2025-09-20T19:47:09+02:00 fatal error running auto migration | error=querying server version pq: the database system is starting up
2025-09-20 19:47:09,945 WARN exited: api (exit status 1; not expected)
2025-09-20 19:47:10,946 INFO gave up: api entered FATAL state, too many start retries too quickly
The health check shows the container as unhealthy, but docker does nothing with that information.
Possible Solution
Potentially a delay on start up to wait for Postgres to come up?
Alternatively/better: If Postgres can't be reached on startup for X tries, especially if it replies with "not ready yet", retry later, or potentially exit out and thus have the container fail/stop, letting Docker handle the restart per its restart policy.
(I used to use autoheal to resurrect ganymede, but autoheal itself is very flawed as it has no "maximum retry" limit and will continuously restart containers if they don't come up quickly enough, which results in extreme load on the host until the latter finally becomes unresponsive.)
Steps to Reproduce
- Have a slow machine like mine (spinning rust, ...).
- Boot machine with ganymede's services in auto start. (Both Postgres and Ganymede will start, but Postgres won't be ready by the time Ganymede wants to access it.)
...
Version
No response
Server logs