An integration-test job passes locally but flakes in CI on about 1 run in 5 with no code change. Find and fix it.
This job boots the API against a database service, then runs the integration suite. It is green locally and on most CI runs, but roughly one run in five fails with connection-refused errors from the tests — same commit, no code change. Find the root cause of the flakiness in the config and fix it.
integration-test:
stage: test
services:
- name: postgres:16
alias: db
script:
- ./start-app.sh & # boots the API, connects to db
- sleep 3 # give it time to come up
- pytest tests/integration/Explain why it is flaky and give the fix.
The sleep 3 is a race — under CI load the app or DB isn't always up in three seconds, so some runs test early and fail with connection-refused. Replace it with a readiness poll of the health endpoint with a timeout before pytest. Wait on the condition, not the clock.
- ✗Fixing a race by lengthening the sleep instead of waiting on readiness
- ✗Masking flakiness with a blanket job retry rather than the root cause
- ✗Blaming CI hardware instead of the timing assumption in the config
- →How would a readiness poll with a timeout look in the script?
- →When is a bounded job retry acceptable, and when does it hide real bugs?
Solution
Cause
sleep 3 is an implicit race. It assumes the app and database come up in exactly three seconds. Locally that is often true, but under CI-runner load startup sometimes takes longer — then pytest starts before the service is ready and hits connection-refused. That is the nondeterministic "1 in 5".
Fix — wait on the condition, not the clock
integration-test:
stage: test
services:
- name: postgres:16
alias: db
script:
- ./start-app.sh &
- |
for i in $(seq 1 30); do
curl -fsS http://localhost:8080/health && break
sleep 1
done
- curl -fsS http://localhost:8080/health # final check, else the job fails
- pytest tests/integration/Polling the health endpoint in a bounded retry loop removes the dependence on wall-clock time: the tests start exactly when the service is actually ready. Separately, isolate any shared mutable state and remove test-ordering dependencies. A blanket retry hides the root cause instead of removing it.
