fix: health check false-negative + add rollback on failure
The deploy pipeline had two critical bugs: 1. Health check used /api/vehicles/ZZZ999 with curl -f. This endpoint returns HTTP 404 for unknown plates (correct behavior), which curl -f treated as a failure. The backend was actually healthy. Fix: use /api/vehicles/ABC123 (seeded in V6 migration, always 200) and remove -f flag from curl. 2. No rollback on failure. If health checks failed, containers stayed running forever because the pipeline exited 1 without stopping them. Fix: combine health checks into one step. If either fails, run 'docker compose down' (without -v, so DB volume is preserved) before exiting with failure.
This commit is contained in:
parent
dfcc8e37c6
commit
e4de2a316a
1 changed files with 26 additions and 16 deletions
|
|
@ -45,36 +45,46 @@ jobs:
|
|||
docker compose -p bilhej-prod -f docker-compose.prod.yml down
|
||||
docker compose -p bilhej-prod -f docker-compose.prod.yml up --build -d
|
||||
|
||||
- name: Wait for services
|
||||
run: sleep 20
|
||||
|
||||
- name: Health check — backend API
|
||||
- name: Health checks with rollback
|
||||
run: |
|
||||
echo "Waiting for services to start..."
|
||||
sleep 20
|
||||
|
||||
BACKEND_OK=false
|
||||
for i in 1 2 3 4 5; do
|
||||
if docker run --rm --network bilhej-prod_default curlimages/curl:8.5.0 \
|
||||
-sf http://bilhej-backend-prod:8080/api/vehicles/ZZZ999; then
|
||||
-s http://bilhej-backend-prod:8080/api/vehicles/ABC123 > /dev/null; then
|
||||
echo "Backend is healthy"
|
||||
exit 0
|
||||
BACKEND_OK=true
|
||||
break
|
||||
fi
|
||||
echo "Attempt $i failed, retrying in 5s..."
|
||||
echo "Backend check attempt $i failed, retrying in 5s..."
|
||||
sleep 5
|
||||
done
|
||||
echo "Backend health check failed"
|
||||
exit 1
|
||||
|
||||
- name: Health check — frontend
|
||||
run: |
|
||||
FRONTEND_OK=false
|
||||
for i in 1 2 3 4 5; do
|
||||
if docker run --rm --network bilhej-prod_default curlimages/curl:8.5.0 \
|
||||
-sf http://bilhej-frontend-prod/ | grep -qi "bilhej\|Bilhej\|BilHej"; then
|
||||
-s http://bilhej-frontend-prod/ | grep -qi "bilhej\|Bilhej\|BilHej"; then
|
||||
echo "Frontend is serving"
|
||||
exit 0
|
||||
FRONTEND_OK=true
|
||||
break
|
||||
fi
|
||||
echo "Attempt $i failed, retrying in 5s..."
|
||||
echo "Frontend check attempt $i failed, retrying in 5s..."
|
||||
sleep 5
|
||||
done
|
||||
echo "Frontend health check failed"
|
||||
exit 1
|
||||
|
||||
if [ "$BACKEND_OK" != "true" ] || [ "$FRONTEND_OK" != "true" ]; then
|
||||
echo ""
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo " HEALTH CHECK FAILED — ROLLING BACK DEPLOYMENT"
|
||||
echo "═══════════════════════════════════════════════════"
|
||||
echo ""
|
||||
docker compose -p bilhej-prod -f docker-compose.prod.yml down
|
||||
echo ""
|
||||
echo "Rolled back. Containers stopped. DB volume preserved."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Print deploy status
|
||||
run: |
|
||||
|
|
|
|||
Loading…
Reference in a new issue