Spinnaker server groups are not shrank when deployed via Halayrd [solved]


#1

Recently I started observing strange behavior for Spinnaker deployments via Halyard.

Earlier when we redeployed Spinnaker via hal deploy apply new versions of Spinnaker pods were started (like orca-v001, fiat-v001) and after that the old ones were stopped (like orca-v000, fiat-v000).

Now the situation is different: after the new pods are started only orca is stopped but the rest of the services keep running:

/tmp → kubectl get pods
NAME                                    READY     STATUS    RESTARTS   AGE
nonhalyard-redis-7f655ddd47-mdkmd       1/1       Running   0          1h
spin-clouddriver-bootstrap-v000-dgv6f   1/1       Running   0          34m
spin-clouddriver-v000-9452n             2/2       Running   0          48m
spin-clouddriver-v001-xhhcl             2/2       Running   0          31m
spin-deck-v000-pk8cc                    1/1       Running   0          49m
spin-deck-v001-5js56                    1/1       Running   0          31m
spin-echo-v000-87rwp                    2/2       Running   0          48m
spin-echo-v001-xnhrp                    2/2       Running   0          31m
spin-fiat-v000-z5zj2                    2/2       Running   0          48m
spin-fiat-v001-xw7h8                    2/2       Running   0          31m
spin-front50-v000-6p6lh                 2/2       Running   0          48m
spin-front50-v001-jvm8n                 2/2       Running   0          31m
spin-gate-v000-mmfd4                    2/2       Running   0          48m
spin-gate-v001-lpqjz                    2/2       Running   0          31m
spin-igor-v000-rgtzs                    2/2       Running   0          48m
spin-igor-v001-tzj6d                    2/2       Running   0          31m
spin-orca-bootstrap-v000-8tpcw          1/1       Running   0          32m
spin-orca-v001-gvbfz                    2/2       Running   0          31m

To overcome this issue I set --deploy.maxRemainingServerGroups=1 when starting halyard and indeed in spin-orca-bootstrap logs I see "maxRemainingAsgs" : "1" but the server groups are still not shrank and removed.

Can you suggest where to look? Halyard and Spinnaker version were not changed. Halyard version: 0.40.0. Spinnaker version: 1.5.4


#2

The issue was solved by upgrade to Spinnaker 1.6.1