Spinnaker Gate spawn a lot of [ssl_client] <defunct>


#1

Hi,

I run Spinnaker on EKS, after around 2 or 3 days. My spinnaker gate (container) created a lot ssl_client like below.

100      19691  0.2  4.7 3654824 370032 ?      Ssl  Feb15  15:07      |   \_ /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Djava.security.egd=file:/dev/./urandom -Dspring.config.location=/opt/spinnaker/config/ ... com.netflix.spinnaker.gate.Main
100      20348  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20395  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20532  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20621  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20689  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20746  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20822  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      20877  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21006  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21317  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21495  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21602  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21683  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21748  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21869  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      21980  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22076  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22204  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22272  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22319  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22389  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22463  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22525  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22617  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22671  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22719  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>
100      22791  0.0  0.0      0     0 ?        Z    Feb15   0:00      |       \_ [ssl_client] <defunct>

I can see a weird status from kubectl command. It showed Gate running but READY 0/1

spin-clouddriver-8645cf8bf7-pdc55   1/1     Running   0          3d
spin-deck-5c5dcd866d-7gc79          1/1     Running   0          3d
spin-echo-c77f994f5-6gthx           1/1     Running   0          3d
spin-fiat-58cf7b9669-26g6f          0/1     Running   0          3d
spin-front50-b6599c7c-hb2rz         1/1     Running   0          3d
spin-gate-fdf5d87c-crlrp            0/1     Running   0          3d
spin-igor-65d848f9db-qkjch          1/1     Running   0          3d
spin-kayenta-8446fd6c59-ktq7t       1/1     Running   0          3d
spin-orca-5dbb7b6f98-hq4kx          1/1     Running   0          3d
spin-rosco-795b9776dc-pj722         1/1     Running   0          3d

#2

I wonder it’s bug or not.
I deleted the old spin-gate-fdf5d87c-crlrp, deployment spin-gate will spin up a new spin-gate-xxx.
I login to worker running a new spin-gate-xxx. Found out it keep creating [ssl_client] and left it into zombie?

root     24944  0.0  0.0 146104  2268 ?        Sl   18:30   0:00      \_ docker-containerd-shim a6ef751fc7c8386ea813e7386e436e07d504d4e7c031ad662f25bd539ef0a656 /var/run/docker/libcontainerd/a6ef751fc7c83
100      24963 28.6  3.5 3544160 280704 ?      Ssl  18:30   0:39          \_ /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Djava.security.egd=file:/dev/./urandom -Dspring.config.location=/opt/spinnaker/conf
100      25266  0.0  0.0      0     0 ?        Z    18:31   0:00              \_ [ssl_client] <defunct>
100      25328  0.0  0.0      0     0 ?        Z    18:31   0:00              \_ [ssl_client] <defunct>
100      25363  0.0  0.0      0     0 ?        Z    18:31   0:00              \_ [ssl_client] <defunct>
100      25402  0.0  0.0      0     0 ?        Z    18:31   0:00              \_ [ssl_client] <defunct>
100      25469  0.0  0.0      0     0 ?        Z    18:31   0:00              \_ [ssl_client] <defunct>
100      25505  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25539  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25595  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25634  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25666  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25732  0.0  0.0      0     0 ?        Z    18:32   0:00              \_ [ssl_client] <defunct>
100      25767  0.0  0.0      0     0 ?        Z    18:33   0:00              \_ [ssl_client] <defunct>

And there was no exception and error in Gate logs.


#3

It’s related to the readinessProbe of gate, if I replace with a readinessProbe using httpGet the zombies disappear.


#4

thanks @dmlr i also got the same answer from github