Clouddriver throw error after running for a while


#1

I setup spinnaker to run on EKS cluster. After running around 3 days. I have some components down because clouddriver threw exception and stopped.
2019-02-08 23:50:19.065 ERROR 1 — [gentScheduler-1] c.n.s.c.r.c.ClusteredAgentScheduler : Unable to run agents

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
        at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:na]
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
        at com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:61) ~[kork-jedis-3.4.0.jar:3.4.0]
        at com.netflix.spinnaker.kork.jedis.telemetry.InstrumentedJedisPool.getResource(InstrumentedJedisPool.java:27) ~[kork-jedis-3.4.0.jar:3.4.0]
        at com.netflix.spinnaker.kork.jedis.JedisClientDelegate.withCommandsClient(JedisClientDelegate.java:46) ~[kork-jedis-3.4.0.jar:3.4.0]
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquireRunKey(ClusteredAgentScheduler.java:166) ~[cats-redis.jar:na]
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquire(ClusteredAgentScheduler.java:124) ~[cats-redis.jar:na]
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.runAgents(ClusteredAgentScheduler.java:148) ~[cats-redis.jar:na]
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.run(ClusteredAgentScheduler.java:141) ~[cats-redis.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_191]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_191]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]
Caused by: java.lang.IllegalStateException: Pool not open
        at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:672) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:412) ~[commons-pool2-2.4.2.jar:2.4.2]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
        at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
        ... 15 common frames omitted