Spinnaker Pods are not in ready state after installation on Kubernates


#1

After starting installing spinnaker on Oracle Kubernates engine(halyard on docker), we are unable to get all spinnaker pods in Ready state as below:
$ kubectl get pod -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-6795648f5c-7695g 0/1 Running 0 10m
spin-deck-6766bfb78-ttqff 1/1 Running 0 10m
spin-echo-76d84bccf4-wqgq4 1/1 Running 0 10m
spin-front50-6fdc874759-jtrfp 1/1 Running 0 6m
spin-gate-6ffdfd7657-xhjh2 0/1 Running 0 10m
spin-igor-6c564fc765-589cz 1/1 Running 0 10m
spin-orca-567879df75-4t2gn 0/1 Running 0 10m
spin-redis-5bd65c56b6-82rxp 1/1 Running 0 10m
spin-rosco-7cdf5858fc-n5vrx 0/1 Running 0 9m

Please find the logs for the pods which are not ready and please help me with any clue to resolve the issue.
LMK in case you need more logs/info.

Logs:

spin-clouddriver logs
#################################################


$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|grep -i error
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:13.842 INFO 1 — [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)


$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|tail -20
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:17.245 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}” onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-09-14 07:35:17.247 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/actuator || /actuator.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public org.springframework.hateoas.ResourceSupport org.springframework.boot.actuate.endpoint.mvc.HalJsonMvcEndpoint.links()
2018-09-14 07:35:17.259 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.276 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.303 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-09-14 07:35:17.304 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.305 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-09-14 07:35:19.677 INFO 1 — [ main] c.n.s.c.security.ProviderUtils : Adding accounts [my-k8s-v2-account] of type KubernetesNamedAccountCredentials…
2018-09-14 07:37:04.151 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:37:04.154 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : There are no namespaces configured (or loadable) – please check that the list of ‘omitNamespaces’ for account ‘my-k8s-v2-account’ doesn’t prevent access from all namespaces in this cluster, or that the cluster is reachable.
2018-09-14 07:38:47.351 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:40:30.099 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:42:12.815 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete

spin-gate logs
##########################################


$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|grep -i error
2018-09-14 07:35:35.599 INFO 1 — [ost-startStop-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:35:42.556 WARN 1 — [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘enableRedisKeyspaceNotificationsInitializer’ defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Error starting ApplicationContext. To display the auto-configuration report re-run your application with ‘debug’ enabled.
2018-09-14 07:35:42.643 ERROR 1 — [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘enableRedisKeyspaceNotificationsInitializer’ defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
2018-09-14 07:36:38.056 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@661f6038 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:39:38.067 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1709e561 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:40:38.069 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@533017fd rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:41:38.070 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@431be4b2 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:42:38.071 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6fc06d32 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]


$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|tail -40
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration$EnableRedisKeyspaceNotificationsInitializer.afterPropertiesSet(RedisHttpSessionConfiguration.java:249) ~[spring-session-1.3.1.RELEASE.jar:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
… 19 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
… 23 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
… 26 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
… 33 common frames omitted

2018-09-14 07:36:38.056 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@661f6038 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

spin-orca logs
##########################################


$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|grep -i error
2018-09-14 07:43:56.069 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.080 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.090 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.101 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.111 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

2018-09-14 07:43:59.065 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.075 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.086 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.097 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.110 ERROR 1 — [ scheduler-3] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: orca.task.queue:buffered:pipeline
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
2018-09-14 07:43:59.110 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.120 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.131 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)


$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|tail -40
2018-09-14 07:45:04.126 ERROR 1 — [ scheduler-6] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at com.netflix.spinnaker.q.redis.RedisQueue.poll(RedisQueue.kt:98)
at com.netflix.spinnaker.q.QueueProcessor.pollOnce(QueueProcessor.kt:76)
at com.netflix.spinnaker.q.QueueProcessor.access$pollOnce(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:62)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor.ifEnabled(QueueProcessor.kt:119)
at com.netflix.spinnaker.q.QueueProcessor.poll(QueueProcessor.kt:57)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)
… 21 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:184)
… 28 common frames omitted

spin-rosco logs
###########################################


$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|grep -i error
2018-09-14 07:36:12.914 INFO 1 — [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:36:24.195 ERROR 1 — [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
2018-09-14 07:36:24.195 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:34.089 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.089 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.099 ERROR 1 — [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:


kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|tail -40 at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGetProperty(GetEffectivePogoPropertySite.java:48) ~[groovy-all-2.4.13.jar:2.4.13] at com.netflix.spinnaker.rosco.executor.BakePoller_onApplicationEvent_closure1.doCall(BakePoller.groovy:81) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.Closure.call(Closure.java:414) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) [groovy-all-2.4.13.jar:2.4.13]
at com.sun.proxy.$Proxy116.call(Unknown Source) [na:na]
at rx.Scheduler$Worker$1.call(Scheduler.java:120) [rxjava-1.0.16.jar:1.0.16]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.0.16.jar:1.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
… 32 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
… 39 common frames omitted


#2

Same issue here. Any help would be appreciated.


#3

Did anyone figure out the cause for this exception. I am seeing the same error too.


#4

Hey guys, I’ve run into the same problem when upgraded from kubernetes v1 provider to v2. I think the error was due to leftover resources that were conflicting(but not sure). When the error appeared I tried to roll back to v1 provider, but got the same result. Redeploying the whole spinnaker namespace solved the issue for me.

Halyard version 1.12.0
Spinnaker version 1.10.5
Kubernetes version(AWS EKS) v1.10.3-eks