Spinnaker Pods are not in ready state after installation on Kubernates


#1

After starting installing spinnaker on Oracle Kubernates engine(halyard on docker), we are unable to get all spinnaker pods in Ready state as below:
$ kubectl get pod -n spinnaker
NAME READY STATUS RESTARTS AGE
spin-clouddriver-6795648f5c-7695g 0/1 Running 0 10m
spin-deck-6766bfb78-ttqff 1/1 Running 0 10m
spin-echo-76d84bccf4-wqgq4 1/1 Running 0 10m
spin-front50-6fdc874759-jtrfp 1/1 Running 0 6m
spin-gate-6ffdfd7657-xhjh2 0/1 Running 0 10m
spin-igor-6c564fc765-589cz 1/1 Running 0 10m
spin-orca-567879df75-4t2gn 0/1 Running 0 10m
spin-redis-5bd65c56b6-82rxp 1/1 Running 0 10m
spin-rosco-7cdf5858fc-n5vrx 0/1 Running 0 9m

Please find the logs for the pods which are not ready and please help me with any clue to resolve the issue.
LMK in case you need more logs/info.

Logs:

spin-clouddriver logs
#################################################


$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|grep -i error
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:13.842 INFO 1 — [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)


$ kubectl log -n spinnaker spin-clouddriver-6795648f5c-7695g|tail -20
log is DEPRECATED and will be removed in a future version. Use logs instead.
2018-09-14 07:35:17.245 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}” onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-09-14 07:35:17.247 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/actuator || /actuator.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public org.springframework.hateoas.ResourceSupport org.springframework.boot.actuate.endpoint.mvc.HalJsonMvcEndpoint.links()
2018-09-14 07:35:17.259 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.276 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.277 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.303 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-09-14 07:35:17.304 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-09-14 07:35:17.305 INFO 1 — [ main] o.s.b.a.e.mvc.EndpointHandlerMapping : Mapped “{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}” onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-09-14 07:35:19.677 INFO 1 — [ main] c.n.s.c.security.ProviderUtils : Adding accounts [my-k8s-v2-account] of type KubernetesNamedAccountCredentials…
2018-09-14 07:37:04.151 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:37:04.154 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : There are no namespaces configured (or loadable) – please check that the list of ‘omitNamespaces’ for account ‘my-k8s-v2-account’ doesn’t prevent access from all namespaces in this cluster, or that the cluster is reachable.
2018-09-14 07:38:47.351 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:40:30.099 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete
2018-09-14 07:42:12.815 WARN 1 — [ main] c.n.s.c.k.v.s.KubernetesV2Credentials : Could not list namespaces for account my-k8s-v2-account: Job took too long to complete

spin-gate logs
##########################################


$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|grep -i error
2018-09-14 07:35:35.599 INFO 1 — [ost-startStop-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:35:42.556 WARN 1 — [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘enableRedisKeyspaceNotificationsInitializer’ defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
Error starting ApplicationContext. To display the auto-configuration report re-run your application with ‘debug’ enabled.
2018-09-14 07:35:42.643 ERROR 1 — [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘enableRedisKeyspaceNotificationsInitializer’ defined in class path resource [com/netflix/spinnaker/gate/config/GateConfig.class]: Invocation of init method failed; nested exception is org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
2018-09-14 07:36:38.056 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@661f6038 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:39:38.067 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1709e561 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:40:38.069 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@533017fd rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:41:38.070 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@431be4b2 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:42:38.071 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6fc06d32 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]


$ kubectl -n spinnaker logs spin-gate-6ffdfd7657-xhjh2|tail -40
Caused by: org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:204) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:348) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration$EnableRedisKeyspaceNotificationsInitializer.afterPropertiesSet(RedisHttpSessionConfiguration.java:249) ~[spring-session-1.3.1.RELEASE.jar:na]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) ~[spring-beans-4.3.14.RELEASE.jar:4.3.14.RELEASE]
… 19 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16) ~[jedis-2.9.0.jar:na]
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:194) ~[spring-data-redis-1.8.10.RELEASE.jar:na]
… 23 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432) ~[commons-pool2-2.4.3.jar:2.4.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~[commons-pool2-2.4.3.jar:2.4.3]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
… 26 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_171]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
… 33 common frames omitted

2018-09-14 07:36:38.056 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@661f6038 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:37:38.065 ERROR 1 — [RxIoScheduler-3] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@1df18e11 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2018-09-14 07:38:38.066 ERROR 1 — [RxIoScheduler-2] c.n.s.gate.services.ApplicationService : Unable to refresh application list, reason: Task java.util.concurrent.FutureTask@6e311e73 rejected from java.util.concurrent.ThreadPoolExecutor@18483b8b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

spin-orca logs
##########################################


$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|grep -i error
2018-09-14 07:43:56.069 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.080 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.090 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.101 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:56.111 ERROR 1 — [ scheduler-9] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

2018-09-14 07:43:59.065 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.075 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.086 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.097 ERROR 1 — [ scheduler-7] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.110 ERROR 1 — [ scheduler-3] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: orca.task.queue:buffered:pipeline
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
2018-09-14 07:43:59.110 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.120 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
2018-09-14 07:43:59.131 ERROR 1 — [ scheduler-4] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)


$ kubectl -n spinnaker logs spin-orca-567879df75-4t2gn|tail -40
2018-09-14 07:45:04.126 ERROR 1 — [ scheduler-6] o.s.s.s.TaskUtils$LoggingErrorHandler : [] Unexpected error occurred in scheduled task.
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:16)
at com.netflix.spinnaker.q.redis.RedisQueue.poll(RedisQueue.kt:98)
at com.netflix.spinnaker.q.QueueProcessor.pollOnce(QueueProcessor.kt:76)
at com.netflix.spinnaker.q.QueueProcessor.access$pollOnce(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:62)
at com.netflix.spinnaker.q.QueueProcessor$poll$1.invoke(QueueProcessor.kt:37)
at com.netflix.spinnaker.q.QueueProcessor.ifEnabled(QueueProcessor.kt:119)
at com.netflix.spinnaker.q.QueueProcessor.poll(QueueProcessor.kt:57)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:432)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
at redis.clients.util.Pool.getResource(Pool.java:49)
… 21 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:184)
… 28 common frames omitted

spin-rosco logs
###########################################


$ kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|grep -i error
2018-09-14 07:36:12.914 INFO 1 — [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped “{[/error]}” onto public java.util.Map com.netflix.spinnaker.kork.web.controllers.GenericErrorController.error(java.lang.Boolean,javax.servlet.http.HttpServletRequest)
2018-09-14 07:36:24.195 ERROR 1 — [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:
2018-09-14 07:36:24.195 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:34.089 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.089 ERROR 1 — [readScheduler-1] c.n.spinnaker.rosco.executor.BakePoller : Update Polling Error:
2018-09-14 07:36:49.099 ERROR 1 — [readScheduler-2] c.n.spinnaker.rosco.executor.BakePoller : Zombie Killer Polling Error:


kubectl -n spinnaker logs spin-rosco-7cdf5858fc-n5vrx|tail -40 at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGetProperty(GetEffectivePogoPropertySite.java:48) ~[groovy-all-2.4.13.jar:2.4.13] at com.netflix.spinnaker.rosco.executor.BakePoller_onApplicationEvent_closure1.doCall(BakePoller.groovy:81) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at com.netflix.spinnaker.rosco.executor.BakePoller$_onApplicationEvent_closure1.doCall(BakePoller.groovy) [rosco-core-0.108.0-SNAPSHOT.jar:0.108.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_171]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) [groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.Closure.call(Closure.java:414) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54) [groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124) [groovy-all-2.4.13.jar:2.4.13]
at com.sun.proxy.$Proxy116.call(Unknown Source) [na:na]
at rx.Scheduler$Worker$1.call(Scheduler.java:120) [rxjava-1.0.16.jar:1.0.16]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.0.16.jar:1.0.16]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_171]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.UnknownHostException: spin-redis.spinnaker
at redis.clients.jedis.Connection.connect(Connection.java:207) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767) ~[jedis-2.9.0.jar:na]
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106) ~[jedis-2.9.0.jar:na]
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435) ~[commons-pool2-2.4.2.jar:2.4.2]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.4.2.jar:2.4.2]
at redis.clients.util.Pool.getResource(Pool.java:49) ~[jedis-2.9.0.jar:na]
… 32 common frames omitted
Caused by: java.net.UnknownHostException: spin-redis.spinnaker
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_171]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_171]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_171]
at redis.clients.jedis.Connection.connect(Connection.java:184) ~[jedis-2.9.0.jar:na]
… 39 common frames omitted


#2

Same issue here. Any help would be appreciated.


#3

Did anyone figure out the cause for this exception. I am seeing the same error too.