Stage dev-app-dac-deploy timed out after 5 minutes


#1

Periodically we get timeouts (currently set to 5 mins) for k8s manifest deployment stages.

We run the pipeline again after the error and it works.

Exception ( Force Cache Refresh )
Stage dev-app-dac-deploy timed out after 5 minutes

sszabo@tor976568e1 [/home/sszabo/tmp] $ cat b | jq -r .stages[].context.exception.details.stackTrace
com.netflix.spinnaker.orca.exceptions.TimeoutException: Stage dev-app-dac-deploy timed out after 5 minutes
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$checkForStageTimeout$1.accept(RunTaskHandler.kt:221)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$checkForStageTimeout$1.accept(RunTaskHandler.kt:54)
at java.util.Optional.ifPresent(Optional.java:159)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.checkForStageTimeout(RunTaskHandler.kt:213)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.checkForTimeout(RunTaskHandler.kt:182)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.access$checkForTimeout(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:80)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:163)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:48)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:32)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:58)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:32)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:67)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:41)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:156)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:66)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:54)
at com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)
at com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)
at com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:54)
at com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)
at com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


#2

Force Cache Refresh: 03:05

Why so much time on cache refresh?


#3

This might be resolved here: https://github.com/spinnaker/spinnaker/issues/3366