Deploy failures with Clouddriver with Kubernetes on AWS


We’re seeing intermittent deploy fails which we’ve been unable to resolve (pipelines fail after 10 sec due to two clouddriver threads both attempting the exact same atomic k8s operation, one of which fails and blows up the pipeline).

Here are relevant clouddriver logs:

We’re on spinnaker v1.7.5.

Things we’ve tried:

  1. We’ve setup GCR PubSub subscriptions for Spinnaker, and checked the logs for possible repeat messages . However, we only see one generated Artifact in the logs, and only one instance of an Artifact build being triggered from the GCR PubSub log.

  2. We’ve throttled the total number of concurrent Redis connections we make to our caching layer to just one to avoid consistency effects. This did not solve the issue.

At this point, we’re suspecting a Spinnaker bug but can’t find anything in the release notes for Spinnaker v1.8.0 or v1.7.6 to indicate a fix, nor can we find a known issue.

Any advice?