Docker provider insecure-registry flag not honored end-to-end


#1

[Spinnaker version 1.7.4]

I am trying to add a docker registry to my spinnaker installation using the ‘–insecure-registry true’ flag.

Halyard was able to verify the registry when adding it via the ‘hal config provider docker-registry account add’ however the clouddriver pod is failing to get into a ready state.

My best effort review of the stack trace below seems to point to a potential bug in the version of the okhttp library that clouddriver is using.

If you look at:


and

you’ll notice that client.getSslSocketFactory is never passed to the http connection that is ultimately made to the docker registry endpoint.

It looks like clouddriver should reevaluate its used of retrofit/okhttp and find an alternative or upgrade to a version that does not have this bug.

Here is the clouddriver stack trace:
retrofit.RetrofitError: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at retrofit.RetrofitError.networkError(RetrofitError.java:27) ~[retrofit-1.9.0.jar:na]
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:395) ~[retrofit-1.9.0.jar:na]
at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) ~[retrofit-1.9.0.jar:na]
at com.sun.proxy.$Proxy116.checkVersion(Unknown Source) ~[na:na]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$DockerRegistryService$checkVersion$0.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$DockerRegistryService$checkVersion$0.call(Unknown Source) ~[na:na]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$_doCheckV2Availability_closure13.doCall(DockerRegistryClient.groovy:379) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$_doCheckV2Availability_closure13.doCall(DockerRegistryClient.groovy) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) ~[groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) ~[groovy-all-2.4.13.jar:2.4.13]
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient.request(DockerRegistryClient.groovy:402) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$request$0.callCurrent(Unknown Source) ~[na:na]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient.doCheckV2Availability(DockerRegistryClient.groovy:378) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:52) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:154) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient.checkV2Availability(DockerRegistryClient.groovy:367) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at com.netflix.spinnaker.clouddriver.docker.registry.api.v2.client.DockerRegistryClient$checkV2Availability$1.call(Unknown Source) ~[na:na]
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) ~[groovy-all-2.4.13.jar:2.4.13]
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117) ~[groovy-all-2.4.13.jar:2.4.13]
at com.netflix.spinnaker.clouddriver.docker.registry.health.DockerRegistryHealthIndicator.checkHealth(DockerRegistryHealthIndicator.groovy:67) ~[clouddriver-docker-2.22.0-SNAPSHOT.jar:2.22.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-4.3.11.RELEASE.jar:4.3.11.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.3.11.RELEASE.jar:4.3.11.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_151]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_151]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.8.0_151]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[na:1.8.0_151]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[na:1.8.0_151]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[na:1.8.0_151]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[na:1.8.0_151]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[na:1.8.0_151]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[na:1.8.0_151]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[na:1.8.0_151]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[na:1.8.0_151]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[na:1.8.0_151]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[na:1.8.0_151]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[na:1.8.0_151]
at com.squareup.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:192) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:149) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.Call.getResponse(Call.java:286) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205) ~[okhttp-2.7.5.jar:na]
at com.squareup.okhttp.Call.execute(Call.java:80) ~[okhttp-2.7.5.jar:na]
at retrofit.client.OkClient.execute(OkClient.java:53) ~[retrofit-1.9.0.jar:na]
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:326) ~[retrofit-1.9.0.jar:na]
… 48 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[na:1.8.0_151]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[na:1.8.0_151]
at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.8.0_151]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[na:1.8.0_151]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[na:1.8.0_151]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[na:1.8.0_151]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[na:1.8.0_151]
… 69 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[na:1.8.0_151]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[na:1.8.0_151]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[na:1.8.0_151]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[na:1.8.0_151]
… 75 common frames omitted


#2

Now I am officially lost, it looks like the sslSocketFactory IS being passed to the connection stream via:

I’ve verified that my docker registry url starts with https:// which should trigger the usage of the sslSockerFactory


#3

Another update. I wrapped the clouddriver docker image in my own image definition that installed the self signed cert for my docker registry. as a result, cloud driver was able to start up and index the docker repositories I had configured for that registry.

This seems to verify that the clouddriver code execution path is not properly allowing for insecure registries.