GCR Docker Registry _catalog endpoint


#1

I’m running spinnaker version: 1.10.11, doing kubernetes v2 manifest based deployments with pipelines that are triggered by docker images residing in GCR.

Recently my spinnaker installation lost its ability to query for docker repositories. I noticed this when I added a new repo and it failed to show up in the dropdown list on the “automated triggers” section during configuration. Confused I did a hal deploy clean then hal deploy apply and now the dropdowns are entirely empty.

However, I can manually execute the pipelines and it does successfully query the tags for the repositories that were previously configured. So something about my docker config is working correctly.

When I hal deploy apply I get the message:

Problems in
default.provider.dockerRegistry.my-docker-registry:
- WARNING Your docker registry has no repositories specified, and
  the registry's catalog is empty. Spinnaker will not be able to deploy any images
  until some are pushed to this registry.
? Manually specify some repositories for this docker registry to
  index.

And my clouddriver logs say:

2019-02-25 14:03:10.446  INFO 1 --- [           main] c.n.s.c.security.ProviderUtils           : Adding accounts [my-docker-registry] of type DockerRegistryNamedAccountCredentials...
2019-02-25 14:03:24.948  INFO 1 --- [ecutionAction-4] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:03:25.409  INFO 1 --- [ecutionAction-4] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 0 tagged images in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:03:25.410  INFO 1 --- [ecutionAction-4] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 0 image ids in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:03:25.488  INFO 1 --- [ecutionAction-4] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.docker.registry.provider.DockerRegistryProvider:my-docker-registry/DockerRegistryImageCachingAgent[1/1] completed in 12.469s
2019-02-25 14:03:30.691  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/dockerRegistry/images/tags],methods=[GET]}" onto public java.util.List<java.lang.String> com.netflix.spinnaker.clouddriver.docker.registry.controllers.DockerRegistryImageLookupController.getTags(java.lang.String,java.lang.String)
2019-02-25 14:03:30.694  INFO 1 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/dockerRegistry/images/find],methods=[GET]}" onto public java.util.List<java.util.Map> com.netflix.spinnaker.clouddriver.docker.registry.controllers.DockerRegistryImageLookupController.find(com.netflix.spinnaker.clouddriver.docker.registry.controllers.DockerRegistryImageLookupController$LookupOptions)
2019-02-25 14:08:14.023  INFO 1 --- [cutionAction-18] .d.r.p.a.DockerRegistryImageCachingAgent : Describing items in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:08:14.024  INFO 1 --- [cutionAction-18] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 0 tagged images in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:08:14.025  INFO 1 --- [cutionAction-18] .d.r.p.a.DockerRegistryImageCachingAgent : Caching 0 image ids in my-docker-registry/DockerRegistryImageCachingAgent[1/1]
2019-02-25 14:08:14.029  INFO 1 --- [cutionAction-18] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.docker.registry.provider.DockerRegistryProvider:my-docker-registry/DockerRegistryImageCachingAgent[1/1] completed in 0.294s

My config file has:
dockerRegistry:
enabled: true
accounts:
- name: my-docker-registry
requiredGroupMembership:
providerVersion: V1
permissions: {}
address: https://us.gcr.io
username: _json_key
email: fake.email@spinnaker.io
cacheIntervalSeconds: 300
clientTimeoutMillis: 60000
cacheThreads: 1
paginateSize: 200
sortTagsByDate: false
trackDigests: false
insecureRegistry: false
repositories:
passwordFile: /Users/nicholas.phillips/.hal/my-spinnaker-keys/spin-account-gcr.json
primaryAccount: my-docker-registry

This config section hasn’t changed, except over the last couple of days I kicked up cacheIntervalSeconds from the default of 30 to 300 because some things I’ve read indicated that I might have exceeded a quota limit.

My gcr service account has “Storage Admin” privileges on the bucket holding the GRC registry, I also added “Storage Object Viewer” (just in case, based on the information in this link https://cloud.google.com/container-registry/docs/access-control#granting_users_and_other_projects_access_to_a_registry)

I have (and have had) the Resource Manager API enabled as per the instructions here:

The only configuration change that I believe that I made was to add Pipeline Permissions (https://www.spinnaker.io/setup/security/authorization/pipeline-permissions/). But when I backed that out the problem persisted.

But, my repositories are there, and this was working last week, I’m baffled.

Does anyone have advice on what steps to take in diagnosing this? If the problem is rate limiting, do you know of a way to check if the quotas are being exceeded?


#2

I have the exact same problem with the same errors and same config, running 10.11.9.
It started on the 22th.


#3

I filed a bug #4125 in the issue tracker corresponding to this behaviour.


#4

Here’s what I hope is a link to a slack thread around this issue (if I pasted it correctly), hopefully that can be of some help:
https://spinnakerteam.slack.com/archives/C091CCWRJ/p1551106771140800

It appears to have been a problem w/ GCR that they managed to fix a few days afterward. It affected a number of people. I can say that in my case it spontaneously started working again and as far as I can tell is still working. I did a triggered deployment as recently as yesterday with a new image.


#5

I’ve set up PubSub on image push to gcr, and Spinnaker is indeed receiving those messages, but it’s still complaining (about this thread’s issue). But, I guess this thread, and your use case, might be more directed towards Spinnaker polling GCR? And in that case; do you think I should just ignore the warning? (because it hasn’t resolved itself for us)