Trigger on push to docker registry


#1

Hi people!
I have a problem with Automated spinnaker triggers with type Docker Registry.
My pipeline is successfully started when I don’t write Tag setting and created new image with new tag and pushed it . But I need pipeline to be triggered when image with special tag is updated.
For example - when the image with Tag ‘test’ is appeared - pipeline should be started. I set up Tag setting - I wrote ‘test’ there - but when image with that tag is updated - my pipeline is not triggered.
There is a message from igor microservice:

[RxIoScheduler-3] c.n.spinnaker.igor.docker.DockerMonitor : Found 0 new images for dockerhub

Is it possible to trigger pipeline if the image with appropriate tag is updated? I’ll appreciate if you help.
Best,
Serhii


#2

@SerhiiK: Where did you add the “test” tag? If you include tag “test” in the configuration of your Docker registry trigger, then the pipeline should only be triggered when that particular tag is pushed. I just tried this configuration myself and it appears to be working.


#3
Thank you @ezimanyi for your response. 
I've added test tag by 'docker push docker_image_name:test'. Please, try to update that image with 'test' tag again, and trigger will not work.

#4

@SerhiiK: Are you trying to push the same image with the same tag multiple times, or changing the image between pushes? I’m not sure if a pipeline will get triggered again if you push the same image with the same tag—it might, but I’m not 100% sure.


#5

@ezimanyi, of course, the image with tag ‘test’ was changed and new version with the same tag '‘test’ was uploaded to docker registry.
It seems that spinnaker searches only for new tags and if new tag was pushed - the trigger starts working. Can you check your spinnaker’s behavior in this situation?
Thanks again.


#6

I just tried this with my set-up and things worked. I build an image, pushed it to my gcr.io repository with docker push, then built the image again and pushed it to the same tag. The pipeline I had listening to that tag triggered on the second push (ie, when I was pushing a new image to a tag that already existed).


#7

@ezimanyi
Is your trigger looks like this?


#8

Yes, my trigger looks exactly like that, the only difference being that I’m using gcr.io instead of dockerhub, but I don’t think that should make a difference.

One thing to test—try removing the entry from “Artifact Constraints”. You can leave the artifact itself in the “Expected Artifacts” section, making sure to uncheck “Use Prior Execution” and “Use Default Artifact”. Then try again; if the pipeline starts but fails due to an unmatched artifact then for some reason the produced artifact is not matching what you have entered, and the error message should provide some more details.


#9

@SerhiiK: I think I may have figured out your issue. In order for Spinnaker to keep track of updates to existing tags, you’ll need to configure your docker registry with the --track-digests flag documented here. When that flag is not passed, Spinnaker will only notice new tags as you observed; if you set that flag to true then Spinnaker will track the digest associated with each tag and will then know when you push a new image to the same tag. I had that set to true in my config, which is why this was working for me.


#10

Unfortunately it doesn’t help.
I’ve changed to:

--track-digests=true

Then I did (tagged ubuntu:xenial as scalecube/scalecube-organization:test):

docker tag ubuntu:xenial scalecube/scalecube-organization:test
docker push scalecube/scalecube-organization:test

The push refers to repository [docker.io/scalecube/scalecube-organization]
49907af65b0a: Layer already exists 
4589f96366e6: Layer already exists 
b97229212d30: Layer already exists 
cd181336f142: Layer already exists 
0f5ff0cf6a1c: Layer already exists 
test: digest: sha256:550f6e26da4b4cb8655a96dd6458ea01e2fb3dcb99d24e6dc427c08ea42c9785 size: 1357

this is Igor logs after pushing:

2018-12-21 12:27:38.501  INFO 1 --- [RxIoScheduler-3] c.n.spinnaker.igor.docker.DockerMonitor  : Updated tagged image: account=dockerhub: image=igor:dockerRegistry:v2:dockerhub:scalecube/scalecube-organization:test. Digest changed from [null] -> [sha256:467118b257722593fdef48091c2e75b01d99288c6f70ae6f75ad2e101c325540].

And still

Found 0 new images for dockerhub

But really SHA is different and in Igor logs it didn’t change. So I guess my spinnaker doesn’t calculate properly SHA.

I’ll try to update spinnaker to the newest version and get back.
Now I have:

spinnaker@hal-d7649f464-wsvzx:/workdir$ hal version list
+ Get current deployment
  Success
+ Get Spinnaker version
  Success
+ Get released versions
  Success
+ You are on version "1.10.5", and the following are available:
 - 1.9.5 (Bright):
   Changelog: https://gist.github.com/spinnaker-release/d24a2c737db49dda644169cf5fe6d56e
   Published: Mon Oct 01 17:15:37 UTC 2018
   (Requires Halyard >= 1.0.0)
 - 1.10.6 (Maniac):
   Changelog: https://gist.github.com/spinnaker-release/8c6e6abe2a0016b823b900523e82cba1
   Published: Wed Dec 19 13:00:39 UTC 2018
   (Requires Halyard >= 1.11)
 - 1.11.1 (Cobra Kai):
   Changelog: https://gist.github.com/spinnaker-release/5cbb402297feb85f82482a73e9428967
   Published: Wed Dec 19 16:44:21 UTC 2018
   (Requires Halyard >= 1.11)

#11

I think the issue may be that Spinnaker doesn’t handle changing --track-digests going from false to true very well. The logic to cache image digests will actually never go back and update the digest for images that existed before the flag was turned on. (This is despite the log message that indicates it is making the update; the next line short-circuits if either the current or former digest is null.)

I’ve made a change to improve this behavior in https://github.com/spinnaker/igor/pull/344. With that PR, Spinnaker will update the null digests in its cache with the current digest value when the flag is turned on.

As a workaround for now, you could carefully delete the key in Redis:
redis-cli del 'igor:dockerRegistry:v2:dockerhub:scalecube/scalecube-organization:test' which will force Spinnaker to re-index it. Be aware, however, that Spinnaker will trigger any pipelines that trigger on that container when it updates the cache after the deletion, as it will no longer realize the image was present before.


#12

Thank you @ezimanyi. It works perfect!