Using PubSub for GCR Trigger [solved]


#1

I’m trying to trigger from PubSub for GCR uploads where I’d love to be able to grab the commit sha (attached as the docker tag) as a variable that I can then provide to my expected artifact and ideally to the artifact substitution process in the v2 kubernetes provider. Are there any examples of this?

I was following the documentation here: https://www.spinnaker.io/guides/user/triggers/pubsub/ but ended up not really understanding the last few steps: https://github.com/spinnaker/spinnaker.github.io/issues/746


#2

I responded in the documentation issue you raised and will copy/paste my response from there to here:

Are you using halyard? For GCR subscriptions specifically you can use the --message-format flag when adding the subscription as follows:

hal config pubsub google subscription add --project my-project --json-path /path/to/service/account.key.json --subscription-name my-subscription --message-format GCR my-gcr-subscription

For messages that aren’t from GCR you can go the route of manually specifying a jinja template using the --template-path flag. Here’s an example usage from the GCS pubsub codelab:

hal config pubsub google subscription add --project my-project --json-path /path/to/service/account.key.json --subscription-name my-subscription --template-path /path/to/jinja/template my-gcs-subscription

After either of these you’ll need to run hal deploy apply for the subscription to become available to stages.

I agree that the documentation needs work around this. We have explicit instructions regarding the jinja templates in some documents but then gloss over it in others. Thanks for the feedback!


#3

Is there something I am missing?

  1. Service Account has the pubsub.subscriber role

  2. Verified Topic and Subscription exist

  3. Confirmed hal config has the correct settings
    hal_config

Is there anything I’m missing? I don’t see the triggers coming through. I’m new to PubSub so I’m probably missing something obvious. Is there a way I can verify that the messages are being sent to the topic?

Also might be nice to show the template that is produced by the GCR message format. I assume that’s essentially what is happening by providing that flag.


#4

Also might be nice to show the template that is produced by the GCR message format. I assume that’s essentially what is happening by providing that flag.

Yes, exactly right. See the template that Echo uses here: https://github.com/spinnaker/echo/blob/master/echo-pubsub-google/src/main/resources/gcr.jinja

Is there anything I’m missing? I don’t see the triggers coming through.

A common source of problems like this is that the user who is triggering the message from GCR (i.e. by pushing an image) must have permissions to publish a message. I believe the required permission is pubsub.topics.publish. Is it possible the user you’re using to initiate the pubsub message doesn’t have the required perms?


#5

This is very possible. Do I have to explicitly fire this message somehow or is this something that GCR does automatically (excuse my ignorance)? If I do, what’s the normal process for this? I would assume the easiest way would be to include this step as part of our container builder steps.


#6

You shouldn’t have to explicitly fire the message. So long as the user performing the action has permissions then Google PubSub should generate that message and publish it automatically.

In regards your earlier question that I missed, “Is there a way I can verify that the messages are being sent to the topic?”, you should be able to see the messages in-flight with the gcloud tool: https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull


#7

I added the roles/pubsub.publisher to our jenkins service account that is kicking off the cloud builder process. Does this change at all when cloud builder is the one publishing the image to GCR and not Jenkins directly.

I tried running:
gcloud pubsub subscriptions pull gcr-subscription

after uploading our dockerfile to cloudbuilder and am getting: Listed 0 items.

Any other suggestions?


#8

I have verified that the messages do seem to be getting published. I think the result above was because spinnaker must be sending an ack before I can pull the message. I was able to see the message come through by turning off the spinnaker subscription temporarily (changing the name). Even still, the pipeline trigger is still not being fired. Is there something else I’m missing?


#9

OK this is good information, thanks. The next step to debugging is to take a look at the logs for Spinnaker’s Echo service.

If you’re running a LocalGit deployment this is as simple as tail -f ~/dev/spinnaker/logs/echo.*
If you’re running a LocalDebian deployment then the logs should be in /var/log/spinnaker
If you’re running a distributed k8s deploy then I think you should be able to see the logs with kubectl log -n spinnaker -f $ECHO_POD_NAME

Edit: submitted too soon. After you start logging, kick off a build / push to GCR. You should see Echo observe the pubsub event and report what it is doing with the information.

Let me know what Echo reports and if possible can you also show the trigger configuration you have set up for your pipeline?


#10

I monitored the logs as the image was uploaded to gcr and saw the following information:

2018-05-11 15:00:16.611  INFO 23059 --- [      Thread-32] c.n.s.e.p.google.GooglePubsubSubscriber  : artifacts Artifact(type=docker/image, name=gcr.io/MYPROJECT/MYIMAGE, version=sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68, location=null, reference=gcr.io/MYPROJECT/MYIMAGE@sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68, metadata=null, artifactAccount=null, provenance=null, uuid=null)
2018-05-11 15:00:16.612  INFO 23059 --- [      Thread-32] c.n.s.echo.pubsub.PubsubMessageHandler   : Processed Pubsub event with payload {"action":"INSERT","digest":"gcr.io/MYPROJECT/MYIMAGE@sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68","tag":"gcr.io/MYPROJECT/MYIMAGE:3aed637"}
2018-05-11 15:00:23.266  INFO 23059 --- [      Thread-34] c.n.s.e.p.google.GooglePubsubSubscriber  : artifacts Artifact(type=docker/image, name=gcr.io/MYPROJECT/MYIMAGE version=sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68, location=null, reference=gcr.io/MYPROJECT/MYIMAGE@sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68, metadata=null, artifactAccount=null, provenance=null, uuid=null)
2018-05-11 15:00:23.268  INFO 23059 --- [      Thread-34] c.n.s.echo.pubsub.PubsubMessageHandler   : Processed Pubsub event with payload {"action":"INSERT","digest":"gcr.io/MYPROJECT/MYIMAGE@sha256:ee58c00cb43575fc4afa40e1ceaf2fb8cfd06aea23e67e4db7db87c7fb3ded68","tag":"gcr.io/MYPROJECT/MYIMAGE:latest"}

My configuration is setup like this:

I’ve tried both with and without the payload constraints. I figured I eventually would want to limit the tag to ‘MYIMAGE’ but wasn’t positive about this part. Also when we send to GCR we tag the commit hash, and then also latest, which is probably why that message is coming through twice (not sure if that is going to cause other issues but ideally we can update the latest tag as well without issue)


#11

@sbws pointed out that I was missing the docker image itself as the “Expected Artifact”. After adding the docker image as an expected artifact my build triggered perfectly!


#12

@sbws So I had mentioned above that we are currently tagging or images with our git hash and the ‘latest’ tag. This does seem to be causing pipelines to trigger twice. Is there a recommended way to deal with this? Would I have to go as far as using a custom message format with my own jinja templates or should I use payload constraints and try some sort of regex that doesn’t match latest or something like that. When deploying the images itself it appears to be using the digest and not the tag so maybe it’d be easier to explicitly match ‘latest’ since that tag isn’t actually used anyways (I think).

Any thoughts on this would be greatly appreciated!


#13

I think your suggestion of the payload constraint is the way to go. My reasoning for this is simply that updating the jinja template will require a hal deploy apply to be performed whereas the payload constraints can be edited in Deck.


#14

@briananderson1222 can you share your JSON pipeline definition? I’m especially interested in the way you were able to reference image tag in your deployment section.

I have a similar issue but for kubernetes v1 provider: Extracting image tag from GCR Pub/Sub trigger (Kubernetes V1 provider)