Introduce Quay build as artifact in pipeline


#1

I am building docker containers in Quay.io, and have successfully set up the webhook to notify Spinnaker that the build is complete. I’ve set up the quay docker registry in Spinnaker as well, but am somewhere missing the concepts on how to get an artifact out of that to introduce to a pipeline.

It could be just too much time spent staring at this, but anything, even “where can I apply something like the Jinja templates specified at https://www.spinnaker.io/reference/artifacts/in-pipelines/ to an incoming webhook to extract an artifact” (if the question even makes sense) would be helpful.

Apologies, I’ve just spent far too long staring at this.


#2

@rhuffman: What’s the format of the message from Quay.io? If you can control the contents of the webhook, you could directly inject an artifact into the pipeline by including it in the webhook payload. (I am actually currently working on a way to apply arbitrary Jinja templates to the result of any trigger to extract artifacts, so that might actually help your use case once this is released.)

Alternatively, if you’re using the docker registry trigger, it should inject the artifact into the pipeline directly. (This would mean configuring the pipeline to start using the registry trigger, rather than the webhook trigger.)


#3

Thank you for your response!

I tried configuring a docker registry trigger but couldn’t figure out how successfully find the artifact from it. If focusing on that is the best option, I’m game, but Find Image from Tags only seems to work for images already in a GCP/AWS registry. Should I be using Find Artifact from Execution? It doesn’t seem so, since there is no prior execution to look to. https://www.spinnaker.io/reference/artifacts/in-pipelines/#find-artifact-from-execution

As for the webhook, here’s the format: https://docs.quay.io/guides/notifications.html

Here’s how the docker registry is configured (configured via hal, this is clouddriver.yaml)

dockerRegistry:
  enabled: true
  accounts:
  - name: quay
    requiredGroupMembership: []
    providerVersion: V1
    permissions: {}
    address: https://quay.io
    username: [redacted]
    password: [redacted]
    email: fake.email@spinnaker.io
    cacheIntervalSeconds: 30
    clientTimeoutMillis: 60000
    cacheThreads: 1
    paginateSize: 100
    sortTagsByDate: false
    trackDigests: false
    insecureRegistry: false
    repositories: []
  primaryAccount: quay

Would love


#4

And here’s the .hal/config

dockerRegistry:
  enabled: true
  accounts:
  - name: quay
    requiredGroupMembership: []
    providerVersion: V1
    permissions: {}
    address: https://quay.io
    username: [redacted]
    password: [redacted]
    email: fake.email@spinnaker.io
    cacheIntervalSeconds: 30
    clientTimeoutMillis: 60000
    cacheThreads: 1
    paginateSize: 100
    sortTagsByDate: false
    trackDigests: false
    insecureRegistry: false
    repositories: []
  primaryAccount: quay

When setting up the registry trigger, the organization and repos are visible, I just don’t know how to get that to the next step in the pipeline. I think it’s from too long being in the forest to find which tree I’m looking for, so to speak.


#5

Have you configured the ‘Expected Artifacts’ section on the ‘Triggers’ page? If you configure an expected artifact in this form:

then the expected artifact should be bound to the actual image that triggered the pipeline at run-time, and you’ll be able to reference that image in downstream stages (such as by choosing to deploy the artifact).


#6

I have. Entered expected artifacts, then tried to create a deploy stage in the pipeline. I get the error “You must have a bake or find image before the deploy stage.” I’m trying to figure out which find image stage to use and how, since it’s not an automagic thing, apparently (not that it should be, I’m just missing a critical step or three here).


#7

What version of Spinnaker are you using, and are you using the V1 (deploy-stage based) or V2 (manifest-based) version of the Kubernetes provider? If you’re using the V1 provider, I just added support for deploying artifacts in Spinnaker 1.9, so you’d need to upgrade to 1.9 for this to work. If you’re on 1.9 already, I’d like to know more as this should work—you shouldn’t need to have a find image stage before. It could be there is just some validation I didn’t update.


#8

I think I’m starting to see why I could be having a problem. Could it be that I need to specify the docker registry in this array?

Also, if I’m running different Kubernetes clusters (in GKE and EKS, for example) I would just create different kube admin accounts?

kubernetes:
  enabled: true
  accounts:
  - name: aws-token-user
    requiredGroupMembership: []
    providerVersion: V2
    permissions: {}
    dockerRegistries: []
    context: aws
    configureImagePullSecrets: true
    cacheThreads: 1
    namespaces: []
    omitNamespaces: []
    kinds: []
    omitKinds: []
    customResources: []
    cachingPolicies: []
    kubeconfigFile: /home/spinuser/.kube/config
    oauthScopes: []
    oAuthScopes: []
  primaryAccount: aws-token-user

#9

I should also say that I’d be very interested in what you’re working on, since it would enable us to pass docker images to ECS clusters via a webhook, instead of only EKS clusters.


#10

Yes, I believe you’d need to link the docker account to the kubernetes account. You can do this via Halyard with the hal config provider kubernetes account edit --docker-registries command (or just manually edit the config).

And, yes if you’re managing multiple clusters, you can just add separate accounts and they’ll work independently with their respective clusters.


#11

Much appreciated. Still trying to nail down a few things, but you’ve been very helpful. Thank you!