Image and manifest as a bundle?


#1

What is the recommended way of dealing with image AND manifest as a bundle.

Sometimes one has image and a manifest as a bundle (i.e. this image won’t work on previous versions of the manifest, nor this manifest won’t work with older images)

Today I see that one can trigger on specific image OR artifact. That works most of the time. Yet every so often one introduces non compatible changes and really need to treat an image and manifest together. And I can’t see a way to do it.

What am I missing? maybe spinnaker has a way which I am not aware of. Or I approach the problem from the wrong angle?

What I was thinking:

  1. Jenkins creates an image AND a manifest… Pipeline should treat them as a bundle.

OR

  1. Jenkins does above AND creates a manifest with a right image printed in. then all pipeline has to do is to take the manifest and use it with no substitution.

Trouble is in nigher #1 nor #2 manifests don’t have versioning/tags (at least I didn’t find). Thus if jenkins makes a new build, we have a new tagged image, but simply overrite the manifest on gcs. Thus have no way back. in case deploy actually fail.

jenkins can ship a manifest to gcs with say manifest_versionxxx.yml yet I don’t see in spinnaker a way to get to pick versions/tags same way I can do for images from the repo .

similar idea, jenkins can expose manifest as an artifact which can be accessed on per build number.(i.e. same way as properties exposed, yet passed later as a manifest)


How to bind artifact to a manifest?
How to bind artifact to a manifest?
#2

Good question, this is an important use case for us to get right. There are a variety of environments, some with good options.

For example, I store my source and manifest in github, and my images are built using Google Cloud Builder, triggered on certain git events. Google Cloud Builder actually emits a pub sub message for such builds, which includes the commit hash. I parse this hash out in a jinja template to inject the associated k8s manifest at that commit hash as a Spinnaker artifact.

If you’re using Jenkins, you have complete flexibility (and work) to pass image and manifest as a bundle to Spinnaker as artifacts. You can, for example, have Jenkins do the work of determining git commit hash and docker tag produced, and generate Spinnaker artifacts to inject as you trigger the Spinnaker pipeline.

More information on configuring artifacts here, and artifact types here.


#3

Steven,

Thanks for the information. I guess I am a little confused on how to get the files from Jenkins into Spinnaker as artifacts. I see there is a recently merge feature for it here. Is that what you mean or is there some current (1.7.4) way to do this?

Thanks.


#4

Jenkins can kick off the Spinnaker pipeline via webhook, injecting Spinnaker artifacts via the payload. In this case, it’ll inject two - the k8s manifest file and the docker image.

The k8s manifest file would reside on GitHub, and the Artifact would specify the URL that it’s to be retrieved at. If you look at the GitHub file artifact type, you’ll see that you can specify the version/commit-hash that the file should be retrieved at (which is supported by the GitHub API).

The Docker image would be a Docker artifact type.

Assuming Jenkins knows these two pieces of information (docker image/tag and the pertaining git commit hash), it would be treating the two as a bundle.

The injected artifacts can be seen in the pipeline execution JSON:

"trigger": {
  "artifacts": [
    {
      "type": "github/file",
      "name": "mdservice2-rs-prod.yml",
      "reference": "https://api.github.com/repos/skim1420/mdservice2/contents/k8s/mdservice2-rs-prod.yml",
      "version": "2bd101e93e5a070acfa974b553fc9fd552308981"
    },
    {
      "type": "docker/image",
      "name": "gcr.io/spinnaker-1022/mdservice2",
      "reference": "gcr.io/spinnaker-1022/mdservice2:2bd101e93e5a070acfa974b553fc9fd552308981"
    }
}

Let me know if anything’s still unclear.


#5

FYI, I accomplish this not via Jenkins, but by processing the pub/sub message that Google Cloud Builder produces. I use a custom jinja template on the GCB subscription in Echo:

[
  {
    "type": "docker/image",
{% set names = images[0].split(':') %}
    "name": "{{ names[0] }}",
    "reference": "{{ images[0] }}"
  },
  {
    "type": "github/file",
    "name": "mdservice2-rs-prod.yml",
    "reference": "https://api.github.com/repos/skim1420/mdservice2/contents/k8s/mdservice2-rs-prod.yml",
    "version": "{{ sourceProvenance.resolvedRepoSource.commitSha }}"
  }
]

But as I mention previously, you can accomplish this a number of ways, to the same effect. Another strategy, for example, is the practice of tagging your images with the commit hash (which I also happen to do) and have a custom jinja template that fills in the version of the github/file artifact from that. GCR also emits pub/sub messages for its docker push events.


#6

Thanks for the ideas. I should try those.

Here is how I finally solved it using Jenkins. Key here is that Jenkins can act as both trigger, source of the information (via property file AND via a Jenkins artifact(s) accessible via http)

  1. Jenkins job builds a new image and puts it in a registry. (this we had before)
  2. Same job takes a template manifest from the same git repo and writes the newly created image tag into the manifest AND renames the deployment to canary (so manifest becomes perfectly deployable using kubectl apply -f manifest.yml ).
  3. Same job publishes two Jenkins artifacts (don’t mix with spinnaker artifacts).
    canary_artifact%20list
    a. the canary manifest
    b. .property file with a link to the above manifest: below is an example of such file
  4. Spinnaker gets triggered of such a job and reads above property file
    trigger
  5. (here is the trick really) Expected Artifact in the configuration is dynamic. A function of the content of the .properties file, in this example value of the ‘canary-manifest’ key
  6. it is used as a manifest for canary NOTE: it is accessed using http interface. i.e. we really use jenkins as a webserver here

    At this point I can manually trigger the pipeline on any of the past Jenkins builds.
  7. Reference build deployed similarly (I describe later how reference.yml obtained)
  8. We do canary analysis and if happy promote to production
  9. To get production manifest we trigger yet another jenkins job. The trick here is that the jenkins job takes a parameter with a link to a manifest we would like to use in production (or more precisely the canary manifest we would like to promote to production). Below is the manipulations we do for this proof of concept. Basically just downloading the manifest and renaming from canary to prod. Of course anything else can be done. We also keep the same (except name and some labels, so autoscaling wouldn’t kick in) production manifest as “last known good” (= reference. i.e. which one be used on the next canary cycle at #7)

    And this job publishes 2 Jenkins artifacts update_artifact_list
    Note above can be accessed via http interface same way as we access canary. although we don’t really need build number. we just use your_normal_jenkins_link_to_a_job/lastSuccessfulBuild/artifact/manifest-prod.yml

I am basically happy now. The only step I am missing is that reference file would be great not to keep in jenkins but download from kubernetes deployment (since this is really what is running in production). But this is a smaller thing now


How to bind artifact to a manifest?