How to bind artifact to a manifest?

#1

I am trying to follow this guide and this. Just reading manifest from gcs and deploying works fine (so manifest on its own is fine). however then I try to bind an image to the manifest pipeline seems to be failing. Tips on further debugging would be appreciated.

I have such triggers and expected artifacts

trigger2
trigger2.png860Ă—815 103 KB

expected
expected.png775Ă—780 34.7 KB

Then I bind it like this

deploy
deploy.png760Ă—616 32.8 KB

And I get the following error (note the account is null. why is that? I trigger the pipeline manually and I get to see the list of available images. I pick one. all works… almost like deploy stage tries to read it on it’s own, but of course I don’t have docker registry account there… there is even no place to specify it…)

error
error.png941Ă—573 47.3 KB

#2

The trick to make this code path working is to make sure that your manifest yml does NOT have tag specified. i.e. normally one has path_to_image:tag whereas I could only make that pipeline working if in my yml I do not specify tag.

well https://www.spinnaker.io/reference/artifacts/in-kubernetes-v2/#versioned-kubernetes-objects also shows that yml should not have tags.

So according to specs it works.

I would love to learn more about the logic.
i.e. yml without a tag is not deploy able. so now one need to keep 2 sets of yml’s: one for spinnaker (without tags) and one for “day to day” test experimentation. Sort feels inefficient. But maybe developers of spinnaker have some other ideas in mind. Would be glad to hear.

#3

It’s not so much a question of having a set of manifests w/ tags and a set w/o tags, but instead think about it like:

  1. The manifests that my deployment solution understands. This might involve templating, assigning images/configs from the context, etc… In the end, it’s the path your tooling understands, and beyond simple examples, always involve extra information/logic to deploy.

  2. The manifests that have been deployed, and are fully hydrated/expanded. These represent what your cluster looks like, and are a sort of “machine-code” representation of the manifests in 1. These aren’t meant to be hand edited, but instead exist for auditability/recoverability.

Spinnaker expects 1, and generates 2 by writing events to Spinnaker’s event bus, echo. There isn’t an out-of-the-box story for easily storing those events yet, but it’s something we’re working on.

Ideally, for development, your manifests in 1 should easily expandable into what’s appropriate for your dev environment, and don’t even need to rely on the artifact substitution if not necessary.

#4

Thanks, @lwander . I see the thinking behind it.

As you pointed out, beyond simple examples it gets very complicated very fast :wink: Now my next complication:

I am now on the next stage. how to keep image and manifest as a same “bundle”. so we never run into problem: ah this image doesn’t work with this manifest and vice versa Image and manifest as a bundle?

#5

If you don’t want to rely on the artifact substitution it’s totally valid to just send the fully expanded image in the manifest to Spinnaker as well. This is a sort of “bundle” as well.

Under what circumstances to you expect to run into that problem?

#6

Yes, I can generate a manifest with an image printed in. But then I have 2 problems

  1. If I generate deployment.yml and make it an artifact. all works. except that every jenkins build just overwrites this file. making it hard to rerun the pipeline.
  2. I can generate deployment_v1.0.2.yml and deployment_v1.0.3.yml but now I don’t know how to point pipeline to such manifest ( say it is on gs://my_bucket/deployment_v1.0.2.yml)

So in case of a jenkins build and associated property file or an image in a repository we can point to a general direction and the system figures out how to deal with versioning/tagging. Thus allowing us to trigger pipelines later and picking build (or image tag) we are interested in. I couldn’t find any functionality like this for manifest. Maybe I am just missing.

And as to where it might be useful: as I point out in this thread Every so often one changes manifest so much that it only works with a specific image. Since manifest is in git and image is build by jenkins based on the same git. one might as well treat them as a bundle. as an alternative, jenkins can generate a manifest with an associated image printed in (i.e. task as binding does today by deploy stage). but then we run into issue #2 above (at least I don’t know how to point to a general direction of the artifact and versioning will be taken care of by giving some regex or similar).

Of course, don’t get me wrong. manifest as an artifact and image binding is great! Just then I wired things together with jenkins generating an image and overwriting deployment.yml I realized that with this system we can only “roll forward”. in case image and manifest are out of sync. just press build again and hope they will be in sync again (and there is no way going back to “let me try image “n-3” and associated manifest. what exactly went wrong there”. since even if image is present, manifest is already only of version “n” ) . So started to brainstorm how this can be improved

#7

What’s wrong with using a regex in the “match artifact” using say
gs://my_bucket/deploymentv.*\.yml?

#8

The use case is you want to try different versions of the manifest & image together?

We’ve been bouncing around the idea of showing the history of artifacts that triggered a pipeline when you select “manual execution”. This would allow you to select a different set of artifacts if you manually trigger a pipeline as opposed to taking all the artifacts at once.

#9

I tried
gs://my_bucket/deploymentv.*\.yml
gs://my_bucket\/deploymentv.*\.yml
gs:\/\/my_bucket\/deploymentv.*\.yml (<- this one one is "proper" regex match to gs://my_bucket/deploymentv1.0.1.yml)

None of them gives me a pop up asking to pick one of the available files.
Is it expected artifacts you mean ? Or maybe I put it in some wrong place?

expected
expected.png750Ă—441 22.5 KB

I also tried to put it on the trigger…still nothing pops up then I manually trigger (whereas if I do the same for container repo or jenkins. If I manually trigger I get a popup asking to pick exact image/execution)

trigger
trigger.png765Ă—448 28.1 KB

#10

This is how I eventually solved the problem. Essentially binding everything in Jenkins.