Helm Support in the V2 Kubernetes Provider


#1

This is something a lot of users have asked about, and it’s worth discussing in detail, especially with the input from more savvy Helm users.

The question is whether or not Helm packages will be supported in the V2 provider. It’s a natural question, since the whole point of the provider is to unlock manifest-driven deployments, and Helm adds a lot of great tooling making deploying Kubernetes manifests easier. However, my initial thoughts on this are:

  1. Helm is best suited for ready-to-deploy style applications that are not being developed by your team. It’s great for dropping in a Redis or Kafka deployment without much worry about how to run it or apply tested updates. This isn’t really what Spinnaker is made for. Generally when you deploy something with Spinnaker, it’s an application that you’re actively developing, where 1-click deployments aren’t guaranteed to succeed. You want that additional safety that Spinnaker adds, whether it’s via an automated canary strategy, traffic guards, artifact promotion across environments, first-class feature-flag support, etc… Helm clearly has a place in your deployment workflow, but it’s really complementary to Spinnaker in what it’s intended for.

  2. As a result, if you want to rely on Helm as a supplement to Spinnaker, at any point in time you’re able to call out to a “Run Job” or “Run Script” stage that invokes helm apply for you. The types of things that work best with Helm don’t need integration with all the extra tooling that Spinnaker provides for applications that you’re developing.

  3. And finally, there are some technical challenges/limitations that make this difficult. For example, the new Kubernetes provider relies heavily on the metadata.annotations field, which Helm does not expose to you. Also, artifacts like ConfigMaps that Spinnaker now treats as a first-class citizen are also handled very closely by Helm. A clear division of concerns is important here.

I’d love to hear what current/prospective users think.


#2

I have a couple thoughts on this as we use both at Skuid.

RE: #1:
I totally agree with this. It’s the reason we chose Spinnaker over Helm for actively developed applications. We use Helm to deploy things that don’t require a full CI/CD pipeline (CronJobs, supporting infrastructure, etc) and are mostly point-and-shoot. However, it still has a place in our workflow. We’ve built tooling around using Helm to automate ConfigMap and Secret management and generation which our Spinnaker pipelines take advantage of. As you said, it’s really a compliment to Spinnaker rather than an alternative. In order to achieve the same, or similar, level of deployments you’d need heavy scripting which requires a ton of glue code.

RE #3:

I’m curious as to what you mean by this. We use this heavily in our internal charts. Some chart values actually allow you to define any annotations you want.

Personally, I think the biggest reason for the desired Helm chart support is adoption. A lot of companies have invested heavily in Helm as a deployment tool and migrating everything over to Spinnaker could prove to be a burden. I’m not sure how feasable this is, but it might be worth trying to support Helm as a templating option to reduce barrier to entry. This may still require some reworking of charts, but could be less work than moving to Spinnaker & learning ksonnet.


#3

I’d like to echo this. One of the biggest inhibitors I’ve seen to Spinnaker adoption in customer conversations is the prospect of having to invest heavily in converting their existing helm/k8s tooling to Spinnaker.


#4

Does helm allow you to spit out the generated k8s manifests? e.g. here is my helm chart → here are the manifests I can deploy.

If that’s the case, integration would be fairly trivial.


#5

@lwander yeah, i’ll look up the command. There’s also a helm plugin called template.

Edit: it is helm install --dry-run --debug -f <valuesfile> <chartname> helm template may be the way to go.


#6

Oh sweet. This shouldn’t be bad at all. We can trigger off of Kafka, SQS, Pubsub, or other message to notify of a template being generated into something deployable.

For example, you run that command, upload the contents to GCS, and Spinnaker is automatically notified via PubSub message of the file changes, fetches them, and deploys the manifests. That can work today with Kubernetes Provider v2.


#7

That would be a really interesting POC and would probably help reduce the barrier to entry as people start to onboard to the v2 provider.


#8

Hi everyone. I’m start working on Helm integration with Spinnaker. I prepared a simply clouddrivder which supports deploying and deleting Helm charts using UI.




#9

Hey @posox we appreciate the support.

This is going to be a heavy integration, and will require some design discussion first. Please submit a doc outlining:

  1. Resource & operation mapping
  2. User flows
  3. Integration plan alongside the k8s provider
  4. Timeline

#10

I’m working on design doc and planned to publish a draft version after the new year.


#11

@lwander @ethanfrogers I’ve prepared design doc
Helm + Spinnaker design doc


#12

I’ve got some ideas on how to make this work with what’s currently available but haven’t had time to do a POC yet. I figured I’d post my thoughts here in case someone wants attempt this.

The first step would be wrapping helm-template in some type of API that Clouddriver would be able to access. This API would be responsible for rendering a Helm chart and outputting the result. I think you could then use the http/text artifact type to treat this result as a manifest which could be deployed with the V2 provider.

Those of you more familiar with the V2 provider might be able to poke holes in this approach so if you see some, let me know.

Also, I realize I’m just kind of “hand waving” away the values files but I think a POC for this is just rendering a chart and deploying it with Spinnaker. Once that’s proven adding facilities to the “template API” shouldn’t be too bad.


#13

I like the approach - what we need to consider is how it sits alongside other templating system support, and how we can make these feel uniform in design and use. It may be worth planning support for at least one other templating language, and seeing what’s similar and different in our implementation of their support.


#14

I’m chiming in out of the blue here but I wanted to share some thoughts from my experience with helm. In my opinion, the most useful aspect of helm is the templating. I suspect adding a bunch of functionality to work with the other aspects of helm (tiller, etc) is going to be a big lift and clash with other Spinnaker primitives. I think the best approach is to provide some mechanism for taking template artifacts, applying environment specific values to them, emitting raw Kubernetes manifest files, and finally applying those directly to a cluster.

Thanks for all the hard work you guys put into the project!


#15

@ethanfrogers - I’ve got a POC that’s doing pretty much exactly what you describe, using the http/text artifact, but using Run Job in a container running helm to do the actual conversion.

When you say wrapping helm-template in an API, where would you envision this API living within the Spinnaker ecosystem?


#16

@ChuckLane for a POC with the Webhook stage the API could live wherever Spinnaker would have access to it. In my case, I’d run it in the same cluster as Spinnaker and use the cluster DNS for the Webhook URL.

I’m not quite sure how that POC would evolve into something that’s part of Spinnaker as a whole, but maybe once we figure out what works we can find a way to integrate the same pattern for other “template providers”.


#17

I think the important thing is to figure out the common functionality and come up with a consistent interface. It would be really nice if we had a “render template” stage that could interact with some pluggable “template backend”.


#18

Agreed - the tricky thing about at “render template” stage is how many different parameters it can take depending on the templating engine. I would also like to be disciplined in allowing people to

  1. Use “base” templates that are versioned somehow, so there is visibility into what’s being deployed.
  2. Allow people to diff between the generated manifests, and what’s running in the cluster. This is easy enough to implement since we do this already when determining if versioned resources need to be updated.
  3. Find a clean separation between what should be hosted on the templating service, and what should be provided in the stage, and how to implement that.

#19

There are some big changes coming for Helm V3. See the proposal here:

This should make integration for things like Spinnaker much easier.


#20

Hey there everyone! I just wanted to give an update! Thanks to some awesome work by @lwander we now have some experimental native support for Helm! Take a look at the docs here! https://www.spinnaker.io/guides/user/kubernetes-v2/deploy-helm/

Also, If you’re not interested in testing out this alpha feature be sure to check out my blog post on different ways to deploy Helm charts! https://blog.armory.io/deploy-helm-charts-with-spinnaker/

We really appreciate everyone’s feedback and input on this!