Helm Support in the V2 Kubernetes Provider


#1

This is something a lot of users have asked about, and it’s worth discussing in detail, especially with the input from more savvy Helm users.

The question is whether or not Helm packages will be supported in the V2 provider. It’s a natural question, since the whole point of the provider is to unlock manifest-driven deployments, and Helm adds a lot of great tooling making deploying Kubernetes manifests easier. However, my initial thoughts on this are:

  1. Helm is best suited for ready-to-deploy style applications that are not being developed by your team. It’s great for dropping in a Redis or Kafka deployment without much worry about how to run it or apply tested updates. This isn’t really what Spinnaker is made for. Generally when you deploy something with Spinnaker, it’s an application that you’re actively developing, where 1-click deployments aren’t guaranteed to succeed. You want that additional safety that Spinnaker adds, whether it’s via an automated canary strategy, traffic guards, artifact promotion across environments, first-class feature-flag support, etc… Helm clearly has a place in your deployment workflow, but it’s really complementary to Spinnaker in what it’s intended for.

  2. As a result, if you want to rely on Helm as a supplement to Spinnaker, at any point in time you’re able to call out to a “Run Job” or “Run Script” stage that invokes helm apply for you. The types of things that work best with Helm don’t need integration with all the extra tooling that Spinnaker provides for applications that you’re developing.

  3. And finally, there are some technical challenges/limitations that make this difficult. For example, the new Kubernetes provider relies heavily on the metadata.annotations field, which Helm does not expose to you. Also, artifacts like ConfigMaps that Spinnaker now treats as a first-class citizen are also handled very closely by Helm. A clear division of concerns is important here.

I’d love to hear what current/prospective users think.


#2

I have a couple thoughts on this as we use both at Skuid.

RE: #1:
I totally agree with this. It’s the reason we chose Spinnaker over Helm for actively developed applications. We use Helm to deploy things that don’t require a full CI/CD pipeline (CronJobs, supporting infrastructure, etc) and are mostly point-and-shoot. However, it still has a place in our workflow. We’ve built tooling around using Helm to automate ConfigMap and Secret management and generation which our Spinnaker pipelines take advantage of. As you said, it’s really a compliment to Spinnaker rather than an alternative. In order to achieve the same, or similar, level of deployments you’d need heavy scripting which requires a ton of glue code.

RE #3:

I’m curious as to what you mean by this. We use this heavily in our internal charts. Some chart values actually allow you to define any annotations you want.

Personally, I think the biggest reason for the desired Helm chart support is adoption. A lot of companies have invested heavily in Helm as a deployment tool and migrating everything over to Spinnaker could prove to be a burden. I’m not sure how feasable this is, but it might be worth trying to support Helm as a templating option to reduce barrier to entry. This may still require some reworking of charts, but could be less work than moving to Spinnaker & learning ksonnet.


#3

I’d like to echo this. One of the biggest inhibitors I’ve seen to Spinnaker adoption in customer conversations is the prospect of having to invest heavily in converting their existing helm/k8s tooling to Spinnaker.


#4

Does helm allow you to spit out the generated k8s manifests? e.g. here is my helm chart → here are the manifests I can deploy.

If that’s the case, integration would be fairly trivial.


#5

@lwander yeah, i’ll look up the command. There’s also a helm plugin called template.

Edit: it is helm install --dry-run --debug -f <valuesfile> <chartname> helm template may be the way to go.


#6

Oh sweet. This shouldn’t be bad at all. We can trigger off of Kafka, SQS, Pubsub, or other message to notify of a template being generated into something deployable.

For example, you run that command, upload the contents to GCS, and Spinnaker is automatically notified via PubSub message of the file changes, fetches them, and deploys the manifests. That can work today with Kubernetes Provider v2.


#7

That would be a really interesting POC and would probably help reduce the barrier to entry as people start to onboard to the v2 provider.


#8

Hi everyone. I’m start working on Helm integration with Spinnaker. I prepared a simply clouddrivder which supports deploying and deleting Helm charts using UI.




#9

Hey @posox we appreciate the support.

This is going to be a heavy integration, and will require some design discussion first. Please submit a doc outlining:

  1. Resource & operation mapping
  2. User flows
  3. Integration plan alongside the k8s provider
  4. Timeline

#10

I’m working on design doc and planned to publish a draft version after the new year.


#11

@lwander @ethanfrogers I’ve prepared design doc
Helm + Spinnaker design doc