Temporally Scale K8 ResourceQuota durring deploy


#1

We’re tightening up our quotas and wondering if it’s possible in Spinnaker to automatically scale up the resourcequotas for blue/green (red/black) deploys and scale them back down automatically?

Otherwise, the quota would always have to be double the nominal limit for this deployment style to work?


#2

By quotas do you mean resource requests/limits? If so, I don’t think that’s possible since those can’t be edited w/o restarting a pod, since it affects scheduling & could create impossible scenarios where a node is overprovisioned.


#3

Sorry, specifically namespace resourceQuotas.

From that page:

When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.
Resource quotas are a tool for administrators to address this concern.

Here is an example ResourceQuota:

$ cat resourceQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: null
  name: test
  namespace: test
spec:
  hard:
    cpu: "10"
    memory: 10G
    pods: 10

Now imagine that I’m running ten pods (single java container per pod), and each only needs 1GB of heap to pre-allocate, now perform a red/black deploy: (pardon this is a V1 provider example)

  1. Execute a pipline that targets the namespace “test”
  2. A second replicaset is created with a new horizontalpodautoscaler set to desired replicas to of (duplicating the pool to a total of 20 pods, each requesting 1 cpu, 1G).

In red/black, I’d expect all of the pods to come up healthy before cutting over. The problem is that the namespace has a quota and we’re essentially exceeding the quota by 2x the pods to perform a red/black deploy. This would be a non-issue if we did a 1:1 rolling update, but we’d prefer faster rollbacks when needed.

Assuming my cluster either has room for the doubling of the pods, or I’ve enabled the Cluster Autoscaler, any recomendation on how I can scale up / down the ResourceQuota in a pipeline?


#4

Thank you for taking a stab at this, I’m going to open a feature request issue. :+1:


#5

I think these apply to pods as they are provisioned, I don’t think you can change these for running pods. It would also probably create strange behavior where only during deployments all resource usage is constrained.