[Solved] Question about Clouddriver and Kinds


#1

Hi,

I am building out a new k8s cluster from the ground up on AWS, and tried reusing all my yaml files from my previous installation but I’ve realized there might be differences with later versions of Spinnaker.

k8s : v1.10.6
spinnaker: v1.9.5

I have noticed that my clouddriver continues to check for Kinds, and it takes forever.

 2018-11-15 02:41:53.402  INFO 1 --- [           main] c.n.s.c.k.v.s.KubernetesV2Credentials    : Checking permissions on configured kinds... [apiService, clusterRole, clusterRoleBinding, configMap, controllerRevision, customResourceDefinition, cronJob, daemonSet, deployment, event, horizontalpodautoscaler, ingress, job, pod, podDisruptionBudget, replicaSet, role, roleBinding, namespace, networkPolicy, persistentVolume, persistentVolumeClaim, secret, service, serviceAccount, statefulSet, storageClass, none]`
 2018-11-15 02:43:36.309  INFO 1 --- [           main] c.n.s.c.k.v.s.KubernetesV2Credentials    : Kind 'apiService' will not be cached in account 'my-k8s-v2-account' for reason: 'Job took too long to complete'
 2018-11-15 02:45:19.024  INFO 1 --- [           main] c.n.s.c.k.v.s.KubernetesV2Credentials    : Kind 'clusterRole' will not be cached in account 'my-k8s-v2-account' for reason: 'Job took too long to complete'

I don’t ever recall running into this before. Cloud drive will keep trying until all the Kinds have been tried. I haven’t let it go all the way yet due to my impatient nature, but I’m letting it run all the way through to see what’s going on.

Could someome tell me what is happening here? I did see a place in the hal config to omit Kinds but don’t want to play around with things I don’t understand.

Thanks!


#2

I was thinking this could be related to RBAC because I remember having troubles with RBAC and Spinnaker in the past so I made all new yml files for rbac based on the lastest docs, which included making the service account a cluster-admin.

I reinstalled everything by running:

hal deploy clean
hal deploy apply

I have even tried this with v1.10.4

Thanks!


#3

Alright all, figured it out.

hal config provider kubernetes account edit my-k8s-v2-account --service-account true

Everything works!

Thanks to this doc:


#4

I’ve been receiving these errors as well in clouddriver startup. From what I understand, clouddriver is going through all your V2 provider accounts and scanning which kubernetes resources it can use (for caching, deploying etc).

I haven’t figured out exactly what my problem is but I think it may be about hitting rate limits on kubernetes master (we use GKE). In my case the problem did not arise always. Sometimes the startup would go just fine and other times it would take hours and hours.

One thing you can try is adding --kinds or --omit-kinds list to you provider account. Then it would not even try to use all kubernetes resources (as long as you know you don’t need them). This should also speed up the startup time