Unable account on Canary settings


#1

Hi.

I used spinnaker. spinnakerVersion : 1.11.7 with kubernetes v2 provider.
Kubernetes is set up by kops on AWS.

When I wanna set up canary on Spinnaker, I set up it, but I cannot use the Canary Stage on Pipeline.

You can see my problem, unable account on the below image.

What’s wrong of me?

It’s my configure for canary.

canary:
    enabled: true
    serviceIntegrations:
    - name: google
      enabled: false
      accounts: []
      gcsEnabled: false
      stackdriverEnabled: false
    - name: prometheus
      enabled: true
      accounts:
      - name: default-prom
        endpoint:
          baseUrl: http://prometheus-service.monitoring.svc.cluster.local:9090
        supportedTypes:
        - METRICS_STORE
    - name: datadog
      enabled: false
      accounts: []
    - name: signalfx
      enabled: false
      accounts: []
    - name: aws
      enabled: true
      accounts:
      - name: default-aws
        bucket: spinnaker-bucket-name
        region: us-east-1
        rootFolder: kayenta
        accessKeyId: AKIAJJUWVGQJUEA3FD8Q
        secretAccessKey: secret
        supportedTypes:
        - CONFIGURATION_STORE
        - OBJECT_STORE
      s3Enabled: true
    reduxLoggerEnabled: true
    defaultMetricsAccount: default-prom
    defaultStorageAccount: default-aws
    defaultJudge: NetflixACAJudge-v1.0
    defaultMetricsStore: prometheus
    stagesEnabled: true
    templatesEnabled: true
    showAllConfigsEnabled: true

#2

Hi,

First thing to try is querying the credentials endpoint of kayenta itself. If you shell into the pod, you can try:
wget -qO- localhost:8090/credentials

Ensure that the response properly represents the desired configuration.

Assuming the configuration is as expected, second thing to check is the logs of the kayenta pod. Start from the beginning and look for stacktraces or issues reaching prometheus. You should see log entries indicating that it is querying your prometheus instance for the available metrics at a regular interval. If those calls are failing, ensure that you are able to reach prometheus yourself from the kayenta pod.

If that looks ok, next thing is to see if you can directly query kayenta for the available metric names:

wget -qO- localhost:8090/metadata/metricsService?metricsAccountName=default-prom

Let us know how those initial troubleshooting steps go.

Thanks,
-Matt


#3

Hi Matt.
Thank you for advise.
I checked my spinnaker following your steps.

Firstly I tried this command into my kayenta pod.

bash-4.4$ wget -qO- localhost:8090/credentials
[ {
  "name" : "default-aws",
  "supportedTypes" : [ "CONFIGURATION_STORE", "OBJECT_STORE" ],
  "bucket" : "my-bucket-name",
  "region" : "us-east-1",
  "rootFolder" : "kayenta",
  "type" : "aws",
  "recommendedLocations" : [ ],
  "locations" : [ ]
}, {
  "name" : "default-prom",
  "supportedTypes" : [ "METRICS_STORE" ],
  "endpoint" : {
    "baseUrl" : "http://prometheus-service.monitoring.svc.cluster.local:9090"
  },
  "type" : "prometheus",
  "recommendedLocations" : [ ],
  "locations" : [ ]
} ]

And I checked my kayenta pod logs, then I could find some warning logs.

2019-01-28 03:48:50.712  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfig) -> Model
Context{groupName=default, type=com.netflix.kayenta.canary.CanaryConfig, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.716  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfig) -> Model
Context{groupName=default, type=com.netflix.kayenta.canary.CanaryConfig, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.726  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfig) -> Model
Context{groupName=default, type=com.netflix.kayenta.canary.CanaryConfig, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.728  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfig) -> Model
Context{groupName=default, type=com.netflix.kayenta.canary.CanaryConfig, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.730  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfigUpdateResp
onse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryConfigUpdateResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.731  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryConfigUpdateResp
onse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryConfigUpdateResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.781  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionStatusR
esponse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionStatusResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.783  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionStatusR
esponse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionStatusResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.795  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionStatusR
esponse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionStatusResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.797  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionStatusR
esponse) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionStatusResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.809  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionRequest
) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionRequest, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.810  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionRequest
) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionRequest, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.811  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionRespons
e) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.812  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryExecutionRespons
e) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryExecutionResponse, isReturnType=true}. java.lang.NullPointerException
2019-01-28 03:48:50.823  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryAdhocExecutionRe
quest) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryAdhocExecutionRequest, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.824  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.CanaryAdhocExecutionRe
quest) -> ModelContext{groupName=default, type=com.netflix.kayenta.canary.CanaryAdhocExecutionRequest, isReturnType=false}. java.lang.NullPointerException
2019-01-28 03:48:50.844  WARN 1 --- [           main] s.d.s.p.CachingModelPropertiesProvider   : Exception calculating properties for model(com.netflix.kayenta.canary.results.CanaryResult) 
-> ModelContext{groupName=default, type=com.netflix.kayenta.canary.results.CanaryResult, isReturnType=true}. java.lang.NullPointerException
...

I think it’s problems because of my configuration.
But I don’t know exactly how do I do.
What would you advise me to do again?

p.s. I could see my metrics on prometheus after execute last command, wget.


#4

Hi,

Based on some quick searches, those warnings look to be related to swagger configuration. In any case, I don’t (at least not yet) think they are the reason your ui is not populated with the prometheus metric names.

Since querying kayenta directly allowed you to retrieve the prometheus metric names, let’s next try querying gate. The ui (deck) calls gate, and gate in turn calls kayenta. So we are troubleshooting from the inside out.

Try this query from within the gate pod:

wget -qO- http://localhost:8084/v2/canaries/metadata/metricsService?metricsAccountName=default-prom

If that works ok, the next thing to do is to bring up the Spinnaker ui and open Chrome’s Network tab in the dev console.

When you type 3 characters or more in the Metric Name field while configuring a Prometheus metric, you should see a call to gate just like the one we tried above. Check the response and also look for console errors.

Thanks,
-Matt


#5

Hi.

I checked that you mentioned.

wget -qO- http://localhost:8084/v2/canaries/metadata/metricsService?metricsAccountName=default-prom

I think it looks good. I confirmed metrics on JSON.

But when I try to create Canary Stage on Pipeline, I can see other error.

I think it’s weird.
404 means it’s client error.

This URL path is right?

/gate/canaryConfig/names

Is that my configuration is fault?