Fail to consume artifact as a manifest in 1.7.2


#1

I am following this manual (using GCS): https://www.spinnaker.io/guides/user/kubernetes-v2/deploy-manifest/ and setup my trigger this way (again GCS) https://www.spinnaker.io/guides/user/triggers/gcs/

I can deploy the app if if I have manifest as text. but then I switch to artifact I fail. and the error message is pretty cryptic

Note, If I just call PUT on http://localhost:7002/artifacts/fetch
with payload
{"artifactAccount":"my-gcs-artifact-account", "type":"gcs/object", "reference":"gs://spin-artifact-test/test/canary_desc.yml", "name":"gs://spin-artifact-test/test/canary_desc.yml" }

I actually get my artifact back (i.e. the manifest)

Any idea how to debug it further? I tried both gcs and custom artifact type. same error ) . Kind of lost :frowning:

:[{"context":{"account":"my-kubernetes-account","cloudProvider":"kubernetes","exception":{"details":{"error":"Unexpected Task Failure","errors":["Cannot deserialize instance ofjava.util.LinkedHashMapout of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]"], "stackTrace":"java.lang.IllegalArgumentException: Cannot deserialize instance ofjava.util.LinkedHashMapout of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]\n\tat com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3738)\n\tat com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3656)\n\tat com.netflix.spinnaker.orca.clouddriver.tasks.manifest.DeployManifestTask.lambda$execute$1(DeployManifestTask.java:99)\n\tat java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)\n\tat java.util.Iterator.forEachRemaining(Iterator.java:116)\n\tat java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)\n\tat java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)\n\tat java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)\n\tat java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)\n\tat java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)\n\tat java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)\n\tat com.netflix.spinnaker.orca.clouddriver.tasks.manifest.DeployManifestTask.execute(DeployManifestTask.java:100)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:82)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAwareKt$sam$Callable$55f02348.call(AuthenticationAware.kt)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$1(AuthenticatedRequest.java:79)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:49)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:81)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:173)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:47)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:31)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:57)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:31)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:66)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:53)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:40)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:166)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:63)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)\n\tat com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:51)\n\tat com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)\n\tat com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:74)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance ofjava.util.LinkedHashMapout of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]\n\tat com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)\n\tat com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)\n\tat com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)\n\tat com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)\n\tat com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:599)\n\tat com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:360)\n\tat com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)\n\tat com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3733)\n\t... 42 more\n"},"exceptionType":"IllegalArgumentException","operation":"deployManifest","shouldRetry":false,"timestamp":1525250615391}, "manifestArtifactAccount":"my-gcs-artifact-account","manifestArtifactId":"0ac314ff-f0f4-4655-a29b-faba3eba9355",


#2

Can you post the “triggers” and “expectedArtifacts” sections of your pipeline json?


#3

This may be related to an issue that @lwander was looking into, and has the following PRs to fix:
https://github.com/spinnaker/orca/pull/2196 (to work around the issue)
https://github.com/spinnaker/orca/pull/2198 (to fix the suspected root cause)


#4

here is the trigger part of the pipeline which failed


#5

@AndreiJ is this behavior (exact error & all) consistent?


#6

yes. 100% repeatable. I can run/modify something if you have ideas. One more piece: I didn’t seem to be able to trigger the pipeline from pup/sub. but since I am starting it manually I parked debugging it. hopefully those are not interconnected

here is how pipeline looks like: trigger part


Screenshot_3

deploy part


#7

Just to be sure. I run above pipeline. and here is the json I get

{  
   "application":"infra",
   "authentication":{  
      "allowedAccounts":[  
         "my-kubernetes-account"
      ],
      "user":"anonymous"
   },
   "buildTime":1525282847231,
   "canceled":false,
   "canceledBy":null,
   "cancellationReason":null,
   "description":null,
   "endTime":1525282849973,
   "id":"5f0582a8-c6c7-4f12-9be5-a2a582b5a360",
   "initialConfig":{  

   },
   "keepWaitingPipelines":false,
   "limitConcurrent":true,
   "name":"test auto",
   "notifications":[  

   ],
   "origin":"deck",
   "paused":null,
   "pipelineConfigId":"72861ce6-2ebc-49f6-9dac-4bd3b3e59cf4",
   "stages":[  
      {  
         "context":{  
            "account":"my-kubernetes-account",
            "cloudProvider":"kubernetes",
            "exception":{  
               "details":{  
                  "error":"Unexpected Task Failure",
                  "errors":[  
                     "Cannot deserialize instance of `java.util.LinkedHashMap` out of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]"
                  ],
                  "stackTrace":"java.lang.IllegalArgumentException: Cannot deserialize instance of `java.util.LinkedHashMap` out of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]
 com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3738) 
 com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:3656) 
 com.netflix.spinnaker.orca.clouddriver.tasks.manifest.DeployManifestTask.lambda$execute$1(DeployManifestTask.java:99)
 java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) 
 java.util.Iterator.forEachRemaining(Iterator.java:116) 
 java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) 
 java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) 
 java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) 
 java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) 
 java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 
 java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) 
 com.netflix.spinnaker.orca.clouddriver.tasks.manifest.DeployManifestTask.execute(DeployManifestTask.java:100) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:82) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1$1.invoke(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.AuthenticationAwareKt$sam$Callable$55f02348.call(AuthenticationAware.kt) 
 com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$1(AuthenticatedRequest.java:79) 
 com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:49) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withAuth(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:81) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$handle$1.invoke(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:173) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler$withTask$1.invoke(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:47) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withTask$1.invoke(OrcaMessageHandler.kt:31) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:57) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:31) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:66) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withExecution(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:53) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withStage(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withTask(OrcaMessageHandler.kt:40) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.withTask(RunTaskHandler.kt:166) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:63) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.handle(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36) 
 com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt) 
 com.netflix.spinnaker.orca.q.handler.RunTaskHandler.invoke(RunTaskHandler.kt:51) 
 com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:47)
 com.netflix.spinnaker.q.QueueProcessor$pollOnce$1$1.run(QueueProcessor.kt:74) 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 java.lang.Thread.run(Thread.java:748)\nCaused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.LinkedHashMap` out of START_ARRAY token\n at [Source: UNKNOWN; line: -1, column: -1]
 com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) 
 com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342) 
 com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138) 
 com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092) 
 com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromEmpty(StdDeserializer.java:599) 
 com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:360) 
 com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) 
 com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:3733)\n\t... 42 more\n"
               },
               "exceptionType":"IllegalArgumentException",
               "operation":"deployManifest",
               "shouldRetry":false,
               "timestamp":1525282849884
            },
            "manifestArtifactAccount":"my-gcs-artifact-account",
            "manifestArtifactId":"5f10b705-1eac-4c72-b295-f6a63229c81f",
            "moniker":{  
               "app":"infra",
               "cluster":"yapt-test"
            },
            "relationships":{  
               "loadBalancers":[  

               ],
               "securityGroups":[  

               ]
            },
            "requiredArtifactIds":[  

            ],
            "source":"artifact"
         },
         "endTime":1525282849933,
         "id":"711f9eed-bee6-49b3-9a6d-a260d7e776bd",
         "lastModified":null,
         "name":"Deploy (Manifest)",
         "outputs":{  

         },
         "parentStageId":null,
         "refId":"1",
         "requisiteStageRefIds":[  

         ],
         "scheduledTime":null,
         "startTime":1525282847285,
         "startTimeExpiry":null,
         "status":"TERMINAL",
         "syntheticStageOwner":null,
         "tasks":[  
            {  
               "endTime":1525282849904,
               "id":"1",
               "implementingClass":"com.netflix.spinnaker.orca.clouddriver.tasks.manifest.DeployManifestTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"deployManifest",
               "stageEnd":false,
               "stageStart":true,
               "startTime":1525282847301,
               "status":"TERMINAL"
            },
            {  
               "endTime":null,
               "id":"2",
               "implementingClass":"com.netflix.spinnaker.orca.clouddriver.tasks.MonitorKatoTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"monitorDeploy",
               "stageEnd":false,
               "stageStart":false,
               "startTime":null,
               "status":"NOT_STARTED"
            },
            {  
               "endTime":null,
               "id":"3",
               "implementingClass":"com.netflix.spinnaker.orca.clouddriver.tasks.manifest.PromoteManifestKatoOutputsTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"promoteOutputs",
               "stageEnd":false,
               "stageStart":false,
               "startTime":null,
               "status":"NOT_STARTED"
            },
            {  
               "endTime":null,
               "id":"4",
               "implementingClass":"com.netflix.spinnaker.orca.clouddriver.tasks.manifest.ManifestForceCacheRefreshTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"forceCacheRefresh",
               "stageEnd":false,
               "stageStart":false,
               "startTime":null,
               "status":"NOT_STARTED"
            },
            {  
               "endTime":null,
               "id":"5",
               "implementingClass":"com.netflix.spinnaker.orca.clouddriver.tasks.manifest.WaitForManifestStableTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"waitForManifestToStabilize",
               "stageEnd":false,
               "stageStart":false,
               "startTime":null,
               "status":"NOT_STARTED"
            },
            {  
               "endTime":null,
               "id":"6",
               "implementingClass":"com.netflix.spinnaker.orca.pipeline.tasks.artifacts.BindProducedArtifactsTask",
               "loopEnd":false,
               "loopStart":false,
               "name":"bindProducedArtifacts",
               "stageEnd":true,
               "stageStart":false,
               "startTime":null,
               "status":"NOT_STARTED"
            }
         ],
         "type":"deployManifest"
      }
   ],
   "startTime":1525282847269,
   "startTimeExpiry":null,
   "status":"TERMINAL",
   "trigger":{  
      "artifacts":[  
         {  
            "metadata":{  
               "kind":"default.gcs"
            },
            "name":"gs://spin-artifact-test/test/canary_desc.yml",
            "reference":"gs://spin-artifact-test/test/canary_desc.yml",
            "type":"gcs/object"
         }
      ],
      "correlationId":null,
      "dryRun":false,
      "notifications":[  

      ],
      "parameters":{  

      },
      "rebake":false,
      "resolvedExpectedArtifacts":[  
         {  
            "boundArtifact":{  
               "metadata":{  
                  "kind":"default.gcs"
               },
               "name":"gs://spin-artifact-test/test/canary_desc.yml",
               "reference":"gs://spin-artifact-test/test/canary_desc.yml",
               "type":"gcs/object"
            },
            "defaultArtifact":{  
               "metadata":{  
                  "kind":"default.gcs"
               },
               "name":"gs://spin-artifact-test/test/canary_desc.yml",
               "reference":"gs://spin-artifact-test/test/canary_desc.yml",
               "type":"gcs/object"
            },
            "id":"5f10b705-1eac-4c72-b295-f6a63229c81f",
            "matchArtifact":{  
               "metadata":{  
                  "kind":"gcs"
               },
               "name":"gs://spin-artifact-test/test/canary_desc.yml",
               "type":"gcs/object"
            },
            "useDefaultArtifact":true,
            "usePriorArtifact":false
         }
      ],
      "strategy":false,
      "type":"manual",
      "user":"[anonymous]"
   },
   "type":"PIPELINE"
}

#8

and just to be 200% sure. I run the same pipeline, just instead of the artifact in the deploy stage copy/pasted the content of the manifest (gs://spin-artifact-test/test/canary_desc.yml) into the text field. all works. so content of the yml is ok.


#9

@lwander, any hints on if I miss something, or if this is a bug in the current builds?


#10

Switching to this build of orca fixed the problem: gcr.io/spinnaker-marketplace/orca:0.12.0-20180506022807