Best practices on multiple vs single pipeline for dependent services


#1

I’m trying to use Spinnaker to manage two related services:

  1. Monolith: complicated SOAP-ish API
  2. Facade: simplified FORM-POST facade on the Monolith

Currently, implementing a feature usually involves making a business logic change in the Monolith then exposing those parameters in the simpler Facade API (the long-term goal is to make Facade a truly independent micro-service).

What’s the Spinnaker community’s take on using multiple vs single pipelines for this situation?

My gut instict is to use a single pipeline to deploy both services since they are coupled, but I could also see multiple pipelines where the Facade pipeline is triggered when the Monolith pipeline is successful. It seems like the choice would depend on which one provides a more coherent process, e.g.: rolling back only Facade vs rolling back both services.

but it’s unclear from the Codelabs what the best practices on Rollbacks are, e.g.:

  • conditional Rollback stages in the pipeline based on an integration test failure
  • manual Rollbacks operation from the Server Group panel (independent of Pipelines)

#2

There’s no definitive answer, but keep in mind that splitting things into separate pipelines can be very helpful for making things individually retryable, but comes at the cost of splitting your workflow (visually) into separate components, perhaps making it harder to follow.

The automation of rollbacks depends on how confident you are in your integration tests – if you can confidently rollback when an integration test fails, then that’s best to automate. If you’re not quite sure, you could follow your integration test with a “manual judgement” stage that prevents the rollout from happening automatically, but will email/ping/sms the team when it’s time to decide what to do.


#3

For the case of separate Pipelines, is there any guidance on how rollback should work: would there be a complimentary Pipelines containing a Edit Load Balancer stage?


#4

Are you asking to rollback load balancer configuration as well?


#5

Yes. The Safe Deployment CodeLab demonstrate using Manual Judgments at various points during a Pipeline to edit the Load Balance configuration:

  • Enable initial Canary traffic split
  • Final switch to new Server Groups

There are discussions of performing manual rollback and also creating a triggered rollback Pipeline.

Going through the mental exercise of creating chained Deploy Pipelines (in a dependency order), it seems straightforward to add a trigger-on-failure Rollback Pipeline for each service in isolation but chaining those Rollbacks together starts to get convoluted, e.g.:

success = deploy(A)
if(success) {
  success = deploy(B)
  if(success) {
    success = deploy(C)
    if(success) {
      // done
    } else {
      rollback(C) // convoluted rollback chains
      rollback(B)
      rollback(A)
    }
  } else {
    rollback(B) // convoluted rollback chains
    rollback(A)
  }
} else {
  rollback(A)
}

so would dependent services tend to sway the decision more towards a single large pipeline?


#6

The rollback will modify the load balancer configuration, you don’t need to edit a load balancer to do that.

Given that you are deploying A before B, (meaning new version of A can run with old version of B), I don’t see why you need to rollback both A & B at once. Ideally, the new A can keep running B fails, and (even better) at some point A & B can be deployed & rolled back independently.

However, those rollback chains (if really needed) aren’t as bad as you might think. You can have rollback(A) trigger on either deploy(A) fails testing, or rollback(B) completes. Same goes for rollback(B) against deploy(B) fails testing or rollback(C) completes. rollback(C) of course only triggers when deploy(C) fails testing.