27. Stream Lifecycle with Skipper

Skipper is a server that allows you to discover Spring Boot applications and manage their lifecycle on multiple Cloud Platforms.

Applications in Skipper are bundled as packages which contain templated configuration files. They also contain an optional values file that contains default values using to fill in template placeholders. You can find out more about the format of the package .zip file in Skipper’s documentation on Packages. Skipper’s templated configuration files contain placeholders for application properties, application version, and deployment properties. Package .zip files are uploaded to Skipper and stored in a package repository. Skipper’s package repository is analogous to those found in tools such as apt-get or brew.

You can override template values when installing or upgrading a package. Skipper orchestrates the upgrade/rollback procedure of applications between different versions, taking the minimal set of actions to bring the system to the desired state. For example, if only one application in a stream has been updated, only that single application is deployed with a new version and the old version undeployed. An application is considered different when upgrading if any of it’s application properties, deployment properties (excluding count), or application version (e.g. 1.0.0.RELEASE) is different from the currently installed application.

Spring Cloud Data Flow is integrated with Skipper by generating a Skipper package when deploying a Stream. The generated package name is the same name as the Stream. The generated package is uploaded to Skipper’s package repository and Data Flow then instructs Skipper to install the package that corresponds to the Stream. Subsequent commands to upgrade and rollback applications within the Stream are passed through to Skipper after some validation checks are performed by Data Flow.

27.1 Creating and Deploying a Stream

You create and deploy a stream using skipper in two steps, creating the stream definition and then deploying the stream.

dataflow:> stream create --name httptest --definition "http --server.port=9000 | log"
dataflow:> stream skipper deploy --name httptest

There is an important optional command argument to the stream skipper deploy command, which is --platformName. Skipper can be configured to deploy to multiple platforms. Skipper is pre-configured with a platform named default which will deploys applications to the local machine where Skipper is running. The default value of the command line argument --platformName is default. If you are commonly deploying to one platform, when installing Skipper you can override the configuration of the default platform. Otherwise, specify the platformName to one of the values returned by the command stream skipper platform-list

[Note]Note

In future releases, only the local Data Flow server will be configured with the default platform.

27.2 Updating a Stream

To update the stream, use the command stream skipper update which takes as a command argument either --properties or --propertiesFile. You can pass in values to these command arguments in the same format as when deploy the stream with or without Skipper. There is an important new top level prefix available when using Skipper, which is version. If the Stream http | log was deployed, and the version of log which registered at the time of deployment was 1.1.0.RELEASE, the following command will update the Stream to use the 1.2.0.RELEASE of the log application.

dataflow:>stream skipper update --name httptest --properties version.log=1.2.0.RELEASE

27.3 Stream versions

Skipper keeps a history of the Streams that were deployed. After updating a Stream, there will be a second version of the stream. You can query for the history of the versions using the command stream skipper history --name <name-of-stream>.

dataflow:>stream skipper history --name httptest
╔═══════╤════════════════════════════╤════════╤════════════╤═══════════════╤════════════════╗
║Version│        Last updated        │ Status │Package Name│Package Version│  Description   ║
╠═══════╪════════════════════════════╪════════╪════════════╪═══════════════╪════════════════╣
║2      │Mon Nov 27 22:41:16 EST 2017│DEPLOYED│httptest    │1.0.0          │Upgrade complete║
║1      │Mon Nov 27 22:40:41 EST 2017│DELETED │httptest    │1.0.0          │Delete complete ║
╚═══════╧════════════════════════════╧════════╧════════════╧═══════════════╧════════════════╝

27.4 Stream Manifests

Skipper keeps an "manifest" of the all the applications, their application properties and deployment properties after all values have been substituted. This represents the final state of what was deployed to the platform. You can view the manifest for any of the versions of a Stream using the command stream skipper manifest --name <name-of-stream> --releaseVersion <optional-version> If the --releaseVersion is not specified, the manifest for the last version is returned.

dataflow:>stream skipper manifest --name httptest

---
# Source: log.yml
apiVersion: skipper.spring.io/v1
kind: SpringCloudDeployerApplication
metadata:
  name: log
spec:
  resource: maven://org.springframework.cloud.stream.app:log-sink-rabbit
  version: 1.2.0.RELEASE
  applicationProperties:
    spring.metrics.export.triggers.application.includes: integration**
    spring.cloud.dataflow.stream.app.label: log
    spring.cloud.stream.metrics.key: httptest.log.${spring.cloud.application.guid}
    spring.cloud.stream.bindings.input.group: httptest
    spring.cloud.stream.metrics.properties: spring.application.name,spring.application.index,spring.cloud.application.*,spring.cloud.dataflow.*
    spring.cloud.dataflow.stream.name: httptest
    spring.cloud.dataflow.stream.app.type: sink
    spring.cloud.stream.bindings.input.destination: httptest.http
  deploymentProperties:
    spring.cloud.deployer.indexed: true
    spring.cloud.deployer.group: httptest
    spring.cloud.deployer.count: 1

---
# Source: http.yml
apiVersion: skipper.spring.io/v1
kind: SpringCloudDeployerApplication
metadata:
  name: http
spec:
  resource: maven://org.springframework.cloud.stream.app:http-source-rabbit
  version: 1.2.0.RELEASE
  applicationProperties:
    spring.metrics.export.triggers.application.includes: integration**
    spring.cloud.dataflow.stream.app.label: http
    spring.cloud.stream.metrics.key: httptest.http.${spring.cloud.application.guid}
    spring.cloud.stream.bindings.output.producer.requiredGroups: httptest
    spring.cloud.stream.metrics.properties: spring.application.name,spring.application.index,spring.cloud.application.*,spring.cloud.dataflow.*
    server.port: 9000
    spring.cloud.stream.bindings.output.destination: httptest.http
    spring.cloud.dataflow.stream.name: httptest
    spring.cloud.dataflow.stream.app.type: source
  deploymentProperties:
    spring.cloud.deployer.group: httptest

The majority of the deployment and application properties were set by Data Flow in order to enable the applications to talk to each other and sending application metrics with identifying labels.

27.5 Rollback a Stream

You can rollback to a previous version of the Stream using the command stream skipper rollback.

dataflow:>stream skipper rollback --name httptest

There is an optional --releaseVersion command argument which is the version of the Stream. If not specified, the rollback goes to the previous stream version.

27.6 Application Count

The application count is a dynamic property of the system. If due to scaling at runtime, the application to be upgraded has 5 instances running, then 5 instances of the upgraded application will be deployed.

27.7 Skipper’s Upgrade Strategy

Skipper has a simple 'red/black' upgrade strategy. It deploys the new version of the applications, as many instances as the currently running version, and checks the /health endpoint of the application. If the health of the new application is good, then the previous application is undeployed. If the health of the new application is bad, then all new applications are undeployed and the upgrade is considered not successful.

The upgrade strategy is not a rolling upgrade, so if 5 applications of the application to upgrade are runningn, then in a sunny day scenario, 5 of the new applications will also be running before the older version is undeployed. Future versions of Skipper will support rolling upgrades and other types of checks, e.g. manual, to continue to upgrade process.