Spring Cloud Data Flow Server for Kubernetes
		Next

Spring Cloud Data Flow Server for Kubernetes

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Florian Rosenberg, Dave Syer, David Turanski, Janne Valkealahti, Oleg Zhurakousky, Vinicius Carvalho

1.3.0.RC1

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting Started

1. Installation

1.1. Kubernetes Compatibility
1.2. Create a Kubernetes cluster
1.3. Deploying using kubectl
1.4. Deploy Skipper

2. Helm Installation

3. Deploying Streams

3.1. Create Streams without Skipper
3.2. Create Streams with Skipper
3.3. Accessing app from outside the cluster

4. Deploying Tasks

5. Application Configuration

5.1. Memory and CPU Settings
5.2. Environment Variables
5.3. Liveness and Readiness Probes

II. Applications

III. Architecture

6. Introduction

7. Microservice Architectural Style

7.1. Comparison to other Platform architectures

8. Streaming Applications

8.1. Imperative Programming Model
8.2. Functional Programming Model

9.1. Topologies
9.2. Concurrency
9.3. Partitioning
9.4. Message Delivery Guarantees

10. Analytics

11. Task Applications

12. Data Flow Server

12.1. Endpoints
12.2. Customization
12.3. Security

13. Runtime

13.1. Fault Tolerance
13.2. Resource Management
13.3. Scaling at runtime
13.4. Application Versioning

IV. Server Configuration

14. Feature Toggles

15. General Configuration

15.1. Using ConfigMap and Secrets

16. Database Configuration

17. Security

18. Spring Cloud Deployer for Kubernetes Properties

18.1. Using Deployments
18.2. CPU and Memory Limits
18.3. Liveness and Rediness Probes Configurations
18.4. Using SPRING_APPLICATION_JSON

19. Monitoring and Management

19.1. Inspecting Server Logs
19.2. Streams
19.3. Tasks

20. Shell Options

21. Listing available commands

22. Tab Completion

23. White space and quote rules

23.1. Quotes and Escaping

23.1.1. Shell rules
23.1.2. DSL parsing rules
23.1.3. SpEL syntax and SpEL literals
23.1.4. Putting it all together

VI. Streams

24. Introduction

24.1. Stream Pipeline DSL
24.2. Application properties

25. Stream Lifecycle

25.1. Register a Stream App

25.2. Register Supported Applications and Tasks

25.2.1. Whitelisting application properties
25.2.2. Creating and using a dedicated metadata artifact
25.2.3. Using the companion artifact

25.3. Creating custom applications

25.4. Creating a Stream

25.4.1. Application properties
25.4.2. Common application properties

25.5. Deploying a Stream

25.5.1. Deployment properties

Passing instance count
Inline vs file based properties
Passing application properties
Passing Spring Cloud Stream properties
Passing per-binding producer consumer properties
Passing stream partition properties
Passing application content type properties
Overriding application properties during stream deployment

25.6. Destroying a Stream

25.7. Undeploying Streams

26. Stream Lifecycle with Skipper

26.1. Register a Versioned Stream App
26.2. Creating and Deploying a Stream
26.3. Updating a Stream
26.4. Stream versions
26.5. Stream Manifests
26.6. Rollback a Stream
26.7. Application Count
26.8. Skipper’s Upgrade Strategy

27. Stream DSL

27.1. Tap a Stream
27.2. Using Labels in a Stream
27.3. Named Destinations
27.4. Fan-in and Fan-out

28. Stream Java DSL

28.1. Overview
28.2. Java DSL styles
28.3. Using the DeploymentPropertiesBuilder

29. Deploying using Skipper

30. Stream applications with multiple binder configurations

31. Examples

31.1. Simple Stream Processing
31.2. Stateful Stream Processing
31.3. Other Source and Sink Application Types

VII. Streams deployed using Skipper

VIII. Tasks

32. Introduction

33. The Lifecycle of a Task

33.1. Creating a Task Application

33.1.1. Task Database Configuration

33.2. Registering a Task Application

33.3. Creating a Task Definition

33.4. Launching a Task

33.4.1. Common application properties

33.5. Reviewing Task Executions

33.6. Destroying a Task Definition

34. Subscribing to Task/Batch Events

35. Composed Tasks

35.1. Configuring the Composed Task Runner

35.1.1. Registering the Composed Task Runner
35.1.2. Configuring the Composed Task Runner

35.2. The Lifecycle of a Composed Task

35.2.1. Creating a Composed Task

Task Application Parameters

35.2.2. Launching a Composed Task

Exit Statuses

35.2.3. Destroying a Composed Task

35.2.4. Stopping a Composed Task

35.2.5. Restarting a Composed Task

36. Composed Tasks DSL

36.1. Conditional Execution

36.2. Transitional Execution

36.2.1. Basic Transition
36.2.2. Transition With a Wildcard
36.2.3. Transition With a Following Conditional Execution

36.3. Split Execution

36.3.1. Split Containing Conditional Execution

37. Launching Tasks from a Stream

37.1. TriggerTask
37.2. TaskLaunchRequest-transform
37.3. Launching a Composed Task From a Stream

IX. Dashboard

38. Introduction

39.1. Bulk Import of Applications

40. Runtime

41. Streams

42. Create Stream

43.1.1. Create a Task Definition from a selected Task App
43.1.2. View Task App Details

43.2. Definitions

43.2.1. Creating Task Definitions using the bulk define interface
43.2.2. Creating Composed Task Definitions
43.2.3. Launching Tasks

43.3. Executions

44.1. List job executions

44.1.1. Job execution details
44.1.2. Step execution details
44.1.3. Step Execution Progress

45. Analytics

X. REST API Guide

XI. Appendices

A. ‘How-to’ guides

A.1. Logging

A.1.1. Deployment Logs
A.1.2. Application Logs

B. Data Flow Template

B.1. Using the Data Flow Template

C. Spring XD to SCDF

C.1. Terminology Changes

C.2. Modules to Applications

C.2.1. Custom Applications
C.2.2. Application Registration
C.2.3. Application Properties

C.3. Message Bus to Binders

C.3.1. Message Bus
C.3.2. Binders
C.3.3. Named Channels
C.3.4. Directed Graphs

C.4. Batch to Tasks

C.5. Shell/DSL Commands

C.6. REST-API

C.7. UI / Flo

C.8. Architecture Components

C.8.1. ZooKeeper
C.8.2. RDBMS
C.8.3. Redis
C.8.4. Cluster Topology

C.9. Central Configuration

C.10. Distribution

C.11. Hadoop Distribution Compatibility

C.12. YARN Deployment

C.13. Use Case Comparison

C.13.1. Use Case #1
C.13.2. Use Case #2
C.13.3. Use Case #3

D. Building

D.1. Documentation

D.2. Working with the code

D.2.1. Importing into eclipse with m2eclipse
D.2.2. Importing into eclipse without m2eclipse

E. Contributing

E.1. Sign the Contributor License Agreement
E.2. Code Conventions and Housekeeping

		Next
		Part I. Getting Started