Spring Cloud Data Flow Server for Kubernetes
		Next

Spring Cloud Data Flow Server for Kubernetes

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Florian Rosenberg, Dave Syer, David Turanski, Janne Valkealahti, Oleg Zhurakousky

1.3.0.M3

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting Started

1. Installation

1.1. Create a Kubernetes cluster
1.2. Deploying using kubectl
1.3. Deploy Skipper

2. Helm Installation

3. Deploying Streams

3.1. Create Streams without Skipper
3.2. Create Streams with Skipper
3.3. Accessing app from outside the cluster

4. Deploying Tasks

5. Application Configuration

5.1. Memory and CPU Settings
5.2. Environment Variables
5.3. Liveness and Readiness Probes

II. Applications

III. Architecture

6. Introduction

7. Microservice Architectural Style

7.1. Comparison to other Platform architectures

8. Streaming Applications

8.1. Imperative Programming Model
8.2. Functional Programming Model

9.1. Topologies
9.2. Concurrency
9.3. Partitioning
9.4. Message Delivery Guarantees

10. Analytics

11. Task Applications

12. Data Flow Server

12.1. Endpoints
12.2. Customization
12.3. Security

13. Runtime

13.1. Fault Tolerance
13.2. Resource Management
13.3. Scaling at runtime
13.4. Application Versioning

IV. Server Configuration

14. Feature Toggles

15. General Configuration

15.1. Using ConfigMap and Secrets

16. Database Configuration

17. Security

18. Spring Cloud Deployer for Kubernetes Properties

18.1. Using Deployments
18.2. CPU and Memory Limits
18.3. Liveness and Rediness Probes Configurations
18.4. Using SPRING_APPLICATION_JSON

19. Monitoring and Management

19.1. Inspecting Server Logs
19.2. Streams
19.3. Tasks

20. Shell Options

21. Listing available commands

22. Tab Completion

23. White space and quote rules

23.1. Quotes and Escaping

23.1.1. Shell rules
23.1.2. DSL parsing rules
23.1.3. SpEL syntax and SpEL literals
23.1.4. Putting it all together

VI. Streams

24. Introduction

24.1. Stream Pipeline DSL
24.2. Application properties

25. Stream Lifecycle

25.1. Register a Stream App

25.2. Register Supported Applications and Tasks

25.2.1. Whitelisting application properties
25.2.2. Creating and using a dedicated metadata artifact
25.2.3. Using the companion artifact

25.3. Creating custom applications

25.4. Creating a Stream

25.4.1. Application properties
25.4.2. Common application properties

25.5. Deploying a Stream

25.5.1. Deployment properties

Passing instance count
Inline vs file based properties
Passing application properties
Passing Spring Cloud Stream properties
Passing per-binding producer consumer properties
Passing stream partition properties
Passing application content type properties
Overriding application properties during stream deployment

25.6. Destroying a Stream

25.7. Undeploying Streams

26. Stream Lifecycle with Skipper

26.1. Creating and Deploying a Stream
26.2. Updating a Stream
26.3. Stream versions
26.4. Stream Manifests
26.5. Rollback a Stream
26.6. Application Count
26.7. Skipper’s Upgrade Strategy

27. Stream DSL

27.1. Tap a Stream
27.2. Using Labels in a Stream
27.3. Named Destinations
27.4. Fan-in and Fan-out

28. Stream Java DSL

28.1. Overview
28.2. Java DSL styles

29. Stream applications with multiple binder configurations

30. Examples

30.1. Simple Stream Processing
30.2. Stateful Stream Processing
30.3. Other Source and Sink Application Types

VII. Streams deployed using Skipper

VIII. Tasks

31. Introduction

32. The Lifecycle of a Task

32.1. Creating a Task Application

32.1.1. Task Database Configuration

32.2. Registering a Task Application

32.3. Creating a Task Definition

32.4. Launching a Task

32.4.1. Common application properties

32.5. Reviewing Task Executions

32.6. Destroying a Task Definition

33. Subscribing to Task/Batch Events

34. Composed Tasks

34.1. Configuring the Composed Task Runner

34.1.1. Registering the Composed Task Runner
34.1.2. Configuring the Composed Task Runner

34.2. The Lifecycle of a Composed Task

34.2.1. Creating a Composed Task

Task Application Parameters

34.2.2. Launching a Composed Task

Exit Statuses

34.2.3. Destroying a Composed Task

34.2.4. Stopping a Composed Task

34.2.5. Restarting a Composed Task

35. Composed Tasks DSL

35.1. Conditional Execution

35.2. Transitional Execution

35.2.1. Basic Transition
35.2.2. Transition With a Wildcard
35.2.3. Transition With a Following Conditional Execution

35.3. Split Execution

35.3.1. Split Containing Conditional Execution

36. Launching Tasks from a Stream

36.1. TriggerTask
36.2. TaskLaunchRequest-transform
36.3. Launching a Composed Task From a Stream

IX. Dashboard

37. Introduction

38.1. Bulk Import of Applications

39. Runtime

40. Streams

41. Create Stream

42.1.1. Create a Task Definition from a selected Task App
42.1.2. View Task App Details

42.2. Definitions

42.2.1. Creating Task Definitions using the bulk define interface
42.2.2. Creating Composed Task Definitions
42.2.3. Launching Tasks

42.3. Executions

43.1. List job executions

43.1.1. Job execution details
43.1.2. Step execution details
43.1.3. Step Execution Progress

44. Analytics

X. REST API Guide

XI. Appendices

A. ‘How-to’ guides

A.1. Logging

A.1.1. Deployment Logs
A.1.2. Application Logs

B. Data Flow Template

B.1. Using the Data Flow Template

C. Spring XD to SCDF

C.1. Terminology Changes

C.2. Modules to Applications

C.2.1. Custom Applications
C.2.2. Application Registration
C.2.3. Application Properties

C.3. Message Bus to Binders

C.3.1. Message Bus
C.3.2. Binders
C.3.3. Named Channels
C.3.4. Directed Graphs

C.4. Batch to Tasks

C.5. Shell/DSL Commands

C.6. REST-API

C.7. UI / Flo

C.8. Architecture Components

C.8.1. ZooKeeper
C.8.2. RDBMS
C.8.3. Redis
C.8.4. Cluster Topology

C.9. Central Configuration

C.10. Distribution

C.11. Hadoop Distribution Compatibility

C.12. YARN Deployment

C.13. Use Case Comparison

C.13.1. Use Case #1
C.13.2. Use Case #2
C.13.3. Use Case #3

D. Building

D.1. Documentation

D.2. Working with the code

D.2.1. Importing into eclipse with m2eclipse
D.2.2. Importing into eclipse without m2eclipse

E. Contributing

E.1. Sign the Contributor License Agreement
E.2. Code Conventions and Housekeeping

		Next
		Part I. Getting Started