Spring Cloud Data Flow Server for Cloud Foundry
   Next

Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.2.4.RELEASE

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting started
1. Deploying on Cloud Foundry
1.1. Provision a Redis service instance on Cloud Foundry
1.2. Provision a Rabbit service instance on Cloud Foundry
1.3. Provision a MySQL service instance on Cloud Foundry
1.4. Download the Spring Cloud Data Flow Server and Shell apps
1.5. Running the Server
1.5.1. Deploying and Running the Server app on Cloud Foundry
Configuring Defaults for Deployed Apps
1.5.2. Running the Server app locally
1.5.3. Sample Manifest Template
1.6. Running Spring Cloud Data Flow Shell locally
2. Application Names and Prefixes
2.1. Using Custom Routes
3. Deploying Docker Applications
4. Application Level Service Bindings
5. A Note About User Provided Services
6. Application Rolling Upgrades
7. Maximum Disk Quota Configuration
7.1. PCF’s Operations Manager Configuration
7.2. Scale Application
7.3. Configuring target free disk percentage
8. Application Resolution Alternatives
II. Applications
III. Architecture
9. Introduction
10. Microservice Architectural Style
10.1. Comparison to other Platform architectures
11. Streaming Applications
11.1. Imperative Programming Model
11.2. Functional Programming Model
12. Streams
12.1. Topologies
12.2. Concurrency
12.3. Partitioning
12.4. Message Delivery Guarantees
13. Analytics
14. Task Applications
15. Data Flow Server
15.1. Endpoints
15.2. Customization
15.3. Security
16. Runtime
16.1. Fault Tolerance
16.2. Resource Management
16.3. Scaling at runtime
16.4. Application Versioning
IV. Server Configuration
17. Feature Toggles
18. Security
18.1. Authentication and Cloud Foundry
18.1.1. Pivotal Single Sign-On Service
18.1.2. Cloud Foundry UAA
19. Configuration Reference
19.1. Understanding what’s going on
19.2. Using Spring Cloud Config Server
19.2.1. Stream, Task, and Spring Cloud Config Server
19.2.2. Sample Manifest Template
19.2.3. Self-signed SSL Certificate and Spring Cloud Config Server
V. Shell
20. Shell Options
21. Listing available commands
22. Tab Completion
23. White space and quote rules
23.1. Quotes and Escaping
23.1.1. Shell rules
23.1.2. DSL parsing rules
23.1.3. SpEL syntax and SpEL literals
23.1.4. Putting it all together
VI. Streams
24. Introduction
24.1. Stream Pipeline DSL
24.2. Application properties
25. Lifecycle of Streams
25.1. Register a Stream App
25.1.1. Whitelisting application properties
25.1.2. Creating and using a dedicated metadata artifact
25.1.3. Using the companion artifact
25.2. Creating custom applications
25.3. Creating a Stream
25.3.1. Application properties
Passing application properties when creating a stream
25.3.2. Deployment properties
Application properties versus Deployer properties
Passing instance count as deployment property
Inline vs file reference properties
Passing application properties when deploying a stream
Passing Spring Cloud Stream properties for the application
Passing per-binding producer consumer properties
Passing stream partition properties during stream deployment
Passing application content type properties
Overriding application properties during stream deployment
25.3.3. Common application properties
25.4. Destroying a Stream
25.5. Deploying and Undeploying Streams
26. Stream DSL
26.1. Tap a Stream
26.2. Using Labels in a Stream
26.3. Named Destinations
26.4. Fan-in and Fan-out
27. Stream applications with multiple binder configurations
28. Examples
28.1. Simple Stream Processing
28.2. Stateful Stream Processing
28.3. Other Source and Sink Application Types
VII. Tasks
29. Introduction
30. The Lifecycle of a Task
30.1. Creating a Task Application
30.1.1. Task Database Configuration
30.2. Registering a Task Application
30.3. Creating a Task Definition
30.4. Launching a Task
30.4.1. Common application properties
30.5. Reviewing Task Executions
30.6. Destroying a Task Definition
31. Subscribing to Task/Batch Events
32. Launching Tasks from a Stream
32.1. TriggerTask
32.2. TaskLaunchRequest-transform
33. Composed Tasks
33.1. Configuring the Composed Task Runner
33.1.1. Registering the Composed Task Runner
33.1.2. Configuring the Composed Task Runner
33.2. The Lifecycle of a Composed Task
33.2.1. Creating a Composed Task
Task Application Parameters
33.2.2. Launching a Composed Task
Exit Statuses
33.2.3. Destroying a Composed Task
33.2.4. Stopping a Composed Task
33.2.5. Restarting a Composed Task
34. Composed Tasks DSL
34.1. Conditional Execution
34.2. Transitional Execution
34.2.1. Basic Transition
34.2.2. Transition With a Wildcard
34.2.3. Transition With a Following Conditional Execution
34.3. Split Execution
34.3.1. Split Containing Conditional Execution
VIII. Tasks on Cloud Foundry
35. Version Compatibility
36. Tooling
37. Task Database Schema
38. Running Task Applications
38.1. Create a Task
38.2. Launch a Task
38.3. View Task Logs
38.4. List Tasks
38.5. List Task Executions
38.6. Destroy a Task
38.7. Deleting Task From Cloud Foundry
IX. Dashboard
39. Introduction
40. Apps
40.1. Bulk Import of Applications
41. Runtime
42. Streams
43. Create Stream
44. Tasks
44.1. Apps
44.1.1. Create a Task Definition from a selected Task App
44.1.2. View Task App Details
44.2. Definitions
44.2.1. Creating Task Definitions using the bulk define interface
44.2.2. Creating Composed Task Definitions
44.2.3. Launching Tasks
44.3. Executions
45. Jobs
45.1. List job executions
45.1.1. Job execution details
45.1.2. Step execution details
45.1.3. Step Execution Progress
46. Analytics
X. REST API Guide
XI. Appendices
A. Data Flow Template
A.1. Using the Data Flow Template
B. Spring XD to SCDF
B.1. Terminology Changes
B.2. Modules to Applications
B.2.1. Custom Applications
B.2.2. Application Registration
B.2.3. Application Properties
B.3. Message Bus to Binders
B.3.1. Message Bus
B.3.2. Binders
B.3.3. Named Channels
B.3.4. Directed Graphs
B.4. Batch to Tasks
B.5. Shell/DSL Commands
B.6. REST-API
B.7. UI / Flo
B.8. Architecture Components
B.8.1. ZooKeeper
B.8.2. RDBMS
B.8.3. Redis
B.8.4. Cluster Topology
B.9. Central Configuration
B.10. Distribution
B.11. Hadoop Distribution Compatibility
B.12. YARN Deployment
B.13. Use Case Comparison
B.13.1. Use Case #1
B.13.2. Use Case #2
B.13.3. Use Case #3
C. Building
C.1. Documentation
C.2. Working with the code
C.2.1. Importing into eclipse with m2eclipse
C.2.2. Importing into eclipse without m2eclipse
D. Contributing
D.1. Sign the Contributor License Agreement
D.2. Code Conventions and Housekeeping
   Next
   Part I. Getting started