Spring Cloud Data Flow Server for Cloud Foundry
   Next

Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.3.0.BUILD-SNAPSHOT

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting started
1. Deploying on Cloud Foundry
1.1. Provision a Redis service instance on Cloud Foundry
1.2. Provision a Rabbit service instance on Cloud Foundry
1.3. Provision a MySQL service instance on Cloud Foundry
1.4. Running the Data Flow Server
1.4.1. Deploying and Running the Server app on Cloud Foundry
Sample Manifest Template
Configuring Defaults for Deployed Apps
1.4.2. Running the Server app locally
1.5. Running Spring Cloud Data Flow Shell locally
2. Spring Cloud Skipper Integration
2.1. Download the Spring Cloud Skipper and Shell apps
2.2. Running the Skipper Server
3. Application Names and Prefixes
3.1. Using Custom Routes
4. Deploying Docker Applications
5. Application Level Service Bindings
6. A Note About User Provided Services
7. Application Rolling Upgrades
8. Maximum Disk Quota Configuration
8.1. PCF’s Operations Manager Configuration
8.2. Scale Application
8.3. Configuring target free disk percentage
9. Application Resolution Alternatives
II. Applications
III. Architecture
10. Introduction
11. Microservice Architectural Style
11.1. Comparison to other Platform architectures
12. Streaming Applications
12.1. Imperative Programming Model
12.2. Functional Programming Model
13. Streams
13.1. Topologies
13.2. Concurrency
13.3. Partitioning
13.4. Message Delivery Guarantees
14. Analytics
15. Task Applications
16. Data Flow Server
16.1. Endpoints
16.2. Customization
16.3. Security
17. Runtime
17.1. Fault Tolerance
17.2. Resource Management
17.3. Scaling at runtime
17.4. Application Versioning
IV. Server Configuration
18. Feature Toggles
19. Security
19.1. Authentication and Cloud Foundry
19.1.1. Pivotal Single Sign-On Service
19.1.2. Cloud Foundry UAA
20. Configuration Reference
20.1. Understanding what’s going on
20.2. Using Spring Cloud Config Server
20.2.1. Stream, Task, and Spring Cloud Config Server
20.2.2. Sample Manifest Template
20.2.3. Self-signed SSL Certificate and Spring Cloud Config Server
V. Shell
21. Shell Options
22. Listing available commands
23. Tab Completion
24. White space and quote rules
24.1. Quotes and Escaping
24.1.1. Shell rules
24.1.2. DSL parsing rules
24.1.3. SpEL syntax and SpEL literals
24.1.4. Putting it all together
VI. Streams
25. Introduction
25.1. Stream Pipeline DSL
25.2. Application properties
26. Stream Lifecycle
26.1. Register a Stream App
26.2. Register Supported Applications and Tasks
26.2.1. Whitelisting application properties
26.2.2. Creating and using a dedicated metadata artifact
26.2.3. Using the companion artifact
26.3. Creating custom applications
26.4. Creating a Stream
26.4.1. Application properties
26.4.2. Common application properties
26.5. Deploying a Stream
26.5.1. Deployment properties
Passing instance count
Inline vs file based properties
Passing application properties
Passing Spring Cloud Stream properties
Passing per-binding producer consumer properties
Passing stream partition properties
Passing application content type properties
Overriding application properties during stream deployment
26.6. Destroying a Stream
26.7. Undeploying Streams
27. Stream Lifecycle with Skipper
27.1. Creating and Deploying a Stream
27.2. Updating a Stream
27.3. Stream versions
27.4. Stream Manifests
27.5. Rollback a Stream
27.6. Application Count
27.7. Skipper’s Upgrade Strategy
28. Stream DSL
28.1. Tap a Stream
28.2. Using Labels in a Stream
28.3. Named Destinations
28.4. Fan-in and Fan-out
29. Stream Java DSL
29.1. Overview
29.2. Java DSL styles
30. Stream applications with multiple binder configurations
31. Examples
31.1. Simple Stream Processing
31.2. Stateful Stream Processing
31.3. Other Source and Sink Application Types
VII. Streams deployed using Skipper
VIII. Tasks
32. Introduction
33. The Lifecycle of a Task
33.1. Creating a Task Application
33.1.1. Task Database Configuration
33.2. Registering a Task Application
33.3. Creating a Task Definition
33.4. Launching a Task
33.4.1. Common application properties
33.5. Reviewing Task Executions
33.6. Destroying a Task Definition
34. Subscribing to Task/Batch Events
35. Composed Tasks
35.1. Configuring the Composed Task Runner
35.1.1. Registering the Composed Task Runner
35.1.2. Configuring the Composed Task Runner
35.2. The Lifecycle of a Composed Task
35.2.1. Creating a Composed Task
Task Application Parameters
35.2.2. Launching a Composed Task
Exit Statuses
35.2.3. Destroying a Composed Task
35.2.4. Stopping a Composed Task
35.2.5. Restarting a Composed Task
36. Composed Tasks DSL
36.1. Conditional Execution
36.2. Transitional Execution
36.2.1. Basic Transition
36.2.2. Transition With a Wildcard
36.2.3. Transition With a Following Conditional Execution
36.3. Split Execution
36.3.1. Split Containing Conditional Execution
37. Launching Tasks from a Stream
37.1. TriggerTask
37.2. TaskLaunchRequest-transform
37.3. Launching a Composed Task From a Stream
IX. Tasks on Cloud Foundry
38. Version Compatibility
39. Tooling
40. Task Database Schema
41. Running Task Applications
41.1. Create a Task
41.2. Launch a Task
41.3. View Task Logs
41.4. List Tasks
41.5. List Task Executions
41.6. Destroy a Task
41.7. Deleting Task From Cloud Foundry
X. Dashboard
42. Introduction
43. Apps
43.1. Bulk Import of Applications
44. Runtime
45. Streams
46. Create Stream
47. Tasks
47.1. Apps
47.1.1. Create a Task Definition from a selected Task App
47.1.2. View Task App Details
47.2. Definitions
47.2.1. Creating Task Definitions using the bulk define interface
47.2.2. Creating Composed Task Definitions
47.2.3. Launching Tasks
47.3. Executions
48. Jobs
48.1. List job executions
48.1.1. Job execution details
48.1.2. Step execution details
48.1.3. Step Execution Progress
49. Analytics
XI. REST API Guide
XII. Appendices
A. Data Flow Template
A.1. Using the Data Flow Template
B. Spring XD to SCDF
B.1. Terminology Changes
B.2. Modules to Applications
B.2.1. Custom Applications
B.2.2. Application Registration
B.2.3. Application Properties
B.3. Message Bus to Binders
B.3.1. Message Bus
B.3.2. Binders
B.3.3. Named Channels
B.3.4. Directed Graphs
B.4. Batch to Tasks
B.5. Shell/DSL Commands
B.6. REST-API
B.7. UI / Flo
B.8. Architecture Components
B.8.1. ZooKeeper
B.8.2. RDBMS
B.8.3. Redis
B.8.4. Cluster Topology
B.9. Central Configuration
B.10. Distribution
B.11. Hadoop Distribution Compatibility
B.12. YARN Deployment
B.13. Use Case Comparison
B.13.1. Use Case #1
B.13.2. Use Case #2
B.13.3. Use Case #3
C. Building
C.1. Documentation
C.2. Working with the code
C.2.1. Importing into eclipse with m2eclipse
C.2.2. Importing into eclipse without m2eclipse
D. Contributing
D.1. Sign the Contributor License Agreement
D.2. Code Conventions and Housekeeping
   Next
   Part I. Getting started