Spring Cloud Data Flow Server for Cloud Foundry
   Next

Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.3.0.M1

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting started
1. Deploying on Cloud Foundry
1.1. Provision a Redis service instance on Cloud Foundry
1.2. Provision a Rabbit service instance on Cloud Foundry
1.3. Provision a MySQL service instance on Cloud Foundry
1.4. Download the Spring Cloud Data Flow Server and Shell apps
1.5. Running the Server
1.5.1. Deploying and Running the Server app on Cloud Foundry
Configuring Defaults for Deployed Apps
1.5.2. Running the Server app locally
1.5.3. Sample Manifest Template
1.6. Running Spring Cloud Data Flow Shell locally
2. Application Names and Prefixes
2.1. Using Custom Routes
3. Deploying Docker Applications
4. Application Level Service Bindings
5. A Note About User Provided Services
6. Application Rolling Upgrades
7. Maximum Disk Quota Configuration
7.1. PCF’s Operations Manager Configuration
7.2. Scale Application
7.3. Configuring target free disk percentage
8. Application Resolution Alternatives
II. Applications
III. Architecture
9. Introduction
10. Microservice Architectural Style
10.1. Comparison to other Platform architectures
11. Streaming Applications
11.1. Imperative Programming Model
11.2. Functional Programming Model
12. Streams
12.1. Topologies
12.2. Concurrency
12.3. Partitioning
12.4. Message Delivery Guarantees
13. Analytics
14. Task Applications
15. Data Flow Server
15.1. Endpoints
15.2. Customization
15.3. Security
16. Runtime
16.1. Fault Tolerance
16.2. Resource Management
16.3. Scaling at runtime
16.4. Application Versioning
IV. Server Configuration
17. Feature Toggles
18. Database Configuration
18.1. Adding a custom JDBC driver
19. Security
19.1. Enabling HTTPS
19.1.1. Using Self-Signed Certificates
19.1.2. Self-Signed Certificates and the Shell
19.2. Traditional Authentication
19.2.1. Single User Authentication
19.2.2. File based authentication
19.2.3. LDAP Authentication
LDAP Transport Security
19.2.4. Shell Authentication
19.2.5. Customizing authorization
19.2.6. Authorization - Shell and Dashboard Behavior
19.2.7. Authorization with Ldap
19.3. OAuth 2.0
19.3.1. OAuth REST Endpoint Authorization
19.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
19.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
19.4. Securing the Spring Boot Management Endpoints
20. Monitoring and Management
20.1. Spring Boot Admin
20.2. Monitoring Deployed Applications
20.3. Log and DataDog MetricWriter
V. Shell
21. Shell Options
22. Listing available commands
23. Tab Completion
24. White space and quote rules
24.1. Quotes and Escaping
24.1.1. Shell rules
24.1.2. DSL parsing rules
24.1.3. SpEL syntax and SpEL literals
24.1.4. Putting it all together
VI. Streams
25. Introduction
25.1. Stream Pipeline DSL
25.2. Application properties
26. Lifecycle of Streams
26.1. Register a Stream App
26.1.1. Whitelisting application properties
26.1.2. Creating and using a dedicated metadata artifact
26.1.3. Using the companion artifact
26.2. Creating custom applications
26.3. Creating a Stream
26.3.1. Application properties
Passing application properties when creating a stream
26.3.2. Deployment properties
Application properties versus Deployer properties
Passing instance count as deployment property
Inline vs file reference properties
Passing application properties when deploying a stream
Passing Spring Cloud Stream properties for the application
Passing per-binding producer consumer properties
Passing stream partition properties during stream deployment
Passing application content type properties
Overriding application properties during stream deployment
26.3.3. Common application properties
26.4. Destroying a Stream
26.5. Deploying and Undeploying Streams
27. Stream DSL
27.1. Tap a Stream
27.2. Using Labels in a Stream
27.3. Named Destinations
27.4. Fan-in and Fan-out
28. Stream applications with multiple binder configurations
29. Examples
29.1. Simple Stream Processing
29.2. Stateful Stream Processing
29.3. Other Source and Sink Application Types
VII. Tasks
30. Introduction
31. The Lifecycle of a Task
31.1. Creating a Task Application
31.1.1. Task Database Configuration
31.2. Registering a Task Application
31.3. Creating a Task Definition
31.4. Launching a Task
31.4.1. Common application properties
31.5. Reviewing Task Executions
31.6. Destroying a Task Definition
32. Subscribing to Task/Batch Events
33. Launching Tasks from a Stream
33.1. TriggerTask
33.2. TaskLaunchRequest-transform
34. Composed Tasks
34.1. Configuring the Composed Task Runner
34.1.1. Registering the Composed Task Runner
34.1.2. Configuring the Composed Task Runner
34.2. The Lifecycle of a Composed Task
34.2.1. Creating a Composed Task
Task Application Parameters
34.2.2. Launching a Composed Task
Exit Statuses
34.2.3. Destroying a Composed Task
34.2.4. Stopping a Composed Task
34.2.5. Restarting a Composed Task
35. Composed Tasks DSL
35.1. Conditional Execution
35.2. Transitional Execution
35.2.1. Basic Transition
35.2.2. Transition With a Wildcard
35.2.3. Transition With a Following Conditional Execution
35.3. Split Execution
35.3.1. Split Containing Conditional Execution
VIII. Tasks on Cloud Foundry
36. Version Compatibility
37. Tooling
38. Task Database Schema
39. Running Task Applications
39.1. Create a Task
39.2. Launch a Task
39.3. View Task Logs
39.4. List Tasks
39.5. List Task Executions
39.6. Destroy a Task
39.7. Deleting Task From Cloud Foundry
IX. Dashboard
40. Introduction
41. Apps
41.1. Bulk Import of Applications
42. Runtime
43. Streams
44. Create Stream
45. Tasks
45.1. Apps
45.1.1. Create a Task Definition from a selected Task App
45.1.2. View Task App Details
45.2. Definitions
45.2.1. Creating Task Definitions using the bulk define interface
45.2.2. Creating Composed Task Definitions
45.2.3. Launching Tasks
45.3. Executions
46. Jobs
46.1. List job executions
46.1.1. Job execution details
46.1.2. Step execution details
46.1.3. Step Execution Progress
47. Analytics
X. REST API Guide
XI. Appendices
A. Data Flow Template
A.1. Using the Data Flow Template
B. Spring XD to SCDF
B.1. Terminology Changes
B.2. Modules to Applications
B.2.1. Custom Applications
B.2.2. Application Registration
B.2.3. Application Properties
B.3. Message Bus to Binders
B.3.1. Message Bus
B.3.2. Binders
B.3.3. Named Channels
B.3.4. Directed Graphs
B.4. Batch to Tasks
B.5. Shell/DSL Commands
B.6. REST-API
B.7. UI / Flo
B.8. Architecture Components
B.8.1. ZooKeeper
B.8.2. RDBMS
B.8.3. Redis
B.8.4. Cluster Topology
B.9. Central Configuration
B.10. Distribution
B.11. Hadoop Distribution Compatibility
B.12. YARN Deployment
B.13. Use Case Comparison
B.13.1. Use Case #1
B.13.2. Use Case #2
B.13.3. Use Case #3
C. Building
C.1. Documentation
C.2. Working with the code
C.2.1. Importing into eclipse with m2eclipse
C.2.2. Importing into eclipse without m2eclipse
D. Contributing
D.1. Sign the Contributor License Agreement
D.2. Code Conventions and Housekeeping
   Next
   Part I. Getting started