Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti, Oleg Zhurakousky

1.2.3.RELEASE

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Getting started
1. System Requirements
2. Deploying Spring Cloud Data Flow Local Server
2.1. Maven Configuration
3. Application Configuration
II. Applications
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Server Configuration
12. Feature Toggles
13. Database Configuration
13.1. Adding a custom JDBC driver
14. Security
14.1. Enabling HTTPS
14.1.1. Using Self-Signed Certificates
14.1.2. Self-Signed Certificates and the Shell
14.2. Traditional Authentication
14.2.1. Single User Authentication
14.2.2. File based authentication
14.2.3. LDAP Authentication
LDAP Transport Security
14.2.4. Shell Authentication
14.2.5. Customizing authorization
14.2.6. Authorization - Shell and Dashboard Behavior
14.2.7. Authorization with Ldap
14.3. OAuth 2.0
14.3.1. OAuth REST Endpoint Authorization
14.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
14.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
14.4. Securing the Spring Boot Management Endpoints
15. Monitoring and Management
15.1. Spring Boot Admin
15.2. Monitoring Deployed Applications
15.3. Log and DataDog MetricWriter
V. Shell
16. Shell Options
17. Listing available commands
18. Tab Completion
19. White space and quote rules
19.1. Quotes and Escaping
19.1.1. Shell rules
19.1.2. DSL parsing rules
19.1.3. SpEL syntax and SpEL literals
19.1.4. Putting it all together
VI. Streams
20. Introduction
20.1. Stream Pipeline DSL
20.2. Application properties
21. Lifecycle of Streams
21.1. Register a Stream App
21.1.1. Whitelisting application properties
21.1.2. Creating and using a dedicated metadata artifact
21.1.3. Using the companion artifact
21.2. Creating custom applications
21.3. Creating a Stream
21.3.1. Application properties
Passing application properties when creating a stream
21.3.2. Deployment properties
Application properties versus Deployer properties
Passing instance count as deployment property
Inline vs file reference properties
Passing application properties when deploying a stream
Passing Spring Cloud Stream properties for the application
Passing per-binding producer consumer properties
Passing stream partition properties during stream deployment
Passing application content type properties
Overriding application properties during stream deployment
21.3.3. Common application properties
21.4. Destroying a Stream
21.5. Deploying and Undeploying Streams
22. Stream DSL
22.1. Tap a Stream
22.2. Using Labels in a Stream
22.3. Named Destinations
22.4. Fan-in and Fan-out
23. Stream applications with multiple binder configurations
24. Examples
24.1. Simple Stream Processing
24.2. Stateful Stream Processing
24.3. Other Source and Sink Application Types
VII. Tasks
25. Introduction
26. The Lifecycle of a Task
26.1. Creating a Task Application
26.1.1. Task Database Configuration
26.2. Registering a Task Application
26.3. Creating a Task Definition
26.4. Launching a Task
26.4.1. Common application properties
26.5. Reviewing Task Executions
26.6. Destroying a Task Definition
27. Subscribing to Task/Batch Events
28. Launching Tasks from a Stream
28.1. TriggerTask
28.2. TaskLaunchRequest-transform
29. Composed Tasks
29.1. Configuring the Composed Task Runner
29.1.1. Registering the Composed Task Runner
29.1.2. Configuring the Composed Task Runner
29.2. The Lifecycle of a Composed Task
29.2.1. Creating a Composed Task
Task Application Parameters
29.2.2. Launching a Composed Task
Exit Statuses
29.2.3. Destroying a Composed Task
29.2.4. Stopping a Composed Task
29.2.5. Restarting a Composed Task
30. Composed Tasks DSL
30.1. Conditional Execution
30.2. Transitional Execution
30.2.1. Basic Transition
30.2.2. Transition With a Wildcard
30.2.3. Transition With a Following Conditional Execution
30.3. Split Execution
30.3.1. Split Containing Conditional Execution
VIII. Dashboard
31. Introduction
32. Apps
32.1. Bulk Import of Applications
33. Runtime
34. Streams
35. Create Stream
36. Tasks
36.1. Apps
36.1.1. Create a Task Definition from a selected Task App
36.1.2. View Task App Details
36.2. Definitions
36.2.1. Creating Task Definitions using the bulk define interface
36.2.2. Creating Composed Task Definitions
36.2.3. Launching Tasks
36.3. Executions
37. Jobs
37.1. List job executions
37.1.1. Job execution details
37.1.2. Step execution details
37.1.3. Step Execution Progress
38. Analytics
IX. REST API Guide
39. Overview
39.1. HTTP verbs
39.2. HTTP status codes
39.3. Headers
39.4. Errors
39.5. Hypermedia
40. Resources
40.1. Index
40.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Links
40.2. Server Meta Information
40.2.1. Retrieving information about the server
Request structure
Example request
Response structure
40.3. Registered Applications
40.3.1. Listing Applications
Request structure
Request parameters
Example request
Response structure
40.3.2. Getting Information on a partical Application
Request structure
Path parameters
Example request
Response structure
40.3.3. Registering a New Application
Request structure
Request parameters
Path parameters
Example request
Response structure
40.3.4. Unregistering an Application
Request structure
Path parameters
Example request
Response structure
40.3.5. Registering Applications in Bulk
Request structure
Request parameters
Example request
Response structure
40.4. Stream Definitions
40.4.1. Creating a new Stream Definition
Request structure
Request parameters
Example request
Response structure
40.4.2. List all Stream Definitions
Request structure
Request parameters
Example request
Response structure
40.4.3. List related Stream Definitions
Request structure
Request parameters
Example request
Response structure
40.4.4. Delete a single Stream Definition
Request structure
Request parameters
Example request
Response structure
40.4.5. Delete all Stream Definitions
Request structure
Request parameters
Example request
Response structure
40.5. Stream Deployments
40.5.1. Example "stream deploy" request for a ticktock stream
40.6. Task Definitions
40.6.1. Creating a new Task Definition
Request structure
Request parameters
Example request
Response structure
40.6.2. List all Task Definitions
Request structure
Request parameters
Example request
Response structure
40.6.3. Retrieve Task Definition Detail
Request structure
Request parameters
Example request
Response structure
40.6.4. Delete Task Definition
Request structure
Request parameters
Example request
Response structure
40.7. Task Executions
40.7.1. Launching a Task
Request structure
Request parameters
Example request
Response structure
40.7.2. List All Task Executions
Request structure
Request parameters
Example request
Response structure
40.7.3. List All Task Executions With a Specified Task Name
Request structure
Request parameters
Example request
Response structure
40.7.4. Task Execution Detail
Request structure
Request parameters
Example request
Response structure
40.7.5. Delete Task Execution
Request structure
Request parameters
Example request
Response structure
40.8. Runtime Information about Applications
40.8.1. Listing All Applications at Runtime
Request structure
Example request
Response structure
40.8.2. Querying all Instances of a Single App
Request structure
Example request
Response structure
40.8.3. Querying a Single Instance of a Single App
Request structure
Example request
Response structure
40.9. Metrics for Stream Applications
40.9.1. Request structure
40.9.2. Example request
40.9.3. Response structure
40.9.4. Example response
X. Appendices
A. Data Flow Template
A.1. Using the Data Flow Template
B. ‘How-to’ guides
B.1. Configure Maven Properties
B.2. Logging
B.3. Deployment Logs
B.4. Application Logs
B.4.1. Log redirect
B.5. Frequently asked questions
B.5.1. Advanced SpEL expressions
B.5.2. How to use JDBC-sink?
B.5.3. How to use multiple message-binders?
C. Spring XD to SCDF
C.1. Terminology Changes
C.2. Modules to Applications
C.2.1. Custom Applications
C.2.2. Application Registration
C.2.3. Application Properties
C.3. Message Bus to Binders
C.3.1. Message Bus
C.3.2. Binders
C.3.3. Named Channels
C.3.4. Directed Graphs
C.4. Batch to Tasks
C.5. Shell/DSL Commands
C.6. REST-API
C.7. UI / Flo
C.8. Architecture Components
C.8.1. ZooKeeper
C.8.2. RDBMS
C.8.3. Redis
C.8.4. Cluster Topology
C.9. Central Configuration
C.10. Distribution
C.11. Hadoop Distribution Compatibility
C.12. YARN Deployment
C.13. Use Case Comparison
C.13.1. Use Case #1
C.13.2. Use Case #2
C.13.3. Use Case #3
D. Building
D.1. Documentation
D.2. Working with the code
D.2.1. Importing into eclipse with m2eclipse
D.2.2. Importing into eclipse without m2eclipse
E. Contributing
E.1. Sign the Contributor License Agreement
E.2. Code Conventions and Housekeeping
   Next
   Part I. Getting started