Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.2.2.RELEASE

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Deploying Spring Cloud Data Flow Local Server
13.1. Maven Configuration
14. Application Configuration
V. Server Configuration
15. Shell Options
16. Feature Toggles
17. Database Configuration
18. Security
18.1. Enabling HTTPS
18.1.1. Using Self-Signed Certificates
18.1.2. Self-Signed Certificates and the Shell
18.2. Traditional Authentication
18.2.1. Single User Authentication
18.2.2. File based authentication
18.2.3. LDAP Authentication
LDAP Transport Security
18.2.4. Shell Authentication
18.2.5. Customizing authorization
18.2.6. Authorization - Shell and Dashboard Behavior
18.2.7. Authorization with Ldap
18.3. OAuth 2.0
18.3.1. OAuth REST Endpoint Authorization
18.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
18.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
18.4. Securing the Spring Boot Management Endpoints
19. Monitoring and Management
19.1. Spring Boot Admin
19.2. Monitoring Deployed Applications
19.3. Log and DataDog MetricWriter
VI. Streams
20. Introduction
21. Stream DSL
22. Register a Stream App
22.1. Whitelisting application properties
22.2. Creating and using a dedicated metadata artifact
22.2.1. Using the companion artifact
23. Creating custom applications
24. Creating a Stream
24.1. Application properties
24.1.1. Passing application properties when creating a stream
24.2. Deployment properties
24.2.1. Application properties versus Deployer properties
24.2.2. Passing instance count as deployment property
24.2.3. Inline vs file reference properties
24.2.4. Passing application properties when deploying a stream
24.2.5. Passing Spring Cloud Stream properties for the application
24.2.6. Passing per-binding producer consumer properties
24.2.7. Passing stream partition properties during stream deployment
24.2.8. Passing application content type properties
24.2.9. Overriding application properties during stream deployment
24.3. Common application properties
25. Destroying a Stream
26. Deploying and Undeploying Streams
27. Other Source and Sink Application Types
28. Simple Stream Processing
29. Stateful Stream Processing
30. Tap a Stream
31. Using Labels in a Stream
32. Explicit Broker Destinations in a Stream
33. Directed Graphs in a Stream
34. Stream applications with multiple binder configurations
VII. Tasks
35. Introducing Spring Cloud Task
36. The Lifecycle of a task
36.1. Creating a custom Task Application
36.2. Registering a Task Application
36.3. Creating a Task
36.4. Launching a Task
36.4.1. Common application properties
36.5. Reviewing Task Executions
36.6. Destroying a Task
37. Task Repository
37.1. Configuring the Task Execution Repository
37.1.1. Local
37.1.2. Task Application Repository
37.2. Datasource
38. Subscribing to Task/Batch Events
39. Launching Tasks from a Stream
39.1. TriggerTask
39.2. TaskLaunchRequest-transform
40. Composed Tasks
40.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
40.1.1. Registering the Composed Task Runner application
40.1.2. Configuring the Composed Task Runner application
40.2. Creating, Launching, and Destroying a Composed Task
40.2.1. Creating a Composed Task
Task Application Parameters
40.2.2. Launching a Composed Task
Exit Statuses
40.2.3. Destroying a Composed Task
40.2.4. Stopping a Composed Task
40.2.5. Restarting a Composed Task
40.3. Composed Task DSL
40.3.1. Conditional Execution
40.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
40.3.3. Split Execution
Split Containing Conditional Execution
VIII. Dashboard
41. Introduction
42. Apps
42.1. Bulk Import of Applications
43. Runtime
44. Streams
45. Create Stream
46. Tasks
46.1. Apps
46.1.1. Create a Task Definition from a selected Task App
46.1.2. View Task App Details
46.2. Definitions
46.2.1. Creating Task Definitions using the bulk define interface
46.2.2. Creating Composed Task Definitions
46.2.3. Launching Tasks
46.3. Executions
47. Jobs
47.1. List job executions
47.1.1. Job execution details
47.1.2. Step execution details
47.1.3. Step Execution Progress
48. Analytics
IX. ‘How-to’ guides
49. Configure Maven Properties
50. Logging
50.1. Deployment Logs
50.2. Application Logs
51. Frequently asked questions
51.1. Advanced SpEL expressions
51.2. How to use JDBC-sink?
51.3. How to use multiple message-binders?
X. REST API Guide
52. Overview
52.1. HTTP verbs
52.2. HTTP status codes
52.3. Headers
52.4. Errors
52.5. Hypermedia
53. Resources
53.1. Index
53.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Links
53.2. Server Meta Information
53.2.1. Retrieving information about the server
Request structure
Example request
Response structure
53.3. Registered Applications
53.3.1. Listing Applications
Request structure
Request parameters
Example request
Response structure
53.3.2. Getting Information on a partical Application
Request structure
Path parameters
Example request
Response structure
53.3.3. Registering a New Application
Request structure
Request parameters
Path parameters
Example request
Response structure
53.3.4. Unregistering an Application
Request structure
Path parameters
Example request
Response structure
53.3.5. Registering Applications in Bulk
Request structure
Request parameters
Example request
Response structure
53.4. Stream Definitions
53.4.1. Creating a new Stream Definition
Request structure
Request parameters
Example request
Response structure
53.4.2. List all Stream Definitions
Request structure
Request parameters
Example request
Response structure
53.4.3. List related Stream Definitions
Request structure
Request parameters
Example request
Response structure
53.4.4. Delete a single Stream Definition
Request structure
Request parameters
Example request
Response structure
53.4.5. Delete all Stream Definitions
Request structure
Request parameters
Example request
Response structure
53.5. Stream Deployments
53.5.1. Example "stream deploy" request for a ticktock stream
53.6. Task Definitions
53.6.1. Creating a new Task Definition
Request structure
Request parameters
Example request
Response structure
53.7. Task Launch
53.7.1. Launching a Task
Request structure
Request parameters
Example request
Response structure
53.8. Runtime Information about Applications
53.8.1. Listing All Applications at Runtime
Request structure
Example request
Response structure
53.8.2. Querying all Instances of a Single App
Request structure
Example request
Response structure
53.8.3. Querying a Single Instance of a Single App
Request structure
Example request
Response structure
53.9. Metrics for Stream Applications
53.9.1. Request structure
53.9.2. Example request
53.9.3. Response structure
53.9.4. Example response
XI. Data Flow Template
54. Overview
55. Using the Data Flow Template
XII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface