Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.2.1.RELEASE

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Deploying Spring Cloud Data Flow Local Server
13.1. Maven Configuration
14. Application Configuration
V. Server Configuration
15. Feature Toggles
16. Database Configuration
17. Security
17.1. Enabling HTTPS
17.1.1. Using Self-Signed Certificates
17.1.2. Self-Signed Certificates and the Shell
17.2. Traditional Authentication
17.2.1. Single User Authentication
17.2.2. File based authentication
17.2.3. LDAP Authentication
LDAP Transport Security
17.2.4. Shell Authentication
17.2.5. Customizing authorization
17.2.6. Authorization - Shell and Dashboard Behavior
17.2.7. Authorization with Ldap
17.3. OAuth 2.0
17.3.1. OAuth REST Endpoint Authorization
17.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
17.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
17.4. Securing the Spring Boot Management Endpoints
18. Monitoring and Management
18.1. Spring Boot Admin
18.2. Monitoring Deployed Applications
18.3. Log and DataDog MetricWriter
VI. Streams
19. Introduction
20. Stream DSL
21. Register a Stream App
21.1. Whitelisting application properties
21.2. Creating and using a dedicated metadata artifact
21.2.1. Using the companion artifact
22. Creating custom applications
23. Creating a Stream
23.1. Application properties
23.1.1. Passing application properties when creating a stream
23.2. Deployment properties
23.2.1. Application properties versus Deployer properties
23.2.2. Passing instance count as deployment property
23.2.3. Inline vs file reference properties
23.2.4. Passing application properties when deploying a stream
23.2.5. Passing Spring Cloud Stream properties for the application
23.2.6. Passing per-binding producer consumer properties
23.2.7. Passing stream partition properties during stream deployment
23.2.8. Passing application content type properties
23.2.9. Overriding application properties during stream deployment
23.3. Common application properties
24. Destroying a Stream
25. Deploying and Undeploying Streams
26. Other Source and Sink Application Types
27. Simple Stream Processing
28. Stateful Stream Processing
29. Tap a Stream
30. Using Labels in a Stream
31. Explicit Broker Destinations in a Stream
32. Directed Graphs in a Stream
33. Stream applications with multiple binder configurations
VII. Tasks
34. Introducing Spring Cloud Task
35. The Lifecycle of a task
35.1. Creating a custom Task Application
35.2. Registering a Task Application
35.3. Creating a Task
35.4. Launching a Task
35.4.1. Common application properties
35.5. Reviewing Task Executions
35.6. Destroying a Task
36. Task Repository
36.1. Configuring the Task Execution Repository
36.1.1. Local
36.1.2. Task Application Repository
36.2. Datasource
37. Subscribing to Task/Batch Events
38. Launching Tasks from a Stream
38.1. TriggerTask
38.2. TaskLaunchRequest-transform
39. Composed Tasks
39.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
39.1.1. Registering the Composed Task Runner application
39.1.2. Configuring the Composed Task Runner application
39.2. Creating, Launching, and Destroying a Composed Task
39.2.1. Creating a Composed Task
Task Application Parameters
39.2.2. Launching a Composed Task
Exit Statuses
39.2.3. Destroying a Composed Task
39.2.4. Stopping a Composed Task
39.2.5. Restarting a Composed Task
39.3. Composed Task DSL
39.3.1. Conditional Execution
39.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
39.3.3. Split Execution
Split Containing Conditional Execution
VIII. Dashboard
40. Introduction
41. Apps
41.1. Bulk Import of Applications
42. Runtime
43. Streams
44. Create Stream
45. Tasks
45.1. Apps
45.1.1. Create a Task Definition from a selected Task App
45.1.2. View Task App Details
45.2. Definitions
45.2.1. Creating Task Definitions using the bulk define interface
45.2.2. Creating Composed Task Definitions
45.2.3. Launching Tasks
45.3. Executions
46. Jobs
46.1. List job executions
46.1.1. Job execution details
46.1.2. Step execution details
46.1.3. Step Execution Progress
47. Analytics
IX. ‘How-to’ guides
48. Configure Maven Properties
49. Logging
49.1. Deployment Logs
49.2. Application Logs
50. Frequently asked questions
50.1. Advanced SpEL expressions
50.2. How to use JDBC-sink?
50.3. How to use multiple message-binders?
X. REST API Guide
51. Overview
51.1. HTTP verbs
51.2. HTTP status codes
51.3. Headers
51.4. Errors
51.5. Hypermedia
52. Resources
52.1. Index
52.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Links
52.2. Server Meta Information
52.2.1. Retrieving information about the server
Request structure
Example request
Response structure
52.3. Registered Applications
52.3.1. Listing Applications
Request structure
Request parameters
Example request
Response structure
52.4. Stream Definitions
52.4.1. Example "stream create" request for a ticktock stream
52.5. Stream Deployments
52.5.1. Example "stream deploy" request for a ticktock stream
52.6. Task Definitions
52.6.1. Creating a new Task Definition
Request structure
Request parameters
Example request
Response structure
52.7. Task Launch
52.7.1. Launching a Task
Request structure
Request parameters
Example request
Response structure
52.8. Metrics for Stream Applications
52.8.1. Request structure
52.8.2. Example request
52.8.3. Response structure
52.8.4. Example response
XI. Data Flow Template
53. Overview
54. Using the Data Flow Template
XII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface