Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.2.0.RC1

Copyright © 2013-2016 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Deploying Spring Cloud Data Flow Local Server
13.1. Maven Configuration
14. Application Configuration
V. Server Configuration
15. Feature Toggles
16. Database Configuration
17. Security
17.1. Enabling HTTPS
17.1.1. Using Self-Signed Certificates
17.1.2. Self-Signed Certificates and the Shell
17.2. Basic Authentication
17.2.1. File based authentication
17.2.2. LDAP Authentication
LDAP Transport Security
17.2.3. Customizing authorization
17.2.4. Authorization - Shell and Dashboard Behavior
17.2.5. Authorization with Ldap
17.3. OAuth 2.0
17.3.1. Authentication using the Spring Cloud Data Flow Shell
17.3.2. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
17.4. Securing the Spring Boot Management Endpoints
18. Monitoring and Management
18.1. Spring Boot Admin
18.2. Monitoring Deployed Applications
18.3. Log and DataDog MetricWriter
VI. Streams
19. Introduction
20. Stream DSL
21. Register a Stream App
21.1. Whitelisting application properties
21.2. Creating and using a dedicated metadata artifact
21.2.1. Using the companion artifact
22. Creating custom applications
23. Creating a Stream
23.1. Application properties
23.1.1. Passing application properties when creating a stream
23.2. Deployment properties
23.2.1. Application properties versus Deployer properties
23.2.2. Passing instance count as deployment property
23.2.3. Inline vs file reference properties
23.2.4. Passing application properties when deploying a stream
23.2.5. Passing Spring Cloud Stream properties for the application
23.2.6. Passing per-binding producer consumer properties
23.2.7. Passing stream partition properties during stream deployment
23.2.8. Passing application content type properties
23.2.9. Overriding application properties during stream deployment
24. Destroying a Stream
25. Deploying and Undeploying Streams
26. Other Source and Sink Application Types
27. Simple Stream Processing
28. Stateful Stream Processing
29. Tap a Stream
30. Using Labels in a Stream
31. Explicit Broker Destinations in a Stream
32. Directed Graphs in a Stream
32.1. Common application properties
33. Stream applications with multiple binder configurations
VII. Tasks
34. Introducing Spring Cloud Task
35. The Lifecycle of a task
35.1. Creating a custom Task Application
35.2. Registering a Task Application
35.3. Creating a Task
35.4. Launching a Task
35.5. Reviewing Task Executions
35.6. Destroying a Task
36. Task Repository
36.1. Configuring the Task Execution Repository
36.1.1. Local
36.1.2. Task Application Repository
36.2. Datasource
37. Subscribing to Task/Batch Events
38. Launching Tasks from a Stream
38.1. TriggerTask
38.2. Translator
39. Composed Tasks
39.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
39.1.1. Registering the Composed Task Runner application
39.1.2. Configuring the Composed Task Runner application
39.2. Creating, Launching, and Destroying a Composed Task
39.2.1. Creating a Composed Task
Task Application Parameters
39.2.2. Launching a Composed Task
Exit Statuses
39.2.3. Destroying a Composed Task
39.2.4. Stopping a Composed Task
39.2.5. Restarting a Composed Task
39.3. Composed Task DSL
39.3.1. Conditional Execution
39.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
39.3.3. Split Execution
Split Containing Conditional Execution
VIII. Dashboard
40. Introduction
41. Apps
41.1. Bulk Import of Applications
42. Runtime
43. Streams
44. Create Stream
45. Tasks
45.1. Apps
45.1.1. Create a Task Definition from a selected Task App
45.1.2. View Task App Details
45.2. Definitions
45.2.1. Creating Task Definitions using the bulk define interface
45.2.2. Creating Composed Task Definitions
45.2.3. Launching Tasks
45.3. Executions
46. Jobs
46.1. List job executions
46.1.1. Job execution details
46.1.2. Step execution details
46.1.3. Step Execution Progress
47. Analytics
IX. ‘How-to’ guides
48. Configure Maven Properties
49. Logging
49.1. Deployment Logs
49.2. Application Logs
X. REST API Guide
50. Overview
50.1. HTTP verbs
50.2. HTTP status codes
50.3. Headers
50.4. Errors
50.5. Hypermedia
51. Resources
51.1. Index
51.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Example "stream create" request for a ticktock stream
Example "stream deploy" request for a ticktock stream
Links
51.2. Server Meta Information
51.2.1. Request structure
51.2.2. Request parameters
51.2.3. Example request
51.2.4. Response structure
51.3. Listing Applications
51.3.1. Request structure
51.3.2. Request parameters
51.3.3. Example request
51.3.4. Response structure
XI. Data Flow Template
52. Overview
53. Using the Data Flow Template
XII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface