Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.1.0.M2

Copyright © 2013-2016 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Controlling features with Data Flow server
14. Deploying Spring Cloud Data Flow
14.1. Deploying 'local'
15. RDBMS configuration
16. Security
16.1. Enabling HTTPS
16.1.1. Using Self-Signed Certificates
16.2. Basic Authentication
16.2.1. File based authentication
16.2.2. LDAP Authentication
16.3. OAuth 2.0
16.3.1. Authentication using the Spring Cloud Data Flow Shell
16.3.2. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
16.4. Securing the Spring Boot Management Endpoints
V. Streams
17. Introduction
18. Stream DSL
19. Register a Stream App
19.1. Whitelisting application properties
20. Creating a Stream
20.1. Application properties
20.1.1. Passing application properties when creating a stream
20.2. Deployment properties
20.2.1. Passing instance count as deployment property
20.2.2. Inline vs file reference properties
20.2.3. Passing application properties when deploying a stream
20.2.4. Passing Spring Cloud Stream properties for the application
20.2.5. Passing per-binding producer consumer properties
20.2.6. Passing stream partition properties during stream deployment
20.2.7. Passing application content type properties
20.2.8. Overriding application properties during stream deployment
20.3. Deployment properties
20.3.1. Passing instance count as deployment property
20.3.2. Inline vs file reference properties
21. Destroying a Stream
22. Deploying and Undeploying Streams
23. Other Source and Sink Application Types
24. Simple Stream Processing
25. Stateful Stream Processing
26. Tap a Stream
27. Using Labels in a Stream
28. Explicit Broker Destinations in a Stream
29. Directed Graphs in a Stream
29.1. Common application properties
30. Stream applications with multiple binder configurations
VI. Tasks
31. Introducing Spring Cloud Task
32. The Lifecycle of a task
32.1. Registering a Task Application
32.2. Creating a Task
32.3. Launching a Task
32.4. Reviewing Task Executions
32.5. Destroying a Task
33. Task Repository
33.1. Configuring the Task Execution Repository
33.1.1. Local
33.1.2. Task Application Repository
33.2. Datasource
34. Subscribing to Task/Batch Events
35. Launching Tasks from a Stream
35.1. TriggerTask
35.2. Translator
VII. Dashboard
36. Introduction
37. Apps
37.1. Bulk Import of Applications
38. Runtime
39. Streams
40. Create Stream
41. Tasks
41.1. Apps
41.1.1. Create a Task Definition from a selected Task App
41.1.2. View Task App Details
41.2. Definitions
41.2.1. Creating Task Definitions using the bulk define interface
41.2.2. Launching Tasks
41.3. Executions
42. Jobs
42.1. List job executions
42.1.1. Job execution details
42.1.2. Step execution details
42.1.3. Step Execution Progress
43. Analytics
VIII. ‘How-to’ guides
44. Configure Maven Properties
45. Logging
45.1. Deployment Logs
45.2. Application Logs
IX. REST API Guide
46. Overview
46.1. HTTP verbs
46.2. HTTP status codes
46.3. Headers
46.4. Errors
46.5. Hypermedia
47. Resources
47.1. Index
47.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Links
47.2. Listing Applications
47.2.1. Request structure
47.2.2. Request parameters
47.2.3. Example request
47.2.4. Response structure
X. Data Flow Template
48. Overview
49. Using the Data Flow Template
XI. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface