Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.1.0.RC1

Copyright © 2013-2016 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Deploying Spring Cloud Data Flow Local Server
14. Application Configuration
V. Server Configuration
15. Feature Toggles
16. Database Configuration
17. Security
17.1. Enabling HTTPS
17.1.1. Using Self-Signed Certificates
17.2. Basic Authentication
17.2.1. File based authentication
17.2.2. LDAP Authentication
LDAP Transport Security
17.3. OAuth 2.0
17.3.1. Authentication using the Spring Cloud Data Flow Shell
17.3.2. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
17.4. Securing the Spring Boot Management Endpoints
18. Monitoring and Management
18.1. Spring Boot Admin
VI. Streams
19. Introduction
20. Stream DSL
21. Register a Stream App
21.1. Whitelisting application properties
22. Creating a Stream
22.1. Application properties
22.1.1. Passing application properties when creating a stream
22.2. Deployment properties
22.2.1. Passing instance count as deployment property
22.2.2. Inline vs file reference properties
22.2.3. Passing application properties when deploying a stream
22.2.4. Passing Spring Cloud Stream properties for the application
22.2.5. Passing per-binding producer consumer properties
22.2.6. Passing stream partition properties during stream deployment
22.2.7. Passing application content type properties
22.2.8. Overriding application properties during stream deployment
22.3. Deployment properties
22.3.1. Passing instance count as deployment property
22.3.2. Inline vs file reference properties
23. Destroying a Stream
24. Deploying and Undeploying Streams
25. Other Source and Sink Application Types
26. Simple Stream Processing
27. Stateful Stream Processing
28. Tap a Stream
29. Using Labels in a Stream
30. Explicit Broker Destinations in a Stream
31. Directed Graphs in a Stream
31.1. Common application properties
32. Stream applications with multiple binder configurations
VII. Tasks
33. Introducing Spring Cloud Task
34. The Lifecycle of a task
34.1. Registering a Task Application
34.2. Creating a Task
34.3. Launching a Task
34.4. Reviewing Task Executions
34.5. Destroying a Task
35. Task Repository
35.1. Configuring the Task Execution Repository
35.1.1. Local
35.1.2. Task Application Repository
35.2. Datasource
36. Subscribing to Task/Batch Events
37. Launching Tasks from a Stream
37.1. TriggerTask
37.2. Translator
VIII. Dashboard
38. Introduction
39. Apps
39.1. Bulk Import of Applications
40. Runtime
41. Streams
42. Create Stream
43. Tasks
43.1. Apps
43.1.1. Create a Task Definition from a selected Task App
43.1.2. View Task App Details
43.2. Definitions
43.2.1. Creating Task Definitions using the bulk define interface
43.2.2. Launching Tasks
43.3. Executions
44. Jobs
44.1. List job executions
44.1.1. Job execution details
44.1.2. Step execution details
44.1.3. Step Execution Progress
45. Analytics
IX. ‘How-to’ guides
46. Configure Maven Properties
47. Logging
47.1. Deployment Logs
47.2. Application Logs
X. REST API Guide
48. Overview
48.1. HTTP verbs
48.2. HTTP status codes
48.3. Headers
48.4. Errors
48.5. Hypermedia
49. Resources
49.1. Index
49.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Links
49.2. Listing Applications
49.2.1. Request structure
49.2.2. Request parameters
49.2.3. Example request
49.2.4. Response structure
XI. Data Flow Template
50. Overview
51. Using the Data Flow Template
XII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface