Spring Cloud Data Flow Reference Guide
   Next

Spring Cloud Data Flow Reference Guide

Authors

Sabby Anandan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Thomas Risberg, Dave Syer, David Turanski, Janne Valkealahti

1.2.0.M1

Copyright © 2013-2016 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Preface
1. About the documentation
2. Getting help
II. Spring Cloud Data Flow Overview
3. Introducing Spring Cloud Data Flow
3.1. Features
III. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
IV. Getting started
12. System Requirements
13. Deploying Spring Cloud Data Flow Local Server
13.1. Maven Configuration
14. Application Configuration
V. Server Configuration
15. Feature Toggles
16. Database Configuration
17. Security
17.1. Enabling HTTPS
17.1.1. Using Self-Signed Certificates
17.1.2. Self-Signed Certificates and the Shell
17.2. Basic Authentication
17.2.1. File based authentication
17.2.2. LDAP Authentication
LDAP Transport Security
17.2.3. Customizing authorization
17.2.4. Authorization with Ldap
17.3. OAuth 2.0
17.3.1. Authentication using the Spring Cloud Data Flow Shell
17.3.2. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
17.4. Securing the Spring Boot Management Endpoints
18. Monitoring and Management
18.1. Spring Boot Admin
18.2. Monitoring Deployed Applications
VI. Streams
19. Introduction
20. Stream DSL
21. Register a Stream App
21.1. Whitelisting application properties
22. Creating custom applications
23. Creating a Stream
23.1. Application properties
23.1.1. Passing application properties when creating a stream
23.2. Deployment properties
23.2.1. Passing instance count as deployment property
23.2.2. Inline vs file reference properties
23.2.3. Passing application properties when deploying a stream
23.2.4. Passing Spring Cloud Stream properties for the application
23.2.5. Passing per-binding producer consumer properties
23.2.6. Passing stream partition properties during stream deployment
23.2.7. Passing application content type properties
23.2.8. Overriding application properties during stream deployment
24. Destroying a Stream
25. Deploying and Undeploying Streams
26. Other Source and Sink Application Types
27. Simple Stream Processing
28. Stateful Stream Processing
29. Tap a Stream
30. Using Labels in a Stream
31. Explicit Broker Destinations in a Stream
32. Directed Graphs in a Stream
32.1. Common application properties
33. Stream applications with multiple binder configurations
VII. Tasks
34. Introducing Spring Cloud Task
35. The Lifecycle of a task
35.1. Creating a custom Task Application
35.2. Registering a Task Application
35.3. Creating a Task
35.4. Launching a Task
35.5. Reviewing Task Executions
35.6. Destroying a Task
36. Task Repository
36.1. Configuring the Task Execution Repository
36.1.1. Local
36.1.2. Task Application Repository
36.2. Datasource
37. Subscribing to Task/Batch Events
38. Launching Tasks from a Stream
38.1. TriggerTask
38.2. Translator
VIII. Dashboard
39. Introduction
40. Apps
40.1. Bulk Import of Applications
41. Runtime
42. Streams
43. Create Stream
44. Tasks
44.1. Apps
44.1.1. Create a Task Definition from a selected Task App
44.1.2. View Task App Details
44.2. Definitions
44.2.1. Creating Task Definitions using the bulk define interface
44.2.2. Launching Tasks
44.3. Executions
45. Jobs
45.1. List job executions
45.1.1. Job execution details
45.1.2. Step execution details
45.1.3. Step Execution Progress
46. Analytics
IX. ‘How-to’ guides
47. Configure Maven Properties
48. Logging
48.1. Deployment Logs
48.2. Application Logs
X. REST API Guide
49. Overview
49.1. HTTP verbs
49.2. HTTP status codes
49.3. Headers
49.4. Errors
49.5. Hypermedia
50. Resources
50.1. Index
50.1.1. Accessing the index
Request structure
Example request
Response structure
Example response
Example "stream create" request for a ticktock stream
Example "stream deploy" request for a ticktock stream
Links
50.2. Listing Applications
50.2.1. Request structure
50.2.2. Request parameters
50.2.3. Example request
50.2.4. Response structure
XI. Data Flow Template
51. Overview
52. Using the Data Flow Template
XII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Documentation
B.2. Working with the code
B.2.1. Importing into eclipse with m2eclipse
B.2.2. Importing into eclipse without m2eclipse
C. Contributing
C.1. Sign the Contributor License Agreement
C.2. Code Conventions and Housekeeping
   Next
   Part I. Preface