Spring Cloud Data Flow Server for Apache Mesos
   Next

Spring Cloud Data Flow Server for Apache Mesos

Sabby Anandan, Artem Bilan, Marius Bogoevici, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Patrick Peralta, Glenn Renfro, Gary Russell, Thomas Risberg, David Turanski, Janne Valkealahti

1.1.0.BUILD-SNAPSHOT

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Introduction
1. Introducing Spring Cloud Data Flow Server for Apache Mesos
2. Spring Cloud Data Flow
3. Spring Cloud Stream
4. Spring Cloud Task
II. Architecture
5. Introduction
6. Microservice Architectural Style
6.1. Comparison to other Platform architectures
7. Streaming Applications
7.1. Imperative Programming Model
7.2. Functional Programming Model
8. Streams
8.1. Topologies
8.2. Concurrency
8.3. Partitioning
8.4. Message Delivery Guarantees
9. Analytics
10. Task Applications
11. Data Flow Server
11.1. Endpoints
11.2. Customization
11.3. Security
12. Runtime
12.1. Fault Tolerance
12.2. Resource Management
12.3. Scaling at runtime
12.4. Application Versioning
III. Getting Started
13. Deploying Streams and Tasks on Apache Mesos and Marathon/Chronos
IV. Server Configuration
14. Feature Toggles
15. General Configuration
V. Streams
16. Introduction
17. Stream DSL
18. Register a Stream App
18.1. Whitelisting application properties
18.2. Creating and using a dedicated metadata artifact
18.2.1. Using the companion artifact
19. Creating custom applications
20. Creating a Stream
20.1. Application properties
20.1.1. Passing application properties when creating a stream
20.2. Deployment properties
20.2.1. Application properties versus Deployer properties
20.2.2. Passing instance count as deployment property
20.2.3. Inline vs file reference properties
20.2.4. Passing application properties when deploying a stream
20.2.5. Passing Spring Cloud Stream properties for the application
20.2.6. Passing per-binding producer consumer properties
20.2.7. Passing stream partition properties during stream deployment
20.2.8. Passing application content type properties
20.2.9. Overriding application properties during stream deployment
20.3. Common application properties
21. Destroying a Stream
22. Deploying and Undeploying Streams
23. Other Source and Sink Application Types
24. Simple Stream Processing
25. Stateful Stream Processing
26. Tap a Stream
27. Using Labels in a Stream
28. Explicit Broker Destinations in a Stream
29. Directed Graphs in a Stream
30. Stream applications with multiple binder configurations
VI. Tasks
31. Introducing Spring Cloud Task
32. The Lifecycle of a task
32.1. Creating a custom Task Application
32.2. Registering a Task Application
32.3. Creating a Task
32.4. Launching a Task
32.4.1. Common application properties
32.5. Reviewing Task Executions
32.6. Destroying a Task
33. Task Repository
33.1. Configuring the Task Execution Repository
33.1.1. Local
33.1.2. Task Application Repository
33.2. Datasource
34. Subscribing to Task/Batch Events
35. Launching Tasks from a Stream
35.1. TriggerTask
35.2. TaskLaunchRequest-transform
36. Composed Tasks
36.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
36.1.1. Registering the Composed Task Runner application
36.1.2. Configuring the Composed Task Runner application
36.2. Creating, Launching, and Destroying a Composed Task
36.2.1. Creating a Composed Task
Task Application Parameters
36.2.2. Launching a Composed Task
Exit Statuses
36.2.3. Destroying a Composed Task
36.2.4. Stopping a Composed Task
36.2.5. Restarting a Composed Task
36.3. Composed Task DSL
36.3.1. Conditional Execution
36.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
36.3.3. Split Execution
Split Containing Conditional Execution
VII. Dashboard
37. Introduction
38. Apps
38.1. Bulk Import of Applications
39. Runtime
40. Streams
41. Create Stream
42. Tasks
42.1. Apps
42.1.1. Create a Task Definition from a selected Task App
42.1.2. View Task App Details
42.2. Definitions
42.2.1. Creating Task Definitions using the bulk define interface
42.2.2. Creating Composed Task Definitions
42.2.3. Launching Tasks
42.3. Executions
43. Jobs
43.1. List job executions
43.1.1. Job execution details
43.1.2. Step execution details
43.1.3. Step Execution Progress
44. Analytics
VIII. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Test Cluster
B.1. Clone and Download files
B.2. Start Vagrant VMs for the Mesos cluster
B.3. Install Chronos as a Service running in Marathon
C. Building
C.1. Documentation
C.2. Working with the code
C.2.1. Importing into eclipse with m2eclipse
C.2.2. Importing into eclipse without m2eclipse
D. Contributing
D.1. Sign the Contributor License Agreement
D.2. Code Conventions and Housekeeping
   Next
   Part I. Introduction