Spring Cloud Data Flow Server for Cloud Foundry
   Next

Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.2.2.BUILD-SNAPSHOT

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Spring Cloud Data Flow for Cloud Foundry
1. Spring Cloud Data Flow
2. Spring Cloud Stream
3. Spring Cloud Task
II. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
III. Getting started
12. Deploying on Cloud Foundry
12.1. Provision a Redis service instance on Cloud Foundry
12.2. Provision a Rabbit service instance on Cloud Foundry
12.3. Provision a MySQL service instance on Cloud Foundry
12.4. Download the Spring Cloud Data Flow Server and Shell apps
12.5. Running the Server
12.5.1. Deploying and Running the Server app on Cloud Foundry
Configuring Defaults for Deployed Apps
12.5.2. Running the Server app locally
12.5.3. Sample Manifest Template
12.6. Running Spring Cloud Data Flow Shell locally
13. Security
13.1. Authentication and Cloud Foundry
13.1.1. Pivotal Single Sign-On Service
13.1.2. Cloud Foundry UAA
14. Application Names and Prefixes
14.1. Using Custom Routes
15. Deploying Docker Applications
16. Configuration Reference
16.1. Understanding what’s going on
16.2. Using Spring Cloud Config Server
16.2.1. Stream, Task, and Spring Cloud Config Server
16.2.2. Sample Manifest Template
16.2.3. Self-signed SSL Certificate and Spring Cloud Config Server
17. Application Level Service Bindings
18. A Note About User Provided Services
19. Application Rolling Upgrades
20. Maximum Disk Quota Configuration
20.1. PCF’s Operations Manager Configuration
20.2. Scale Application
20.3. Configuring target free disk percentage
IV. Server Configuration
21. Shell Options
22. Feature Toggles
23. Database Configuration
24. Security
24.1. Enabling HTTPS
24.1.1. Using Self-Signed Certificates
24.1.2. Self-Signed Certificates and the Shell
24.2. Traditional Authentication
24.2.1. Single User Authentication
24.2.2. File based authentication
24.2.3. LDAP Authentication
LDAP Transport Security
24.2.4. Shell Authentication
24.2.5. Customizing authorization
24.2.6. Authorization - Shell and Dashboard Behavior
24.2.7. Authorization with Ldap
24.3. OAuth 2.0
24.3.1. OAuth REST Endpoint Authorization
24.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
24.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
24.4. Securing the Spring Boot Management Endpoints
25. Monitoring and Management
25.1. Spring Boot Admin
25.2. Monitoring Deployed Applications
25.3. Log and DataDog MetricWriter
V. Streams
26. Introduction
27. Stream DSL
28. Register a Stream App
28.1. Whitelisting application properties
28.2. Creating and using a dedicated metadata artifact
28.2.1. Using the companion artifact
29. Creating custom applications
30. Creating a Stream
30.1. Application properties
30.1.1. Passing application properties when creating a stream
30.2. Deployment properties
30.2.1. Application properties versus Deployer properties
30.2.2. Passing instance count as deployment property
30.2.3. Inline vs file reference properties
30.2.4. Passing application properties when deploying a stream
30.2.5. Passing Spring Cloud Stream properties for the application
30.2.6. Passing per-binding producer consumer properties
30.2.7. Passing stream partition properties during stream deployment
30.2.8. Passing application content type properties
30.2.9. Overriding application properties during stream deployment
30.3. Common application properties
31. Destroying a Stream
32. Deploying and Undeploying Streams
33. Other Source and Sink Application Types
34. Simple Stream Processing
35. Stateful Stream Processing
36. Tap a Stream
37. Using Labels in a Stream
38. Explicit Broker Destinations in a Stream
39. Directed Graphs in a Stream
40. Stream applications with multiple binder configurations
VI. Tasks
41. Introducing Spring Cloud Task
42. The Lifecycle of a task
42.1. Creating a custom Task Application
42.2. Registering a Task Application
42.3. Creating a Task
42.4. Launching a Task
42.4.1. Common application properties
42.5. Reviewing Task Executions
42.6. Destroying a Task
43. Task Repository
43.1. Configuring the Task Execution Repository
43.1.1. Local
43.1.2. Task Application Repository
43.2. Datasource
44. Subscribing to Task/Batch Events
45. Launching Tasks from a Stream
45.1. TriggerTask
45.2. TaskLaunchRequest-transform
46. Composed Tasks
46.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
46.1.1. Registering the Composed Task Runner application
46.1.2. Configuring the Composed Task Runner application
46.2. Creating, Launching, and Destroying a Composed Task
46.2.1. Creating a Composed Task
Task Application Parameters
46.2.2. Launching a Composed Task
Exit Statuses
46.2.3. Destroying a Composed Task
46.2.4. Stopping a Composed Task
46.2.5. Restarting a Composed Task
46.3. Composed Task DSL
46.3.1. Conditional Execution
46.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
46.3.3. Split Execution
Split Containing Conditional Execution
VII. Tasks on Cloud Foundry
47. Version Compatibility
48. Tooling
49. Task Database Schema
50. Running Task Applications
50.1. Create a Task
50.2. Launch a Task
50.3. View Task Logs
50.4. List Tasks
50.5. List Task Executions
50.6. Destroy a Task
50.7. Deleting Task From Cloud Foundry
VIII. Dashboard
51. Introduction
52. Apps
52.1. Bulk Import of Applications
53. Runtime
54. Streams
55. Create Stream
56. Tasks
56.1. Apps
56.1.1. Create a Task Definition from a selected Task App
56.1.2. View Task App Details
56.2. Definitions
56.2.1. Creating Task Definitions using the bulk define interface
56.2.2. Creating Composed Task Definitions
56.2.3. Launching Tasks
56.3. Executions
57. Jobs
57.1. List job executions
57.1.1. Job execution details
57.1.2. Step execution details
57.1.3. Step Execution Progress
58. Analytics
IX. ‘How-to’ guides
59. Configure Maven Properties
60. Logging
60.1. Deployment Logs
60.2. Application Logs
61. Frequently asked questions
61.1. Advanced SpEL expressions
61.2. How to use JDBC-sink?
61.3. How to use multiple message-binders?
X. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Basic Compile and Test
B.2. Documentation
B.3. Working with the code
B.3.1. Importing into eclipse with m2eclipse
B.3.2. Importing into eclipse without m2eclipse
   Next
   Part I. Spring Cloud Data Flow for Cloud Foundry