Spring Cloud Data Flow Server for Cloud Foundry
   Next

Spring Cloud Data Flow Server for Cloud Foundry

Authors

Sabby Anandan, Eric Bottard, Mark Fisher, Ilayaperumal Gopinathan, Gunnar Hillert, Mark Pollack, Thomas Risberg, Marius Bogoevici, Josh Long, Michael Minella, David Turanski

1.2.1.BUILD-SNAPSHOT

Copyright © 2013-2017 Pivotal Software, Inc.

Copies of this document may be made for your own use and for distribution to others, provided that you do not charge any fee for such copies and further provided that each copy contains this Copyright Notice, whether distributed in print or electronically.

Table of Contents

I. Spring Cloud Data Flow for Cloud Foundry
1. Spring Cloud Data Flow
2. Spring Cloud Stream
3. Spring Cloud Task
II. Architecture
4. Introduction
5. Microservice Architectural Style
5.1. Comparison to other Platform architectures
6. Streaming Applications
6.1. Imperative Programming Model
6.2. Functional Programming Model
7. Streams
7.1. Topologies
7.2. Concurrency
7.3. Partitioning
7.4. Message Delivery Guarantees
8. Analytics
9. Task Applications
10. Data Flow Server
10.1. Endpoints
10.2. Customization
10.3. Security
11. Runtime
11.1. Fault Tolerance
11.2. Resource Management
11.3. Scaling at runtime
11.4. Application Versioning
III. Getting started
12. Deploying on Cloud Foundry
12.1. Provision a Redis service instance on Cloud Foundry
12.2. Provision a Rabbit service instance on Cloud Foundry
12.3. Provision a MySQL service instance on Cloud Foundry
12.4. Download the Spring Cloud Data Flow Server and Shell apps
12.5. Running the Server
12.5.1. Deploying and Running the Server app on Cloud Foundry
Configuring Defaults for Deployed Apps
12.5.2. Running the Server app locally
12.5.3. Sample Manifest Template
12.6. Running Spring Cloud Data Flow Shell locally
13. Security
13.1. Authentication and Cloud Foundry
13.1.1. Pivotal Single Sign-On Service
13.1.2. Cloud Foundry UAA
14. Application Names and Prefixes
14.1. Using Custom Routes
15. Deploying Docker Applications
16. Configuration Reference
16.1. Understanding what’s going on
16.2. Using Spring Cloud Config Server
16.2.1. Stream, Task, and Spring Cloud Config Server
16.2.2. Sample Manifest Template
16.2.3. Self-signed SSL Certificate and Spring Cloud Config Server
17. Application Level Service Bindings
18. A Note About User Provided Services
19. Application Rolling Upgrades
20. Maximum Disk Quota Configuration
20.1. PCF’s Operations Manager Configuration
20.2. Scale Application
20.3. Configuring target free disk percentage
IV. Server Configuration
21. Feature Toggles
22. Database Configuration
23. Security
23.1. Enabling HTTPS
23.1.1. Using Self-Signed Certificates
23.1.2. Self-Signed Certificates and the Shell
23.2. Traditional Authentication
23.2.1. Single User Authentication
23.2.2. File based authentication
23.2.3. LDAP Authentication
LDAP Transport Security
23.2.4. Shell Authentication
23.2.5. Customizing authorization
23.2.6. Authorization - Shell and Dashboard Behavior
23.2.7. Authorization with Ldap
23.3. OAuth 2.0
23.3.1. OAuth REST Endpoint Authorization
23.3.2. OAuth Authentication using the Spring Cloud Data Flow Shell
23.3.3. OAuth2 Authentication Examples
Local OAuth2 Server
Authentication using GitHub
23.4. Securing the Spring Boot Management Endpoints
24. Monitoring and Management
24.1. Spring Boot Admin
24.2. Monitoring Deployed Applications
24.3. Log and DataDog MetricWriter
V. Streams
25. Introduction
26. Stream DSL
27. Register a Stream App
27.1. Whitelisting application properties
27.2. Creating and using a dedicated metadata artifact
27.2.1. Using the companion artifact
28. Creating custom applications
29. Creating a Stream
29.1. Application properties
29.1.1. Passing application properties when creating a stream
29.2. Deployment properties
29.2.1. Application properties versus Deployer properties
29.2.2. Passing instance count as deployment property
29.2.3. Inline vs file reference properties
29.2.4. Passing application properties when deploying a stream
29.2.5. Passing Spring Cloud Stream properties for the application
29.2.6. Passing per-binding producer consumer properties
29.2.7. Passing stream partition properties during stream deployment
29.2.8. Passing application content type properties
29.2.9. Overriding application properties during stream deployment
29.3. Common application properties
30. Destroying a Stream
31. Deploying and Undeploying Streams
32. Other Source and Sink Application Types
33. Simple Stream Processing
34. Stateful Stream Processing
35. Tap a Stream
36. Using Labels in a Stream
37. Explicit Broker Destinations in a Stream
38. Directed Graphs in a Stream
39. Stream applications with multiple binder configurations
VI. Tasks
40. Introducing Spring Cloud Task
41. The Lifecycle of a task
41.1. Creating a custom Task Application
41.2. Registering a Task Application
41.3. Creating a Task
41.4. Launching a Task
41.4.1. Common application properties
41.5. Reviewing Task Executions
41.6. Destroying a Task
42. Task Repository
42.1. Configuring the Task Execution Repository
42.1.1. Local
42.1.2. Task Application Repository
42.2. Datasource
43. Subscribing to Task/Batch Events
44. Launching Tasks from a Stream
44.1. TriggerTask
44.2. TaskLaunchRequest-transform
45. Composed Tasks
45.1. Configuring the Composed Task Runner in Spring Cloud Data Flow
45.1.1. Registering the Composed Task Runner application
45.1.2. Configuring the Composed Task Runner application
45.2. Creating, Launching, and Destroying a Composed Task
45.2.1. Creating a Composed Task
Task Application Parameters
45.2.2. Launching a Composed Task
Exit Statuses
45.2.3. Destroying a Composed Task
45.2.4. Stopping a Composed Task
45.2.5. Restarting a Composed Task
45.3. Composed Task DSL
45.3.1. Conditional Execution
45.3.2. Transitional Execution
Basic Transition
Transition With a Wildcard
Transition With a Following Conditional Execution
45.3.3. Split Execution
Split Containing Conditional Execution
VII. Tasks on Cloud Foundry
46. Version Compatibility
47. Tooling
48. Task Database Schema
49. Running Task Applications
49.1. Create a Task
49.2. Launch a Task
49.3. View Task Logs
49.4. List Tasks
49.5. List Task Executions
49.6. Destroy a Task
49.7. Deleting Task From Cloud Foundry
VIII. Dashboard
50. Introduction
51. Apps
51.1. Bulk Import of Applications
52. Runtime
53. Streams
54. Create Stream
55. Tasks
55.1. Apps
55.1.1. Create a Task Definition from a selected Task App
55.1.2. View Task App Details
55.2. Definitions
55.2.1. Creating Task Definitions using the bulk define interface
55.2.2. Creating Composed Task Definitions
55.2.3. Launching Tasks
55.3. Executions
56. Jobs
56.1. List job executions
56.1.1. Job execution details
56.1.2. Step execution details
56.1.3. Step Execution Progress
57. Analytics
IX. ‘How-to’ guides
58. Configure Maven Properties
59. Logging
59.1. Deployment Logs
59.2. Application Logs
60. Frequently asked questions
60.1. Advanced SpEL expressions
60.2. How to use JDBC-sink?
60.3. How to use multiple message-binders?
X. Appendices
A. Migrating from Spring XD to Spring Cloud Data Flow
A.1. Terminology Changes
A.2. Modules to Applications
A.2.1. Custom Applications
A.2.2. Application Registration
A.2.3. Application Properties
A.3. Message Bus to Binders
A.3.1. Message Bus
A.3.2. Binders
A.3.3. Named Channels
A.3.4. Directed Graphs
A.4. Batch to Tasks
A.5. Shell/DSL Commands
A.6. REST-API
A.7. UI / Flo
A.8. Architecture Components
A.8.1. ZooKeeper
A.8.2. RDBMS
A.8.3. Redis
A.8.4. Cluster Topology
A.9. Central Configuration
A.10. Distribution
A.11. Hadoop Distribution Compatibility
A.12. YARN Deployment
A.13. Use Case Comparison
A.13.1. Use Case #1
A.13.2. Use Case #2
A.13.3. Use Case #3
B. Building
B.1. Basic Compile and Test
B.2. Documentation
B.3. Working with the code
B.3.1. Importing into eclipse with m2eclipse
B.3.2. Importing into eclipse without m2eclipse
   Next
   Part I. Spring Cloud Data Flow for Cloud Foundry