6. Spring XD Limitations
• How to scale up/down instances at runtime?
• How to upgrade/downgrade module instances at runtime?
• How to specify resources unique to each module, e.g. memory?
• Container architecture lead to parent/child class loader issues
• Too many libraries in root class path
6
7. Refactoring to a Microservice Architecture
7
「コンテナに組み込まれた複数モジュール」から
「単体で実行されるスタンドアロンアプリ」 へ
「独自ランタイム」 から
「プラットフォーム委任」 へ
16. Simple Real Time Analytics
16
tweets = twitterstream | hdfs
analytics = :ingest.twitterstream > field-value-counter --fieldName=lang
HTTP
s1.http
HDFS COUNTER
Data Flow Server
REST API
17. Spring Cloud Stream Programming Model
17
@EnableBinding(Processor.class)
public class TransformProcessor {
@StreamListener(“input”)
@SendTo(“output”)
public String transform(String s) {
return s.toUpperCase();
}
}
18. Spring Cloud Stream Programming Model
18
@EnableBinding(Processor.class)
public class TransformProcessor {
@StreamListener
@Output(“output”)
public Flux<String> transform(@Input(“input”) Flux<String> input) {
return input.map(s -> s.toUpperCase());
}
}
19. Spring Cloud Stream Programming Model
19
@EnableBinding(Processor.class)
public class TransformProcessor {
@StreamListener
@Output(“output”)
public Flux<WordCount> countWords(@Input("output") Flux<String> words) {
return words.window(ofSeconds(5), ofSeconds(1))
.flatMap(window -> window.groupBy(word -> word)
.flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
.map(count -> new WordCount(group.key(), count))));
}
}
25. Spring Cloud Task
• Spring Boot based framework for short lived processes
• Auto-configuration provides a task repository and pluggable data source
• Result of each process persists beyond the life of the task for future
reporting
• Tasks can be any arbitrary short lived code
• Well integrated with Spring Batch
25
26. Task Orchestration in Spring Cloud Data Flow
26
>task create jdbc2hdfs –sql=‘select * from table’
>task launch jdbc2hdfs
jdbc2hdfs
Data Flow Server DB
Task Name
Start Time
End Time
Exit Code
Exit Message
Last Updated Time
Parameters
task-event
Message Broker
job-execution-events
step-execution-events
item-read-events
item-process-events
item-write-events
skip-events
27. Spring Cloud Data Flow Tasks
27
spark
Data Flow Server DB
http | task-launcher
sqoop
Message Broker
task-event
Platform Runtime
28. Spring Cloud Task Programming Model
28
@SpringBootApplication
@EnableTask
public class ExampleApplication {
@Bean
public CommandLineRunner commandLineRunner() {
return strings ->
System.out.println("Executed at :" +
new SimpleDateFormat().format(new Date()));
}
public static void main(String[] args) {
SpringApplication.run(ExampleApplication.class, args);
}
}
33. Upcoming features
• Some ‘porting’ from XD
• Batch Job DSL + Designer
• Role based access
• Looking forward
• Spring Cloud Sleuth
• JavaDSL
• In-place application version upgrades with Spinnaker
• Application Groups
• Polyglot
• Expanded analytics with Redis and Python/R ecosystem
• More provided apps/tasks
33
34. Related Talks
• Building Resilient and Evolutionary Data Microservices – Tuesday 2:00pm
• Cloud Native Java – Tuesday 2:00pm
• Task Madness - Modern On Demand Processing – Tuesday 2:40pm
• Spinnaker – Land of a 1000 Builds – Tuesday 5:00pm
• Spring and Big Data – Tuesday 5:00pm
• Migrating from Spring XD to Spring Cloud Data Flow – Thursday 10:10am
• Orchestrate All the Things! with Spring Cloud Data Flow – Thursday 11:10am
• Cloud Native Streaming and Event-Driven Microservices – Wednesday 4:20pm
34
