Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Data Integration and Real-Time Data Processing with Spring Boot

932 Aufrufe

Veröffentlicht am

SF JUG talk by Sabby Anandan at Pivotal Software.
https://www.meetup.com/sfjava/events/256850303/

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Data Integration and Real-Time Data Processing with Spring Boot

  1. 1. Data Integration and Real-time Data Processing with Spring Boot Sabby Anandan @sabbyanandan
  2. 2. - Introduce Spring projects - Discuss how they address Data Integration / Data Processing Challenges Goal:
  3. 3. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  4. 4. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  5. 5. - Decentralization (no ESB) - Lightweight applications that contain integration logic - Loose coupling through message channels Spring Integration
  6. 6. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  7. 7. Stream processing is much more than just EIP
  8. 8. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public String replaceStringMsgHandler(String payload) { return StringUtils.replace(payload, "foo", "bar"); } } B I N D I N G E V E N T S foo channel B I N D I N G channel bar C O N S U M E R S Programming Model: Message Channel Abstraction
  9. 9. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public KStream<Object, Foo> handler(KStream<Object, Event> input){ return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Kafka Streams
  10. 10. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public Flux<Average> sensorAverage(Flux<Sensor> data) { return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Native Reactor Flux’s
  11. 11. Spring Cloud Stream a event-driven microservices framework @EnableBinding(Processor.class) public class Application { @Bean public Function<String, String> toUpperCase() { return s -> s.toUpperCase(); } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Programming Model: Plain Old Java Functions
  12. 12. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  13. 13. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace Same code + Same test-harness Drop-in replacement for a variety of Messaging SystemsOpportunities:
  14. 14. Spring Cloud Stream a event-driven microservices framework Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  15. 15. Demo #1 Events and Data Intensive Applications
  16. 16. :UserCreated :UserCreated :UserNameChanged :UserActivated
  17. 17. User activity in the last 30s Users created in the last 2 mins User interaction by region in the last 1hr window
  18. 18. 1 3 2 1 2 3 Websocket Connection REST request/response REST request/response
  19. 19. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  20. 20. JSR 352
  21. 21. - Closely related processing steps that perform a discrete business process - Deployable unit comprised of one or more job steps - Lifecycle management for jobs/steps Spring Batch JSR 352
  22. 22. Spring Cloud Task a short-lived microservices framework @EnableTask @EnableBatchProcessing public class BatchJobApplication { @Bean public Step extractStep() { // extract business logic } @Bean public Step transformStep() { // transformation logic } @Bean public Step loadStep() { // persistence logic } @Bean public Job etlJob() { return this.jobBuilderFactory.get("etlJob") .start(extractStep()) .next(transformStep()) .next(loadStep()) .build(); } } Database R E P O S I T O R Y Programming Model: Spring Batch Job as Short-lived Application
  23. 23. Spring Cloud Task a short-lived microservices framework @EnableTask public class TimestampTask { @Bean public TimestampTask timeStampTask() { return new TimestampTask(); } public static class TimestampTask implements CommandLineRunner { @Override public void run(String... strings) throws Exception { DateFormat dateFormat = . . logger.info(dateFormat.format(new Date())); } } } Database R E P O S I T O R Y Programming Model: An arbitrary business-logic as Short-lived Application
  24. 24. Spring Cloud Task a short-lived microservices framework Lifecycle Management Transactions Bookkeeping for Restarts/Replay Historical Representation Remote Partitions
  25. 25. Demo #2 24/7 ETL: Cloud-native File Ingest
  26. 26. SFTP Source TaskLauncher ETL Job/Task Database Orchestrated by Spring Cloud Data FlowSFTP Server poll for new files publish each file launch task for each file persist parsed data
  27. 27. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow
  28. 28. A toolkit for building data integration, real- time streaming, and batch data processing pipelines. Spring Cloud Data Flow But wait, there’s more!
  29. 29. Developer Focus Ideation Implementation Production
  30. 30. Demo #3 CI/CD for Data Pipelines
  31. 31. Mask each Payload 111-22-3333 444-55-6666 777-88-9999 . . . The Security Number = xxx-xx-3333 The Security Number = xxx-xx-6666 The Security Number = xxx-xx-9999 . . . Don’t Disturb Don’t Disturb Fix This!
  32. 32. Let’s Recap
  33. 33. Spring Cloud Stream Build highly scalable event-driven microservices connected with shared messaging systems. Spring Cloud Task Build short-lived microservices to perform data processing locally or in the cloud. Spring Cloud Skipper Discover applications and manage their lifecycle on multiple Cloud Platforms. Spring Cloud Data Flow Orchestrate data pipelines made of Spring Cloud Stream or Spring Cloud Task microservices. Consolidate Development and Testing Practices Standardize CI/CD Tooling & AutomationOpportunities:
  34. 34. Next Spring Boot 2.1 Compatibility: Stream, Task, Skipper, and SCDF Function Composition / Function Chaining OAuth2 + OpenID Connect by Default Deeper Integration with Micrometer for Metrics/Monitoring New Data Integration Apps
  35. 35. Resources Spring Cloud Stream Samples | Gitter | StackOverflow Spring Cloud Task Samples | Gitter | StackOverflow Spring Cloud Skipper Samples | Gitter | StackOverflow Spring Cloud Data Flow Samples | Gitter | StackOverflow Demo #1: Events + Kafka Streams Demo #2: File-ingest Demo #3: CI/CD for Data Pipelines
  36. 36. Q+A

×