Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Develop Powerful Big Data Applications Easily with SpringXD

4.820 Aufrufe

Veröffentlicht am

Veröffentlicht in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Develop Powerful Big Data Applications Easily with SpringXD

  1. 1. © 2014 SpringOne2GX. All rights reserved. Do not distribute without permission. Develop Powerful Big Data Applications Easily with SpringXD Mark Fisher & Mark Pollack
  2. 2. Mark Fisher •Spring XD –Co Lead •Spring Integration •Spring Framework •Spring AMQP Mark Pollack •Spring XD –Co Lead •Spring Data •Spring Framework •Spring .NET 2 Speakers
  3. 3. Spring XDXD= eXtremeData 3
  4. 4. “One stop shop for developing and deploying Big Data Applications” 4
  5. 5. What is a Big Data Application?
  6. 6. Big Data Architecture Stream Processing Analytics Ingest Workflow Orchestration Spring XD Export FILES SOCIAL SENSORS MOBILE XD> MASTER DATASET Predictive Modeling BATCH VIEWS REALTIME VIEWS Spring BOOT Spring BOOT Spring BOOT
  7. 7. REALTIME VIEWS BATCH VIEWS Spring XD MASTER DATASET Spring BOOT Spring BOOT Spring BOOT FILES SOCIAL SENSORS MOBILE Stream Processing Analytics Ingest Workflow Orchestration Spring XD Export XD> Predictive Modeling Lambda Architecture SPEED LAYER BATCH LAYER SERVING LAYER
  8. 8. REALTIME VIEWS BATCH VIEWS Spring XD MASTER DATASET Spring BOOT Spring BOOT Spring BOOT FILES SOCIAL SENSORS MOBILE Stream Processing Analytics Ingest Workflow Orchestration Spring XD Export XD> GemFireXD Predictive Modeling GemFireXD SPEED LAYER BATCH LAYER SERVING LAYER
  9. 9. Spring IO Platform 9 Jobs, Steps, Readers, Writers Ingestion, Export, Orchestration, Hadoop Controllers, REST, WebSocket Channels, Adapters, Filters, Transformers WEB INTEGRATION BATCH BIG DATA SPRING CORE FRAMEWORK SECURITY GROOVY REACTOR DATA RELATIONALDATA ACCESS NON-RELATIONALDATA ACCESS BOOT Bootable, Minimal, Ops-Ready GRAILS Full-stack, Web XD Stream, Taps, Jobs IO EXECUTION IO FOUNDATION IO COORDINATION SPRING CLOUD
  10. 10. Spring XD: Unified Platform for Big Data 10 Spring XD Runtime BIDIRECTIONAL Compute HDFS RDBMS NoSQL R, SAS Streams Jobs ingest workflow export taps Predictive Modelling >_ Redis
  11. 11. Streams Spring XD HTTP Tail File Mail Twitter Gemfire Syslog TCP UDP JMS RabbitMQ MQTT Trigger Reactor TCP/UDP Filter Transformer Object-to-JSON JSON-to-Tuple Splitter Aggregator HTTP Client Groovy Scripts Java Code JPMML Evaluator File HDFS JDBC TCP Log Mail RabbitMQ Gemfire Splunk MQTT Dynamic Router Counters
  12. 12. Demo: Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommerciallicense: http://creativecommons.org/licenses/by-nc/3.0/ Spring XD -Streams 12
  13. 13. Taps Spring XD •“Listen” to data on another stream
  14. 14. Analytics •Counters and Gauges –Simple & Field Value Counter •How many tweets for #java –Aggregate Counter •How many tweets for #java in the week/day/hour –Gauge & Rich Gauge •How many requests per minute? •Abstract API. Implemented in –In-Memory –Redis
  15. 15. Demo: Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommerciallicense: http://creativecommons.org/licenses/by-nc/3.0/ Spring XD -Taps 15
  16. 16. Spring XD Runtime XD Container XD Container XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport ZooKeeper Container State XD Admin XD Admin
  17. 17. Spring XD Runtime XD Container XD Container XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport Spring App Context M1 ZooKeeper Container State XD Admin XD Admin
  18. 18. Spring XD Runtime XD Container XD Container XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport Spring App Context M1 ZooKeeper Container State XD Admin XD Admin M2
  19. 19. Deployment Manifest Spring XD
  20. 20. Deployment Manifest Spring XD •The stream/job definition defines the logical view of processing •The deployment manifest defines the physical view of processing •Important properties relate to module count and data partitioning xd:>stream create test1 --definition "http | transform --expression=payload.toUpperCase() | log” xd:>stream deploy --name test1 --properties "module.transform.count=3"
  21. 21. Deployment Manifest –Data Partitioning Spring XD stream create words --definition "http | splitter --expression=payload.split(' ') | log" stream deploy words --properties module.splitter.producer.partitionKeyExpression=payload,module.log.count=2 http post --data "How much wood would a woodchuck chuck if a woodchuck could chuck wood"
  22. 22. Demo: Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommerciallicense: http://creativecommons.org/licenses/by-nc/3.0/ Spring XD -Partitioning 22
  23. 23. Distributed, Fault Tolerant Runtime Spring XD
  24. 24. Spring XD –Runtime –Fault Tolerance Spring XD XD Container XD Container XD XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport Spring App Context M1 ZooKeeper Container State XD Admin XD Admin M2
  25. 25. XD Container Spring XD –Runtime –Fault Tolerance Spring XD XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport ZooKeeper Container State XD Admin XD Admin M2
  26. 26. XD Container Spring XD –Runtime –Fault Tolerance Spring XD XD Admin (leader) XD Shell HTTP POST /streams/aStream “M1 | M2” Data Transport ZooKeeper Container State XD Admin XD Admin M2 M1
  27. 27. XD Container Spring XD –Runtime –Fault Tolerance Spring XD XD Shell Data Transport ZooKeeper Container State XD Admin XD Admin (leader) M2 M1
  28. 28. XD Container Spring XD –Runtime –Fault Tolerance Spring XD XD Shell Data Transport ZooKeeper Container State XD Admin XD Admin (leader) XD Container M2 M1
  29. 29. Spring XD –Runtime –Fault Tolerance Spring XD XD Shell Data Transport ZooKeeper Container State XD Admin XD Admin (leader) XD Container XD Container XD Container M2 M1
  30. 30. Spring XD –Runtime –Fault Tolerance Spring XD XD Shell Data Transport ZooKeeper Container State XD Admin XD Admin (leader) XD Admin XD Container XD Container XD Container M2 M1
  31. 31. XD Container Spring XD –Runtime –Fault Tolerance Spring XD XD Shell HTTP POST /streams/aStream “M3| M4” Data Transport ZooKeeper Container State XD Admin XD Admin (leader) XD Container XD Admin M3 XD Container M4 M2 M1
  32. 32. Predictive Models Spring XD
  33. 33. Predictive Models
  34. 34. Concepts •Model –Parameterized algorithm •Model Building –Derive a parameterized algorithm from the data –Slow process. Done offline, as a batch process, due to amount of data involved •Model Scoring –Use the model to predict new information –Fast process. Can be done as part of stream processing
  35. 35. PMML •Predictive Model Markup Language •XML interchange format for analytical models •From the Data Mining Group http://www.dmg.org •Processing + models •Supported by statistics and data minigtools –R/Rattle, SAS Enterprise Miner, SPSS, Weka •Java Evaluator API –JPMML-Evaluator project –Provides model scoring
  36. 36. Demo: Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under aCreative Commons Attribution-NonCommerciallicense: http://creativecommons.org/licenses/by-nc/3.0/ Spring XD –Predictive Models 36
  37. 37. Spring XD: Unified Platform for Big Data 37 Spring XD Runtime BIDIRECTIONAL Compute HDFS RDBMS NoSQL R, SAS Streams Jobs ingest workflow export taps Predictive Modelling >_ Redis
  38. 38. Jobs Spring XD CSV to JDBC FTP to HDFS JDBC to HDFS HDFS to JDBC HDFS to MongoDB
  39. 39. Learn More… Spring XD •Project: http://projects.spring.io/spring-xd/ •GitHub: https://github.com/spring-projects/spring-xd/ •Issues: https://jira.springsource.org/browse/XD •Wiki: https://github.com/spring-projects/spring-xd/wiki •Samples: https://github.com/spring-projects/spring-xd-samples

×