Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR.
GENEVA | HAMBURG | LAUSANNE |...
Agenda
1. Introduction
2. Blueprints Oracle RDBMS => Apache Kafka
3. Blueprints Apache Kafka => Oracle RDBMS
4. Summary
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MA...
Introduction
Microservices / Modern Applications
• Highly decoupled
• Independently deployable
• Bounded Context/Aggregate (DDD)
• Resp...
Microservices / Modern Applications
Integrate with Traditional System
M3M2
ACL
Event
Hub
M1
ACL
• Highly decoupled
• Indep...
Use Case
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Processing System
{ }
Order API OrderOrder Lo...
MessageMessageMessageMessage
MessageMessage
Properties - Message
Message Message
A1 A2 A3
Message
B1 B2 B3 B4
A1 A2 A3 B
B...
Properties - Latency
Traditional System Event
Hub
Data
Flow
RDBMS
latency
latency
Flat Aggregate
Low Latency High Latency
...
Properties – Anti-Corruption Layer (ACL)
Traditional System Event
Hub
Data
Flow
RDBMS
Traditional System Event
Hub
Data
Fl...
Blueprints Oracle RDBMS =>
Apache Kafka
Blueprints Oracle RDBMS => Apache Kafka (DB-K)
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Process...
DB-K_1: Polling of RDBMS table/view
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data FlowRDBMS
Appli...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data FlowRDBMS
Application
Logic
Stream Data
Integration...
Kafka Connect & JDBC Connector
• Many connectors available
• Single Message Transforms (SMT)
• declarative style, simple d...
DB-K_1 – Configure JDBC Connector
#!/bin/bash
curl -X "POST" "http://192.168.69.138:8083/connectors" 
-H "Content-Type: ap...
DB-K_2: Change Data Capture (CDC) on RDBMS
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
Inte...
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
Integration
API
Data Flow
Application / Data So...
DB-K_3: Polling of RDBMS API
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data Flow
RDBMS
Application...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data Flow
RDBMS
Application
Logic
API
Stream Data
Integr...
Oracle REST Data Services (ORDS)
• makes it easy to develop modern REST interfaces for relational data in the
Oracle Datab...
DB-K_3 – Setup ORDS (I)
ORDS.ENABLE_SCHEMA(
p_enabled => TRUE,
p_schema => 'ORDER_PROCESSING',
p_url_mapping_type => 'BASE...
DB-K_3 – Setup ORDS (II)
ORDS.DEFINE_HANDLER(
p_module_name => 'order_processing',
p_pattern => 'changes/:offset',
p_metho...
StreamSets Data Collector
• GUI-based, drag-and
drop Data Flow Pipelines
• Both stream and batch
processing
• custom sourc...
DB-K_4: Produce to Event Hub from RDBMS
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Applicatio...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Application
Logic
API
Stream Data
Integration & An...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Application
Logic
API
Stream Data
Integration & An...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Application
Logic
API
Stream Data
Integration & An...
DB-K_5: RDBMS Queue with bridge to Event Hub
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
In...
DB-K_5: RDBMS Queue with bridge to Event Hub
Oracle Advanced Queuing & Kafka Connect JMS
Stream Data
Integration & Analyti...
BEGIN
dbms_aqadm.create_queue_table (
queue_table => 'ORDER_QT',
queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE',
sort_li...
curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" 
-H "Content-Type: application/json" 
--data '{
"name": "jms-source",
"co...
DB-K_5: RDBMS Queue with bridge to Event Hub
Oracle AQ with Kafka API & MirrorMaker
Stream Data
Integration & Analytics
St...
Blueprints Apache Kafka =>
Oracle RDBMS
Blueprints Apache Kafka => Oracle RDBMS (K-DB)
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Process...
K-DB_1: Write to RDBMS table/view
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data FlowRDBMS
Applica...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data FlowRDBMS
Application
Logic
Stream Data
Integration...
K-DB_2: Write over RDBMS API
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data Flow
RDBMS
Application...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
Data Flow
RDBMS
Application
Logic
API
Stream Data
Integr...
DB-K_2 – Setup ORDS (I)
ORDS.DEFINE_HANDLER(
p_module_name => 'customer',
p_pattern => 'customer',
p_method => 'POST',
p_s...
K-DB_3: Consume from Event Hub
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Application
Logic
A...
Event
Hub
Stream Data
Integration
API
Applications / Data Sources
RDBMS
Application
Logic
API
Stream Data
Integration & An...
K-DB_4: Event Hub with bridge to RDBMS queue
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
In...
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
Integration
API
Data Flow
Application / Data So...
Stream Data
Integration & Analytics
Stream
Analytics
Event
Hub
Stream Data
Integration
API
Data Flow
Application / Data So...
Summary
Summary
Customer Microservice
{ }
Customer API CustomerCustomer Logic
Order Processing System
{ }
Order API OrderOrder Log...
Bulk Source
Ref Architecture
Data Platform
Service
Event
Stream
Bulk
Data
Flow
Event Source
Location
DB
Extract
File
Weath...
Bulk Source
Ref Architecture
Data Platform
Service
Event
Stream
Bulk
Data
Flow
Event Source
Location
DB
Extract
File
Weath...
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka (Guido Schmutz, Trivadis)  Kafka Summit SF ...
Nächste SlideShare
Wird geladen in …5
×

Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka (Guido Schmutz, Trivadis) Kafka Summit SF 2019

635 Aufrufe

Veröffentlicht am

A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Today's enterprises have their core systems often implemented on top of relational databases, such as the Oracle RDBMS. Implementing a new solution supporting the digital strategy using Kafka and the ecosystem can not always be done completely separate from the traditional legacy solutions. Often streaming data has to be enriched with state data which is held in an RDBMS of a legacy application. It's important to cache this data in the stream processing solution, so that It can be efficiently joined to the data stream. But how do we make sure that the cache is kept up-to-date, if the source data changes? We can either poll for changes from Kafka using Kafka Connect or let the RDBMS push the data changes to Kafka. But what about writing data back to the legacy application, i.e. an anomaly is detected inside the stream processing solution which should trigger an action inside the legacy application. Using Kafka Connect we can write to a database table or view, which could trigger the action. But this not always the best option. If you have an Oracle RDBMS, there are many other ways to integrate the database with Kafka, such as Advanced Queueing (message broker in the database), CDC through Golden Gate or Debezium, Oracle REST Database Service (ORDS) and more. In this session, we present various blueprints for integrating an Oracle RDBMS with Apache Kafka in both directions and discuss how these blueprints can be implemented using the products mentioned before.

Veröffentlicht in: Technologie
  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka (Guido Schmutz, Trivadis) Kafka Summit SF 2019

  1. 1. BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Guido Schmutz (guido.schmutz@trivadis.com)
  2. 2. Agenda 1. Introduction 2. Blueprints Oracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Summary
  3. 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 170th edition
  4. 4. Introduction
  5. 5. Microservices / Modern Applications • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! M3M2 ACL Event Hub M1
  6. 6. Microservices / Modern Applications Integrate with Traditional System M3M2 ACL Event Hub M1 ACL • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! Traditional App
  7. 7. Use Case Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense “Modern Apps”Traditional Apps (Legacy)
  8. 8. MessageMessageMessageMessage MessageMessage Properties - Message Message Message A1 A2 A3 Message B1 B2 B3 B4 A1 A2 A3 B B1 B2 B3 B4 Table A A1 A2 A3 Table B B1 B2 B3 B4 FlatDB Model Aggregate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  9. 9. Properties - Latency Traditional System Event Hub Data Flow RDBMS latency latency Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  10. 10. Properties – Anti-Corruption Layer (ACL) Traditional System Event Hub Data Flow RDBMS Traditional System Event Hub Data Flow RDBMS ACL ACL Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Examples: • View Layer • Storage Procedure • JSON Support in DB • … Examples: • StreamSets • Kafka Connect • Kafka Streams / KSQL • … Database Dataflow
  11. 11. Blueprints Oracle RDBMS => Apache Kafka
  12. 12. Blueprints Oracle RDBMS => Apache Kafka (DB-K) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Que with bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5
  13. 13. DB-K_1: Polling of RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  14. 14. Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_1: Polling of RDBMS table/view Kafka Connect with JDBC Source Connector
  15. 15. Kafka Connect & JDBC Connector • Many connectors available • Single Message Transforms (SMT) • declarative style, simple data flows • framework is part of Apache Kafka
  16. 16. DB-K_1 – Configure JDBC Connector #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
  17. 17. DB-K_2: Change Data Capture (CDC) on RDBMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  18. 18. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy DB-K_2: Change Data Capture (CDC) on RDBMS Using Oracle GoldenGate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Alternatives: StreamSets Data Collector Attunity Debezium …
  19. 19. DB-K_3: Polling of RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  20. 20. Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow DB-K_3: Polling of RDBMS API StreamSets invokes Oracle Rest Data Service Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  21. 21. Oracle REST Data Services (ORDS) • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
  22. 22. DB-K_3 – Setup ORDS (I) ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
  23. 23. DB-K_3 – Setup ORDS (II) ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name ''emailAddress'' VALUE cu.email), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''id'' VALUE li.product_id, ''name'' VALUE li.product_name, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
  24. 24. StreamSets Data Collector • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing • custom sources, sinks, processors • Monitoring and Error Detection
  25. 25. DB-K_4: Produce to Event Hub from RDBMS Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  26. 26. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Native Kafka Producer using Java in DB Does not feel right!
  27. 27. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Invoke REST Proxy from PL/SQL Invoking a REST Service from DB not well-supported
  28. 28. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon … DB-K_4: Produce to Event Hub from RDBMS Oracle Big Data SQL integrates with Kafka Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  29. 29. DB-K_5: RDBMS Queue with bridge to Event Hub Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  30. 30. DB-K_5: RDBMS Queue with bridge to Event Hub Oracle Advanced Queuing & Kafka Connect JMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  31. 31. BEGIN dbms_aqadm.create_queue_table ( queue_table => 'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
  32. 32. curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.princ ipal=order_processing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonS impleConverter`" } }' DB-K_5 – Setup AQ and Kafka Connect (I)
  33. 33. DB-K_5: RDBMS Queue with bridge to Event Hub Oracle AQ with Kafka API & MirrorMaker Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense AQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing
  34. 34. Blueprints Apache Kafka => Oracle RDBMS
  35. 35. Blueprints Apache Kafka => Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4
  36. 36. K-DB_1: Write to RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  37. 37. Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_1: Write to RDBMS table/view Kafka Connect and JDBC Sink Connector
  38. 38. K-DB_2: Write over RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  39. 39. Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_2: Write over RDBMS API Kafka Connect invokes Oracle Rest Data Service
  40. 40. DB-K_2 – Setup ORDS (I) ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
  41. 41. K-DB_3: Consume from Event Hub Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  42. 42. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon… K-DB_3: Consume from Event Hub Oracle Big Data SQL exposes topic as table Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  43. 43. K-DB_4: Event Hub with bridge to RDBMS queue Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  44. 44. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ K-DB_4: Event Hub with bridge to RDBMS queue Oracle Advanced Queuing & Kafka Connect JMS Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  45. 45. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing K-DB_4: Event Hub with bridge to RDBMS queue Oracle AQ with Kafka API & MirrorMaker Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  46. 46. Summary
  47. 47. Summary Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 https://github.com/gschmutz/various-demos/tree/master/bidirectional-integration-oracle-kafka DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Queue with bridge to Event Hub
  48. 48. Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
  49. 49. Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service sEvent Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream

×