Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Page 1Gunther Pippèrr © 2014 http://www.pipperr.de
ORACLE NOSQL – A NEW ALTERNATIVE
FOR THE ORACLE RDBMS DATABASE?
Just an...
Page 2Gunther Pippèrr © 2014 http://www.pipperr.de
Gunther Pippèrr – IT-Architect – Consultant
• High-Tech
• Real Estate
•...
Page 3Gunther Pippèrr © 2014 http://www.pipperr.de
LinkedIn Contact
https://www.linkedin.com/pub/gunther-pipp%C3%A8rr/5b/6...
Page 4Gunther Pippèrr © 2014 http://www.pipperr.de
Use Case
Our challenge to solve
Page 5Gunther Pippèrr © 2014 http://www.pipperr.de
Our Use Case – Internet Simple Password Store
 We like to build:
– A s...
Page 6Gunther Pippèrr © 2014 http://www.pipperr.de
How we can store the Data?
Performance, Security, Availability
Page 7Gunther Pippèrr © 2014 http://www.pipperr.de
OracleRDBMSOracleNoSQL
Shared Everything
Shared Nothing
Instance
SQL*Ne...
Page 8Gunther Pippèrr © 2014 http://www.pipperr.de
Use for each Problem the right tool
 NoSQL – General Concept
– More a ...
Page 9Gunther Pippèrr © 2014 http://www.pipperr.de
NOSQL – Definition
 A real NoSQL database is:
– none relational
– Sche...
Page 10Gunther Pippèrr © 2014 http://www.pipperr.de
Classification Oracle NoSQL in the NoSQL World
 Key-Value
– One key p...
Page 11Gunther Pippèrr © 2014 http://www.pipperr.de
Idea - Key-Value store database
– Oracle NoSQL – a pure Key Value Stor...
Page 12Gunther Pippèrr © 2014 http://www.pipperr.de
The Mayor –Minor Key Concept
 Key – Composed of two components
Data
K...
Page 13Gunther Pippèrr © 2014 http://www.pipperr.de
Implementation
 Key = Hash Mayor Key => Partition Mapping
– When you ...
Page 14Gunther Pippèrr © 2014 http://www.pipperr.de
Query the store
 Java API
– Alternative C API (Enterprise Feature!)
–...
Page 15Gunther Pippèrr © 2014 http://www.pipperr.de
Variants / versions of the Oracle NoSQL
 CC – Community Edition
– Ope...
Page 16Gunther Pippèrr © 2014 http://www.pipperr.de
Integration in the Hadoop Ecosystem
Hadoop Map Reduce Read the Data fr...
Page 17Gunther Pippèrr © 2014 http://www.pipperr.de
Evolution form Key-Value / Avro to table
 Release 1 – Key Value
– Pur...
Page 18Gunther Pippèrr © 2014 http://www.pipperr.de
The architecture of the Oracle
NoSQL database in detail
A distributed ...
Page 19Gunther Pippèrr © 2014 http://www.pipperr.de
The store = the database
 Store – all nodes together over all storage...
Page 20Gunther Pippèrr © 2014 http://www.pipperr.de
Overview
 Depending on the replications factor the data will
duplicat...
Page 21Gunther Pippèrr © 2014 http://www.pipperr.de
The distribution of the data in the store
 The client API hashes the ...
Page 22Gunther Pippèrr © 2014 http://www.pipperr.de
The distribution of the data in the store
 Read is possible from each...
Page 23Gunther Pippèrr © 2014 http://www.pipperr.de
Internal management
 Admin Service for the internal management
– Sepa...
Page 24Gunther Pippèrr © 2014 http://www.pipperr.de
Consistency => all nodes see the same data at the same time
Availabili...
Page 25Gunther Pippèrr © 2014 http://www.pipperr.de
The engine (1)
 The Java Version of the Berkeley DB
– JE Version 5.0....
Page 26Gunther Pippèrr © 2014 http://www.pipperr.de
The engine (2)
 Particularity of the Berkeley DB:
• Redo Log entries ...
Page 27Gunther Pippèrr © 2014 http://www.pipperr.de
Example – Analyze the Data files
 Read the size of the DB with the Be...
Page 28Gunther Pippèrr © 2014 http://www.pipperr.de
Installation
Unzip Java archive, create directory ,
ready?
Download So...
Page 29Gunther Pippèrr © 2014 http://www.pipperr.de
The planning of a NoSQL Installation - Network
 Network Infrastructur...
Page 30Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL installation - Ports
 Necessary ports - Network I...
Page 31Gunther Pippèrr © 2014 http://www.pipperr.de
Oracle NoSQL Client Connect
 Access from the Client to the „registry ...
Page 32Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL Installation - Hardware
 CPU and Memory
– Cache S...
Page 33Gunther Pippèrr © 2014 http://www.pipperr.de
Planning of a NoSQL Installation – Disk space
 Disk space
– Calculati...
Page 34Gunther Pippèrr © 2014 http://www.pipperr.de
Installation
 Preparation on each Node
– Java JDK installation and cr...
Page 35Gunther Pippèrr © 2014 http://www.pipperr.de
Install our own local test system
 Download from oracle.com and extra...
Page 36Gunther Pippèrr © 2014 http://www.pipperr.de
Work with the NoSQL Store
Maintenance and operation
Page 37Gunther Pippèrr © 2014 http://www.pipperr.de
Maintenance (1) – command Line
 Kv – For the Store Administration
 k...
Page 38Gunther Pippèrr © 2014 http://www.pipperr.de
Maintenance (2) – Web interface
– Web interface based on a internal je...
Page 39Gunther Pippèrr © 2014 http://www.pipperr.de
JMX Interface
 Start JMC – Java Mission Control
– JAVA_HOME/bin/jmc.e...
Page 41Gunther Pippèrr © 2014 http://www.pipperr.de
Reliability
Availability and Backup
Page 42Gunther Pippèrr © 2014 http://www.pipperr.de
Backup per Snapshot possible
 Backup:
– A complete Store can be saved...
Page 43Gunther Pippèrr © 2014 http://www.pipperr.de
Performance
I/O and Network
Page 44Gunther Pippèrr © 2014 http://www.pipperr.de
Performance considerations
 read
– Network
• All records must be read...
Page 45Gunther Pippèrr © 2014 http://www.pipperr.de
Security?
Oracle NoSQL and Security?
Page 46Gunther Pippèrr © 2014 http://www.pipperr.de
Security?
 Version 2 ?
 The developer has to care about it – not imp...
Page 47Gunther Pippèrr © 2014 http://www.pipperr.de
Read and write to the store
with Java
No SQL necessary – all is done i...
Page 48Gunther Pippèrr © 2014 http://www.pipperr.de
Client
Software
Client Driver : Hashing der ID
Store over n Storage No...
Page 49Gunther Pippèrr © 2014 http://www.pipperr.de
First – Connect to the store (1)
 Create the connection to the store
...
Page 50Gunther Pippèrr © 2014 http://www.pipperr.de
First – Connect to the store (2)
 Create the connection to the store
...
Page 51Gunther Pippèrr © 2014 http://www.pipperr.de
First Example - Write the data
 Address the DB object over the key
 ...
Page 52Gunther Pippèrr © 2014 http://www.pipperr.de
First Example - Read the data over the key
 Create the key
 Read dat...
Page 53Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 54Gunther Pippèrr © 2014 http://www.pipperr.de
TRANSACTION CONCEPTS
Are we reading the last version of data?
Page 55Gunther Pippèrr © 2014 http://www.pipperr.de
The consistency of the data – Transactionality
 In one session
transa...
Page 56Gunther Pippèrr © 2014 http://www.pipperr.de
The write consistency in the Store - Durability
 The developer is res...
Page 57Gunther Pippèrr © 2014 http://www.pipperr.de
Durability Policies
 Commit-Policy for the Master and the replicates
...
Page 58Gunther Pippèrr © 2014 http://www.pipperr.de
The read consistency in the store - Consistency
 The developer is res...
Page 59Gunther Pippèrr © 2014 http://www.pipperr.de
Consistency
 Consistency.ABSOLUTE
– The session can read only from th...
Page 60Gunther Pippèrr © 2014 http://www.pipperr.de
THE TABLE IS BACK
Structured data access
Page 61Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1)
 Our Key Model
 Our Table Model
User Id ...
Page 62Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1.1)
 Create a table definition in the Store...
Page 63Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1.2)
# Add the primary Key
primary-key -field...
Page 64Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (1)
 Write the source code to use the table d...
Page 65Gunther Pippèrr © 2014 http://www.pipperr.de
Use the table construct (2)
 Read from this table again
// get the Ta...
Page 66Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 67Gunther Pippèrr © 2014 http://www.pipperr.de
INTEGRATION INTO THE HADOOP DATA
WAREHOUSE
Archive your data
Page 68Gunther Pippèrr © 2014 http://www.pipperr.de
Oracle NoSQL – MapReduce Hadoop
 Traditional MapReduce Pattern
 Inpu...
Page 69Gunther Pippèrr © 2014 http://www.pipperr.de
Main Part – Job Configuration
 Define the Input Format Class in the J...
Page 70Gunther Pippèrr © 2014 http://www.pipperr.de
Example
 Demo
Demo
Page 71Gunther Pippèrr © 2014 http://www.pipperr.de
Conclusion
Are the benefits worth the effort?
Page 72Gunther Pippèrr © 2014 http://www.pipperr.de
Why we should use the Oracle NoSQL? (1)
 What is really important for...
Page 73Gunther Pippèrr © 2014 http://www.pipperr.de
Why we should use the Oracle NoSQL? (2)
 What is for a strategic deve...
Page 74Gunther Pippèrr © 2014 http://www.pipperr.de
Use Cases
 Use Cases
– Personalization of web pages
– Authentication
...
Page 75Gunther Pippèrr © 2014 http://www.pipperr.de
Case Study: Web Application
 In a Web application each user can have ...
Page 76Gunther Pippèrr © 2014 http://www.pipperr.de
Architecture of a NoSQL Stores
3 Storage Nodes (SN) on three servers
w...
Page 77Gunther Pippèrr © 2014 http://www.pipperr.de
Direct Comparison Oracle RDBMS NoSQL
 ~35 years of development
 Pow...
Page 78Gunther Pippèrr © 2014 http://www.pipperr.de
Fragen
Oracle NoSQL
NO:SQL
Page 79Gunther Pippèrr © 2014 http://www.pipperr.de
Sources
 Oracle - ORACLE
– English
http://www.oracle.com/technetwork/...
Nächste SlideShare
Wird geladen in …5
×

Oracle nosql twjug-oktober-2014_taiwan_print_v01

1.505 Aufrufe

Veröffentlicht am

Oracle NoSQL Development TWJUG - Taiwan Java User Group Treffen - 04.10.2014 - Oracle Taipei Office

Developing high performance application with the Oracle NoSQL Database

The Oracle NoSQL Database is a distributed key value store. Based on the proven Oracle Java Berkley DB Oracle developed an new database, the Oracle NoSQL.
This new database, also available as free community edition, can be optimal used in high performance applications as persistent cache or for queuing usages.

A easy to use Java API helps to implement the read and write operations on the store with low programming effort.

The Java API also allows the developer to choose the transactional behavior directly in his own code. The data of the store can be defined with the help of Avro or over a simple table construct, but is also possible to use the Key-Value store schema less. With the new Version 3 the connecting to the store can be encrypted and secured to satisfy high security requirements.
In the talk we will show the general architecture , the first steps to implement the store and how to develop the first lines of code to use the store in your own software.

Veröffentlicht in: Daten & Analysen
  • Als Erste(r) kommentieren

Oracle nosql twjug-oktober-2014_taiwan_print_v01

  1. 1. Page 1Gunther Pippèrr © 2014 http://www.pipperr.de ORACLE NOSQL – A NEW ALTERNATIVE FOR THE ORACLE RDBMS DATABASE? Just another Key/Values store? Java Meeting – 10/04/2014
  2. 2. Page 2Gunther Pippèrr © 2014 http://www.pipperr.de Gunther Pippèrr – IT-Architect – Consultant • High-Tech • Real Estate • Utility • Communications • IT Systems Architect • Technical project manager • Design and implementation of database applications • Design and implementation of IT infrastructures for data management Gunther Pippèrr is working intensively for more than 15 years with the products of Oracle in database / application server and document management. Mr. Pippèrr has deep knowledge about the structure of complex IT architecture and successfully implemented this in practice. Mr. Pippèrr has a degree in computer science as Dipl. Ing. Technische Informatik (FH) - FH Weingarten. Software manufacture telecommunication - Architect, database design and implementation of a mass data processing system for communication data German technology group - Architect and project management, database design and implementation of an order management for control of external contractors for the language service provider of a German technology group. Utility Consulting - Architect and technical project responsibility for a smart metering portal for meter data collection and asset management Architect and technical project responsibility for IT infrastructure projects, such as: - Central data storage for Munich hotel group with over 25 hotels worldwide - Data collector for monitoring Russian cable operator - Redundant database cluster infrastructure for various Web applications such as insurance portals , fond platforms, loyalty cards Huge Munich building contractor - CRM and tendering portal Industry Expertise Background Functional Expertise Selected Experience • German: http://www.pipperr.de • English: http://orapowershell.codeplex.com Web
  3. 3. Page 3Gunther Pippèrr © 2014 http://www.pipperr.de LinkedIn Contact https://www.linkedin.com/pub/gunther-pipp%C3%A8rr/5b/623/23
  4. 4. Page 4Gunther Pippèrr © 2014 http://www.pipperr.de Use Case Our challenge to solve
  5. 5. Page 5Gunther Pippèrr © 2014 http://www.pipperr.de Our Use Case – Internet Simple Password Store  We like to build: – A secure password store for identity credentials App REST Interface PWD Save User Id PWD Name Username Password URL Access Token URL PWD Name PWD Name Value Value Value Value Key Mayor Minor
  6. 6. Page 6Gunther Pippèrr © 2014 http://www.pipperr.de How we can store the Data? Performance, Security, Availability
  7. 7. Page 7Gunther Pippèrr © 2014 http://www.pipperr.de OracleRDBMSOracleNoSQL Shared Everything Shared Nothing Instance SQL*Net Central Storage Storage Node Storage Node Storage Node Distributed Storage Client SQL Client Java API RMI
  8. 8. Page 8Gunther Pippèrr © 2014 http://www.pipperr.de Use for each Problem the right tool  NoSQL – General Concept – More a approach for data management and processing then a special technologies  Basic Concept – Distributed, scalable solutions (Scale-out Concept) • CAP Theorem – Consistency – Availability - Partition Tolerance • BASE - Basically Available, Soft-state, Eventually versus ACID – Atomicity, Consistency, Isolation, Durability – Highly specialized solutions for exactly one problem with the handling of very large or very quickly required data • Dozens of solutions available in the market
  9. 9. Page 9Gunther Pippèrr © 2014 http://www.pipperr.de NOSQL – Definition  A real NoSQL database is: – none relational – Schema free – distributed multi-node architecture • Goal: a shared nothing cluster solution – Simple replications mechanism – A easy API ( Not only SQL = NoSQL) – Not necessarily a ACID Consistency Model – Open Source But – the major disadvantage => Usually no easy query language => Complex joins are not so easy to implement
  10. 10. Page 10Gunther Pippèrr © 2014 http://www.pipperr.de Classification Oracle NoSQL in the NoSQL World  Key-Value – One key points to a set of data • Oracle  Document oriented – All data of a data set is organized in one document • Lotus Notes  Graph – networked structures • Neo4J
  11. 11. Page 11Gunther Pippèrr © 2014 http://www.pipperr.de Idea - Key-Value store database – Oracle NoSQL – a pure Key Value Store • A distributed and scalable high available database • Consistent hashing approach Key Value Key Value Key Value Key Value Data store is distributed over all nodes Intelligent driver • Implementation of the Hash Mechanisms • Knows the status of the store • Read the partition Map at the first connect to the store Client Software Key Value Key Value Key Value Key Value Key Value Key Value Key Value Key Value One ACID Transaction is possible for all records of a mayor key with one Single API call.
  12. 12. Page 12Gunther Pippèrr © 2014 http://www.pipperr.de The Mayor –Minor Key Concept  Key – Composed of two components Data Key Mayor Minor String Byte Array Partitioning over the mayor key Example: User identifier Password Name / User Name Access Token / URL Value user name Token value The structure of the data is NOT self-describing
  13. 13. Page 13Gunther Pippèrr © 2014 http://www.pipperr.de Implementation  Key = Hash Mayor Key => Partition Mapping – When you create the store in the first step a fixed count of partitions will be created • Each Partition corresponds essentially to his own Berkeley DB „Database“ • The Hash of the key determine the partition • The partition will be stored redundant over the store Client Driver : Hashing the ID Store over n Storage Nodes Partition 1 Partition 2 Partition 3 Partition … Partition N Replication Group Replication Group Replication Group MD5(major key) % n Partitions →Partition ID Replication Group = Master + Replicates
  14. 14. Page 14Gunther Pippèrr © 2014 http://www.pipperr.de Query the store  Java API – Alternative C API (Enterprise Feature!) – Command Line Interface for maintenance  The key and ONLY the Key can be queried – For a search in the values all the data in the store have to be read – For a count you have to iterate over all values
  15. 15. Page 15Gunther Pippèrr © 2014 http://www.pipperr.de Variants / versions of the Oracle NoSQL  CC – Community Edition – Open Source Code • Open Source with the GNU Affero General Public License – Support is possible on a annual basis  EE – Enterprise Edition – Full support – Support of Oracle external tables - exchange data with the normal Oracle database – Oracle Event Processing – RDF Adapter – SNMP Monitoring – Jena Adapter (Graph/SPARQL) – Since Version 3 - Security options with the Oracle Wallet
  16. 16. Page 16Gunther Pippèrr © 2014 http://www.pipperr.de Integration in the Hadoop Ecosystem Hadoop Map Reduce Read the Data from the NoSQL Store Hadoop File system Store results 11g 12c Read as External Table pure Java Hive Read as External Table Only EE Edition of the Oracle NoSQL!
  17. 17. Page 17Gunther Pippèrr © 2014 http://www.pipperr.de Evolution form Key-Value / Avro to table  Release 1 – Key Value – Pure Key-Value concept – No „Data Dictionary“ – Entire logic in the application  Release 2 – Avro – Schema Definition with Apache Avro Standard – Declaration of the structure of the Key-Value Data inside the database AND in the application necessary  Release 3 – Tables – Structure definition as table • A secondary index on columns of the table is possible Value
  18. 18. Page 18Gunther Pippèrr © 2014 http://www.pipperr.de The architecture of the Oracle NoSQL database in detail A distributed key value store
  19. 19. Page 19Gunther Pippèrr © 2014 http://www.pipperr.de The store = the database  Store – all nodes together over all storage nodes Server 1 Storage Node 1 Server 2 Storage Node 2 Server 4 Storage Node 4 Server 3 Storage Node 3 Example of a store with a replication factor of 3 1 1 13332 2 24 44 Master 1 Replika sn1 Replika sn1 The replication factor is defined as „master + count replicates“. In our example one master and two replicates = replication factor 3 Zone = one store node environment Master Master Master Replication Group = Master + Replicate Shared A Shared D Zone
  20. 20. Page 20Gunther Pippèrr © 2014 http://www.pipperr.de Overview  Depending on the replications factor the data will duplicated over the store  The count of partitions will be distributed over all master nodes  A topology describes the whole store  Example– 4 nodes - replication factor 3 – 800 partitions – One Master per node + two replica per storage node – 800 partitions distributed over the four master Nodes • 200 for each master – Inserts only over the master node • The Master Node distribute the data to the replicas – It is possible to read the data from each node The count of partition can not be changed after the create of the store!
  21. 21. Page 21Gunther Pippèrr © 2014 http://www.pipperr.de The distribution of the data in the store  The client API hashes the key und distributes the Key over the count of partitions Java Code Java API Key 7 B1A1 1 2 8 1 1 D1C1 9 1 0 1 2 88 B2B3 Insert The hash of the key defines the partition AND The hash defines the master node holds the appropriate partition.
  22. 22. Page 22Gunther Pippèrr © 2014 http://www.pipperr.de The distribution of the data in the store  Read is possible from each replicate Java Code Java API Key read 7 B1A1 1 2 8 1 1 D1C1 9 1 0 1 2 88 B2B3 The hash of the key defines the partition AND The hash defines the master node, holds the appropriate partition.
  23. 23. Page 23Gunther Pippèrr © 2014 http://www.pipperr.de Internal management  Admin Service for the internal management – Separate admin database – Separate port  The Admin Service is responsible for implementing of the Master – Slave concept – If one Node fails a new master will be defined
  24. 24. Page 24Gunther Pippèrr © 2014 http://www.pipperr.de Consistency => all nodes see the same data at the same time Availability => a guarantee that every request receives a response about whether it was successful or failed Partition tolerance => the system continues to operate despite arbitrary message loss or failure of part of the system Availability  What happens in case of a failure of a storage node? – If the Master is unavailable – it can now not longer be written to the store? • Admin Process supervise the whole store – Has his own Admin Database – Recognizes errors – Select from the remaining Replicas the replica with the highest LSN (log sequence number) for a new master If the developer decide to use ReplicaAckPolicy.ALL you will get an error message if a node fails! The developer has to choose : • Consistency and Availabitlity OR • Availabitlity and Partition Tolerance
  25. 25. Page 25Gunther Pippèrr © 2014 http://www.pipperr.de The engine (1)  The Java Version of the Berkeley DB – JE Version 5.0.83 - NoSQL 2.2.18 – JE Version 6.0.14 - NoSQL 3.0.14 – Each partition is a database – Berkeley DB Replication for the master slave concept – The DB files can be analyzed with the Berkeley DB classes Tipp: You can configure the original Berkeley DB Mechanism per Storage Node See: http://www.pipperr.de/dokuwiki/doku.php?id=nosql:log_file_verhalten_oracle_nosql_db_11gr2 Show the version with: java -classpath ".libkvclient.jar;.libkvstore.jar;.libje.jar" com.sleepycat.je.util.DbVerify -V
  26. 26. Page 26Gunther Pippèrr © 2014 http://www.pipperr.de The engine (2)  Particularity of the Berkeley DB: • Redo Log entries and the data are stored to the same „data“ file • Background Jobs take care about the cleaning of the unnecessary entries je.log.bufferSize Minimal Size of the log je.log.fileMax Maximal Size of the log je.cleaner.minUtilization % must be used of this file je.cleaner.minFileUtilization Clean Logfile, if this value is reached je.cleaner.bytesInterval Start cleaning if x Bytes written je.cleaner.minAge Only clean if x files are older then the latest je.cleaner.threads Number of Cleaner Threads
  27. 27. Page 27Gunther Pippèrr © 2014 http://www.pipperr.de Example – Analyze the Data files  Read the size of the DB with the Berkeley classes: – Ensure the correct class path to the jar files • export KVLIB=/opt/oracle/produkt/11.2.0/kv-2.1.8/lib • export KVCLASS=$KVLIB/kvclient.jar:$KVLIB/kvstore.jar:$KVLIB/je.jar – Path to the data files • export JEENV=/opt/oracle/kvdata/GPIDB/sn1/rg1-rn1/env/ – Analyze with: java -classpath $KVCLASS com.sleepycat.je.util.DbSpace -h $JEENV File Size (KB) % Used -------- --------- ------ 00000000 110808 21 TOTALS 110808 21 (LN size correction factor: NaN)
  28. 28. Page 28Gunther Pippèrr © 2014 http://www.pipperr.de Installation Unzip Java archive, create directory , ready? Download Software from : http://www.oracle.com/technetwork/database/database-technologies/nosqldb/overview/index.html
  29. 29. Page 29Gunther Pippèrr © 2014 http://www.pipperr.de The planning of a NoSQL Installation - Network  Network Infrastructure – Physic • High performante internal communication between the nodes necessary – pay attention for low latencies in the network » Keyword Infiband » Own 10Gbit Network? – Define TCP Ports • For the internal communication of the Storage nodes • For the communication of the Clients with the Master nodes – One Port for external connection + one Port for the Admin Console + ServicePortRange for the RMI Connection from the client – Maintainability • SSH between the den Storage Nodes – Script for maintenance over many nodes Define some company standard , with the time you will get a lot of environments!
  30. 30. Page 30Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL installation - Ports  Necessary ports - Network Infrastructure Ports for one Storage Node 5000 Application Admin Client -port 5001 HTTP Admin Client -admin 5010-5020 Replication -harange … -servicerange RMI Port for the Client communication 5021-5040 … Parameter when equipped with the "makebootconfig" haPortRange servicePortRange Parameter for changing with the "change-policy -params " registry portApplication
  31. 31. Page 31Gunther Pippèrr © 2014 http://www.pipperr.de Oracle NoSQL Client Connect  Access from the Client to the „registry port“ – For example Port 5000 on node A  The query is take through over the Java RMI protocol on the auf den respective node – Define enough remote ports! -servicerange -port Node A Node B Node C Client Port 5000 ServiceRange Port 5021 ServiceRange Port 5021 ServiceRange Port 5021 Firewall!
  32. 32. Page 32Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL Installation - Hardware  CPU and Memory – Cache Size can be adjusted from 2 until n GB • Watch carefully GC effects of Java – Oracle NoSQL is explicit optimized for clean, small objects sizes to not overwhelm the Java garbage collector  Storage Infrastructure – One Store Node can server also more then one Master und replicates • If you have more then one Storage node on a server you should use separate disks/luns to have best I/O on the server – In the Background the “Cleaner” processes need some I/O resources
  33. 33. Page 33Gunther Pippèrr © 2014 http://www.pipperr.de Planning of a NoSQL Installation – Disk space  Disk space – Calculation is not entirely trivial • The Berkeley DB store all actions like the redo data in the same file • Not until some Thresholds are reached the data files will be optimized • => The delete of data enlarge the store in the first step! Rule of thumb: Pro Master/ Replika => Count of Transactions (Insert/Update/Delete) * (Net Data Size + Overhead) => Filling of the data files per Default optimized to 40%
  34. 34. Page 34Gunther Pippèrr © 2014 http://www.pipperr.de Installation  Preparation on each Node – Java JDK installation and create Store User – Setup SSH Connection – Create the KVROOT (Data Storages!)  Oracle NoSQL installation – Extract the code – create the KVHOME – Create the Store Node Base directory (KVROOT) – Create the Boot configuration – Start the Admin Node – Define a plan for the new store • Define the database topologies – Rollout the plan – test Per Script: 5min
  35. 35. Page 35Gunther Pippèrr © 2014 http://www.pipperr.de Install our own local test system  Download from oracle.com and extract the zip file to a directory  Create a local store and start the NoSQL Database  Start a admin Shell REM go to your DB directory cd R:KVSTORE REM set the libraries Path set KVHOME=D:kv-3.0.14 REM Create the Store java -jar %KVHOME%libkvstore.jar kvlite -root R:KVSTORE -store kvstore -host localhost -port 5000 -admin 5001 java -jar %KVHOME%libkvstore.jar runadmin -port 5000 -host localhost kv-> ping … Pinging components of store kvstore based upon topology sequence #14 … Rep Node [rg1-rn1] Status: RUNNING,MASTER at sequence number: 39 ha
  36. 36. Page 36Gunther Pippèrr © 2014 http://www.pipperr.de Work with the NoSQL Store Maintenance and operation
  37. 37. Page 37Gunther Pippèrr © 2014 http://www.pipperr.de Maintenance (1) – command Line  Kv – For the Store Administration  kvshell – Select / insert data kvshell-> put -key "/Gunther/name" -value "Pipperr" Operation successful, record inserted. kvshell-> get -key "/Gunther/name" Pipperr
  38. 38. Page 38Gunther Pippèrr © 2014 http://www.pipperr.de Maintenance (2) – Web interface – Web interface based on a internal jetty server http://nosqldb03:5005/
  39. 39. Page 39Gunther Pippèrr © 2014 http://www.pipperr.de JMX Interface  Start JMC – Java Mission Control – JAVA_HOME/bin/jmc.exe – Connect over the „registry port“
  40. 40. Page 41Gunther Pippèrr © 2014 http://www.pipperr.de Reliability Availability and Backup
  41. 41. Page 42Gunther Pippèrr © 2014 http://www.pipperr.de Backup per Snapshot possible  Backup: – A complete Store can be saved consistent per Snapshot – pay attention for enough disk space!  Restore – A Store can be completely reset • Create a Store snapshot • Stop the Store • Provide the Snapshot (Link in the File system for example) • Start the Store, Storage Nodes read die Data, create the data again and DELETE the snapshot! – Use as a kind of import • The generated files can be read from an other store
  42. 42. Page 43Gunther Pippèrr © 2014 http://www.pipperr.de Performance I/O and Network
  43. 43. Page 44Gunther Pippèrr © 2014 http://www.pipperr.de Performance considerations  read – Network • All records must be read over the network  write – I/O • Background process need enough recourse for the optimization of the data files! ALL record are processed on the client side! A simple Count(*) have to read all data!
  44. 44. Page 45Gunther Pippèrr © 2014 http://www.pipperr.de Security? Oracle NoSQL and Security?
  45. 45. Page 46Gunther Pippèrr © 2014 http://www.pipperr.de Security?  Version 2 ?  The developer has to care about it – not implemented  Since Version 3 – SSL encoding – Password security • But Wallet usage is a EE feature!
  46. 46. Page 47Gunther Pippèrr © 2014 http://www.pipperr.de Read and write to the store with Java No SQL necessary – all is done in Java
  47. 47. Page 48Gunther Pippèrr © 2014 http://www.pipperr.de Client Software Client Driver : Hashing der ID Store over n Storage Nodes Partition 1 Partition 2 Partition 3 Partition … Partition N Replication Group Replication Group Replication Group MD5(major key) % n Partitions →Partition ID Replication Group = Master + Replication Node Key Value Key Value Partition … Intelligent driver • Implementation of the Hash Mechanisms • Knows the status of the store • Read the partition Map at the first connect to the store Overview
  48. 48. Page 49Gunther Pippèrr © 2014 http://www.pipperr.de First – Connect to the store (1)  Create the connection to the store • Set the store properties with the class KVStoreConfig Store Name // Node liste String[] kvhosts = {"nosqldb01:5000","nosqldb02:5000","nosqldb03:5000"} // Config erzeugen KVStoreConfig kvconfig = new KVStoreConfig("GPIDB", kvhosts); Port Node
  49. 49. Page 50Gunther Pippèrr © 2014 http://www.pipperr.de First – Connect to the store (2)  Create the connection to the store • Define the Read and Write Rules // Define Consistency kvconfig.setConsistency(Consistency.NONE_REQUIRED); // Define Durabiltiy kvconfig.setDurability(Durability.COMMIT_NO_SYNC); // Connect to the store KVStore kstore = KVStoreFactory.getStore(kvconfig); Rule read Rules write Open connection to the Store of a Factory Do not forget to close the Store! kstore.close();
  50. 50. Page 51Gunther Pippèrr © 2014 http://www.pipperr.de First Example - Write the data  Address the DB object over the key  Create the key  Write the data Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME"); Mayor Key Minor Key // Wert als Byte Array anlegen byte[] b = ("gpipperr@googelmail.com").getBytes(); // Wert in den Store schreiben kvstore.put(k, Value.createValue(b)); Write the data to the store
  51. 51. Page 52Gunther Pippèrr © 2014 http://www.pipperr.de First Example - Read the data over the key  Create the key  Read data Key k = Key.createKey("USER_ID", "PWD_NAME_GOOGLE/USERNAME"); Mayor Key Minor Key // Read the Data from the Store with the key ValueVersion vv = kvstore.get(k); // Read the Daten Value Value vdata = vv.getValue(); // Recreate the Values String username= new String(vdata.getValue()); // Work with the data System.out.println(username); read
  52. 52. Page 53Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  53. 53. Page 54Gunther Pippèrr © 2014 http://www.pipperr.de TRANSACTION CONCEPTS Are we reading the last version of data?
  54. 54. Page 55Gunther Pippèrr © 2014 http://www.pipperr.de The consistency of the data – Transactionality  In one session transactionality for write one mayor key is possible  The developer has to decide when a record is apply as written! read  The developer has to decide if the read record is actual write FastSave Durability Consistency Performance Performance Latest Version first available answer On all nodes to disk On one node to the cache FastSave
  55. 55. Page 56Gunther Pippèrr © 2014 http://www.pipperr.de The write consistency in the Store - Durability  The developer is responsible for the write consistency!  Will be defined at the connect to the store – Class oracle.kv.Durability • Commit-Policy for the Master • Commit-Policy for the replicas • Replica Acknowledgement-Policy FastSave Performance On all nodes to disk On one node to the cache
  56. 56. Page 57Gunther Pippèrr © 2014 http://www.pipperr.de Durability Policies  Commit-Policy for the Master and the replicates – SyncPolicy.SYNC • Wait until the Storage Node write the transaction to the log file and synchronized to the disk – SyncPolicy.WRITE_NO_SYNC • Wait until the Storage Node write the transaction to the log file in the cache (not written to disk!) – SyncPolicy.NO_SYNC • Don’t wait  Replica Acknowledgement – ReplicaAckPolicy.ALL • Wait until all replicates confirm, that all transaction has been written – ReplicaAckPolicy.SIMPLE_MAJORITY • Wait until the simple majority the replicates confirm the transaction – ReplicaAckPolicy.NONE • Don’t wait for the answers of the replicates
  57. 57. Page 58Gunther Pippèrr © 2014 http://www.pipperr.de The read consistency in the store - Consistency  The developer is responsible for the read consistency!  Each key object owns a version information – A kind of transaction ID  The behavior will be defined with the connect to the store – read – Class oracle.kv.Consistency Performance FastSave Latest Version first available answer
  58. 58. Page 59Gunther Pippèrr © 2014 http://www.pipperr.de Consistency  Consistency.ABSOLUTE – The session can read only from the master for this key. – This ensures that the data is relay valid  Consistency.Time – The session can read from a replica – With the Parameter you can define how „old“ the data can be to accepted as valid data  Consistency.Version – The session can read from a replica – The version information will be used instead of the time  Consistency.NONE_REQUIRED – Very result is valid, can be any old
  59. 59. Page 60Gunther Pippèrr © 2014 http://www.pipperr.de THE TABLE IS BACK Structured data access
  60. 60. Page 61Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1)  Our Key Model  Our Table Model User Id PWD Name Username Password URL Access Token URL PWD Name PWD Name Value Value Value Value USER_ID | PWD_NAME | USER_NAME | PASSWORD| URL USER_ID | ACESS_TOCKEN | URL | Value Table USER_PASSWORDS Table USER_ACESS_TOCKENS
  61. 61. Page 62Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1.1)  Create a table definition in the Store with the client utility set KVHOME=D:entwicklunglibrarieskv-ee-3.0.14 java -jar %KVHOME%libkvcli.jar -host localhost -port 5000 -store kvstore #Create table table create -name USER_PASSWORDS # Add table fields add-field -name USER_ID -type STRING add-field -name PWD_NAME -type STRING add-field -name USER_NAME -type STRING add-field -name PASSWORD -type STRING add-field -name URL -type STRING
  62. 62. Page 63Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1.2) # Add the primary Key primary-key -field USER_ID #Leave table Mode exit #create the Table in the Store plan add-table -name USER_PASSWORDS -wait #check the Table Definition show table -name USER_PASSWORDS #insert first Record of data in to the table with the client tools put table -name USER_PASSWORDS -json "{"USER_ID":"gunther@pipperr.de","PWD_NAME":"google.de","USER_NA ME":"gpipperr","PASSWORD":"Password01","URL":"www.google.de"}" #create index on the table plan add-index -name IDX_USER_PASSWORDS -table USER_PASSWORDS -field USER_ID –wait #read the data get table -name USER_PASSWORDS -index IDX_USER_PASSWORDS
  63. 63. Page 64Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (1)  Write the source code to use the table defintion // get the Table API TableAPI tableAPI = kvstore.getTableAPI(); //get a reference to the table Table empTable = tableAPI.getTable("USER_PASSWORDS"); // Get a Row instance Row empRow = empTable.createRow(); // put data in the row empRow.put("empno", 10); // write the data to the store tableAPI.put(empRow, null, null);
  64. 64. Page 65Gunther Pippèrr © 2014 http://www.pipperr.de Use the table construct (2)  Read from this table again // get the Table API TableAPI tableAPI = kvstore.getTableAPI(); //get a reference to the table Table empTable = tableAPI.getTable("emp"); //Create a primary key and assign the field value PrimaryKey key = table.createPrimaryKey(); key.put("userID", 1); //Get the matching row Row row = tableAPI.get(key, new ReadOptions(null, 0, null)); if (row == null) { throw new Error("No matching row found"); } //Print the full row as JSON System.out.println(row.toJsonString(true)); //Access a specific field System.out.println("firstName field as JSON: " + row.get("firstName").toJsonString(false));
  65. 65. Page 66Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  66. 66. Page 67Gunther Pippèrr © 2014 http://www.pipperr.de INTEGRATION INTO THE HADOOP DATA WAREHOUSE Archive your data
  67. 67. Page 68Gunther Pippèrr © 2014 http://www.pipperr.de Oracle NoSQL – MapReduce Hadoop  Traditional MapReduce Pattern  Input Format Class – KVInputFormat.class Mapper Mapper Mapper Mapper Reducer HDFSOracle NoSQL Store KVInputFormat oracle.kv.Key.toString() lib/kvclient.jar must in the class path! Like: hadoop jar secure-pwd-store.jar gpi.secstore.PWDStoreHadoop -libjars $KVHOME/lib/kvclient.jar /tmp/nosqlTest2 Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class oracle.kv.hadoop.KVInputFormat not found at Better solution -put all in one jar fie
  68. 68. Page 69Gunther Pippèrr © 2014 http://www.pipperr.de Main Part – Job Configuration  Define the Input Format Class in the Job Configuratoin // set the Input Format Classe job.setInputFormatClass(KVInputFormat.class); //Parameter for the Input Format Class KVInputFormat.setKVStoreName("KVStore"); String [] kvhostList = {"StoreDB01:5000", "StoreDB02:5000",}; KVInputFormat.setKVHelperHosts(kvhostList); Set the Input Class for the MapReduce Job Define the connection to the Store Default: Key and Value are read as Text Format
  69. 69. Page 70Gunther Pippèrr © 2014 http://www.pipperr.de Example  Demo Demo
  70. 70. Page 71Gunther Pippèrr © 2014 http://www.pipperr.de Conclusion Are the benefits worth the effort?
  71. 71. Page 72Gunther Pippèrr © 2014 http://www.pipperr.de Why we should use the Oracle NoSQL? (1)  What is really important for a strategic development? – Maintainability • Is the product easy to use? – Long-term strategy of the manufacturer • Is a product cycle from 4-5 year possible? – Release and Update Problematic • Java 8 and higher? Linux 7?
  72. 72. Page 73Gunther Pippèrr © 2014 http://www.pipperr.de Why we should use the Oracle NoSQL? (2)  What is for a strategic development really important? – Fits the base concept to my technical needs? • Are my requirements changing with the time? – Costs Oracle promises this: – Version 3 ready – The Software Berkeley DB since year in use
  73. 73. Page 74Gunther Pippèrr © 2014 http://www.pipperr.de Use Cases  Use Cases – Personalization of web pages – Authentication – Stream Processing • machine data Goal: Get for one Key fast as possible the results Requirement: Provide large amounts of data with low latency – Persistent Cache
  74. 74. Page 75Gunther Pippèrr © 2014 http://www.pipperr.de Case Study: Web Application  In a Web application each user can have a very high number of different "bonus offers, vouchers"  This bonus offers are calculated individually according to the profiles of a user on a weekly base – The NoSQL Store holds the UserID as key with a list of all metadata for the bonus offers • Avoid complex joins over the DWH with the user profiles • Immediate display results on the smart phone or in the Web without a time delay – very low latencies Requirement: Provide large amounts of data with low latency – Persistent Cache
  75. 75. Page 76Gunther Pippèrr © 2014 http://www.pipperr.de Architecture of a NoSQL Stores 3 Storage Nodes (SN) on three servers with a Replication Factor of three = three Replication Groups (RG) Server 1 Storage Node 1 SN 1 Server 2 Storage Node 2 SN 2 Server 3 Storage Node 3 SN 3 Rg1- n1 Master 1 Replication Node Replica 1 Replication Node Replica 2 Replication Node Shared A Store Shared B Shared C Zone A Admin Srv Admin Srv Admin Srv Storage Node Agent (SNA) Storage Node Agent (SNA) Storage Node Agent (SNA) Rg2- n1 Rg3- n1 Rg1- n2 Rg2- n2 Rg3- n2 Rg1- n3 Rg2- n3 Rg3- n3
  76. 76. Page 77Gunther Pippèrr © 2014 http://www.pipperr.de Direct Comparison Oracle RDBMS NoSQL  ~35 years of development  Powerful SQL API  Joins  Constraints  Many Features  OLTP  Shared anything approach  A general all purpose weapon Oracle NoSQL  ~2-3 years  Simple Java API  Joins only over the application  ?  Pure data store  Mostly only read operation  Shared Nothing Cluster  Specialized niche solution Oracle RDBMS
  77. 77. Page 78Gunther Pippèrr © 2014 http://www.pipperr.de Fragen Oracle NoSQL NO:SQL
  78. 78. Page 79Gunther Pippèrr © 2014 http://www.pipperr.de Sources  Oracle - ORACLE – English http://www.oracle.com/technetwork/database/database- technologies/nosqldb/documentation/index.html – German http://www.oracle.com/webfolder/technetwork/de/community/dojo/index.ht ml  See also: – German http://www.pipperr.de/dokuwiki/doku.php?id=nosql_datenbank

×