Weitere ähnliche Inhalte Ähnlich wie MySQL JSON Document Store - A Document Store with all the benefits of a Transactional RDBMS (20) Mehr von Olivier DASINI (14) Kürzlich hochgeladen (20) MySQL JSON Document Store - A Document Store with all the benefits of a Transactional RDBMS1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
MySQL Document Store
Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
A Document Store with all the benefits of a Transactional RDBMS
Olivier Dasini
MySQL Principal Solutions Architect EMEA
olivier.dasini@oracle.com
Twitter : @freshdaz
Blog : http://dasini.net/blog
2. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.
2
3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 3
Me, Myself & I
MySQL Geek
Addicted to MySQL for 15+ years!
Playing with databases for 20+ years
MySQL Writer, Blogger and Speaker
Also former : DBA, Consultant, Architect, Trainer, ...
MySQL Principal Solutions Architect EMEA at Oracle
Stay tuned! :
Twitter: @freshdaz
Blog: http://dasini.net/blog
Olivier DASINI
4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 4
The world's most popular open source database
5. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Powers Social
5
6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Powers eCommerce
6
7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Powers FinTech
7
8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Powers SaaS
8
9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Powers the Cloud
9
10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 10
• Data Integrity
– Normalization
– Constraints (Foreign keys, …)
• Atomicity, Consistency, Isolation, Durability
– ACID Compliant
– Transactions
• SQL
– Powerful Query Language
– Schema, Table, Row
– SELECT / UPDATE / DELETE / INSERT
Relational Databases
10
11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 11
• An object that can represent structured data
• Structure is implicit in the document; usually no external/central schema
• Compact, popular and standardized
• Can be represented natively in many languages (JavaScript,Python, etc.)
11
• JSON is a lightweight data-interchange format.
• It is easy for humans to read and write.
• It is easy for machines to parse and generate.
• It is based on a subset of the JavaScript Programming.
• Language, Standard ECMA-262 3rd Edition - December 1999.
• Use for storing and exchanging data.
What is a JSON Document ?
JavaScript Object Notation
12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 12
JSON Support
• Native File Format
– Standardized as ECMA-404 (http://json.org)
– Binary Storage
• Generated Columns
• 20+ Functions
Search Functions
Aggregations Functions
• Query structured data and semi-structured JSON data
13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13
JSON_ARRAY_APPEND()
JSON_ARRAY_INSERT()
JSON_ARRAY()
JSON_CONTAINS_PATH()
JSON_CONTAINS()
JSON_DEPTH()
JSON_EXTRACT()
JSON_INSERT()
JSON_KEYS()
JSON_LENGTH()
JSON_MERGE[_PRESERVE]()
JSON_OBJECT()
JSON_QUOTE()
JSON_REMOVE()
JSON_REPLACE()
JSON_SEARCH()
JSON_SET()
JSON_TYPE()
JSON_UNQUOTE()
JSON_VALID()
JSON_PRETTY()
JSON_STORAGE_SIZE()
JSON_STORAGE_FREE()
JSON_ARRAYAGG()
JSON_OBJECTAGG()
JSON_MERGE_PATCH()
JSON_TABLE()
JSON Functions
Perform operations on JSON documentsPerform operations on JSON documents
https://dev.mysql.com/doc/refman/8.0/en/json-functions.html
14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 14
• Schemaless
– No schema design, No normalization, No Foreign Keys, No data types, …
– Very quick initial development
• Flexible data structure
– Embedded arrays or objects
– Valid solution when natural data can’t be modeled optimally into a relational model
– Objects persistence without the use of any ORM – Mapping Object-Oriented
• JSON
– Close to frontend
– Native in JavaScript
– Easy to learn
NoSQL : Document Store
14
15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
How DBAs see data
15
How Devs see data
{
" G N P " : 2 7 3 1 ,
" _ i d " : " M T Q " ,
" N a m e " : " M a r t i n i q u e " ,
" I n d e p Ye a r " : n u l l ,
" g e o g r a p h y " : {
" R e g i o n " : " C a r i b b e a n " ,
" C o n t i n e n t " : " N o r t h A m e r i c a " ,
" S u r f a c e A r e a " : 11 2 8
} ,
" g o v e r n m e n t " : {
" H e a d O f S t a t e " : " E m m a n u e l M a c r o n " ,
" G o v e r n m e n t F o r m " : " O v e r s e a s D e p a r t m e n t o f F r a n c e "
} ,
" d e m o g r a p h i c s " : {
" P o p u l a t i o n " : 3 9 5 0 0 0 ,
" L i f e E x p e c t a n c y " : 7 8 . 3 0 0 0 0 3 0 5 1 7 5 7 8 1
}
}
16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 16
• Schema (Database)
• Collection
– Group of JSON documents.
– Equivalent of table
• Document
– Set of key-value pairs in JSON format
– Equivalent of row (tuple)
Document Store Databases
16
• CRUD (basic functions of persistent storage)
– CREATE
– READ
– UPDATE
– DELETE
TerminologiesTerminologies
17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 17
Document Oriented Databases
Usability & ScalabilityUsability & Scalability
• Schemaless: No centralized database schema
– Data model enforcement and validation (if any) at application layer
– Simpler schema updates (no ALTER TABLE penalty)
• NoSQL APIs: Simpler programming interfaces
– No specialized language for queries and data manipulation
– Complex queries handled at application layer (no complex SELECTs, JOINs)
– Document in, document out, manipulations at client side
• Scalability, but some drawbacks:
– Limited database features (no foreign keys, no transactions, etc.)
– Weak consistency guarantees
18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 18
Document Store
An easy, straight forward way to work with JSON documents in MySQL
19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 19
RDBMS or NoSQL ?
Why not both ?
20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
• One Extensible database
– Do more with the MySQL database
– Many can manage (with deep skills)
– Stable
– Cost-effective
– Easy to move data between like database
types
– Fewer Drivers
– Few Tools
– SQL works, CRUD works
– Operational and Analytical Together
• Many different databases
– Requires larger skill repertoire, more
complex development …
• Harder to find deep skills
– Many Drivers
– Many Tools
– More effort to share and exchange data
– It’s a lot more work
– Operational and Analytical Separate
One Database Many Models VS Many Databases Many Models
20
21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
NoSQL, MySQL, Why not…
• Have both schema-less AND schema in the same technology stack?
• One that checks all the boxes of all stakeholders:
21
Developers:
[✔] Schemaless or/and Schema
[✔] Rapid Prototyping/Simpler APIs
[✔] Document Model
[✔] Transactions
Operations:
[✔] Performance Management/Visibility
[✔] Robust Replication, Backup, Restore
[✔] Comprehensive Tooling Ecosystem
[✔] Simpler application schema upgrades
Business Owner:
[✔] Don’t lose my data = ACID transactions
[✔] Capture all my data = Extensible/Schemaless
[✔] Products On Schedule/Time to Market = Rapid Development
22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
- A One Stop Shop
• Expensive to manage many data stores
• Better few databases – more flexibility
– More developers
• Can harness both NoSQL and SQL savvy
– More DBAs
• No shortage of highly experienced MySQL DBAs
– Less training – don’t need to learn many products
– Cross data store exchange – easier to move from from docs to tables etc.
– One connector/driver needed for apps
22
Combining Relational and Document StoresCombining Relational and Document Stores
23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Document Store
23
24. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Document Store: Components
• MySQL X PluginMySQL X Plugin
• Introduces X Protocol for relational- and
document operations
• Maps CRUD operations to standard SQL
(relational tables, JSON datatype and
functions)
• X ProtocolX Protocol
• New MySQL client protocol based on top
of industry standard (Protobuf)
• Works for both, CRUD and SQL
operations
• InnoDB ClusterInnoDB Cluster
• Read-Scaling, Write-Scaling, HA
• X DevAPIX DevAPI
• New, modern, async developer API for
CRUD and SQL operations on top of X
Protocol
• Introduces Collections as new Schema obj.
• MySQL ShellMySQL Shell
• Offers interactive X DevAPI mode for app
prototyping
• MySQL ConnectorsMySQL Connectors
• Support for X DevAPI for
• JavaScript, Python, PHP, Java, C#, C++
24
25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Document Store: Architecture
27
26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 34
Document Store
SQL is now optional ! - Write applications using X DevAPIX DevAPI
https://insidemysql.com/mysql-8-0-welcome-to-the-devapi/
27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 35
Welcome to the X DevAPI!
MySQL X devAPI connector for:
• Java
– https://insidemysql.com/connector-j-8-0-11-the-face-for-your-brand-new-document-oriented-database/
• .NET
– https://insidemysql.com/introducing-connector-net-with-full-support-for-mysql-8-0/
• Node.JS
– https://insidemysql.com/introducing-connector-node-js-for-mysql-8-0/
• C++
– https://insidemysql.com/what-is-new-in-connector-c-8-0/
• Python
– https://insidemysql.com/using-mysql-connector-python-8-0-with-mysql-8-0/
• PHP
– https://insidemysql.com/introducing-the-mysql-x-devapi-php-extension-for-mysql-8-0/
• ODBC
– https://insidemysql.com/what-is-new-in-connector-odbc-8-0/
35
Motivation
• We are doing something that has not been done before
• Document databases exist! Relational databases exist!
• We even see databases that support relational and document querying over the same data set
• However we have yet to see a relational database include a document model so that a user can use
document objects alongside their existing relational data.
28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 36
Welcome to the X DevAPI! - PHP Example
36
Writing a program using MySQL 8.0 Document Store
$user = 'my_user';
$passwd = 's0S3kR*T';
$host = 'localhost';
$port = '33060';
$connection_uri = 'mysqlx://'.$user.':'.$passwd.'@'.$host.':'.$port;
$session = mysql_xdevapigetSession($connection_uri);
$schema = $session->getSchema("test");
$collection = $schema->getCollection("my_collection");
$result = $collection->find("Name like :param")->bind(["param" => "Olivier"])->execute();
$docs = $result->fetchAll();
//… print results …
$session->close();
Requirements / Installation
https://dev.mysql.com/doc/apis-php/en/apis-php-mysql-xdevapi.setup.html
https://insidemysql.com/introducing-the-mysql-x-devapi-php-extension-for-mysql-8-0/
29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 37
Welcome to the X DevAPI! - Python Example
https://insidemysql.com/using-mysql-connector-python-8-0-with-mysql-8-0/
37
Writing a program using MySQL 8.0 Document Store
import mysqlx
session = mysqlx.get_session({
"host": "localhost",
"port": 33060,
"user": "my_user",
"password": "s0S3kR*T"
})
schema = session.get_schema("test")
collection = schema.get_collection("my_collection")
result = collection.find("name like :param").bind("param", "Olivier").limit(1).execute()
docs = result.fetch_all()
print("Name: {0}".format(docs[0]["name"]))
session.close()
Connector/Python 8.0 installation
shell> pip install mysql-connector-python
30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 38
Document Store – MySQL Shell
An integrated development & administration shell where all MySQL
products will be available through a common scripting interface
31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Shell 8.0.11+
39Confidential - Oracle Internal 39
MySQL
Server 5.7
MySQL 8.0
Upgrade Checker
Prompt Themes
Auto Completion
&
Command History
MySQL
Server 8.0
Document Store
X DevAPI
InnoDB ClusterSQL CLI
Output Formats
(Table, JSON,
Tabbed)
Batch Execution
JavaScript
Python
SQL
32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Shell – Key Features
40
Interface for Development and Administration of MySQLInterface for Development and Administration of MySQL
• Scripting for Javascript, Python, and SQL mode
• Supports MySQL Standard and X Protocols
• Document and Relational Models
• CRUD Document and Relational APIs via scripting
• Traditional Table, JSON, Tab Separated output results formats
• Both Interactive and Batch operations
33. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 44
Document Store
Under The Hood
34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 45
js> session.createSchema('demo')
js> use demo
js> db.getCollections()
js> db.createCollection('myCollection')
js> db.getCollections()
js> db.myCollection.add({"param1":"value1", "param2":"value2"})
js> db.myCollection.find()
js> db.myCollection.find().limit(1)
js> db.myCollection.find("_id = '00005af018430000000000000002'")
js> db.myCollection.modify("_id = '1234'").set("param","value")
js> db.myCollection.remove("_id = '1234'")
js> session.startTransaction()
js> …
js> session.rollback()
js> db.myCollection.createIndex("ageIdx",{fields:[{"field":"$.age","type":"INT",required:false}]})
Copyright @ 2018 Oracle and/or its affiliates. All rights reserved.
MySQL Document Store “cheat sheet”
CCreate
RRead
UUpdate
DDelete
Index
TTransaction
35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Documents and Collections
• Collections are containers for documents
– These documents share a purpose
– Possibly share one or more indexes
– Each collection has a unique name
– Exists within a single schema
• Within a Collection you can
– Add(), Find(), Modify(), and Remove() - JSON documents
• Collections can be
– Create(), List(), Drop()
46
36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Operation Document Relational
Create Collection.add() Table.insert()
Read Collection.find() Table.select()
Update Collection.modify() Table.update()
Delete Collection.remove() Table.delete()
47
• Use SQL, CRUD APIs – Document (NoSQL) and Relational (SQL), or “All of the Above”
– All of this is in addition to the Classic APIs
Connectors include X DevAPI
37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Collection Search – find(), bind(), fields()
• Supports many operators to specify searches
– ||, &&, XOR, IS, NOT, BETWEEN, IN, LIKE, !=, <>, >, >=, <, <=, &, |, <<, >>, +, -, *, /, ~, %.
• Searching - find()
– db.CountryInfo.find("GNP > 500000 and demographics.Population < 100000000")
– db.CountryInfo.find("GNP*1000000/demographics.Population > 30000")
• Binding - bind()
– db.CountryInfo.find("Name = :country").bind("country", "Italy")
• Project Results – fields() – returns specific fields
– db.CountryInfo.find("Name = :country").bind("country", "Italy")
48
38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Setup
50
$ mysqlsh -u user -p --sql
-e"SELECT PLUGIN_NAME, PLUGIN_STATUS, PLUGIN_DESCRIPTION FROM
information_schema.plugins WHERE PLUGIN_NAME='mysqlx'"
+-------------+---------------+--------------------+
| PLUGIN_NAME | PLUGIN_STATUS | PLUGIN_DESCRIPTION |
+-------------+---------------+--------------------+
| mysqlx | ACTIVE | X Plugin for MySQL |
+-------------+---------------+--------------------+
• MySQL 8.0.11 or higher
• X plugin is enabled by default and is part of the MySQL Server
• Check the X plugin installation:
• Disabling at startup:
– Set mysqlx=0 in your MySQL configuration file
– Or by passing in either --mysqlx=0 or --skip-mysqlx when starting the MySQL server
39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Shell
51
Interface for Development and Administration of MySQLInterface for Development and Administration of MySQL
• Scripting for Javascript, Python, and SQL mode
• Supports MySQL Standard and X Protocols
• Document and Relational Models
• CRUD Document and Relational APIs via scripting
• Traditional Table, JSON, Tab Separated output results formats
• Both Interactive and Batch operations
40. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 52
$ mysqlsh -u <user> -p
…
s
MySQL Shell version 8.0.11
Session type: X
Connection Id: 17
Default schema:
Current schema:
Current user: root@localhost
SSL: Cipher in use: DHE-RSA-AES128-GCM-SHA256 TLSv1.2
Using delimiter: ;
Server version: 8.0.11-commercial MySQL Enterprise Server - Commercial
Protocol version: X protocol
Client library: 8.0.11
Connection: localhost via TCP/IP
TCP port: 33060
Server characterset: utf8mb4
Schema characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
Uptime: 30 min 16.0000 sec
• Following commands provides instructions to begin prototyping database applications
interactively with MySQL Shell for Javascript
• MySQL Shell for Python provide same functionalities
41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 53
h
===== Global Commands =====
help (?,h) Print this help.
sql Switch to SQL processing mode.
js Switch to JavaScript processing mode.
py Switch to Python processing mode.
source (.) Execute a script file. Takes a file name as an argument.
Start multi-line input when in SQL mode.
quit (q) Quit MySQL Shell.
exit Exit MySQL Shell. Same as quit
connect (c) Connect to a server.
reconnect Reconnect with a server.
option Manage MySQL Shell options.
warnings (W) Show warnings after every statement.
nowarnings (w) Don't show warnings after every statement.
status (s) Print information about the current global connection.
use (u) Set the current schema for the active session.
rehash Update the auto-completion cache with database names.
history View and edit command line history.
For help on a specific command use the command as ? <command>
===== Global Objects =====
dba Enables you to administer InnoDB clusters using the AdminAPI.
mysql Used to work with classic MySQL sessions using SQL.
mysqlx Used to work with X Protocol sessions using the MySQL X DevAPI.
session Represents the currently open MySQL session.
shell Gives access to general purpose functions and properties.
sys Gives access to system specific parameters.
util Global object that groups miscellaneous tools like upgrade checker.
For more help on a global variable use <var>.help(), e.g. dba.help()
42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 54
MySQL localhost:33060+ ssl JS> session
<Session:root@localhost>
mysql-js> session.help()
...
The following functions are currently supported.
- close Closes the session.
- commit Commits all the operations executed after a call to startTransaction().
- createSchema Creates a schema on the database and returns the corresponding object.
- dropSchema Drops the schema with the specified name.
- getCurrentSchema Retrieves the active schema on the session.
- getDefaultSchema Retrieves the Schema configured as default for the session.
- getSchema Retrieves a Schema object from the current session through it's name.
- getSchemas Retrieves the Schemas available on the session.
- getUri Retrieves the URI for the current session.
- help Provides help about this class and it's members
- isOpen Returns true if session is known to be open.
- quoteName Escapes the passed identifier.
- releaseSavepoint Removes a savepoint defined on a transaction.
- rollback Discards all the operations executed after a call to startTransaction().
- rollbackTo Rolls back the transaction to the named savepoint without terminating the transaction.
- setCurrentSchema Sets the current schema for this session, and returns the schema object for it.
- setFetchWarnings Enables or disables warning generation.
- setSavepoint Creates or replaces a transaction savepoint with the given name.
- sql Creates a SqlExecute object to allow running the received SQL statement on the target
MySQL Server.
- startTransaction Starts a transaction context on the server.
43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Collection creation
55
44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 56
js> session.createSchema('doc')
<Schema:doc>
js> use doc
Default schema `doc` accessible through db.
js> db
<Schema:doc>
js> db.getCollections()
[]
js> db.createCollection('posts')
<Collection:posts>
js> db.getCollections()
[
<Collection:posts>
]
js> db.posts.add({"title":"MySQL 8.0 rocks", "text":"My first post!", "code": "42"})
Query OK, 1 item affected (0.0221 sec)
js> db.posts.add({"title":"Polyglot database", "text":"Developing both SQL and NoSQL applications"})
Query OK, 1 item affected (0.0172 sec)
45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Find documents
58
https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-find.html
46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 59
js> db.posts.add("This is not a valid JSON document")
CollectionAdd.add: Argument #1 expected to be a document, JSON expression or a list of documents (ArgumentError)
js> db.posts.find()
[
{
"_id": "00005aec27830000000000000001",
"code": "42",
"text": "My first post!",
"title": "MySQL 8.0 rocks"
},
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
js> db.posts.find().fields("text")
[
{
"text": "My first post!"
},
{
"text": "Developing both SQL and NoSQL applications"
}
]
47. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 60
js> db.posts.find().limit(1)
[
{
"_id": "00005aec27830000000000000001",
"code": "42",
"text": "My first post!",
"title": "MySQL 8.0 rocks"
}
]
js> db.posts.add({"_id": "00005aec27830000000000000001", "code": "13"})
ERROR: 5116: Document contains a field value that is not unique but required to be
js> db.posts.find().limit(1).skip(1)
[
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 61
js> db.posts.find("title = 'Polyglot database'")
[
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
js> db.posts.find("title = :what").bind("what", "Polyglot database")
[
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 62
js> db.posts.find()
[
{
"_id": "00005aec27830000000000000001",
"code": "42",
"text": "My first post!",
"title": "MySQL 8.0 rocks"
},
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
js> db.posts.find().sort("text") /* same as : db.posts.find().sort("text asc")*/
[
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
},
{
"_id": "00005aec27830000000000000001",
"code": "42",
"text": "My first post!",
"title": "MySQL 8.0 rocks"
}
]
50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Document modification
63
https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-modify.html
51. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 64
js> db.posts.find("_id = '00005aec27830000000000000002'")
[
{
"_id": "00005aec27830000000000000002",
"text": "Developing both SQL and NoSQL applications",
"title": "Polyglot database"
}
]
js> db.posts.modify("_id = '00005aec27830000000000000002'").set("title", "Hybrid database")
Query OK, 1 item affected (0.0182 sec)
js> db.posts.find("_id = '00005aec27830000000000000002'").fields("title", "text")
[
{
"text": "Developing both SQL and NoSQL applications",
"title": "Hybrid database"
}
]
js> db.posts.modify("_id = '00005aec27830000000000000002'").set("author" ,"Daz").set("code" , 2018)
Query OK, 1 item affected (0.0282 sec)
js> db.posts.find("_id = '00005aec27830000000000000002'").fields("author","text","code")
[
{
"author": "Daz",
"code": 2018,
"text": "Developing both SQL and NoSQL applications"
}
]
52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 65
js> db.posts.modify("_id = '00005aec27830000000000000002'").set("autor", "Daz le magnifique")
Query OK, 1 item affected (0.0117 sec)
js> db.posts.find("_id = '00005aec27830000000000000002'")
[
{
"_id": "00005aec27830000000000000002",
"author": "Daz",
"autor": "Daz le magnifique",
"code": 2018,
"text": "Developing both SQL and NoSQL applications",
"title": "Hybrid database"
}
]
js> db.posts.modify("_id = '00005aec27830000000000000002'").unset("autor")
Query OK, 1 item affected (0.0123 sec)
js> db.posts.find("_id = '00005aec27830000000000000002'")
[
{
"_id": "00005aec27830000000000000002",
"author": "Daz",
"code": 2018,
"text": "Developing both SQL and NoSQL applications",
"title": "Hybrid database"
}
]
53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Full ACID
66
https://dev.mysql.com/doc/x-devapi-userguide/en/transaction-handling.html
54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
It relies on the proven MySQL InnoDB’s strength & robustness:
• innodb_flush_log_at_trx_commit = 1
• innodb_doublewrite = ON
• sync_binlog = 1
• transaction_isolation = REPEATABLE-READ | READ-COMMITTED | ...
We do care about your data!
Document Store is Full ACID
ACID transactions = Don’t lose my dataACID transactions = Don’t lose my data
67
55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Document Store supports transactions
ACID transactions = Don’t lose my dataACID transactions = Don’t lose my data
js> session.startTransactionstartTransaction() // Start a transaction
Query OK, 0 rows affected (0.0004 sec)
js> db.posts.modify("_id ='00005aec27830000000000000002'").set("title","ACID Transaction").set("text","Don’t lose my data")
js> db.posts.find("_id ='00005aec27830000000000000002'").fields(["title", "text"])
[
{
"text": "ACID Transaction",
"title": "Don’t lose my data"
}
]
js> db.restaurants.remove("_id='5ad5b645f88c5bb8fe3fd337'")
js> db.restaurants.find("_id='5ad5b645f88c5bb8fe3fd337'")
Empty set (0.0005 sec)
js> session.rollback() // Rollback the transaction (if necessary e.g. in case of an error)
Query OK, 0 rows affected (0.0862 sec)
js> db.posts.find("_id ='00005aec27830000000000000002'").fields(["title", "text"])
[
{
… [snip] …
js> db.restaurants.find("_id='5ad5b645f88c5bb8fe3fd337'")
[
{
… [snip] …
69
56. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Document deletion
70
https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-documents-remove.html
57. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 72
js> db.posts.add({"actions": {"type": 1, "site": "http://dasini.net/blog/fr"}})
js> db.posts.add({"actions": {"type": 2, "site": "http://dasini.net/blog/en"}})
js> db.posts.find().sort("actions.type desc").limit(1)
[
{
"_id": "00005aec2783000000000000000b",
"actions": {
"site": "http://dasini.net/blog/en",
"type": 2
}
}
]
js> db.posts.remove("true").sort("actions.type desc").limit(1)
Query OK, 1 item affected (0.1212 sec)
js> db.posts.find('actions.type IN (1, 2)')
[
{
"_id": "00005aec2783000000000000000a",
"actions": {
"site": "http://dasini.net/blog",
"type": 1
}
}
]
js> db.posts.remove('actions.type = 1')
Query OK, 1 item affected (0.0192 sec)
js> db.posts.find('actions.type IN (1, 2)')
Empty set (0.0005 sec)
58. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations
Indexes
73
https://dev.mysql.com/doc/refman/8.0/en/mysql-shell-tutorial-javascript-indexes-create.html
59. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 74
// Add a Non Unique Index
js> db.posts.createIndex("idxAuthor", {fields: [{field: "$.author", type: "TEXT(20)"}]})
// Add a Unique Index
js> db.posts.createIndex("idxCode", {fields: [{field: "$.code", type: "INT"}], unique: true})
js> db.posts.add({code: 42})
ERROR: 5116: Document contains a field value that is not unique but required to be
// required:true => field is required to exist in the document i.e. NOT NULL
js> db.posts.createIndex("idxTitle", {fields: [{field: "$.title", type: "TEXT(30)", required:
true}]})
js> db.posts.add({code: 11})
ERROR: 5115: Document is missing a required field
// Drop an Index
js> db.posts.dropIndex("idxTitle")
https://dev.mysql.com/doc/x-devapi-userguide/en/collection-indexing.html
60. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Using SQL with Session
75
61. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 76
// pics is a regular SQL table that contains JSON documents
js> session.sql("SELECT * FROM pics")
+----+------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 14:56:19 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 2018 | 2018-04-25 14:47:15 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
+----+------+---------------------+----------------------------------------------------------------+
// SQL join between a collection posts and a table pics
js> session.sql("SELECT po.doc->'$.title' title, po.doc->'$.text' msg, ts time, pi.doc details FROM
posts po INNER JOIN pics pi ON (pi.code = po.doc->>'$.code') WHERE pi.code = 2018")
+-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+
| title | msg | time | details |
+-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+
| "Hybrid database" | "Developing both SQL and NoSQL applications" | 2018-04-25 14:47:15 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
+-------------------+----------------------------------------------+---------------------+------------------------------------------------------------+
// Many other SQL commands available
js> session.sql("CREATE …
js> session.sql("CALL...
js> session.sql("DROP...
Using SQL with Session
• The Session object has a sql() function that takes any SQL statement as a string
• Possible to join collections with tables :-O
62. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations – SQL / Relational
77
63. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
CRUD operations – SQL / Relational Tables
78
• You can use MySQL Shell to manipulate not just JSON documents, but also relational tables.
• Basic operations scoped by tables include:
Operation Form Description
db.name.insert()
The insert() method inserts one or more records into the
named table.
db.name.select()
The select() method returns some or all records in the
named table.
db.name.update() The update() method updates records in the named table.
db.name.delete()
The delete() method deletes one or more records from the
named table.
64. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 79
js> db.getTables()
[
<Table:pics>
]
// INSERT Records into table pics
js> db.pics.insert('code', 'doc').values(42, '{"data": {"tags": [1, 11], "type": "image", "location": "Fr"}}')
js> db.pics.insert('code', 'doc').values(11, '{"data": {"tags": [2], "type": "image", "location": "Fr"}}')
js> db.pics.insert('code', 'doc').values(11, '{"data": {"tags": [9, 99], "type": "sound"}}')
// Select table pics
js> db.pics.select()
+----+--------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+--------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
| 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+--------+---------------------+----------------------------------------------------------------+
js> db.pics.select(["id", "ts", "doc"])
+----+---------------------+----------------------------------------------------------------+
| id | ts | doc |
+----+---------------------+----------------------------------------------------------------+
| 1 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
| 3 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+---------------------+----------------------------------------------------------------+
js> db.pics.select(["ts", "doc"]).where("id = 1")
+---------------------+----------------------------------------------------------------+
| ts | doc |
+---------------------+----------------------------------------------------------------+
| 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
+---------------------+----------------------------------------------------------------+
65. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 80
js> db.pics.select(["ts", "doc"]).where("id like :id").bind("id", 1)
+---------------------+----------------------------------------------------------------+
| ts | doc |
+---------------------+----------------------------------------------------------------+
| 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
+---------------------+----------------------------------------------------------------+
js> db.pics.select(["ts", "doc"]).where('id IN (1, 2) AND ts < NOW()').orderBy(["ts desc"]).limit(2)
+---------------------+----------------------------------------------------------------+
| ts | doc |
+---------------------+----------------------------------------------------------------+
| 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
| 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
+---------------------+----------------------------------------------------------------+
js> db.pics.select().where("doc->'$.data.type' = 'image'")
+----+--------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+--------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
+----+--------+---------------------+----------------------------------------------------------------+
js> db.pics.select().where("doc->'$.data.tags[0]' = 9")
+----+------+---------------------+----------------------------------------------+
| id | code | ts | doc |
+----+------+---------------------+----------------------------------------------+
| 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+------+---------------------+----------------------------------------------+
// Is pics a view?
js> db.pics.isView()
False
66. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 81
js> db.pics.select()
+----+--------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+--------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
| 3 | 11 | 2018-04-25 12:12:12 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+--------+---------------------+----------------------------------------------------------------+
// UPDATE Table pics
js> db.pics.update().set('code', 2018).where("doc->'$.data.type' = 'sound'")
Query OK, 1 item affected (0.0208 sec)
js> db.pics.select()
+----+------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 2 | 11 | 2018-04-25 12:12:01 | {"data": {"tags": [2], "type": "image", "location": "Fr"}} |
| 3 | 2018 | 2018-04-25 12:31:09 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+------+---------------------+----------------------------------------------------------------+
// DELETE records from table pics
js> db.pics.delete().where("doc->'$.data.tags[0]' = 2")
Query OK, 1 item affected (0.0512 sec)
js> db.pics.select()
+----+------+---------------------+----------------------------------------------------------------+
| id | code | ts | doc |
+----+------+---------------------+----------------------------------------------------------------+
| 1 | 42 | 2018-04-25 12:11:33 | {"data": {"tags": [1, 11], "type": "image", "location": "Fr"}} |
| 3 | 2018 | 2018-04-25 12:12:01 | {"data": {"tags": [9, 99], "type": "sound"}} |
+----+------+---------------------+----------------------------------------------------------------+
js> //db.pics.delete() // Drop all rows
67. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Relational / SQL / Analytics on MySQL Tables
82
68. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 84
SQL for running Complex and/or analytic queries
• Relational Analytics
– Collations are MySQL tables
– So you can use ALL the power of MySQL’s SQL commands i.e.
●
NATURAL [ INNER | CROSS | {LEFT|RIGHT} OUTER ] JOIN
●
CTE
●
Window functions
●
JSON_ARRAYAGG() / JSON_OBJECTAGG() / JSON_PRETTY()
●
…
69. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 85
// Calculate the average of the restaurant rating
sql>
SELECT rname AS Restaurant, Style, avg(rating) AS 'Average Rating'
FROM restaurants, JSON_TABLE(doc, "$" columns( rname varchar(50) path "$.name",
style varchar(50) path "$.cuisine", nested path "$.grades[*]" columns (rating int
path "$.score" ))) as jt
GROUP BY rname, style
ORDER BY avg(rating) DESC;
+--------------------------------+------------+----------------+
| Restaurant | Style | Average Rating |
+--------------------------------+------------+----------------+
| Wendy'S | Hamburgers | 13.7500 |
| Dj Reynolds Pub And Restaurant | Irish | 9.2500 |
| Morris Park Bake Shop | Bakery | 8.2000 |
+--------------------------------+------------+----------------+
SQL for running Complex and/or analytic queries
• Simple query with JSON_TABLE
– JSON_TABLE: Extracts data from a JSON document and returns it as a relational table having the specified columns
70. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 86
// Calculate the average of the restaurant rating (more elegant form)
sql>
SELECT doc->>"$.name" AS Restaurant, doc->>"$.cuisine" AS Cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH
"$.score")) AS r) AS 'Avg Rating'
FROM restaurants
ORDER BY 'Avg Rating' DESC;
+--------------------------------+------------+------------+
| Restaurant | Cuisine | Avg Rating |
+--------------------------------+------------+------------+
| Morris Park Bake Shop | Bakery | 8.2000 |
| Wendy'S | Hamburgers | 13.7500 |
| Dj Reynolds Pub And Restaurant | Irish | 9.2500 |
+--------------------------------+------------+------------+
SQL for running Complex and/or analytic queries
• Simple query with JSON_TABLE
– JSON_TABLE: Extracts data from a JSON document and returns it as a relational table having the specified columns
71. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 87
// Calculate the average of the restaurant rating (Adding CTE and window functions)
sql>
WITH cte1 AS (
SELECT doc->>"$.name" AS Restaurant, doc->>"$.cuisine" AS Cuisine,
(SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH
"$.score")) AS r) AS AvgScore FROM restaurants
)
SELECT *, RANK() OVER (PARTITION BY cuisine ORDER BY AvgScore) AS `Rank`
FROM cte1
ORDER BY `Rank`, AvgScore DESC;
+--------------------------------+------------+----------+------+
| Restaurant | Cuisine | AvgScore | Rank |
+--------------------------------+------------+----------+------+
| Wendy'S | Hamburgers | 13.7500 | 1 |
| Dj Reynolds Pub And Restaurant | Irish | 9.2500 | 1 |
| Morris Park Bake Shop | Bakery | 8.2000 | 1 |
+--------------------------------+------------+----------+------+
SQL for running Complex and/or analytic queries
• Commont Table Expression (CTE) : https://dev.mysql.com/doc/refman/8.0/en/with.html
• Windows functions : https://dev.mysql.com/doc/refman/8.0/en/window-functions.html
72. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 94
Document Store
Wrap-up
73. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 95
Document Store
✔ Built on Proven SQL/InnoDB/Replication technology
✔ Schema-less/Relational/Hybrid
✔ ACID/Transactions
✔ No SQL required - CRUD/JSON/Documents/Indexes
✔ Modern Dev API
✔ Modern/Efficient Protocol
✔ SQL Queries/Analytics over JSON Documents
✔ Transparent and Easy HA/Scaling with MySQL InnoDB Cluster
95
74. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Hybrid database that covers 4 core areas
Relational
MySQL Tables and SQL
Relational
MySQL Tables and SQL
Relational Analytics
MySQL Tables and SQL
Relational Analytics
MySQL Tables and SQL
Document Store
MySQL Collections and
Table with JSON
Datatype
Key/Value
MySQL Memcached
Document Store
MySQL Collections and
Table with JSON
Datatype
Key/Value
MySQL Memcached
Mass Scale Analytics
Sharded MySQL
Relational and/or
Document Store
Mass Scale Analytics
Sharded MySQL
Relational and/or
Document Store
96
Operational Analytical
Relational/SQL
NoSQL
75. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 97
Clients and Applications
InnoDB Storage Engine
mysqld process
Firewall
Audit
Encryption
Authentication
Online Backup
Monitoring
Integration
Support
3rd
Party
Tools3rd
Party
Tools3rd
Party
Tools3rd
Party
Tools3rd
Party
Tools
NoSQLNoSQL
Simple access
patterns
Compromise on consistency
for performance
Ad-hoc data format
Simple operation
SQLSQL
Complex queries with
joins
ACID transactions
Well defined schemas
Rich set of tools
Document Store
Works transparently with MySQL Enterprise FeaturesWorks transparently with MySQL Enterprise Features
76. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 99
Resources
Topic Link(s)
Top 10 reasons for NoSQL with MySQL http://lefred.be/content/top-10-reasons-for-nosql-with-mysql/
MySQL as a Document Store https://dev.mysql.com/doc/refman/8.0/en/document-store.html
MySQL Shell User Guide https://dev.mysql.com/doc/mysql-shell-excerpt/8.0/en/index.html
MySQL Shell Documentation https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/
https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/
X Dev API User Guide https://dev.mysql.com/doc/x-devapi-userguide/en/
X Plugin https://dev.mysql.com/doc/refman/8.0/en/x-plugin.html
MySQL Engineering Blog https://insidemysql.com/mysql-8-0-welcome-to-the-devapi/
https://insidemysql.com/mysql-document-store-crud-quick-start/
MySQL JSON Data Type https://dev.mysql.com/doc/refman/8.0/en/json.html
Blogs http://dasini.net/blog/2018/07/23/30-mins-with-mysql-json-functions/
http://dasini.net/blog/2015/11/17/30-mins-avec-json-en-mysql/
http://dasini.net/blog/2015/11/30/json-et-colonnes-generees-avec-mysql
http://mysqlserverteam.com/tag/json/
http://mysqlserverteam.com/category/docstore/
78. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Thank you!