Presentation given in April 2009, during the kickoff of the "Oracle ACEcase" series for the Dutch Oracle Usergroup, the OGh, at the Oracle HQ, The Meern, Holland
Ogh Ace Case, Part 1 and 2, Oracle Xml Database, Marco Gralike
1. OGh Ace Case
Oracle XML Database – Part 1
Marco Gralike – AMIS
1
2. A shortlist…
Started as DBA with Oracle 7 on Windows NT 3.1
(1994)
Experienced with Oracle 6 / 7.x / 8.x / 9.x / 10.x and
11.1
Started with Oracle XMLDB in 2003 (Oracle 10.1.3)
Oracle 11g Beta tester for Oracle XMLDB
Dedicated XMLDB Blog (blog.gralike.com)
Active Oracle OTN XMLDB Forum Member
Oracle ACE Award for XMLDB Community
Contributions
2
3. Overview
What is Oracle XML Database
The Protocol Server
Configuration and Basic Concepts
HTTP API’s
PL/SQL Gateways, ORADB, NDWS
URITypes in the Database
DBUri-, HTTPUri- and XDBUriTypes
Combining Worlds
Selecting and Saving of Files Outside the Database
Repository Events
3
4. Storing XML in Oracle XML Database
File Storage
XML Database Oracle XML DB
Native A World within a World
XML Enabled
Complies to Standards
License Free
Core part of (OXJR)DBMS
The (relational) foundation
is there and can be used for
solving XML “imperfections”
4
6. XMLType
XML data exchange
SQL, C, C++, Java, PL/SQL, .Net, etc
Via Protocol Listener supported methods
Via Oracle Advanced Queuing methods (Oracle 11gR1)
XML data storage
CLOB, Object Relational, Binary XML (11gR1)
XML data validation
XMLSchema, Programmatically, via Storage definitions
XML data creation
XML Operators, Packages
6
16. Native Database WebService (NDWS)
1. Enable the Protocol Server for HTTP access
2. Enable the orawsv entry point in xdbconfig.xml
3. Create a user and grant access
DBMS_NETWORK_ACL_ADMIN
XDB_WEBSERVICES
XDB_WEBSERVICES_OVER_HTTP
XDB_WEBSERVICES_WITH_PUBLIC
4. Create your procedure
For instance a procedure called OBJECTTYPES
5. Test (demo02.sql)
http://localhost:8080/orawsv/SCOTT/OBJECTTYPES?wsdl
16
17. DBURI Servlet
Access Any Table or View From a Browser
DBUriType
Security is handled via DB roles
Transformation via XSL
Query options via XPath Expressions
http://localhost:8080/public/demo03/
17
19. URITypes
DBUriType
Represents an URI that targets database data
HTTPUriType
Represents an URL that starts with http://
Uses UTL_HTTP
XDBUriType
Represents an URI that targets a resource in the Repository
19
20. HTTPUriType - Google API
SQL> SELECT extractvalue(value(t),'/Hostip/gml:name'
2 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot;
3 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;Namequot;
4 , extractvalue(value(t),'/Hostip/countryName'
5 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot;
6 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;CNamequot;
7 , extractvalue(value(t),'/Hostip/countryAbbrev'
8 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot;
9 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;CAbbrevquot;
10 , extractvalue(value(t)
,'/Hostip/ipLocation/gml:PointProperty/gml:Point/gml:coordinates'
11 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot;
12 xmlns=quot;http://www.hostip.info/apiquot;') AS quot;Coord.quot;
13 FROM TABLE (XMLSEQUENCE (EXTRACT
(HTTPURITYPE ('http://api.hostip.info/?ip=141.146.8.66').getxml ()
14 , '/HostipLookupResultSet/gml:featureMember/Hostip'
15 , 'xmlns:gml=quot;http://www.opengis.net/gmlquot;
16 xmlns=quot;http://www.hostip.info/apiquot;' ))) t;
Name CName CAbbrev Coordinates
-------------------- -------------------- -------------------- --------------------
Redwood City, CA UNITED STATES US -122.206,37.5164
20
22. XML/SQL Output
TITLE LINK PUBLICATION_DATE CREATOR
-----------------------------------------------------------------
DESCRIPTION CATEGORY
-----------------------------------------------------------------
Quering RSS Feeds The XMLDB Way
http://feeds.feedburner.com/~r/Bloggralikecom/~3/3
Wed, 25 Jun 2008 16:47:19 +0000
Marco Gralike
Actually this IS old stuff (2006), but it got lost IN a…
< ![CDATA[11g]]>< ![CDATA[Howto]]>
demo04.sql
22
23. XDBUriType
Useful for accessing resources in the XML DB
Repository
Resources = files and folders
Helpful during XML Schema and XML Document
registration
Helpful while insert XML document content from the
XML DB Repository into XMLType Tables, Columns or
Views
23
25. Combining BFileName with XML
The bfilename function returns a BFILE locator for a
physical LOB binary file
Syntax: bfilename( 'directory', 'filename' )
Often used for inserting huge XML documents into the
database
But you can do more…
25
26. Selecting XML Data outside the Database
SQL> SELECT
extract((XMLTYPE(bfilename('XMLSTORE','data.xml'),NLS_CHARSET_ID('AL32UTF8'))),'*')
AS quot;XDATAquot;
2 FROM dual;
XDATA
-------------------------------------------------------
<root>
<id>0</id><info><info_id>0</info_id>
<info_content>Text</info_content></info>
</root>
demo06.sql
SQL> SELECT xdata
2 FROM (XMLTABLE('*'
3 PASSING
(XMLTYPE(bfilename('XMLSTORE','data.xml'),NLS_CHARSET_ID('AL32UTF8')))
4 COLUMNS xdata xmltype PATH '/*'
5 )
6 );
XDATA
-------------------------------------------------------------------------------------
<root>
<id>0</id><info><info_id>0</info_id><info_content>Text</info_content></info></root>
26
27. Using dbms_xmldom or dbms_xslprocessor
Ways to write to disk
dbms_xslprocessor.clob2file
dbms_xmldom.writetofile
Syntax: bfilename( 'directory', 'filename' )
Often used for inserting huge XML documents into the
database
But you can do more…
27
28. Saving XML Data outside the Database
SQL> declare
2 rc sys_refcursor;
3 BEGIN
4 open rc FOR
SELECT * FROM
( SELECT rownum FROM dual connect BY level < 100 );
5 dbms_xslprocessor.clob2file( xmltype( rc ).getclobval( ) , ‘STORE’,'anton.xml’);
6 END;
7/
SQL> declare
2 rc sys_refcursor;
3 doc DBMS_XMLDOM.DOMDocument;
4 begin
5 open rc for
select * from
( select rownum from dual connect by level < 100 );
6 doc := DBMS_XMLDOM.NewDOMDocument(xmltype( rc ));
7 DBMS_XMLDOM.WRITETOFILE(doc, ‘STORE/marco.xml’);
8 end;
9 /
demo07.sql
28
30. Repository Events
Repository Events are events which get triggered via
defined methods on the Repository
For Example:
creating, deleting, locking, unlocking, placing under version
control, checking in, checking out, unchecking out, opening,
and/or updating
Repository Events <> Database Triggers
Are specially designed to act on Repository actions
Repository operation can be associated with one or
more repository events
30
31. Repository Events
Event Listener
Java class or a PL/SQL package or an object type
A repository event listener can be configured for a particular
resource or for the entire repository
It comprises a set of PL/SQL procedures or Java methods,
each of which is called an event handler
Event Handler
Each event handler processes a single event
Resource Configuration File
Associates a repository event listener with a resource by
mapping a resource configuration file to the resource (=
folder or file)
31
33. End Part 1
marco.gralike@amis.nl
http://www.amis.nl
33
34. OGh Ace Case
Oracle XML Database – Part 2
Marco Gralike – AMIS
34
35. Disclaimer
The following are “Rules of
Numb”
Every environment has its own
unique
criteria and needs…
Pay attention
to: “Maintainability”
“Testing”
“Choice” “Extendibility”
“Performance”
“Design”
35
36. Overview
Storage of XML data in the Oracle database
Physical and Logical Design
The General “Issues” with XML
Performance !
An Customer Use Case
36
39. The “XML Relational” Mindset
CREATE TABLE MYTABLE
xmltype);
(data
Here is your
Design !
39
40. “Hidden” Oracle Design Features
CREATE TABLE “AMISquot;.“OBJECTquot; OF quot;SYSquot;.quot;XMLTYPEquot;
XMLSCHEMA quot;http://www.myserver.com/myapp/1.0/xsd/object.xsdquot;
ELEMENT “ROOTquot; ID 4733
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 65536Schema: your container for
XML NEXT 1048576 MINEXTENTS 1 MAXEXTENTS
2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 Design
Logical and Physical BUFFER_POOL DEFAULT)
TABLESPACE quot;USERS“
VARRAY quot;XMLEXTRAquot;.quot;NAMESPACESquot; STORE AS BASICFILE LOB
quot;NAMESPACES660_Lquot;
(ENABLE STORAGE IN ROW CHUNK 8192 PCTVERSION 10 CACHE
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS
2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT))
VARRAY quot;XMLEXTRAquot;.quot;EXTRADATAquot; STORE AS BASICFILE LOB
quot;EXTRADATA659_Lquot;
(ENABLE STORAGE IN ROW CHUNK 8192 PCTVERSION 10 CACHE
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS
Physical Design
2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT))
40
41. Designing Your Building Blocks
Logical Design
XML Instance / Document
XML Schema (XDB Annotations)
XML Evolution, Validation
Physical Design
Oracle Database / Storage Design based on XML Logical
Design
Binary Storage (Encrypted, Compressed, Validation,
Evolution?)
XML Schema (XDB Annotations)
Indexes (BTree, IOT, XMLIndex, Function Based, Text)
Is your XML data CONTENT or DOCUMENT driven ?
Think: Access Paths, The Optimizer, Performance
41
42. Control your design: XML Schemata
Logical, Physical Data Design
Maintained via the Repository
Storage, Evolution, Security
Storage can be a Black Box
(without schemata)
CLOB
Binary XML (ALLOW NONSCHEMA)
Namespaces (Recursiveness ?!)
Overhead due to Parsing
XDB Annotations can define:
Storage, Control, Name
42
43. When to use what kind of Storage
Document centric XML CLOB XML Document
No shredding, document “keeps intact”
Content centric XML Object Relational
Shredding, rebuilding document info XML Document
part
43
45. Choices…
XMLType CLOB
Document Driven
Only in 11gR1 and upwards XMLIndex
XMLType OR
10gRx Index Organized Tables
11gR1 and upwards BTree indexes
XML Schema “overhead”, Data is “shredded”
Query Re-write
XMLType Binary XML
XMLType OR features kind off behavior PLUS
Post Parse Presentation
45
49. xdb:annotations – XML Schema Declaration
xdb:storeVarrayAsTable (“false” < 11g >= “true”)
If true, the varray is stored as a table (OCT)
If false, the varray is stored in a LOB
xdb:mapUnboundedStringToLob (default=“false”)
If true, then unbounded strings are mapped to CLOB
instances by default. Similarly, unbounded binary data gets
mapped to a BLOB value, by default.
If false, then unbounded strings are mapped to
VARCHAR2(4000) values, and unbounded binary
components are mapped to RAW(2000) values.
Don’t forget: xmlns:xdb=quot;http://xmlns.oracle.com/xdbquot;
49
50. xdb:annotations – Global ComplexTypes
xdb:maintainDOM (default=“true”)
If false, then the output is not guaranteed to have the same
DOM action as the input.
xdb:SQLSchema
Name of the database user who
owns the type specified by SQLType
xdb:SQLType
Name of the SQL type that corresponds
to this XML element declaration
Any SQL data type except NCHAR, NVARCHAR, and NCLOB
50
54. Result Object Relational Storage
SQL> select * from tab;
TNAME TABTYPE
------------------------------ -------
INFO_TABLE TABLE
ROOT_TABLE TABLE
SQL> select dbms_metadata.get_ddl('TABLE','ROOT_TABLE') from dual;
DBMS_METADATA.GET_DDL('TABLE','ROOT_TABLE')
------------------------------------------------------
CREATE TABLE “AMISquot;.quot;ROOT_TABLEquot; OF quot;SYSquot;.quot;XMLTYPEquot;
XMLSCHEMA quot;http://www.myserver.com/root.xsdquot; ELEMENT quot;ROOTquot;
PCTFREE 10 PCTUSED 40 INITRANS 1 MAXTRANS 255 NOCOMPRESS LOGGING
STORAGE(INITIAL 65536 NEXT 1048576 MINEXTENTS 1 MAXEXTENTS 2147483645
PCTINCREASE 0 FREELISTS 1 FREELIST GROUPS 1 BUFFER_POOL DEFAULT)
TABLESPACE quot;USERSquot;
VARRAY quot;XMLEXTRAquot;.quot;NAMESPACESquot; STORE AS BASICFILE LOB quot;NAMESPACES780_Lquot;
...
VARRAY quot;XMLDATAquot;.quot;INFOquot;.quot;SYS_XDBPD$quot; STORE AS BASICFILE LOB quot;SYS_XDBP...quot;
DISALLOW NONSCHEMA
...
54
55. XMLType CLOB Storage (simplistic)
SQL> create table XMLCLOB of XMLType
XMLTYPE store as CLOB;
Table created.
55
56. XMLType Object Relational Storage (simplistic)
SQL> create table XMLOR of XMLType
XMLTYPE store as OBJECT RELATIONAL
XMLSCHEMA
quot;http://www.amis.nl/xsd/root.xsdquot;
ELEMENT quot;RootElementquot;;
Table created.
56
57. XMLType Binary XML Storage (simplistic)
SQL> create table XMLBIN of XMLType
XMLTYPE store as BINARY XML;
Table created.
SQL> create table XMLBIN_XSD of XMLType
XMLTYPE store as BINARY XML
XMLSCHEMA
quot;http://www.amis.nl/xsd/root.xsdquot;
ELEMENT quot;RootElementquot;;
Table created.
57
61. Relational Fundamentals
What goes IN, must come OUT (IN=OUT)
Regarding Data Integrity, Meaning and Context
Why do we DESIGN in the First Place?
ACID: Atomicity, Consistency, Isolation, Durability
Edgar F Codd’s 12 Rules defining a “Relational Database”
Database Normalization (eg. 3NF)
Nullology
Imperfections of the (Procedural / Query) Language
Maintainability, Performance, Usability
Decades of Innovation, Methods and Standards
61
64. Optimal Access Paths for a Tree ?
Keep it intact?
To much empty
space!
Access
Optimization?
Should we shred
it?
Rebuilding and
shredding the tree
structure…?
There is no
solution for
64
65. Relational contra XML Data
ONE…yes ONE
XML document
can contain a
whole database,
schema of objects
!
65
66. Impedance Mismatch
Different data models.
E.g. XPath, XQuery models an XML document as a tree,
while most general purpose programming languages have no
native data types for a tree.
Different programming paradigms.
XSLT is a functional language, while Java is object-
oriented, and Perl is a procedural one.
Source: Wikipedia SXML
66
67. So in General
NO PROBLEM
Data Shipping / Transport XML via XML
UI’s that “Understand”, are based on XML
PROBLEM
Accessing Data in XML structures
• CPU Intensive
• Memory Intensive
PGA, XML Fragments, Document Object Model
Transformation:
• XML Relational (Risk: Data / Information Loss)
• Document Object Model
67
68. Issues you will encounter with XML
The nature of XML:
It is (most of the time) not Structured
It can not be stored in a Universal XML “Container”
It can not be accessed via a standard Access Path Method
that “understands” the free format XML structure all the time…
It is all about free format
OK and Not OK
Standards: W3C, Oasis, etc
Performance problem induced by its Nature…
Nature is not structured…(as is Human Nature)
68
69. Nope…!
XML can handle (package) structured and semi-
structured data and sometimes even unstructured
data.
The open standard for data exchange
If it is XML, use XML !
Oracle XML Database = Continues Innovation
“There is essentially no computer in the world, desktop,
handheld, or backroom, that doesn't process XML
sometimes...”
69
73. Recap…
XMLDB Relational
XMLDB Data Storage
World
World
XMLType XMLType
v?
Views
(Tables)
Structured Binary XML Hybrid
Unstructured
(Object)
Document Content Mixed? Content
Relational
na complex[n] complex[y] complex[n]
Objects
CLOB OR BINARY OR/CLOB
XSD [y] XSD [y] XSD [y/n] XSD [y]
Relational
XMLIndex BTree, IOT XMLIndex BTree, IOT
Tables
73
74. Some XML Storage Issues
O.R. CLOB Binary XML
Schema Based + - +
XSD Flexibility - (in xml) ++
DML ++ - +
XPath Performance ++ - +
Throughput - + ++
Memory Management ++ - +
XMLIndex - + +
Function / Text + + +
BTree (/ IOT) + - -
74
75. Indexation of Data
BTree Indexes (11g Schema
based)
Cluster Indexes
Bitmap Indexes
Index Organized Table
(10g Schema
Indexes
based)
Function Based Indexes
Partition Indexes
>= V 11.1.0.7.0
Domain Indexes
Oracle Text Context Index
Unstructured XMLIndex
75
76. XML Objects (XOBs) DOM
String
Char
bookstore
Varchar
String
2(20)
book whitepaper
title author author chapter title author id paragraph
Number
Float
content content
CLOB
76
77. Query Re-Write
BTre
BTre
BTre
ee
e
Index
Index
Index
bookstore
Function
based Index
(XPath)
book whitepaper
title author author chapter title author id paragraph
XMLIndex
content content
Oracle
Text Index
77
79. ETL in Memory Processing
Memor
y
/ DOM
Oracle
BLOB CLOB XMLType
Advanced Queue
Validation
Process Shred Elements Store in Oracle
XML Schema
Checks Via XMLDOM ETL Tables Workflow
(JAVA)
79
80. ETL - XMLType Table (OR) Processing
Oracle
BLOB CLOB
Advanced Queue
Validation
XMLType Table Store in Oracle
Against Checks
(O.R) ETL Tables Workflow
XML Schema
Rewrite on Disk
/ XOB
(Relational)
80
81. Results XML Relational Approach
100.000 “zaken” shredded, validated in 5 minutes
(instead of 1000 “zaken” in 3 minutes)
Avoiding
ORA-31186: Document contains too many nodes
Scalable
Efficient with Memory and CPU
Checked on 9.2.0.5 database version
81
82. References
Oracle 11g XMLDB Developers Guide
The XMLDB Forum
http://forums.oracle.com/forums/forum.jspa?forumID=34
OTN XML DB FAQ Thread
http://forums.oracle.com/forums/thread.jspa?threadID=410714
Technology Blogs
http://blog.gralike.com
http://technology.amis.nl/blog
Thanks to:
Mark Drake, Tom Kyte
Keith Matthew, Anton Scheffer, Laurent Schneider
82