Incomplete And Missing Data In Geoscience Databases
•Als PPT, PDF herunterladen•
1 gefällt mir•826 views
Presentation at an Edinburgh e-science workshop in 2006
![Not just geoscience ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Andere mochten auch
Andere mochten auch (14)
Incomplete And Missing Data In Geoscience Databases
- 1. Incomplete and missing data in geoscience databases Towards the OWA relational model ? Stephen Henley Presented at the eSI workshop The Closed World of Databases Meets the Open World of the Semantic Web, Edinburgh 12-13 Oct 2006 Resources Computing International Ltd
- 4. Typical imprecise data Sample SiO2 % Cu ppm #101 53.5 128 #102 49.2 185 #103 66.3 163
- 5. Typical imprecise data Sample SiO2 % Cu ppm #101 53.5 128 #102 49.2 185 #103 66.3 163
- 7. Each value has its own error distribution Sample SiO2 % Cu ppm #101 ~53.5 ~128 #102 ~49.2 ~185 #103 ~66.3 ~163
- 10. Incomplete data Hole_ID Total_Depth D_green #301 320.0 250.0 #302 300.0 270.0 #303 200.0 Unknown ?
- 11. Incomplete data Hole_ID Total_Depth D_green #301 320.0 250.0 #302 300.0 270.0 #303 200.0 > 200.0
- 12. Incomplete data
- 16. Missing data item Sample SiO2 % Cu ppm #101 53.5 128 #102 - 185 #103 66.3 163
- 21. Decompose into ‘null-free’ relations Sample SiO2% Cu ppm #101 53.5 128 #102 - 185 #103 66.3 163 Sample SiO2 % Sample Cu ppm #101 53.5 #101 128 #103 66.3 #102 185 #103 163
- 25. No tuple for sample #102 in the SiO2 relation Sample SiO2 % Sample Cu ppm #101 53.5 #101 128 #103 66.3 #102 185 #103 163
- 30. CWA - 2VL T represents TRUE F represents FALSE
- 31. 2VL with probabilities T represents p=1; F represents p=0 p(A B) , p(A B) in general need statistical computation
- 32. OWA - 3VL T represents TRUE F represents FALSE U represents UNKNOWN
Hinweis der Redaktion
- What I am going to talk about is an old problem. It’s one that we were aware of when I was working with Keith Jeffery on relational systems development 30 years ago. Unfortunately, rather than solving the problem, insistence on the relational model being restricted to the closed world assumption is pushing us backwards
- - as are data in many other fields too
- Laboratory analyses are typically reported – and recorded in databases – as single numbers with greater or less precision , but usually the accuracy is ignored.
- … so this number 49.2 may well be what has been reported by the laboratory but of course it is unlikely to be exactly the ‘true’ value for the silica content of sample 102.
- Each data value consists of the reported ‘mean’ or ‘best estimate’ together with a definition of the error distribution about that estimate.
- With such data, a simple > test will no longer give absolute true or false results, but computed probability estimates based on the error distributions of the operands.
- OK, let’s move on to a different problem. Three drillholes. What is the depth of the ‘top of green’ in drill hole 303 ?
- Is it actually unknown ? Not completely. We do know that it hasn’t been intersected to the maximum depth that hole 303 has been drilled. We have some information about it.
- So instead of ‘missing data’ or the dreaded NULL we need to put a ‘partial’ data value in there.
- Let’s just take another look at the geology before moving on.
- So we have the strange situation of a data value which sometimes gives absolute true or false truth values in queries, and other times gives unknown.
- Now to the thorny question of completely missing data items.
- OK, sample 102 wasn’t analysed for SiO2, by mistake, or oversight, or the instrument malfunctioned. The value is just missing in the purest sense of the word.
- Among those who insist that relational means closed-world, there have been some very inventive methods devised to avoid the NULL representation for missing data – and a strong feeling that all data ought really to be complete.
- The default-value or special-value solution simply does not hold water. Not only is it more complicated than a global ‘null’ representation, but it also requires applications (or the user) to carry and manipulate a lot of extra garbage. At best it consists of a domain-specific null value which needs the same sort of processing as a global null.
- Effectively what we are looking at is normalisation to 6NFwhere all relations are, at most, binary.
- So at least we have established that any occurrence of NULL can be converted to a missing tuple. Does this actually get us anywhere ?
- This is important. We can discuss later what the “corresponding proposition” is or ought to be, because this may lie at the heart of the question.
- Decomposition has eliminated the tuple that we had, with the missing-data placeholder for SiO2 in sample 102.
- Here is the nub of it. What is the “corresponding proposition” ? “ Sample N contains X% SiO2” or “Sample N is reported to contain X% SiO2” ? In other words is the database a record of (a) what is or of (b) what is known ? This reasoning works for (a). For (b) it is perfectly valid for a value to be unknown and for a null or some other missing value placeholder to be used, so the problem does not arise
- Basically the same but allowing a continuity of values between True and False
- Please note that U is a truth value. It should not be confused with NULL or any missing-data placeholder. Indeed, it would be perfectly valid to have a type 3VL domain in which T,F, and U all appear as genuine values, as well as a missing-data placeholder (let’s call it null?)