This document discusses metadata quality and metrics for evaluating metadata. It defines metadata as structured information that describes something else. Metadata quality is described as fulfillment of specifications and goals. General metrics for metadata quality include completeness, accuracy, consistency, objectiveness, appropriateness, and correctness. For linked data, additional dimensions and metrics are proposed such as accessibility, intrinsic qualities, contextual relevance, and representational properties. Good metrics are said to be clear, realistic, measurable, discriminating, and universal. The document discusses using RDFUnit, SHACL and ShEx for evaluating linked data and using clustering algorithms like K-means to analyze metadata quality.

1. Researching metadata quality
Péter Király <pkiraly@gwdg.de>
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Open Research Knowledge Graph workshop
TIB, Hannover, 2018-11-22
these slides: http://bit.ly/qa-orkg2018

2. metadata
structured information that describes, explains, locates, or otherwise
represents something else.
NISO (2004)
http://bit.ly/qa-orkg2018

3. quality and ‘fitness for purpose’
★ fulfilment of a specification or stated outcomes
★ measured against what is seen to be the goal of the unit
★ achieving institutional mission and objectives
’We know it when we see it, but conveying the full bundle of assumptions and
experience that allow us to identify it is a different matter.’
http://bit.ly/qa-orkg2018

4. general metrics
★ completeness: number of metadata elements filled out
★ accuracy: data correspond to the resource that is being described
★ consistency: values compliant to what is defined by the metadata scheme
★ objectiveness: values describe the resource in an unbiased way
★ appropriateness: values are facilitating the deployment of search
★ correctness: syntactically and grammatically correct language
Bruce and Hillman (2004); Ochoa and Duval (2009); Palavitsinis (2014)
http://bit.ly/qa-orkg2018

5. linked data dimensions and metrics
accessibility
★ Availability
★ Licensing
★ Interlinking
★ Security
★ Performance
intrinsic
★ Syntactic validity
★ Semantic
accuracy
★ Consistency
★ Conciseness
★ Completeness
contextual
★ Relevancy
★ Trustworthiness
★ Understandability
★ Timeliness
representational
★ Representational
conciseness
★ Interoperability
★ Interpretability
★ Versatility
Stvilia et al. (2007); Zaveri et al. (2015)
http://bit.ly/qa-orkg2018

6. The good metrics are
★ clear
★ realistic
★ discriminating
★ measurable
★ universality
http://fairmetrics.org – https://github.com/FAIRMetrics/Metrics/blob/master/ALL.pdf
FAIR metrics
http://bit.ly/qa-orkg2018

7. http://bit.ly/qa-orkg2018

8. RDFUnit, SHACL and ShEx
★ Linked Data is based on Open World assumption
★ No “record”, no clear boundaries
★ RDF Data Shapes: reinventing the schema
★ ShEx and SHACL: two competiting approaches
★ Finding individual data issues
http://bit.ly/qa-orkg2018
ex:PersonCountShape
a sh:NodeShape ;
sh:targetNode ex:Person ;
sh:property [
sh:path [ sh:inversePath rdf:type ] ;
sh:minCount 1 ;
] .

9. variation to weighted completeness
Thompson and Traill (2017)
9
http://bit.ly/qa-orkg2018

10. K-means clustering
Spark (Scala)
increasing number of clusters
decreasing the distance from
the centroids
after a point this gain is not
so big (“elbow effect”) -- in
theory
Big number or low
quality records
small clusters with ‘in
between’ quality records
the acceptable average
clusters with good quality
records
10
http://bit.ly/qa-orkg2018

11. https://www.zotero.org/groups/488224/metadata_assessment