Weitere ähnliche Inhalte
Ähnlich wie Data Tactics Unified Dataspace Architecture and Description (20)
Mehr von DataTactics (19)
Data Tactics Unified Dataspace Architecture and Description
- 1. Data Tactics
Unified DataSpace
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 3. Systems Engineering & Integration
SYSTEMS ENGINEERING
• Data Ingestion Frameworks (structured, unstructured, semi-structured)
• Semantic DataSpace Enrichment SYSTEM INTEGRATION
• Cloud Management Systems (CMS) • Ingestion
• Cloudbase/Accumulo – Generalized Ingest /
– Pig (Big Data) Plug-in NiagraFiles
• Dissemination and Reporting Tools • Geospatial Capabilities
• Data Mining, Exploitation, and Correlation Tools • Biometric Capabilities
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 4. Cloud Experience
17 Enclaves at SECRET//NOFORN 4 Enclaves for NATO ISAF
• 3 in Tyson’s • 2 in Afghanistan
• 1 at GISA, Ft. Bragg • 1 at GISA, Fort Bragg
• 2 in Hawaii • 1 in Germany
• 2 in Germany US BICES Cloud in Germany
• 7 at Aberdeen Over a dozen at UNCLASS//FOUO
• 2 in Afghanistan • Supporting real-world missions on
6 Enclaves at TS//SCI contract
• AF TENCAP • At various levels of complexity
• NRL
• DARPA
• INSCOM
• DCGS-A
• DHS OI&A
Cloud Domains is where we live
Data, is the Hard Problem
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 5. Data – The Hard Part
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 6. BigData Architecture
Data Tactics has delivered solutions that manage PETABYTES of data
and provide mission relevant analytics, metrics and user interfaces
• DESIGN, DEVELOPMENT AND INTEGRATION OF REFERENCE ARCHITECTURES
– Ghost Machine
– Stratus
• SECURE DATABASE ARCHITECTURES
– Secure Entity Database (SED)
– Defense Cross-Domain Analytic Capability (DCAC)
• DATA MIGRATION, EXTRACTION, TRANSFORM AND PARSING
• FEDERATED DATA MANAGEMENT
– Federated Search, Multi-Source / Multi-Vendor Integration
– Storage Cluster Management
• DATA MINING AND FORENSIC ANALYSIS
• SPATIAL, MULTI-DOMAIN, AND CLOUD DATA SERVICES
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 7. Unified DataSpace
The Wild
• Data sources
with rich data & Segment 3 - Model Description
semantic context
locked in domain Data
Rich semantic
silos Models
context
• Data tightly
coupled to
data-models
• Data-models
Segment 2 - Data Description
tightly coupled to Structured
Integration
Enrichment
storage models Data Exploitation
Exploration
Silos isolated by Across all sources
• Implementation Segment 1 - Artifact Description
technology
• Storage structure Unstructured Rich data
• Data Data context
representation
• Data modality
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 8. Unified DataSpace
High-Level Conceptual Model of the DataSpace
and Ingest/Extraction Flows
Segment 3 - Model Semantics
.
2 .
2
CONCEPT CONCEPT_ASSOCIATION PREDICATE PREDICATE_ASSOCIATION
.
.
.
Uses Uses
Segment 1 - Artifact Semantics Segment 2 - Data Semantics Semantics
SOURCE . +
2 . 2 . Metadata
ARTIFACT ARTIFACT_ASSOCIATION TERM . STATEMENT
. .
. .
Data
Uses +
Metadata Metadata
Segment 0 - Artifacts
Ingest Extraction
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 9. Unified DataSpace
•Segment 0 is an artifact store (i.e., binary
representation of artifacts). High-Level Conceptual Model of the DataSpace
and Ingest/Extraction Flows
Segment 3 - Model Semantics
•Segment 1 represents artifact semantics
.
2 .
2
CONCEPT CONCEPT_ASSOCIATION PREDICATE PREDICATE_ASSOCIATION
.
.
.
and includes artifact metadata and Uses Uses
associations between the artifacts. Indexing Segment 1 - Artifact Semantics Segment 2 - Data Semantics Semantics
of Segment 1 supports search on text
SOURCE . +
2 . 2 . Metadata
ARTIFACT ARTIFACT_ASSOCIATION TERM . STATEMENT
. .
.
content, geospatial, and artifact meta data.
.
Data
Uses +
Metadata Metadata
Segment 0 - Artifacts
•Segment 2 represents data and semantics
Ingest Extraction
of structured data elements extracted from
artifacts. Indexing of Segment 2 supports
search on properties of entities (e.g., Person,
Location) based on their properties and
relationships.
•Segment 3 represents data-models
extracted from artifacts and models used for
aligning, disambiguating, and enriching the
elements of Segments 1 and 2.
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 10. Data Description Framework
• DDF – looks at data in the following ways
– Mention: A chunk of data, either physically located within a tangible artifact,
or contained within an analyst’s mind
• “Washington” at offset x in file Y
– Sign: A representation of all disambiguated mentions that are identical except
for their indexicality
• E.g., “Washington”
– Concept: An abstract idea, defined explicitly or implicitly by a source data-
model
• E.g., City, Person, Name, Address, Photo
– Predicate: An abstract idea used to express a relationship between “things”
• E.g., isCity, isPerson, hasName, hasAddress, hasPhoto
– Term: A disambiguated sign abstracted from the source artifact or asserting
analyst
• E.g., Washington Person; Washington Location
– Statement: Encodes a binary relationship between a subject (term) and an
object mediated by a predicate
• E.g.,[Washington, Person] hasPhoto [GeorgeWashingtonImage.jpg]
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
- 14. Elastic Data Ingest
Java Messaging Service
Artifact Processor Persistence Index Manager Error Manager
Queue Manager Queue Queue Queue
Queue Artifact Persistence Index Error
Loader Processor Manager Manager Manager
UDS Components
Lucene
File
System
Custom Components
Hadoop DFS
Artifact Processor Persistence
Modules Manager Modules
BigTable
WWW.DATA–TACTICS.COM © 2012 Data Tactics ARCHITECT – ENGINEER – INTEGRATE – SOLUTIONS
