5. Key Domain Characteristics
Governance-related
• Many independent Data Providers
• Data usage restrictions and auditing
Process-related
• Research groups / teams
• Resource search & discovery
• Routine steps to prepare data for research
Data-related
• Large size and volume of data sets
• Heterogeneity: instruments, data formats
Technology-related
• Heterogeneity: repository technologies, analytic / visualization
technologies
• Cost of technology vs funding models
CSSDP - NEP 21 5
11. Federating Resources
Local Site Overall System
Local Site
Local Site
Federated
Local Site
System
Local
Data
Repository Federated
Instrument System
Instrument
Data Site Data
Data Copied to Portal
Local Site
Enterprise Service Bus
Instrument
Researcher Data
Data Copied to Portal Instrument
Portal Federated
Data Site
System
Data Federated
Local Site
by Portal
Instrument Federated
Data Site System
Local Site
Local Analytics & Instrument
Visualizations Data
Federated
System
Overall System is a Federation of
Each Local Site is a Federation
Federations
CSSDP - NEP 21 11
12. Block Diagram
UI
Presentation Forms Reports Charts Visualizations
External Process Executon &
Layer Page Flow
Management
Java Business Logic
Analytics
POJOs
Business
Business Business
Components Rules
Layer
Events and
Resource
Notifications
Management
Scheduler Digital Rights
Management
Transport Resource
Factory Registry
Transaction
Data Data Set Management
Transport
Layer
Storage
Persistence
Data Set Management
Transformation
Common Services
Collaboration Configuration
Business Process & Instrumentation, Auditing &
Workflow Logging
Security & Identity Messaging & Web Services
CSSDP - NEP 21 12
15. Scientific Workflows
D a ta L o a d e r
G e n e ra te
D a ta
A cq u ire D a ta
A va ila b ility
E ve n t
S ch e d u le r
M o n ito r /
In itia te
M o n ito r R e trie ve N o tify E ve n t
W o rkflo w s
W o rkflo w s E ve n ts S u b scrib e rs
fo r D a ta S e t
W o rkflo w
Run S to re R u n D a ta S to re Run S to re
A n a lytics R e su lts M a rke rs R e su lts V isu a liza tio n s R e su lts
D a ta M a rk e r
A n an a ly D a ta
A lyze S to re E ve n ts
R e s e a rc h e r
R e ce ive R e ce ive
R e vie w E ve n ts
D a ta S e t R e vie w P e rfo rm D a ta M a rke r
and
A va ila b ility D a ta S e ts A n a lysis E ve n t
V isu a liza tio n s
N o tifica tio n s N o tifica tio n s
CSSDP - NEP 21 15
22. Some of the Lessons Learned (So Far)
MySQL vs Postgres
Human-readable vs automated URI formation
Wicket vs Struts
Hibernate query formation
Patterns
XML binding
CSSDP - NEP 21 22
23. More Lessons Learned (So Far)
Scrum and kanban (BVC)
UI prototyping
Unit vs integration testing
Administration and operations
External dependencies
Make it work, make it work better
CSSDP - NEP 21 23
24. Outstanding Questions
Web service file transfer
– MTOM? Redirection?
REST vs WS-* (or both)
– SPASE vs IVOA, OGSA
Workflow engine
– Scripting language? Taverna? ESB++?
CSSDP - NEP 21 24