Presented at the APS Workshop, a virtual event on October 15, 2021.

1. Globus Portal Framework
An introduction to Globus Data Portals
October 15, 2021

2. Agenda
• Globus Data Portals
– FAIR data
– Data discovery
– Custom use-cases
• Django Globus Portal Framework
– Globus Auth
– Globus Services
– Integration with custom resource servers
• Portal Architecture
– Lifecycle of a request
• Live Demo: Build your own portal

3. What is a Globus Data Portal
A Web interface for a Globus Search index with
integrated Globus capabilities to find, access, and act
on data regardless of location

4. Why we use portals
• Different experiments (beamlines, electron
microscopes, biology, etc) generate data with
different types, size and experimental information
• Processing, curation, and cataloging need to
happen as soon as possible so data are not lost
• Standardize secure access between users
• Work toward FAIR datasets to enable more science

5. Advantages of a portal
• Make data FAIRer
• Track lots of data
– Tens of thousands of datasets
• Discovery
– Free text search in Globus Search
– Filtering on specific values
– User Friendly GUI
• Authorized Access
– Public/private, group-based, subject-level ACLs
• Integration with Globus Services
• Customizability

6. Portal Features
• Watch curated data appear in your portal!
– A Web interface to data collection
– Data acquisition in real time
o Synchrotron serial crystallography
• HTTPS Access to data
– Show images in a portal from a Globus Endpoint
– Add links to specific files for user download
• Easy customizability
– Add your own templates
– Rewrite existing views
– Create and call out to custom services

7. ALCF Community Data Co-Op: acdc.alcf.anl.gov
• A collection of data portals in collaboration with ALCF
• Leverage ALCF resources
– Eagle: 100PB store, 650GB/s transfer, Globus interfaces

8. CosmoFlow Search

9. Serial Crystallography

10. Leveraging the Globus
data platform…

11. Globus Search
Globus Search
• Scalable, secure search
for research data
• Metadata store with fine-
grained access controls
• Schema agnostic
• Free text, faceted search
• Integrated with core
platform services, e.g.
Auth, Groups
Input form Extract
Metadata
Ingest metadata, set
visibility policies
Discovery
POST /index/123
{"filters": [
{"field_name": "record_year",
"values": ["2020"],
"type": "match_all" },
{"field_name": "temp_farenheit",
"values": [{"from": 90, "to": "*"}],
"type": "range" ]}
]}
Query
Bulk
ingest

12. Globus Portal Framework
• Django framework for rapidly creating portals
• Python Package for bootstrapping a portal
– Install from PyPi using pip
– github.com/globus/django-globus-portal-framework
• Globus Portal Framework Deployment
– ALCF Community Data Co-Op (acdc.alcf.anl.gov)
• Example Repo
– github.com/globus/django-globus-portal-framework-example

13. Globus Portal Framework Features
• Authorized Access with Globus Auth
• Plugins for Globus Services
– Globus Search
– Globus Transfer
– Globus Flows
– Third-Party Globus Services
• Customizability
– Redesign how your portal talks to these services
– Extend Globus Portal Framework with new features

14. APS XPCS: secure data discovery
14
Globus Auth
Globus Groups
Globus Search

15. APS XPCS: data access & preview
15
Globus Transfer
HTTPS access

16. APS XPCS: automated processing & indexing
16
Globus Flows:
Transfer,
analysis, and
ingest to
search index

17. Portal Architecture

18. Portal Architecture
• User Authentication
• Globus Portal Framework
– Portal URL Mappings
– Token Loading
• Service Calls to Globus Search
• Lifecycle of a Request
• Post Processing search requests

19. User authentication
• Scopes are configured in the portal
• Users authenticate with Globus using 3-legged flow
– Python Social Auth used for Authentication backend
• User tokens are saved in the database
• Future requests authorized with user access tokens
– New searches use Search bearer token.

20. Service Calls
• Most Portal Service calls use the Globus SDK
– Globus Portal Framework loads tokens from DB
– Globus SDK object instantiated with token
– Call to Globus Service
– Portal renders result in templates

21. Globus Portal Framework URLs
• URLs are laid out in three major categories
– Index Selection
– Index Search page
– Search Subject detail page
• Many Globus Search Indices are supported
• Index Search page links to many result subjects
• Each subject has a unique URL

22. Example: Cosmology
22

23. Format of a URL

24. An index is Configuration Driven
• A Search Index can be setup through configuration
– Add Globus Search index uuid
– Add a name
– Add facets
– Add fields
• Start Searching!

25. Lifecycle of a request
• User makes a query
• Portal sends request to Globus Search
– Request contains user bearer token
• Portal receives response
• Portal does processing on response
– Parse Dates, build URL for Globus webapp, etc.
• Portal renders data into templates
• User receives a search page

26. Pilot
• Tooling to simplify ingesting metadata
• Create valid GMeta entries from JSON
• Upload files to an associated endpoints and track
HTTPS URLs to add to GMeta

27. Demo: Creating your own portal
• Django Globus Portal Framework
– github.com/globus/django-globus-portal-framework
– django-globus-portal-framework.readthedocs.io/en/stable/
• Example Repo
– github.com/globus/django-globus-portal-framework-example