Introduction to Research Automation with Globus

Vas Vasiliadis
vas@uchicago.edu
February 27, 2024
Introduction to Research Automation

What do we mean by
research “automation”?
Executing research tasks* reliably,
at scale, with minimal (or no)
human intervention when required.
*data management and computation
2

Stepping into automation using Globus
• Level 1: Use the web app; it’s manual, but it may be
more automated than your current process J
• Level 2: Semi-automated, recurring tasks
• Level 3: Automation using the Globus CLI
• Level 4: Automation using Globus Flows
• Level 5: “Lights-out” automation using Globus Flows
with event triggers

A simple, and very common, use case
Transfer data
to a system for
sharing
Transfer
Set access
controls for
sharing data
Share
We’ll use this to
demonstrate
5 levels of automation

Level 1
Point-and-click using
the web app
6
Example: Ad hoc sharing of
results with a collaborator
Transfer Share

Level 2
Timer + manual setting
of permissions
7
Example: Scheduled backup
set to read-only access
Transfer Share

Timers
8
Scheduled and/or
recurring file
transfers
Support all transfer
and sync options
hpc.nih.gov/docs/globus/globus_cron.php#cron

Level 3
Parameterized CLI script
9
Transfer Share
Example: On-demand data
distribution from analysis

Globus Command Line Interface
Automation of
simple data
management tasks
Integration with
existing scripts
(job submission …)
Open source, uses
the Python SDK
$ globus
Usage: globus [OPTIONS] COMMAND [ARGS]...
Interact with Globus from the command line
All `globus` subcommands support `--help` documentation.
Use `globus login` to get started!
The documentation is also online at https://docs.globus.org/cli/
Options:
-v, --verbose Control level of output
-h, --help Show this message and exit.
-F, --format [unix|json|text] Output format for stdout. Defaults to
text
--jmespath, --jq TEXT A JMESPath expression to apply to json
output. Forces the format to be json
processed by this expression
--map-http-status TEXT Map HTTP statuses to any of these exit
codes:
0,1,50-99. e.g. "404=50,403=51"
Commands:
api Make API calls to Globus services
bookmark Manage endpoint bookmarks
cli-profile-list List all CLI profiles which have been used
collection Manage your Collections
delete Submit a delete task (asynchronous)
endpoint Manage Globus endpoint definitions
flows Interact with the Globus Flows service

Transfer and share CLI commands
11
$ globus transfer
> --recursive
> source_collection_uuid:source_path
> guest_collection_uuid:destination_path
Message: The transfer has been accepted and a task has been created and
queued for execution
Task ID: f5eb855c-4098-11ee-8ba2-2197ca2bfedc
$ globus endpoint permission create
> --group $group_uuid
> --permissions $permissions
> guest_collection_uuid:destination_path
Granting group, ............., read access to the destination directory
Message: Access rule created successfully.
Rule ID: 7fe723a4-413b-11ee-88f9-03dc0e0dcc45

Exercise: Run script using the Globus CLI
• Log into your instance
• Go to the ~/globus-tutorials directory
• Run the transfer_share.sh script
$ ./transfer_share.sh
> --source-collection a6f165fa-aee2-4fe5-95f3-97429c28bf82
> --source-path /cli
> --guest-collection fe2feb64-4ac0-4a40-ba90-94b99d06dd2c
> --sharing-path /rpi/YOUR_NAME
> --group-id 50b6a29c-63ac-11e4-8062-22000ab68755

Level 4
Using a Globus Flow
13
Transfer Share
Example: Moving data from
instrument to campus
cluster for analysis

Level 4: Automation with Globus Flows
• Flows Service: A platform for managed, secure,
reliable task orchestration
• Flows comprise Actions à invoke Globus services;
extensible to support your own services
• Run via web app, CLI, API, event-based triggers*

Common tasks in most instrument scenarios
Transfer raw
images to HPC
cluster
Transfer
Set access
controls to allow
analysis
Share
Flows
Service
2
Actions
1
:set_permission
Action Provider
:transfer
Action Provider

Flow lifecycle
16
• Define using JSON

Flow lifecycle
17
• Define using JSON/YAML
• Deploy to Flows service

Flow lifecycle
18
• Set access policy for
visibility and execution

Flow lifecycle
19
• Run (debug) and monitor

Flow lifecycle: Write once, run many
20
• Run (debug) and monitor
• …and run again!

Flow definition
21
"StartAt": "TransferFiles",
"States": {
"TransferFiles": {
"Comment": "Transfer to a guest collection",
"Type": "Action",
"ActionUrl": "https://actions.automate.globus.org/transfer/transfer",
"Parameters": {
"source_endpoint_id.$": "$.input.source.id",
"destination_endpoint_id.$": "$.input.destination.id",
"transfer_items": [
{
"source_path.$": "$.input.source.path",
"destination_path.$": "$.input.destination.path",
"recursive.$": "$.input.recursive_tx"
}
]
},
"ResultPath": "$.TransferFiles",
"WaitTime": 60,
"Next": "SetPermission",
},
"SetPermission": {
.....
"End": True
}
}
Action
Action Provider URL
Action inputs
Timeout (seconds)
Next state

Flow input schema
22
{
....
"properties": {
"input": {
"type": "object",
"required": [
"source",
"destination",
"recursive_tx"
],
"properties": {
"source": {
"type": "object",
"title": "Select source collection and path",
"description": ”Source collection/path (MUST end with '/')",
"format": "globus-collection",
"required": [
"id",
"path"
],
"transfer_label": {
"type": "string",
"title": "Label for Transfer Task",
"pattern": "^[a-zA-Z0-9-_, ]+$",
"maxLength": 128,
}
....
Required inputs
Custom schema
Input type
Input type

Run a flow
app.globus.org/flows
(make sure you’re a member
of the “Tutorial Users” group)
25
Transfer Share

Exercise: Run Globus Flow using the web app
• Find “Tutorial - Transfer and Share” in flows library
• Click “Start”
• Confirm the source and destination collections
• Change the name of target path: /rpi/YOUR_NAME
• Enter a label for the flow run
• Click “Start Run”
• Monitor flow progress on the “Event log” tab

Level 5
Triggering flows
automatically
27
Transfer Share

EC2
Instance
“Instrument”
Simulating an instrument flow
Monitor
script
transfer control
Access data
and run
analysis
0 trigger
flow run
set
permissions
2
Globus
Connect
Server
Globus
Connect Server
ALCF
Eagle
transfer
files
1

Illustrating the
possible…
29

A more interesting scenario: cryoEM
Globus
Flows
Carbon!
Correct,
classify, …
Transfer
Transfer
raw files
Compute
Launch
analysis job
Share
Set access
controls
Transfer
Move final
files to repo

Globus
Flows
End-to-end Automation: Serial Crystallography
Image
processing
Data capture
Carbon!
Check
threshold
Data publication
Transfer
Transfer
raw files
Transfer
Move results
to repo
Compute
Analyze
images
Compute
Visualize
Compute
Gather
metadata
Share
Set access
controls
Compute
Launch QA
job
Search
Ingest to
index

Extending the Flows
ecosystem
32

Extending the ecosystem: Action providers
33
• Action Provider is a
service endpoint
– Run
– Status
– Cancel
– Release
– Resume
• Action Provider Toolkit
action-provider-tools.readthedocs.io
compute
ACLs
delete
identifier
transfer
notify ingest
mkdir
search
ls
Xtract describe
web form
Custom developed
docs.globus.org/api/flows/hosted-action-providers

Support resources
• Flows service in web app: app.globus.org/flows
• Flows documentation: docs.globus.org/api/flows
• Helpdesk: support@globus.org
• Customer engagement team can advise on flows
• Professional services team can help build flows

Introduction to Research Automation with Globus

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Ähnlich wie Introduction to Research Automation with Globus

Ähnlich wie Introduction to Research Automation with Globus (20)

Mehr von Globus

Mehr von Globus (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Introduction to Research Automation with Globus