1.
August 15, 2016
The Community Software Repository from XSEDE:
A Resource for the National Research Community
JP Navarro, Craig Stewart, Rich Knepper, Lee Liming, David Lifka, Maytal Dahan
July 13, 2017

2.
The Community Software
Repository (CSR) from XSEDE
July 13 / 12:00 pm / Strand 12
Enabling Software Sharing in the XSEDE community
https://software.xsede.org/
For end users, service operators,
software operators and integrators,
and developers
Sharing software requirements, priorities,
gaps, designs, installable packages,
ready-to-use software, delivery plans,
documentation, and community discussions
July 9-13, 2017 — New Orleans — pearc17.pearc.org 2

3.
3
XSEDE Proposal
XSEDE aims to be a connector of cyberinfrastructure (CI) resources,
software, and services that enable collaborations to seamlessly access
advanced computing resources and share data.
XSEDE Cyberinfrastructure Integration (XCI) Team Mission
The mission of the XCI team is to facilitate interaction, sharing and
compatibility of all relevant software and related services across the
national CI community building on and improving on the foundational
efforts of XSEDE. … including seamlessly integrating new CI resources
supported by XSEDE and CI resources located on campuses throughout the
US.

4.
XSEDE Stakeholders
Users Funding
Sources
Service Providers Software Providers, Vendors,
and Integrators

5.
5
Software and services must be well designed and well documented
in a clear, transparent, publicly accessible fashion.
Raises two important and difficult questions?
1. How can XSEDE promote this when we are just one of many stakeholders?
2. How can XSEDE make what we do sustainable beyond our current award?
Important Observation #1

6.
6
NSF Advisory Committee on Cyberinfrastructure (2011).
Finding:
“The current state of cyberinfrastructure software and current levels of expert
support for use of cyberinfrastructure create barriers in use of the many and varied
campus and national cyberinfrastructure facilities. These barriers prevent the US
open science and engineering research community from using the existing, open US
cyberinfrastructure as effectively and efficiently as possible.”
Recommendation:
“The NSF should fund activities that support the evolution and maturation of
cyberinfrastructure through careful analyses of needs (in advance of creating new
cyberinfrastructure facilities) and outcomes (during and after the use of
cyberinfrastructure facilities). The NSF should establish and fund processes for
collecting disciplinary community requirements and planning long-term
cyberinfrastructure software road maps to support disciplinary community research
objectives.”
Important Observation #2

7.
The XSEDE 1 Past – We Built
Elaborate
Engineering
Process
Specific
Architecture
Specific
Software
Partners

8.
Why Change?
• Make XSEDE’s software activities more
streamlined and transparent
• Enable the community to use XSEDE’s
software and service delivery best practices
• Enable more software and services to
integrate into XSEDE
• Enable more infrastructure to integrate into
XSEDE
• Expand capabilities available to XSEDE users
8

9.
The XSEDE 2 Present – We Facilitate
Streamlined
Delivery
Process
Pragmatic
Architecture
New Tool:
CSR
Increased Focus on
Community Software
Provider
Increased Focus on
Un-Allocated Service
Provider

10.
XSEDE 2 – Areas of Change
Streamlined Integration Process
1. Engineering Process (software, services, and infrastructure)
2. Pragmatic Architecture (community driven)
Greater Community Focus
3. Campuses and un-allocated service providers
4. Community Software Providers
New Community Enabling Tool
5. The Community Software Repository (CSR)

11.
Change - Streamlined Integration Process
11
CI Users
CI
Providers
Software
Partners
CI
Providers
Use
Cases
Transparent
Process
Software
Catalogs
Docs

12.
Greater Focus on:
1) Unallocated (infrastructure) Service Providers
Integrating campuses and provide special software services for campuses
Examples: SSO Hub, working with L3s to advertise their HPC resources
1) Community Software Providers
Integrating pre-existing software and services from the community and from
vendors
Examples: Globus Services, Kepler
Change - Greater Community Focus

13.
13
VISION: To enabling transparent and traceable software
requirements, priorities, delivery, sharing, and discovery.
Supports the entire software delivery process
(not just a software catalog):
1. Understanding Requirements
2. Preparing Software Solutions
3. Sharing Software Solutions
4. Discovering Software Solutions
5. Return on Investment (ROI)
Community Software Repository

14.
Key roles:
• Front door to all XCI software activities
• “Home base” for the extended XSEDE software ecosystem where developers,
integrators, operators, users, and funding agencies collaborate to accelerate
resource, software, and software based service integration into the national CI
XSEDE must participate and enable:
• XSEDE software activities are discoverable thru the CSR
• Community can pick and choose CSR elements to share and discover
14
CSR Roles

15.
15
• XSEDE organization
• End Users
• Campus CI Operators
• Service Providers (SPs)
• Software Developers and Integrators
• Organizations competing for XSEDE related software awards
• Funding agencies
Each stakeholder has a different set of needs!
CSR Customers / Stakeholders

16.
1. Understanding Requirements
2. Preparing Software Solutions
3. Sharing Software Solutions
4. Discovering Software Solutions
CSR Focus Areas

17.
17
Supporting capabilities:
• Requirement discussions
• A Use Case Registry tracks the driving requirements
• Capability Delivery Plans detail use case support components
and gaps
• Stakeholder use case priorities
• Proposed and planned Engineering activities needed to fill
capability caps
CSR - 1. Understanding Requirements

18.
18
Supporting capabilities:
• Track active engineering effort delivering capabilities
• Sharing engineering and security designs
• Managing engineering reviews
• Engineering discussions
CSR - 2. Preparing Software Solutions

19.
19
Supporting capabilities:
• Repositories:
–RPM Repository
–Tar Repository
–Future VM, container, and build recipe repositories
• Software metadata catalogs
–From repositories
–Operational
NOTE: our expectation is that the vast majority of software described in the CSR will
be hosted externally in other repositories.
CSR - 3. Sharing Software Solutions

20.
20
Supporting capabilities:
• Use case discovery interfaces
• Repository and operational software and service discovery
interfaces
• Discussion Forums:
–Deployment support discussions
–Use discussions
CSR - 4. Discovering Software Solutions

21.
Campus examples
XNIT
“Pull the lever, build a cluster from scratch”XCBC
Add XSEDE-like functionality to a existing
and functioning cluster
Install community software (packaged by
XCI) via Community Software Repository (by
hand or by subscription from YUM repo)
At present XNIT toolkits (YUM repo) contains a total of
37 software titles packaged as relocatable RPMs.
Counting across different packaging methods (different
MPI implementations, different libraries, development
versions vs. production versions) this is a total of 142
separate packages

22.
22
End Users examples
• What can users do
(use cases)?
• What software is
available on SP
resources?
• XSEDE User Portal

23.
23
Developers examples
• API information
• Capability gaps

24.
24
Service Provider examples
• Software packages:
• Integration status:
• Operations status:

25.
25
https://software.xsede.org/

26.
26
Future CSR capabilities
• Science Gateway software
• Community Software Area (CSA) software
• Campus and L3 Service Provider software
• XSEDE enterprise services
• VMs, containers, and build recipe sharing
• Community software and service ratings
• Community discussion forums
and … what will help you?
https://software.xsede.org/feedback

27.
END OF PRESENTATION
27

28.
Supplemental material
28

29.
XCI Provided Software -
operating XSEDE
Service Provider Resources
XSEDE Federation
Accounting & Allocations
Description & Status
Login
Data Movement
Execution
Common User
Environment
User Portal
Information
Services
Security
Services
Accounting
Service
Login Services
Data
Services
Monitoring
Services
Enterprise
Services
Blue boxes are XCI provided

30.
30
Architecture Design, Decomposition
Requirements: Use Cases, Qualities
Detailed Implementation Plans
Design, Integration, (Development)
Coordinated Deployment
Production Support
Architecture Active Design Review
Integration Test
Team(s): A&D, S&SE, TIS
XSEDE Software Engineering Flow
Acceptance Test
Team(s): SD&I
Area(s): Operations, User Services
Users Pilots Service
Providers
Team(s): TEOS, ECSS, …
User & Operator
experience
Operations
enhancements
Bug fixes, routine
enhancements
Need for new
capabilities
Implementation-ready
architecture
Deployment-ready
software and service
Operational
infrastructure