xAPI Camp DevLearn 2018 presentation by Aaron Silvers

1. xAPI For Reals:
The State of xAPI in 2018
Aaron E. Silvers
xAPI Camp: DevLearn
23 October 2018
aaron@makingbetter.us

2. In 2018, It’s Difficult to Scale Productivity with xAPI
(But We Know What We Need…)
• Available Talent Who Work with xAPI to a Set of Shared Expectations
• A Way to Automate Complicated, Contextual Data Operations Unique
to xAPI
• Policies that Make xAPI Fit-for-Market

3. So Today, We’ll Share Out the Results of Our 2018
Research Efforts
• xAPI-related Knowledge, Skills & Abilities for Learning Professionals
• xAPI Profiles
• xAPI:
The Specification,
The US Department of Defense Instruction,
The IEEE Standardization

4. xAPI-related Knowledge, Skills &
Abilities for Learning Professionals

5. Take a Picture: Two 2018 Case Studies in xAPI’s
ROI
$1 Billion $1.6 Billion
The US Navy appropriated $1B spend to
modernize enlisted training programs,
converting content and upgrading systems to
leverage xAPI for data-driven "Ready
Relevant Learning,” saving $1.3B by 2021.
The Sailor 2025 Program
The Office of Personnel Management FY19
Budget designates $1.6B to modernize career
development across US Government to
leverage xAPI to enable data-driven
approaches to Learning & Development.
USA Learning
https://news.usni.org/2018/02/23/ready-relevant-learning-pushing-out-to-the-fleet
https://www.opm.gov/about-us/budget-performance/budgets/
congressional-budget-justification-fy2019.pdf

6. What’s Do People Need to Know to Rock xAPI
Projects that Make Data?
• A knowledge of xAPI’s fundamentals
• The development and maintenance of an xAPI data strategy
• A knowledge of market-available Learning Record Providing solutions
and their capabilities

7. A Common Challenge with xAPI…
• Organizations derive the most value from xAPI when the data
generated from one source aligns with similar data from other sources.
• In practice, the tools and content widely available on the market
generate data that follows the structure required of xAPI, but how xAPI
is implemented to create data varies.
• This results in data that is technically interoperable (same data
structure) but lacks semantic interoperability (same meaning): two
“completion” statements from two different authoring tools might not
be recognized as similar.

8. Department of Defense Implementation Plan for Advanced Distributed Learning. Office of the
Deputy Undersecretary of Defense (Readiness & Training), 1999, prhome.defense.gov/
portals/52/documents/rfm/readiness/docs/adl_stratplan.pdf.
“In short, the strategy is to: pursue emerging network-based
technologies; create common standards that will enable reuse
and interoperability of learning content; lower development
costs; promote widespread collaboration that can satisfy
common needs; enhance performance with next-generation
learning technologies; work closely with industry to influence
COTS product development cycle; and establish a coordinated
implementation process.”

9. Investigating Performance
Do I currently have this Knowledge and these Skills and Abilities?
If not, what am I willing to do to obtain these KSAs?
Effron, Janet Laane, and Sean Putman. Investigating Performance: Design and Outcomes with xAPI.
Edited by Megan Bowe, MakingBetter, 2017.

10. Get Ready for 14 Slides of All New,
SRSLY Serious Competencies
(nobody’s joking)

11. Knowledge: xAPI’s Fundamentals
Skill/Ability Performance Measure
Understands/Explains
Observed by Supervisor to a) engage in
educational opportunities and b) explain xAPI
fundamentals
Required elements of an xAPI activity statement
Optional elements of an xAPI activity statement

12. Knowledge: xAPI’s Fundamentals
Skill/Ability Performance Measure
Describes
Observed by Supervisor to a) engage in
educational opportunities and b) explain xAPI
fundamentalsVerbs, Activities, Activity Types, Attachment Usage
Types, Document Resources, and Extensions that
may be required of the Design

13. Knowledge: xAPI’s Fundamentals
Skill/Ability Performance Measure
Determines
Observed by Supervisor to a) engage in
educational opportunities and b) explain xAPI
fundamentals
A well-written activity statement from
poorly written statements
What should go into an Extension

14. Knowledge: Development and Maintenance of an
xAPI Data Strategy
Skill/Ability Performance Measure
Understands/Explains
Established ontology and/or taxonomy used in the
organization
Observed by Supervisor to produce or maintain a
conformant xAPI Profile
Observed by Enterprise-level Data Architecture
Stakeholders that the xAPI Profile conforms to or
supports required taxonomy/ontology conventions

15. Knowledge: Development and Maintenance of an
xAPI Data Strategy
Skill/Ability Performance Measure
Identifies
Observed by Supervisor to produce or maintain a
conformant xAPI Profile
Observed by Enterprise-level Data Architecture
Stakeholders that the xAPI Profile conforms to or
supports required taxonomy/ontology conventions
Key interactions in the Design
Key performance indicators in the Design
Established xAPI vocabulary, activities and Profiles
required in the Design

16. Knowledge: Development and Maintenance of an
xAPI Data Strategy
Skill/Ability Performance Measure
Defines
Observed by Supervisor to produce or maintain a
conformant xAPI Profile
Observed by Enterprise-level Data Architecture
Stakeholders that the xAPI Profile conforms to or
supports required taxonomy/ontology conventions
xAPI activity statements related to key interactions
Groupings or patterns of xAPI activity statements
that evidence key performance indicators
A maintenance plan that addresses evolutionary
changes and graceful retirement of vocabulary no
longer used

17. Knowledge: Development and Maintenance of an
xAPI Data Strategy
Skill/Ability Performance Measure
Develops/Maintains
Observed by Supervisor to produce or maintain a
conformant xAPI Profile
Observed by Enterprise-level Data Architecture
Stakeholders that the xAPI Profile conforms to or
supports required taxonomy/ontology conventions
An xAPI Profile conformant to the xAPI Profile
specification

18. Knowledge: Market-available Learning Record Provider Solutions and
Capabilities
Skill/Ability Performance Measure
Understands/Explains Validated by third-party services that the activity
statements generated by the Learning Record
Provider conform to the xAPI specification
Confirmed by Enterprise-level Data Architecture
Stakeholders that the generated activity
statements support or conform to Design
requirements
Commercial, Off-the-Shelf Tools on the market that
potentially support the Design requirements
Open-source code libraries that potentially support
the Design requirements

19. Knowledge: Market-available Learning Record Provider Solutions and
Capabilities
Skill/Ability Performance Measure
Identifies Validated by third-party services that the activity
statements generated by the Learning Record
Provider conform to the xAPI specification
Confirmed by Enterprise-level Data Architecture
Stakeholders that the generated activity
statements support or conform to Design
requirements
Activity statements and any other xAPI-related
capabilities generated by the given Learning
Record Provider

20. Knowledge: Market-available Learning Record Provider Solutions and
Capabilities
Skill/Ability Performance Measure
Evaluates Validated by third-party services that the activity
statements generated by the Learning Record
Provider conform to the xAPI specification
Confirmed by Enterprise-level Data Architecture
Stakeholders that the generated activity
statements support or conform to Design
requirements
Generated activity statements against defined
needs

21. Common Roles & Phases in Agile xAPI Projects
that Generate Data
Role Manage Design Develop Deliver
Product Owner
Business Analyst
Instructional Designer
User Experience (UX) Designer
Engineer (Back-end)
Engineer (Front-end)
Engineer (QA)
Delivery Manager

22. Quality Controls and Applicable KSAs: Manage
Quality Controls KSAs Roles
Develop the Business Case
Knowledge of xAPI’s Fundamentals
• Product Owner
• Business Analyst
Perform SWOT Analysis
Create the Opportunity Statement
Define Project Objectives
Develop the Project Scope • Knowledge of xAPI’s Fundamentals
• Development and Maintenance of an
xAPI Data StrategyComplete the Project Plan
Finalize Project Charter Knowledge of xAPI’s Fundamentals
Develop a Process Flowchart
• Knowledge of xAPI’s Fundamentals
• Development and Maintenance of an
xAPI Data Strategy

23. Quality Controls and Applicable KSAs: Design
Quality Controls KSAs Roles
Translate Requirements into Critical-to-Quality
Components
Development and Maintenance of an
xAPI Data Strategy
• Business Analyst
• Instructional Designer
• UX Designer
Conduct Benchmarking
Knowledge of Market-Available
Learning Record Provider Solutions and
Capabilities
• Product Owner
• Business Analyst
• Instructional Designer
• UX Designer
Reduce the Set of Potential Design Concepts
Development and Maintenance of an
xAPI Data Strategy
• Product Owner
• Instructional Designer
• UX Designer
Evaluate Potential Design Concepts
Development and Maintenance of an
xAPI Data Strategy
• Product Owner
• Business Analyst
• Instructional Designer
• UX Designer
Evaluate the Prototyped Design
• Knowledge of xAPI’s Fundamentals
• Development and Maintenance of an xAPI Data
Strategy
• Knowledge of Market-Available
Learning Record Provider Solutions and
Capabilities
• Product Owner
• Business Analyst
• Instructional Designer
• UX Designer

24. Quality Controls and Applicable KSAs: Develop
Quality Controls KSAs Roles
Evaluate the work product
functions to design
specifications
• Knowledge of xAPI’s
Fundamentals
• Development and
Maintenance of an xAPI
Data Strategy
• Product Owner
• Business Analyst
• UX Designer
• Instructional Designer
• Engineer (Back-end)
• Engineer (Front-end)
• Engineer (QA)

25. Quality Controls and Applicable KSAs: Deliver
Quality Controls KSAs Roles
Compare Delivery Approaches
Development and
Maintenance of an xAPI Data
Strategy
• Product Owner
• Delivery Manager
Confirm the Product Meets
Business Requirements
• Product Owner
• Engineer (QA)
• Delivery Manager
Confirm the Product Achieves
Strategic Goals and Objectives
• Product Owner
• Business Analyst
• Delivery Manager

26. That’s a lot… is it
worth it?
When we’re dealing with high stakes
performance (Clinical Medicine, Legal
Compliance, Military Operations, Safety), the
level of professionalism and effort needed is
intense…
When high stakes performance matters, the
level of effort required is less for a greater
return on investment (ROI).

27. In Case It’s Buried by Knowledge…
Two 2018 Case Studies in xAPI’s ROI
$1 Billion $1.6 Billion
The US Navy appropriated $1B spend to
modernize enlisted training programs,
converting content and upgrading systems to
leverage xAPI for data-driven "Ready
Relevant Learning,” saving $1.3B by 2021.
The Sailor 2025 Program
The Office of Personnel Management FY19
Budget designates $1.6B to modernize career
development across US Government to
leverage xAPI to enable data-driven
approaches to Learning & Development.
USA Learning
https://news.usni.org/2018/02/23/ready-relevant-learning-pushing-out-to-the-fleet
https://www.opm.gov/about-us/budget-performance/budgets/
congressional-budget-justification-fy2019.pdf

28. xAPI Profiles

29. What challenges with SCORM
did xAPI address?

30. What challenges with xAPI does xAPI Profiles
address?
• Aligns Data Generated by Multiple Sources
• Scales xAPI’s Best Practices
• Reduces the Skill Level Required to Produce High Quality Learning
Data
• Allows for Version Control and Retrospective Analysis

31. What challenges with xAPI does xAPI Profiles
address… for YOU?
• Supports better, more accurate reporting of learning experiences
involving multiple learning activities.
• Embeds xAPI’s best practices into workflows, making it easier to work
well with xAPI.

32. Case Study:
Elsevier Transition-
to-Practice
Almost half of all newly graduated registered
nurses quit within the first two years. A new
product being developed leverages multiple
approaches to identify newly graduated
registered nurses who are at-risk. These activities
yield targeted feedback for nursing managers
about engagement, competency (and gaps) to
help nurses feel more confident and more
supported, and improve on that attrition rate.

33. Artisanal, Hand-Crafted Data
• I have to look up what vocabulary already
exists that may be related to the goals of our
content, before we have all the content.
• Next, I have to draft model statements that
our content, assessments, surveys and other
apps should be making.
• After reviewing the approach with
stakeholders, I have to validate that these
statements actually conform well to the
xAPI spec.
• Then, after documenting all this, developers
need to translate the specifications I write
into actual code, which has to be tested
multiple times to make sure the data
generated is to-spec.

34. WHAT ARE WE TRACKING?
…is the question I imagine you’re asking or gonna ask me.

35. Our First Prototype uses xAPI to Track…
• Content-Specific eLearning Events
• Assessment Events
• Feature Usage Events
• Generalized eLearning Events
• Navigation
• Media Events
• Reading Events

36. BETTER xAPI DATA IS
HANDCODED (TODAY)!

37. Current State: Vocabulary (not quite xAPI Profiles)
Vocab Server - https://xapi.vocab.pub
The approaches to publishing and using linked data are the same as xAPI Profiles, but there
are differences between “vocabulary” and profiles.
xAPI Profiles have more requirements and metadata than what’s available through the Vocab
Server, the interface to ADL’s ontology set.
• ADL has a vocabulary and it also has an xAPI Profile for SCORM. The SCORM profile can
use the ADL vocabulary and other vocabularies in addition to custom data elements,
statement templates/patterns defined within the profile.
• Medbiquitous has a core vocabulary, but plan to build several different profiles based on
their various medical scenarios or use cases. They have already built one profile called
Virtual Patient.

38. Current State: xAPI Profiles
The xAPI Profile specification has two main parts.
• One section describes how a profile should be documented with
metadata describing an xAPI Concept. Multiple xAPI Concepts
make up one profile. This section of the xAPI Profile specification
offers guidance on how vocabulary should linked together with
metadata across different profiles.
• The other section of the xAPI Profile specification details
conformance requirements for minimal functionality from a profile
server. This section requires that a profile server be able support:
• Statement template validation
• Statement pattern validation
7%
93%
The current Vocal Server meets only 3 of
38 identified requirements for an xAPI Profile Server

40. xAPI: The IEEE Standard

41. Four Stages of IEEE Standard Development
1. Project Initiation & Standard Development: Projects are initiated by existing Working Groups or by external
groups who propose new standardization projects. =The bulk of the work in creating a standard involves
developing a draft standard that is ready for the formal IEEE approval process. This work is carried out by the
individual working groups listed in the Navigation panel to the left. Work is done via email lists, teleconferences,
and face-to-face meetings as needed. Any materially interested person may participate. LTSC membership is
required for posting to the mailing lists and accessing working group and study group documents and IEEE
Standards Association membership is required for voting. Membership is only available on an individual basis –
organizations may send individual representatives but may not join as “entities.”
2. Standards Approval: When a Working Group decides it is ready (and the LTSC Sponsor Executive Committee
approves), a draft standard is put into the formal IEEE Sponsor Ballot process. This is managed by and requires
membership in the IEEE
3. Standards Association: Requires membership in either the IEEE Computer Society, the IEEE or both. Please
see the membership page for more details.
4. Standards Maintenance: Maintaining a standard is the responsibility of the Working Group that produced it.
Please contact the appropriate Working Group chair if you would like to comment on an existing standard.

42. STANDARDIZATION!
P92741 (xAPI) Work Group is a
spinoff of the work of the xAPI
Technical Working Group. There
are three components –
• P92741.1 (xAPI Specification)
• P92741.2 (xAPI Profile Spec)
• P92741.2.* (Individual Profiles)

43. Where will xAPI be in 2019?
in 2020?
xAPI
2019
2020

44. Thank You
The State of xAPI in 2018
Aaron E. Silvers
xAPI Camp: DevLearn
23 October 2018
aaron@makingbetter.us