Slides from PNSQC's Culture Jam 2021. This year we had 7 great speakers at our annual kick-off for the #PNSQC2021 conference year.

1. 1
Pacific Northwest Software
Quality Conference
Achieving higher quality software through knowledge exchange
#PNSQC2021
Culture Jam 2021

2. 2
•
•
•
•
•
•
About PNSQC

3. 3

4. • Opening Context w/ Phil Lew - Program Chair
• Lightning Talks 1-3
• Break With 2020 Awards
• Lightning Talks 4-7
• Event Closing
• Happy Hour
4
Welcome

6. 6
“It is not the strongest of the species that
survives, nor the most intelligent that survives.
It is the one that is most adaptable to change.”

7. WE NEED TO LEARN
PROACTIVELY
7

8. ● Providing valuable
content
● Fostering community
● Networking
● Learning
● Sharing
PNSQC
Your Learning Platform

9. 9
Some Great Lightning Talks Lined Up
1. Ensuring A Quality-first Approach In Software Development And
Implementation -- A Manager's Perspective with Amy Sawyer
2. Cybersecurity by Design Without Compromising UX and Quality
with John Klassen
3. Why Testers Belong at the Hackathon with Rachael Lovallo
4. New Age of Test Automation with Iryna Suprun
5. Useful UAT with Robin Goldsmith
6. From 3 to 1 Easier Said Than Done with Shiva Srinivasan
7. Automation Does Not Replace Collaboration with Stephen Kilbourn
#PNSQC2021
2020 Awards-Refreshments/Networking
Closing and Happy Hour

10. 10

11. Welcome
(1) Ensuring A Quality-first Approach In Software
Development And Implementation -- A
Manager's Perspective
– Amy Sawyer
#PNSQC2021

15. Welcome
(2) Cybersecurity by Design Without
Compromising UX and Quality
- John Klassen
#PNSQC2021

16. Cybersecurity by Design
without compromising UX and Quality

17. Kudos to last year’s security talks!
•Daniel Kranowski: Security as the new Non-Functional
Requirement (NFR)
•Bhushan Gupta: Be Agile with Security

19. Cybersecurity, UX, and Software Quality
Pick 2
● Why isn’t every system Secure by Design?
● If you make security hard, people may work around it.
● What is the quality of your software supply chain?
● Cybersecurity, UX and Software Quality have become
interwoven. What are the ramifications?

20. Security cannot be an Afterthought
● The internet took off because of an open-trust model. But
that has also been a key failure from a security standpoint.
● It’s not as if the hacking threat is new; it’s been going on,
and its scope and possible fixes have been known, for a
very long time.
● In April 1967, just before the ARPANET’s rollout, an
engineer named Willis Ware wrote a paper called
“Security and Privacy in Computer Systems”

21. Kick the can down the road
● The ARPANET dev team was annoyed. They begged their leader not to
saddle them with a security requirement.
● Let’s do this step by step, the team said. It had been hard enough to get
the system to work; the Russians wouldn’t be able to match it for decades.
● So vast systems and networks would sprout up throughout the U.S. and
much of the world, without any provisions for security.
● Some security provisions would be backfitted later, but the vulnerability
that Ware and the later studies observed was built into the technology.
● And the Russians did catch up.

22. Get it working, then secure it
Security takes a backseat to new features, reliability, performance,
scalability, ease of use, integration, etc.
● Email
● Microsoft Windows
● Web Browser
“If the Greeks were creating the Trojan Horse today, it would be a
web browser.”

23. “Detect not Prevent” Legacy
● Security isn’t intrinsic and flaws are easy to introduce into products, so
we resort to scanning each file with 3rd
party tools to detect malware.
That’s not enough.
● Think evil, do good. Penetration testing is a good start.
● “We didn’t focus on how you could wreck this system intentionally,” said
Vinton G. Cerf, a parent of the Internet
● “Cybersecurity is kind of like safe sex. There’s this sense that the
[Internet] provider’s not going to protect you. The government’s not
going to protect you. It’s kind of up to you to protect yourself.”
- Janet Abbate, the Virginia Tech historian

24. Usable Cybersecurity
● Gold Standard for UX Security: Your users can still do everything they
did before installing [Security Tool] but now they’re protected.
● No change to user behavior
● Imperceptible latency
● Does not block users from getting to the internet resources they need
to do their work
● "Ultimately for the user, security should be automatic and barely
noticeable.“*

25. Supply Chain Security
● Russians use known vulnerabilities to attack customers of SolarWinds by
adding malicious code to SolarWinds’ product* before updates are sent to
customers.
● Most software providers regularly send out updates to their systems,
whether it's fixing a bug or adding new features. SolarWinds is no
exception. Beginning as early as March 2020, SolarWinds unwittingly
sent out software updates to its customers that included the hacked code.
● The code created a backdoor to customer's systems, which hackers then
used to install even more malware that helped them spy on companies
and organizations.
● The idea is not new, just the hacker’s ability to execute.

26. SolarWinds
15 months from penetration to identification

27. Is it safe?
● Do you test software updates from your security vendors before you
install them?
● Or do you trust the security vendor to test for cybersecurity issues
before releasing their releases and patches?
● McAfee team Beaverton Oregon runs a multi-vendor “Soak Test”
before sending code to customers

28. The Takeaway
Attackers are financially motivated, innovative and
persistent. Tools we use for good they use for evil.
Cybersecurity, UX and Software Quality have become
interwoven.
• Security can’t be an island, a set of features to kick down the road.
• Security isn’t effective if users refuse to use it
• Software Quality for Security: Think evil, do good.

29. Welcome
(3) Why Testers Belong at the Hackathon
- Rachael Lovallo
#PNSQC2021

35. 35

36. Break, Drinks, and
#PNSQC2020 Award Ceremony
• Communication is Key: Lessons Learned from Testing in
Healthcare Technology - Rachael Lovallo
#PNSQC2021
• Iron Chef Cucumber: Cooking Up Software
Requirements That Get Great Results - Chris Cowell
• Of Machines and Men - Iryna Suprun

37. 37

38. A Special Thanks to Our 2020 Sponsors
#PNSQC2021

39. Welcome
(4) New Age of Test Automation
- Iryna Suprun
#PNSQC2021

40. New Age of Test Automation
Iryna Suprun
Xand, NYC

41. Why it is so hard?
1. Still a challenging task after many many years of doing this.
2. Automation code should be better than software under test code. More
stable, more reliable, more trustworthy.
3. Growing complexity of the software under tests

42. Homegrown VS third-party
1. Homegrown tools a tailored suit - fits you perfectly and looks great if
a. The people who build have adequate set of skills
b. There is enough time
c. The language and supporting technologies are selected correctly
2. Third party testing tool
a. Will never check all the boxes unless you building something trivial
and very standard
b. Good tool might be costly, you might need more than one

43. New(er) players on the market
● Why newer, not new? Too risky. Too many tried to solve automation
problems and failed
● “Industry-Leading”, “Game changing”, “Trusted by many”, “AI-based”,
“Future of testing”, “Fully Autonomous”

44. Easier but not easy
Visual Testing*
● Applitools https://applitools.com/
● Percy https://percy.io
● SmartBear https://crossbrowsertesting.com/
● SauceLabs https://saucelabs.com/
AI - based (test recording and automatic test generation)
● Test.ai https://www.test.ai/
● Mabl https://www.mabl.com/
● TestIm https://www.testim.io/
● Appvance IQ https://www.appvance.ai/

45. New approaches
Test generation based on production Data
● ProdPerfect Prodperfect.com - E2E test suites based on data-driven,
machine-led analysis of live user traffic
Selenium alternative
● Cypress https://www.cypress.io/

46. New Problems to solve
Data quality testing
● BigEval https://bigeval.com/
● iCEDQ https://icedq.com/
AI-bias
● AI BRAT (AI Bias Risk Assesment Tool) http://bias.test.ai/
● Google What-If https://pair-code.github.io/what-if-tool/

47. Will it ever be easy?
● When software development will be easy
● New technologies can improve automation quality and speed
● We can use tools to make some processes easier and some less time
consuming

48. Welcome
(5) Useful UAT
- Robin Goldsmith
#PNSQC2021

49. Useful UAT
- 49
©2021 GO PRO MANAGEMENT, INC.
Useful UAT
GO PRO MANAGEMENT, INC.
SYSTEM ACQUISITION & DEVELOPMENT
QUALITY/TESTING
PRODUCTIVITY
22 CYNTHIA ROAD
NEEDHAM, MA 02494-1461
INFO@GOPROMANAGEMENT.COM
WWW.GOPROMANAGEMENT.COM
(781) 444-5753
Robin F. Goldsmith, JD

50. Useful UAT
- 50
©2021 GO PRO MANAGEMENT, INC.
What’s Acceptance Testing
and Why Do Users Need to Do It?
Self-defense:
Acceptance Testing is the user’s chance
and duty to confirm the system works
properly, when used in a real manner,
from the user’s standpoint, before users
and the organization rely upon it.
Can’t we just trust the developers to do their jobs right?

51. Useful UAT
- 51
©2021 GO PRO MANAGEMENT, INC.
Traditional Development and Testing Gurus Tend
to Miss the Boat on UAT
⦸ Consider UAT a test to confirm product’s system
requirements--often expect a rubber stamp
⦸ Some testing books/courses say
⦸ UAT should be only positive/valid proof-of-concept tests to
demonstrate normal functionality
⦸ UAT should be a repeated subset of the System Test, run by
users
⦸ UAT simply needs one test for each functional requirement or
use case scenario
Yet, organizations continually rely on UAT to catch many missed problems

52. Useful UAT
- 52
©2021 GO PRO MANAGEMENT, INC.
In Agile, Demo Purports to Be UAT
www.pixabay.com

53. Useful UAT
- 53
©2021 GO PRO MANAGEMENT, INC.
Most Acceptance Testing Is Reactive
● At the end, based on system as
written
● Unplanned, unanticipating
● A lot of work, often with little payback
– Doesn’t find many of the errors
– Adversarial, arguments
– Too late to fix errors anyhow

54. Useful UAT
- 54
©2021 GO PRO MANAGEMENT, INC.
Users Often Lack Confidence in
How to Perform Acceptance Testing
● Don’t know how to test (and often neither do
developers)
– “Play with it”
– “Try it out”
● Don’t know how to use system or how it’s
supposed to work
– “Push this button” Why?
– Isn’t this the developer’s job?
– Feel like they broke it
● User view often compromised
Proactive Testing™
provides the basis for more
confident and effective user
participation

55. Useful UAT
- 55
©2021 GO PRO MANAGEMENT, INC.
Proactive User Acceptance Criteria™--Not Just
Reacting to the Requirements
● What the users/customers/stakeholders (may be multiple perspectives)
must have demonstrated to be confident the delivered system works
– 5 categories
– Bet their jobs relying on the system
● Determination will be made whether or not it is conscious, planned, or
explicit
● True empowerment builds cooperation
– Ability to accept or reject delivered system
– Just asking for how decision will be made

56. Useful UAT
- 56
©2021 GO PRO MANAGEMENT, INC.
Go Pro Management, Inc. Seminars/Consulting--Relation to Life Cycle
Proactive Systems/Software Quality Assurance (SQA)™
Feasibility
Analysis
Systems
Analysis
System
Design
Develop-
ment Implement-
ation Operations
Maintenance
Proactive Testing:
Risk-Based Test Planning,
Design, and Management
Testing Early in the Life Cycle
Credibly Managing Projects and Processes with Metrics
21 Ways to Test Requirements
Making You a Leader
Managing Software Acquisition and Outsourcing:
> Purchasing Software and Services
> Controlling an Existing Vendor’s Performance
Proactive User Acceptance Testing
Reusable Test Designs
Test Estimation
Risk
Analysis
Defining and Managing Business
Requirements
Writing Right Agile User Story and Acceptance
Test Requirements Right
System Measurement Test Process Management
ROI

57. Useful UAT
- 57
©2021 GO PRO MANAGEMENT, INC.
Robin F. Goldsmith, JD
www.gopromanagement.com robin@gopromanagement.com
∙ President of Go Pro Management, Inc. consultancy since 1982, working directly with and training professionals in business engineering, requirements
analysis, software acquisition, project management, quality and testing.
∙ Partner with ProveIT.net in REAL ROI™ and ROI Value Modeling™.
∙ Previously a developer, systems programmer/DBA/QA, and project leader with the City of Cleveland, leading financial institutions, and a “Big 4”
consulting firm.
∙ Degrees: Kenyon College, A.B.; Pennsylvania State University, M.S. in Psychology; Suffolk University, J.D.; Boston University, LL.M. in Tax Law.
∙ Published author and frequent speaker at leading professional conferences.
∙ Formerly International Vice President of the Association for Systems Management and Executive Editor of the Journal of Systems Management.
∙ Founding Chairman of the New England Center for Organizational Effectiveness.
∙ Member of the Boston SPIN and SEPG’95 Planning and Program Committees.
∙ Attendee Networking Coordinator for STAR, Better Software, and Test Automation Conferences.
∙ Chair of record-setting attendance BOSCON 2000 and 2001, ASQ Boston Section‘s Annual Quality Conferences.
∙ Member IEEE Std. 829 for Software Test Documentation Standard Revision Committee.
∙ Member IEEE P730 standard for Software Quality Assurance Revision Committee.
∙ International Institute of Business Analysis (IIBA) Business Analysis Body of Knowledge (BABOK) subject expert.
∙ TechTarget SearchSoftwareQuality.com requirements and testing expert.
∙ Admitted to the Massachusetts Bar and licensed to practice law in Massachusetts.
∙ Author of book: Discovering REAL Business Requirements for Software Project Success
∙ Author of forthcoming book: Cut Creep—Write Right Agile User Stories and Acceptance Tests

58. Welcome
(6) From 3 to 1 Easier Said Than Done
- Shiva Srinivasan
#PNSQC2021

59. Welcome
(7) Automation Does Not Replace Collaboration
- Stephen Kilbourn
#PNSQC2021

60. 60

61. Today’s Speakers – Thank You!
The speakers today are your colleagues
with a story to tell.
If you enjoyed their presentation,
connect with and thank them.
#PNSQC2021

62. “What Do You Want To Go Today?”
62
What Do You Want To Learn Today?

63. Give us feedback
Let us know how to
improve your learning
effectiveness
○ What content?
○ What delivery
mechanism?
PNSQC
Your Learning Platform
63

64. 64

65. PNSQC 2020 Diversity Scholarships
Thank you Partners!
Women Who Code
PDX Blacks in Tech
Latinx Tech PDX
Future Ada
#PNSQC2021
Thank You Scholarship Sponsors!

66. Ways To Get Involved with PNSQC
• Volunteer your services
– Edit technical blogs
– Review papers
– Work with your company to become a sponsor
– Organize volunteers & events
– Website mods
– Adwords
• Become an author at our annual conference
– or a contributor to our online blog and article
archive
• Support us on social media
• Become the program chair, or other chair-board
member
#PNSQC2021

67. Closing
Opening the Call for Proposals for #PNSQC2021
1.
a.
2.
a.
b.
c.

68. 68

69. Call for Volunteers
Please step up and volunteer at PNSQC
• Benefits of volunteering:
• Professional development
• Contribution to industry
• Recognition by peers
Contact Us: Email Robert Anderson to get a free t-shirt!
Or add your name to the conference survey or contact us via PNSQC.org
PNSQC
is a non-profit managed
by volunteers passionate
about software quality.
Our mission
is to enable knowledge
exchange to produce
higher quality software
#PNSQC2021

70. 70

71. #PNSQC2021