The Pinto, Pareto, and Performance

1. The Pinto,
Pareto and
Performance
By
Leandro Melendez
-Señor Performo

2. ‘BoutMe
• I am Leandro Melendez
• Performance tester, scripter, engineer and idealist. Perf Manager
@QualiTest
• Lot’sa experience on several performance projects for 10 years.
• Found lots of vices, weird paths of action and plain ignorance.
• Decided to get a secret identity and wear the spandex moustache to fight
against bad practices, process inertia and plain ignorance as Señor
Performo.
• All this through a blog (www.srperf.com), Social NW’s (@srperf), hosting the
Spanish version of the performance testing podcast PerfBytes en Español.
• Last, but not least, through public speaking, spreading the word of
performance, which we will do here today.

3. Economical impacts of
Performance testing practices
(or lack of)

4. Boring title?
• How about make it bout moolah!
• I am gonna be talkin’a lot about MONEY.

5. Boring title?
• How about make it bout moolah!
• I am gonna be talkin’a lot about MONEY.
• And performance of course.
• Especially long term impact.

6. Boring title?
• How about make it bout moolah!
• I am gonna be talkin’a lot about MONEY.
• And performance of course.
• Especially long term impact.
• I will give you two different options.
• Analyze we will.

7. Butfirst youneed…
• I have noticed many do not have
• We need to develop here
• … Sight beyond sight
• On the intersection of:
• Performance
• Money
• Vision
• So you don’t gotta come from the future!
PERFORMANCE!
OOOOHHHHH

8. Onelittle trick
• To make the analysis easier I will go to numbers
• We will do some calculations here and there
• Don’t worry won't hurt
• … much

9. Does thishappen inreal life?
• I am not full of anything.
• On the corp world, the judgement
on short term money impacts is
good, but the long term is not
always good.
• So to start with the doubters here
will come an example for all the nay
sayers.
• Based on a true story.

10. THE PINTO

11. Da Facts
• Young or unfamiliar
• Launched in 1971
• By Ford Motor Co.
• Named after a Pony
• Economy market
• Fastest design ever
• Goal Price $2K USD
• Cheap cheap

12. Hiddencostsofraced project(pun intended)
• When hurrying corners are cut
• The pony had issues… several.
• One was outstanding
• Gas tank issues
• Collision>30mph=Fire
• Solvable with some $

13. Optionsanalysis
APPLY A FIX TO THE FUEL TANK.
• Consisted of adding a rear bumper
• Estimated fix cost per car $11USD
• Number of cars 12.5 million*
• Applied also for trucks
• Trucks had same issue a la
microservice
• Total cost
$137 million USD
~$870 million-Adjusted to 2019
LEAVE IT AND PAY FINES.
• Assign a cost to each event
• 2100 cars burned each @ $700USD
• 180 burns each @ $67,000USD
• 180 deaths each @ $200,000USD
• WTF a human life for $200k???
• Total cost
$49.5 million USD
~$313 million-Adjusted to 2019
*Not only Ford cars

14. Again… True story

15. Optionsanalysis
APPLY A FIX TO THE FUEL TANK.
• Total cost
$137 million USD
LEAVE IT AND PAY FINES.
• Total cost
$49.5 million USD

16. Sightbeyondsight
APPLY A FIX TO THE FUEL TANK.
• Total cost
$137 million USD
• Future cost
1. Less trials.
2. Better reputation.
3. Less production costs.
LITTLE COST
LEAVE IT AND PAY FINES.
• Total cost
$49.5 million USD
• Future cost
1. Multiple trials and investigations.
2. Severely damaged reputation.
3. 1.5 million cars recalled. Largest ever
HOLLY MOTHER OF
UNTANGIBLE HUGE COSTS!!!

17. PERFORMANCE
& IMPORTANCE

18. Performance automation
• Common misconception is to try
to automate everything.
• Or at least what is deemed as
“Important”.
• VIP processes must be
Performance tested. Or
automated.
• But… how do we know what is
IMPORTANT?
• Important for the business or for
automation?

19. Justsayin
In performance testing, automation is generally
recommended
only for LOAD simulation

20. Performance automation
• How to know if process is
actually important?
• Something important for the
solution or company may not be
for performance automation.
• We just saw, performance
automation is for load.
• Then the processes that
generate most load might be
important.
• Meaning multiple interactions.

21. Sill notconvinced?
• I know cause you have been told
for so long what is important,
you may still not buy this very
much.
• It is important to get response
times.
• So how about we throw again
some $$$ numbers?

22. MOOLAH
EXCERCISES

23. Money-xercize
• Special project, code name SNAIL
• It has 50 BPs marked as important for automation
• One process called SNOWFLAKE, happens only once
a week. It is critical for the business as it is so special.
• SNOWFLAKE is actually very important for other
things to work.
• Will happen only once in our load tests.
• It is also critical to know if it performs under load
within time and resources indicated.

24. Money-xercize
• Automate SNOWFLAKE
• Need consultant or scripter
• Between $50 and $100 x hour… Let’s do $75 (not
counting travel and other expenses)
• Average script or automation (well done, no silly
record n’play, test case, documented and data
tested) takes from 4 to 16 hours depending on
complexity.
• Let’s do 8 hours.
• Process will be used 1 (once) per test.

25. Numbers time– Scenario A
• $75USD per hour
• 8 Hours
• 1 time per test.
*Most processes are snowflakes

26. Scenario B–Meet Bob
• Bob knows very well how to run
SNOWFLAKE
• He is so good that it takes him just a
few minutes
• Let’s say it takes him an hour
• Bob does well and is paid about
$5000 USD per month
• A month has about 170 hours

27. Bob’sNumbers– Scenario B
• $5000USD per month
• 170 hours per month
• 1 hour for snowflake
• Ask him to run it

28. Snowflake’s optionsanalysis
SCRIPT
- $600 USD x click
- Requires
maintenance
- Time to create
BOB
- $30 USD per click
- Already on the payroll
- Just call him
- Tells good jokes

29. What aboutresponse time?
• Add manually
• Stop Watch
• Mississippi
• Integrated
• APMs

30. ANOTHER OPTION

31. Snowflakes andincessant
• Most of the 50 BPS are
snowflakes.
• There is one that
happens 1000 times
each hour called
INCESSANT
• To script it takes the
same time as
snowflake.
• Numbers time

32. Numbers time– Scenario Incessant
• $75USD per hour
• 8 Hours
• 1000 times per test.
*Most processes are snowflakes

33. Snowflake VSIncessant optionsanalysis
SNOWFLAKE
- $600 USD x click
- Happens once
- Bob can do it
INCESSANT
- $0.6 USD per click
- Happens A LOT!
- Bob cannot do it

34. Happens more=important
• Automation for
performance and load
has different definition
for important
• Important is what
happens the most (for
automation)
• How do I know which?
• How can I select?

35. PARETO & MOOLAH

36. Meet Pareto
• Vilfredo Pareto
• Italian economist
• Discovered the
PARETO PRINCIPLE
• AKA 80-20 rule
• 80% of land in Italy
owned by 20% of
people
• Applies everywhere!

37. Performance Pareto
• Load testing applies
• 80% of load comes from
• 20% of processes
• Quick guide
• Focus on that 20% that
generates 80% (or more)
• Get all the processes
• List order by occurrence
• Sum
• Find top 80%

38. Example morenumbers
• Big list of processes
• Total 15 important
• Sorted by occurrence
• Sum total of events
• Percentage each
• Sum top percentages
until 80% or more
• Pick the ones who
generate at least 80%
• A B C – YEAH
• D E F G - MEH
Process Times
A 4,088
B 4,087
C 2,057
D 977
E 471
F 169
G 91
H 48
I 33
J 28
K 28
L 23
M 23
N 20
O 20
TOTAL 12,163
34%
34%
17%
8%
4%
1%
1%
0%
0%
0%
0%
0%
0%
0%
0%
84%
14%
2%

39. Only 3= 84%
• Original 15 processes
• Each automation
~$600USD
• Bob costs ~$30USD/hour
• Bang for the buck
• A, B, C generate 80%
• 3 processes vs 15
• Differences

40. Pareto = Savings andmore efficiency
• One decision out of the picture.
• Still one decision

41. Thanks Pareto!
• Thanks to Pareto
• Increase the efficiency and money allocation in performance automation
• Save money, spend it better
• Performance smarter
• Many other sight beyond sight
• Everywhere
• Release rush impact
• Cloud needs no performance
• Many others

42. Thank you

43. QUESTIONS?