2. What is this talk about?
How do we do Perfomance testing with
SCRUM at InfoJobs?
Real User eXperience performance
monitorization
2
3. Who are we?
We are the leading employment job board where professionals and companies converge to satisfy their hiring and
employment needs. Since our begginings in (1998) we have beaten the market’s timing
More than 40% of the time inverted in Interenet job
search in Spain is inverted in Infojobs.net (*)
(*) Source: Nielsen Netratings 2010
3
4. Our people and vocation
200 workers with a shared objective:
“To make it easy for everyone to find the best possible job”
InfoJobs Team
4
5. During this 30 minutes…
51 people will be hired using InfoJobs(1)
5 companies will post 58 job ads(2)
1,000,000 requests will be processed by
our Real User eXperience monitoring
systems
(1) Contracts signed during 2009 in Spain through InfoJobs, according to an independent study from Salvetti & Llombart
(2) Data from InfoJobs (jan-abr 2011)
6
6. In the last month…
8 out of 10 big Spanish companies used
InfoJobs(1)
3,000 companies signed up at InfoJobs to
post job offers (1)
33,5 million job offer searches have been
executed (2)
(1) Source: InfoJobs.net (April 2011)
(2) Source: InfoJobs .net (February 2011)
7
7. Software development
P.O ?*
T.L P.O ?*
T.L P.O ?*
T.L
S.M S.M S.M
Team Team Team
#* Dev ? Test #* Dev ? Test #* Dev ? Test
6 SCRUM teams working together on the
# Dev ? Test # Dev ? Test # Dev ? Test
#
employment and training sites
Dev UX # Dev UX # Dev UX
# Dev <> Front # Dev <> Front # Dev <> Front
P.O ?*
T.L P.O ?*
T.L P.O ?*
T.L
1 integrated production release every 2
S.M S.M S.M
Team Team Team
weeks
#* Dev ? Test #* Dev ? Test #* Dev ? Test
# Dev ? Test # Dev ? Test # Dev ? Test
# Dev UX # Dev UX # Dev UX
# Dev <> Front # Dev <> Front # Dev <> Front
10
9. But what about real users?
We want to
measure how
InfoJobs
performs for
every real
user request.
13
10. QA Performance Testing
Performance Testing in Infojobs
• Performance Test of the code generated by
dev team.
• Give standards for external developments.
• Test Infrastructure changes before being in
production.
• Keep track of performance evolution of the
application and systems.
• Keep track of Real User Experience
including 3rd party services such as banners
11. QA Performance Testing
Performance Testing in Infojobs
• Running tests in an isolated
environment similar to
production and sharing
networking infrastructure.
• Load tests with 150 Virtual
users generating 8 transactions
per second that is the load
peak in a front end during a
Monday morning.
• Benchmarks from Database,
Application server … generated
by the same monitoring tools as
in production.
• Tests are ran every release
before going online.
12. QA Performance Testing
Key Performance Indicators: Client Side
• Response Time
• Number of
Transactions
• Errors
• Recovery Capacity
13. QA Performance Testing
Key Performance Indicators: Server Side
• CPU
• SQL/second
• Jboss Cache
• Connection Pools
• Exceptions
14. RUX QA Performance Testing
• Google announces performance will be
considered for their SEO algorithms.
• RUX, real user experience is the part of the
Syslog application that provides information about
the session time of a user in Infojobs
• The metrics are adjusted so they fit the ones
provided by Google
15. QA Performance Testing
RUX
• Infojobs tracks every
request that arrives to
our systems
• Capability of
immediate response to
lack of availability
• Running Test A/B
frontend
16. QA Performance Testing
RUX: What does it measure
• In the web site: Every page includes in the footer a
pixel timer.jpeg that is called after the WindowLoad
event of the browser.
• In the server: the request arrives to the web server
and the time is calculated
• Basic graphic about what is measured
Internet Apache JBoss Busy sending rendering Internet Syslog
User
RUX
Infojobs Browser Timer.jpeg
17. QA Performance Testing
RUX: Median and Quartiles
• Not all the requests are measured. Only those that load
timer.jpeg and those that are between reasonable values that
we calculate considering median and 1st 3rd and quartile
18. QA Performance Testing
RUX: Application
• Weekly measurement for all servers all requests.
• Exactly at minute measurements are available
filtered by page and front-end.
• Accurate measurement of infojobs User’s
Experience.
23. QA Performance Testing
Goals: Operations – QA - DEV
• Cooperation amongst
departments. Keep up to
date information from the
systems and the
application.
• Maintenance of Systems
( DB, Application Servers ) for
functional testing
24. Challenge: SCRUM and Performance
• 6 Scrum Teams 1
performance
environment
• Load Tests vs Real
Load
• Production vs Pre
Production
• Comparison
reports not
available
