Alice Till-Carty presented on testing at startups. She discussed her experience growing the testing team at Lyst, a fashion aggregation startup, from 1 to 3 testers as the development team grew from 20 to 36 developers. Some challenges included justifying fixing bugs, attitudes towards testing, and lack of communication. Process changes included adopting a "test, learn, adapt" approach and shifting responsibility for automation to developers. Lessons included that one size does not fit all, communication is key, and quality upfront is important for startups focused on growth.

1. T13
Test
Management
5/5/16
13:30
Testing
at
Startup
Companies:
What,
When,
Where,
and
How
Presented
by:
Alice
Till-­‐Carty
Lyst
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.stareast.techwell.com/

2. Alice
Till-­‐Carty
Lyst
A
senior
QA
engineer
for
Lyst,
one
of
London's
fastest-growing
startups,
Alice
Till-Carty
landed
her
career
in
testing
sideways.
Having
studied
film
at
the
university,
Alice's
obsession
with
technology
and
problem
solving
gave
her
a
head
start
in
testing.
With
six
years
of
experience
across
two
countries,
four
companies,
and
more
than
one hundred
projects,
Alice
has
worn
many
different
testing
hats: consultant,
analyst,
engineer,
and
quality
evangelist.
She
considers
herself
a
fierce
advocate
for
the
end
user.
Alice's
determination
for
always
making
things
better
for
the
customer
has
given
her
a
unique
perspective
on
test
process
improvement.
Follow
her
on
Twitter
@aliceness.

3. Alice Till-Carty (@aliceness)
what, when, where and how
or, how to show value when none is perceived
testing & startups:
1

4. • originally from New Zealand
• background in manual testing, with a focus on end user & usability testing
• worked in a number of small companies, mostly eCommerce-related
• but you don’t really care about any of this, you’re here to learn about startups and
testing!
hi, I’m Alice
2

5. so, what is a startup?
3

6. • fashion aggregation website
• continuous deployment
• heavily data-based
• half-tech, half-fashion. engineering/product team of 58, the remaining 69
made up of marketing, comms, operations & sales
the startup
4

7. … and, what is value?
5
noun. the regard that something is held to deserve; the importance, worth, or usefulness of
something.

8. A lot of companies are hiring for testers.
(and this was just when I looked at jobs in London!)

9. ‘QA Engineer’
7
lyst is founded in 2010
first dedicated QA (QA lead) hired in 2013
second dedicated QA (me!) hired in march 2015 (total count: 2)
third dedicated QA hired in may 2015 (total count: 3)
… third dedicated QA leaves in october 2015 (total count: 2)
the second third dedicated (mobile) QA joins in december 2015 (total count: 3)
The originally hired QA (the QA lead, and my boss!) leaves in april 2016 (total count:
2)

10. 1 tester
20 developers
then, 2 testers to 20 developers
then 25, then 30
now:
2 testers to 36 developers
the situation
8

11. to improve quality across the website and the iOS application
to encourage teams to think about quality
to come up with and implement a process that addresses each individual team’s needs in the best
way possible
to educate and improve developer’s testing skills
the challenge
9

12. there were a few!
challenges
10

13. always needing a justification for fixing a
bug

14. attitude towards
test

15. test as a separate team from the rest of engineering

16. responsibility of automation - who, and why?

17. working primarily with developers who had never
worked with testers before

18. lack of communication between the business and
engineering, and even between teams within
engineering

19. 17

20. - Test, learn, adapt
- one size fits all doesn’t work - but we tried it anyway!
- a shift in responsibility: quality up front instead of at the
end
- finding ways to show ‘value’
- automation responsibility now belongs to the developers
process changes
18

21. - the one size fits all approach: didn’t work.
- communication is the lifeblood of the development
process.
- we definitely still hadn’t solved all of the problems.
So, what impact did these changes have?

22. - quality upfront is important as quality at the end
- the smaller the change, the smaller the amount of testing that needs to
be done
- encouraging a routine when it comes to QA/test helps developers to
think about quality regardless of how much (or how little) risk there is in a
change
- get to know your developers - having a good working relationship with
them will always make your job easier!
Lessons I wish I had known from the start
20

23. 21
What do I do now?

24. then now
Test responsible for automation Developers responsible for UI automation tests
Primarily automated tests, with some manual testing
Extensive unit test coverage, a suite of automated
tests that are maintained and kept up to date, as well
as exploratory manual testing of all front-end features
Head of product + developers would decide on and
implement new features
All developers, tester, designers, and product owner
size and define new features together
No focus / monitoring of performance, security and
other non-functional issues like usability and UX
experience
Have a dedicated in-house UX Researcher,
tools/resources to measure performance and security
issues
Test in a separate team
Test as an integrated part of the respective team they
are responsible for
now vs. then
22

25. Some things to remember!

26. an inevitable evil.
context switching
24

27. a test role at a startup is NOT going to be the same from one company to the
next.
being flexible is important!
25

28. for startups, the most important thing is growth.
the relationship between startups & testing
26

29. A startup is a company designed to grow fast. Being newly founded does not in itself make a
company a startup. Nor is it necessary for a startup to work on technology, or take venture funding,
or have some sort of "exit." The only essential thing is growth. Everything else we associate with
startups follows from growth.
Paul Graham on startups
27

30. the most important thing is that everyone is on the same
page about what QA is there to achieve.
so, what should test do at a startup?
28

31. we haven’t solved all the problems yet!
- grow the test team!
- find out and define the role that automation plays within our code base.
is it a priority? Why?
- continue to encourage a culture where everyone believes quality is
important.
the future
29

32. - iteration is important when improving process. A complete overhaul once every quarter will just confuse
people.
- if you’re going to use a metric to define quality: it’s important to decide what you’re going to measure,
AND start to measure it before you decide to use it as a metric. This is so you have a yardstick to define
whether the process has improved the metric or not.
- having allies will make everything a lot easier!
- use EVERY opportunity to have to show value.
lessons learned
30

33. Should startups have test?

34. value
32

35. thank you
questions?