1. Making the most out
of your test suite
Eric Holscher
http://ericholscher.com
Djangocon.eu 2010

2. Overview
» Defining your tests
» Running your tests on every commit
» Monitoring and deployment
» Playing with test data

3. Defining your tests

4. I download your code...

5. Now I want to run your tests

6. How do I do that?

7. Ways to run Django tests
» ./manage.py test <app>
» ./manage.py test testapp
» nosetests
» setup.py test
» run_tests.py
» ...

8. Python has even more

9. We need a standard way to
run Python tests

10. setup.py test is the answer

11. ...but it’s not ready

12. setup.py test
» Part of setuptools/distribute
» Will be a part of distutils2
» Django has no support for this yet
» We can lead standardization efforts

13. Pony Build

14. Pony Build
» Started by Titus Brown
» Super simple Continuous Integration
Server
» Interesting part is the client
» github.com/ericholscher/pony_barn

15. Define how to run tests in
Python

16. Full Example
class PonyBuild(BaseBuild):
def __init__(self):
super(PonyBuild, self).__init__()
self.name = "sphinx"
self.required = ['nose']
self.repo_url = 'http://bitbucket.org/birkenfeld/sphinx/'
def define_commands(self):
self.commands = [
pony.HgClone(self.repo_url, egg=self.get_name()),
pony.BuildCommand([self.context.python, 'setup.py', 'install']),
pony.TestCommand([self.context.python, 'tests/run.py']),
]

17. Supports standard ways of
running tests out of the box

18. Ships with support for
» ./manage.py test <app>
» ./manage.py test <testapp>
» nose
» setup.py test

19. Simple Pony Build
class PonyBuild(DjangoGitBuild):
def __init__(self):
super(PonyBuild, self).__init__()
self.repo_url = "git://github.com/alex/django-filter.git"
self.name = "django-filter"
self.package_name = 'django_filters'
self.installed_apps = ['django_filters', 'django_filters.tests']

20. Lets you run your build in
different environments

21. Different Environments
» Local Environment
» Temporary Directory
» Virtualenv

22. At least ship with a shell
script

23. Document how to run tests

24. Continuous Integration

25. “Run your tests on every
commit”

26. A fancy wrapper around a
shell script

27. Without this, your tests are
useless

28. Make sure you setup.py
install your package!

29. --noinput is your friend

30. Use Virtualenv

31. Watch Titus’s Pycon Talk

32. Three main options
» Buildbot
» Hudson
» Devmason

33. Hudson

34. Hudson stats
» Written in Java :x
» Great plugin community
» Super simple to setup
» Becoming Defacto
» Worthwhile if you have resources

35. We use this at work and it
kicks ass

36. hudson.djangoproject.com

37. Setting up your own

38. Useful Capabilities
» Coverage
» XML Output
» PyLint
» Profiling

39. Plugins
» Hudson Cobertura Plugin
» Hudson Violations Plugin
» IRC Plugin

40. pip install coverage
(Compiles a C extension)

41. Coverage.py
coverage run manage.py test
or
coverage run 1.1.X/tests/runtests.py
coverage xml -o 1.1.X/coverage.xml

43. XML Output
» Gareth Rushgrove’s django-test-
extensions
» Requires a subclassed TestRunner
» github.com/garethr/django-test-
extensions
» github.com/danielfm/unittest-xml-
reporting

44. Example Build Bits
from my_test_settings import *
TEST_RUNNER =
‘test_extensions.test_runners.xmloutput’

46. Nightly Tasks
» pylint
» profiling

47. django-nose makes a lot of
this easier

48. Devmason

49. Language Agnostic Build
Reporting API

50. <3 Piston

51. devmason.com/docs

52. Integrates with Github and
Bitbucket

53. Have simple API support for
Python unittest and Pony
Build

54. Keep your builds in the
Pony Barn

55. “Free” for Python
community

56. Will hopefully help
standardize tests

57. github.com/ericholscher/
devmason-server

58. github.com/ericholscher/
devmason-utils

59. Monitoring & Deployment

61. “Django Kong”
» “Functional Testing”
» “Devops”
» Uses Twill: twill.idyll.org
» github.com/ericholscher/django-kong

63. Simple Example
go {{ site.url }}/
code 200
find “Awesome”
follow “Events”
code 200
find “Today’s Events”

64. Check to see if your code
works on live sites

65. Easily run your tests across
all types of a site

66. Catch errors users are
having but never reporting

67. Gives you a neat “feel” for
your sites

68. We check all of our sites
every 3 minutes

69. A neat hack allows you to
test your site locally too!

70. Stuff to do
» Nagios integration
» “Run levels”
» Better aggregates timing data
» Have tests live on disk/in VCS

71. Run nightly automated
tools to check your sites

72. Patu

73. Test Utils Crawler

74. Get a Dashboard!

76. Test Data

77. My Pony

78. We all run tests a lot.

79. There is valuable data about
our software there.

80. CPANTS
» Perl’s Effort of aggregating test data
» Lots of pretty graphs

83. Cpants pivots
» Operating System
» Architecture
» Language Versions

84. We need this for Python

85. Devmason can serve as the
reporting platform

86. We have all the tech we
need, just need to tie it
together

87. Things to Remember

88. Document how to run your
tests (in executable form)

89. Get a CI server setup

90. What you don’t measure
you don’t improve

91. Get some basic monitoring
for your sites functionality

92. URLS
» github.com/ericholscher/django-kong
» hudson-ci.org
» github.com/ctb/pony-build
» pypants.org
» cpantesters.org
» devmason.com
» github.com/ericholscher/devmason-server
» github.com/ericholscher/devmason-utils

93. Questions?
» eric@ericholscher.com
» twitter.com/ericholscher