Logging in Python for large scale applications.

1. Why use logging
Basics
When to use what?
Logbook
Logging in Python
For complex applications
Fayaz Yusuf Khan
20 December 2013
Error reporting
The end

2. Why use logging
Basics
When to use what?
Logbook
Why use logging?
Error reporting
The end

3. Why use logging
Basics
When to use what?
Logbook
Why use logging?
• All errors happen in runtime.
Error reporting
The end

4. Why use logging
Basics
When to use what?
Logbook
Why use logging?
• All errors happen in runtime.
• Testing can never be 100%
Error reporting
The end

5. Why use logging
Basics
When to use what?
Logbook
Error reporting
import logging
logging . warning ( ’ Watch out ! ’) # will print a message
logging . info ( ’I told you so ’) # will not print a n y t h i n g
The end

6. Why use logging
Basics
When to use what?
Logbook
Error reporting
import logging
logging . warning ( ’ Watch out ! ’) # will print a message
logging . info ( ’I told you so ’) # will not print a n y t h i n g
Output:
WARNING:root:Watch out!
The end

7. Why use logging
Basics
When to use what?
Logbook
Error reporting
Logging to a file
import logging
logging . basicConfig ( filename = ’ example . log ’ ,
level = logging . DEBUG )
logging . debug ( ’ This message should go to the log file ’)
logging . info ( ’ So should this ’)
logging . warning ( ’ And this , too ’)
The end

8. Why use logging
Basics
When to use what?
Logbook
Error reporting
Logging to a file
import logging
logging . basicConfig ( filename = ’ example . log ’ ,
level = logging . DEBUG )
logging . debug ( ’ This message should go to the log file ’)
logging . info ( ’ So should this ’)
logging . warning ( ’ And this , too ’)
Output:
DEBUG:root:This message should go to the log file
INFO:root:So should this
WARNING:root:And this, too
The end

9. Why use logging
Basics
When to use what?
Logbook
Initializing
logger = logging . getLogger ( __name__ )
Error reporting
The end

10. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
print()
Display console output for ordinary usage of a command line script
or program

11. Why use logging
Basics
When to use what?
Logbook
Error reporting
logging.info()
Report events that occur during normal operation of a program
(e.g. for status monitoring or fault investigation)
The end

12. Why use logging
Basics
When to use what?
Logbook
Error reporting
logging.debug()
Very detailed output for diagnostic purposes
The end

13. Why use logging
Basics
When to use what?
Logbook
Error reporting
warnings.warn()
In library code if the issue is avoidable and the client application
should be modified to eliminate the warning.
The end

14. Why use logging
Basics
When to use what?
Logbook
Error reporting
logging.warning()
If there is nothing the client application can do about the
situation, but the event should still be noted
The end

15. Why use logging
Basics
When to use what?
Logbook
Error reporting
Report an error regarding a particular runtime event
The end

16. Why use logging
Basics
When to use what?
Logbook
Error reporting
Raise an exception
Report an error regarding a particular runtime event
The end

17. Why use logging
Basics
When to use what?
Logbook
Error reporting
logging.error() or logging.exception()
Report suppression of an error without raising an exception (e.g.
error handler in a long-running server process)
The end

18. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
logging.critical()
A serious error, indicating that the program itself may be unable to
continue running.

19. Why use logging
Basics
When to use what?
Logbook
Error reporting
Logbook
pythonhosted.org/Logbook
Logbook is a logging sytem for Python that replaces the standard
library’s logging module.
The end

20. Why use logging
Basics
When to use what?
Logbook
Error reporting
>>> from logbook import Logger
>>> log = Logger ( ’ Logbook ’)
>>> log . info ( ’ Hello , World ! ’)
[2010 -07 -23 16:34] INFO : Logbook : Hello , World !
The end

21. Why use logging
Basics
When to use what?
Logbook
Error reporting
Context managers everywhere!
from logbook import SyslogHandler
error_handler = SyslogHandler ( ’ logbook example ’ ,
level = ’ ERROR ’)
with error_handler . a p p l i c a t i o n b o u n d ():
# wh a t e v e r is e x e c u t e d here and an error is logged to
# the error handler
...
The end

22. Why use logging
Basics
When to use what?
Logbook
Error reporting
Getting your libraries and old code to behave
from logbook . compat import r e d i r ec t _ l o g g i n g
r e d i r e ct _ l o g g i n g ()
The end

23. Why use logging
Basics
When to use what?
Logbook
Error reporting
Injecting context
import os
from logbook import Processor
def inject_cwd ( record ):
record . extra [ ’ cwd ’] = os . getcwd ()
with my_handler . a p p l i c at i o n b o u n d ():
with Processor ( inject_cwd ). a p p l i c a t i o n b o u n d ():
# e v e r y t h i n g logged here will have the current
# working d i r e c t o r y in the log record .
...
The end

24. Why use logging
Basics
When to use what?
Logbook
Error reporting
Formatting in Logging
import logging
logging . basicConfig ( format = ’ %( levelname ) s :%( message ) s ’ ,
level = logging . DEBUG )
The end

25. Why use logging
Basics
When to use what?
Logbook
Error reporting
Formatting in Logbook
>>> from logbook import StderrHandler
>>> handler = StderrHandler (
...
format_string = ’{ record . channel }: { record . message } ’)
>>> handler . formatter
< logbook . handlers . St ri n gF or ma t te r object at 0 x100641b90 >
The end

26. Why use logging
Basics
When to use what?
Logbook
Custom formatters
>>> def my_formatter ( record , handler ):
... return record . message
...
>>> handler . formatter = my_formatter
Error reporting
The end

27. Why use logging
Basics
When to use what?
Logbook
Custom formatters
>>> def my_formatter ( record , handler ):
... return record . message
...
>>> handler . formatter = my_formatter
Tip:
Or use Jinja!
Error reporting
The end

28. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
Extra fields
handler = StderrHandler ( format_string =
’{ record . channel }: { record . message ) [{ record . extra [ cwd ]}] ’)

29. Why use logging
Basics
When to use what?
Logbook
Redis
import logbook
from logbook . queues import RedisHandler
handler = RedisHandler ()
l = logbook . Logger ()
with handler :
l . info ( ’ Your log message ’)
Error reporting
The end

30. Why use logging
Basics
When to use what?
Logbook
Error reporting
RabbitMQ
Handler
handler = R a bb it MQ H an dl er ( ’ amqp :// guest : gu e st @l oc a lh os t // ’ ,
queue = ’ my_log ’)
Subscriber
with target_h andler :
subscriber = R a b b i t M Q S u b s c r i b e r (
’ amqp :// guest : gu e st @l oc a lh os t // ’ , queue = ’ my_log ’)
subscriber . d i s p a t ch _ f o r e v e r ()
The end

31. Why use logging
Basics
When to use what?
Logbook
ZeroMQ
Don’t use it. (It’s broken)
Error reporting
The end

32. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
Minimize logging
>>> from logbook import FingersCrossedHandler , StderrHandler
>>> handler = F i n g e r s C r o s s e d H a n d l e r ( StderrHandler ())
>>> with handler . a p p l i c a t i o n b o u n d ():
...
logbook . info ( ’ Hello ’)
...
...
>>> handler = F i n g e r s C r o s s e d H a n d l e r ( StderrHandler ())
>>> with handler . a p p l i c a t i o n b o u n d ():
...
logbook . info ( ’ Hello ’)
...
logbook . error ( ’ Bye ’)
...
...
[2013 -12 -16 11:12] INFO : Generic : Hello
[2013 -12 -16 11:12] ERROR : Generic : Bye

33. Why use logging
Basics
When to use what?
Logbook
Error reporting
TestHandler
>>> from logbook import TestHandler , Logger
>>> logger = Logger ( ’ Testing ’)
>>> handler = TestHandler ()
>>> handler . push_thread ()
>>> logger . warn ( ’ Hello World ’)
>>> handler . records
[ < logbook . base . LogRecord object at 0 x100640cd0 >]
>>> handler . f o r m a t t e d _ r e c o r d s
[ u ’[ WARNING ] Testing : Hello World ’]
The end

34. Why use logging
Basics
When to use what?
Logbook
Github
github.com/mitsuhiko/logbook
Error reporting
The end

35. Why use logging
Basics
When to use what?
Logbook
Error reporting
Error reporting
The end

36. Why use logging
Basics
When to use what?
Logbook
Error reporting
Emails?
Error reporting
The end

37. Why use logging
Basics
When to use what?
Logbook
Error reporting
Error reporting
The end

38. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
Airbrake
Airbrake is the leading exception reporting service, currently
providing error tracking for 50,000 applications with support for 18
programming languages.

39. Why use logging
Basics
When to use what?
Logbook
Sign up
airbrake.io
Error reporting
The end

40. Why use logging
Basics
When to use what?
Logbook
Error reporting
Flask
from flask import Flask , request
from airbrake import A i r b r a k e E r r o r H a n d l e r
import gevent
@app . errorhandler (500)
def interna l_error ( error ):
if app . config [ ’ E X C E P T I O N _ L O G G I N G ’ ]:
handler = A i r b r a k e E r r o r H a n d l e r (
api_key = app . config [ ’ A I R B R E A K _ A P I _ K E Y ’] ,
env_name = ENV , request_url = request . url ,
request_path = request . path ,
requ est_meth od = request . method ,
request_args = request . args ,
r eq ue st _ he ad er s = request . headers )
gevent . spawn ( handler . emit , error )
The end

41. Why use logging
Basics
When to use what?
Logbook
Error reporting
Django
AIRBRAKE = {
’ API_KEY ’: ’ y o u r _ a p i _ k e y _ h e r e ’ ,
’ USE_SSL ’: True ,
’ TIMEOUT ’: 5 ,
’ ENVIRONMENT ’: ’ u n i q u e _ n a m e _ f o r _ e n v i r o n m e n t ’ ,
}
The end

42. Why use logging
Basics
When to use what?
Logbook
Errbit
github.com/errbit/errbit
Error reporting
The end

43. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
Sentry
getsentry.com
Sentry is a realtime event logging and aggregation platform. At its
core it specializes in monitoring errors and extracting all the
information needed to do a proper post-mortem without any of the
hassle of the standard user feedback loop.

44. Why use logging
Basics
When to use what?
Detail
Logbook
Error reporting
The end

45. Why use logging
Basics
When to use what?
Logbook
More detail
Error reporting
The end

46. Why use logging
Basics
When to use what?
Logbook
Install the server
sentry.readthedocs.org
Error reporting
The end

47. Why use logging
Basics
When to use what?
Logbook
Error reporting
Sentry + Logbook
raven.readthedocs.org/en/latest/config/logbook.html
Install raven
pip install raven
Your app code will have this at the entry point
from raven . handlers . logbook import SentryHandler
client = Client (...)
sent ry_handl er = SentryHandler ( client )
with sentry_h andler . a p pl i c a t i o n b o u n d ():
# e v e r y t h i n g logged here will go to sentry .
...
The end

48. Why use logging
Basics
When to use what?
Logbook
Error reporting
Pay attention to formatting!
Sentry will intelligently group messages if you use proper string formatting.
logger . error ( ’ There was some {} error ’ , ’ crazy ’)
logger . error ( ’ There was some {} error ’ , ’ fun ’)
logger . error ( ’ There was some {} error ’ , 1)
The end

49. Why use logging
Basics
When to use what?
Logbook
Tagging
logger . error ( ’ Could not do that ’ ,
tags ={ ’ user email ’: ’ foo@bar ’ })
Error reporting
The end

50. Why use logging
Basics
When to use what?
Logbook
Error reporting
The end
TicketingHandler
Create unique tickets for log records and keep track of the number of times these log
records were created
from logbook import ERROR
from logbook . ticketing import T i c k e t i n g H a n d l e r
handler = T i c k e t i n g Ha n d l e r ( ’ postgres :// localhost / database ’ ,
level = ERROR )
with handler :
# e v e r y t h i n g in this block and thread will be handled by
# the t i c k e t i n g d a t a b a s e handler
...

51. Why use logging
Basics
When to use what?
Logbook
Error reporting
Phew! It’s finally over!
Reach me at fayaz at dexetra dot com.
The end