Diese Präsentation wurde erfolgreich gemeldet.
Die SlideShare-Präsentation wird heruntergeladen. ×

The Onward Journey: Porting Twisted to Python 3

Wird geladen in …3

Hier ansehen

1 von 37 Anzeige

Weitere Verwandte Inhalte

Diashows für Sie (19)

Andere mochten auch (19)


Ähnlich wie The Onward Journey: Porting Twisted to Python 3 (20)

Aktuellste (20)


The Onward Journey: Porting Twisted to Python 3

  1. 1. The Onward Journey: Porting Twisted to Python 3 Craig Rodrigues <rodrigc@crodrigues.org>
  2. 2. Twisted
  3. 3. What is Twisted? Python library, written by Glyph Lefkowitz and many others provides basic building blocks for writing networking clients and servers: protocol implementations: SSH, IRC, SMTP, IMAP event “reactors”: select, kqueue, epoll, IOCP, asyncio…...
  4. 4. More resources Web site with docs, examples, tutorials: http://www.twistedmatrix.com Main code repository: https://github.com/twisted/twisted O’Reilly book:
  5. 5. Twisted sub-projects Klein (similar to Flask, Bottle) https://github.com/twisted/klein Treq (similar to Requests) https://github.com/twisted/treq
  6. 6. Which projects Twisted? buildbot scrapy Lots of other projects: https://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted
  7. 7. Who uses Twisted? Hipchat Apple Calendar Server Lots of companies: https://twistedmatrix.com/trac/wiki/SuccessStories
  8. 8. Twisted and asynchronous programming Twisted has promoted the technique of using callbacks and asynchronous programming for writing network servers for a long time Twisted’s Deferred class is central to this, and has been used extensively Guido van Rossum consulted with Glyph in the design of Python’s new asyncio framework ( Python 3.4 and higher )
  9. 9. Interesting things First commit to Twisted was in 2001 Twisted includes a tool trial which runs unittest-style tests. (similar to pytest or nose) Unittests and code coverage are very important to Twisted development process
  10. 10. Why bother?
  11. 11. Motivation I like Twisted and the community behind it I had some time in between jobs and wanted to improve my Python skills and learn more about Python 3 I wanted to help out projects which depended on Twisted, but couldn’t move to Python 3, such as buildbot Core Python devs are dropping Python 2 support in 2020: https://pythonclock.org/
  12. 12. Major motivation: Twisted moved to GitHub in 2016!! Following process to submit patches via Subversion was cumbersome Moving to GitHub and pull requests made things easier for submitters and reviewers Integration with Continuous Integration (CI) was improved: codecov, travis, appveyor, buildbot Submitting patches is easier!!
  13. 13. Moving Twisted to GitHub
  14. 14. Moving to Python 3
  15. 15. Twisted started moving to Python 3 Original Python 3 porting plan developed in 2012 Worked on by various developers: Jean-Paul Calderone, Itamar Turner- Trauring, Amber Brown, Glyph Lefkowitz, Ralph Meijer, and others Canonical funded some Python 3 porting work Some parts ported, many parts still unported
  16. 16. Moving Twisted to Python 3 is a tough job! Old codebase (since 2001) Advanced framework which uses many, many features of Python Twisted development process requires unit tests and coverage Submitting hundreds of patches in Subversion workflow was slow moving
  17. 17. Porting to Python 3
  18. 18. What has changed in Python 3? Lots of little changes to make the language cleaner Deprecated code has been removed Some changes are backwards incompatible. Previously working code is now broken on Python 3 http://python3porting.com has an extensive list of changes in Python 3
  19. 19. print is now a function print “hello world” now must be: print(“hello world”)
  20. 20. dict.has_key() is gone some_dict = { “one” : 1, “two”: 2} some_dict.has_key(“one”) Now should be: “one” in some_dict
  21. 21. obj.__cmp__() and cmp() is gone Developers are supposed to implement __lt__(), __gt__(), __ge__(), __le__(), __ne__(), __eq__() functions on an object instead of __cmp__() Developers need to use <, >, >=, <=, !=, == operators instead of cmp() which is gone
  22. 22. Less things allocate lists These functions no longer allocate lists in Python 3: range(), dict.items(), dict.values(), map(), filter() Users should iterate over these functions, which only allocate items as they are needed: for n in range(99): ...
  23. 23. C API for Python C extensions changed C API changed in a backwards incompatible way Very challenging when porting the Twisted IOCP reactor (Windows only) which has parts written in C
  24. 24. Python str type has changed Python 2: u”Some unicode string 銩” is of type unicode “Some string” is of type str and also of type bytes b”Some string” is of type str and also of type bytes type(unicode) != type(str), type(str) == type(bytes) Python 3: u”Some string 銩” is of type str
  25. 25. Python str type has changed Twisted protocols must send out bytes over the wire on sockets Lots of code written assuming type(str) == type(bytes), did not account for unicode This type of porting needs extensive analysis and testing, cannot be automated
  26. 26. Python str type has changed Ned Batchelder unicode presentation very good: https://nedbatchelder.com/text/unipain/unipain.html “Unicode sandwich” technique (write bytes to sockets and files, keep data as unicode internally in application) cannot be used 100% when dealing with network protocols
  27. 27. Technique for porting to Python 3
  28. 28. Porting technique: use virtualenvs Checkout the code from git Create a python2 virtualenv in one window: virtualenv myenv_2 source myenv_2/bin/activate python setup.py develop Create a python3 virtualenv in another window: python3 -m venv myenv_3
  29. 29. Porting technique: run unit tests After modifying code, run unittests using tox and trial See what breaks, make sure it works on Python 2.7 and Python 3 Write new unit tests if necessary Always try to improve code coverage: https://codecov.io/gh/twisted/twisted/
  30. 30. Python 3 status for Twisted
  31. 31. June 3, 2016 Python 2.7: 8425 tests PASS Python 3.5: 4834 tests PASS ( approx. 57% of Python 2.7 tests)
  32. 32. March 21, 2017 Python 2.7: 9692 tests PASS Python 3.5: 9025 tests PASS ( approx. 93% of Python 2.7 tests) My contributions: 325 pull requests!
  33. 33. What’s left? A few modules need to be ported such as: twisted.mail twisted.news twisted.web
  34. 34. Lessons learned
  35. 35. What I learned Extensive unittests and code coverage are very important for this kind of effort Porting an old and large codebase to Python 3 can be a lot of work Benefits of porting: code cleanliness and keeping up with Python direction...the benefits vs. the effort required sometimes doesn’t feel worth it
  36. 36. Hope for the future and performance CPython 3.6 and 3.7 performance seems to be improving and is comparable to Python 2.7: http://speed.python.org Pypy 3.5 just came out….hopefully better performance with that Now that Python has asyncio built in, the “Twisted way” of doing things has some validation, and is pervading more libraries and projects in Python
  37. 37. Thanks All who started before me on the Python 3 effort: Jean-Paul, Itamar, Amber, Glyph, Ralph, many others All who helped code review my patches: Adi Roiban, Alex Gaynor, Glyph, many others Special thanks to Abhishek Choudhary for help on code reviews