A short presentation for PyWeb-IL 8th meeting.

1. Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

3. Addressing Scalability

4. Experience with Django

5. Use-case 1: automated data integration service

6. Use-case 2: social media analysis service

7. Recommendations

8. Links

10. Web back-ends become thinner - just services

11. Applications become service providers, usually over HTTP

12. All reasons for using Django for almost any back-end offering services

14. Usually involve long processing

15. May involve continuous communications, & not just request-response

16. Reliability & high-availability are usually more important with non-human users

17. Lots of communication with other back-ends

19. Workflow Engines manage long processing

20. Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

21. Clouds & auto-scaling enable high-availability

22. Can use SOAP/REST for protocols against other back-ends

24. Even when processing took days to complete, Django was 100% robust

26. Large data

27. Protocols against other back-ends

29. Integrating data & services across several back-end systems

32. Used a great library called Suds

36. Used REST interface to push & pull messages with message queues, such as ActiveMQ

37. Used Celery for AMQP-based message queues

39. According to metadata, different format handlers were invoked

44. Exports logic to Java-based scripts

46. Used Nicolas Toll's workflow engine, which allows users to define & manage complex workflows

48. Added a graphical dashboard

51. Used task-names for flexible handlers invocation

52. Used periodic tasks for driving the workflow engine

54. Horizontal & vertical scaling

55. Reliable & easy to manage

56. Message queues allow distributing load horizontally

57. Used script-based auto-scaling – starting new instances based on load

63. Recommend people to follow based on interests

65. Objects are organized in hierarchical layers

66. Objects have autonomous micro behavior aggregating to the macro behavior of the system

67. Layers are organized in spatial grids, which enable easy sharding & parallel processing

69. Dbpedia

70. ConceptNet

73. Celery & RabbitMQ are used

74. JSON is used as message payload

76. most of the data processing

77. Very intensive

79. Not always-on

80. Requires 3 rd party libraries, not limited to Python

83. Can be started just upon need

87. Frameworks such as Fuzed can be integrated with Python/Django

88. We're working on it as a coding session & hope to deliver a prototype soon

90. Suds

91. Workflow

92. django-xmpp

93. Fuzed

94. Google Chart API

95. Talend

96. Thanks! @dibau_naum_h