1. Week 5

2. Advanced Architectures
COAST architecture: http://isr.uci.edu/projects/coast/about.html

3. Data Silos
• Typical in many 'small' data architectures.
• Data are persisted in different databases with imposed borders.
• They are created due to organizational boundaries and divergent
goals between individuals or teams.

4. Data Silos (cont)

5. A silo breaker, the data lake

6. Data Lake Architecture
Credit: EMC

7. Data Lake Architecture
Credit: Zaloni

8. Data Lake vs Data Warehouse
Credit: Martin Fowlerr

9. Data Lake Architecture, Batch Processing
Credit: EMC

10. Data Lake Architecture, Stream Processing
https://www.confluent.io/blog/stream-data-platform-1/

11. Data Lake Architecture, Stream Processing (cont.)
https://www.confluent.io/blog/stream-data-platform-1/

12. Data Lake Architecture, Stream Processing (cont.)
https://www.confluent.io/blog/stream-data-platform-1/

13. What happens when we want to combine batch
and stream processing?

14. λ!
• Invented by Nathaniel Marz, it is intended to combine the pros of both
worlds.
• Processing is separated but persistence is common.
• It introduces more complexity.
• Many more moving parts.
• Start getting out of favour in the last couple of years.

15. Lambda Architecture (cont.)

16. Lambda Architecture (cont.)

17. Lambda Architecture for Web Apps
Credit: Amazon

18. How far we can go in reducing elements?

19. Serverless Architecture
• Do we still need servers?
• - Seems not.
• Do we need storage clusters?
• - Seems not.
• Do we need developers?
• - Seems yes.

20. Serverless Architecture (cont.)
Credit: Amazon

21. Serverless Architecture (cont.)
• Serverless systems don't need to be limited to web services or
contellations of microservices.
• They can also be used for Continuous Deployment (CI), Continuous
Delivery (CD) or monitoring.

22. Since we mentioned Serverless Microservices
• Functionality of Lambda functions can be grouped together and form
serverless microservices (logically).
• Same principles regarding communication and separation of concerns
apply here.
• Operations become much, much simpler and cheaper.

23. Serverless Microservices (cont.)
Credit: Amazon

24. Serverless Continuous Delivery
• Jenkins is not needed any more.
• There is no limiation in compiling/building capacity (except your credit
cards).
• Highly available (no more dead Jenkins workers).
• No risk of over or under-provisioning.
• No OS to maintain or worry about.
https://stelligent.com/2016/03/17/serverless-delivery-architecture-part-1/

25. Serverless Continuous Delivery (cont.)
https://stelligent.com/2016/03/17/serverless-delivery-architecture-part-1/

26. Bottom line...
https://stelligent.com/2016/03/17/serverless-delivery-architecture-part-1/

27. COAST Architecture
• COmputAtional State Transfer (COAST).
• The principal goal is the construction of Internet-scale decentralized
applications.
• Computation exchange is the bilateral exchange of computations
among peers (code to data).
• A PoC has been implemented in Racket.
http://isr.uci.edu/projects/coast/about.html

28. COAST Architecture (cont.)
http://isr.uci.edu/projects/coast/about.html

29. Zeta Architecture
• An enterprise-scale architecture aiming to meet always rising
expectations.
• Inspired by Google's top-level architecture.
• There are seven pluggable components of the architecture, and all of
them must work together.
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

30. Zeta Architecture in Google (cont.)
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

31. Poor man's Zeta (cont.)
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

32. Zeta Architecture (cont.)
• Data protection schemes, backing up data, recovering from failures,
load balancing, and even running multiple versions of software are
simplified with Zeta.
• These properties are offered as services and are available to every
guest application.
• Resource allocation is optimized and results in lower HW
requirements.
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

33. Web Service before Zeta
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

34. Web Service after Zeta
https://www.oreilly.com/ideas/zeta-architecture-hexagon-is-the-new-circle

35. Unikernels
• Unikernels are specialised, single-address-space machine images
constructed by using library operating systems.
• They are built by compiling high-level languages directly into
specialised machine images that run directly on a hypervisor.
• Unikernels benefits are improved security, smaller footprints, more
optimisation and faster boot times.
http://unikernel.org

36. Unikernels (cont.)
http://unikernel.org

37. ClickOS, exercise in lightness
• "A minimalistic, tailor-made, virtualized operating system to run Click-
based middleboxes."
• Small (6MB)
• Boot quickly (in about 30 milliseconds)
• Quick response time (45 microseconds)
• Targeting C++
http://cnp.neclab.eu/clickos/

38. MirageOS, Docker's successor?
• The most widely used Unikernel system.
• In production as a microservice platform.
• Unikernel Systems has been acquired by Docker, making it an inherent
part of the Docker ecosystem.
• Used in serverless, non-AWS deployment scenarios.
https://mirage.io

39. Looking in the (not so deep) future
https://mirage.io

40. Self-organizing Architectures
• Self-adaptive software architecture.
• Autonomously exploring all possible architectures that can be used
to realise a given software system.
• Monitoring that system in execution in terms of its performance and
its operating environment.
• Identifying the optimal architecture for each set of operating
environment conditions that are encountered
http://ieeexplore.ieee.org/document/7573144

41. Train your server
https://www.wired.com/2016/05/the-end-of-code/

42. Deepcoder Learning to Write Programs
https://arxiv.org/abs/1611.01989
• A Cambridge Project.
• Deep Neural network which is not performing function aproximation
but assembling simple pieces of code which approximate the
function.
• It can solve the simplest problems on programming competition
websites.

43. Deep Learning Architecture – High Level
http://www.slideshare.net/odsc/arno-candel-scalabledatascienceanddeeplearningwithh2oodscboston2015-48909623

44. Deep Learning Architecture – Neural Net
http://www.slideshare.net/odsc/arno-candel-scalabledatascienceanddeeplearningwithh2oodscboston2015-48909623

45. Deep Learning Architecture – Infrastructure

46. Thank you!