Talk by @aitorciki at BcnDevCon'11.

1. Ducksboard
a real-time data oriented
webservice architecture
aitor@ducksboard.com
@aitorciki on twitter

2. What is Ducksboard?

3. What is Ducksboard?
●
it's a web-based dashboard...

4. What is Ducksboard?
●
it's a web-based dashboard...
●
to display SaaS / your own metrics...

5. What is Ducksboard?
●
it's a web-based dashboard...
●
to display SaaS / your own metrics...
●
with heavy focus on UX...

6. What is Ducksboard?
●
it's a web-based dashboard...
●
to display SaaS / your own metrics...
●
with heavy focus on UX...
●
huge bet on REAL real-time updates...

7. What is Ducksboard?
●
it's a web-based dashboard...
●
to display SaaS / your own metrics...
●
with heavy focus on UX...
●
huge bet on REAL real-time updates...
●
and much more in the near future!

8. Architectural Goals

9. Architectural Goals
●
data grabber / visualizer independence

10. Architectural Goals
●
data grabber / visualizer independence
●
real-time updates (push vs pull)

11. Architectural Goals
●
data grabber / visualizer independence
●
real-time updates (push vs pull)
●
scalability (web scale, bitches!)

12. Architectural Goals
●
data grabber / visualizer independence
●
real-time updates (push vs pull)
●
scalability (web scale, bitches!)
●
eat your own duck food

13. Architecture Concepts

14. Architecture Concepts
●
Back-end (Twisted)

15. Architecture Concepts
●
Back-end (Twisted)
●
Parser: obtains data from external services

16. Architecture Concepts
●
Back-end (Twisted)
●
Parser: obtains data from external services
●
Gateway: stores data

17. Architecture Concepts
●
Back-end (Twisted)
●
Parser: obtains data from external services
●
Gateway: stores data
●
API: provides read access to stored data

18. Architecture Concepts
●
Front-end

19. Architecture Concepts
●
Front-end
●
web application

20. Architecture Concepts
●
Front-end
●
web application
●
mobile web application (not yet available)

21. Architecture Concepts
●
Front-end
●
web application
●
mobile web application (not available)
●
mobile / desktop apps?

22. Common Components

23. Common Components
●
PostgreSQL (the “DB”)

24. Common Components
●
PostgreSQL (the “DB”)
●
stores application configuration

25. Common Components
●
PostgreSQL (the “DB”)
●
stores application configuration
●
stores data for historic queries

26. Common Components
●
PostgreSQL (the “DB”)
●
stores application configuration
●
stores data for historic queries
●
starts / stops components (PL/Python!)

27. Common Components
●
RabbitMQ (the “queue”)

28. Common Components
●
RabbitMQ (the “queue”)
●
handles RT updates

29. Common Components
●
RabbitMQ (the “queue”)
●
handles RT updates
● simplifies publisher / consumer scenarios

30. Architecture Overview

31. Backend Components: Parser

32. Backend Components: Parser
●
a parser per service and user

33. Backend Components: Parser
●
a parser per service and user
●
many API models supported (RT preferred)

34. Backend Components: Parser
●
a parser per service and user
●
many API models supported (RT preferred)
●
configuration stored in the DB

35. Backend Components: Parser
●
a parser per service and user
●
many API models supported (RT preferred)
●
configuration stored in the DB
●
obtained data sent to the gateway for storage

36. Backend Components: Gateway

37. Backend Components: Gateway
●
stores data in the DB for persistence

38. Backend Components: Gateway
●
stores data in the DB for persistence
●
pushes data to the queue for RT notifications

39. Backend Components: Gateway
●
stores data in the DB for persistence
●
pushes data to the queue for RT notifications
●
receives updates from all the parsers

40. Backend Components: Gateway
●
stores data in the DB for persistence
●
pushes data to the queue for RT notifications
●
receives updates from all the parsers
●
exposes webservice for custom metrics

41. Backend Components: API

42. Backend Components: API
●
gets historic data from the DB and ...

43. Backend Components: API
●
gets historic data from the DB and ...
●
… RT updates from the queue

44. Backend Components: API
●
gets historic data from the DB and ...
●
… RT updates from the queue
●
exposes HTTP interface for data retrieval

45. Backend Components: API
●
gets historic data from the DB and ...
●
… RT updates from the queue
●
exposes HTTP interface for data retrieval
●
exposes Websocket server for RT updates

46. Frontend Components: WebApp

47. Frontend Components: WebApp
●
Django application

48. Frontend Components: WebApp
●
Django application
●
Nginx + uWSGI

49. Frontend Components: WebApp
●
Django application
●
Nginx + uWSGI
●
uses the DB as a normal webapp (SQLAlchemy)

50. Frontend Components: WebApp
●
Django application
●
Nginx + uWSGI
●
uses the DB as a normal webapp (SQLAlchemy)
●
Celery handles async operations

51. Frontend Components: WebApp
●
Backbone.js empowers the UI

52. Frontend Components: WebApp
●
Backbone.js empowers the UI
●
Raphaël draws the widgets

53. Frontend Components: WebApp
●
Backbone.js empowers the UI
●
Raphaël draws the widgets
●
a little bit of jQuery everywhere :)

54. Frontend Components: WebApp
●
Backbone.js empowers the UI
●
Raphaël draws the widgets
●
a little bit of jQuery everywhere :)
●
Websocket or Flash fallback for RT updates

55. Architecture Overview

56. Scale (or die trying)

57. Scale (or die trying)
●
every component is standalone (no code deps)

58. Scale (or die trying)
●
every component is standalone (no code deps)
●
components talk using HTTP (Nginx + DNS win!)

59. Scale (or die trying)
●
every component is standalone (no code deps)
●
components talk using HTTP (Nginx + DNS win!)
●
RabbitMQ allows m-gateways / n-apis scenarios

60. Security

61. Security
●
HTTPS everywhere

62. Security
●
HTTPS everywhere
●
pgcrypto to provide credentials encryption

63. Security
●
HTTPS everywhere
●
pgcrypto to provide credentials encryption
●
bcrypted webapp passwords

64. Security
●
HTTPS everywhere
●
pgcrypto to provide credentials encryption
●
bcrypted webapp passwords
●
relational integrity ensures data sanity

65. Well, that's it! Questions?

66. Thanks for listening!
and buying ;)