Apresentação do pessoal da WebCo Internet (BlogBlogs e Brasigo) no Rails Summit Latin America 2008. Manoel Lemos, Ronaldo Ferraz e Nando Vieira falam de desafios reais vividos por empreendimentos Web 2.0. Desde o início do BlogBlogs até a formação da WebCo.

1. Uma história de
guerra: Enfrentando
problemas reais em
aplicações Web 2.0
Manoel Lemos
Ronaldo Ferraz
Nando Vieira
Rails Summit Latin America 2008
São Paulo, Brasil, 16.Outubro

3. 1983

4. 1983
1984

5. 1983 1993
1984

6. 1983 1993
1984 1994

7. 1983 1993
1984 1994 1997

8. 1983 1993 1999
1984 1994 1997

9. 1983 1993 1999
1984 1994 1997

10. 1983 1993 1999
1984 1994 1997 2000

11. 1983 1993 1999 2006
1984 1994 1997 2000

13. Abr.06

14. Nov.06
Abr.06

15. Fev.07
Nov.06
Abr.06

16. Fev.08
Fev.07
Nov.06
Abr.06

17. Ago.08
Fev.08
Fev.07
Nov.06
Abr.06

18. Ago.08
Fev.08
Fev.07
Nov.06
Abr.06

20. BlogBlogs Status
Business Numbers Traffic Numbers
110.000 usuários 5.2M PageViews
200.000 blogs 2.6M Uniques
2.000.000 posts Top 4000 Alexa Global
250 usuários / dia Top 120 Alexa Brasil
150 blogs / dia Top 102 IBOPE Brasil
5.000 posts / dia 50M Widget Impressions

21. Crescer é um processo
complicado e doloroso!
É impossível de se prever com precisão
Existem gargalos escondidos
Às vezes você precisa voltar atrás
Às vezes você precisa mudar tudo
Não existem “silver-bullets”
A complexidade vai crescer com você

22. Crescimento real nunca
acontece em um só eixo!
Tráfego : mais usuários, transações, imagens...
Tamanho : banco, memória, cache...
Complexidade : algoritmos, arquitetura, infraestrutura...
Organização : equipe, comunicação, processos...
Negócio : modelos, parceiros, fornecedores...
Na maioria dos casos os problemas crescem mais que
linearmente!!!

26. Hoje
1.5 Milhões
por dia

35. httpd
mongrel
mysql

36. httpd
mongrel
mongrel
mongrel
mysql

37. httpd
mongrel
mongrel
mongrel
memcached
mysql

38. httpd
mongrel
mongrel
mongrel
memcached
mysql

39. httpd
mongrel
mongrel
mongrel
memcached
mysql

40. load balancing
httpd httpd
mongrel mongrel
mongrel mongrel
mongrel mongrel
memcached memcached
mysql

41. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

42. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

43. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

44. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

45. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

46. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

47. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

48. load balancing
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

49. load balancing 1 (web)
httpd httpd httpd
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
mongrel mongrel mongrel
memcached memcached memcached
mysql

50. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql

51. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql (master)
mysql (slave)

52. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

53. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

54. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
memcached memcached memcached memcached memcached memcached
mysql mysql (master)
mysql (slave)

55. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached
memcached memcached

56. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached
memcached memcached

57. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
memcached memcached Robot Robot
memcached memcached Robot Robot

58. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
memcached memcached Robot Robot

59. load balancing 1 (web) load balancing 2 (widgets)
httpd httpd httpd httpd httpd httpd
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mongrel mongrel mongrel mongrel mongrel mongrel
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
httpd httpd
mongrel mongrel
mongrel mongrel
memcached memcached Robot Robot
mongrel mongrel
mongrel mongrel
mongrel mongrel
load balancing 3 (API)

60. load balancing 1 (web) load balancing 2 (widgets)
mongrels mongrels mongrels mongrels mongrels mongrels
mysql mysql (master)
mysql (slave)
Indexer Indexer
memcached memcached Robot Robot
(Search) (Search)
memcached memcached Robot Robot mongrels mongrels
load balancing 3 (API)

61. load balancing
mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels
Robot
Robot
Indexer
memcached memcached
(Search)
mysql mysql
(master) (master)
Indexer
memcached memcached
(Search)

62. load balancing
mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels
Robot
Robot
Indexer
memcached memcached
(Search)
mysql mysql
(master) (master)
Indexer
memcached memcached
(Search)

63. load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

64. load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

65. load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Eureca...Idéias...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Single Sign On
Shared Resources
Robot Robot
Integrated Search
Robot
E=MC2 Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

66. load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

67. load balancing load balancing
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
E ai, #comofaz ?
Robot Robot
Robot Robot
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)
mysql mysql mysql mysql
(master) (master) (master) (master)
Indexer Indexer
memcached memcached memcached memcached
(Search) (Search)

68. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels

69. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

70. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

71. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA
Credentials
(PHP)
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

72. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA
Credentials Profile
(PHP) (PHP)
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

73. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA
Credentials Profile Avatars
(PHP) (PHP) (PHP)
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

74. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph
(PHP) (PHP) (PHP) (C++) (Python)
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

75. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

76. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

77. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

78. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

79. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP)
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

80. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
memcached memcached memcached Robot Robot
(Search) (Search)

82. Active DNA
Problema: Várias features necessárias (Profile, SSO,
Avatar, etc) e pouco tempo
Serendipity: Companhia irmã com vários componentes
REST
Resultado: Active DNA, uma biblioteca REST flexível
mais cache e serviços

83. Active DNA
Por que não Active Resource
Interface fixa
Somente Rails sem overrides
Pouca integração com external services

84. Busca
Ferret: FAIL
Sphinx: Muito limitado
Lucene / Solr: Java
Xapian: Interessante
Mas, e se precisarmos mudar?

85. Busca
Solução: Sapien
Cliente / encapsulamento de servidor de busca
Driver-based, naturalmente distribuído
Drivers podem ser mudados e o índice refeito em um
passo
Brasigo Query Language

86. API
Controle de Acesso
HTTP Auth (email & senha)
OAuth para suportar usuários de OpenID e outros
Restrições de uso
Frameworks mais leves
Sinatra
Thin

87. Dia-a-dia (SCRUM)
Scrum
Várias equipes
Coordenação e priorização
Rotacionamento de equipes
Socialização de conhecimento
Cultura compartilhada

88. Dia-a-dia (Git)
Git
Vários repositórios internos, todos Git
Branches por estória
Repositório sempre releasable

89. Dia-a-dia (Sharing)
Cultura de bibliotecas
Plugins
Gems / Gem server interno
Funcionalidade compartilhada via API

90. Dia-a-dia (Testes)
Test all the f***ing time
Sem cobertura, não há conhecimento
Socialização de conhecimento
Integração contínua

91. Continuous Integration
Dedicated integration environment with automated
integration and tests execution with the most up to
dated code base.
Automated Reporting
Test Cases Results
Tests Coverage

92. Tests, tests & more tests

93. Tests, tests & more tests

94. Tests, tests & more tests

95. Continuous Integration

96. Continuous Integration

97. Continuous Integration

98. Continuous Integration

99. Continuous Integration

100. Continuous Integration

101. Continuous Integration

102. Continuous Integration

103. Continuous Integration

104. Continuous Integration

105. Continuous Integration

106. Continuous Integration

107. Continuous Integration

108. Continuous Integration

109. Continuous Integration

110. Continuous Integration

111. Venha trabalhar com a
gente!!!
vagas@blogblogs.com.br

112. Obrigado!
mlemos@webcointernet.com
rferraz@webcointernet.com
fvieira@webcointernet.com
http://brasigo.com.br/
http://blog.brasigo.com.br/
http://blogblogs.com.br/
http://blog.blogblogs.com.br/