5. BlogBlogs Status
Business Numbers Traffic Numbers
115.000 users 7.5M PageViews / month
200.000 blogs 3.5M Uniques / month
2.500.000 posts Top 5000 Alexa Global
250 users / day Top 120 Alexa Brasil
150 blogs / day 50M Widget
Impressions / month
7.000 posts / day
6. Scaling is a process
complicated and painful!
Its impossible to predict with accuracy
There are hidden bottlenecks
Sometimes you need to step back
Sometimes you need to re-factor everything
There are no “silver-bullets”
Complexity will grow with you
7. Growth never happens in a
single dimension
Traffic : more users, more transactions, more downloads...
Size : database, memory, bandwidth...
Complexity : algorithms, architecture, infrastructure...
Organization : team, communication, processes...
Business : models, partners, suppliers...
Mostly the issues grow faster than linearly!!!
58. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA
Credentials Profile Avatars
(PHP) (PHP) (PHP)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
59. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph
(PHP) (PHP) (PHP) (C++) (Python)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
60. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
61. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
62. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
63. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
ActiveDNA (REST)
load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
64. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
Credentials
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
(PHP)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
65. load balancing load balancing
mongrels mongrels mongrels sinatra mongrels mongrels mongrels mongrels
humm...
mongrels mongrels mongrels mongrels mongrels mongrels mongrels mongrels
PHP... ActiveDNA (REST)
WTF??!! load balancing
DNA DNA DNA DNA DNA DNA DNA DNA DNA DNA
Credentials Profile Avatars Scoring SocialGraph Messaging Messaging Messaging Messaging Messaging
(PHP) (PHP) (PHP) (C++) (Python) (Erlang) (Erlang) (Erlang) (Erlang) (Erlang)
ActiveDNA (REST)
load balancing
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
mysql mysql mysql
(master) (slave) (slave)
Indexer Indexer
Robot Robot
memcached memcached memcached
(Search) (Search)
66. Active DNA
The problem : time-to-market & development time
The solution : ready-for-use SWAT DNA components
The layer : Active DNA, a lightweight, RESTful, SWAT
DNA client API translating REST calls into fully realized
Ruby objects
Pros
Complete abstraction of the component’s complexity.
It’s just Ruby, very small learning curve for devs.
67. Active DNA
Application
REST API
SWAT DNA Component
68. Active DNA
Application
REST API
SWAT DNA Component
69. Active DNA
Application
REST Client
REST API
SWAT DNA Component
70. Active DNA
Application
Query Runner
REST Client
REST API
SWAT DNA Component
71. Active DNA
Application
Query Runner Query Translator
REST Client
REST API
SWAT DNA Component
72. Active DNA
Application
Component Layer
Query Runner Query Translator
REST Client
REST API
SWAT DNA Component
73. class ActiveDNA::ProfileEntry < ActiveDNA::Base
uses :profile
key :key, :suffix => :entry
depends_on :profile_id
attributes :type, :group, :level, :value => :json, :is_unique => :boolean
validates_presence_of :value, :message => quot;The value must be informedquot;
validates_presence_of :profile_id, :message => quot;Profile must be informedquot;
validates_presence_of :group, :message => quot;The group must be informedquot;
validates_inclusion_of :type,
:in => %w(boolean email float general guid integer url),
:message => quot;The type must be informedquot;
end
74. class RegistrationController < ApplicationController
def create_credential(profile_id)
returning credential = ActiveDNA::Credential.new do
credential.credential_identifier = self.identifier
credential.credential_type = self.identifier_type ||
IDENTIFIER_TYPE_EMAIL
credential.credential_status = quot;enabledquot;
credential.user_id = Profile.find(:first, :conditions => { :profile_id
=> profile_id }).user_id
credential.password = self.password if self.identifier_type ==
IDENTIFIER_TYPE_EMAIL
credential.save!
end
end
end
75. Technical Learnings
GIT
Distributed version control that works
No branches on SVN to dozens of branches on Git
Fine-grained control of revisions
Fewer conflicts
A branch per story
83. Test all the f**king time!!!
TDD/BDD is a muscle, you need to exercise it
Without coverage, there is no understanding
Knowledge socialization
Continuous integration
84. Continuous Integration
Dedicated integration environment with automated
integration and tests execution with the most up to
dated code base.
Automated Reporting
Test Cases Results
Tests Coverage
104. Looking Forward
Queues are the new black : starling & others
Asynchronous is the newer black : workling & others
Erlang + Cloud = massive continuous computing space
Distributed objects : Distributed search
Predictive caching
Algorithms & Computer Science : latent semantics,
vector space model, etc
105. Three Lines on Scrum
Fastest way to go to nowhere
Scrum without a present Product Owner
Slowest way to go to somewhere
Scrum without a dedicated Scrum Master
Embrace it fully with the best team.