You're Going To Need A Bigger Toolbox

You’re Going To Need A
Bigger Toolbox

DIBI 28th April 2010

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/booleansplit/2376359338/

morethanseven.net

Gareth Rushgrove

gareth rushgrove | morethanseven.net

We Used To Just Build Websites


Probably Simple Websites


Maybe the odd Ecommerce Site


Then We Built Web Applications


Now We Just Build.


Your Toolbox is Growing

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/gordonr/42555739

1
Embrace Heterogeneous Environments

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/mk1971/2548492513/

1
No Development Tool Silver Bullet

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/mk1971/2548492513/

We Build Websites With...

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/freefoto/3436970425/

PHP, MySQL, Apache


.NET, MSSQL, IIS


Java, Oracle, Tomcat


Lots of Others


Just One Stack?

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/harrygoldenfeld/4263710200/

The Guardian Java, Python, Oracle, App Engine


LastFM PHP, C++, Java, Hadoop, Python


GitHub Ruby, Erlang, MySQL, Redis, Sinatra


Twitter Ruby, Scala, Java, C/C++


Facebook PHP, Erlang, C, MySQL, Cassandra


2
2. Know When Your Stack is out of its Depth

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/dk_spook/2421009077/

What Should You Know About?

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/takomabibelot/220265100/

wiki.nginx.org

Serve Web Pages With Nginx


“Apache is like Microsoft Word, it has a
million options but you only need six. Nginx
does those six things, and it does ﬁve of
them 50 times faster than Apache.
Chris Lea

Why Nginx


server {
listen 80;
server_name www.example.com;

location / {
root /var/www/example.com;
}
}

Nginx Example


http {
upstream php {
server localhost:8002;
}
upstream python {
server localhost:8003;
}
}

server {
server_name www.example.com;
location / {
proxy_pass http://python;
}
location ~ /basket/* {
proxy_pass http://php;
}
}

Nginx Example


- Thin - http://code.macournoyer.com/thin/
- Mongrel - http://github.com/fauna/mongrel/
- Spawning - http://pypi.python.org/pypi/Spawning/
- Unicorn - http://github.com/defunkt/unicorn/

Also See


memcached.org

Caching with Memcached


from django.core.cache import cache

key = "/about/"
content = cache.get(key)
if not content:
# expensive query to get content
cache.set(key, content, 300)

Memcached Example


- Squid - http://www.squid-cache.org
- Varnish - http://varnish-cache.org

Also See


lucene.apache.org/solr/

Search with Solr


http://solr:8983/solr/products/select/
?q=colour:red
&sort=price%20desc
&rows=60
&wt=json

Solr Example


- Xapian - http://xapian.org
- Sphinx - http://sphinxsearch.com
- Nutch - http://lucene.apache.org/nutch/
- Whoosh - http://whoosh.ca

Also See


rabbitmq.com

Asynchronous Processing with RabbitMQ


require 'carrot'

q = Carrot.queue('carrot', :durable => true)
10.times do |num|
q.publish(num.to_s)
end

RabbitMQ Enqueue Example


puts "count: #{q.message_count}"

while msg = q.pop(:ack => true)
puts msg
q.ack
end

Carrot.stop

RabbitMQ Dequeue Example


- Apache ActiveMQ - http://activemq.apache.org
- Beanstalk - http://kr.github.com/beanstalkd/
- Stomp Server - http://stomp.codehaus.org
- MemcacheQ - http://memcachedb.org/memcacheq/

Also See


couchdb.apache.org

Data Storage With CouchDB


<?php

$options['host'] = "localhost";
$options['port'] = 5984;

$couch = new CouchSimple($options);

$resp = $couch->send("PUT", "/test");

$resp = $couch->send("PUT", "/test/123",
'{"_id":"123","data":"Foo"}');

?>

CouchDB Example


<?php

$options['host'] = "localhost";
$options['port'] = 5984;

$couch = new CouchSimple($options);

$resp = $couch->send("GET", "/test/_all_docs");

$resp = $couch->send("GET", "/test/123");

$resp = $couch->send("DELETE", "/test/");

?>

CouchDB Example


- MongoDB - http://www.mongodb.org
- Tokyo Tyrant - http://1978th.net/tokyotyrant/
- Cassandra - http://cassandra.apache.org
- Redis - http://code.google.com/p/redis/

Also See


hadoop.apache.org

Data Mining With Hive


CREATE TABLE u_data (
userid INT,
movieid INT,
rating INT,
unixtime STRING)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 't'
STORED AS TEXTFILE;

Hive Create Example
Cucumber DSL Example


LOAD DATA LOCAL INPATH 'data.txt' OVERWRITE
INTO TABLE u_data;

SELECT COUNT(1) FROM u_data;

Hive Load DSL Example
Cucumber Example


add FILE weekday_mapper.py;

INSERT OVERWRITE TABLE u_data_new
SELECT
TRANSFORM (userid, movieid, rating, unixtime)
USING 'python weekday_mapper.py'
AS (userid, movieid, rating, weekday)
FROM u_data;

SELECT weekday, COUNT(1)
FROM u_data_new
GROUP BY weekday;

Hive Map Reduce Example


- Hadoop - http://hadoop.apache.org
- Hive - http://wiki.apache.org/hadoop/Hive/
- Pig - http://hadoop.apache.org/pig/
- Dumbo - http://lastfm.com/dumbo/
- Disco - http://discoproject.org

Also See


cukes.info

Testing with Cucumber


Feature: google.co.uk
To broaden their knowledge
A user should be able
To search for things

Scenario: Searching for things
Given I visit "http://www.google.co.uk"
When I fill in "q" with "wikipedia"
And I press "Google Search"
Then I should see "www.wikipedia.org"



Feature: google.co.uk
To broaden their knowledge
A user should be able
To search for things

Scenario: Searching for things
Given I visit "http://www.google.co.uk"
When I fill in "q" with "wikipedia"
And I press "Google Search"
Then I should see "www.wikipedia.org"

1 scenario (1 failed)
4 steps (1 failed, 2 skipped, 1 passed)
0m0.332s

Cucumber Results Example


puppetlabs.com

Server Provisioning with Puppet


class baseclass {
$packagelist = ["sudo", "openssh-server"]
package { $packagelist: ensure => installed }
service { sshd:
name => "ssh",
enable => true,
ensure => running
}
}

Puppet Class Example


node 'example.com' inherits basenode {
$packagelist = ["nginx"]
package { $packagelist: ensure => installed }
service { "nginx":
ensure => running,
require => Package["nginx"]
}
}

Puppet Node Example


- Chef - http://wiki.opscode.com/display/chef/Home/

Also See


Conclusions

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/batega/1596898776/

1
Know What's Possible

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/docman/36125185/

2
Look for Opportunities

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/kgregory/500456103/

3
Experiment

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/seeminglee/3967329241/

4
Manage Complexity

gareth rushgrove | morethanseven.net http://www.ﬂickr.com/photos/30890318@N06/3510161637/

Questions?

gareth rushgrove | morethanseven.net http://ﬂickr.com/photos/psd/102332391/

You're Going To Need A Bigger Toolbox

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Viewers also liked

Viewers also liked (7)

Similar to You're Going To Need A Bigger Toolbox

Similar to You're Going To Need A Bigger Toolbox (20)

More from Gareth Rushgrove

More from Gareth Rushgrove (20)

Recently uploaded

Recently uploaded (20)

You're Going To Need A Bigger Toolbox