Lessons from building the Dutch public broadcaster's website- The team built a CMS from scratch in Rails to handle a high-traffic website for the Dutch public broadcaster. It received 30,000-40,000 pageviews daily.- They used BDD with RSpec and Cucumber for testing. Code was well-tested with over 2,000 specs. - Performance was optimized through caching, moving static assets to a CDN, and using Memcache. NewRelic RPM was used for monitoring.- External data from RSS feeds, EPG data, etc. required robust error handling due to inconsistencies. Data was
This document provides lessons learned from building the Dutch public broadcasting company's website omroep.nl. Key points include using Ruby on Rails, BDD with RSpec and Cucumber, caching everything possible, rescuing errors, testing extensively, and handling large amounts of external data from various XML/RSS feeds and APIs. Performance was optimized through techniques like moving static assets to a front proxy, page caching, fragment caching, and using Memcache. The team of 6 people built the CMS from scratch over 6 months.
OpenStack Austin Meetup January 2014: Chef + OpenStack
Ähnlich wie Lessons from building the Dutch public broadcaster's website- The team built a CMS from scratch in Rails to handle a high-traffic website for the Dutch public broadcaster. It received 30,000-40,000 pageviews daily.- They used BDD with RSpec and Cucumber for testing. Code was well-tested with over 2,000 specs. - Performance was optimized through caching, moving static assets to a CDN, and using Memcache. NewRelic RPM was used for monitoring.- External data from RSS feeds, EPG data, etc. required robust error handling due to inconsistencies. Data was
Ähnlich wie Lessons from building the Dutch public broadcaster's website- The team built a CMS from scratch in Rails to handle a high-traffic website for the Dutch public broadcaster. It received 30,000-40,000 pageviews daily.- They used BDD with RSpec and Cucumber for testing. Code was well-tested with over 2,000 specs. - Performance was optimized through caching, moving static assets to a CDN, and using Memcache. NewRelic RPM was used for monitoring.- External data from RSS feeds, EPG data, etc. required robust error handling due to inconsistencies. Data was (20)
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Lessons from building the Dutch public broadcaster's website- The team built a CMS from scratch in Rails to handle a high-traffic website for the Dutch public broadcaster. It received 30,000-40,000 pageviews daily.- They used BDD with RSpec and Cucumber for testing. Code was well-tested with over 2,000 specs. - Performance was optimized through caching, moving static assets to a CDN, and using Memcache. NewRelic RPM was used for monitoring.- External data from RSS feeds, EPG data, etc. required robust error handling due to inconsistencies. Data was
1. Lessons learned while
building omroep.nl
Bart Zonneveld (@bartzon)
Sjoerd Tieleman (@tieleman)
2. Nederlandse Publieke Omroep
Dutch Public broadcasting Company
AVRO Joodse Omroep NMO Teleac
BNN KRO NOS TROS
BOS LLiNK NPS VARA
EO MAX OHM VPRO
HUMAN NCRV RKK ZvK
IKON NIO RVU
3. Rails sites
• Beetlejuice • Radio 1
• Centrale • Z@PP
navigatie
• Omroep.nl • Z@ppelin
• Nederland 1 • Zelda
• Nederland 3 • Various tools
4.
5. Team
• 2 coders
• 1 designer
• 1 editor
• 1 project manager
6 months, CMS built from scratch
17. 3 slide intro to BDD:
RSpec
describe Article do
it_should_behave_like "all objects with userstamps"
it_should_behave_like "all objects than can be published"
it_should_behave_like "all objects that have an url"
it_should_behave_like "all objects that can be searched"
it_should_behave_like "all objects with related articles"
it "should not be valid without a name" do
@article.attributes = @valid_attributes.except(:name)
@article.should_not be_valid
end
it "should not be valid without contents" do
@article.attributes = @valid_attributes.except(:contents)
@article.should_not be_valid
end
end
18. 3 slide intro to BDD:
Cucumber features
Feature: Articles on the homepage
As a visitor
I want to view articles on the homepage
So that I can see the latest content
Scenario: 5 most recent articles
Given there are 8 articles
When I visit the homepage
Then I should see the 5 last published articles
19. 3 slide intro to BDD:
Cucumber steps
Given "there are $num articles" do |num|
num.to_i.times { create_article }
end
When "I visit the homepage" do
visit root_path
end
Then "I should see the $num last published articles" do |num|
Article.last_published(num).each do |article|
response.should contain(article.title)
end
end
21. Shared behaviours
module UserStamps
def self.included(klass)
class Article < ActiveRecord::Base
klass.instance_eval do
include Shared::UserStamps
include InstanceMethods
end include Shared::Published
end include Shared::Url
include Shared::Search
module InstanceMethods include Shared::RelatedArticles
def created_by
User.find_by_id(creator_id) # stuff
end end
def updated_by
User.find_by_id(updater_id)
end
end
end
29. External data
• RSS feeds
• EPG data
• Zelda
• Babel
• News/sport/teletekst
• Twitter
• Lots of custom XML formats
30. External data: XML/RSS
• Empty feeds
• Encodings are off (Windows-1252,
ISO-8859-1, UTF-8)
• “Custom” fields
• Incorrect fields (dates, unescaped HTML)
• Timeouts
• Everything that can go wrong, will go wrong
41. Case: banner items
• Fast requests (<10ms)
• ETags (304 Not Modified)
• Static resource → page cache
• Move to front proxy, frees up Rails cluster
• 1100rpm → 130rpm
• 20ms → 40ms
• Average response time going up? Oh nooooes!
42. Caching external data
• Don’t expire cache (preferrably)
• Explicitly overwrite
• Update in background (feeeeeeeds)
• memcache FTW!
43. memcache
• Escape your keys using CGI::escape
• Max keylength is 250
• Max value size is 1MB
44. Rescueing
def self.get_feed_contents(url)
content = ""
open(url) { |s| content = s.read }
RSS::Parser.parse(content, false).items
rescue => e
logger.warn "Feed #{url} raised error: #{e.message}"
[]
rescue Timeout::Error => e
logger.warn "Feed #{url} timed out: #{e.message}"
[]
end
Timeout::Error is an interrupt...
45. Testing
• rcov
• Refactor your tests
• Peer reviews, external audits
• Run specs/features
(continuously) in parallel
(your Cucumber is too slooooow!)
46. Cucumber salad
num_of_processes.times do |count|
pids << Process.fork do
setup_database(conn, count)
Cucumber::Cli::Main.execute(
["-f", "progress", "-l", "nl", "-r", "features"] +
feature_sets[count]
)
end
end
MacBook Pro
“Regular” Mac Pro (8)
(4)
12:12 4:34 2:12