Cloud, Cache, and Configs

Cloud, Cache, and Conﬁgs
WordCamp NYC 2012

Saturday, June 9, 12

Scott Taylor
Lead PHP Developer, eMusic
@wonderboymusic
www.scotty-t.com
#projectmanagementsoftware


eMusic

Multisite
Regionalized Content
Regionalized Caching
Shared Content across Sites
Tons of Custom Post Types
Post Formats
Tons of Web Services


eMusic Architecture
• 12-24 Amazon EC2 instances (CentOS)
• 4 MySQL (1 Write, 3 Read)
• 4 Memcached (~28GB RAM)
• Amazon S3 / Cloudfront CDN
• PHP 5.3 / MySQL 5.5
• PECL: APC, Memcached, HTTP
• Batcache

APC = duh

• Essential for PHP
• Opcode cache
• apc.shm_size = 64M or higher


Logs

• Never allow any PHP-related notices,
errors, warnings, exceptions, etc
• Check access logs regularly for 404s, 500s
etc
• make a constant for toggling debug logging


Production Data

• Always pull down, never push up
• We only push imported legacy content up
• output buffering
• code ﬁltering
• more to come on this....


Output Buffering

ob_start( $callback )

ob_start();

echo ‘Daryl’;

$daryl = ob_get_clean();


ob_start( function ( $data ) {

return str_replace(
$urls_array,
EMUSIC_CURRENT_HOST,
$data
);

} );


Configs

• Mandatory machine configs
• HyperDB config
• Overridable sunrise.php


Machine Config
• DB credentials local to that machine
• Amazon S3 bucket
• Web Service Endpoints
• Memcached Servers
if ( file_exists( ‘/wp-config/config.php’ ) ) {
require_once( ‘/wp-config/config.php’ );
} else {
die( ‘You must have a local config!’ );
}

Memcached

• Memcached PHP Extension (not Memcache)
• Can be used locally (127.0.0.1)
• Memcached Redux supports
wp_cache_get_/set_multi( )
• Johnny Cache


Batcache
• Full-page caching
• Can be conﬁgured
• You can partition cache by unique values
• Loads before plugins - any code you need
has to be duped or added early
(sunrise.php)


class batcache {

// This is the base configuration. You can edit these variables or move
them into your wp-config.php file.
var $max_age = 300;

// Expire batcache items aged this many seconds (zero to disable batcache)

var $remote = 0;

// Zero disables sending buffers to remote datacenters (req/sec is never sent)

var $times = 5;

// Only batcache a page after it is accessed this many times... (two or more)

var $seconds = 120;

// ...in this many seconds (zero to ignore this and use batcache immediately)

var $group = 'batcache';

// Name of memcached group. You can simulate a cache flush by changing this.

var $unique = array( BATCACHE_REGION, BATCACHE_COUNTRY );

// If you conditionally serve different content, put the variable values here.

var $headers = array(
'X-nananana' => 'Batcache'
);

. . . .
}


Sunrise
• Used to alter multisite context
• sets $current_blog and $current_site
• ﬁlters all URL functions to resolve all URLs
to your current domain
• registers custom locations for media
• ﬁlters Admin URLs


switch_to_blog( $blog_id )

• All dynamic functions need to account for
this
• Shared content needs to resolve proper
URLs
• Different sites have different media
locations


Fix switch_to_blog()
$current_blog = new stdClass();
$current_blog->site_id = 1;
$current_blog->archived = 0;
$current_blog->mature = 0;
$current_blog->spam = 0;
$current_blog->deleted = 0;
$current_blog->lang_id = 0;
$current_blog->public = 1;
$current_blog->registered = '2011-02-20 03:38:22';
$current_blog->last_updated = $_SERVER['REQUEST_TIME'];
$current_blog->domain = EMUSIC_CURRENT_HOST;

function emusic_switch_to_blog( $blog_id, $prev_blog_id = 0 ) {
if ( $blog_id === $prev_blog_id )
return;

global $current_blog, $emusic_paths;
$current_blog->blog_id = $blog_id;
$current_blog->path = $emusic_paths[$blog_id];
}

emusic_switch_to_blog( $the_id );

add_action( 'switch_blog', 'emusic_switch_to_blog', 10, 2 );

$blog_id = $the_id;
$site_id = 1;

$current_site = new stdClass();
$current_site->blog_id = $the_id;
$current_site->id = 1;
$current_site->domain = EMUSIC_CURRENT_HOST;
$current_site->site_name = 'eMusic';
$current_site->path = $the_path;

Filter URLs

add_filter( 'pre_option_upload_path', function () {
$id = get_current_blog_id();
if ( 1 < $id )
return $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$id}/files";

return $_SERVER['DOCUMENT_ROOT'] . '/' . EMUSIC_UPLOADS;
} );

add_filter( 'pre_option_upload_url_path', function () {
$id = get_current_blog_id();
if ( 1 < $id )
return 'http://' . EMUSIC_CURRENT_HOST . "/blogs/{$id}/files";

return 'http://' . EMUSIC_CURRENT_HOST . '/' . EMUSIC_UPLOADS;
} );

add_filter( 'pre_option_siteurl', function () {
global $current_blog;
$extra = rtrim( $current_blog->path, '/' );
return 'http://' . EMUSIC_CURRENT_HOST . $extra;
} );

add_filter( 'pre_option_home', function () {
global $current_blog;
$extra = rtrim( $current_blog->path, '/' );
return 'http://' . EMUSIC_CURRENT_HOST . $extra;
} );


Plugins
• Filter active network plugins (don’t rely on
database being correct)
• Filter each site’s plugins (if you have a
manageable number)
• Use classes, not a bunch of functions
• Extend before you copy / paste


Site Conﬁgs
require_once( 'site-configs/global.php' );

if ( $the_id > 1 ) {
define( 'UPLOADBLOGSDIR', 0 );
define( 'UPLOADS', 0 );
define( 'BLOGUPLOADDIR', $_SERVER['DOCUMENT_ROOT'] . "/blogs/{$the_id}/files" );

switch ( $the_id ) {
case 2:
require_once( 'site-configs/bbpress.php' );
break;

case 3:
require_once( 'site-configs/dots.php' );
break;

case 5:
require_once( 'site-configs/support.php' );
break;
}

add_filter( 'pre_option_template', function () {
return 'dark';
} );
} else {
require_once( 'site-configs/emusic.php' );
}


Global Plugins
add_filter( 'pre_site_option_active_sitewide_plugins', function () {
return array(
'batcache/batcache.php' => 1,
'akismet/akismet.php' => 1,
'avatar/avatar.php' => 1,
'bundle/bundle.php' => 1,
'cloud/cloud.php' => 1,
'download/download.php' => 1,
'emusic-notifications/emusic-notifications.php' => 1,
'emusic-ratings/emusic-ratings.php' => 1,
'emusic-xml-rpc/emusic-xml-rpc.php' => 1,
'johnny-cache/johnny-cache.php' => 1,
'like-buttons/like-buttons.php' => 1,
'members/members.php' => 1,
//'minify/minify.php' => 1,
'apc-admin/apc-admin.php' => 1
//'debug-bar/debug-bar.php' => 1
);
} );


Site Plugins

add_filter( 'pre_option_active_plugins', function () {
return array(
'artist-images/artist-images.php',
'catalog-comments/catalog-comments.php',
'emusic-post-types/emusic-post-types.php',
'discography.php',
'emusic-radio/emusic-radio.php',
'gravityforms/gravityforms.php',
'super-ghetto/super-ghetto.php'
//,'theme-check/theme-check.php'
);
} );


class MyPlugin {
function init() {
add_action( ‘init’, array( $this, ‘register’ ) );
}

function register() {}
}

$my_plugin = new MyPlugin();
$my_plugin->init();

class MyPlugin {
function init() {
add_action( ‘init’, array( ‘MyPlugin’, ‘register’ ) );
}

function register() {}
}

MyPlugin::init();


Share code when possible

class FeaturePack extends eMusicPostTypes implements PostType {
..........
}


MySQL
• Use 5.5, better handlng of weird multibyte
strings
• Use InnoDB, not MyISAM, pretty much in
all cases
• HyperDB handles scaling for you
• Benchmark queries, don’t be afraid to roll
your own SQL, tables, use $wpdb


HyperDB

• Can be empty locally
• Inherits wp-conﬁg DB defaults
• contains functions for replication lag
detection (when used with mk-heartbeat)


Themes

• Theme setup is a class
• Extend before you repeat
• Classes are better than preﬁxing function
names


Theme Conﬁg in functions.php

class Theme_17Dots extends Regionalization {
function __construct() {
global $dots_regions_tax_map, $dots_regions_map;

$this->regions_map = $dots_regions_map;
$this->regions_tax_map = $dots_regions_tax_map;

parent::__construct();
}

function init() {
add_action( 'init', array( $this, 'register' ) );
add_action( 'after_setup_theme', array( $this, 'setup' ) );
add_action( 'add_meta_boxes_post', array( $this, 'boxes' ) );
add_action( 'save_post', array( $this, 'save' ), 10, 2 );
add_filter( 'embed_oembed_html', '_feature_youtube_add_wmode' );
}
. . . . . . .
}


Use base classes

class Regionalization {
var $regions_map;

add_filter( 'manage_posts_columns', array( $this, 'manage_columns' ) );
add_action( 'manage_posts_custom_column', array( $this,
'manage_custom_column' ), 10, 2 );
add_filter( 'posts_clauses', array( $this, 'clauses' ), 10, 2 );
add_filter( 'manage_edit-post_sortable_columns',array( $this, 'sortables' ) );
add_filter( 'pre_get_posts', array( $this, 'pre_posts' ) );
}
. . . . . .
}


pre_get_posts
function regionalize( $query ) {
global $regions_map;
if ( $query->is_main_query() && !is_admin() && ( is_search() || is_archive() || is_home() ) ) {

$types = get_post_types( array( 'publicly_queryable' => true, '_builtin' => false ) );
$tax_region = array(
'taxonomy' => 'region',
'field' => 'term_id',
'terms' => array(
$regions_map[ 'ALL' ],
$regions_map[ THE_REGION ]
),
'operator' => 'IN'
);

if ( is_home() ) {
$query->set( 'posts_per_page', 10 );

} else if ( is_search() && get_option( 'editorial_search_enabled' ) ) {
$ctx = get_query_var( 'search_context' );
$query->set( 's', stripslashes( urldecode( get_query_var( 's' ) ) ) );

if ( in_array( $ctx, array( 'features', 'features-books' ) ) ) {
} else {
}

if ( in_array( $ctx, array( 'books', 'features-books' ) ) ) {
foreach ( $types as $type )
if ( false === strpos( $type, 'book' ) )
unset( $types[$type] );
}

if ( empty( $query->posts ) && $query->is_paged() )
$query->is_paged = false;

} else if ( is_tag() ) {
if ( empty( $query->posts ) && $query->is_paged() )
$query->is_paged = false;
}

//error_log( 'REGIONALIZING' );

if ( !is_post_type_archive() )
$query->set( 'post_type', array_keys( $types ) );

$query->set( 'tax_query', array( $tax_region ) );
}

return $query;
}


Assets

• Use remote storage
• Use a CDN
• Replace hosts using output buffer
• Cloud - pieces of W3 Total Cache


Minify
• JS / CSS concatenation speed up your
front-end loading / perceived loading
• Minify is automagic
• HTML5 CSS properties are automatically
inﬂated
• Admin tool to cache-bust URLs
• Auto-locking while ﬁles are generated

Web Services
• cURL PHP extension
• curl and curl_multi()
• Memcached is essential
• hooks into parse_request to load data
along with WordPress, allows us to cause
WP to 404 and bail early when required
data response fails
• Parallelization with curl_multi()


class eMusicRequest {
var $request;
var $sub_request;

var $path;
var $page;

function load( $page = '' ) {
if ( !empty( $page ) )
$this->page = $page;

$file = $this->path . $this->page . '.php';
if ( file_exists( $file ) )
require_once( $file );
}

function parse() {
$requests = array( $this->request, $this->sub_request );

foreach ( $requests as $request ) {
if ( !empty( $request ) ) {
$keys = array_keys( get_class_vars( get_class( $request ) ) );
foreach ( $keys as $var ) {
if ( !empty( $request->$var ) ) {
$GLOBALS[$var] = $request->$var;
}
}
}
}
}
}


class DarkRequest extends eMusicRequest {
var $genre;
var $post_type;

function init( $request ) {
global $_GENRES;

$vars =& $request->query_vars;

.........
}

}


switch ( get_current_blog_id() ) {
case 1:
$_dark_request = new DarkRequest();
add_action( 'parse_request', array( $_dark_request, 'init' ) );
break;
case 4:
$_my_emusic_request = new MyEMusicRequest();
add_action( 'parse_request', array( $_my_emusic_request,
'init' ) );
break;
}


<?php
class HomeRequest extends RequestMap {
var $recommendations;

parent::__construct();

if ( !get_option( 'recs_enabled' ) )
return;

if ( is_user_logged_in() && 'US' === THE_REGION ) {
$user = wp_get_current_user();
$user_id = isset( $_GET['user_id'] ) ? $_GET['user_id'] : $user->ID;
$params = array( 'userId' => $user_id, 'return' => true );

$this->add( get_user_recommendations( $params ), array( $this, 'parse_recommendations' ) );
$this->send();
}
}

function parse_recommendations( $data ) {
if ( empty( $data['recommendations'] ) )
return;

$data = $data['recommendations'];

if (
!empty( $data ) &&
isset( $data[key($data)]['items'] ) &&
!empty( $data[key($data)]['items'] )
) {
$this->recommendations = array_slice( $data[key($data)]['items'], 0, 18 );
shuffle( $this->recommendations );

foreach ( $this->recommendations as &$rec ) {
$rec = array( 'work_id' => $rec['catalogId'] );
}
}
}
}


<?php
class RequestMap extends API {
private $requests;
private $responses;
private $ttl;
private $useCache = true;

protected $error = false;

public function __construct() {
$this->flush();
}

public function is_error() {
return $this->error;
}

public function flush() {
$this->requests = array();
}

public function getTtl() {
if ( empty( $this->ttl ) ) {
$this->ttl = CACHE::API_CACHE_TTL;
}
return $this->ttl;
}

public function add( $url, $callback, $vars = array() ) {
$params = new stdClass();
$params->url = $url;
$params->callback = $callback;
$params->params = (array) $vars;
$this->requests[] = $params;
}

private function exec( $item, $response ) {
$params = array_merge( array( $response ), $item->params );
call_user_func_array( $item->callback, $params );
}

public function send() {
if ( !empty( $this->requests ) ) {
$this->responses = self::batch( $this->getRequestUrls(), $this->getTtl(), $this->useCache );

if ( is_array( $this->responses ) ) {
foreach ( $this->responses as $i => $response ) {
if ( !empty( $this->requests[$i] ) ) {
$this->exec( $this->requests[$i], self::parse_response( $response ) );
}
}
}
}
}
}

class API {

public static function batch( $urls, $ttl = '', $usecache = true ) {
$response = array();
ob_start();

if ( empty( $ttl ) ) {
$ttl = CACHE::API_CACHE_TTL;
}

if ( is_array( $urls ) ) {
if ( $usecache ) {
foreach ( $urls as $index => $url ) {
$in = Cache::get( Cache::API, $url );

if ( $in ) {
$response[$index] = $in;
unset( $urls[$index] );
}
}
}

$keys = array_keys( $urls );
}
$calls = self::multi_request( $urls );

. . . . . . . .


if ( is_array( $calls ) && count( $calls ) > 0 ) {
$calls = array_combine( $keys, array_values( $calls ) );

foreach ( $calls as $index => $c ) {
if ( $c ) {
$response[$index] = self::parse_response( $c );
if ( isset( $response[$index]['status']['code'] ) && $response[$index]
['status']['code'] < 400 ) {
if ( isset( $response[$index]['results'] ) && !
empty( $response[$index]['results'] ) ) {
Cache::put( Cache::API, $urls[$index], $response[$index], $ttl );
} else {
}
} else if ( !isset( $response[$index]['status']['code'] ) ) {
}
} else {
$response[$index] = null;
}
}
} else if ( !empty( $calls ) && isset( $urls[0] ) ) {
$data = self::parse_response( $calls );
if ( isset( $data['status']['code'] ) && $data['status']['code'] < 400 ) {
if ( isset( $data['results'] ) && !empty( $data['results'] ) ) {
Cache::put( Cache::API, $urls[0], $data, $ttl );
} else {
}
} else if ( !isset( $data['status']['code'] ) ) {
}
$response[] = $data;
}
ob_end_clean();

return $response;

Because we used
classes, everything is
abstracted


Custom Authentication
• WP stores slashed passwords
• wp_authentication arguments are slashed
• Inherit your base system’s rules
• DO NOT sanitize email / password
(WordPress does by default)
• User tables in this case are really a
transient cache


Questions / complaints?


Cloud, Cache, and Configs

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Cloud, Cache, and Configs

Ähnlich wie Cloud, Cache, and Configs (20)

Mehr von Scott Taylor

Mehr von Scott Taylor (11)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Cloud, Cache, and Configs