1.
WebPagetest
Private Instances
Patrick Meenan
pmeenan@webpagetest.org
@patmeenan

2.
The Basics
● Presentation is available on slideshare
○ http://www.slideshare.net/patrickmeenan
○ Mostly contains links to everything I’ll be showing
● Video will be uploaded to YouTube
○ https://www.youtube.com/user/patmeenan
● I’ll send links and announcements of both to Twitter and Google+
○ https://twitter.com/patmeenan - @patmeenan
○ https://plus.google.com/u/0/+PatrickMeenan/posts

3.
WPT Web Server
Sites being tested
System Architecture
Desktop Agent
Node.js Mobile Agent
Mobitest Agent

4.
WPT Web Server
Sites being tested
Agent
Data Flow
Submit test (UI or API)
http://<server>/runtest.php
User

5.
Server:
- Creates test ID
YYMMDD_<hash>_<num>
- Creates directory for test
results/YY/MM/DD/<hash>/<num>
- Writes test job to work directory for location
work/jobs/<location>/<ID>.<priority>
- Adds test to queue file (lossy, can be re-built from the work directory)
tmp/<location hash>.queue
- Redirects to result page

6.
WPT Web Server
Sites being tested
Agent
Poll for test status
http://<server>/result/<ID>/
User

7.
WPT Web Server
Sites being tested
Agent
User
Poll for work for location
http://<server>/work/getwork.php

8.
WPT Web Server
Sites being tested
Agent
User
- Launch Browser
- Navigate to page
- Measure/Collect Data

9.
WPT Web Server
Sites being tested
Agent
User
Upload large files individually
(images, tcpdump, timeline, etc)
http://<server>/work/resultimage.php

10.
WPT Web Server
Sites being tested
Agent
User
Complete run
(or full test if all runs are done)
http://<server>/work/workdone.php

11.
WPT Web Server
Sites being tested
Agent
View completed test
http://<server>/result/<ID>/
User

12.
Server Config
● Any server that can run php
○ Linux tends to work best
○ Can run on the same PC as test agents if needed
● Copy www/ directory from distribution to site docroot
● Configure rewrites for friendly URLs (optional)
○ .htaccess included for apache
○ nginx.conf can be included into nginx server config
● Fix anything that http://<server>/install/ complains about
○ PHP ini settings: allow_url_fopen, memory and post file size limits
○ PHP modules: GD, zip, zlib, curl, apc, sqlite
○ Command-line utilities: ffmpeg, imagemagick, jpegtran, exiftool
○ Filesystem permissions (and directories)

13.
Nginx config
server {
listen 80;
server_name webpagetest.example.com;
root /var/www;
location ~ .php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME /var/www$fastcgi_script_name;
fastcgi_param HTTP_MOD_REWRITE On;
include fastcgi_params;
}
include /var/www/nginx.conf;
}

14.
/install/

15.
Server WPT Config
● Copy/modify settings/*.sample
○ settings/settings.ini
■ Contact information
■ Maximum number of runs
■ Screen shot quality
■ Show Slow integration
■ Test archiving (covered later)
○ settings/locations.ini
■ Definition for all of the test locations

16.
Locations.ini
Grouping
Browsers

17.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...

18.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
List of the group IDs in order to be displayed

19.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
“Location ID”s that are part of the group in the
order the browsers are to be displayed.
String displayed for the “Test Location”

20.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
These are the location ID’s the agents connect to
and that are used when submitting tests through
the API

21.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
All of the browsers from all of the locations in a
group are collected and displayed in the Browser
select box.

22.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
All of the agents for a given location ID must
support all of the browsers listed

23.
[NYC]
browser=Chrome,Firefox
label=”New York”
[NYC_IE10]
browser=IE 10
label=”New York”
[NYC_IE11]
browser=IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
A single agent can pull from multiple locations
(configure it as a comma-separated list).
location=NYC,NYC_IE10

24.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
lat=40.712918
lng=-74.005469
group=North America
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...
Add map=1 to settings.ini

25.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
lat=40.712918
lng=-74.005469
group=North America
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
[NYC_Mobile]
browser=Motorola G - Chrome,Motorola G - Chrome Beta
label=”New York”
[Group_2]
...

26.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
When traffic shaping is not supported.
Shows up in the UI as the connectivity information.

27.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
Shared key for securing agent communications
(optional)

28.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
Expected number of agents (for alerting)

29.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
Email addresses for alerting when agents stop
connecting.

30.
[locations]
1=Group_1
2=Group_2
default=Group_1
[Group_1]
1=NYC
2=NYC_Mobile
label=”New York (Chrome, Firefox, IE, Android)”
[NYC]
browser=Chrome,Firefox,IE 11
label=”New York”
connectivity="Google Fiber (1Gbps)"
key=123456
agents=6
notify=pmeenan@webpagetest.org,admin@webpagetest.org
hidden=1
Hide the location from the UI (still works through the API
and from http://<server>/?hidden=1 )

31.
Desktop Agent Config
● Prepare Windows
○ Auto logon
○ Disable screen savers and screen power management (never sleep)
○ Disable UAC
○ Copy agent/* to local disk (c:/webpagetest/ in the examples)
○ Install dummynet (optional for traffic shaping)
○ Create startup shortcut to wptdriver.exe (and urlblast.exe if used)
● Install browsers
○ Firefox can self-install (necessary for updates)
● Configure wptdriver.ini
○ Server URL
○ Location ID
○ Browsers

32.
wptdriver.ini
[WebPagetest]
url=http://127.0.0.1/
location=Test
browser=chrome
Time Limit=120
;key=TestKey123
;Automatically install and update support software (Flash, Silverlight, etc)
software=http://www.webpagetest.org/installers/software.dat
[chrome]
exe="%PROGRAM_FILES%GoogleChromeApplicationchrome.exe"
options='--load-extension="%WPTDIR%extension" --user-data-dir="%PROFILE%" --no-proxy-server'
installer=http://www.webpagetest.org/installers/browsers/chrome.dat
[Firefox]
exe="%PROGRAM_FILES%Mozilla Firefoxfirefox.exe"
options='-profile "%PROFILE%" -no-remote'
installer=http://www.webpagetest.org/installers/browsers/firefox.dat
template=firefox
[Safari]
exe="%PROGRAM_FILES%SafariSafari.exe"
[IE]
exe="%PROGRAM_FILES%Internet Exploreriexplore.exe"

33.
EC2 Test Agents
● AMI’s available in all regions
○ IE8-11, Chrome, Firefox, Safari 5 (Windows)
● Pass configuration through user data when launching
○ wpt_server=www.webpagetest.org wpt_loc=EC2_East
● No need to ever touch/login to instance
● Kill/Start as needed
● c1.medium or better recommended

34.
EC2 Test Agents - User Data

35.
EC2 Test Agents - Locations

36.
Sites being tested
WPT Web Server
Mobile Agent (Node)
Node.js Mobile Agent

37.
Mobile Agents - Android
● Android 4.4+ recommended
● Root (Nexus and Moto phones are easiest)
● Install Chrome and Chrome Beta from Play store
● Configure Play to auto-update
● Disable screen lock (security settings)
● Configure developer mode
○ Enable USB debugging
○ Disable screen sleep

38.
Mobile Agents - iOS (alpha)
● Jailbreak
● install/configure iOS webkit debug proxy
● build openurl ipa
○ Needs to be signed with your dev key

39.
Mobile Agents - Tethered host config
● Linux or Windows (Mac may work)
○ Multiple phones per tethered agent
● Node.js
● imagemagick
● Launch wptdriver.bat (or wptdriver.sh)
● Config is passed by command-line
○ Device ID
○ Server URL
○ Location ID
● Windows - consider using adbwatch
○ Kills/restarts adb is it gets hung
○ Launches/restarts agents if they crash

40.
Sites being tested
WPT Web Server
Mobile Agent (Node) - Cell Network
Node.js Mobile Agent

41.
Sites being tested
WPT Web Server
Mobile Agent (Node) - RNDIS
Node.js Mobile Agent
IPFW
(Optional)

42.
WPT Web Server
Sites being tested
Mobile Agent (Node) - WiFi
Node.js Mobile Agent
IPFW Bridge
WiFi Access (Optional)
Point

43.
Priority Queues
● Tests are pulled from queues based on test priority
● 10 Priority levels (0-9)
○ 0 = highest, 9 = lowest
● Defaults
○ 0 - Tests submitted through the UI
○ 4 - Bulk tests submitted through the UI
○ 5 - Tests submitted through the API
● Queue lengths reported on location check UI
○ http://<server>/getLocations.php

44.
https://www.flickr.
com/photos/lunchtimemama/110765169/

45.
API Keys
settings/keys.ini (disabled if file is missing)
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
description=API Key
contact=notme@example.com
limit=100

46.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Arbitrary string
used for hashing keys sent to browser
(I usually use GUIDs)

47.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Server API key
To be used by the Web UI

48.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Each API key defined as
an ini file section

49.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Contact info shows up in
usage stats

50.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Allowed “Page Loads” per day
(tests x runs)
x 2 if also recording repeat view

51.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Forced test priority
Defaults to 5 for API
(can be specified in request)

52.
API Keys
[server]
secret=7a5c742c021c414ca0dd1bec42e8f64b
key=acdd95754c654042a526847acb96f692
[acdd95754c654042a526847acb96f692]
description=Web UI
contact=me@example.com
limit=0
[6d3a661b7401486b8a5d4f78962f8b80]
description=API Key
contact=batch@example.com
limit=5000
priority=8
[acd5ffd840094cdda054a0b7e57d4546]
contact=notme@example.com
limit=100
Minimal key configuration

53.
Firewalls...
Intranet WPT Web Server
Desktop Agent
Private Sites
Public Desktop
Agent
Public Sites

54.
Server Lock-down
● Use keys for test locations
● Open access to /work/* for test agents
● Auth access (or IP rules) for rest of the UI

55.
WPT Web Server
Firewalls...
Private Sites
Intranet
Desktop Agent
Public Desktop
Agent
Public Sites
WPT Relay
Server

56.
Relay Server
● Configure locations for public agents
● Configure API keys for internal instance to use for relay
● add headless=1 to settings.ini
○ Disbles UI for submitting tests, requires API key for all tests
● Results are pulled back to internal instance
○ never visible on external

57.
Relay Server - locations.ini
● Configure locations.ini like normal on the relay server
● Local server configures 1:1 mapping of locations
○ forwarded to the relay server
[Public_EC2_East]
browser=IE 11,Chrome,Firefox
label="EC2 East (Public)"
relayServer="http://<relay server>/"
relayKey=<your API key>
relayLocation=EC2_East
Config like normal location
API Key and Location ID
on the Relay Server

58.
Results Storage
● Each test is self-contained in a directory under results/
● Grouped by year, month and day
○ results/YY/MM/DD/…
● Prune old tests? Delete directories.

59.
Results Archiving
● WPT can seamlessly restore missing tests from “archive”
● Supports archiving to local path or S3-like storage
● Configure archive settings in settings.ini
● Test copied to archive after it completes (as 1 zip file per test)
● To delete old archived tests need to run/schedule script
○ php cli/archive.php
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● S3-like
○ zip files uploaded to bucket with <id>.zip filename
○ Can restore from URL or S3 API

60.
Results Archiving - Local Path
● Settings.ini
○ archive_dir=/archive/
○ archive_days=2
● Local Path:
○ zip files copied to specified directory
○ YY/MM/DD/<hash>/<id>.zip
● 2nd Level archiving
○ settings.ini: archive2_dir=/mnt/archive/
○ archive zip files merged into daily zip files
○ cli/archive2.php archive script

61.
Results Archiving - S3
● Settings.ini
○ archive_s3_server=s3.amazonaws.com
○ archive_s3_key=<access key>
○ archive_s3_secret=<secret>
○ archive_s3_bucket=<bucket>
● zip files uploaded to bucket with <id>.zip filename
● Can use URL access to restore
○ Defaults to using API
○ archive_s3_url=http://s3.amazonaws.com/

62.
Bulk Testing in the UI (Private Instances)

63.
Bulk Testing in the UI (Private Instances)
● Can be disabled through settings.ini
○ noBulk=1
● Submit list of URLs to test
● UI options apply to each test
● Scripts can be run against each URL
○ %URL% in script gets replaced with test URL
○ %HOST% in script gets replaced with host from test URL
○ %HOSTR% in script gets replaced with host from URL after redirects
● Aggregate results

64.
API - Submitting a test
● Anything you can do with the UI you can do with the API
○ If a field isn’t documented, just inspect the HTML on the UI to get the
field name :-)
● Make sure to encode parameters
○ urlencode if GET
○ form encode if POST

65.
API Response
http://www.webpagetest.org/runtest.php?url=www.aol.com&f=json
{
"statusCode":200,
"statusText":"Ok",
"data":{
"testId":"140610_FY_N2F",
"ownerKey":"04121a1e5b17a59ba9ddd666ae020cf52cee4abe",
"jsonUrl":"http://www.webpagetest.org/jsonResult.php?test=140610_FY_N2F",
"xmlUrl":"http://www.webpagetest.org/xmlResult/140610_FY_N2F/",
"userUrl":"http://www.webpagetest.org/result/140610_FY_N2F/",
"summaryCSV":"http://www.webpagetest.org/result/140610_FY_N2F/page_data.csv",
"detailCSV":"http://www.webpagetest.org/result/140610_FY_N2F/requests.csv"
}
}

66.
Node API Wrapper
Created by Marcel Duran (@marcelduran)
https://github.com/marcelduran/webpagetest-api
$ npm install webpagetest -g
webpagetest test -k <key> http://twitter.com/marcelduran
webpagetest status 140610_F0_T5K
webpagetest results 140610_F0_T5K
webpagetest test http://twitter.com/marcelduran --poll 5 --timeout 60

67.
CI Integration
https://github.com/marcelduran/webpagetest-api/wiki/Test-Specs
Match result to provided specs
{
"median": {
"firstView": {
"requests": 20,
"render": 400,
"loadTime": 3000,
"score_gzip": {
"min": 90
}
}
}
}

68.
CI Integration cont.
Uses Mocha with reporters suitable for integration with:
● Jenkins
● Travis-CI
● Drone.io
● Just about anything else you can imagine

69.
grunt-perfbudget
Created by Tim Kadlec (@tkadlec)
https://github.com/tkadlec/grunt-perfbudget

70.
Results logging
● As each test comes in, full result data is written to log files
○ Page Data (load time, Speed index, requests, custom metrics, etc)
○ Request Data (details for every individual request)
● Each test run or request is on it’s own log file line
● JSON-formatted
● Config information is included
○ Test URL
○ Run #
○ First/Repeat View
○ Test label
○ Location
○ Browser
○ Connectivity
○ Test ID
○ URL to test result

71.
Results logging - integration
● Splunk
● Logster
○ Logster -> StatsD = Arbitrary metrics trending automatically
● Track individual requests across all pages
○ ads JS performance
○ CDN response times
○ Effectiveness of image compression
○ so much more

72.
Benchmarks (Private Instances)
● Still pretty experimental
● Recurring tests run automatically
○ Configuration is a bit ugly but flexible
● Tests are scheduled, run and aggregated automatically
○ Top-level trended aggregate view
○ All-metrics trended aggregate view
○ Per-page trended view
○ Scatter Plot for a given run
○ Filmstrip comparisons

73.
Monitoring the System
● Hourly Agent Monitoring
● getLocations.php
● getTesters.php
● checkTesters.php
● Custom cron jobs/scripts

74.
bit.ly/usingwebpagetest

75.
Feedback Please!

76.
Thank You!