Weitere ähnliche Inhalte Ähnlich wie Simple Log Analysis and Trending (20) Mehr von Mike Brittain (14) Kürzlich hochgeladen (20) Simple Log Analysis and Trending1. Simple Log Analysis and Trending
Mike Brittain
DIRECTOR OF ENGINEERING, ETSY
@mikebrittain
EXAMPLES: bit.ly/velocity-etsy
2. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no
A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
4. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no
A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
6. A B
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_state
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_s
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.fa
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorit
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gat
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lis
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listi
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Moz
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_l
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communit
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
8. Plotting log data for ad hoc analysis and
real-time trending
EXAMPLES: bit.ly/velocity-etsy
10. Access log format
Time, remote address, http method, request uri, referrer, user-agent,
response size, response code, execution time, memory consumed,
plus custom fields...
• Signed-in/out (user_id vs. “-”)
• display mode (“desktop” vs. “mobile”)
• i10n/i18n (“en-US”)
• A/B experiment buckets
12. apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...
LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
13. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
15. Ad hoc analysis using the newest,
hottest, bleeding-edge plotting tool.
18. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
19. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
20. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
| sed "s/[//"
21. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
| sed "s/[//" > response_times.dat
# response_times.dat
11/May/2011:17:08:53 505.780
11/May/2011:17:08:54 311.455
11/May/2011:17:08:53 376.745
11/May/2011:17:08:54 322.278
22. # response_times.dat
11/May/2011:17:08:53 505.780
11/May/2011:17:08:54 311.455
11/May/2011:17:08:53 376.745
11/May/2011:17:08:54 322.278
# scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#f28705"
24. #!/usr/local/bin/gnuplot
set terminal pngcairo enhanced font "arial,11" size 800,600
set output 'response_times.png'
# Graph formatting
set title "Listings, generation times (ms)"
set ylabel "millisec"
set key off
set yrange [0:2000]
# Setup time formatting
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot
'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#f28705"
26. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ exercise_for_reader.sh
# response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416
27. # response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416
# trends.gp
plot 'response_median_perc95.dat'
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile",
"" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median",
800 title "SLA" linewidth 2 linecolor rgb "#f28705"
28. # trends.gp
plot 'response_median_perc95.dat'
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile",
"" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median",
800 title "SLA" linewidth 2 linecolor rgb "#f28705"
29. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ php data_buckets.php <filename> <bucket-size>
30. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ php data_buckets.php access.log 50 > response_histogram.dat
# response_histogram.dat
# bucket (ms), count
0 0
50 0
100 12
150 412
200 1203
250 2312
300 2078
31. #!/usr/local/bin/gnuplot
set terminal pngcairo enhanced font "arial,11" size 800,600
set output 'histogram.png'
# Graph formatting
set title "Listings response times (ms)"
set key off
set yrange [0:*]
# Histogram rendering
set style data histograms
set style histogram clustered gap 1
set style fill solid 1.00 border lt -1
set boxwidth 0.9 absolute
plot 'response_histogram.dat' using 2:xtic(1)
33. Warning for Mac / homebrew users
The pngcairo terminal does not work with pango 1.30 (latest).
Use 1.28 or 1.29 instead.
set terminal pngcairo enhanced font "arial,11" size 800,600
“Gnuplot pdfcairo problem”
https://github.com/mxcl/homebrew/issues/11901
35. Logster
Run by cron, maintaining a cursor on log files
Simple parsing & aggregation
Output to Ganglia, Graphite, stdout
Combined log, via syslog
1 minute schedule
Output to Graphite github.com/etsy/logster
36. Report script execution time (ms) segmented by signed-in/out visitors?
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-
gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038
web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-
diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/
72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU like
Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/
419.3" mobile 8513282 15249296 563978 586038
37. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
# Parse fields from log file
self.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+)
(?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)
38. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
# Collect data from each matching line
# signed_out_ms and signed_in_ms are lists.
fields = regMatch.groupdict()
php_ms = int(fields['php_usec']) / 1000
user_id = fields['user_id']
if (user_id == "-"):
self.signed_out_ms.append(php_ms)
else:
self.signed_in_ms.append(php_ms)
39. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
return [
MetricObject("signed_in_reqs", len(self.signed_in_ms), "Requests"),
MetricObject("signed_in_ms_min", self.signed_in_ms[0], "ms"),
MetricObject("signed_in_ms_median", percentile(self.signed_in_ms, 0.5), "ms"),
MetricObject("signed_in_ms_perc95", percentile(self.signed_in_ms, 0.95), "ms"),
MetricObject("signed_in_ms_max", self.signed_in_ms[-1], "ms"),
]
40. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
signed_out_reqs 8
signed_out_ms_min 201
signed_out_ms_median 227.5
signed_out_ms_perc95 236.9
signed_out_ms_max 239
signed_in_reqs 3
signed_in_ms_min 258
signed_in_ms_median 268
signed_in_ms_perc95 268.0
signed_in_ms_max 268
41. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
$ logster --output graphite --graphite-host graphite.example.com:2003
PerfLogster access.log
42. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
$ logster --output graphite --graphite-host graphite.example.com:2003
PerfLogster access.log
# Put that into a crontab to run every minute, and...
43. 600
450
300
MEDIAN PAGE GENERATION TIMES (MS)
BY SIGNED-IN/SIGNED-OUT
150
0
44. Thanks!
Code samples and slides at bit.ly/velocity-etsy
github.com/etsy/logster
We are hiring... etsy.com/careers
(or just come talk to me afterward)