New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
eCommerce performance, what is it costing you and what can you do about it?
1. eCommerce Performance
what is it costing you, and what can
you do about it?
Peter Holditch
Technologist
pholditch@appdynamics.com
2. The Business Impact of One Second
“One second increase in
Amazon‟s page load
would annually cost $1.6
billion in sales”
Borland Research - March 2013
3. Because a 1 second delay equates to…
3
11% fewer page views
A 16% decrease in customer satisfaction
A 7% loss in conversions
4. Google and Microsoft research
• Experiments to introduce delay into web
searches to measure the impact
4
http://velocityconf.com/velocity2009/public/schedule/detail/8523
http://vimeo.com/5310021
5. Server Delays Experiment: Results
• Strong negative impacts
• Roughly linear changes with increasing delay
• Time to Click changed by roughly double the delay
DistinctQueries/UserQuery
RefinementRevenue/User
AnyClicks
Satisfaction
TimetoClick
(increaseinms)
50ms - - - - - -
200ms - - - -0.3% -0.4% 500
500ms - -0.6% -1.2% -1.0% -0.9% 1200
1000ms -0.7% -0.9% -2.8% -1.9% -1.6% 1900
2000ms -1.8% -2.1% -4.3% -4.4% -3.8% 3100
- Means no statistically significant change
6. Impact measured by
• Slower performance abandoned searches
• More active users more sensitive to this
• Effect got worse over time, and persisted once
performance was restored
6
dailysearchesperuserrelativetocontrol
wk1 wk2 wk3 wk4 wk5 wk6
-1%-0.8%-0.6%-0.4%-0.2%0%0.2%
200 ms delay
400 ms delay
actual
trend
Impact of Post-header Delays Over Time
dailysearchesperuserrelativetocontrol
wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11
-1%-0.8%-0.6%-0.4%-0.2%0%0.2%
delay
removed
Persistent Impact of Post-header Delay
200 ms delay
400 ms delay
actual
trend
7. Conclusion
• Revenue is a function of user behaviour
• User behaviour is quite sensitive to
performance
• Effects of poor performance outlast the
problems
• It is necessary to have a constant watch on
performance of critical transactions, fix
problems quickly and continuously improve
over time
7
8. BIG DATA
Hadoop
Cassandra
MongoDB
Coherence
Memcached
CLOUD
Amazon EC2
Windows Azure
VMWare
This is made very hard by the modern technology landscape
DistributedMonolithic
Login
Search Flight
View Flight Status
Make Reservation
Weblogic
Oracle
.NET
MQ
ATG, Vignette,
Sharepoint
SQL
Server
JBoss
Tomcat
Tomcat
Mule, Tibco, AG
ESB
.NET
Tomcat
SOA
WEB 2.0
Browser Logic
AJAX
Web Frameworks
Release 3.4
Release 3.5
Release 3.6
Release 4.0
AGILE
Release 1.1
Release 1.2
Release 1.23
Release 1.5
Release 4.4
Release 4.5
Release 4.6
Release 5.0
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
8
9. BIG DATA
Hadoop
Cassandra
MongoDB
Coherence
Memcached
CLOUD
Amazon EC2
Windows Azure
VMWare
Where and what is the problem?
Weblogic
Oracle
.NET
MQ
ATG, Vignette,
Sharepoint
SQL
Server
JBoss
Tomcat
Tomcat
Mule, Tibco, AG
ESB
.NET
Tomcat
SOA
WEB 2.0
Browser Logic
AJAX
Web Frameworks
Release 3.4
Release 3.5
Release 3.6
Release 4.0
AGILE
Release 1.1
Release 1.2
Release 1.23
Release 1.5
Release 4.4
Release 4.5
Release 4.6
Release 5.0
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
9
Login
Search Flight
View Flight Status
Make Reservation
10. BIG DATA
Hadoop
Cassandra
MongoDB
Coherence
Memcached
CLOUD
Amazon EC2
Windows Azure
VMWare
Where and what is the problem?
Weblogic
Oracle
.NET
MQ
ATG, Vignette,
Sharepoint
SQL
Server
JBoss
Tomcat
Tomcat
Mule, Tibco, AG
ESB
.NET
Tomcat
SOA
WEB 2.0
Browser Logic
AJAX
Web Frameworks
Release 3.4
Release 3.5
Release 3.6
Release 4.0
AGILE
Release 1.1
Release 1.2
Release 1.23
Release 1.5
Release 4.4
Release 4.5
Release 4.6
Release 5.0
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
10
Login
Search Flight
View Flight Status
Make Reservation
11. BIG DATA
Hadoop
Cassandra
MongoDB
Coherence
Memcached
CLOUD
Amazon EC2
Windows Azure
VMWare
What if the problem is outside the application?
Weblogic
Oracle
.NET
MQ
ATG, Vignette,
Sharepoint
SQL
Server
JBoss
Tomcat
Tomcat
Mule, Tibco, AG
ESB
.NET
Tomcat
SOA
11
Login
Search Flight
View Flight Status
Make Reservation
WEB 2.0
Browser Logic
AJAX
Web Frameworks
Release 3.4
Release 3.5
Release 3.6
Release 4.0
AGILE
Release 1.1
Release 1.2
Release 1.23
Release 1.5
Release 4.4
Release 4.5
Release 4.6
Release 5.0
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
12. Real-User Monitoring gets Real Results*
12
>10% decrease in
end-user complaints
>30% increase in
App Availability
>91% transaction
completion
End-users
„completely satisfied‟
BusinessesdoingRealUser
Monitoring
BusinessesNOTdoingRealUser
Monitoring
*Source:AberdeenGroup,July2012
13. BIG DATA
Hadoop
Cassandra
MongoDB
Coherence
Memcached
CLOUD
Amazon EC2
Windows Azure
VMWare
And beyond performance monitoring…
Weblogic
Oracle
.NET
MQ
ATG, Vignette,
Sharepoint
SQL
Server
JBoss
Tomcat
Tomcat
Mule, Tibco, AG
ESB
.NET
Tomcat
SOA
WEB 2.0
Browser Logic
AJAX
Web Frameworks
Release 3.4
Release 3.5
Release 3.6
Release 4.0
AGILE
Release 1.1
Release 1.2
Release 1.23
Release 1.5
Release 4.4
Release 4.5
Release 4.6
Release 5.0
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Release 1.4
Release 1.5
Release 1.6
Release 2.0
13
Login
Search Flight
View Flight Status
Make Reservation
14. Case Study – One Year
Dev QA Ops Business
ProductionPre-Production
• Agile Releases 12 > 18
• Spent 3,060 hours less firefighting
• Delivered More Innovation
• Identify & Fix Defect 20 hours > 13 hours
• Spent 4,024 hours less testing
• Faster Time to Market
• Availability 99.91% > 99.95%
• MTTR 40 hours > 22 hours
• 1,528 hours less troubleshooting
• End User Experience 500ms > 150ms
• $167,475 lost revenue savings
• $627,691 productivity savings
• $795,166 Total savings
14
A study by Borland identified an overwhelming correlation between sales-generated traffic rises and increases in website response times – a nightmare situation for any retailer hoping to capitalize on the seasonal online rush of bargain-hunting consumers. Research has shown that even minor delays to website response times can have a sizable impact on customer satisfaction, page views, conversion rates and site abandonment. A one second delay in website response time equals11% fewer page views,16% decrease in customer satisfaction and a 7% loss in conversions.The study thus concludes that a one second increase in Amazon’s page load would annually cost $1.6 billion in sales, and 38% of UK online shoppers abandon websites or apps that take more than 10 seconds to load.The average online shopper expects web pages to load in 2 seconds or less, after 3 seconds, up to 40% will abandon the site. Seventy four per cent of users will abandon a mobile site after waiting only five seconds for it to load.Once visitors leave, it’s very difficult to get them back. 88% of online consumers are less likely to return to a site after a bad experience.Play.com, the UK arm of the Rakuten Group, saw performance drop by 500% as its site slowed from a load time of 2 seconds to 12 when site traffic peaked on the 4th January. Other online retailers that also suffered significant increases in load times during the first few days of the January sales included John Lewis, Amazon.co.uk, Asos.com and Tesco.com. Increases ranged between 3 and 4.5 seconds for their landing page to load.“There is lots of data available showing that users are losing patience with poor performing websites,” said Archie Roboostoff, product director at Borland. “It looks like a number of the sites monitored over the seasonal period will have missed out on potential revenue as a result of their website’s inability to process high levels of traffic. The sites we monitored in the UK had normal load times averaging 2.9 seconds, but saw load times increase by an average of 4.5 seconds during peak traffic periods – a 55% deterioration.Developing a robust performance strategy takes time, and peak period preparation should begin early with testing starting about six months beforehand. Putting in this groundwork is crucial if retailers are to take full advantage of peak shopping times throughout the year.”http://www.retail-digital.com/retail_technology/one-second-delay-on-amazon-16-billion-loss-a-year[source data: http://www.aberdeen.com/aberdeen-library/5136/RA-performance-web-application.aspx]
The application landscape is complex, and so is the transaction landscapeSome transactions will be more important to track than others – with conventional monitoring it’s impossible to focus on the important things, and impossible to understand if monitoring anomalies have any business impactMoreover, it’s impossible to troubleshoot the important things – just
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Objective of SlideHighlight our value proposition across Development, QA, Operations and the business.ScriptFor example, here’s a customer case study from Edmunds.com which highlights the annual benefits of AppDynamics across their organization and lifecycle.Development was able to double their innovation as a result of spending less time firefighting, and implementing more business requirements.QA were able to detect performance defects twice as fast, therefore increasing testing productivity and accelerating time to market.Operations increased application availability by .04%, and cut MTTR in half which had a significant impact on the business.All these benefits translated an enhanced end user experience combined with significant lost revenue and productivity annual savings totaling almost $800,000.Bank of New Zealand, Expedia and Fox News also had similar savings to Edmunds.com.