1.
CASE STUDY
Leading Retailer Chooses
Microsoft Visual Studio for
Load Testing Critical Web Services
Background
RTTS’ client operated numerous heterogeneous customer-facing
(i.e. revenue producing) web applications, as well as administrative
and customer support applications, that relied on a common set of
services, including customer account retrieval, delivery date
availability, and address validation. Using IBM’s Integration Bus
solution (formerly known as WebSphere Message Broker), RTTS’
client was launching an enterprise server bus (ESB), so that these
individual applications (and future applications) could more easily
scale in terms of business growth.
Challenges
High availability and responsiveness were critical factors to the customer’s business model, since these
factors directly affected order conversion for the customer-facing applications, as well as mean time to
resolution regarding customer service inquiries. RTTS’s customer required a load testing solution that
possessed a minimal total cost of ownership (TCO) that could scale in proportion to its business growth and
the consumption of the web services provided by the ESB.
Industry
Retail
Solution & Services
Performance/Load Testing
Software Test Management
Test tool evaluation
Tools
 Microsoft Visual Studio Ultimate version

2.
Licensing expenditures and load generation infrastructure costs were paramount in being able to meet
budget constraints, while the ability to scale to the transaction rates and concurrency were imperative to the
application and infrastructure teams. The short-term load testing goal was to be able to support the demands
of at least one (1) of the customer’s primary brand’s e-retail web site using a single instance of the IBM
Integration Bus. Peak usage had been observed to be approximately 7,000 online customers.
Strategy
RTTS put forth a proposal that would assess the available load testing solutions. Proof-of-concepts were
arranged that addressed the gamut of open source load testing solutions and commercial load testing
solutions. The results of the proof-of-concepts were presented to the stakeholders in conjunction with the
projected costs of the solution in order to determine the best return on investment. RTTS also took
ownership of the rollout of the load testing solution by providing professional services that installed the
testing products and engineered the baseline performance tests.
Solution
Several load testing solutions were evaluated using the criteria set forth by RTTS’ customer. Open source,
commercial, and cloud-based load testing solutions were evaluated via proof-of-concepts against a subset of
the ESB’s suite of web services. Load test solutions that were provided by open source testing tools were
deemed a risk in terms of having product support. Cloud-based testing solutions turned out not to be a fit a
the time due to concerns regarding security, proprietary scripting languages, and the client’s current
corporate culture. However, RTTS’ client did express interest in having the flexibility to move to a cloud-
based solution, especially regarding its load generation infrastructure. In the end Microsoft Visual Studio
Ultimate was selected to be the best fit based upon its capabilities, familiarity, and cost.
The initial cost of Microsoft Visual Studio Ultimate was significantly less expensive when compared to some
of the more established load testing solutions in the market. Microsoft’s licensing model permitted unlimited
numbers of virtual users, so that testing could easily scale along with the growth of the customer’s business.
Along with the ability to systematically test the ESB using a familiar Visual Studio IDE, the ability to add
capacity to its load generation infrastructure was seamless. RTTS was able to incrementally add to the
customer’s load generation infrastructure by simply spinning up additional virtual machines and adding them

3.
to the current Visual Studio Load Test Rig. If and when the need arises, these tests could be ported to the
Windows Azure cloud platform for greater flexibility for implementing its load generation infrastructure.
Benefits
RTTS successfully installed and configured the necessary Microsoft Visual Studio components and was able to
jumpstart the load testing initiative by testing a subset of the ESB web services related to product availability,
delivery dates, and shipping options for one (1) of the customer’s family of brands. Using a Visual Studio Load
Test Rig of six (6) agents, load tests reaching 12,000 concurrent users were executed. Microsoft Visual
Studio’s load testing capabilities were able to assist in discovering bottlenecks related to the threading model
being used by the ESB’s database connection pool, as well as within the delivered HTTP configuration of the
ESB’s web interface. Once the environment was optimized, 90% of all web service requests were returned in
under one (1) second. The IBM Integration Bus was shown to support the processing of 55 web service
requests per second with its CPU utilization averaging about 60%.
At the conclusion o the project RTTS was tasked with leading the testing effort by augmenting the collection
of web services being tested, incorporating additional retail brands and back-office applications, and
subsequently increasing the end-user consumption of the web services.
RTTS is a Microsoft Gold Partner. Since 2006, RTTS has been an education & consulting partner with
Microsoft – implementing solutions and providing educational services around Microsoft's Visual Studio
testing tools. RTTS has worked with the Visual Studio team to develop top-notch solutions around their
excellent toolset. For more information about our solutions and services, go to www.rttsweb.com
copyright Real-Time Technology Solutions, Inc. 2014