PushToTest implemented a multi-step business workflow on Oracle, IBM, and TIBCO platforms to compare developer productivity and application performance. TIBCO was found to require 29% less time and costs than Oracle and 22% less than IBM based on service construction, orchestration, and other metrics. The study also explored why platform choices impact costs, expertise needs, and ability to quickly build and deploy scalable applications.

1. SOA Knowledge Kit 2011
How To Evaluate Highly Scalable SOA Component Architecture
November 29, 2011
The Results Are In: Developer Productivity and Performance Comparison Analysis
PushToTest implemented a multi-step business workflow on Oracle, IBM, and TIBCO
platforms to surface developer productivity and application performance differences. The
results are ideal lessons and patterns for enterprise architects, development managers,
and business managers.
In this Workshop, PushToTest will present a developer's journal of lessons learned and
metrics to compare costs, as well as load and performance results. They will also explore
why platform choices count – covering the planning, expertise, and knowledge required
to quickly and flexibly build and deploy highly scalable applications.
Not only will they cover the results of the study and the open-sourced code and materials
to run the tests, they will also share the methodology used so you can apply this
knowledge and evaluate the results in your own environment.
Come to this Workshop to learn the results. Take the SOA Knowledge Kit back to your
organization to make the best decisions with the latest information.
http://soakit.pushtotest.com
fcohen@pushtotest.com

2. http://soakit.pushtotest.com
SOA Knowledge Kit
The Results Are In: Developer Productivity
and Performance Comparison Analysis
Frank Cohen, +01 (408) 871-0122, fcohen@pushtotest.com
November 29, 2011

3. http://soakit.pushtotest.com
About PushToTest
‣Our Mission: Test, To Make A More Reliable World
‣Make It Easier To Move From Manual To Automated Testing
‣Support Advanced Testing, including Agile, Load, Integration
‣Open Source Testing (OST) For Everyone
3

4. http://soakit.pushtotest.com
Results
• ActiveMatrix + BusinessWorks proven to require
less time and costs in side-by-side implementation
• 29% less time/costs vs. Oracle
• 22% less time/costs vs IBM
• Greatest productivity gains: $8,000
$7,074
Service Construction $6,469
$6,000
Service Orchestration $5,026
Policy Management $4,000
Security Configuration
Change Management $2,000
$0
Oracle
IBM
4
TIBCO

5. http://soakit.pushtotest.com
What We Need
‣Models versus Code/Annotations
‣Agile, Rapid, Performance
‣Significant Maintenance Costs Over Long Term
5

6. http://soakit.pushtotest.com
What We Get
‣Oracle
‣(especially of theout-of-the-box, required community support
Nothing worked
BEA components)
‣Tools focused on developers, not architects and business analysts
‣Fusion strategy gives more concise installations, still version problems
‣TIBCO
‣SOA Oriented Tools and Deployment
‣Tools are more robust, but still have some conflicts
‣Knowledge (Guides, Tutorials) need improvement
‣IBM
‣Several Big Platforms Integrated Together
‣Parts Don’t Play Well Together
6
‣Problems With Standards Adherence

7. http://soakit.pushtotest.com
Methodology
‣Define A Manufacturing Use Case
‣Implement The Use Case
‣Make Changes To The Implementation
‣Add HTTPS/SSL Security
‣Change Message Schema
‣Change To Asynchronous Message Delivery
‣Run a Functional and Performance Test
‣Total Cost of Ownership (TCO) Model
Implemented using 2 engineering teams.
Team A - TIBCO and Oracle
7 Team B - IBM

8. http://soakit.pushtotest.com
Download Now
‣http://soakit.pushtotest.com
‣Compares Developer Productivity between TIBCO, Oracle, IBM
‣Open Source Code Implementations
‣Developer Journals
‣Executive Slide Deck
‣Invitation to SOA Knowledge Kit Workshop - Free Webinar
‣Total Cost of Ownership Model
‣PushToTest TestMaker Platform
‣Free Open Source License
8

9. http://soakit.pushtotest.com
Next let’s talk about
Oracle
‣(especially of theout-of-the-box, required community support
Nothing worked
BEA components)
‣analysts on developers, not architects and business
Tools focused
9

10. http://soakit.pushtotest.com
Oracle Summary
‣Oracle 11g has problems not work out of the box
‣Delivered solutions may
‣external services and configurations required knowledge from
Special
blogs and forums
‣Most improved came from workarounds found by third parties
solutions
‣Oracle concise, still has versioningtools
the packaging of
‣Moresolution creation for engineersproblems
‣Quick dive is expected
‣Code adjustments to descriptors, classpaths and general
‣configuration
Manual
‣analysts tools are for developers - not for architects and business
Oracle's
‣Based on the non-standard JDeveloper IDE
10

11. http://soakit.pushtotest.com
Oracle Getting Started
‣From Idea Tothe tools for the task using only the web site was easy
Code
‣Identifying Guide straight forward and useful
‣Quickdays for installation
Start
‣Plan 3 size: 14 Gbyte, No download manager (register each time)
‣Download has problems installing different versions of the
‣application in the environment, not properly described in the
Oracle Suite
installation guide
‣ Tutorials Surface Problems
‣ Web Service creation requires Java coding experience: annotations
‣ Proprietary annotations like
oracle.webservices.annotations.PortableWebService
not found/included by the tool, workarounds needed
‣ The generated web services failed to generate the correct WSDL
11 • Another manual intervention to quick fix the issue was needed

12. http://soakit.pushtotest.com
Oracle Learning
‣For each of the services we found -severalfor simple solutions - that
tutorials
‣didn’tneeded towork workarounds even
We
actually
find
as they should have
‣chose tutorials on Web Service creation on the Oracle site. The one we
Many
failed:
‣ws.html#t5
http://st-curriculum.oracle.com/obe/jdev/obe11jdev/ps1/webservices/
‣SOAPUI web service mocks can not be called from Oracle
12

13. http://soakit.pushtotest.com
Oracle Learning
‣Found 4 tutorials on building Asynchronous Servers
‣asynch.htm#CBHECBFG
http://download.oracle.com/docs/cd/E17904_01/web.1111/e15184/
‣We used annotations that are not available on the app server
‣@AsyncWebService and @PortableWebService tags
‣import oracle.webservices.annotations.PortableWebService
‣import oracle.webservices.annotations.async.AsyncWebService
13

14. http://soakit.pushtotest.com
Next let’s talk about
IBM
‣Several Big Platforms Integrated Together
‣Parts Don’t Play Well Together
‣Problems With Standards Adherence
14

15. http://soakit.pushtotest.com
IBM Summary
‣Several Big Platforms Integrated Together Integration Developer,
‣Rational Application Developer, WebSphere Process Server
WebSphere Application Server, WebSphere
‣Each7.0 requires WASothers above 1Gbyte, RAD was 8 Gbytes
tool 600Mbyte+,
‣WID a day at Disneyland toand RADall the options WAS 8.0.3
7.0 8.0.3 requires
‣LikeDon’t Play Well Togetherexplore
‣Parts 7 Refactoring Problems (namespace, name, schema)
‣WID and process development happen in different tools (WID
‣and WPS)
Integration
‣RAD cannot develop business process, can’t be deployed to WAS
‣Problems With Standards Adherence
‣WID generated the corrupt WSDLs service
‣Could not create the Client for WPS
15

16. http://soakit.pushtotest.com
IBM Installation
‣Each tool 600Mbyte+, others above 1Gbyte, RAD was 8 Gbytes
‣WID 7.0 requires WAS 7.0 and RAD 8.0.3 requires WAS 8.0.3
‣Rational Application Developer
‣1.5 hours to install
‣Automatically starts the installation for Websphere Application Server
‣with Googlegosearch.failedthe “reconcile” property asFound and start
In the first it got
Set
and gave a plug-in error.
“true”
solution
the installation manager through command prompt. That again took
1.5 hrs.
‣Started the RAD, Encountered another error
‣JVM could not be started
‣SDP) “-Xmx1024m”, I changed it to “-Xmx896m”
VM parameters required: Eclipse config file (C:Program FilesIBM
16

17. http://soakit.pushtotest.com
IBM Construction
‣Can’t Get There from Here
‣WID generated the corrupt WSDLs
‣Could not create the Client for WPS service
‣RAD doesn’t have capability to develop business processes
‣Business process cannot be deployed in WAS
‣WID and WPS process development will happen in different tools
Integration and
‣WID V7.0
‣not updated interface, target Namespace, name and schema location
Refactoring
17

18. http://soakit.pushtotest.com
Next let’s talk about
TIBCO
‣SOA Oriented Tools and Deployment
‣Tools are more robust, but still have some conflicts
‣Knowledge (Guides, Tutorials) need improvement
18

19. http://soakit.pushtotest.com
TIBCO
‣SOA Oriented Tools and Deployment
‣abstractingare model oriented, keeping a betterThere is of code and
The tools
away the underlying technologies.
control
no need to
look at code, just models with SOA Concepts.
‣ Code oriented developers non-SOA has a steep learning curve
‣ The documentation does not help as it is more oriented to describing
the functionality rather than showing how to do things
‣ Tools are more robust, but still have some conflicts
‣ The modeling language is a proprietary one, reducing intuitiveness.
‣ Eclipse Based IDE that is familiar to many developers
‣ Missed Admin configuration for BW SE and deployed to Tomcat
‣ Engineers used to Message Driven Beans (MDB) approach should
be using JMS for Asynchronous Services
‣
19
BW WSDL modeling using a single field with complex data type failed

20. http://soakit.pushtotest.com
TIBCO Installation
‣Lack of Trial Software
‣installation instructionswas anot available there are no trials and the
Getting the tool suite
are
problem as
in the site.
‣order isisveryquick start guide to follow and it seems the installation
There no
important.
‣After registration, the tool suite download was quick
‣Even when the tools are split in 6 different modules.
‣Installation using default settings caused some problems
‣problemcoordination between toolscharacter (underscore)Although the
So the
was related to an invalid
is not totally correct.
that is
widely used.
20

21. http://soakit.pushtotest.com
TIBCO Installation
‣2.5 hours to learn where to begin
‣architects
Much commercial info at the site but no real guide for developers or
‣Knowledge Is There, Challenge Is To Find It
‣Documentation abundant and quite good, finding right guide is hard
‣The Eclipse base IDE is a plus in productivity for Eclipse users
‣Failed BW SE to create or select a TIBCO server for deployment
‣Asynchronous use case, references for using JMS, but no how
‣Mediation flow was required to implement asynchronous services
‣business_studio_process_developer_guide.pdf
http://developer.tibco.com/resources/business_studio/
• Helped us realized thatBusiness Studio IDE. Java web
services using TIBCO
developers must create
21

22. http://soakit.pushtotest.com
TCO Model
TCO Summary TIBCO Oracle IBM
System Setup $609 $844 $750
Design/Review $1,125 $1,125 $0
Learning $1,148 $1,688 $750
Support $0 $0 $375
Service Construction - Normal Serv $375 $234 $750
Service Construction - Async Serv $784 $933 $750
Service Orchestration $281 $891 $938
Deployment $188 $609 $375
Change - Add Security $94 $422 $609
Change - Transport to JMS $188 $188 $516
Change - Complex XSD $234 $141 $281
$5,026 $7,074 $6,469
TIBCO vs Oracle 29%
TIBCO vs IBM 22%
22

23. http://soakit.pushtotest.com
Where To Go From Here
‣Download the SOA Knowledge Kit
‣http://soakit.pushtotest.com
‣Attend The Open Source Test Workshop for CIOs, CTOs
‣http://workshop.pushtotest.com
‣Watch A Screencast and Tutorial
‣http://www.pushtotest.com/screencastcentral
‣ActiveMatrix Service Grid, ActiveSpaces screencast
• http://www.pushtotest.com/soa-deployment-enterprise-applications
‣PushToTest Services Available
‣+01 408 871 0122 (USA), sales@pushtotest.com
23