JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway

Jboss Fuse in
Vodafone’s Global
Integration Gateway
5th December 2013 / Bruno Meseguer
Jboss Architect Meetup

Jboss Architect Meetup | 05 December 20132
Contents of the presentation
• Who we are within Vodafone
• The performance challenge
• Selection process for the new technology
• Creation of a new framework
• Some final thoughts
Welcome !

Global Integration Gateway (GIG)
Vodafone Group Enablers

• Enablers team, responsible to enable capabilities in Vodafone Group
– Charging
– Messaging
– Customer Registry
 GIG
– … and more
• GIG (Global Integration Gateway) (100+ systems)
– Intelligent Gateway which allows flexible Partners/Vodafone integration.
– Local/Global Partners and Vodafone internal systems can access services provided by OpCo’s.
– Opco’s can access services provided by Partners and Vodafone internal systems.
Who we are and what we do

Vodafone’s Global Integration Gateway (GIG)
HTTP HTTP
Technical capabilities view
Authentication
Authorisation
Protocol
Bridging
Orchestration
Throttling
Routing
Header
Enrichment
Error Handling
Data
Transformation
Logging &
Reporting
Asynchronous
Very fast > 800 TPS

HTTP HTTP
Bridge
In
(north)
IBM DP
Bridge
Out
(south)
IBM DP
Orchestration
(Tibco BE)
Atomic
(Tibco BE)
Atomic
(Tibco BE)
Orchestration
Orchestration
Orchestration
Atomic
Atomic
Atomic
System layers view

• In GIG’s ecosystem there are currently 100+ systems
• GIG’s current traffic is 1000+ TPS
• GIG’s original design aimed to be capable of handling 10,000 TPS
– This principle has been maintained on every evolutionary step of the solution
– GIG scales horizontally: 10,000 doesn’t represent a barrier
• The latency that GIG introduces is <0.1 seconds on slowest use cases
• The average payload size is around 2KB
– As GIG typically exposes API’s, payloads are not significantly big.
Some GIG numbers

• 2008:
– Poor HTTP performance from TIBCO component
– Had to find a fast HTTP gateway
– IBM's DataPower was the answer > 800 TPS
• 2012:
– Vision to port systems to a cloud environment
– IBM's hardware not portable
– the quest to find a replacement for IBM's DataPower starts
A little bit of history

• To find a software based solution, current DataPowers not portable to the cloud.
– With very high levels of performance, ideally > 800 TPS
– Fully asynchronous socket handling
– EMS compatible
• Minimise impact on clients.
– Able to maintain the same port definitions as in DataPower
• Minimise migration efforts
– Quick adoption of the technology
– 70+ flows to be migrated
The goals to replace IBM’s DataPowers

The performance
challenge

The performance challenge
• A normal HTTP server retrieves resources or generates some content and quickly responds to the
client
• What happens when a system acts as a pass-through and needs to wait for backend responses?
• delays from the backend could be disastrous to the system

• The worker thread-pool gets exhausted, where are the workers?
Quickly running out of workers!

• Worker threads need to wait for the backend response to resume.
Threads busy… doing nothing!

• A waiting queue of requests rapidly grows
Requests keep coming!

• Clients get increasingly angry!
Unhappiness!

• The system is exposed:
– Delays from the backend could be disastrous to the system
– The workers thread pool gets exhausted
– Worker threads need to wait for the backend response to resume
– A waiting queue of requests rapidly grows
• Systems will need to be proved to be robust:
– Under high rates of client requests
– Possible delays from back-ends
• Systems need to be Asynchronous

Selection Process

Selection Process
• Direct competitor to DataPower
• Highly recommended by Group Architecture
– Pre-selected by Group Architecture to replace DataPower
• Can be used as a Hardware or Software based solution
• A promise of raw performance, up to 30000 TPS
Layer7… first candidate

Selection Process
• How can we stress a system capable of 30000 TPS without the need of a farm of HTTP clients?
• There is an old friend to us called Netty used back in 2008 (a beta library back then) to test
DataPower on the lab.
• Using Netty we created a powerful client and stub proved to handle 5000+ TPS per instance
• Netty is an NIO client/server framework (event driven) with HTTP capability:
– Non-blocking socket management
– threads are freed up when posting requests, they don't wait for responses
– difficult to exhaust the thread pool
– responses are taken care of by a second thread pool
– the socket and HTTP context are kept in memory awaiting for a response thread to pick them up and close
Lab preparation

Selection Process
Lab preparation
• To stress the bridge we bombard it with a constant TPS rate and delay the response by 5 seconds.

Selection Process
• Within the space of two weeks we realised Layer7 was not the right choice
– By the third day of testing we aborted the performance study for Layer7
• Layer7 proved not to fulfil our performance requirements
– delays at the backend proved to be deadly to Layer7
– it was not asynchronous
Layer7 results

Selection Process
• As Netty proved to be very efficient we approached Red Hat and asked to provide us with Netty
support
• Initially Red Hat recommends Fuse + Jetty (Netty equivalent)
– 1st round of performance test show the solution works (400+ TPS), and it is indeed asynchronous
• From our Netty benchmarks we knew there was more potentially to exploit
• Red Hat agrees to integrate Netty HTTP into their product (it already included the Netty TCP
component)
– Red Hat delivers a Netty HTTP concept for us to try out
– 2nd round of performance show an improvement by a factor of 2 at least
Red Hat comes into play

Selection Process
• We triggered parallel assessments for:
– JBoss7
– TIBCO BusinessEvents 5.0
– WSO2
More candidates…

Selection Process
Performance results

Selection Process
• Although only one product could get selected, all of them had an opportunity to identify
weaknesses in their product
• They all immediately put engineers at work to focus around those areas to be improved
• They all managed to boost up their engines:
– some more than others (x2, x3, x4)
– some adopted the asynchronous mechanisms (x100)
Everyone wins

Selection Process
• Fastest NIO product available (Netty enabled)
• Integration product (Camel)
• OSGI container
• Fully covering our migration requirements
• Java based solution
• Red Hat’s engineers: main open source committers
• Red Hat quality assurance
Red Hat’s JBOSS FUSE gets selected

Selection Process
• It was very critical to make to right choice
• We invested the right amount of time on each one of the steps
• We know we got it RIGHT!
No rushing !

Vodafone’s Framework

• Jboss Fuse offering an opportunity to improve DEV, Deployment & Support processes
– Traditionally our developments are based on frameworks
– Upgrades on frameworks require:
 from DEV a new build (per component)
 and from D&S another testing and deployment process
 … per component !
Jboss Fuse’s potential

Traditional build and deployment processes
Customisation
Vodafone’s
Framework
Build
Customisation
Vodafone’s
Framework
Deployable Unit
Deployable Unit
Source Code
Vodafone’s
Framework
Customisation
Customisation
Customisation
JVM
JVM

• JBoss Fuse container is OSGI based
– It encourages to think differently
– Perhaps Vodafone can exploit the benefits
 Could we change our processes to reduce efforts?
 Could these benefits help the various involved teams
 Development
 Deployment
 Support
OSGI and bundles

New processes: OSGI orientated
Build
Vodafone’s
Framework
Deployable Unit
Customisation
Deployable Unit
Source Code
Vodafone’s
Framework
Customisation
Customisation
Customisation
Customisation
Deployable Unit
JVM
Customisation
Customisation
Vodafone’s
Framework

• The Framework is decoupled from business components and sits on the OSGI container as a
reusable module
• Implemented using Camel route definitions
• It contains 90% of the execution logic of a flow
• The Framework is responsible for
– Standarising flow definitions
– Reduce complexity in customisations
– Include all common functionality
– Ensure stability, robustness and performance
Framework's main bundle

Framework's main bundle
common common common
Direct-VM JMS
common
Framework 1.0.0
1. Accept internal
invocations.
2. Throtteling.
3. Header
Enrichment.
4. System Logging.
5. Collect Reporting
Information.
6. Bridge to JMS.

• Business components (called Connectors) are configured to communicate with the framework
• Implemented using Camel route definitions
• Connectors are responsible for
– Listening to incoming requests
– Concentrate on the business logic
– Delegate execution control to the Framework
Connector’s bundle

Connector’s bundle
HTTP
custom
Direct-VM
out
Flow A
1. Open an HTTP
port.
2. Custom manipulation, typically
extraction of the Service ID, MSISDN,
and country code.
3. Invoke the Framework
route definition.

• Connectors are configured to communicate with the framework
• N connectors talk to the same single Framework module within the same OSGI container
• Upgrades on the framework do not require new deliveries from Development and do not require
redeployments
Framework and connectors interaction

Framework and connectors interaction

• Upgrades on the Framework do not cause an impact on Connectors
• When a Framework fix is delivered, Connectors are simple reconfigured
• Multiple Framework versions can coexist if necessary
Framework upgrades

Framework upgrades
Direct-VM
in
JMS
common
Framework 1.0.0
custom
HTTP Direct-VM
out
HTTP
custom
Direct-VM
out
Direct-VM
in
JMS
common
Framework 2.0.0
Flow A
Flow B

• An early performance round shows a degradation (when
compared to POC results) well within the acceptable
margin
– around 10-15% loss only to the raw potential available
– with 5 seconds delay at the back and 2K payload we obtain 700
TPS
 Note: The Lab environment used is SLOW !
 We think the Lab environment is around half the speed of a
modern environment.
• An official data gathering is pending to be collected
Framework's performance
0
100
200
300
400
500
600
700
800
PoC Framework
TPS

Final overview

Current status of the project
• Vodafone’s Framework
– Implemented internally at
– Nearly 100% of the functionality completed
• Migration of DataPower flows
– The migration is outsourced to our trusted vendor Bulgaria
– Sprint 6 out of 8 is on its way
– 70+ flows in total
• Deployment
– First migrated flows started to be deployed in the pre-production environment

How JBoss Fuse compares to DataPower?
• Of course, JBoss Fuse is software based instead of Hardware based
– this means it is “cloudable”
– this means we are free to choose the hardware to run on
– it means the hardware is much cheaper (this is specially important when scaling)
– it means everyone can run it on its laptop
• Performance results suggest a substantial gain in overall TPS rate (1000+ TPS per instance)
• We can exploit the potential of 64 bits environments
– DataPower runs on 32 bits unless money is invested to acquire the new 64 DataPower XI52s
• The community of open source based solutions is much bigger

Are there any other advantages?
• it is Java based
– the skills required are much more common
• It is developer friendly
– DataPower development is painful
• One single Framework for all (instead of several in DataPower)
• RED HAT Customer support represents a quality leap coming from IBM's
– fast responses to customer cases
– skilled agents at the other end of the line
– proactive and collaborative reaction
– open to customer's point of view
– a user friendly Customer support Portal !!
Absolutely !!

What about the disadvantages?
• It's new
– we don't have OSGI experience
– we don't know yet the pitfalls
• New Component
– Red Hat has included Netty HTTP to satisfy our requirements
– we may find some integration defects
• New Framework
– will need some time to mature
– surely problems will be found
– migrated components may differ in behaviour from DataPower

The good and the bad
• The good
– Plenty has been said on previous slides
– The raw performance of the Camel framework and the Netty HTTP component
– The incredible number of available alternatives on how to do a particular task
– Very active product, with constant addition of features
– Customer support is first class
• The bad
– Not intuitive with the mix of available frameworks
– The available IDE to graphically define flows is not mature enough
– Having many options to do one task is seen as a positive, but in some cases is a negative
– Steep learning curve for less experienced developers

Questions and answers

Thank you
Jboss Architect Meetup | 5th December 2013

JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway

Similar to JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway (20)

Recently uploaded

Recently uploaded (20)

JBoss Architect Meetup - December 2013 - JBoss Fuse in Vodafone’s Global Integration Gateway