2. 2
Me
Who I am
French, please bear with my accent…
What I was
A consultant in system integration and Service Oriented
Architecture
What I am
Since 2013, I am part of the E-Commerce Architecture
group. I am currently working on integrating the Rakuten
Group's e-commerce systems.
3. 3
The history of Rakuten Ichiba globalisation
1997: Opening of Japan Ichiba
2008: Opening of Taiwan Ichiba
2010-: Rakuten acquired several e-commerce companies
around the world
2011: Japanese merchants can sell their products
overseas (outside of Japan)
2012-: Rakuten started to open new E-commerce
websites in new countries on a regular basis
2013-: Rakuten started to consolidate the group E-commerce
systems
5. 5
From a Japan-only platform to a global platform
The challenges
Part 1 – To a unified platform
Part 2 – To a multi-language system
Part 3 – To an expendable and extensible system
Part 4 – To a global ‘one team’
6. From a Japan-only platform to a global
platform
Part 1 – To a unified platform
Part 2 – To a multi-language system
Part 3 – To an expendable and extensible system
Part 4 – To a global ‘one team’
7. 7
The idea seems great but
1. Are the features even compatible from one
system to another?
2. How well do we know our system?
3. How can we inter-connect systems?
8. 8
Borders between marketplace specific and global features
• Each marketplace does things differently
• Where does global start/end?
=> Our product management team manages
case by case
=> Build a feature gaps backlog
Image source: http://commons.wikimedia.org/wiki/File:Information1.svg
9. 9
Know yourself first
• We have system metrics, but are they
meaningful?
• How can we compare 2 systems that use
different technologies?
=> Go back to user stories
10. 10
And our user stories look like
Shopper Merchant
Buy a product
Cancel an order
Register as a member
…
Update product price
Update order status
…
Mall UI Merchant UI
Ichiba Backend
11. 11
Inter-connecting systems, but how?
• In complex environments, integration often
requires an additional layer (ESB, EAI, …)
• It is quite nice, but we want to keep it simple
and extensible
=> We chose to use HTTP based “APIs” in
our system design
• ESB: Enterprise Service Bus
• EAI: Enterprise Application Integration
12. 12
Let’s go back in time…
UI Application External System UI Application
Data
source
Data
source
13. 13
UI Application External System
Data API
Data
source
Data API
Data
source
UI Application
And fast forward
14. 14
UI Application
And fast forward
UI Application External System
Data
source
Service API
Data API
Data API
Data
source
Service API
Caching
Quality of Service
…
15. 15
API
• Did you say API?
• How do we use them in our systems
– They provide data to the presentation layer
– Some level of abstraction is good!
• How do they help integrating systems?
– Simplify the technical integration
16. 16
We can go from
System
For marketplace A
System
For marketplace B
???
17. 17
To the following
System A System B ?
Item API
Campaign API
Order API
Mall UI
Item API
Billing system
?
?
18. 18
Test: what, when, how?
• Integration tests, but what to test and how?
=> 60% to 75 % of our UIs tests are
automated
=> Elaborate end-to-end scenarios
19. From a Japan-only platform to a global
platform
Part 1 – To a unified platform
Part 2 – To a multi-language system
Part 3 – To an expendable and extensible system
Part 4 – To a global ‘one team’
20. 20
Global Shopping
• We want to help our merchants to sell their
products in other markets
=> Therefore, we offer the ability to
merchants to translate their items to different
languages
Fact: 97% of our translations are automatic!
21. 21
Translation IS rocket science
• No translation engine is perfect so we are using
a mix of several translation engines
• Each of those engine has its specificities that we
have to learn
• We are also improving the quality of those
translations (e.g.: katakana processing)
22. 22
Our next target: images
2個セット
50%OFF
ポイント10倍
Buy 1, get 1 free
and x10 points
23. From a Japan-only platform to a global
platform
Part 1 – To a unified platform
Part 2 – To a multi-language system
Part 3 – To an expendable and extensible system
Part 4 – To a global ‘one team’
24. 24
Expandable = we can scale
• For our Global Platform, all our applications are
deployed in an R-PaaS Platform
• It abstracts the Operating System layer, as well
as the Virtual Machine layer and the physical
layer
25. 25
RPaaS modules
rpaas instances my_application +1
Application
Server
O.S. (Linux)
How does R-PaaS work?
Load balancer
Application
WAR, Ruby…
RPaaS modules
Application
Server
Application
WAR, Ruby…
O.S. (Linux)
26. 26
How does R-PaaS work?
RPaaS modules
Application
Server
O.S. (Linux)
Load balancer
Application
WAR, Ruby…
RPaaS modules
Application
Server
Application
WAR, Ruby…
O.S. (Linux)
RPaaS modules
Application
Server
Application
WAR, Ruby…
O.S. (Linux)
27. 27
Serving users from all over the world
• We use Akamai for caching and serving
customers all over the world
• Because our system is based on HTTP calls, we
can dispatch those components in different
data centres
• Our short term goal is to be able to do the same
with our data storage layer
=> If the application supports it, scale out
28. 28
Extensible = we can add new features easily
• As the system grows, how can we add new
features?
• How do I know that my new feature does not
break anything?
• How do I develop features that make a
difference to the business?
=> We use event driven processing
29. 29
We have users and we have events
Shoppers Merchants
Buy a product
Cancel an order
Register as a member
…
Update product price
Update order status
…
Mall UI Merchant UI
Ichiba Backend An order has been created
An order status has changed
A product has been updated
A new user has registered
…
30. 30
So we have a database and we have an event bus
Shoppers Merchants
Buy a product
Cancel an order
Register as a member
…
Update product price
Update order status
…
Mall UI Merchant UI
Ichiba Backend
An order has been created
An order status has changed
A product has been updated
A new user has registered
…
Update product
Update membership
Update order
…
31. 31
Events are meaningful, use them wisely
• Search is expensive, generating / consuming
events is not
• For example, we use this pattern for:
– invalidating cache
– sending out notifications
– rebuilding our search engine index
• We can go further and bring new capabilities to
our businesses
32. From a Japan-only platform to a global
platform
Part 1 – To a unified platform
Part 2 – To a multi-language system
Part 3 – To an expendable and extensible system
Part 4 – To a global ‘one team’
33. 33
Language is only one problem
In a global environment:
• How do we assign tasks?
• How do we share progress?
• How do we make decisions?
=> Collaborative tools do help but mostly,
we have to educate ourselves
34. 34
About the usefulness of a glossary…
• Like I mentioned in part 1, each team knows
their own system
• The problem is, that when you have similar
systems, you tend to make easy shortcuts
• Those shortcuts create a lot of
misunderstanding
=> Create a glossary
35. 35
About the usefulness of a glossary…
Concept Meaning
Glossary A table like this one
Item An item is defined by a
merchant […]
Product A product is an item sold on
a marketplace […]
=> Create a glossary
37. 37
Today I suggested to
Describe your system with user stories
Automate your tests
Offer automatic translation to merchants
and shoppers
Use APIs to make system integration easier
Design your application for scaling out
Use the event driven paradigm
Learn how to work in a global environment
Build a glossary
40. 40
Bonus slide – how do we track progress when working
remotely
• Atlassian on-demand and Atlassian on premise
• Jira for issue, features tracking
• Confluence for information sharing
Editor's Notes
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues:
Of course, from all marketplace perspective, the way they do business is different. However, if we want to provide a global platform, we have to make some concession.
We try to make our global platform customisable by a marketplace but there are limits of what we can do.
One of the big challenges of data migration is that the data topology can be different: one country could have an item with 20 images associated to it whereas another only 1. What should we chose?
On this topic, we see it case by case, there is no generic way for handling this issue
Systems tend to grow complex over time. One issue that systematically comes with all the integration works we have done was the following:
"We know our system can support our needs, prove that yours can". This is a very tricky question. Of course, when migrating from one system to another, the legacy system could handle what it was handling. But a new system will have a completely different topology, architectures and concepts.
The way we are trying to tackle this is to try to come to a common ground, as much as possible. If we look at a complex website, knowing only "how many users come and visit the website" is far from being enough. We need to translate
Those numbers that are very contextual to a more abstract way.
Some systems cache a lot, some systems do pre-rendering, some systems integrate in a different way with a search engine ...
Legacy systems = a big box that has a lot of dependencies and that works "as it is"
Our solution for that has been to promote the use of APIs between systems.
We usually split our systems into the following:
UI for shoppers
UI for merchants
UI for Business Units
Open APIs that are accessed by third parties (merchants, partner companies, …)
API layer, across the above layers
=> Our basic rule is to centralize in the API layer what can be re-used in any of the above layers. We want to code once and for all.
=> By standardizing the way a system communicates with the outside, we can reduce the effort for replacing a system by another.
=> By abstracting the data layer by an API layer, we can freely change the backend without impacting the UI components (add a caching layer, replace an RDBMS by a NoSQL solution, etc…).
=> In my experience, it is almost impossible to do a big-bang approach of integrating "System A" with "System B". By separating the API layer, that offers us the ability to look at one functional area after the other. We can say "First, let's integrate the order API of System A with the order system of System B".
What we do: 60% to 75 % of UIs automated tests
Then the rest of the standard package is tested manually
Then, the new features that come into the platform, we test those separately
In parallel, we run load and stress tests
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues:
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues:
Please note that this diagram is a temporary diagram
Basically, RPaaS is based on Cloud Foundry and on a top of a Virtualization layer.
It provides the application with the ability of easily deploying and scaling.
It also provides environment variables so we can deploy the same artifact (WAR file, ruby runtime, php modules etc…) in all the environments without having to modify them or building custom scripts.
It also supports deploying a new application without any service stop. We can deploy a new application then map it to the endpoint to which external services should call.
It is basically HTTP based traffic so that the UI calls API using JSON APIs over HTTP, then the APIs would connect to a database via JDBC or other protocol.
Please note that this diagram is a temporary diagram
Basically, RPaaS is based on Cloud Foundry and on a top of a Virtualization layer.
It provides the application with the ability of easily deploying and scaling.
It also provides environment variables so we can deploy the same artifact (WAR file, ruby runtime, php modules etc…) in all the environments without having to modify them or building custom scripts.
It also supports deploying a new application without any service stop. We can deploy a new application then map it to the endpoint to which external services should call.
It is basically HTTP based traffic so that the UI calls API using JSON APIs over HTTP, then the APIs would connect to a database via JDBC or other protocol.
Is it okay to say that we have a partnership with Akamai?
Regarding the data storage layer, we can of course easily split the data by marketplace. But then what happens when a merchant in Indonesia wants to sell an item in Malaysia?
We could then split the data by region. But then, what happens when a merchant in Asia wants to sell an item in Europe? How do we keep the consistency of the item and inventory information?
To address those challenges, we are building a data storage that is not based on relational databases because, unfortunately, those kind of features are missing from traditional relational databases
During our design phase, we realised that our platform would be communicating with a lot of external systems (dataware house, internal search engines, …)
Usually, from an E-Commerce system perspective, external systems are mostly interested in changes that occur in our platform.
Those changes can be of any type: a merchant registers a new item, an item runs out of stock, an order is cancelled, and so on.
However, having a lot of systems polling for changes is very expensive system wise.
That is why we are doing event driven mechanisms for those cases. A system will register on a message bus saying “I am interested in being notified when an order changes”.
We can then use this mechanism, without any change in the original source for: invalidating cache, sending emails, rebuilding our search engine indexes and so on and so forth
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues:
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues:
With all those expansions, we ended up building several E-Commerce systems. However, we are in the process of consolidating those systems.
When consolidating systems that have been around from quite some time, you usually get into the following issues: