This document provides background on a project to build an online poker environment for a Dutch company. Key details include:
- The project involved building a complete online poker game environment within 12 months using technologies like Java, C++, and MySQL.
- The application included components like a player portal, game server, lobby server, and database connected via RabbitMQ.
- Major challenges included the asynchronous nature of the game, random elements, maintaining game history, and supporting many concurrent players.
- The initial architecture had some problems and underwent changes over time to improve performance and integration of components like the lobby server.
3. Is to show that
Complex projects can be executed by
following Scrum
Adaptability to emerging architectural
needs is achievable
Adoption of Scrum mindset helps in
succeeding
Is to give some concrete examples
Overview
4. Project Background
Dutch company – Online lottery giant,
decided to enter Online Poker market
Scope: Building a complete offering of
online betting Poker game environment
(first version in 12 months)
Technology choices: Java, C++, GWT,Test
Scripting, Linux, MySQL, RabbitMQ, XUnit,
Cruisecontrol, Artifactory, Maven,
RedDwarf
5. Application: a bird’s eye view
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server
6. Poker Lobby – showing all table types, tournament types etc
9. Servers must be ...
Robust
Network ready
Support for ‘Statefulness’
Easy to interface with external systems
Highly threaded
Minimum supportable concurrent player base: ~20000
Logged in players don't always play but add load
Speed and Order of message exchange between Client
and Server are of essence
10. Crucial Challenges
Inherently asynchronous
Arbitrarily fired Timers affect software's behaviour
Players may decide to sit out and re-join anytime
Network connection with player can drop anytime: game
must continue
Randomness must be satisfactorily built in and proven
(legal requirement)
Game History must be retained (legal requirement)
Multiple simultaneous tables, Multiple simultaneous
tournaments
11. Example of asynchronous behaviour
P1
P2P4
P3
step1. P2 plays
step2. P3 sits-out /
gets disconnected
NEW HAND!
step3. Hand over!
ASYNCHRONOUS
MESSAGE!
Now P3 is
informed of
new Hand
allowed to re-
join
12. Tricky to test software which..
is inherently asynchronous
is supposed to support many users, while
maintaining performance
is asking testers to be good/bad Poker players
is affected indeterminately by Timers (arbitrariness)
requires preservation of history of every move
is expected to display minimum latency always
13. `
Infrastructure - revisited
Player Portal
Game Server
Lobby Server
Database
RabbitMQ
Tournament
Server
.
.
.
.
.
.
.
.
...
.
BOTs
Human Players
JMX Console
14. Some initial Architectural
decisions
Game logic implemented as a handcrafted FSM
Game Server separate from Tournament Server
All components inside a Game/Tournament Server
communicate through messages
{ Game | Tournament | Lobby } server exchange
information through Database
Clients ‘pull’ information from Lobby Server
15. Poker Lobby – showing all table types, tournament types etc
16. Problems popped up: Pig headed
or headless chicken
Understanding of tournament server was clearer over
time
Performance bottleneck:
Identified earlier that lobby would be heavily loaded
Clients ‘pulling’ information from Lobby Server was a
bottleneck
Exchanging information through Database was
difficult
Plugging in 3rd party authentication mechanism was not
straightforward
17. Architectural adaptations led to
Replacing Game Server with Tournament Server
We realised that every game played can be considered to be a
part of ‘default’ tournament with no competition
Game/Tournament and Lobby Servers decoupling
through a Queue
Lobby Server keeps its own database
Lobby Server gets latest update from queue
Previous updates can be forgotten – saves memory and
processing
18. Some architectural changes made..
Replaced Game Server with Tournament Server
We realised that every game played can be considered to be a
part of ‘default’ tournament with no competition
Game/Tournament and Lobby Servers decoupled
through a Queue
Lobby Server keeps its own database
Lobby Server gets latest update from queue
Previous updates can be forgotten – saves memory and
processing
19. Building functionality
Significant focus on unit testing
Regular and frequent builds
By not having adequate automated tests, we paid the
price in
Incorrectly calculating how to divide money at
stake, amongst players
Incorrectly recording the HandHistory of games
(both of the above, legal requirements and hence,
extremely important)
Debugging effort and cost was huge
20. Our Scrum experience
reminds us that
Collaboration between product management,
developers, testers and architect is crucial
Changes from the product management could be
accommodated at any point of time
Architectural alterations can be made to the initial
architecture