The team at Picnic Software giving a detailed walkthrough of their application architecture and development processes for a large Angular and .NET Event Sourcing application.
4. Disclaimer
The views expressed here are solely those of the
authors in their private capacity and do not in any
way represent the views of the Picnic Software Pty
Ltd, or any other associated entity or shareholder.
Picnic Software Pty Ltd has not approved, endorsed,
embraced, friended, liked, tweeted, google-plused,
pinterested, dugg, reddited, hacker-newsed,
sanctioned or authorized this presentation.
5.
6. Agenda
• App & Tech
• Infrastructure & Data Flow
• Deployment & Scalability
• Permissions with Neo4j
• Client Side Technologies
• Development & Testing Workflow
7. What we do
• We are an ISV (Independent Software Vendor)
– Building and running a workflow/collaboration
application
• Partnerships with large businesses in the
Advertising/Marketing sector
• Our customers are primarily large retailers
8. Our App
• Media Library
– High resolution files; PSDs, Video
• Collaborative workflows
– Coordinating inputs
• Photography, Illustrations, Graphic Design
– Producing advertising outputs
• Over multiple media channels
– Catalogues, Billboards/Print, TV, Radio, Web
10. Flexibility
• Architecture choices to support
– Changes in requirements
– Future customers working in same domain
• When we started building we had
– Known customer workflows
– Known unknown customer workflows
– Unknown unknown customers workflows
28. Deployment
• Server Infrastructure – AWS
– Starting new instances largely a manual affair at
this point
• Configuration Management – Chef
• Application deployment – Team City, Octopus
Deploy
29. Chef
“Give me six hours to chop down a tree and I will
spend the first four sharpening the axe.”
30. Chef
• Windows / Linux deployment nodes
• Each node is in an Environment
• Each node performs one or more Roles
• Each Role requires the running of one or more
Recipes
• Recipes are stored in cookbooks
• Keep configuration in Git
• Keep CHEF Server configuration in Git
31.
32. Octopus Deploy
• Windows code deployments only – Linux coming
soon.
• Environments, Roles, Apps, Releases.
• Deployment Process – steps executed on Roles to
“Tentacles”
– Nuget packages retrieved from Team City
– Store configuration as variables
– Variable snapshot + Nuget packages = release
– IIS, Windows Service, PowerShell steps available
– Partial and Rolling deploys
– Easy to roll back – just re-deploy last working release.
33. SOON: Blue / Green Deployments
• Asgard brings up a NEW (GREEN) copy of
production AWS infrastructure.
• Automatically Bootstrap instances against Chef
and Octopus.
• Smoke test GREEN environment
• Add GREEN web servers to load balancer
• Remove OLD (BLUE) web servers from load
balancer
• Asgard tears down BLUE production AWS
infrastructure.
34. Message delivery between processes
• Requirements
– Reliable.
– Easy to manage.
– Easy to use.
– Low latency.
35. Things we looked at (2+ years ago)
• NServiceBus
– Tied to MSMQ at the time.
• MassTransit
– Lacked documentation.
– Not ready for prime-time at that point.
• RabbitMQ + EasyNetQ
– Simple, best fit for us.
– Wrote our own client – bad idea.
36.
37. RabbitMQ
• Written in Erlang, maintained by Pivotal.
• Linux / Windows.
• Easy administration (Web, command line, JSON).
• Supports
– Clustering and failover.
– Durable and HA queues.
– ‘At least once’ delivery guaranteed.
– Direct, Fan-out and Topic exchanges.
– Partitioning (vhosts), Federation & Shovelling.
38. How Picnic use RabbitMQ
• Setup
– Cluster of RabbitMQ servers behind ELB in
multiple AZs.
• You can use HAProxy instead of ELB.
– EasyNetQ library by Mike Hadlow.
• Handles subscription, publish and reconnection logic.
– So solid now we hardly think about it.
39. Use case: Scaling of long-running CPU
and IO intensive operations
• File format conversion, Zip bundling, PDF &
InDesign creation etc.
• Uses Topic exchange. Currently just one topic!
• Subscribers are round-robined by the broker.
• Subscribers are isolated – no clustering.
• Scaling - just launch new instances.
• Redundancy – launch in multiple Azs
• This has worked really well for us.
40. Use case: Distribution to SignalR
clients
• In-app notifications, long running task
progress etc. to the browser.
• Each web server receives all messages (Fan-
out exchange)
• Messages delivered to users / groups via
SignalR
50. Original Implementation
Write
Read
Me
ApprovalOwner
• RavenDB document for each permission
• Records which permissions directly inherit this permission
Read <footwear-folder-guid>:
[Write <footwear-folder-guid>]
Write <footwear-folder-guid>:
[Me <user-dave-guid>]
Owner <approval-1-guid>:
[Me <user-dave-guid>]
Me <user-dave-guid>:
[]
51. Original Implementation
Write
Read
Me
ApprovalOwner
• Worker task builds a second state document for each
permission
• Records all permissions which inherit this permission
Read <footwear-folder-guid>:
[Write <footwear-folder-guid>,
Me <user-dave-guid>]
Write <footwear-folder-guid>:
[Me <user-dave-guid>]
Owner <approval-1-guid>:
[Me <user-dave-guid>]
Me <user-dave-guid>:
[]
53. Original Implementation - Issues
• State documents can get large
• Introduce intermediate groups
• Takes time for state documents to be updated
• Cache and update permission graph in process
• Other processes can still sometimes see out-of-date permissions
• Use a graph database!
54.
55. New Implementation
• In process of switching to Neo4j
• Transactional updates
• No need to calculate intermediate state
• Faster
• Simpler
• Still need to send some state data across to RavenDB for permissions when
searching
63. Pull Requests
• Just over 2 months now using the PR based
workflow
• Approx 120 closed pull requests so far
• How
– Features branches, GitHub Tagging, TeamCity build
process comments against PR
– Asynchronous task for a team member
• “Here’s a PR please review when you can”
65. Development Workflow
• Why
– It was recommended to us.
– Supports consistent and frequent code reviews.
• Improves code quality
• Shares knowledge amongst the team
– Lets us catch some bugs much earlier.
66.
67. Development Workflow
• The wins
– Build server is more often in a green state.
• Can push to your PR branch to rely on CI to give feedback
– Knowledge sharing
• “Oh, that’s how you solved that”
• Reducing silo effects
– Offer constructive feedback to others
• “This could be made better by…”
– Bugs / issues caught
• Typos, debug code left in, incomplete/missed features
68. Development Workflow
• Testing
– Each PR builds as if it was already merged
– Unit/Integration tests run against PR in TeamCity
– YouTrack bugs marked with build numbers to track
deployment
70. Development Workflow
• Agility
– Tracking feature changes as they evolve alongside
code
• As with all documentation - trying our best to keep up
to date
• PRs feedback can “code change not reflected in docs”
– Testing team
• Can review these changes and be more up to date with
• Review PRs for an idea of scope of changes and where
to look for issues