Atlassian Connect add-ons are SaaS applications. Building and running them means planning with operations in mind: where should you host your add-on? What's the best way to deploy it? How can you monitor it once it's live? How much will it cost to run? We'll draw from Atlassian's experiences building Who's Looking for OnDemand (a production Connect add-on installed in 750 instances with 15,000 active users) to explore tips and best practices to help answer these questions and more.

1. June 3-5, 2014 | Berlin, Germany

2. Robin Fernandes, Dev Team Lead, Atlassian
Building a Production-
Ready Connect Add-On

3. Who’s
Looking?
19500 users
950 hosts

4. Overview / Stack
× many!
× many!
Who’s Looking?
Multi-tenancy
Scalability
https://bitbucket.org/atlassian/whoslooking-connect

5. What Connect Provides
Tenant registration
Authentication & tenant context
Webhooks
Integrated look & feel
Local dev & test
…
Handled by Connect!

6. A Production-Ready Add-On
Hosting
Upgrading
Scaling
Monitoring
…
Lots of decisions…

7. Hosting

8. Stateless Request Processing
Load Balancer
Data Store
Hosting on a PaaS can encourage
statelessness, which enables
horizontal scalability and
improves resilience.
Requests
app
processes
(dynos)
whoslooking.herokuapp.com

9. Rich ecosystem of
external services
3rd Party Services

10. Basic deployment is just a git push
Low Barrier to Entry
> git push heroku master
Free tiers allow you to experiment at no cost

11. Testing

12. Testing
Unit testing: business as usual
Browser tests: similar to in-process add-ons
Run add-on and host app locally
Atlassian SDK, WebDriver, Atlassian Page Objects…
Add post-deployment verification tests
Reliable sanity tests, run after every deployment
Must be runnable against a production instance

13. Testing & Muti-tenancy
Mock out products’ REST APIs to test multi-tenant interactions
(see Who’s Looking source on Bitbucket for examples)
ac-mock-tenants.herokuapp.com
Connect
add-on

14. Load Testing
Load test on free tier, and extrapolate
Use e.g. New Relic to monitor the add-on under load

15. Load Testing
Tool: ac-load-test
Sets up Gatling-based load tests for Connect add-ons
Simulates authenticated requests from 1000s of hosts & users
Connect
add-on
ac-mock-tenants> ./sbt run
ac-load-test
(Gatling)
heavy load
bitbucket.org/atlassianlabs/ac-load-test.g8
Mock OD
instances
JWT-signed
requests

16. Load Testing
> ./sbt run
…
================================================================================
2014-05-17 17:36:22 20s elapsed
---- A mixture of iframe and XHR requests --------------------------------------
[--------------------------------------------------------------------------] 0%
waiting: 0 / running: 200 / done:0
---- Requests ------------------------------------------------------------------
> Global (OK=899 KO=0 )
> iframe-request (OK=200 KO=0 )
> token-based-xhr-request (OK=699 KO=0 )
================================================================================
…
bitbucket.org/atlassianlabs/ac-load-test.g8

17. Deployment

18. Staged Deployments
whoslooking-dev.
herokuapp.com
dev
run PDV tests
deploy

19. Staged Deployments
whoslooking-stg.
herokuapp.com
stg
run PDV tests
promote from
dev to stg
(using pipelines)
Green dev
deployment?
+ soak time

20. Staged Deployments
whoslooking.
herokuapp.com
run PDV tests
All good on stg?
prod
promote from
stg to prod
(using pipelines)

21. Further Precautions
the goggles…

22. Things can still go wrong!
Log management
Set up alerts on log patterns, search logs
Add-ons: Logentries, FlyData, Papertrail…
Feature flags
Turn new features on & off via configuration
Metrics / analytics
Get insight into application behaviour to detect issues
Libraries for in-memory metrics: Simon, Metrics, Servo…
Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…

23. Healthcheck & Monitoring
http://whoslooking.herokuapp.com/healthcheck
Implement a Health Check
REST resource that runs sanity-checks & reports results
Monitor it (Pingdom, New Relic…)
Example:
(Checks Postgres & Redis connections, reports config
settings, resource usage, weekly/daily active users/hosts)

24. Healthcheck & Monitoring

25. To build a production-ready add-on…
Design for multi-tenancy and horizontal scalability
Ensure deploying is easy
Make the most of 3rd party services
Load test to help estimate running cost
Implement a health check resource & monitor it
Run post-deployment verification tests
Start simple!

26. Who’s Looking Stats
2-4 dynos (average ~3)
5 external services (3 free, 2 paid)
19800 active users last week
960 active instances last week
3500 average rpm (~7500 peak)
<1s average page load time
!
$89.50 / month (~$0.005 / user / month)
Slides:
go.atlassian.com/prod-connect
robin@atlassian.com / @rewbs