This document summarizes a 1 year journey to improve the stability, reliability, and performance of an app and platform with over 1 million users and 2 million euros in monthly revenue. Key improvements included implementing dashboards and improved logs to monitor performance, setting alarms to detect issues, prioritizing a backlog of improvements, conducting weekly refactor sessions, and analyzing metrics like latency averages and percentiles to identify and address bottlenecks. These efforts led to significant benefits for clients like much less downtime, less time spent debugging issues, faster app performance, and issues being detected before users were impacted.

2. 1 year journey
to improve the stability

3. +1 million users
+2 million € /month

4. App
+
Platform
!

5. Platform
"
App
!
Me
#

6. “Performance is a feature”
Stability and Reliability
are also a feature

7. (we had almost nothing)
1 year ago

8. Today
Dashboards

9. Today
Improved Logs

10. Your card was not
accepted
The payment was rejected
for lack of funds
Contact Support
Retry
Today
Better user messages

11. 🔔
Alarms
Today

12. let’s go to details

13. Challenges
I’m not the DEV
Convince the Client
???

14. Improvements Backlog

15. ♜♜♜ ♖♖♖♖♖
♖♖♖♖♖
♖♖♖♖
Time Allocation

16. Improvements Day
1-week refactor
Talks
Initiatives

17. API Visibility

18. Latency
average
400
300
200
100

19. “Why Averages suck and
Percentiles are great”
Michael Kopp

20. Average vs Percentiles
0
225
450
675
900
00:00 02:00 04:00 08:00 10:00 12:00 14:00 16:00 18:00

21. 800
600
400
200
Latency
average p95

22. p99 🔥
30000
20000
10000
Latency

23. Latency
DELETE /api/fines/pay
GET /api/request
GET /api/request
GET /api/request
GET /api/request
GET /api/request
8s2.1s 21s
p99 🔥

24. 20000
10000
p99 🔥
Latency
improvement 😀

25. “Performance is a feature”
Stability and Reliability
are also a feature

26. 100% transparent to the client

27. Improving logs

28. Improving logs

29. 🔔
Being alert

30. Alarms

32. Client benefits
MUCH less downtime
LESS time to debug user issues
DETECT issues before the users
FASTER app

33. Thank you
dfigueiredo@pminds.pt