The document discusses debugging techniques at different levels from basic to advanced. It begins with level 1 debugging which involves tools like echo and var_dump and strategies like researching issues online and asking peers for help. Level 2 debugging introduces logging, advanced local environments, monitoring and profiling tools. It discusses strategies like replicating issues and eliminating variables. Level 3 involves preventative techniques like defensive programming, code quality tools, and ensuring a learning culture. The document also provides an example case study of randomly occurring cache corruption that was solved through these debugging steps.

2. Elite Bug Squashing

3. Technical Director, Space 48
@Tonegolf71
@Space48ers
Tony Brown

4. Mage Titans UK – Developer Conference
10th November 2018
@MageTitans
Tony Brown

5. Cache corruption
Randomly occurring, site down

6. Random cache corruption
• A pre-launch Magento 2 site
• Would randomly end up with exceptions
• The site would be unavailable until…
• …a cache clear would resolve the issue

7. Random cache corruption
• A pre-launch Magento 2 site
• Would randomly end up with exceptions
• The site would be unavailable until…
• …a cache clear would resolve the issue
Where do you begin?

8. Level 1
Finding your debugging feet

9. Debugging starts on day 1
• Debugging is an integral part of coding
• I started with 20% coding, 80% debugging
• We should strive for the opposite!

10. Types of debugging
• Bugs during development
• Bugs that are “live”
– Feature broken
– Performance issues
– Site down

11. Impact of bugs
• The further along the development lifecycle that bugs appear, the
greater the impact.

12. Level 1 Tooling
• Echo / var_dump / print_r … die()

13. Level 1 Strategy: Research
• The majority of problems you will face have been solved already.

14. Level 1 Strategy: Research
• The majority of problems you will face have been solved already.
• Opensource community rocks!
– Stack exchange
– Forums
– Github
– Contribute if you can!

15. Level 1 Strategy: Ask your peers
• You shouldn’t be fearful of leaning on your team

16. Level 1 Strategy: Ask your peers
• You shouldn’t be fearful of leaning on your team
• Articulating a problem (even to someone non-technical) can result
in your brain sparking off new ideas
– They may have ideas or even the solution!

17. Random cache corruption
• Google revealed it to be a config cache issue
– But the recommended resolution didn’t apply

18. Random cache corruption
• Google revealed it to be a config cache issue
– But the recommended resolution didn’t apply
• No-one in the team had seen the issue before

19. Random cache corruption
• Google revealed it to be a config cache issue
– But the recommended resolution didn’t apply
• No-one in the team had seen the issue before
• Collaboration did happen, which accelerated progress

20. Level 2
Walking the debugging walk

21. Level 2 Tooling
• Logging

22. Level 2 Tooling
• Logging
Why is the hard drive nearly full?!

23. Level 2 Tooling
• Logging
• Advanced local environments

24. Level 2 Tooling
• Logging
• Advanced local environments
• Monitoring - New Relic

25. Level 2 Tooling
• Logging
• Advanced local environments
• Monitoring - New Relic
• Profiling – Blackfire

26. Level 2 Strategy
• Can I replicate the issue?

27. Level 2 Strategy
• Can I replicate the issue?
• What has changed?

28. Level 2 Strategy
• Can I replicate the issue?
• What has changed?
• Are there any errors?

29. Level 2 Strategy
• Can I replicate the issue?
• What has changed?
• Are there any errors?
• Eliminate as much as possible at each step

30. Level 2 Strategy
• Can I replicate the issue?
• What has changed?
• Are there any errors?
• Eliminate as much as possible at each step
• Protect through process:
– Code Reviews
– QA Function
– Automated tests where possible

31. Random cache corruption
• Couldn’t replicate it locally

32. Random cache corruption
• Couldn’t replicate it locally
• What had changed?
– Recently gone to multi-server setup

33. Single-server setup
Cluster
RedisDB
Servers
Pre-
prod
Load Balancer

34. Multi-server setup
Cluster
RedisDB
Servers
Web 1 Web 3Web 2 Admin
Load Balancer

35. Random cache corruption
• Couldn’t replicate it locally
• What had changed?
– Recently gone to multi-server setup
• Eliminate as much as possible
– When set to single server, the issue disappeared

36. Multi-server setup (restricted)
Cluster
RedisDB
Servers
Web 1 Web 3Web 2 Admin
Load Balancer

37. Random cache corruption
• Couldn’t replicate it locally
• What had changed?
– Recently gone to multi-server setup
• Eliminate as much as possible
– When set to single server, the issue disappeared
• Put logging in place to capture state
– Found that the config cache was getting zapped repeatedly

38. Level 3
Clear vision and prevention

39. Level 3 Tooling
• Xdebug

40. Level 3 Tooling
• Xdebug
• Static analysis for quality / best practice
– Mess Detector, Copy paste detector

41. Level 3 Strategy
• Defensive Programming

42. Level 3 Strategy
• Defensive Programming
• Conscious structure of code
– SOLID principles

43. Level 3 Strategy
• Defensive Programming
• Conscious structure of code
– SOLID principles
• Tight feedback loops

44. Level 3 Strategy: Culture of learning – Reduce FEAR!
• Mistakes and failures are learning opportunities
• Ensure people understand they will be supported
• Put in place processes to reduce risk and protect people
• Actively encourage people to seek improvements

45. Random cache corruption
• Used Xdebug to find it was a 3rd party module zapping the cache

46. Random cache corruption
• Used Xdebug to find it was a 3rd party module zapping the cache
• Used Redis Monitor to observe the config cache being written to
– This revealed that when it was written to with bad data, it always
came from the admin IP address

47. Random cache corruption
• Used Xdebug to find it was a 3rd party module zapping the cache
• Used Redis Monitor to observe the config cache being written to
– This revealed that when it was written to with bad data, it always
came from the admin IP address
The old pre-production environment still existed!

48. Multi-server setup
Cluster
RedisDB
Servers
Web 1 Web 3Web 2 Admin
Load Balancer

49. Multi-server setup (pre-prod!!)
Cluster
RedisDB
Servers
Web 1 Web 3Web 2 Admin
Load Balancer
Pre-prod

50. Bonus Level
Increasing efficiency

51. Bonus Level: The Pomodoro Technique
• 25 minute cycles

52. Bonus Level: The Pomodoro Technique
• 25 minute cycles
• At each break, question progress and approach

53. Bonus Level: The Pomodoro Technique
• 25 minute cycles
• At each break, question progress and approach
• Encourages focus and reduces chances of going “down the rabbit
hole”

54. Thank you!
• Any Questions?
Manchester
10th November 2018
@MageTitans
@tonegolf71