This document discusses troubleshooting RabbitMQ and services that use it. It provides tips for troubleshooting publishers and consumers, including common issues like I/O exceptions, inadequate delivery quality of service, and lack of confirmations. It also recommends collecting logs, metrics, exceptions and traces system-wide to analyze issues. Distributed system debugging remains a challenging problem.

1. Troubleshooting
RabbitMQ
and services that use it

2. Who am I?
• Staff Engineer, RabbitMQ @ Pivotal

3. Who am I?
• Staff Engineer, RabbitMQ @ Pivotal
• @michaelklishin, github.com/michaelklishin

5. The monolith problem

21. Troubleshooting publishers

22. Troubleshooting publishers
• I/O exceptions (shutdown handlers)

23. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms

25. When in doubt, borrow
ideas from TCP

26. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers

27. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers
• Invalid payload (e.g. fails to deserialize or
decrypt)

28. Troubleshooting publishers
• I/O exceptions (shutdown handlers)
• Publisher confirms
• Returned message handlers
• Invalid payload (e.g. fails to deserialize or
decrypt)
• Identifying publisher instances

29. Troubleshooting publishers
• identifying blocked (throttled) publishers

30. Client-provided
connection names in
RabbitMQ 3.6.3+

31. Troubleshooting publishers
• identifying blocked (throttled) publishers
• retries

32. Troubleshooting publishers
• spring-amqp can cover all of the above

35. Troubleshooting consumers

36. Troubleshooting consumers
• I/O exceptions

37. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS

38. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming

40. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming

41. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming
• Redelivery metrics

42. Troubleshooting consumers
• I/O exceptions
• Inadequate delivery QoS
• Lack of confirmations; double-confirming
• Redelivery metrics
• Identifying consumer instances

43. Troubleshooting consumers
• Consumer utilization (reported by HTTP API)

44. Troubleshooting consumers
• spring-amqp can help with some of the above

46. — W. Edwards Deming
“In God we trust, all others must bring data…”

47. — W. Edwards Deming
“In God we trust, all others must bring data…”

48. — What do you do for
a living?

49. — What do you do for a living?
— Tell people to read the logs.

50. Sources of data useful for
debugging

51. Sources of data useful for
debugging
• Metrics

52. Sources of data useful for
debugging
• Metrics
• Your logs

53. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs

54. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs
• Tracing data

55. Sources of data useful for
debugging
• Metrics
• Your logs
• Someone else's logs
• Tracing data
• Wireshark (tcpdump, libpcap)

58. Collecting data from
RabbitMQ

59. Collecting data from
RabbitMQ
• Logs

60. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status

61. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment

62. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)

63. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)
• HTTP API (lots of metrics)

64. http://{hostname}:15672/api

65. curl -u guest:guest http://127.0.0.1:15672/api/overview | python -m json.tool
curl -u guest:guest http://127.0.0.1:15672/api/nodes/{node} | python -m json.tool
curl -u guest:guest http://127.0.0.1:15672/api/queues | python -m json.tool

66. Collecting data from
RabbitMQ
• Logs
• rabbitmqctl status
• rabbitmqctl environment
• rabbitmq-top (ships with RabbitMQ as of 3.6.3)
• HTTP API (lots of metrics)
• Message tracing ("firehose")

67. Collecting data from
RabbitMQ
• HTTP API (lots of metrics)
• Message tracing ("firehose")
• Infrastructure metrics

69. Common theme?

70. Common theme?
• Collect logs system-wide

71. Common theme?
• Collect logs system-wide
• Collect metrics system-wide

72. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide

73. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)

74. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)
• Analyze

76. Common theme?
• Collect logs system-wide
• Collect metrics system-wide
• Collect exceptions system-wide
• Trace requests (e.g. with Zipkin)
• Analyze
• Sounds like something a structured platform can help
with!

77. Distributed system
debugging is a problem
far from being solved.

79. Thank you

80. Thank you
• @michaelklishin

81. Thank you
• @michaelklishin
• github.com/michaelklishin

82. Thank you
• @michaelklishin
• github.com/michaelklishin
• mklishin@pivotal.io