MidoNet Trouble Shooting 講師：鈴木 孝彰（株式会社ミドクラ） 「Midokura OpenStack Training」の人気講師が本セミナーのためにOpenStackネットワークについて分かりやすく解説いたします。これからOpenStackを始める人やOpenStackをふたたび始める人に最適のセッションとなっております。

1. MidoNet - Troubleshooting
1MidoNet Troubleshooting, Copyright (c) 2015 Midokura

2. Troubleshooting
● General Approach
● Typical Issues
● MidoNet Tools
● Getting Help

3. General Approach
3MidoNet Troubleshooting, Copyright (c) 2015 Midokura

4. Network Stack
● Multiple layers
Underlay Network
Overlay Network
Topology Simulation
Virtual Network Topology

5. Network Stack
● Multiple layers
Underlay Network → Physical Environment
Overlay Network → Datapath / Tunnel
Topology Simulation → MidoNet Agent
Virtual Network Topology → Neutron, NSDB

6. Physical Environment
6MidoNet Troubleshooting, Copyright (c) 2015 Midokura

7. Physical Network
● Cabling
o Cannot connect to physical nodes
o ethtool → link detected?
o ip link → device status?
● Routing
o Cannot access VMs on
particular node
o netstat -nr
o ip route Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

8. Security Configuration
● Firewall
o Cannot connect to services
o iptables -L
● ACL / SELinux
o Cannot access services
o getenforce
o setenforce
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

9. Linux Kernel
● Open vSwitch module
o MidoNet Agent does not start
o VMs are not reachable
o modinfo openvswitch
o lsmod | grep openvswitch
● Load manually
o modprobe openvswitch
o insmod /lib/modules/…
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

10. Clock Synchronization
● Sync clock on all nodes
o Same time zone? JTC != UTC
o ntpd
o ntpdate
● Clocks out of sync?
o NSDB troubles
o 50% packet loss
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

11. Datapath / Tunnel
11MidoNet Troubleshooting, Copyright (c) 2015 Midokura

12. Datapath
● VMs unreachable, physical network ok
● Check status of MidoNet datapath
o mm-dpctl --show-dp midonet
o mm-dpctl --dump-dp midonet
o mm-ctl --list-hosts
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

13. Tunnel
● Tunnel Zone
o Check tunnel-zone with midonet-cli
o Check GRE/VXLAN interfaces
→ mm-dpctl --show-dp midonet
● MTU
o Overhead of tunnel protocols
 GRE (MTU 1454)
 VxLAN (MTU 1450)
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

14. MidoNet Agent
14MidoNet Troubleshooting, Copyright (c) 2015 Midokura

15. MidoNet Agent
● MidoNet configuration issues
o e.g. cannot start MidoNet Agent
o Errors show up in the log file
o Check NSDB connections
o ping [NSDB_IP]
o telnet [NSDB_IP] 2181
o telnet [NSDB_IP] 9160
● Otherwise MidoNet bugs...
Overlay Network
Topology Simulation
Virtual Network Topology
Underlay Network

16. Neutron / NSDB
16MidoNet Troubleshooting, Copyright (c) 2015 Midokura

17. Neutron / NSDB setup
● Security Groups
o Proper rules added?
●
ICMP, SSH, HTTP
● Gateway Configuration
o neutron router-gateway-set
o neutron router-interface-add
Overlay Network
Virtual NW Topology
Simulation
Virtual NW Topology
Underlay Network

18. Neutron / NSDB consistency
● DO NOT delete components via MidoNet
o neutron port-list returns error if you delete a port!
o Neutron server error if you delete a bridge!
● Manual rebuild
o Neutron → NSDB
● Use Neutron / Horizon
Overlay Network
Virtual NW Topology
Simulation
Virtual NW Topology
Underlay Network

19. MidoNet Tools
19MidoNet Troubleshooting, Copyright (c) 2015 Midokura

20. General CLI
● midonet-cli
o Interactive or script-controlled
o Client to the REST API
o Browse and manipulate topology
o Maintain tunnel zone
o Upgrade MidoNet
o man midonet-cli

21. Control
● mm-ctl
o Show Agent hosts
o Show VM tap interface mappings
o Bind / unbind ports

22. Datapath
● mm-dpctl
o Show flows
o Delete flows
o grep by IP / port

23. Packet Tracing
● mm-trace
o Filters
●
Protocol
●
IP / MAC address
●
Port number
●
…
o mm-trace –help

24. Statistics
● mm-stat
o Statistics
●
Packets processed
●
Packets dropped
●
Packets on-hold
●
Flows created
●
Latency
●
...

25. Getting Help
25MidoNet Troubleshooting, Copyright (c) 2015 Midokura

26. Getting Help
● Chat
– https://slack.midonet.org
● Mailing Lists
– https://lists.midonet.org
● Troubleshooting Guide
– http://docs.midonet.org

27. Getting Help
● Chat
– https://slack.midonet.org
● Mailing Lists
– https://lists.midonet.org
● Troubleshooting Guide
– http://docs.midonet.org

28. Thank You!
28MidoNet Troubleshooting, Copyright (c) 2015 Midokura