Openstack Networking Internals - first part
Die SlideShare-Präsentation wird heruntergeladen.
×
![関連する設定パラメータ
最低限の設定
● /etc/neutron/neutron.conf
– [Default]
● l3_ha = True
その他のパラメータ
● /etc/neutron/neutron.conf
– [...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-12-2048.jpg?cb=1668061332)
![フェイルオーバー
● 初期状態
– master: juno-network1
– backup: juno-network2
● keepalivedを殺す
[root@juno-network1 ~]# date; killall...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-23-2048.jpg?cb=1668061332)
![フェイルオーバー
● OSのログ
juno-network1
12月 01 12:51:59 juno-network1 Keepalived[3668]: Stopping Keepalived v1.2.10 (06/10,2014)...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-24-2048.jpg?cb=1668061332)
![フェイルオーバー
● VMへの疎通
[Mon Dec 1 12:51:52 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=417 ttl=62 time=1.13 ms
[Mon Dec ...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-25-2048.jpg?cb=1668061332)
![フェイルオーバー前
● ip netns
juno-network1 juno-network2
[root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-26-2048.jpg?cb=1668061332)
![フェイルオーバー後
● ip netns
juno-network1 juno-network2
[root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-27-2048.jpg?cb=1668061332)
![素朴な疑問
● HAルータを2個作ると？
[root@juno-network1 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns}
ip a; ...](https://image.slidesharecdn.com/l3ha-vrrp-20141201-141201231404-conversion-gate01/75/l3havrrp20141201-35-2048.jpg?cb=1668061332)
Hier ansehen
Weitere Verwandte Inhalte
Diashows für Sie (20)
Andere mochten auch (20)
Ähnlich wie L3HA-VRRP-20141201 (20)
Aktuellste (20)
L3HA-VRRP-20141201
- 1. 日本OpenStackユーザ会 第20回勉強会 Neutron L3 HA (VRRP) 2014-12-01 Manabu Ori
- 2. 自己紹介 ● 名前: 織 学 (おり まなぶ) ● 所属: レッドハット ● Twitter: @orimanabu
- 3. 自己紹介 ● 名前: 織 学 (おり まなぶ) ● 所属: レッドハット ● Twitter: @orimanabu ● slideshareに上げた資料たち – POWER + PostgreSQL – POWER + KVM
- 4. Neutron L3 agentとは ● OpenStackのネットワーク仮想化で使用するエージェ ントのひとつ – 他にはMetadata Proxy agentとかDHCP agentとか ● テナントネットワークのサブネット間接続、テナント VMと外部ネットワークとの接続等を制御する ● 複数ノードで起動可能 – 仮想ルータはどこか1ノード上で実行される 仮想ルータ #1 仮想ルータ #2 L3 Agent #1 L3 Agent #2 VM VM VM パブリック ネットワーク テナント ネットワーク
- 5. これまでのL3 agentの冗長化 ● コントローラーノードによるリスケジューリング – allow_automatic_l3agent_failover = True – Juno以降で使用可能 – https://review.openstack.org/#/c/110893/ – https://bugs.launchpad.net/neutron/+bug/1174591 ● L3 Agent Healthcheck – OpenStackと独立して開発 (by eNovance) – Grizzly, Havana, Icehouseで使用可能 – https://github.com/enovance/neutron-l3-healthcheck ● Pacemaker等でHAクラスター化 フェイルオーバーに 時間がかかる フェイルオーバーに 時間がかかる 設定が大変
- 6. L3 HA概要 ● L3 agentの冗長化機能を拡張/ドライバの形で実装 ● 今の実装はkeepalivedを使用 – VRRPを用いて死活監視 – フェイルオーバーはRPC layerから独立 ● テナントごとにひとつVRRPハートビートネットワークを作 成 – ハートビートネットワークも通常のNeutronネットワーク – 物理的にはテナントネットワークを通る ● 全てのテナント分割方式 (VXLAN, VLAN, GRE, ...)で使用可能 ● フェイルオーバー時は、ゲートウェイのIPアドレスおよび Floating IPを仮想IPアドレスとして引き継ぐ
- 7. VRRPとは ● VRRP: Virtual Router Redundancy Protocol ● RFC3768 ● 複数のルータから仮想的なルータを構成 – マスタールータ ● 一時点に1台のみ ● 実際にフォワーディング処理を行う – バックアップルータ ● 通常は待機 ● 障害時にマスターになる ● それぞれのルータは、ステータスの確認メッセージを交換するこ とで障害を検知 ● First Hop Redundancy Protocol – サーバ、PC等のデフォルトルートとしてIPアドレス、MACアドレスを 提供
- 8. ネットワークノードの構成 - non-HAの場合 - Network Node #1 Network Node #2 eth2 br-ex qg-BBB qr-AAA br-int IP address int-br-tenant phy-br-tenant br-tenant eth1 eth2 br-ex qg-EEE qr-DDD br-int IP address int-br-tenant phy-br-tenant br-tenant eth1 パブリックネットワーク テナントネットワーク IP address IP address qrouter-xxxx qrouter-yyyy
- 9. ネットワークノードの構成 - HAの場合 - Network Node #1 Network Node #2 eth2 br-ex qg-BBB br-int qr-AAA qrouter-xxxx qrouter-yyyy ha-CCC int-br-tenant phy-br-tenant br-tenant eth1 eth2 br-ex qg-EEE br-int ha-FFF qr-DDD int-br-tenant phy-br-tenant br-tenant eth1 パブリックネットワーク VRRPハートビート テナントネットワーク VIP VIP VIP VIP Gateway Floating IP Fixed IP keepalived keepalived IP address IP address
- 10. 制限事項 ● VRRPハートビート用のネットワークは1つのみ – テナントネットワークを流れる ● テナント当たりの仮想ルータは最大256個 – ハートビートネットワークが1つ かつ VRIDが8ビット ● セッションの引き継ぎは現時点では未サポート ● DVRとの同時使用は現時点では未サポート ● (LB|FW|VPN)aaSとの統合は現時点では未サポート ● 今後の改良予定 – どれがマスターがわかりにくい問題 – 手動フェイルオーバーの方法問題 – L2populationと仲良くなる
- 11. Gerrit Topics ● https://review.openstack.org/#/q/topic:bp/l3-h igh-availability,n,z ●
- 12. 関連する設定パラメータ 最低限の設定 ● /etc/neutron/neutron.conf – [Default] ● l3_ha = True その他のパラメータ ● /etc/neutron/neutron.conf – [Default] ● max_l3_agents_per_router 3 ● min_l3_agents_per_router 2 ● l3_ha_net_cidr 169.254.192.0/18 ● /etc/neutron/l3_agent.ini – [Default] ● agent_mode legacy ● ha_confs_path $state_path/ha_confs ● ha_vrrp_auth_type PASS ● ha_vrrp_auth_password openstack ● ha_vrrp_advert_int 2
- 13. 動き ● neutron router-createすると – ハートビート用のNeutronネットワークを作成 – ハートビート用のインタフェースをルータの Namespace内に作成 – keepalivedを起動 Public Network Management/API Network eth0 eth0 eth2 eth0 eth2 eth0 controller network1 network2 compute1 eth1 eth1 eth1 eth1 Tenant Network
- 14. ルータ作成 (1) ● neutron router-createすると... # neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') router_demo Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | False | | external_gateway_info | | | ha | True | | id | e7e98ef7-36d5-4544-be07-c83ddf7e67ea | | name | router_demo | | routes | | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | +-----------------------+--------------------------------------+
- 15. ルータ作成 (2) ● 両ノードのL3 agentで動いている # neutron l3-agent-list-hosting-router router_demo +--------------------------------------+---------------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+---------------+----------------+-------+ | 1c407f95-5a62-4577-a86d-623943ab0283 | juno-network1 | True | :-) | | 8ad4e4df-cd8e-4c6e-bea2-8101a1225019 | juno-network2 | True | :-) | +--------------------------------------+---------------+----------------+-------+ ● ハートビート用のネットワークができている # neutron net-list +--------------------------------------+----------------------------------------------------+----------------------------------| id | name | subnets +--------------------------------------+----------------------------------------------------+----------------------------------| 616a2195-162c-4e09-8afa-a347c56b9c11 | external | 3c705be7-6d51-| 218b2842-1832-4ec6-a979-5efdf01453bf | HA network tenant 328ffb467eb24c3595b549516b8acbd0 | 288d389b-33c9-+--------------------------------------+----------------------------------------------------+----------------------------------
- 16. ルータ作成 (3) ● ハートビート用ネットワークの情報 # neutron net-show 218b2842-1832-4ec6-a979-5efdf01453bf +---------------------------+----------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------+ | admin_state_up | True | | id | 218b2842-1832-4ec6-a979-5efdf01453bf | | name | HA network tenant 328ffb467eb24c3595b549516b8acbd0 | | provider:network_type | vlan | | provider:physical_network | physnet-tenant | | provider:segmentation_id | 2001 | | router:external | False | | shared | False | | status | ACTIVE | | subnets | 288d389b-33c9-4559-8ec8-98b4faac2274 | | tenant_id | | +---------------------------+----------------------------------------------------+ # neutron subnet-show 288d389b-33c9-4559-8ec8-98b4faac2274 +-------------------+------------------------------------------------------+ | Field | Value | +-------------------+------------------------------------------------------+ | allocation_pools | {"start": "169.254.192.1", "end": "169.254.255.254"} | | cidr | 169.254.192.0/18 | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | | | host_routes | | | id | 288d389b-33c9-4559-8ec8-98b4faac2274 | | ip_version | 4 | | ipv6_address_mode | | | ipv6_ra_mode | | | name | HA subnet tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | 218b2842-1832-4ec6-a979-5efdf01453bf | | tenant_id | | +-------------------+------------------------------------------------------+ ● network_type、physical_networkは テナント用ネットワークと同じ ● 独立したsegmentation_idが振られる ● テナントIDは空 ● /etc/neutron/neutron.confで変更可能 [DEFAULT] l3_ha_net_cidr = 169.254.192.0/18
- 17. ルータ作成 (4) ● ハートビート用のポートの確認 # neutron port-list +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ | 00ab5988-dedd-421d-921a-61698de65155 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:7c:52:0d | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.2"} | | ef4a5a34-17a1-4008-916a-72ff19424b61 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:16:91:0f | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.1"} | +--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+ # for port_id in $(neutron port-list | awk '/HA port/ {print $2}'); do neutron port-show ${port_id}; done +-----------------------+--------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | juno-network1 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b | | device_owner | network:router_ha_interface | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.2"} | | id | 208ace4e-077c-4b23-86a3-d67a7d5ca612 | | mac_address | fa:16:3e:99:54:58 | | name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 | | security_groups | | | status | ACTIVE | | tenant_id | | +-----------------------+--------------------------------------------------------------------------------------+ +-----------------------+--------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | juno-network2 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b | | device_owner | network:router_ha_interface | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.1"} | | id | e6a1529c-6a52-418e-8507-7531d343a84a | | mac_address | fa:16:3e:a3:bc:e6 | | name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | | network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 | | security_groups | | | status | ACTIVE | | tenant_id | | +-----------------------+--------------------------------------------------------------------------------------+
- 18. keepalived (1) ● keepalived関連のファイル置き場 # ls -F /var/lib/neutron/ dhcp/ external/ ha_confs/ lock/ metadata_proxy= ● keepalived関連のファイルたち # find /var/lib/neutron/ha_confs/ -type f /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid-vrrp router id 状態変化に応じて 呼ばれるスクリプト 設定ファイル master or backup
- 19. keepalived (2) ● 設定ファイル # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf vrrp_sync_group VG_1 { group { VR_1 } notify_master "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh" notify_backup "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh" notify_fault "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh" } vrrp_instance VR_1 { state BACKUP interface ha-ef4a5a34-17 virtual_router_id 1 priority 50 nopreempt advert_int 2 track_interface { ha-ef4a5a34-17 } virtual_ipaddress { 172.16.99.100/24 dev qg-a6ee9669-8f } virtual_ipaddress_excluded { 172.16.99.101/32 dev qg-a6ee9669-8f 192.168.10.1/24 dev qr-3d8666f0-c4 } virtual_routes { 0.0.0.0/0 via 172.16.99.1 dev qg-a6ee9669-8f } } ● パブリックネットワークの ゲートウェイ ● Floating IP ● テナントネットワークの ゲートウェイ
- 20. keepalived (3) ● 通知スクリプトたち マスターになったら metadata proxyも起動する # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh #!/usr/bin/env bash kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid) echo -n backup > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh #!/usr/bin/env bash kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid) echo -n fault > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state # cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh #!/usr/bin/env bash neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid --metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=e7e98ef7-36d5-4544-be07-c83ddf7e67ea --state_path=/var/lib/neutron --metadata_port=9697 --verbose --log-file=neutron-ns-metadata-proxy-e7e98ef7- 36d5-4544-be07-c83ddf7e67ea.log --log-dir=/var/log/neutron echo -n master > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state マスターかバックアップかを stateファイルに書いておく
- 21. keepalived (4) ● keepalivedはプロセスが2 個見える – VRRP用に1個 fork(2) する – 親はVRRP用の子供を監視 して、死んでいたら respawnする # pgrep keepalived 2948 5130 # pstree -p -N net [4026531956] systemd(1)─┬─agetty(908) ├─agetty(909) ├─auditd(594)───{auditd}(605) ├─avahi-daemon(624)───avahi-daemon(635) ├─chronyd(674) ├─crond(701) ├─dbus-daemon(680)───{dbus-daemon}(698) ├─httpd(1189)─┬─httpd(1553) │ ├─httpd(1554) │ ├─httpd(1555) │ ├─httpd(1556) │ ├─httpd(1557) │ ├─httpd(1558) │ ├─httpd(1559) │ └─httpd(1560) ├─iprdump(745) ├─iprinit(668) ├─iprupdate(671) ├─irqbalance(644) ├─lvmetad(471) ├─master(1676)─┬─pickup(27341) │ └─qmgr(1690) ├─monitor(704)───ovsdb-server(705) ├─monitor(727)───ovs-vswitchd(729)─┬─{ovs-vswitchd}(732) │ ├─{ovs-vswitchd}(733) │ ├─{ovs-vswitchd}(734) │ └─{ovs-vswitchd}(735) ├─neutron-dhcp-ag(2291) ├─neutron-l3-agen(2290) ├─neutron-metadat(1192)─┬─neutron-metadat(2167) │ └─neutron-metadat(2168) ├─neutron-meterin(1191) ├─neutron-openvsw(2293)───sudo(2378)───neutron-rootwra(2380)───ovsdb-client(2382) ├─nrpe(1198) ├─rsyslogd(626)─┬─{rsyslogd}(646) │ └─{rsyslogd}(647) ├─sshd(1199)───sshd(1842)───bash(1969)───pstree(28359) ├─systemd-journal(470) ├─systemd-logind(679) ├─systemd-udevd(473) └─tuned(642)─┬─{tuned}(774) ├─{tuned}(775) ├─{tuned}(776) └─{tuned}(788) [4026532345] dnsmasq(3834) [4026532262] keepalived(2948)───keepalived(5130) neutron-ns-meta(4043)
- 22. マスタールータの確認 ● VIPがついている方がマスター juno-network1 juno-network2 # ip netns list qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea # ip netns list qdhcp-802e69c8-b064-4eb1-90b9-327e712472d8 qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07- c83ddf7e67ea/state; echo backup [ro[root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07- c83ddf7e67ea/state; echo master ● ここにも書いてある
- 23. フェイルオーバー ● 初期状態 – master: juno-network1 – backup: juno-network2 ● keepalivedを殺す [root@juno-network1 ~]# date; killall keepalived Mon Dec 1 12:51:59 JST 2014 [root@juno-controller rdo-juno(keystone_demo)]# nova show vm1 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2014-12-01T03:37:44.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2014-12-01T03:37:01Z | | demo_net network | 192.168.10.100, 172.16.99.101 | | flavor | m1.tiny (1) | | hostId | 826d63fe7c11232775d45dd1edc8bf9db15b01d01ce908c143434750 | | id | da10da5e-6b61-40f7-b5ab-18409b4d4267 | | image | cirros (69d97e8f-24b5-4f0b-8977-7295d8ba8edb) | | key_name | sshkey | | metadata | {} | | name | vm1 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | | updated | 2014-12-01T03:37:44Z | | user_id | 01c4dc5c8ec0494eaaecb84fd4170556 | +--------------------------------------+----------------------------------------------------------+ [root@juno-controller rdo-juno(keystone_demo)]# nova ssh --login cirros vm1 $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:be:02:20 brd ff:ff:ff:ff:ff:ff inet 192.168.10.100/24 brd 192.168.10.255 scope global eth0 inet6 fe80::f816:3eff:febe:220/64 scope link valid_lft forever preferred_lft forever $ ping 172.16.99.11 PING 172.16.99.11 (172.16.99.11): 56 data bytes 64 bytes from 172.16.99.11: seq=0 ttl=63 time=2.659 ms 64 bytes from 172.16.99.11: seq=1 ttl=63 time=0.687 ms 64 bytes from 172.16.99.11: seq=2 ttl=63 time=0.981 ms ^C --- 172.16.99.11 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.687/1.442/2.659 ms $ exit Connection to 172.16.99.101 closed. [root@juno-controller rdo-juno(keystone_demo)]#
- 24. フェイルオーバー ● OSのログ juno-network1 12月 01 12:51:59 juno-network1 Keepalived[3668]: Stopping Keepalived v1.2.10 (06/10,2014) 12月 01 12:51:59 juno-network1 Keepalived_vrrp[5025]: VRRP_Instance(VR_1) sending 0 priority juno-network2 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Transition to MASTER STATE 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Group(VG_1) Syncing instances to MASTER state 12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: Opening script file /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh 12月 01 12:52:02 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Entering MASTER STATE
- 25. フェイルオーバー ● VMへの疎通 [Mon Dec 1 12:51:52 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=417 ttl=62 time=1.13 ms [Mon Dec 1 12:51:53 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=418 ttl=62 time=1.12 ms [Mon Dec 1 12:51:54 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=419 ttl=62 time=1.41 ms [Mon Dec 1 12:51:55 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=420 ttl=62 time=1.24 ms [Mon Dec 1 12:51:56 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=421 ttl=62 time=0.617 ms [Mon Dec 1 12:51:57 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=422 ttl=62 time=0.855 ms [Mon Dec 1 12:51:58 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=423 ttl=62 time=1.39 ms [Mon Dec 1 12:51:59 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=424 ttl=62 time=1.20 ms [Mon Dec 1 12:52:03 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=428 ttl=62 time=4.30 ms [Mon Dec 1 12:52:04 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=429 ttl=62 time=1.63 ms [Mon Dec 1 12:52:05 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=430 ttl=62 time=1.16 ms [Mon Dec 1 12:52:06 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=431 ttl=62 time=0.952 ms [Mon Dec 1 12:52:07 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=432 ttl=62 time=1.25 ms [Mon Dec 1 12:52:08 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=433 ttl=62 time=0.906 ms [Mon Dec 1 12:52:09 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=434 ttl=62 time=1.26 ms [Mon Dec 1 12:52:10 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=435 ttl=62 time=1.22 ms
- 26. フェイルオーバー前 ● ip netns juno-network1 juno-network2 [root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state master [root@juno-network2 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state backup
- 27. フェイルオーバー後 ● ip netns juno-network1 juno-network2 [root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe16:910f/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe7a:d916/64 scope link valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# ip netns exec $(ip netns list | grep qrouter) ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7c:520d/64 scope link valid_lft forever preferred_lft forever 10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet 172.16.99.101/32 scope global qg-a6ee9669-8f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-3d8666f0-c4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe43:438c/64 scope link valid_lft forever preferred_lft forever
- 28. フェイルオーバー ● VRRP 12:51:51.991662 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:53.992827 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:55.993186 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:57.994432 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:51:59.674908 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 2s, length 20 12:52:00.481158 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:02.483882 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:04.484732 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:06.485926 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20 12:52:08.485989 IP 169.254.192.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20
- 29. フェイルオーバー ● Gratuitous ARP 12:52:02.476000 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476075 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476121 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476133 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476144 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477325 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477376 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477403 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477429 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477439 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:51:59.679351 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679373 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679377 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679380 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679383 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679385 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679388 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679391 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:51:59.679393 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:51:59.679396 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680231 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680307 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680313 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680315 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680317 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680319 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680321 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680322 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:04.680324 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:04.680326 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28
- 30. フェイルオーバー ● network nodes (external) 12:52:02.475961 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476059 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476114 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476127 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476138 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476150 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476155 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476161 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.476175 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.476181 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:03.331280 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 428, length 64 12:52:03.334090 ARP, Request who-has 172.16.99.254 tell 172.16.99.101, length 28 12:52:03.334767 ARP, Reply 172.16.99.254 is-at 52:54:00:9e:a3:85 (oui Unknown), length 28 12:52:03.335070 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 428, length 64 12:52:04.332680 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 429, length 64 12:52:04.333804 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 429, length 64 12:52:05.334435 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 430, length 64 12:52:05.335160 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 430, length 64 12:52:06.335625 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 431, length 64 12:52:06.336258 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 431, length 64 12:52:07.335902 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 432, length 64 12:52:07.336705 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 432, length 64 12:52:07.477275 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477364 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477397 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477424 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477433 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477446 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477458 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477462 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.477466 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.477475 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:08.337331 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 433, length 64 12:52:08.337924 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 433, length 64 12:52:09.337900 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 434, length 64 12:52:09.338739 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 434, length 64 12:52:10.339405 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 435, length 64 12:52:10.340197 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 435, length 64 juno-network1 12:51:57.328686 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 422, length 64 12:51:57.329325 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 422, length 64 12:51:58.329837 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 423, length 64 12:51:58.330791 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 423, length 64 12:51:59.331426 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 424, length 64 12:51:59.332194 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 424, length 64 12:52:00.332890 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 425, length 64 12:52:01.332965 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 426, length 64 12:52:02.332935 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 427, length 64 12:52:02.478520 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.478549 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479044 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479059 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479062 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479064 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479067 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479069 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:02.479071 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:02.479073 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:03.335874 ARP, Request who-has 172.16.99.254 tell 172.16.99.101, length 28 12:52:07.479212 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479867 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479877 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479879 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479881 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479882 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479884 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479885 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 12:52:07.479887 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28 12:52:07.479889 ARP, Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28 juno-network2
- 31. フェイルオーバー ● network nodes (tenant) juno-network1 juno-network2 12:51:59.675765 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:02.476000 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476075 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476121 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476133 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.476144 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:03.331701 ARP, Request who-has 192.168.10.100 tell 192.168.10.1, length 28 12:52:03.332823 ARP, Reply 192.168.10.100 is-at fa:16:3e:be:02:20 (oui Unknown), length 28 12:52:03.333103 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 428, length 64 12:52:03.333832 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 428, length 64 12:52:04.332750 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 429, length 64 12:52:04.333764 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 429, length 64 12:52:04.403080 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:05.334499 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 430, length 64 12:52:05.335128 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 430, length 64 12:52:06.335673 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 431, length 64 12:52:06.336219 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 431, length 64 12:52:07.335970 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 432, length 64 12:52:07.336659 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 432, length 64 12:52:07.477325 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477376 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477403 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477429 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.477439 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:08.337381 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 433, length 64 12:52:08.337903 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 433, length 64 12:52:09.337972 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 434, length 64 12:52:09.338713 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 434, length 64 12:52:10.339497 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 435, length 64 12:52:10.340170 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 435, length 64 12:51:57.329293 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 422, length 64 12:51:58.329928 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 423, length 64 12:51:58.330740 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 423, length 64 12:51:59.331478 IP 172.16.99.254 > 192.168.10.100: ICMP echo request, id 6456, seq 424, length 64 12:51:59.332152 IP 192.168.10.100 > 172.16.99.254: ICMP echo reply, id 6456, seq 424, length 64 12:51:59.677264 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:02.478590 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478605 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478625 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478631 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:02.478636 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:03.333519 ARP, Request who-has 192.168.10.100 tell 192.168.10.1, length 28 12:52:04.404617 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s) 12:52:07.479383 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479396 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479401 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479404 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28 12:52:07.479408 ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 28
- 32. 素朴な疑問 (1) ● Q: non-HA構成で作ったルータをHA構成に変更 できる？ ● A: できませんでした # neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') --ha False router_demo Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | distributed | False | | external_gateway_info | | | ha | False | | id | f747119c-6183-4ae4-b6a5-397628a3cc4a | | name | router_demo | | routes | | | status | ACTIVE | | tenant_id | 328ffb467eb24c3595b549516b8acbd0 | +-----------------------+--------------------------------------+ # neutron router-list +--------------------------------------+-------------+-----------------------+-------------+-------+ | id | name | external_gateway_info | distributed | ha | +--------------------------------------+-------------+-----------------------+-------------+-------+ | f747119c-6183-4ae4-b6a5-397628a3cc4a | router_demo | null | False | False | +--------------------------------------+-------------+-----------------------+-------------+-------+ [root@juno-controller rdo-juno(keystone_admin)]# neutron router-update router_demo --ha True Bad Request (HTTP 400) (Request-ID: req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c) 2014-11-30 23:19:41.715 3526 INFO neutron.wsgi [-] (3526) accepted ('10.0.1.111', 54480) 2014-11-30 23:19:41.776 3526 INFO neutron.wsgi [req-f2181436-4a6c-4ab8-b63d-f12c5770d6f2 None] 10.0.1.111 - - [30/Nov/2014 23:19:41] "GET /v2.0/routers.json?fields=id&name=router_demo HTTP/1.1" 200 250 0.059557 2014-11-30 23:19:41.785 3526 ERROR neutron.api.v2.resource [req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c None] update failed 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource Traceback (most recent call last): 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 87, in resource 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource result = method(request=request, **args) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 501, in update 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource allow_bulk=self._allow_bulk) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 623, in prepare_request_body 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource raise webob.exc.HTTPBadRequest(msg) 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute ha 2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource
- 33. 素朴な疑問 (2) ● Q: マスタールータはどっち？ ● A: VIPがついている方 ● でもたまに動きがあやしい... ● この辺りから総合的に判断するのが確実 – VIPがついている – /var/lib/neutron/ROUTER_ID/stateに "master" と書かれて いる – keepalivedのプロセスが見える
- 34. 素朴な疑問 ● Q: セッションの引き継ぎ？ ● A: 今後conntrackdと組み合わせてやる予定 – Wikiにはconntrackdの設定テンプレートとかが 載っていていかにもできそうですが、まだできませ ん ● https://wiki.openstack.org/wiki/Neutron/L3_High_Avai lability_VRRP – Specには "Phase 2でやるよ" と書いてあります ● https://github.com/openstack/neutron-specs/blob/maste r/specs/juno/l3-high-availability.rst
- 35. 素朴な疑問 ● HAルータを2個作ると？ [root@juno-network1 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns} ip a; done => qrouter-767318c2-8dfa-4713-9018-27ef072383dd 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 12: ha-12860419-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:65:92:ed brd ff:ff:ff:ff:ff:ff inet 169.254.192.4/18 brd 169.254.255.255 scope global ha-12860419-7f valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe65:92ed/64 scope link valid_lft forever preferred_lft forever 13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff inet 172.16.99.101/24 scope global qg-6214b83a-d7 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:febc:8d2f/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-56c2e065-77 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe66:8d63/64 scope link valid_lft forever preferred_lft forever => qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-51456562-ca: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:7b:10:1d brd ff:ff:ff:ff:ff:ff inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-51456562-ca valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe7b:101d/64 scope link valid_lft forever preferred_lft forever 10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fee7:7d5c/64 scope link valid_lft forever preferred_lft forever 11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:feab:52f0/64 scope link valid_lft forever preferred_lft forever [root@juno-network2 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns} ip a; done => qrouter-767318c2-8dfa-4713-9018-27ef072383dd 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 12: ha-fe1f12b3-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ae:78:91 brd ff:ff:ff:ff:ff:ff inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-fe1f12b3-8c valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feae:7891/64 scope link valid_lft forever preferred_lft forever 13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:febc:8d2f/64 scope link valid_lft forever preferred_lft forever 14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff inet6 fe80::f816:3eff:fe66:8d63/64 scope link valid_lft forever preferred_lft forever => qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 9: ha-1dff8969-fc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:db:a4:68 brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-1dff8969-fc valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fedb:a468/64 scope link valid_lft forever preferred_lft forever 10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff inet 172.16.99.100/24 scope global qg-5f3287c0-e0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fee7:7d5c/64 scope link tentative dadfailed valid_lft forever preferred_lft forever 11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff inet 192.168.10.1/24 scope global qr-48ea3761-b0 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feab:52f0/64 scope link valid_lft forever preferred_lft forever
- 36. Packstackのハマりどころ ● Cinder – Bug 1164359 - KeyError: 'CONFIG_CINDER_VOLUMES_SIZE' ● https://bugzilla.redhat.com/show_bug.cgi?id=1164359 – Fix LVM Cinder Volume Creation ● https://review.openstack.org/#/c/131803/ – Avoid KeyError: 'CONFIG_CINDER_VOLUMES_SIZE' error ● https://review.openstack.org/#/c/136766/ ● mongodb – Bug 1167888 - [CENTOS7] packstack all in one install fails at MongoDB Service[mongodb]: Execution of '/usr/bin/systemctl start mongod' returned 1: Job for mongod.service failed ● https://bugzilla.redhat.com/show_bug.cgi?id=1167888 – mongodb.pp error when installing RDO on CentOS 7 ● https://ask.openstack.org/en/question/54015/mongodbpp-error-when-installing-rdo -on-centos-7/
- 37. SELinux問題 ● SELinux=enforcingの場合... – HA Routerを起動するとエラーになります 2014-11-29 15:28:11.800 2269 ERROR neutron.agent.l3_agent [-] Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter- 002a56ba-beda-43ad-a5d6-adff4dc1a725', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/002a56ba-beda- 43ad-a5d6-adff4dc1a725/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6- adff4dc1a725.pid', '-r', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid-vrrp'] Exit code: 99 Stdout: '' Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-002a56ba-beda-43ad-a5d6- adff4dc1a725 keepalived -P -f /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6- adff4dc1a725/keepalived.conf -p /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid -r /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6adff4dc1a725.pid-vrrp (no filter matched)n' "no filter matched" と言われている → 一見rootwrap関連の不具合にみえる → だが実は...
- 38. SELinux問題 ● SELinux関連が原因だったのでした – keepalivedの起動に失敗している – permissiveにすると問題なし # ausearch -m avc ---- time->Sat Nov 29 15:28:11 2014 type=SYSCALL msg=audit(1417242491.790:1992): arch=c000003e syscall=21 success=no exit=-13 a0=1b17a00 a1=1 a2=7fc4466fcf88 a3=7fff3728eb20 items=0 ppid=3390 pid=3391 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="neutron-rootwra" exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null) type=AVC msg=audit(1417242491.790:1992): avc: denied { execute } for pid=3391 comm="neutron-rootwra" name="keepalived" dev="dm-1" ino=9038320 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file (snip) avc: access vector cache
- 39. 参考文献 ● Blueprint – https://blueprints.launchpad.net/neutron/+spec/l3-high-availability ● Spec – https://github.com/openstack/neutron-specs/blob/master/specs/juno/l3-high-availability .rst ● Neutron/L3 High Availability VRRP – https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP ● Atlanta Summit: High Availability in Neutron – Getting the L3 Agent Right – https://www.youtube.com/watch?v=8kBPlIBq0Dc – http://techs.enovance.com/wp-content/uploads/2014/05/Neutron-L3-HA-Summit-eNova nce-template.pdf ● Paris Summit: Neutron Network Node High Availability – https://www.youtube.com/watch?v=go4fOYOUkmE – https://assafmuller.files.wordpress.com/2014/11/l3-ha.pdf ● Layer 3 High Availability – http://assafmuller.com/2014/08/16/layer-3-high-availability/ ● Keepalived Documentation – http://www.keepalived.org/documentation.html
