解决Lvs上行丢包的过程和收获

2. LVS
-

3. 100ms

4. 200ms

5. 10.11.128.29 qf.56.com - [25/Jul/2017:18:46:33 +0800] GET /chatpay/v1/
sendGiftHistory.do?roomId=7991332&callback=jsonp_callback_qf56com
sendGiftHistory HTTP/1.1 "200" 711 "https://qf.56.com/7991332" "Mozilla/
5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/45.0.2454.101 Safari/537.36" "60.6.202.75" "0.212"
"MISS" "0.212"
10.10.83.166 qf.56.com - [25/Jul/2017:18:46:33 +0800] GET /chatpay/v1/
sendGiftHistory.do?roomId=7991332&callback=jsonp_callback_qf56com
sendGiftHistory HTTP/1.1 "200" 711 "https://qf.56.com/7991332" "Mozilla/
5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/45.0.2454.101 Safari/537.36" "60.6.202.75,
10.11.128.29" "0.010" "10.10.193.228:8080" "0.009"

6. 200ms

7. 3 5

8. proxy -> vip:
location ^~ /chatpay/v1/xxx {
proxy_pass http://10.10.91.173;
}

9. netstat -s

10. 1w reset

11. ipvsadm -lnc
TCP 00:21 FIN_WAIT 10.10.91.126:57236 10.10.91.173:80 10.10.93.152:80
TCP 01:56 FIN_WAIT 10.10.83.166:37574 10.10.91.173:80 10.10.93.158:80

12. FIN_WAIT

13. LVS
tcpdump -i eth0 “ip[9]==4"

14. TCP Retransmission

15. TCP segment of a reassembled PDU

16. [@zw_91_126 /opt/nginx_logs]# cat access.log|grep "/chatpay/v1/
getMsg.do?ts=1500994855&withTop=1&callback=head_line_cb"
122.228.21.182 qf.56.com - [25/Jul/2017:23:01:59 +0800] GET /chatpay/
v1/getMsg.do?ts=1500994855&withTop=1&callback=head_line_cb
HTTP/1.1 "200" 442 "https://qf.56.com/1012148" "Mozilla/5.0 (Windows
NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87
Safari/537.36" "115.194.33.144" "0.209" "BYPASS" "0.209"

17. /etc/sysctl.conf
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 3
net.ipv4.tcp_keepalive_time = 30
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_timestamps = 1

18. /usr/sbin/ntpdate ntp.sohu.com > /dev/null 2>&1

19. lb_algo

20. RS
frame 200ms

21. RS mtu IPIP
/sbin/ifconfig tunl0 mtu 1440

23. Retransmission
mtu

24. LVS
LVS RS “ ”

25. ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on

26. LVS LRO/GRO
LVS LRO/GRO
sequence LVS LVS
MTU
IMCP

27. LVS LRO/GRO
ethtool -K eth0 lro off
ethtool -K eth0 gro off
ethtool -k eth0
……
generic-receive-offload: off
large-receive-offload: off
……

29. http://blog.csdn.net/hzsunshine/article/details/58638310
http://blog.csdn.net/wzyzzu/article/details/51381578
http://www.bo56.com/
%E4%BD%BF%E7%94%A8lvs%EF%BC%8C%E5%85%B3%E9%97%AD
%E7%BD%91%E5%8D%A1lrogro%E5%8A%9F%E8%83%BD/
http://blog.chinaunix.net/uid-20639775-id-3529535.html
http://man.linuxde.net/ethtool