23. CephFS - Jewel
• Single Active MDS,Active-Standby MDSs
• Single CephFS within a single Ceph Cluster
• CephFS requires at least kernel 3.10.x
• CephFS – Production Ready
• Experimental Features
• Multi Active MDSs
• Multiple CephFS file systems within a single Ceph Cluster
• Directory Fragmentation
38. CephFS测试分析-稳定性测试
• 读写数据模式
• 选择工具fio
# fio循环测试读写
while now < time
fio write 10G file
fio read 10G
file delete file
• 读写元数据模式
• 采用自写脚本,大规模创建目录、文件、写很小数据到文件中
# 百万级别的文件个数
while now < time
create dirs
touch files
write little data to each file
delete files
delete dirs
39. CephFS测试分析-稳定性测试
• 结论
• 几天的连续测试,CephFS一切正常
• 在上亿级别小文件的测试中,有些问题
• 问题与解决
• 日志中报“Behind on trimming”告警
调整参数 mds_log_max_expiring,mds_log_max_segments
• rm删除上亿文件时报“No space left on device”错误
调大参数 mds_bal_fragment_size_max,mds_max_purge_files,mds_max_purge_ops_per_pg
• 日志中报“_send skipping beacon, heartbeat map not healthy”
调大参数 mds_beacon_grace,mds_session_timeout,mds_reconnect_timeout
MDS log信息 -> 搜索相关Ceph代码 -> 分析原因 -> 调整参数
45. 展望 – Ceph Luminous
• Ceph Luminous (v12.2.0) - next long-term stable release series
1.The new BlueStore backend for ceph-osd is now stable and the new
default for newly created OSDs
2.Multiple active MDS daemons is now considered stable
3.CephFS directory fragmentation is now stable and enabled by default
4.Directory subtrees can be explicitly pinned to specific MDS daemons