hl10502的博客

CentOS7.2上Jewel版ceph-10.2.7 rbd map failed (6) No such device or address

rbd map错误信息

1
2
3
4
5
[root@node210 ~]# rbd map rbd/rbdtest
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (6) No such device or address

ceph集群dmesg查看

1
2
3
4
5
6
7
8
9
10
11
[root@ceph01 ceph]# dmesg | tail
[258857.936153] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
[258857.936165] libceph: mon0 192.168.212.201:6789 socket error on read
[258867.936151] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
[258867.936163] libceph: mon0 192.168.212.201:6789 socket error on read
[258877.936155] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
[258877.936167] libceph: mon0 192.168.212.201:6789 socket error on read
[258887.936153] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
[258887.936165] libceph: mon0 192.168.212.201:6789 socket error on read
[258897.936139] libceph: mon0 192.168.212.201:6789 feature set mismatch, my 4a042a42 < server's 2004a042a42, missing 20000000000
[258897.936150] libceph: mon0 192.168.212.201:6789 socket error on read

环境

  • CentOS Linux release 7.2.1511 (Core)
  • ceph version 10.2.7

CentOS7.2的内核版本3.10

1
2
[root@node210 ~]# uname -r
3.10.0-327.el7.x86_64

分析

ceph支持rbd块两种格式:

  • format 1:新建 rbd 映像时使用最初的格式。此格式兼容所有版本的 librbd 和内核模块,但是不支持较新的功能,像克隆。
  • format 2:使用第二版 rbd 格式, librbd 和 3.11 版以上内核模块才支持(除非是分拆的模块)。此格式增加了克隆支持,使得扩展更容易,还允许以后增加新功能。

Jewel版10.2.7新建rbd默认使用格式2,默认全部开启如下特性:

  • layering: 支持分层
  • striping: 支持条带化 v2
  • exclusive-lock: 支持独占锁
  • object-map: 支持对象映射(依赖 exclusive-lock )
  • fast-diff: 快速计算差异(依赖 object-map )
  • deep-flatten: 支持快照扁平化操作
  • journaling: 支持记录 IO 操作(依赖独占锁)

查看下支持了哪些特性

1
2
3
4
5
6
7
8
[root@node210 ~]# rbd info rbd/rbdtest
rbd image 'rbdtest':
size 102400 MB in 25600 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.fa6d2ae8944a
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:

CentOS7.2内核版本 3.10,仅支持此特性(layering),所以映射失败。其它特性需要使用更高版本内核,或者从新编译内核加载特性模块才行。

解决方法

对于当前rbd,直接disable这个rbd不支持的特性exclusive-lock、 object-map、 fast-diff、 deep-flatten

1
2
3
4
[root@node210 ~]# rbd feature disable rbd/rbdtest fast-diff
[root@node210 ~]# rbd feature disable rbd/rbdtest deep-flatten
[root@node210 ~]# rbd feature disable rbd/rbdtest object-map
[root@node210 ~]# rbd feature disable rbd/rbdtest exclusive-lock

再次查看rbd信息,只有layering特性

1
2
3
4
5
6
7
8
[root@node210 ~]# rbd info rbd/rbdtest
rbd image 'rbdtest':
size 102400 MB in 25600 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.fa6d2ae8944a
format: 2
features: layering
flags:

rbd map成功

1
2
[root@node210 ~]# rbd map rbd/rbdtest
/dev/rbd1

对于后续其他rbd的操作,可以使用以下方法:

1、创建rbd时使用 --image-feature 选项指定需要的特性,不用全部开启,开启layering即可。

1
[root@node210 ~]# rbd create rbdtest2 --size 100G --image-feature layering

2、修改/etc/ceph/ceph.conf,增加或修改以下配置

rbd default features = 3

只开启layering,禁用exclusive-lock、 object-map、fast-diff、 deep-flatten这些特性

参考

http://cephnotes.ksperis.com/blog/2014/01/21/feature-set-mismatch-error-on-ceph-kernel-client

http://www.zphj1987.com/2016/06/07/rbd%E6%97%A0%E6%B3%95map-rbd-feature-disable/