hl10502的博客

ceph-deploy激活OSD出错 experimental feature 'rocksdb' is not enabled

使用 ceph-deploy --overwrite-conf osd prepare --bluestore 的OSD,在激活OSD时出错 experimental feature ‘rocksdb’ is not enabled ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ioG4dF: (5) Input/output error

环境

  • CentOS Linux release 7.2.1511 (Core)
  • Jewel版ceph-10.2.7
  • ceph-deploy-1.5.37

创建OSD成功

1
[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare --bluestore node201:/dev/sdb

激活OSD出错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[root@node201 ceph-cluster]# ceph-deploy --overwrite-conf osd activate node201:/dev/sdb1
...
...
[node201][WARNIN] command: Running command: /usr/bin/timeout 300 ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /var/lib/ceph/tmp/mnt.ioG4dF/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.ioG4dF --osd-uuid 110bea02-b6ff-4ae1-9a76-536933fec1a2 --keyring /var/lib/ceph/tmp/mnt.ioG4dF/keyring --setuser ceph --setgroup ceph
[node201][WARNIN] mount_activate: Failed to activate
[node201][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.ioG4dF
[node201][WARNIN] command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.ioG4dF
[node201][WARNIN] Traceback (most recent call last):
[node201][WARNIN] File "/usr/sbin/ceph-disk", line 9, in <module>
[node201][WARNIN] load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5047, in run
[node201][WARNIN] main(sys.argv[1:])
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4998, in main
[node201][WARNIN] args.func(args)
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3357, in main_activate
[node201][WARNIN] reactivate=args.reactivate,
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3114, in mount_activate
[node201][WARNIN] (osd_id, cluster) = activate(path, activate_key_template, init)
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3290, in activate
[node201][WARNIN] keyring=keyring,
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2756, in mkfs
[node201][WARNIN] '--setgroup', get_ceph_user(),
[node201][WARNIN] File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2720, in ceph_osd_mkfs
[node201][WARNIN] raise Error('%s failed : %s' % (str(arguments), error))
[node201][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'0', '--monmap', '/var/lib/ceph/tmp/mnt.ioG4dF/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.ioG4dF', '--osd-uuid', u'110bea02-b6ff-4ae1-9a76-536933fec1a2', '--keyring', '/var/lib/ceph/tmp/mnt.ioG4dF/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-07-20 17:15:38.195590 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
[node201][WARNIN] 2017-07-20 17:15:38.195842 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
[node201][WARNIN] 2017-07-20 17:15:38.195953 7ff0f5078800 -1 WARNING: experimental feature 'bluestore' is enabled
[node201][WARNIN] Please be aware that this feature is experimental, untested,
[node201][WARNIN] unsupported, and may result in data corruption, data loss,
[node201][WARNIN] and/or irreparable damage to your cluster. Do not use
[node201][WARNIN] feature with important data.
[node201][WARNIN]
[node201][WARNIN] 2017-07-20 17:15:38.219843 7ff0f5078800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore
[node201][WARNIN] 2017-07-20 17:15:38.229498 7ff0f5078800 -1 *** experimental feature 'rocksdb' is not enabled ***
[node201][WARNIN] This feature is marked as experimental, which means it
[node201][WARNIN] - is untested
[node201][WARNIN] - is unsupported
[node201][WARNIN] - may corrupt your data
[node201][WARNIN] - may break your cluster is an unrecoverable fashion
[node201][WARNIN] To enable this feature, add this to your ceph.conf:
[node201][WARNIN] enable experimental unrecoverable data corrupting features = rocksdb
[node201][WARNIN]
[node201][WARNIN] 2017-07-20 17:15:38.229514 7ff0f5078800 -1 bluestore(/var/lib/ceph/tmp/mnt.ioG4dF) _open_db error creating db
[node201][WARNIN] 2017-07-20 17:15:38.724290 7ff0f5078800 -1 OSD::mkfs: ObjectStore::mkfs failed with error -5
[node201][WARNIN] 2017-07-20 17:15:38.724341 7ff0f5078800 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ioG4dF: (5) Input/output error
[node201][WARNIN]
[node201][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/sdb1

解决

分析错误提示需要在ceph.conf文件中配置rocksdb

enable_experimental_unrecoverable_data_corrupting_features = bluestore rocksdb

修改ceph.conf之后,重启ceph,再重新激活

1
[root@node201 ceph-cluster]# systemctl restart ceph.target