hl10502的博客

MON数据恢复

MON的数据丢失或毁坏的情况下,使用ceph-monstore-tool工具来恢复MON。ceph-monstore-tool工具需要手动安装ceph-test包才有,例如ceph-10.2.7版本的ceph-test-10.2.7-0.el7.x86_64.rpm。

环境

  • CentOS Linux release 7.2.1511 (Core) Mini版
  • Jewel版ceph-10.2.7

测试步骤

毁坏数据

停止MON

1
[root@node201 ~]# systemctl stop ceph-mon.target

制造mon毁坏

1
[root@node201 ~]# mv /var/lib/ceph/mon/ceph-node201/ /var/lib/ceph/mon/ceph-node201_bak

OSD收集MON数据

创建临时目录

1
[root@node201 ~]# mkdir /tmp/mon-store

停止OSD

1
2
[root@node201 ~]# systemctl stop ceph-osd@0
[root@node201 ~]# systemctl stop ceph-osd@1

从OSD收集MON相关的数据,存放到/tmp/mon-store/。

  • 如果有多台服务器,需要依次将/tmp/mon-store/上传到这些服务器,从OSD收集MON相关的数据
  • 将最终的/tmp/mon-store/放到一台需要恢复的服务器上
    1
    2
    3
    4
    5
    6
    7
    [root@node201 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0/ --op update-mon-db --mon-store-path /tmp/mon-store/
    osd.0 : 0 osdmaps trimmed, 56 osdmaps added.
    64 pgs added.
    [root@node201 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --op update-mon-db --mon-store-path /tmp/mon-store/
    osd.1 : 0 osdmaps trimmed, 0 osdmaps added.
    0 pgs added.

重建MON数据

ceph-monstore-tool恢复MON数据

1
[root@node201 ~]# ceph-monstore-tool /tmp/mon-store rebuild

创建MON目录

1
[root@node201 ~]# mkdir /var/lib/ceph/mon/ceph-node201

copy MON数据文件

1
[root@node201 ~]# cp -ra /tmp/mon-store/* /var/lib/ceph/mon/ceph-node201

创建MON的done、systemd文件

1
2
[root@node201 ~]# touch /var/lib/ceph/mon/ceph-node201/done
[root@node201 ~]# touch /var/lib/ceph/mon/ceph-node201/systemd

重建kering

如果未开启cephx,那么不需要重建kering。开启cephx,需要做以下操作

查看client.admin.keyring

1
2
3
[root@node201 ~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg==

设置client.admin.keyring的caps

1
2
3
4
5
6
7
[root@node201 ~]# ceph-authtool /etc/ceph/ceph.client.admin.keyring -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'
[root@node201 ~]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg==
caps mds = "allow *"
caps mon = "allow *"
caps osd = "allow *"

创建MON keyring

1
2
3
4
5
6
[root@node201 ~]# ceph-authtool --create-keyring /var/lib/ceph/mon/ceph-node201/keyring --gen-key -n mon. --cap mon 'allow *'
creating /var/lib/ceph/mon/ceph-node201/keyring
[root@node201 ~]# cat /var/lib/ceph/mon/ceph-node201/keyring
[mon.]
key = AQBQoHVZRQ8KIBAA+Ml7P3qXIPw1h+xCFkzxWA==
caps mon = "allow *"

导入client.admin.keyring到MON的keyring

1
2
3
4
5
6
7
8
9
10
11
12
[root@node201 ~]# ceph-authtool /var/lib/ceph/mon/ceph-node201/keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /var/lib/ceph/mon/ceph-node201/keyring
[root@node201 ~]# cat /var/lib/ceph/mon/ceph-node201/keyring
[mon.]
key = AQBQoHVZRQ8KIBAA+Ml7P3qXIPw1h+xCFkzxWA==
caps mon = "allow *"
[client.admin]
key = AQB7snVZ1Ha/MRAAOmQNSoLuoep/qcbZiNvBIg==
auid = 0
caps mds = "allow *"
caps mon = "allow *"
caps osd = "allow *"

启动MON

设置MON的目录权限

1
[root@node201 ~]# chown ceph:ceph -R /var/lib/ceph/mon/

启动MON

1
[root@node201 ~]# systemctl start ceph-mon.target

参考

http://www.zphj1987.com/2016/09/20/Ceph%E7%9A%84Mon%E6%95%B0%E6%8D%AE%E9%87%8D%E6%96%B0%E6%9E%84%E5%BB%BA%E5%B7%A5%E5%85%B7/

http://letus.club/2017/03/30/ceph-rebuild-mon-data-from-osds/

https://gist.github.com/jingniao/839c6fe3b635a90b0fcfafa19c1b1557