hl10502的博客

CentOS7.2部署Jewel版Ceph-10.2.7

在CentOS7.2上安装部署Jewel版最新Ceph-10.2.7。

环境

每台主机

  • CentOS Linux release 7.2.1511 (Core) Mini版
  • 两个100G的磁盘做OSD
1
2
3
4
5
6
7
8
9
10
11
12
13
[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@localhost ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
xvda 202:0 0 10G 0 disk
├─xvda1 202:1 0 500M 0 part /boot
└─xvda2 202:2 0 9.5G 0 part
├─centos-root 253:0 0 8.5G 0 lvm /
└─centos-swap 253:1 0 1G 0 lvm [SWAP]
xvdb 202:16 0 100G 0 disk
xvdc 202:32 0 100G 0 disk

主机ceph-210作为管理节点,部署ceph-deploy。三台主机配置如下

主机 IP 安装组件
ceph-210 192.168.217.210 ceph-deploy、mon、osd、ntp
ceph-211 192.168.217.211 mon、osd、ntpdate
ceph-212 192.168.217.212 mon、osd、ntpdate

域名解析

修改每台主机的hostname

1
[root@localhost ~]# hostnamectl set-hostname [hostname]

hostname分别为ceph-210、ceph-211、ceph-212

修改/etc/hosts

1
[root@localhost ~]# vi /etc/hosts

添加
192.168.217.210 ceph-210
192.168.217.211 ceph-211
192.168.217.212 ceph-212

NTP 配置

NTP配置请参考 CentOS7.2配置NTP服务

ceph-210作为NTP服务端,ceph-211、ceph-212作为NTP客户端

yum源配置

在每台主机上执行。

删除默认的源

1
2
3
4
5
6
7
8
9
10
11
[root@localhost ~]# yum clean all
已加载插件:fastestmirror
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
正在清理软件源: Ceph Ceph-noarch base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
[root@localhost ~]# rm -rf /etc/yum.repos.d/*.repo

下载阿里云的base源

1
2
3
4
5
6
7
8
9
10
11
[root@localhost ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
--2017-06-07 14:26:39-- http://mirrors.aliyun.com/repo/Centos-7.repo
正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 112.124.140.210, 115.28.122.210
正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|112.124.140.210|:80... 已连接。
已发出 HTTP 请求,正在等待回应... 200 OK
长度:2573 (2.5K) [application/octet-stream]
正在保存至: “/etc/yum.repos.d/CentOS-Base.repo”
100%[===================================================================================================================================>] 2,573 --.-K/s 用时 0s
2017-06-07 14:26:39 (28.0 MB/s) - 已保存 “/etc/yum.repos.d/CentOS-Base.repo” [2573/2573])

下载阿里云的epel源

1
2
3
4
5
6
7
8
9
10
11
[root@localhost ~]# wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
--2017-06-07 14:27:08-- http://mirrors.aliyun.com/repo/epel-7.repo
正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 115.28.122.210, 112.124.140.210
正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|115.28.122.210|:80... 已连接。
已发出 HTTP 请求,正在等待回应... 200 OK
长度:1084 (1.1K) [application/octet-stream]
正在保存至: “/etc/yum.repos.d/epel.repo”
100%[===================================================================================================================================>] 1,084 --.-K/s 用时 0s
2017-06-07 14:27:08 (121 MB/s) - 已保存 “/etc/yum.repos.d/epel.repo” [1084/1084])

修改里面的系统版本为7.3.1611,当前用的CentOS7.2.1511版本的yum源已经清空了

1
2
3
[root@localhost ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/CentOS-Base.repo
[root@localhost ~]# sed -i '/aliyuncs/d' /etc/yum.repos.d/epel.repo
[root@localhost ~]# sed -i 's/$releasever/7.3.1611/g' /etc/yum.repos.d/CentOS-Base.repo

添加网易的ceph源(阿里云的ceph源没有最新的10.2.7版本)

1
[root@localhost ~]# vi /etc/yum.repos.d/ceph.repo

添加
[ceph]
name=ceph
baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.163.com/ceph/rpm-jewel/el7/noarch/
gpgcheck=0

进行yum的makecache

1
[root@localhost ~]# yum makecache

安装ceph

下载ceph的相关rpm到本地

1
[root@localhost ~]# yum install --downloadonly --downloaddir=/tmp/ceph ceph

在每台主机上安装ceph

1
[root@localhost ~]# yum localinstall -C -y --disablerepo=* /tmp/ceph/*.rpm

安装成功,查看ceph版本

1
2
[root@localhost yum.repos.d]# ceph -v
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)

关闭selinux

关闭每台主机的selinux

1
2
[root@localhost ~]# sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
[root@localhost ~]# setenforce 0

部署ceph

在管理节点ceph-210上执行。

配置无密码访问

生成本机密钥

1
[root@ceph-210 ~]# ssh-keygen -t rsa

将本机密钥copy到ceph-211、ceph-212

1
2
[root@ceph-210 ~]# ssh-copy-id root@ceph-211
[root@ceph-210 ~]# ssh-copy-id root@ceph-212

验证配置是否成功

1
2
[root@ceph-210 ~]# ssh root@ceph-211
[root@ceph-210 ~]# ssh root@ceph-212

安装ceph-deploy

下载ceph-deploy

1
[root@ceph-210 ~]#yum install --downloadonly --downloaddir=/tmp/ceph-deploy/ ceph-deploy

安装ceph-deploy

1
[root@ceph-210 ~]#yum localinstall -C -y --disablerepo=* /tmp/ceph-deploy/*.rpm

安装成功,查看ceph-deploy版本

1
2
[root@ceph-210 yum.repos.d]# ceph-deploy --version
1.5.37

部署集群

创建部署目录,部署集群

1
2
3
[root@ceph-210 ~]# mkdir ceph-cluster
[root@ceph-210 ~]# cd ceph-cluster
[root@ceph-210 ceph-cluster]# ceph-deploy new ceph-210 ceph-211 ceph-212

修改ceph.conf文件,添加 public_network=192.168.217.0/24

1
[root@ceph-210 ~]# vi ceph.conf

部署mon

1
[root@ceph-210 ceph-cluster]# ceph-deploy mon create-initial

创建osd

1
[root@ceph-210 ceph-cluster]# ceph-deploy --overwrite-conf osd prepare ceph-210:/dev/xvdb ceph-210:/dev/xvdc ceph-211:/dev/xvdb ceph-211:/dev/xvdc ceph-212:/dev/xvdb ceph-212:/dev/xvdc --zap-disk

激活osd

1
[root@ceph-210 ceph-cluster]# ceph-deploy --overwrite-conf osd activate ceph-210:/dev/xvdb1 ceph-210:/dev/xvdc1 ceph-211:/dev/xvdb1 ceph-211:/dev/xvdc1 ceph-212:/dev/xvdb1 ceph-212:/dev/xvdc1

查看集群

1
2
3
4
5
6
7
8
9
10
[root@ceph-210 ceph-cluster]# ceph -s
cluster b1945f0d-38df-4f17-83df-42b4749d2bed
health HEALTH_OK
monmap e1: 3 mons at {ceph-210=192.168.217.210:6789/0,ceph-211=192.168.217.211:6789/0,ceph-212=192.168.217.212:6789/0}
election epoch 10, quorum 0,1,2 ceph-210,ceph-211,ceph-212
osdmap e51: 6 osds: 6 up, 6 in
flags sortbitwise,require_jewel_osds
pgmap v214: 64 pgs, 1 pools, 0 bytes data, 0 objects
204 MB used, 569 GB / 569 GB avail
64 active+clean

参考

http://www.xuxiaopang.com/2016/10/09/ceph-quick-install-el7-jewel/

http://linux.it.net.cn/CentOS/course/2016/1213/25330.html