hl10502的博客

ceph-deploy源码分析(三)——mon模块

ceph-deploy的mon.py模块是用来管理mon守护进程。

mon 子命令格式如下

1
ceph-deploy mon [-h] {add,create,create-initial,destroy} ...

  • create: 创建mon
  • add: 添加mon到集群,添加之前先在ceph.conf配置public_network
  • create-initial: 创建mon并初始化
  • destroy: 删除mon,如果集群只有一个mon则不能删除

在创建mon时,建议使用 ceph-deploy mon create-initial 来创建。

mon管理

make函数priority为30,子命令设置的默认函数为mon函数。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
@priority(30)
def make(parser):
"""
Ceph MON Daemon management
"""
parser.formatter_class = ToggleRawTextHelpFormatter
mon_parser = parser.add_subparsers(dest='subcommand')
mon_parser.required = True
mon_add = mon_parser.add_parser(
'add',
help=('R|Add a monitor to an existing cluster:\n'
'\tceph-deploy mon add node1\n'
'Or:\n'
'\tceph-deploy mon add --address 192.168.1.10 node1\n'
'If the section for the monitor exists and defines a `mon addr` that\n'
'will be used, otherwise it will fallback by resolving the hostname to an\n'
'IP. If `--address` is used it will override all other options.')
)
mon_add.add_argument(
'--address',
nargs='?',
)
mon_add.add_argument(
'mon',
nargs=1,
)
mon_create = mon_parser.add_parser(
'create',
help=('R|Deploy monitors by specifying them like:\n'
'\tceph-deploy mon create node1 node2 node3\n'
'If no hosts are passed it will default to use the\n'
'`mon initial members` defined in the configuration.')
)
mon_create.add_argument(
'--keyrings',
nargs='?',
help='concatenate multiple keyrings to be seeded on new monitors',
)
mon_create.add_argument(
'mon',
nargs='*',
)
mon_create_initial = mon_parser.add_parser(
'create-initial',
help=('Will deploy for monitors defined in `mon initial members`, '
'wait until they form quorum and then gatherkeys, reporting '
'the monitor status along the process. If monitors don\'t form '
'quorum the command will eventually time out.')
)
mon_create_initial.add_argument(
'--keyrings',
nargs='?',
help='concatenate multiple keyrings to be seeded on new monitors',
)
mon_destroy = mon_parser.add_parser(
'destroy',
help='Completely remove Ceph MON from remote host(s)'
)
mon_destroy.add_argument(
'mon',
nargs='+',
)
parser.set_defaults(
func=mon,
)

mon子命令

mon 函数,mon有四个subcmd为create、add、destroy、create-initial,分别对应mon_create、mon_add、mon_destroy、mon_create_initial四个函数

1
2
3
4
5
6
7
8
9
10
11
def mon(args):
if args.subcommand == 'create':
mon_create(args)
elif args.subcommand == 'add':
mon_add(args)
elif args.subcommand == 'destroy':
mon_destroy(args)
elif args.subcommand == 'create-initial':
mon_create_initial(args)
else:
LOG.error('subcommand %s not implemented', args.subcommand)

创建mon

命令行格式为: ceph-deploy mon create [node1] [node2] [node3] …

mon_create函数创建mon

  • args参数校验
  • 调用hosts.get函数获取操作系统版本信息,检查是否安装ceph包,目前支持的操作系统为centos/debian/fedora/rhel/suse,如果需要修改代码支持其他操作系统,可以从hosts入手修改,比如增加支持XenServer
  • 调用相应操作系统模块下的create函数创建mon,比如操作系统为centos,即hosts/centos下的mon模块
  • 调用mon_status函数检测mon状态
  • 调用catch_mon_errors函数获取mon的错误信息,写入logger
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    def mon_create(args):
    # 获取配置文件
    cfg = conf.ceph.load(args)
    if not args.mon:
    # 参数没指定mon,调用get_mon_initial_members函数从配置文件获取mon_initial_members作为mon
    args.mon = get_mon_initial_members(args, error_on_empty=True, _cfg=cfg)
    if args.keyrings:
    monitor_keyring = concatenate_keyrings(args)
    else:
    keyring_path = '{cluster}.mon.keyring'.format(cluster=args.cluster)
    try:
    # 获取ceph.mon.keyring文件信息
    monitor_keyring = files.read_file(keyring_path)
    except IOError:
    LOG.warning('keyring (%s) not found, creating a new one' % keyring_path)
    new_mon_keyring(args)
    monitor_keyring = files.read_file(keyring_path)
    LOG.debug(
    'Deploying mon, cluster %s hosts %s',
    args.cluster,
    ' '.join(args.mon),
    )
    errors = 0
    # 循环mon
    for (name, host) in mon_hosts(args.mon):
    try:
    # TODO add_bootstrap_peer_hint
    LOG.debug('detecting platform for host %s ...', name)
    # 获取操作系统版本信息,检查是否安装ceph包,如果需要修改代码支持其他操作系统,可以从hosts入手修改
    distro = hosts.get(
    host,
    username=args.username,
    callbacks=[packages.ceph_is_installed]
    )
    LOG.info('distro info: %s %s %s', distro.name, distro.release, distro.codename)
    rlogger = logging.getLogger(name)
    # ensure remote hostname is good to go
    hostname_is_compatible(distro.conn, rlogger, name)
    rlogger.debug('deploying mon to %s', name)
    # 创建mon,调用hosts目录的相应操作系列目录,比如系统是centos,那就是hosts/centos下的mon模块
    distro.mon.create(distro, args, monitor_keyring)
    # tell me the status of the deployed mon
    time.sleep(2) # give some room to start
    # 检测mon的状态
    mon_status(distro.conn, rlogger, name, args)
    # 获取mon的错误信息,写入logger
    catch_mon_errors(distro.conn, rlogger, name, cfg, args)
    distro.conn.exit()
    except RuntimeError as e:
    LOG.error(e)
    errors += 1
    if errors:
    raise exc.GenericError('Failed to create %d monitors' % errors)

以centos为例,hosts/centos下的mon模块的init.py

1
2
from ceph_deploy.hosts.common import mon_add as add # noqa
from ceph_deploy.hosts.common import mon_create as create # noqa

mon_create函数在hosts/common.py模块

  • 创建/etc/ceph/ceph.conf文件,从cph-deploy new创建的ceph.conf与args中写入数据到/etc/ceph/ceph.conf
  • 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring
  • 创建mon,初始化keyring数据
  • 创建空白的done文件,并将文件的拥有者设置成uid、gid
  • 创建init文件systemd,并将文件的拥有者设置成uid、gid
  • 设置mon服务开机启动,启动mon服务
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    def mon_create(distro, args, monitor_keyring):
    hostname = distro.conn.remote_module.shortname()
    logger = distro.conn.logger
    logger.debug('remote hostname: %s' % hostname)
    # mon目录,比如:/var/lib/ceph/mon/ceph-1
    path = paths.mon.path(args.cluster, hostname)
    # 获取/var/lib/ceph目录的用户id
    uid = distro.conn.remote_module.path_getuid(constants.base_path)
    # 获取/var/lib/ceph目录的用户组gid
    gid = distro.conn.remote_module.path_getgid(constants.base_path)
    # 获取mon下的done文件路径,比如:/var/lib/ceph/mon/ceph-1/done
    done_path = paths.mon.done(args.cluster, hostname)
    # 获取mon下的systemd文件路径,比如:/var/lib/ceph/mon/ceph-1/systemd
    init_path = paths.mon.init(args.cluster, hostname, distro.init)
    # 获取ceph-deploy创建的ceph.conf文件数据
    conf_data = conf.ceph.load_raw(args)
    # write the configuration file
    # 写入/etc/ceph/ceph.conf
    distro.conn.remote_module.write_conf(
    args.cluster,
    conf_data,
    args.overwrite_conf,
    )
    # if the mon path does not exist, create it
    # 如果mon文件目录不存在,创建mon目录,并将目录的拥有者改成uid、gid
    distro.conn.remote_module.create_mon_path(path, uid, gid)
    logger.debug('checking for done path: %s' % done_path)
    if not distro.conn.remote_module.path_exists(done_path):
    # done文件不存在
    logger.debug('done path does not exist: %s' % done_path)
    if not distro.conn.remote_module.path_exists(paths.mon.constants.tmp_path):
    # /var/lib/ceph/tmp目录不存在
    logger.info('creating tmp path: %s' % paths.mon.constants.tmp_path)
    # 创建/var/lib/ceph/tmp目录
    distro.conn.remote_module.makedir(paths.mon.constants.tmp_path)
    # 获取/var/lib/ceph/tmp/ceph.mon.keyring
    keyring = paths.mon.keyring(args.cluster, hostname)
    logger.info('creating keyring file: %s' % keyring)
    # 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring
    distro.conn.remote_module.write_monitor_keyring(
    keyring,
    monitor_keyring,
    uid, gid,
    )
    user_args = []
    if uid != 0:
    user_args = user_args + [ '--setuser', str(uid) ]
    if gid != 0:
    user_args = user_args + [ '--setgroup', str(gid) ]
    # 创建mon
    remoto.process.run(
    distro.conn,
    [
    'ceph-mon',
    '--cluster', args.cluster,
    '--mkfs',
    '-i', hostname,
    '--keyring', keyring,
    ] + user_args
    )
    logger.info('unlinking keyring file %s' % keyring)
    distro.conn.remote_module.unlink(keyring)
    # create the done file
    # 创建空白的done文件,并将文件的拥有者设置成uid、gid,表示mon创建完成
    distro.conn.remote_module.create_done_path(done_path, uid, gid)
    # create init path
    # 创建init文件,并将文件的拥有者设置成uid、gid
    distro.conn.remote_module.create_init_path(init_path, uid, gid)
    # start mon service
    # 启动mon服务
    start_mon_service(distro, args.cluster, hostname)

添加mon到集群

命令行格式为: ceph-deploy mon add [--address [ADDRESS]] mon …

mon_add函数

  • 调用admin模块admin函数(即ceph-deploy admin子命令),远程host上写入/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件
  • 调用操作系统类型相应的mon的add函数,创建mon
  • 调用catch_mon_errors函数校验error
  • 调用mon_status校验mon的状态
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    def mon_add(args):
    cfg = conf.ceph.load(args)
    # args.mon is a list with only one entry
    mon_host = args.mon[0]
    # 获取ceph.mon.keyring文件配置信息
    try:
    with open('{cluster}.mon.keyring'.format(cluster=args.cluster),
    'rb') as f:
    monitor_keyring = f.read()
    except IOError:
    raise RuntimeError(
    'mon keyring not found; run \'new\' to create a new cluster'
    )
    LOG.info('ensuring configuration of new mon host: %s', mon_host)
    args.client = args.mon
    # 调用admin模块admin函数,在远程host上写入/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件
    admin.admin(args)
    LOG.debug(
    'Adding mon to cluster %s, host %s',
    args.cluster,
    mon_host,
    )
    mon_section = 'mon.%s' % mon_host
    cfg_mon_addr = cfg.safe_get(mon_section, 'mon addr')
    # 校验mon ip
    if args.address:
    LOG.debug('using mon address via --address %s' % args.address)
    mon_ip = args.address
    elif cfg_mon_addr:
    LOG.debug('using mon address via configuration: %s' % cfg_mon_addr)
    mon_ip = cfg_mon_addr
    else:
    mon_ip = net.get_nonlocal_ip(mon_host)
    LOG.debug('using mon address by resolving host: %s' % mon_ip)
    try:
    LOG.debug('detecting platform for host %s ...', mon_host)
    distro = hosts.get(
    mon_host,
    username=args.username,
    callbacks=[packages.ceph_is_installed]
    )
    LOG.info('distro info: %s %s %s', distro.name, distro.release, distro.codename)
    rlogger = logging.getLogger(mon_host)
    # ensure remote hostname is good to go
    hostname_is_compatible(distro.conn, rlogger, mon_host)
    rlogger.debug('adding mon to %s', mon_host)
    args.address = mon_ip
    # 添加mon到集群
    distro.mon.add(distro, args, monitor_keyring)
    # tell me the status of the deployed mon
    time.sleep(2) # give some room to start
    # 获取mon的错误信息写入logger warning
    catch_mon_errors(distro.conn, rlogger, mon_host, cfg, args)
    # 校验mon的状态
    mon_status(distro.conn, rlogger, mon_host, args)
    distro.conn.exit()
    except RuntimeError as e:
    LOG.error(e)
    raise exc.GenericError('Failed to add monitor to host: %s' % mon_host)

add函数,以centos为例,在hosts/centos/mon/init.py

1
2
from ceph_deploy.hosts.common import mon_add as add # noqa
from ceph_deploy.hosts.common import mon_create as create # noqa

hosts/common.py模块的mon_add函数

  • 创建/etc/ceph/ceph.conf文件,从cph-deploy new创建的ceph.conf与args中写入数据到/etc/ceph/ceph.conf
  • 将ceph-deploy new创建的ceph.mon.keyring文件内容写入临时文件/var/lib/ceph/tmp/ceph.mon.keyring
  • 获取mon的monmap
  • 创建mon,初始化monmap和keyring数据
  • 创建空白的done文件,并将文件的拥有者设置成uid、gid
  • 创建init文件systemd,并将文件的拥有者设置成uid、gid
  • 设置mon服务开机启动,启动mon服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
def mon_add(distro, args, monitor_keyring):
hostname = distro.conn.remote_module.shortname()
logger = distro.conn.logger
path = paths.mon.path(args.cluster, hostname)
uid = distro.conn.remote_module.path_getuid(constants.base_path)
gid = distro.conn.remote_module.path_getgid(constants.base_path)
monmap_path = paths.mon.monmap(args.cluster, hostname)
done_path = paths.mon.done(args.cluster, hostname)
init_path = paths.mon.init(args.cluster, hostname, distro.init)
conf_data = conf.ceph.load_raw(args)
# write the configuration file
distro.conn.remote_module.write_conf(
args.cluster,
conf_data,
args.overwrite_conf,
)
# if the mon path does not exist, create it
distro.conn.remote_module.create_mon_path(path, uid, gid)
logger.debug('checking for done path: %s' % done_path)
if not distro.conn.remote_module.path_exists(done_path):
logger.debug('done path does not exist: %s' % done_path)
if not distro.conn.remote_module.path_exists(paths.mon.constants.tmp_path):
logger.info('creating tmp path: %s' % paths.mon.constants.tmp_path)
distro.conn.remote_module.makedir(paths.mon.constants.tmp_path)
keyring = paths.mon.keyring(args.cluster, hostname)
logger.info('creating keyring file: %s' % keyring)
distro.conn.remote_module.write_monitor_keyring(
keyring,
monitor_keyring,
uid, gid,
)
# get the monmap
remoto.process.run(
distro.conn,
[
'ceph',
'--cluster', args.cluster,
'mon',
'getmap',
'-o',
monmap_path,
],
)
# now use it to prepare the monitor's data dir
user_args = []
if uid != 0:
user_args = user_args + [ '--setuser', str(uid) ]
if gid != 0:
user_args = user_args + [ '--setgroup', str(gid) ]
remoto.process.run(
distro.conn,
[
'ceph-mon',
'--cluster', args.cluster,
'--mkfs',
'-i', hostname,
'--monmap',
monmap_path,
'--keyring', keyring,
] + user_args
)
logger.info('unlinking keyring file %s' % keyring)
distro.conn.remote_module.unlink(keyring)
# create the done file
distro.conn.remote_module.create_done_path(done_path, uid, gid)
# create init path
distro.conn.remote_module.create_init_path(init_path, uid, gid)
# start mon service
start_mon_service(distro, args.cluster, hostname)

删除mon

命令行格式为: ceph-deploy mon destroy [-h] mon [mon …]

mon_destroy函数,调用destroy_mon函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def mon_destroy(args):
errors = 0
for (name, host) in mon_hosts(args.mon):
try:
LOG.debug('Removing mon from %s', name)
distro = hosts.get(
host,
username=args.username,
callbacks=[packages.ceph_is_installed]
)
hostname = distro.conn.remote_module.shortname()
# 删除mon
destroy_mon(
distro.conn,
args.cluster,
hostname,
)
distro.conn.exit()
except RuntimeError as e:
LOG.error(e)
errors += 1
if errors:
raise exc.GenericError('Failed to destroy %d monitors' % errors)

destroy_mon函数

  • 从集群中删除mon
  • 停止mon服务
  • 将mon数据文件移动到/var/lib/ceph/mon-removed目录归档文件
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    def destroy_mon(conn, cluster, hostname):
    import datetime
    import time
    retries = 5
    # mon的目录,比如/var/lib/ceph/mon/ceph-node1
    path = paths.mon.path(cluster, hostname)
    if conn.remote_module.path_exists(path):
    # remove from cluster
    # 从集群中删除mon
    remoto.process.run(
    conn,
    [
    'ceph',
    '--cluster={cluster}'.format(cluster=cluster),
    '-n', 'mon.',
    '-k', '{path}/keyring'.format(path=path),
    'mon',
    'remove',
    hostname,
    ],
    timeout=7,
    )
    # stop
    if conn.remote_module.path_exists(os.path.join(path, 'upstart')) or system.is_upstart(conn):
    status_args = [
    'initctl',
    'status',
    'ceph-mon',
    'cluster={cluster}'.format(cluster=cluster),
    'id={hostname}'.format(hostname=hostname),
    ]
    elif conn.remote_module.path_exists(os.path.join(path, 'sysvinit')):
    status_args = [
    'service',
    'ceph',
    'status',
    'mon.{hostname}'.format(hostname=hostname),
    ]
    elif system.is_systemd(conn):
    # 停止mon服务
    status_args = [
    'systemctl',
    'stop',
    'ceph-mon@{hostname}.service'.format(hostname=hostname),
    ]
    else:
    raise RuntimeError('could not detect a supported init system, cannot continue')
    while retries:
    conn.logger.info('polling the daemon to verify it stopped')
    if is_running(conn, status_args):
    time.sleep(5)
    retries -= 1
    if retries <= 0:
    raise RuntimeError('ceph-mon deamon did not stop')
    else:
    break
    # archive old monitor directory
    fn = '{cluster}-{hostname}-{stamp}'.format(
    hostname=hostname,
    cluster=cluster,
    stamp=datetime.datetime.utcnow().strftime("%Y-%m-%dZ%H:%M:%S"),
    )
    # 创建/var/lib/ceph/mon-removed目录
    remoto.process.run(
    conn,
    [
    'mkdir',
    '-p',
    '/var/lib/ceph/mon-removed',
    ],
    )
    # 将mon数据文件移动到/var/lib/ceph/mon-removed目录归档文件
    conn.remote_module.make_mon_removed_dir(path, fn)

创建mon并初始化

命令行格式为: ceph-deploy mon create-initial [-h] [--keyrings [KEYRINGS]]

mon_create_initial函数

  • 调用mon_create函数,创建mon,可参考create部分
  • 判断集群中的mon_in_quorum与ceph.conf中的mon_initial_members是否完全匹配
  • 调用gatherkeys模块gatherkeys函数,收集用于配置新节点的keys
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    def mon_create_initial(args):
    # 获取ceph.conf中的mon_initial_members
    mon_initial_members = get_mon_initial_members(args, error_on_empty=True)
    # create them normally through mon_create
    args.mon = mon_initial_members
    # 创建mon
    mon_create(args)
    # make the sets to be able to compare late
    mon_in_quorum = set([])
    mon_members = set([host for host in mon_initial_members])
    for host in mon_initial_members:
    mon_name = 'mon.%s' % host
    LOG.info('processing monitor %s', mon_name)
    sleeps = [20, 20, 15, 10, 10, 5]
    tries = 5
    rlogger = logging.getLogger(host)
    distro = hosts.get(
    host,
    username=args.username,
    callbacks=[packages.ceph_is_installed]
    )
    while tries:
    # 获取mon的状态
    status = mon_status_check(distro.conn, rlogger, host, args)
    has_reached_quorum = status.get('state', '') in ['peon', 'leader']
    if not has_reached_quorum:
    LOG.warning('%s monitor is not yet in quorum, tries left: %s' % (mon_name, tries))
    tries -= 1
    sleep_seconds = sleeps.pop()
    LOG.warning('waiting %s seconds before retrying', sleep_seconds)
    time.sleep(sleep_seconds) # Magic number
    else:
    mon_in_quorum.add(host)
    LOG.info('%s monitor has reached quorum!', mon_name)
    break
    distro.conn.exit()
    # 集群中的mon_in_quorum与ceph.conf中的mon_initial_members完全匹配
    if mon_in_quorum == mon_members:
    LOG.info('all initial monitors are running and have formed quorum')
    LOG.info('Running gatherkeys...')
    # 调用gatherkeys模块gatherkeys函数,收集用于配置新节点的keys
    gatherkeys.gatherkeys(args)
    else:
    LOG.error('Some monitors have still not reached quorum:')
    for host in mon_members - mon_in_quorum:
    LOG.error('%s', host)
    raise SystemExit('cluster may not be in a healthy state')

手工管理mon

创建mon

以ceph-deploy创建mon:ceph-deploy mon create ceph-231为例,对应的手工操作。

生成fsid

1
2
[root@ceph-231 ~]# uuidgen
a3b9b0aa-01ab-4e1b-bba3-6f5317b0795b

创建ceph.conf

1
2
3
4
5
6
7
8
[root@ceph-231 ~]# vi /etc/ceph/ceph.conf
[global]
fsid = a3b9b0aa-01ab-4e1b-bba3-6f5317b0795b
mon_initial_members = ceph-231
mon_host = 192.168.217.231
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

创建ceph-ceph-231.mon.keyring

创建keyring

1
2
3
4
5
[root@ceph-231 ~]# ceph-authtool --create-keyring /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring --gen-key -n mon. --cap mon 'allow *'
[root@ceph-231 ~]# cat /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring
[mon.]
key = AQBzv0hZAAAAABAAJLiETzmegHWmVO7JwvsMdQ==
caps mon = "allow *"

改变文件拥有者为ceph

1
[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring

创建mon

获取ceph用户的uid、gid

1
2
[root@ceph-231 ~]# id ceph
uid=167(ceph) gid=167(ceph) groups=167(ceph)

创建mon

1
[root@ceph-231 ~]# ceph-mon --cluster ceph --mkfs -i ceph-231 --keyring /var/lib/ceph/tmp/ceph-ceph-231.mon.keyring --setuser 167 --setgroup 167

创建done文件

1
2
[root@ceph-231 ~]# touch /var/lib/ceph/mon/ceph-ceph-231/done
[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-231/done

创建init文件

查看init

1
2
[root@ceph-231 ~]# cat /proc/1/comm
systemd

创建systemd文件

1
2
[root@ceph-231 ~]# touch /var/lib/ceph/mon/ceph-ceph-231/systemd
[root@ceph-231 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-231/systemd

启动mon

1
2
3
[root@ceph-231 ~]# systemctl enable ceph.target
[root@ceph-231 ~]# systemctl enable ceph-mon@ceph-231
[root@ceph-231 ~]# systemctl start ceph-mon@ceph-231

查看mon状态

  • rank大于等于0,表示mon正在运行
  • rank等于-1,state为mon的状态
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    [root@ceph-231 ceph-ceph-231]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-231.asok mon_status
    {
    "name": "ceph-231",
    "rank": 0,
    "state": "leader",
    "election_epoch": 3,
    "quorum": [
    0
    ],
    "outside_quorum": [],
    "extra_probe_peers": [],
    "sync_provider": [],
    "monmap": {
    "epoch": 1,
    "fsid": "a3b9b0aa-01ab-4e1b-bba3-6f5317b0795b",
    "modified": "2017-06-20 15:53:44.533604",
    "created": "2017-06-20 15:53:44.533604",
    "mons": [
    {
    "rank": 0,
    "name": "ceph-231",
    "addr": "192.168.217.231:6789\/0"
    }
    ]
    }
    }

添加mon到集群

集群ceph-231上添加ceph-232这个mon。

ceph.conf添加public_network

修改ceph.conf文件,添加 public_network=192.168.217.0/24

1
[root@ceph-231 ~]# vi /etc/ceph/ceph.conf

copy配置与keyring

复制/etc/ceph/ceph.conf与/etc/ceph/ceph.client.admin.keyring文件

1
2
[root@ceph-231 ~]# scp /etc/ceph/ceph.conf root@ceph-232:/etc/ceph
[root@ceph-231 ~]# scp /etc/ceph/ceph.client.admin.keyring root@ceph-232:/etc/ceph

复制mon的keyring

1
[root@ceph-231 ~]# scp /var/lib/ceph/mon/ceph-ceph-231/keyring root@ceph-232:/var/lib/ceph/tmp/ceph-ceph-232.mon.keyring

设置keyring的用户权限

1
[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/tmp/ceph-ceph-232.mon.keyring

创建mon

获取ceph-232的monmap

1
[root@ceph-232 ~]# ceph --cluster ceph mon getmap -o /var/lib/ceph/tmp/ceph.ceph-232.monmap

获取用户ceph的uid、gid

1
2
[root@ceph-232 ~]# id ceph
uid=167(ceph) gid=167(ceph) groups=167(ceph)

创建ceph-232的mon

1
[root@ceph-232 ~]# ceph-mon --cluster ceph --mkfs -i ceph-232 --monmap /var/lib/ceph/tmp/ceph.ceph-232.monmap --keyring /var/lib/ceph/tmp/ceph-ceph-232.mon.keyring --setuser 167 --setgroup 167

创建done文件

1
2
[root@ceph-232 ~]# touch /var/lib/ceph/mon/ceph-ceph-232/done
[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-232/done

创建init文件

查看init

1
2
[root@ceph-232 ~]# cat /proc/1/comm
systemd

创建systemd文件

1
2
[root@ceph-232 ~]# touch /var/lib/ceph/mon/ceph-ceph-232/systemd
[root@ceph-232 ~]# chown ceph:ceph /var/lib/ceph/mon/ceph-ceph-232/systemd

启动mon

1
2
3
[root@ceph-232 ~]# systemctl enable ceph.target
[root@ceph-232 ~]# systemctl enable ceph-mon@ceph-232
[root@ceph-232 ~]# systemctl start ceph-mon@ceph-232

查看mon状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[root@ceph-232 ~]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-232.asok mon_status
{
"name": "ceph-232",
"rank": 1,
"state": "peon",
"election_epoch": 6,
"quorum": [
0,
1
],
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 4,
"fsid": "e1d69cc3-f40c-44c7-a758-fdfd4e5ef43e",
"modified": "2017-06-21 16:46:01.518137",
"created": "2017-06-21 16:16:34.913008",
"mons": [
{
"rank": 0,
"name": "ceph-231",
"addr": "192.168.217.231:6789\/0"
},
{
"rank": 1,
"name": "ceph-232",
"addr": "192.168.217.232:6789\/0"
}
]
}
}

删除mon

如果集群只有一个mon则不能删除mon

1
2
[root@ceph-232 ~]# ceph --cluster=ceph -n mon. -k /var/lib/ceph/mon/ceph-ceph-232/keyring mon remove ceph-232
removing mon.ceph-232 at 192.168.217.232:6789/0, there will be 1 monitors

创建mon并初始化

创建mon

创建mon与前面【创建mon】相同

初始化

初始化,创建keys

client.admin

1
2
3
4
5
6
7
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.admin
exported keyring for client.admin
[client.admin]
key = AQAEI0JZupXTFRAAmFF56vYMzKkzc5nxLit6mA==
caps mds = "allow *"
caps mon = "allow *"
caps osd = "allow *"

client.bootstrap-mds

1
2
3
4
5
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-mds
exported keyring for client.bootstrap-mds
[client.bootstrap-mds]
key = AQBjzklZ5D6oLRAAMSwQ169JjzNnPBMzIv6vCw==
caps mon = "allow profile bootstrap-mds"

如果client.bootstrap-mds在keyring中不存在,则创建

1
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-mds mon "allow profile bootstrap-mds"

client.bootstrap-osd

1
2
3
4
5
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-osd
exported keyring for client.bootstrap-osd
[client.bootstrap-osd]
key = AQBkzklZf/NpKhAAeJwteZDEGfZj66BUnbxC1Q==
caps mon = "allow profile bootstrap-osd"

如果client.bootstrap-osd在keyring中不存在,则创建

1
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-osd mon "allow profile bootstrap-osd"

client.bootstrap-rgw

1
2
3
4
5
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get client.bootstrap-rgw
exported keyring for client.bootstrap-rgw
[client.bootstrap-rgw]
key = AQBlzklZT3BEIRAA9w+/G+Sp6zJ+aTh+VQwTUQ==
caps mon = "allow profile bootstrap-rgw"

如果client.bootstrap-rgw在keyring中不存在,则创建

1
[root@ceph-231 ~]# /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-231/keyring auth get-or-create client.bootstrap-rgw mon "allow profile bootstrap-rgw"

mon文件结构

mon目录,比如/var/lib/ceph/mon/ceph-ceph-231有四个文件

  • done: 空白文件,是否创建完成的标志
  • keyring: mon的keys
  • store.db:mon数据库文件
  • systemd:空白文件,init的标志文件(如果Centos上的init为sysvinit,那么这个文件名称就是sysvinit)