hl10502的博客

ceph-deploy源码分析(一)——源码结构与cli

ceph-deploy是部署Ceph集群的工具,可以通过SSH方式往远程主机上安装 Ceph 软件包、创建集群、增加监视器、收集(或销毁)密钥、增加 OSD 和元数据服务器、配置管理主机,甚至拆除集群。

ceph-deploy使用Python开发,GitHub为 https://github.com/ceph/ceph-deploy。本次源码分析的ceph-deploy版本为1.5.37。

源码结构

ceph-deploy-1.5.37源码目录

1
2
3
4
5
6
7
8
9
10
11
12
├── ceph_deploy # ceph-deploy源码目录
├── ceph_deploy.egg-info # egg-info目录
├── CONTRIBUTING.rst # 贡献指南
├── LICENSE # MIT LICENSE
├── MANIFEST.in # 打包规则
├── PKG-INFO # PKG-INFO文件
├── README.rst # ceph-deploy介绍
├── scripts # 启动脚本目录
├── setup.cfg # setup.py配置
├── setup.py # ceph-deploy安装脚本
├── tox.ini # 标准化测试
└── vendor.py

ceph_deploy源码目录文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
├── admin.py # 子命令admin模块,将ceph.conf和client.admin key push到远程主机
├── calamari.py # 子命令calamari模块,连接calamari master
├── cli.py # CLI入口
├── cliutil.py # 为装饰器函数增加priority
├── conf # ceph.conf与cephdeploy.conf读取与写入相关操作目录
├── config.py # 子命令config模块,push ceph.conf文件到远程主机;从远程主机 pull ceph.conf文件
├── connection.py # 连接本地主机、远程主机
├── exc.py # 异常处理Error
├── forgetkeys.py # 子命令forgetkeys模块,本地移除authentication keys
├── gatherkeys.py # 子命令gatherkeys模块,从mon主机上拉取authentication keys
├── hosts # ceph-deploy在不同操作系统(centos、debian、fedora、rhel、suse)的操作
├── __init__.py # 初始化版本号信息,当前版本1.5.37
├── install.py # 子命令install模块,安装、卸载ceph包,清除数据
├── lib # vendor类库
├── mds.py # 子命令mds模块,mds管理
├── misc.py # 其他工具类,比如:mon host组装tuples
├── mon.py # 子命令mon模块,mon管理
├── new.py # 子命令new模块,部署集群
├── osd.py # 子命令osd模块,osd管理
├── pkg.py # 子命令pkg模块,逗号分隔的包安装、卸载
├── repo.py # 子命令repo模块,添加 yum repo
├── rgw.py # 子命令rgw模块,rgw管理
├── tests # 测试文件目录
├── util # util目录
└── validate.py # 参数校验函数

源码入口

script目录下的ceph-deploy文件是ceph-deploy的入口,安装之后是/usr/bin/ceph-deploy。

ceph-deploy的__main__函数调用ceph_deploy.cli的main函数

1
2
3
4
5
6
...
...
from ceph_deploy.cli import main
if __name__ == '__main__':
sys.exit(main())

cli模块

cli.py是命令行操作模块。

main函数调用_main函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
def main(args=None, namespace=None):
try:
_main(args=args, namespace=namespace)
finally:
# This block is crucial to avoid having issues with
# Python spitting non-sense thread exceptions. We have already
# handled what we could, so close stderr and stdout.
if not os.environ.get('CEPH_DEPLOY_TEST'):
try:
sys.stdout.close()
except:
pass
try:
sys.stderr.close()
except:
pass

_main函数

  • 设置日志:Console Logger与File Logger添加到root_logger
  • 调用argparse模块,解析cli参数
  • 调用conf目录下的ceph-deploy模块set_overrides函数,从当前目录的cephdeploy.conf或~/.cephdeploy.conf文件获取ceph-deploy-global、ceph-deploy-[subcmd]配置项写入args
  • 调用执行subcmd相应的模块
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    @catches((KeyboardInterrupt, RuntimeError, exc.DeployError,), handle_all=True)
    def _main(args=None, namespace=None):
    # Set console logging first with some defaults, to prevent having exceptions
    # before hitting logging configuration. The defaults can/will get overridden
    # later.
    # Console Logger
    # 命令行控制台日志
    sh = logging.StreamHandler()
    # 不同级别的日志,使用不同的颜色区别:DEBUG蓝色;WARNIN黄色;ERROR红色;INFO白色
    sh.setFormatter(log.color_format())
    # 设置日志级别为WARNING
    sh.setLevel(logging.WARNING)
    # because we're in a module already, __name__ is not the ancestor of
    # the rest of the package; use the root as the logger for everyone
    # root_logger日志
    root_logger = logging.getLogger()
    # allow all levels at root_logger, handlers control individual levels
    # 设置root_logger日志级别为DEBUG
    root_logger.setLevel(logging.DEBUG)
    # 将 sh添加到root_logger
    root_logger.addHandler(sh)
    # 获取解析cli的argparse,调用argparse模块
    parser = get_parser()
    if len(sys.argv) < 2:
    parser.print_help()
    sys.exit()
    else:
    # 解析获取sys.argv中的ceph-deploy子命令和参数
    args = parser.parse_args(args=args, namespace=namespace)
    # 设置日志级别
    console_loglevel = logging.DEBUG # start at DEBUG for now
    if args.quiet:
    console_loglevel = logging.WARNING
    if args.verbose:
    console_loglevel = logging.DEBUG
    # Console Logger
    sh.setLevel(console_loglevel)
    # File Logger
    # 文件日志
    fh = logging.FileHandler('ceph-deploy-{cluster}.log'.format(cluster=args.cluster))
    fh.setLevel(logging.DEBUG)
    fh.setFormatter(logging.Formatter(log.FILE_FORMAT))
    # 将 fh添加到root_logger
    root_logger.addHandler(fh)
    # Reads from the config file and sets values for the global
    # flags and the given sub-command
    # the one flag that will never work regardless of the config settings is
    # logging because we cannot set it before hand since the logging config is
    # not ready yet. This is the earliest we can do.
    # 从当前目录的cephdeploy.conf或~/.cephdeploy.conf文件获取ceph-deploy配置覆盖命令行参数
    args = ceph_deploy.conf.cephdeploy.set_overrides(args)
    LOG.info("Invoked (%s): %s" % (
    ceph_deploy.__version__,
    ' '.join(sys.argv))
    )
    log_flags(args)
    # args.func为cli中的subcmd子命令,调用相应的模块
    return args.func(args)

get_parser函数

[ceph_deploy.cli]以以下方式配置:

模块名 = 模块包名:执行函数

比如: new = ceph_deploy.new:make
new作为ceph-deploy的子命令,执行ceph-deploy new命令时,执行make函数

其他的模块也类似:
mon = ceph_deploy.mon:make
osd = ceph_deploy.osd:make
rgw = ceph_deploy.rgw:make
mds = ceph_deploy.mds:make
config = ceph_deploy.config:make

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def get_parser():
# 调用argparse模块
parser = argparse.ArgumentParser(
prog='ceph-deploy',
formatter_class=argparse.RawDescriptionHelpFormatter,
description='Easy Ceph deployment\n\n%s' % __header__,
)
verbosity = parser.add_mutually_exclusive_group(required=False)
verbosity.add_argument(
'-v', '--verbose',
action='store_true', dest='verbose', default=False,
help='be more verbose',
)
verbosity.add_argument(
'-q', '--quiet',
action='store_true', dest='quiet',
help='be less verbose',
)
parser.add_argument(
'--version',
action='version',
version='%s' % ceph_deploy.__version__,
help='the current installed version of ceph-deploy',
)
parser.add_argument(
'--username',
help='the username to connect to the remote host',
)
parser.add_argument(
'--overwrite-conf',
action='store_true',
help='overwrite an existing conf file on remote host (if present)',
)
parser.add_argument(
'--cluster',
metavar='NAME',
help='name of the cluster',
type=validate.alphanumeric,
)
parser.add_argument(
'--ceph-conf',
dest='ceph_conf',
help='use (or reuse) a given ceph.conf file',
)
sub = parser.add_subparsers(
title='commands',
metavar='COMMAND',
help='description',
)
sub.required = True
# 获取ceph_deploy.cli下的entry_points
entry_points = [
(ep.name, ep.load())
for ep in pkg_resources.iter_entry_points('ceph_deploy.cli')
]
# 根据priority排序
entry_points.sort(
key=lambda name_fn: getattr(name_fn[1], 'priority', 100),
)
# 将模块加入到子命令
for (name, fn) in entry_points:
p = sub.add_parser(
name,
description=fn.__doc__,
help=fn.__doc__,
)
if not os.environ.get('CEPH_DEPLOY_TEST'):
p.set_defaults(cd_conf=ceph_deploy.conf.cephdeploy.load())
# flag if the default release is being used
p.set_defaults(default_release=False)
fn(p)
p.required = True
parser.set_defaults(
cluster='ceph',
)
return parser

argparse模块

cli命令的解析使用了argparse.py模块。argparse是Python标准库中命令行选项、参数和子命令的解析器,其是为替代已经过时的optparse模块,argparse在Python2.7中被引入。

argparse模块请参考
https://docs.python.org/2.7/library/argparse.html
http://python.usyiyi.cn/translate/python_278/library/argparse.html

set_overrides函数

conf目录下的ceph-deploy模块set_overrides函数

  • 调用load()函数
  • 判断ceph-deploy配置文件ceph-deploy-global、ceph-deploy-[subcommand]配置项,调用override_subcommand()函数写入args。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    def set_overrides(args, _conf=None):
    """
    Read the configuration file and look for ceph-deploy sections
    to set flags/defaults from the values found. This will alter the
    ``args`` object that is created by argparse.
    """
    # Get the subcommand name to avoid overwritting values from other
    # subcommands that are not going to be used
    subcommand = args.func.__name__
    command_section = 'ceph-deploy-%s' % subcommand
    # 加载ceph-deploy配置
    conf = _conf or load()
    for section_name in conf.sections():
    if section_name in ['ceph-deploy-global', command_section]:
    # ceph-deploy-global、ceph-deploy-[subcommand]配置项写入args
    override_subcommand(
    section_name,
    conf.items(section_name),
    args
    )
    return args

load函数,调用location()函数

1
2
3
4
5
def load():
parser = Conf()
# 读取解析ceph-deploy配置文件
parser.read(location())
return parser

location函数,调用_locate_or_create()函数

1
2
3
4
5
6
def location():
"""
Find and return the location of the ceph-deploy configuration file. If this
file does not exist, create one in a default location.
"""
return _locate_or_create()

_locate_or_create函数,判断当前目录的cephdeploy.conf或~/.cephdeploy.conf文件是否存在。

  • 如果都不存在则调用create_stub函数创建一个~/.cephdeploy.conf文件。这个文件是根据模板创建的,内容为空。
  • 如果存在(提前创建)cephdeploy.conf或~/.cephdeploy.conf文件,可以在文件中配置public_network、cluster_network、overwrite-conf等。
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    def _locate_or_create():
    home_config = path.expanduser('~/.cephdeploy.conf')
    # With order of importance
    locations = [
    path.join(os.getcwd(), 'cephdeploy.conf'),
    home_config,
    ]
    for location in locations:
    if path.exists(location):
    logger.debug('found configuration file at: %s' % location)
    return location
    logger.info('could not find configuration file, will create one in $HOME')
    create_stub(home_config)
    return home_config