openstack 与 ceph (monitor初始化)-白红宇

Monitor 节点初始化

执行下面步骤

操作系统初始化
当前环境中, ceph 存储集群统一部署 centos 7.1 操作系统, 并在 /etc/hosts 中登记集群中的服务器主机名, 并完成时间同步

主机名验证

[root@hh-yun-ceph-cinder015-128055 ~]# hostnamehh-yun-ceph-cinder015-128055.vclound.com[root@hh-yun-ceph-cinder017-128057 ~]# hostnamehh-yun-ceph-cinder017-128057.vclound.com[root@hh-yun-ceph-cinder024-128074 ~]# hostnamehh-yun-ceph-cinder024-128074.vclound.com

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4::1         localhost localhost.localdomain localhost6 localhost6.localdomain6240.30.128.55 hh-yun-ceph-cinder015-128055.vclound.com hh-yun-ceph-cinder015-128055240.30.128.56 hh-yun-ceph-cinder016-128056.vclound.com hh-yun-ceph-cinder016-128056240.30.128.57 hh-yun-ceph-cinder017-128057.vclound.com hh-yun-ceph-cinder017-128057240.30.128.73 hh-yun-ceph-cinder023-128073.vclound.com hh-yun-ceph-cinder023-128073240.30.128.74 hh-yun-ceph-cinder024-128074.vclound.com hh-yun-ceph-cinder024-128074240.30.128.75 hh-yun-ceph-cinder025-128075.vclound.com hh-yun-ceph-cinder025-128075240.30.128.76 hh-yun-ceph-cinder026-128076.vclound.com hh-yun-ceph-cinder026-128076

时间同步验证

[root@hh-yun-ceph-cinder024-128074 ~]# chronyc trackingReference ID    : 10.199.129.21 (hh-yun-129021.vclound.com)Stratum         : 12Ref time (UTC)  : Tue Jul 21 07:45:32 2015System time     : 0.000002526 seconds fast of NTP timeLast offset     : 0.000005405 secondsRMS offset      : 0.007192081 secondsFrequency       : 18.428 ppm slowResidual freq   : 0.001 ppmSkew            : 0.060 ppmRoot delay      : 0.000141 secondsRoot dispersion : 0.011696 secondsUpdate interval : 386.4 secondsLeap status     : Normal

ceph 软件安装
注: 假如之前安装过 ceph, 并计划清除之前的集群配置, 那么执行下面命令可以把 ceph 配置进行安装

清除方法: ceph-deploy purgedata {node-name},     ceph-deploy purge {node-name}

yum 源

yum install -y epel-release.noarchecho "sslverify=0" >> /etc/yum.conf

ceph 软件安装

yum install -y ceph* bash-completion

创建配置文件, 默认是 /etc/ceph/ceph.conf
vim /etc/ceph/ceph.conf

[global]fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074mon host = 240.30.128.55, 240.30.128.57, 240.30.128.74public network = 240.30.128.0/21auth cluster required = cephxauth service required = cephxauth client required = cephxosd journal size = 1024filestore xattr use omap = trueosd pool default size = 2osd pool default min size = 1osd pool default pg num = 333osd pool default pgp num = 333osd crush chooseleaf type = 1

创建集群密钥对创建监控密钥

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'creating /etc/ceph/ceph.mon.keyring[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring[mon.]        key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==        caps mon = "allow *"

注意: ceph-0.87 版本需要把文件 /etc/ceph/ceph.mon.keyring 重命名为 /etc/ceph/ceph.keyring 或 /etc/ceph/keyring 或 /etc/ceph/ceph.mon.(hostname).keyring 否则无法执行注册

生成管理员密钥, 生成 client.admin 用户并增加用户密钥对

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'creating /etc/ceph/ceph.client.admin.keyring[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.client.admin.keyring[client.admin]        key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==        auid = 0        caps mds = "allow"        caps mon = "allow *"        caps osd = "allow *"

增加 client.admin 密钥对到 ceph.mon.keyring 中

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyringimporting contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring-rw------- 1 root root 214 7月  23 16:20 /etc/ceph/ceph.mon.keyring

注意, 这里 /etc/ceph/ceph.mon.keyring 从之前的 77 字节变成 214 字节, 并自动添加如下的内容

[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring[mon.]        key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==        caps mon = "allow *"[client.admin]        key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==        auid = 0        caps mds = "allow"        caps mon = "allow *"        caps osd = "allow *"

使用主机名, IP 地址及 FSID 生成 monitor map 保存到 /etc/ceph/monmap 注意, 命令创建时候需要一次过把所有的 mon 节点都直接进行定义

[root@hh-yun-ceph-cinder015-128055 tmp]#  monmaptool --create --add  hh-yun-ceph-cinder015-128055 240.30.128.55 --add hh-yun-ceph-cinder017-128057 240.30.128.57 --add hh-yun-ceph-cinder024-128074 240.30.128.74 --fsid dc4f91c1-8792-4948-b68f-2fcea75f53b9 /etc/ceph/monmapmonmaptool: monmap file /etc/ceph/monmapmonmaptool: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9monmaptool: writing epoch 0 to /etc/ceph/monmap (3 monitors)

[root@hh-yun-ceph-cinder015-128055 /]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055[root@hh-yun-ceph-cinder017-128057 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057[root@hh-yun-ceph-cinder024-128074 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074

定义监控进程与监控 MAP 及密钥对

[root@hh-yun-ceph-cinder015-128055 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder015-128055 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyringceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 for mon.hh-yun-ceph-cinder015-128055

把 /etc/ceph 目录下的文件: ceph.conf ceph.client.admin.keyring ceph.mon.keyring monmap 同步到另外两台 mon 节点中

[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder017-128057:/etc/ceph/.ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00ceph.conf                                                                                   100%  544     0.5KB/s   00:00ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00monmap                                                                                      100%  550     0.5KB/s   00:00[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder024-128074:/etc/ceph/.ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00ceph.conf                                                                                   100%  544     0.5KB/s   00:00ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00monmap                                                                                      100%  550     0.5KB/s   00:00

为另外两台 mon 节点进行监控数据初始化

[root@hh-yun-ceph-cinder017-128057 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder017-128057 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyringceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 for mon.hh-yun-ceph-cinder017-128057[root@hh-yun-ceph-cinder024-128074 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder024-128074 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyringceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074 for mon.hh-yun-ceph-cinder024-128074

分别启动三台 mon 节点

[root@hh-yun-ceph-cinder015-128055 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/sysvinit[root@hh-yun-ceph-cinder017-128057 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/sysvinit[root@hh-yun-ceph-cinder024-128074 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/sysvinit

[root@hh-yun-ceph-cinder015-128055 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder015-128055=== mon.hh-yun-ceph-cinder015-128055 ===Starting Ceph mon.hh-yun-ceph-cinder015-128055 on hh-yun-ceph-cinder015-128055...Running as unit run-53061.service.Starting ceph-create-keys on hh-yun-ceph-cinder015-128055...[root@hh-yun-ceph-cinder017-128057 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder017-128057=== mon.hh-yun-ceph-cinder017-128057 ===Starting Ceph mon.hh-yun-ceph-cinder017-128057 on hh-yun-ceph-cinder017-128057...Running as unit run-52736.service.Starting ceph-create-keys on hh-yun-ceph-cinder017-128057...[root@hh-yun-ceph-cinder024-128074 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder024-128074=== mon.hh-yun-ceph-cinder024-128074 ===Starting Ceph mon.hh-yun-ceph-cinder024-128074 on hh-yun-ceph-cinder024-128074...Running as unit run-51440.service.Starting ceph-create-keys on hh-yun-ceph-cinder024-128074...