5. 创建文件系统并启动。以下命令在监控节点进行！
#mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
遇以下问题：
（1）scp: /etc/ceph/ceph.conf: No such file or directory
[root@ceph_mds ceph]# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
[/usr/local/etc/ceph/fetch_config/tmp/fetched.ceph.conf.2693]
The authenticity of host 'ceph_mds (127.0.0.1)' can't be established.
RSA key fingerprint is a7:c8:b8:2e:86:ea:89:ff:11:93:e9:29:68:b5:7c:11.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph_mds' (RSA) to the list of known hosts.
ceph.conf 100% 4436 4.3KB/s 00:00
temp dir is /tmp/mkcephfs.tIHQnX8vkw
preparing monmap in /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool --create --clobber --add 0 222.31.76.178:6789 --print /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: monmap file /tmp/mkcephfs.tIHQnX8vkw/monmap
/usr/local/bin/monmaptool: generated fsid f998ee83-9eba-4de2-94e3-14f235ef840c
epoch 0
fsid f998ee83-9eba-4de2-94e3-14f235ef840c
last_changed 2013-05-31 08:22:52.972189
created 2013-05-31 08:22:52.972189
0: 222.31.76.178:6789/0 mon.0
/usr/local/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.tIHQnX8vkw/monmap (1 monitors)
=== osd.0 ===
pushing conf and monmap to ceph_osd0:/tmp/mkfs.ceph.0b3c65941572123eb704d9d614411fc1
scp: /etc/ceph/ceph.conf: No such file or directory
解决：编写一个脚本，将配置文件同步到/etc/ceph和/usr/local/etc/ceph目录下（需手动先建立/etc/ceph目录）：
[root@ceph_mds ceph]# cat cp_ceph_conf.sh
cp /usr/local/etc/ceph/ceph.conf/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.conf root@ceph_osd0:/usr/local/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.conf root@ceph_osd0:/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.conf root@ceph_osd1:/usr/local/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.conf root@ceph_osd1:/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.conf root@ceph_osd2:/usr/local/etc/ceph/ceph.conf
scp /usr/local/etc/ceph/ceph.confroot@ceph_osd2:/etc/ceph/ceph.conf
（2）
[root@ceph_mds ceph]# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
temp dir is /tmp/mkcephfs.hz1EcPJjtu
preparing monmap in /tmp/mkcephfs.hz1EcPJjtu/monmap
/usr/local/bin/monmaptool--create--clobber--add0222.31.76.178:6789--print/tmp/mkcephfs.hz1EcPJjtu/monmap
/usr/local/bin/monmaptool: monmap file/tmp/mkcephfs.hz1EcPJjtu/monmap
/usr/local/bin/monmaptool: generated fsid62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
epoch 0
fsid 62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
last_changed 2013-05-3108:39:48.198656
created 2013-05-3108:39:48.198656
0: 222.31.76.178:6789/0 mon.0
/usr/local/bin/monmaptool: writing epoch0 to/tmp/mkcephfs.hz1EcPJjtu/monmap (1 monitors)
=== osd.0===
pushing conf and monmap to ceph_osd0:/tmp/mkfs.ceph.2e991ed41f1cdca1149725615a96d0be
umount: /data/osd0:not mounted
umount: /dev/mapper/VolGroup-lv_ceph:not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-31 12:39:04.073438 7f02cd9ac760 -1 filestore(/data/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-3112:39:04.3620107f02cd9ac760-1 created object store/data/osd0 journal/data/osd.0/journalfor osd.0 fsid62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
2013-05-31 12:39:04.362074 7f02cd9ac760 -1 auth: error reading file: /data/osd0/keyring: can't open /data/osd0/keyring: (2) No such file or directory
2013-05-31 12:39:04.362280 7f02cd9ac760 -1 created new key in keyring /data/osd0/keyring
collecting osd.0 key
=== osd.1 ===
pushing conf and monmap to ceph_osd1:/tmp/mkfs.ceph.9a9f67ff6e7516b415d30f0a89bfe0dd
umount: /data/osd1: not mounted
umount: /dev/mapper/VolGroup-lv_ceph: not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-31 08:39:13.237718 7ff0a2fe4760 -1 filestore(/data/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-31 08:39:13.524175 7ff0a2fe4760 -1 created object store /data/osd1 journal /data/osd.1/journal for osd.1 fsid 62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
2013-05-31 08:39:13.524241 7ff0a2fe4760 -1 auth: error reading file: /data/osd1/keyring: can't open/data/osd1/keyring: (2) No such fileor directory
2013-05-3108:39:13.5244307ff0a2fe4760-1 created new keyin keyring/data/osd1/keyring
collecting osd.1 key
=== osd.2===
pushing conf and monmap to ceph_osd2:/tmp/mkfs.ceph.51a8af4b24b311fcc2d47eed2cd714ca
umount: /data/osd2:not mounted
umount: /dev/mapper/VolGroup-lv_ceph:not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-3109:01:49.3718537ff422eb1760-1 filestore(/data/osd2) couldnot find23c2fcde/osd_superblock/0//-1in index: (2) No such fileor directory
2013-05-3109:01:49.5830617ff422eb1760-1 created object store/data/osd2 journal/data/osd.2/journalfor osd.2 fsid62fdb8b1-8d98-42f2-9cef-b95e2ad7bd43
2013-05-3109:01:49.5831237ff422eb1760-1 auth: error reading file:/data/osd2/keyring: can't open /data/osd2/keyring: (2) No such file or directory
2013-05-31 09:01:49.583312 7ff422eb1760 -1 created new key in keyring /data/osd2/keyring
collecting osd.2 key
=== mds.alpha ===
creating private key for mds.alpha keyring /data/keyring.mds.alpha
creating /data/keyring.mds.alpha
bufferlist::write_file(/data/keyring.mds.alpha): failed to open file: (2) No such file or directory
could not write /data/keyring.mds.alpha
can't open /data/keyring.mds.alpha: can't open /data/keyring.mds.alpha: (2) No such file or directory
failed: '/usr/local/sbin/mkcephfs -d /tmp/mkcephfs.hz1EcPJjtu --init-daemon mds.alpha'
解决：手动建立这个文件：
#mkdir /data
#touch /data/keyring.mds.alpha
【创建成功】
[root@ceph_mds ceph]# mkcephfs -a -c /usr/local/etc/ceph/ceph.conf --mkbtrfs
temp dir is /tmp/mkcephfs.v9vb0zOmJ5
preparing monmap in /tmp/mkcephfs.v9vb0zOmJ5/monmap
/usr/local/bin/monmaptool--create--clobber--add0222.31.76.178:6789--print/tmp/mkcephfs.v9vb0zOmJ5/monmap
/usr/local/bin/monmaptool: monmap file/tmp/mkcephfs.v9vb0zOmJ5/monmap
/usr/local/bin/monmaptool: generated fsid652b09fb-bbbf-424c-bd49-8218d75465ba
epoch 0
fsid 652b09fb-bbbf-424c-bd49-8218d75465ba
last_changed 2013-05-3108:50:21.797571
created 2013-05-3108:50:21.797571
0: 222.31.76.178:6789/0 mon.0
/usr/local/bin/monmaptool: writing epoch0 to/tmp/mkcephfs.v9vb0zOmJ5/monmap (1 monitors)
=== osd.0===
pushing conf and monmap to ceph_osd0:/tmp/mkfs.ceph.8912ed2e34cfd2477c2549354c03faa3
umount: /dev/mapper/VolGroup-lv_ceph:not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-3112:49:36.5483297f67d293e760-1 journal check: ondisk fsid919417f1-0a79-4463-903c-3fc9df8ca0f8 doesn't match expected 3b3d2772-4981-46fd-bbcd-b11957c77d47, invalid (someone else's?) journal
2013-05-3112:49:36.9536667f67d293e760-1 filestore(/data/osd0) couldnot find23c2fcde/osd_superblock/0//-1in index: (2) No such fileor directory
2013-05-3112:49:37.2443347f67d293e760-1 created object store/data/osd0 journal/data/osd.0/journalfor osd.0 fsid652b09fb-bbbf-424c-bd49-8218d75465ba
2013-05-3112:49:37.2443977f67d293e760-1 auth: error reading file:/data/osd0/keyring: can't open /data/osd0/keyring: (2) No such file or directory
2013-05-31 12:49:37.244580 7f67d293e760 -1 created new key in keyring /data/osd0/keyring
collecting osd.0 key
=== osd.1 ===
pushing conf and monmap to ceph_osd1:/tmp/mkfs.ceph.69d388555243635efea3c5976d001b64
umount: /dev/mapper/VolGroup-lv_ceph: not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-31 08:49:45.012858 7f82a3d52760 -1 journal check: ondisk fsid 28f23b77-6f77-47b3-b946-7eda652d4488 doesn't match expected65a75a4f-b639-4eab-91d6-00c985118862, invalid (someone else's?) journal
2013-05-31 08:49:45.407962 7f82a3d52760 -1 filestore(/data/osd1) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-31 08:49:45.696990 7f82a3d52760 -1 created object store /data/osd1 journal /data/osd.1/journal for osd.1 fsid 652b09fb-bbbf-424c-bd49-8218d75465ba
2013-05-31 08:49:45.697052 7f82a3d52760 -1 auth: error reading file: /data/osd1/keyring: can't open/data/osd1/keyring: (2) No such fileor directory
2013-05-3108:49:45.6972387f82a3d52760-1 created new keyin keyring/data/osd1/keyring
collecting osd.1 key
=== osd.2===
pushing conf and monmap to ceph_osd2:/tmp/mkfs.ceph.686b9d63c840a05a6eed5b5781f10b27
umount: /dev/mapper/VolGroup-lv_ceph:not mounted
WARNING! - Btrfs Btrfs v0.20-rc1 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/mapper/VolGroup-lv_ceph
nodesize 4096 leafsize 4096 sectorsize 4096 size 100.00GB
Btrfs Btrfs v0.20-rc1
2013-05-3109:12:20.7087337fa54ae8f760-1 journal check: ondisk fsid dc21285e-3bde-4f53-9424-d059540ab920 doesn't match expected cae83f10-d633-48d1-b324-a64849eca974, invalid (someone else's?) journal
2013-05-3109:12:21.0571547fa54ae8f760-1 filestore(/data/osd2) couldnot find23c2fcde/osd_superblock/0//-1in index: (2) No such fileor directory
2013-05-3109:12:21.2536897fa54ae8f760-1 created object store/data/osd2 journal/data/osd.2/journalfor osd.2 fsid652b09fb-bbbf-424c-bd49-8218d75465ba
2013-05-3109:12:21.2537497fa54ae8f760-1 auth: error reading file:/data/osd2/keyring: can't open /data/osd2/keyring: (2) No such file or directory
2013-05-31 09:12:21.253931 7fa54ae8f760 -1 created new key in keyring /data/osd2/keyring
collecting osd.2 key
=== mds.alpha ===
creating private key for mds.alpha keyring /data/keyring.mds.alpha
creating /data/keyring.mds.alpha
Building generic osdmap from /tmp/mkcephfs.v9vb0zOmJ5/conf
/usr/local/bin/osdmaptool: osdmap file '/tmp/mkcephfs.v9vb0zOmJ5/osdmap'
/usr/local/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.v9vb0zOmJ5/osdmap
Generating admin key at /tmp/mkcephfs.v9vb0zOmJ5/keyring.admin
creating /tmp/mkcephfs.v9vb0zOmJ5/keyring.admin
Building initial monitor keyring
added entity mds.alpha auth auth(auid = 18446744073709551615 key=AQCXnKhRiL/QHhAA091/MQGD25V54smKBz959w== with 0 caps)
added entity osd.0 auth auth(auid = 18446744073709551615 key=AQDhK6hROEuRDhAA9uCsjB++Szh8sJy3CUgeoA== with 0 caps)
added entity osd.1 auth auth(auid = 18446744073709551615 key=AQBpnKhR0EKMKRAAzNWvZgkDWrPSuZaSttBdsw== with 0 caps)
added entity osd.2 auth auth(auid = 18446744073709551615 key=AQC1oahReP4fDxAAR0R0HTNfVbs6VMybLIU9qg== with 0 caps)
=== mon.0 ===
/usr/local/bin/ceph-mon: created monfs at /data/mon0 for mon.0
placing client.admin keyring in /etc/ceph/keyring
【启动】
#/etc/init.d/ceph -a start//必要时先关闭防火墙（#service iptables stop）
[root@ceph_mds ceph]# /etc/init.d/ceph -a start
=== mon.0===
Starting Ceph mon.0 on ceph_mds...
starting mon.0 rank 0 at 222.31.76.178:6789/0 mon_data /data/mon0 fsid652b09fb-bbbf-424c-bd49-8218d75465ba
=== mds.alpha===
Starting Ceph mds.alpha on ceph_mds...
starting mds.alpha at :/0
=== osd.0===
Mounting Btrfs on ceph_osd0:/data/osd0
Scanning for Btrfs filesystems
Starting Ceph osd.0 on ceph_osd0...
starting osd.0 at :/0 osd_data/data/osd0/data/osd.0/journal
=== osd.1===
Mounting Btrfs on ceph_osd1:/data/osd1
Scanning for Btrfs filesystems
Starting Ceph osd.1 on ceph_osd1...
starting osd.1 at :/0 osd_data/data/osd1/data/osd.1/journal
=== osd.2===
Mounting Btrfs on ceph_osd2:/data/osd2
Scanning for Btrfs filesystems
Starting Ceph osd.2 on ceph_osd2...
starting osd.2 at :/0 osd_data/data/osd2/data/osd.2/journal
【查看Ceph集群状态】
[root@ceph_mds ceph]# ceph -s
health HEALTH_OK
monmap e1: 1 mons at {0=222.31.76.178:6789/0}, election epoch 2, quorum 0 0
osdmap e7: 3 osds:3 up,3in
pgmap v432: 768 pgs:768 active+clean;9518 bytes data, 16876 KB used,293 GB/300 GB avail
mdsmap e4: 1/1/1 up {0=alpha=up:active}
[root@ceph_mds ceph]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
300M 293M 16876 0
POOLS:
NAME ID USED %USED OBJECTS
data 0 0 0 0
metadata 1 9518021
rbd 2 0 0 0
疑问：空间统计有问题吧?！"ceph -s"查看是300GB，"ceph df"查看是300M。
(责任编辑：IT) |
