Accueil > 1- Blog-Notes > Au boulot > Notes Susecon 2016 - Washington
Notes Susecon 2016 - Washington

Workshop CEPH
lundi 7 novembre 2016, par Olivier Duquesne aka DaffyDuke
CEPH, OpenSorce, Software Defined Storage, LTS de 12 mois, Suse Storage 3 basee sur Jewel release

jusqu qu 6 milliards de noeuds par cluster
CERN et toutes les distrib ou constructeurs sont membres du management CEPH
usual strategy is to scale up (vertical) ceph is for scaling out (horizontal)
suse is the first to ship a iscsi gateway wih ceph (supported)
RADOS = CEPH cluster, stockage objet
Librados = app pour acceder aux donnees (C, PHP, Java etc)
Radosgw = gateway REST co;\mpatible S3, Swift
RBD = block sur host qui heberge une instance
CEPH FS = fs pour un client
la gatway irbd presente les donnees en iscsi ou nfs etc

2 co;posants i;portants :
- un OSD pour manager les disques (one per path)
- MONS (3, $, ....), pour monitorer le cluster
Un algo CRUSH de stockage

Disk-> FileSystem (XFS) -> OSD
Les donnes sont stockees via algo CRUSH
Mimu;m de 3 MONS, utilise algo PAXOS pour monitorer les nodes et le reseau
=> i;pair absoluement
monitor map, contient placement group map, CRUSH map, epoch pour garantir integrite des donnes
rssayer de mettre le moins possible de mons
un client accede a la donnee objet en ;ode osd directement
PAXOS election de l ip la plus petite pour elire un MON
les objets sont ranges dans des storage pools
defaultpool rbd
regles de placement de snapshot par pool
l osd voit les placement group, chaque objet est associe a um placement group
il faut faire attention au nbre de PG

CRUSH
controllable
Replication
Under

les donnes ne sont pas centralisees
OSD run CRUSH => besoin de CPU

Step2, install, hardware voir la doc suse
ceph unprivileged, 167 if possible
DNS recommande
ipv4 tres bien
timesync
and at least 2 networks : pub for client/admin and priv for osd to osd
SES3 dispo en extension de SLES 12sp0
zypper in ceph-deploy and that s all folks
ceph-deploy install <node list>
ceph-deply new -cluster moncluster <node list>
ceph.conf contient les cles et le rep du journal

ceph-deploy mon create-initial <node list>
utiliser create pour ajouter des nodes MON

puis ajouter les OSD
ceph-deploy osd prepare node:dev

when check with
ceph -k ceph.client.admin.keyring status

ceph -w (watch)

geeko@ses-admin:~/Desktop> sudo su -

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

  #1) Respect the privacy of others.
  #2) Think before you type.
  #3) With great power comes great responsibility.

root's password:
ses-admin:~ # yast
yast yast2
ses-admin:~ # yast2
ses-admin:~ # useradd -m cephadm
ses-admin:~ # passwd cephadm
New password:
BAD PASSWORD: it is too short
BAD PASSWORD: is too simple
Retype new password:
passwd: password updated successfully
ses-admin:~ # visudo
visudo: /etc/sudoers.tmp unchanged
ses-admin:~ # visudo
ses-admin:~ # su – cephadm
su: user – does not exist
ses-admin:~ # su - cephadm
cephadm@ses-admin:~> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cephadm/.ssh/id_rsa):
Created directory '/home/cephadm/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cephadm/.ssh/id_rsa.
Your public key has been saved in /home/cephadm/.ssh/id_rsa.pub.
The key fingerprint is:
9e:b2:bd:31:6f:32:10:b6:7e:84:77:33:79:86:a3:e6 [MD5] cephadm@ses-admin
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| |
| o |
| . +S o |
| +.o.B o |
| ..+=o * |
| .+*+. |
| .+E=. |
+--[MD5]----------+
cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node1
The authenticity of host 'ses-node1 (192.168.100.11)' can't be established.
ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5].
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'cephadm@ses-node1'"
and check to make sure that only the key(s) you wanted were added.

cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node2
The authenticity of host 'ses-node2 (192.168.100.12)' can't be established.
ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5].
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'cephadm@ses-node2'"
and check to make sure that only the key(s) you wanted were added.

cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node3
The authenticity of host 'ses-node3 (192.168.100.13)' can't be established.
ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5].
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh 'cephadm@ses-node3'"
and check to make sure that only the key(s) you wanted were added.

cephadm@ses-admin:~> ssh ses-node3 id
uid=1001(cephadm) gid=100(users) groups=100(users)
cephadm@ses-admin:~> exit
logout
ses-admin:~ # zypper in ceph ceph-deploy
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 42 NEW packages are going to be installed:
  FastCGI babeltrace ceph ceph-base ceph-common ceph-deploy ceph-mds ceph-mon ceph-osd gperftools
  libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1
  libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-Flask
  python-Jinja2 python-MarkupSafe python-Werkzeug python-apipkg python-argparse python-cephfs python-ecdsa
  python-execnet python-itsdangerous python-paramiko python-pushy python-pycrypto python-rados python-rbd
  python-remoto python-requests supportutils-plugin-ses

The following package is not supported by its vendor:
  libxio

42 new packages to install.
Overall download size: 48.7 MiB. Already cached: 0 B. After the operation, additional 211.7 MiB will be used.
Continue? [y/n/? shows all options] (y): y
Retrieving package python-Werkzeug-0.9.6-5.2.noarch (1/42), 483.9 KiB ( 2.2 MiB unpacked)
Retrieving package python-apipkg-1.2-1.2.noarch (2/42), 9.2 KiB ( 18.2 KiB unpacked)
Retrieving package python-ecdsa-0.13-2.1.noarch (3/42), 77.4 KiB (261.8 KiB unpacked)
Retrieving package python-itsdangerous-0.24-2.1.noarch (4/42), 20.2 KiB ( 69.1 KiB unpacked)
Retrieving package supportutils-plugin-ses-1.0+git.1463485157.269428c-4.1.noarch
  (5/42), 13.7 KiB ( 26.0 KiB unpacked)
Retrieving package FastCGI-2.4.0-167.1.x86_64 (6/42), 184.8 KiB (587.6 KiB unpacked)
Retrieving package libboost_random1_54_0-1.54.0-13.1.x86_64 (7/42), 20.0 KiB ( 10.3 KiB unpacked)
Retrieving package libleveldb1-1.18-1.3.x86_64 (8/42), 132.8 KiB (361.3 KiB unpacked)
Retrieving package python-MarkupSafe-0.18-7.1.x86_64 (9/42), 24.6 KiB ( 66.0 KiB unpacked)
Retrieving package python-pycrypto-2.6.1-4.1.x86_64 (10/42), 371.5 KiB ( 2.0 MiB unpacked)
Retrieving package python-execnet-1.4.1-1.3.noarch (11/42), 73.7 KiB (249.5 KiB unpacked)
Retrieving package python-Jinja2-2.7.3-17.1.noarch (12/42), 278.5 KiB ( 1.7 MiB unpacked)
Retrieving package python-paramiko-1.15.2-1.1.noarch (13/42), 860.0 KiB ( 5.1 MiB unpacked)
Retrieving package python-remoto-0.0.27-1.1.noarch (14/42), 80.6 KiB (298.2 KiB unpacked)
Retrieving package python-Flask-0.10.1-2.1.noarch (15/42), 178.0 KiB (827.7 KiB unpacked)
Retrieving package python-pushy-0.5.3-1.5.noarch (16/42), 58.3 KiB (238.5 KiB unpacked)
Retrieving package python-argparse-1.2.1-15.8.noarch (17/42), 248.6 KiB (490.1 KiB unpacked)
Retrieving package python-requests-2.3.0-6.5.2.noarch (18/42), 334.6 KiB ( 1.7 MiB unpacked)
Retrieving package babeltrace-1.2.4-3.2.x86_64 (19/42), 159.0 KiB (463.9 KiB unpacked)
Retrieving package gperftools-2.2-2.2.x86_64 (20/42), 554.3 KiB ( 2.2 MiB unpacked)
Retrieving package libboost_iostreams1_54_0-1.54.0-13.1.x86_64 (21/42), 39.8 KiB (103.1 KiB unpacked)
Retrieving package libboost_program_options1_54_0-1.54.0-13.1.x86_64 (22/42), 125.5 KiB (456.7 KiB unpacked)
Retrieving package libibverbs1-1.1.7-12.2.x86_64 (23/42), 34.1 KiB ( 71.4 KiB unpacked)
Retrieving package liburcu0-0.8.8-1.2.x86_64 (24/42), 49.4 KiB (182.6 KiB unpacked)
Retrieving package librdmacm1-1.0.18.1-1.14.x86_64 (25/42), 53.4 KiB (129.1 KiB unpacked)
Retrieving package lttng-ust-2.7.0-1.2.x86_64 (26/42), 171.4 KiB (754.7 KiB unpacked)
Retrieving package ceph-deploy-1.5.32+git.1464082518.5aab689-1.1.noarch (27/42), 174.7 KiB (740.3 KiB unpacked)
Retrieving package libxio-1.6+git.1461071064.e0ed264-2.1.x86_64 (28/42), 242.5 KiB (885.9 KiB unpacked)
Retrieving package librados2-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (29/42), 1.7 MiB ( 6.3 MiB unpacked)
Retrieving package libcephfs1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (30/42), 1.8 MiB ( 6.0 MiB unpacked)
Retrieving package python-rados-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (31/42), 209.0 KiB (493.4 KiB unpacked)
Retrieving package librgw2-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (32/42), 2.7 MiB ( 10.3 MiB unpacked)
Retrieving package librbd1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (33/42), 2.1 MiB ( 7.5 MiB unpacked)
Retrieving package libradosstriper1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64
  (34/42), 1.8 MiB ( 6.9 MiB unpacked)
Retrieving package python-cephfs-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64
  (35/42), 138.1 KiB (227.4 KiB unpacked)
Retrieving package python-rbd-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (36/42), 133.1 KiB (184.9 KiB unpacked)
Retrieving package ceph-common-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (37/42), 14.9 MiB ( 67.1 MiB unpacked)
Retrieving package ceph-base-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (38/42), 3.9 MiB ( 33.1 MiB unpacked)
Retrieving package ceph-osd-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (39/42), 9.0 MiB ( 33.2 MiB unpacked)
Retrieving package ceph-mon-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (40/42), 2.7 MiB ( 9.3 MiB unpacked)
Retrieving package ceph-mds-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (41/42), 2.7 MiB ( 9.3 MiB unpacked)
Retrieving package ceph-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (42/42), 79.1 KiB ( 0 B unpacked)
Checking for file conflicts: .............................................................................[done]
( 1/42) Installing: python-Werkzeug-0.9.6-5.2 ............................................................[done]
( 2/42) Installing: python-apipkg-1.2-1.2 ................................................................[done]
( 3/42) Installing: python-ecdsa-0.13-2.1 ................................................................[done]
( 4/42) Installing: python-itsdangerous-0.24-2.1 .........................................................[done]
( 5/42) Installing: supportutils-plugin-ses-1.0+git.1463485157.269428c-4.1 ...............................[done]
( 6/42) Installing: FastCGI-2.4.0-167.1 ..................................................................[done]
( 7/42) Installing: libboost_random1_54_0-1.54.0-13.1 ....................................................[done]
( 8/42) Installing: libleveldb1-1.18-1.3 .................................................................[done]
( 9/42) Installing: python-MarkupSafe-0.18-7.1 ...........................................................[done]
(10/42) Installing: python-pycrypto-2.6.1-4.1 ............................................................[done]
(11/42) Installing: python-execnet-1.4.1-1.3 .............................................................[done]
(12/42) Installing: python-Jinja2-2.7.3-17.1 .............................................................[done]
(13/42) Installing: python-paramiko-1.15.2-1.1 ...........................................................[done]
(14/42) Installing: python-remoto-0.0.27-1.1 .............................................................[done]
(15/42) Installing: python-Flask-0.10.1-2.1 ..............................................................[done]
(16/42) Installing: python-pushy-0.5.3-1.5 ...............................................................[done]
(17/42) Installing: python-argparse-1.2.1-15.8 ...........................................................[done]
(18/42) Installing: python-requests-2.3.0-6.5.2 ..........................................................[done]
(19/42) Installing: babeltrace-1.2.4-3.2 .................................................................[done]
(20/42) Installing: gperftools-2.2-2.2 ...................................................................[done]
(21/42) Installing: libboost_iostreams1_54_0-1.54.0-13.1 .................................................[done]
(22/42) Installing: libboost_program_options1_54_0-1.54.0-13.1 ...........................................[done]
(23/42) Installing: libibverbs1-1.1.7-12.2 ...............................................................[done]
(24/42) Installing: liburcu0-0.8.8-1.2 ...................................................................[done]
(25/42) Installing: librdmacm1-1.0.18.1-1.14 .............................................................[done]
(26/42) Installing: lttng-ust-2.7.0-1.2 ..................................................................[done]
(27/42) Installing: ceph-deploy-1.5.32+git.1464082518.5aab689-1.1 ........................................[done]
(28/42) Installing: libxio-1.6+git.1461071064.e0ed264-2.1 ................................................[done]
(29/42) Installing: librados2-10.2.1+git.1464009581.b8ee9e4-1.1 ..........................................[done]
(30/42) Installing: libcephfs1-10.2.1+git.1464009581.b8ee9e4-1.1 .........................................[done]
(31/42) Installing: python-rados-10.2.1+git.1464009581.b8ee9e4-1.1 .......................................[done]
(32/42) Installing: librgw2-10.2.1+git.1464009581.b8ee9e4-1.1 ............................................[done]
(33/42) Installing: librbd1-10.2.1+git.1464009581.b8ee9e4-1.1 ............................................[done]
(34/42) Installing: libradosstriper1-10.2.1+git.1464009581.b8ee9e4-1.1 ...................................[done]
(35/42) Installing: python-cephfs-10.2.1+git.1464009581.b8ee9e4-1.1 ......................................[done]
(36/42) Installing: python-rbd-10.2.1+git.1464009581.b8ee9e4-1.1 .........................................[done]
(37/42) Installing: ceph-common-10.2.1+git.1464009581.b8ee9e4-1.1 ........................................[done]
(38/42) Installing: ceph-base-10.2.1+git.1464009581.b8ee9e4-1.1 ..........................................[done]
Additional rpm output:
Updating /etc/sysconfig/ceph...

(39/42) Installing: ceph-osd-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done]
(40/42) Installing: ceph-mon-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done]
(41/42) Installing: ceph-mds-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done]
(42/42) Installing: ceph-10.2.1+git.1464009581.b8ee9e4-1.1 ...............................................[done]
ses-admin:~ # su – cephadm
su: user – does not exist
ses-admin:~ # su - cephadm
cephadm@ses-admin:~> ls
bin public_html
cephadm@ses-admin:~> ceph-deploy install ses-node1 ses-node2 ses-node3
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy install ses-node1 ses-node2 ses-node3
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f12cbf6cef0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : True
[ceph_deploy.cli][INFO ] func : <function install at 0x7f12cc1b0668>
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['ses-node1', 'ses-node2', 'ses-node3']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version infernalis on cluster ceph hosts ses-node1 ses-node2 ses-node3
[ceph_deploy.install][DEBUG ] Detecting platform for host ses-node1 ...
[ses-node1][DEBUG ] connection detected need for sudo
[ses-node1][DEBUG ] connected to host: ses-node1
[ses-node1][DEBUG ] detect platform information from remote host
[ses-node1][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[ses-node1][INFO ] installing Ceph on ses-node1
[ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[ses-node1][DEBUG ]
[ses-node1][DEBUG ] The following 35 NEW packages are going to be installed:
[ses-node1][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[ses-node1][DEBUG ]
[ses-node1][DEBUG ] The following package is not supported by its vendor:
[ses-node1][DEBUG ] libxio
[ses-node1][DEBUG ]
[ses-node1][DEBUG ] 35 new packages to install.
[ses-node1][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[ses-node1][DEBUG ] Continue? [y/n/? shows all options] (y): y
[ses-node1][INFO ] Running command: sudo ceph --version
[ses-node1][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
[ceph_deploy.install][DEBUG ] Detecting platform for host ses-node2 ...
[ses-node2][DEBUG ] connection detected need for sudo
[ses-node2][DEBUG ] connected to host: ses-node2
[ses-node2][DEBUG ] detect platform information from remote host
[ses-node2][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[ses-node2][INFO ] installing Ceph on ses-node2
[ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[ses-node2][DEBUG ]
[ses-node2][DEBUG ] The following 35 NEW packages are going to be installed:
[ses-node2][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[ses-node2][DEBUG ]
[ses-node2][DEBUG ] The following package is not supported by its vendor:
[ses-node2][DEBUG ] libxio
[ses-node2][DEBUG ]
[ses-node2][DEBUG ] 35 new packages to install.
[ses-node2][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[ses-node2][DEBUG ] Continue? [y/n/? shows all options] (y): y
[ses-node2][INFO ] Running command: sudo ceph --version
[ses-node2][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
[ceph_deploy.install][DEBUG ] Detecting platform for host ses-node3 ...
[ses-node3][DEBUG ] connection detected need for sudo
[ses-node3][DEBUG ] connected to host: ses-node3
[ses-node3][DEBUG ] detect platform information from remote host
[ses-node3][DEBUG ] detect machine type
[ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[ses-node3][INFO ] installing Ceph on ses-node3
[ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[ses-node3][DEBUG ]
[ses-node3][DEBUG ] The following 35 NEW packages are going to be installed:
[ses-node3][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[ses-node3][DEBUG ]
[ses-node3][DEBUG ] The following package is not supported by its vendor:
[ses-node3][DEBUG ] libxio
[ses-node3][DEBUG ]
[ses-node3][DEBUG ] 35 new packages to install.
[ses-node3][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[ses-node3][DEBUG ] Continue? [y/n/? shows all options] (y): y
[ses-node3][INFO ] Running command: sudo ceph --version
[ses-node3][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
cephadm@ses-admin:~> ls
bin ceph-deploy-ceph.log public_html
cephadm@ses-admin:~> cat ceph-deploy-ceph.log
[2016-11-07 11:37:32,854][ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy install ses-node1 ses-node2 ses-node3
[2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] ceph-deploy options:
[2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] verbose : False
[2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] testing : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f12cbf6cef0>
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] cluster : ceph
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] dev_commit : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_mds : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] stable : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] default_release : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] username : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] adjust_repos : True
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] func : <function install at 0x7f12cc1b0668>
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_all : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] repo : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] host : ['ses-node1', 'ses-node2', 'ses-node3']
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_rgw : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_tests : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] repo_url : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] ceph_conf : None
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_osd : False
[2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] version_kind : stable
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] install_common : False
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] overwrite_conf : False
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] quiet : False
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] dev : master
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] local_mirror : None
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] release : None
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] install_mon : False
[2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] gpg_url : None
[2016-11-07 11:37:32,856][ceph_deploy.install][DEBUG ] Installing stable version infernalis on cluster ceph hosts ses-node1 ses-node2 ses-node3
[2016-11-07 11:37:32,856][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node1 ...
[2016-11-07 11:37:33,075][ses-node1][DEBUG ] connection detected need for sudo
[2016-11-07 11:37:33,221][ses-node1][DEBUG ] connected to host: ses-node1
[2016-11-07 11:37:33,221][ses-node1][DEBUG ] detect platform information from remote host
[2016-11-07 11:37:33,256][ses-node1][DEBUG ] detect machine type
[2016-11-07 11:37:33,260][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[2016-11-07 11:37:33,260][ses-node1][INFO ] installing Ceph on ses-node1
[2016-11-07 11:37:33,263][ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[2016-11-07 11:37:33,491][ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[2016-11-07 11:37:33,758][ses-node1][DEBUG ]
[2016-11-07 11:37:33,758][ses-node1][DEBUG ] The following 35 NEW packages are going to be installed:
[2016-11-07 11:37:33,758][ses-node1][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[2016-11-07 11:37:33,758][ses-node1][DEBUG ]
[2016-11-07 11:37:33,758][ses-node1][DEBUG ] The following package is not supported by its vendor:
[2016-11-07 11:37:33,758][ses-node1][DEBUG ] libxio
[2016-11-07 11:37:33,758][ses-node1][DEBUG ]
[2016-11-07 11:37:33,759][ses-node1][DEBUG ] 35 new packages to install.
[2016-11-07 11:37:33,759][ses-node1][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[2016-11-07 11:37:33,759][ses-node1][DEBUG ] Continue? [y/n/? shows all options] (y): y
[2016-11-07 11:37:44,565][ses-node1][INFO ] Running command: sudo ceph --version
[2016-11-07 11:37:44,683][ses-node1][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
[2016-11-07 11:37:44,683][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node2 ...
[2016-11-07 11:37:44,876][ses-node2][DEBUG ] connection detected need for sudo
[2016-11-07 11:37:45,016][ses-node2][DEBUG ] connected to host: ses-node2
[2016-11-07 11:37:45,017][ses-node2][DEBUG ] detect platform information from remote host
[2016-11-07 11:37:45,055][ses-node2][DEBUG ] detect machine type
[2016-11-07 11:37:45,060][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[2016-11-07 11:37:45,060][ses-node2][INFO ] installing Ceph on ses-node2
[2016-11-07 11:37:45,062][ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[2016-11-07 11:37:45,335][ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[2016-11-07 11:37:45,602][ses-node2][DEBUG ]
[2016-11-07 11:37:45,602][ses-node2][DEBUG ] The following 35 NEW packages are going to be installed:
[2016-11-07 11:37:45,602][ses-node2][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[2016-11-07 11:37:45,602][ses-node2][DEBUG ]
[2016-11-07 11:37:45,603][ses-node2][DEBUG ] The following package is not supported by its vendor:
[2016-11-07 11:37:45,603][ses-node2][DEBUG ] libxio
[2016-11-07 11:37:45,603][ses-node2][DEBUG ]
[2016-11-07 11:37:45,603][ses-node2][DEBUG ] 35 new packages to install.
[2016-11-07 11:37:45,603][ses-node2][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[2016-11-07 11:37:45,603][ses-node2][DEBUG ] Continue? [y/n/? shows all options] (y): y
[2016-11-07 11:37:56,699][ses-node2][INFO ] Running command: sudo ceph --version
[2016-11-07 11:37:56,819][ses-node2][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
[2016-11-07 11:37:56,820][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node3 ...
[2016-11-07 11:37:57,028][ses-node3][DEBUG ] connection detected need for sudo
[2016-11-07 11:37:57,173][ses-node3][DEBUG ] connected to host: ses-node3
[2016-11-07 11:37:57,173][ses-node3][DEBUG ] detect platform information from remote host
[2016-11-07 11:37:57,210][ses-node3][DEBUG ] detect machine type
[2016-11-07 11:37:57,215][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64
[2016-11-07 11:37:57,216][ses-node3][INFO ] installing Ceph on ses-node3
[2016-11-07 11:37:57,218][ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet refresh
[2016-11-07 11:37:57,494][ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw
[2016-11-07 11:37:57,762][ses-node3][DEBUG ]
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] The following 35 NEW packages are going to be installed:
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses
[2016-11-07 11:37:57,762][ses-node3][DEBUG ]
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] The following package is not supported by its vendor:
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] libxio
[2016-11-07 11:37:57,762][ses-node3][DEBUG ]
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] 35 new packages to install.
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used.
[2016-11-07 11:37:57,762][ses-node3][DEBUG ] Continue? [y/n/? shows all options] (y): y
[2016-11-07 11:38:08,867][ses-node3][INFO ] Running command: sudo ceph --version
[2016-11-07 11:38:08,987][ses-node3][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
cephadm@ses-admin:~> ceph -v
ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe)
cephadm@ses-admin:~> ceph-deploy new ses-node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy new ses-node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fbd48a4df80>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['ses-node1']
[ceph_deploy.cli][INFO ] func : <function new at 0x7fbd48a285f0>
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ses-node1][DEBUG ] connected to host: ses-admin
[ses-node1][INFO ] Running command: ssh -CT -o BatchMode=yes ses-node1
[ses-node1][DEBUG ] connection detected need for sudo
[ses-node1][DEBUG ] connected to host: ses-node1
[ses-node1][DEBUG ] detect platform information from remote host
[ses-node1][DEBUG ] detect machine type
[ses-node1][DEBUG ] find the location of an executable
[ses-node1][INFO ] Running command: sudo /bin/ip link show
[ses-node1][INFO ] Running command: sudo /bin/ip addr show
[ses-node1][DEBUG ] IP addresses found: ['192.168.100.11', '192.168.200.11']
[ceph_deploy.new][DEBUG ] Resolving host ses-node1
[ceph_deploy.new][DEBUG ] Monitor ses-node1 at 192.168.100.11
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ses-node1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.100.11']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
cephadm@ses-admin:~>

Creation de pool
pool name
number of placement groups : (nber of OSD * 100 ) / nber of replicas
pool size : 2,10 g take 20 g of storage and 20 g of netw transfert
pool size : 3,10 g take 30 g of storage and 30 g of netw transfert
=> il vaut ;ieux bcp de petits data pool plutot qu un gros
rados attaque en direct le pool sans passer par la couche osd

default, replicate all datas
=> full replication, 3x stockage et bande passante

RDB Storage
pour avoir du block storage, il faut creer une image d abord puis ;appe en block device
format1 or 2 => 2 support snapshots
rbd create, puis map pour connect
si besoin, faire le pool avant
rbd create -p poolname rdbimagename --size xxx --image format 2
size en mb par defaut
rbd list

cephadm@ses-admin:~> ceph osd map test myobj
osdmap e23 pool 'test' (1) object 'myobj' -> pg 1.c3ca3d86 (1.6) -> up ([2,1], p2) acting ([2,1], p2)

cephadm@ses-admin:~> ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.04376 root default
-2 0.01459 host ses-node1
 0 0.01459 osd.0 up 1.00000 1.00000
-3 0.01459 host ses-node2
 1 0.01459 osd.1 up 1.00000 1.00000
-4 0.01459 host ses-node3
 2 0.01459 osd.2 up 1.00000 1.00000

cephadm@ses-admin:~> ceph osd pool set test size 3
set pool 1 size to 3

cephadm@ses-admin:~> ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.04376 root default
-2 0.01459 host ses-node1
 0 0.01459 osd.0 up 1.00000 1.00000
-3 0.01459 host ses-node2
 1 0.01459 osd.1 up 1.00000 1.00000
-4 0.01459 host ses-node3
 2 0.01459 osd.2 up 1.00000 1.00000

Mapping pour utilisation en direct
persistence de mapping se fait dans fstab

cephadm@ses-admin:~> rbd create -p block-pool block-storage --size 1024
cephadm@ses-admin:~> rbd -p block-pool info block-storage
rbd image 'block-storage':
size 1024 MB in 256 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.108f238e1f29
format: 2
features: layering
flags:
cephadm@ses-admin:~> sudo su -
ses-admin:~ # cat /etc/fstab
UUID=a8cecc6a-ed55-45ec-9092-08cccd1d1fa5 swap swap defaults 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb / btrfs defaults 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /boot/grub2/i386-pc btrfs subvol=@/boot/grub2/i386-pc 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /boot/grub2/x86_64-efi btrfs subvol=@/boot/grub2/x86_64-efi 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /home btrfs subvol=@/home 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /opt btrfs subvol=@/opt 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /srv btrfs subvol=@/srv 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /tmp btrfs subvol=@/tmp 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /usr/local btrfs subvol=@/usr/local 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/crash btrfs subvol=@/var/crash 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/libvirt/images btrfs subvol=@/var/lib/libvirt/images 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mailman btrfs subvol=@/var/lib/mailman 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/named btrfs subvol=@/var/lib/named 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/log btrfs subvol=@/var/log 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/opt btrfs subvol=@/var/opt 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/spool btrfs subvol=@/var/spool 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/tmp btrfs subvol=@/var/tmp 0 0
UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /.snapshots btrfs subvol=@/.snapshots 0 0
ses-admin:~ # rbd map -p block-pool --image block-storage
/dev/rbd0
ses-admin:~ # rbd showmapped
id pool image snap device
0 block-pool block-storage - /dev/rbd0
ses-admin:~ # ls /dev/rbd*
/dev/rbd0

/dev/rbd:
block-pool

Snapshot sans overhead
rbd snap (etc)

Copy on write
snapshot layering, i;age au for;at 2 obligatoirement
mais les snapshots sont alors indispensables, i faut les proteger avec la fonction lock

rfc3720 for iscsi

json configurqtion quthenticqtion to iscsi authentication
https://www.suse.com/docrep/documents/kgu61iyowz/suse_enterprise_storage_3_and_iscsi.pdf

ses-node3:~ # targetcli ls
o- / ..................................................................................................... [...]
  o- backstores .......................................................................................... [...]
  | o- fileio ............................................................................... [0 Storage Object]
  | o- iblock ............................................................................... [1 Storage Object]
  | | o- iscsi .................................................................. [/dev/rbd/rbd/iscsi activated]
  | o- pscsi ................................................................................ [0 Storage Object]
  | o- rbd .................................................................................. [0 Storage Object]
  | o- rd_mcp ............................................................................... [0 Storage Object]
  o- ib_srpt ....................................................................................... [0 Targets]
  o- iscsi .......................................................................................... [1 Target]
  | o- iqn.2003-01.org.linux-iscsi.ses-node3.x8664:sn.76494aa74ebc ..................................... [1 TPG]
  | o- tpg1 ........................................................................................ [enabled]
  | o- acls ....................................................................................... [0 ACLs]
  | o- luns ........................................................................................ [1 LUN]
  | | o- lun0 .......................................................... [iblock/iscsi (/dev/rbd/rbd/iscsi)]
  | o- portals .................................................................................. [1 Portal]
  | o- 192.168.100.13:3260 ........................................................... [OK, iser disabled]
  o- loopback ...................................................................................... [0 Targets]
  o- qla2xxx ....................................................................................... [0 Targets]
  o- tcm_fc ........................................................................................ [0 Targets]
  o- vhost ......................................................................................... [0 Targets]

erasure code profile
une sorte de qos pour econo;iser du reseau pour limiter les controles d integrite
chaque erreur part dans une sorte de poubelle

cephadm@ses-admin:~> ceph osd erasure-code-profile get default
k=2
m=1
plugin=jerasure
technique=reed_sol_van

exemple k+m (10+6), on peut perdre 6 sur 16 disks (totalite des disques)
10 disks requis
prend 60% de capacite
attention aux perf, se rapproche d un raid6
si erasure profile fixe, ne peut pas etre change
utile sur des donnes chaudes/donnees froides/cout de stockage

ceph status : look at placement group
ex : recovering en bas de page

monitor :
health, mon quorum, osd status, pg status, full disk, nodes status
=> ceph health
=> ceph status

cephadm@ses-admin:~> ceph -w
  cluster 9a5b2edf-4d00-4b46-a010-3420d96450e5
  health HEALTH_WARN
  too many PGs per OSD (408 > max 300)
  monmap e3: 3 mons at {ses-node1=192.168.100.11:6789/0,ses-node2=192.168.100.12:6789/0,ses-node3=192.168.100.13:6789/0}
  election epoch 6, quorum 0,1,2 ses-node1,ses-node2,ses-node3
  osdmap e57: 3 osds: 3 up, 3 in
  flags sortbitwise
  pgmap v3256: 408 pgs, 15 pools, 70949 kB data, 505 objects
  322 MB used, 45724 MB / 46046 MB avail
  408 active+clean

ceph calamari backend, romana frontend
inclus avec suse enterprise storage

install calamari puis deploy --master monitoring-mode

alaister@suse.com