[
fr
]
L’Imp’Rock Scénette (by @_daffyduke_)
Rechercher :
>>
Rechercher dans le site
Accueil du site
>
1- Blog-Notes
>
Au boulot
>
Notes Susecon 2016 - Washington
Notes Susecon 2016 - Washington
Workshop CEPH
lundi 7 novembre 2016
par
Olivier Duquesne aka DaffyDuke
popularité : 8%
CEPH, OpenSorce, Software Defined Storage, LTS de 12 mois, Suse Storage 3 basee sur Jewel release jusqu qu 6 milliards de noeuds par cluster CERN et toutes les distrib ou constructeurs sont membres du management CEPH usual strategy is to scale up (vertical) ceph is for scaling out (horizontal) suse is the first to ship a iscsi gateway wih ceph (supported) RADOS = CEPH cluster, stockage objet Librados = app pour acceder aux donnees (C, PHP, Java etc) Radosgw = gateway REST co;\mpatible S3, Swift RBD = block sur host qui heberge une instance CEPH FS = fs pour un client la gatway irbd presente les donnees en iscsi ou nfs etc 2 co;posants i;portants : - un OSD pour manager les disques (one per path) - MONS (3, $, ....), pour monitorer le cluster Un algo CRUSH de stockage Disk-> FileSystem (XFS) -> OSD Les donnes sont stockees via algo CRUSH Mimu;m de 3 MONS, utilise algo PAXOS pour monitorer les nodes et le reseau => i;pair absoluement monitor map, contient placement group map, CRUSH map, epoch pour garantir integrite des donnes rssayer de mettre le moins possible de mons un client accede a la donnee objet en ;ode osd directement PAXOS election de l ip la plus petite pour elire un MON les objets sont ranges dans des storage pools defaultpool rbd regles de placement de snapshot par pool l osd voit les placement group, chaque objet est associe a um placement group il faut faire attention au nbre de PG CRUSH controllable Replication Under les donnes ne sont pas centralisees OSD run CRUSH => besoin de CPU Step2, install, hardware voir la doc suse ceph unprivileged, 167 if possible DNS recommande ipv4 tres bien timesync and at least 2 networks : pub for client/admin and priv for osd to osd SES3 dispo en extension de SLES 12sp0 zypper in ceph-deploy and that s all folks ceph-deploy install <node list> ceph-deply new -cluster moncluster <node list> ceph.conf contient les cles et le rep du journal ceph-deploy mon create-initial <node list> utiliser create pour ajouter des nodes MON puis ajouter les OSD ceph-deploy osd prepare node:dev when check with ceph -k ceph.client.admin.keyring status ceph -w (watch) geeko@ses-admin:~/Desktop> sudo su - We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things: #1) Respect the privacy of others. #2) Think before you type. #3) With great power comes great responsibility. root's password: ses-admin:~ # yast yast yast2 ses-admin:~ # yast2 ses-admin:~ # useradd -m cephadm ses-admin:~ # passwd cephadm New password: BAD PASSWORD: it is too short BAD PASSWORD: is too simple Retype new password: passwd: password updated successfully ses-admin:~ # visudo visudo: /etc/sudoers.tmp unchanged ses-admin:~ # visudo ses-admin:~ # su – cephadm su: user – does not exist ses-admin:~ # su - cephadm cephadm@ses-admin:~> ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/cephadm/.ssh/id_rsa): Created directory '/home/cephadm/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/cephadm/.ssh/id_rsa. Your public key has been saved in /home/cephadm/.ssh/id_rsa.pub. The key fingerprint is: 9e:b2:bd:31:6f:32:10:b6:7e:84:77:33:79:86:a3:e6 [MD5] cephadm@ses-admin The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | o | | . +S o | | +.o.B o | | ..+=o * | | .+*+. | | .+E=. | +--[MD5]----------+ cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node1 The authenticity of host 'ses-node1 (192.168.100.11)' can't be established. ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5]. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys Password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'cephadm@ses-node1'" and check to make sure that only the key(s) you wanted were added. cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node2 The authenticity of host 'ses-node2 (192.168.100.12)' can't be established. ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5]. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys Password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'cephadm@ses-node2'" and check to make sure that only the key(s) you wanted were added. cephadm@ses-admin:~> ssh-copy-id cephadm@ses-node3 The authenticity of host 'ses-node3 (192.168.100.13)' can't be established. ECDSA key fingerprint is 33:4d:5a:db:e7:69:26:2b:90:11:91:ad:ab:3d:49:eb [MD5]. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys Password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'cephadm@ses-node3'" and check to make sure that only the key(s) you wanted were added. cephadm@ses-admin:~> ssh ses-node3 id uid=1001(cephadm) gid=100(users) groups=100(users) cephadm@ses-admin:~> exit logout ses-admin:~ # zypper in ceph ceph-deploy Loading repository data... Reading installed packages... Resolving package dependencies... The following 42 NEW packages are going to be installed: FastCGI babeltrace ceph ceph-base ceph-common ceph-deploy ceph-mds ceph-mon ceph-osd gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-Flask python-Jinja2 python-MarkupSafe python-Werkzeug python-apipkg python-argparse python-cephfs python-ecdsa python-execnet python-itsdangerous python-paramiko python-pushy python-pycrypto python-rados python-rbd python-remoto python-requests supportutils-plugin-ses The following package is not supported by its vendor: libxio 42 new packages to install. Overall download size: 48.7 MiB. Already cached: 0 B. After the operation, additional 211.7 MiB will be used. Continue? [y/n/? shows all options] (y): y Retrieving package python-Werkzeug-0.9.6-5.2.noarch (1/42), 483.9 KiB ( 2.2 MiB unpacked) Retrieving package python-apipkg-1.2-1.2.noarch (2/42), 9.2 KiB ( 18.2 KiB unpacked) Retrieving package python-ecdsa-0.13-2.1.noarch (3/42), 77.4 KiB (261.8 KiB unpacked) Retrieving package python-itsdangerous-0.24-2.1.noarch (4/42), 20.2 KiB ( 69.1 KiB unpacked) Retrieving package supportutils-plugin-ses-1.0+git.1463485157.269428c-4.1.noarch (5/42), 13.7 KiB ( 26.0 KiB unpacked) Retrieving package FastCGI-2.4.0-167.1.x86_64 (6/42), 184.8 KiB (587.6 KiB unpacked) Retrieving package libboost_random1_54_0-1.54.0-13.1.x86_64 (7/42), 20.0 KiB ( 10.3 KiB unpacked) Retrieving package libleveldb1-1.18-1.3.x86_64 (8/42), 132.8 KiB (361.3 KiB unpacked) Retrieving package python-MarkupSafe-0.18-7.1.x86_64 (9/42), 24.6 KiB ( 66.0 KiB unpacked) Retrieving package python-pycrypto-2.6.1-4.1.x86_64 (10/42), 371.5 KiB ( 2.0 MiB unpacked) Retrieving package python-execnet-1.4.1-1.3.noarch (11/42), 73.7 KiB (249.5 KiB unpacked) Retrieving package python-Jinja2-2.7.3-17.1.noarch (12/42), 278.5 KiB ( 1.7 MiB unpacked) Retrieving package python-paramiko-1.15.2-1.1.noarch (13/42), 860.0 KiB ( 5.1 MiB unpacked) Retrieving package python-remoto-0.0.27-1.1.noarch (14/42), 80.6 KiB (298.2 KiB unpacked) Retrieving package python-Flask-0.10.1-2.1.noarch (15/42), 178.0 KiB (827.7 KiB unpacked) Retrieving package python-pushy-0.5.3-1.5.noarch (16/42), 58.3 KiB (238.5 KiB unpacked) Retrieving package python-argparse-1.2.1-15.8.noarch (17/42), 248.6 KiB (490.1 KiB unpacked) Retrieving package python-requests-2.3.0-6.5.2.noarch (18/42), 334.6 KiB ( 1.7 MiB unpacked) Retrieving package babeltrace-1.2.4-3.2.x86_64 (19/42), 159.0 KiB (463.9 KiB unpacked) Retrieving package gperftools-2.2-2.2.x86_64 (20/42), 554.3 KiB ( 2.2 MiB unpacked) Retrieving package libboost_iostreams1_54_0-1.54.0-13.1.x86_64 (21/42), 39.8 KiB (103.1 KiB unpacked) Retrieving package libboost_program_options1_54_0-1.54.0-13.1.x86_64 (22/42), 125.5 KiB (456.7 KiB unpacked) Retrieving package libibverbs1-1.1.7-12.2.x86_64 (23/42), 34.1 KiB ( 71.4 KiB unpacked) Retrieving package liburcu0-0.8.8-1.2.x86_64 (24/42), 49.4 KiB (182.6 KiB unpacked) Retrieving package librdmacm1-1.0.18.1-1.14.x86_64 (25/42), 53.4 KiB (129.1 KiB unpacked) Retrieving package lttng-ust-2.7.0-1.2.x86_64 (26/42), 171.4 KiB (754.7 KiB unpacked) Retrieving package ceph-deploy-1.5.32+git.1464082518.5aab689-1.1.noarch (27/42), 174.7 KiB (740.3 KiB unpacked) Retrieving package libxio-1.6+git.1461071064.e0ed264-2.1.x86_64 (28/42), 242.5 KiB (885.9 KiB unpacked) Retrieving package librados2-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (29/42), 1.7 MiB ( 6.3 MiB unpacked) Retrieving package libcephfs1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (30/42), 1.8 MiB ( 6.0 MiB unpacked) Retrieving package python-rados-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (31/42), 209.0 KiB (493.4 KiB unpacked) Retrieving package librgw2-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (32/42), 2.7 MiB ( 10.3 MiB unpacked) Retrieving package librbd1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (33/42), 2.1 MiB ( 7.5 MiB unpacked) Retrieving package libradosstriper1-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (34/42), 1.8 MiB ( 6.9 MiB unpacked) Retrieving package python-cephfs-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (35/42), 138.1 KiB (227.4 KiB unpacked) Retrieving package python-rbd-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (36/42), 133.1 KiB (184.9 KiB unpacked) Retrieving package ceph-common-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (37/42), 14.9 MiB ( 67.1 MiB unpacked) Retrieving package ceph-base-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (38/42), 3.9 MiB ( 33.1 MiB unpacked) Retrieving package ceph-osd-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (39/42), 9.0 MiB ( 33.2 MiB unpacked) Retrieving package ceph-mon-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (40/42), 2.7 MiB ( 9.3 MiB unpacked) Retrieving package ceph-mds-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (41/42), 2.7 MiB ( 9.3 MiB unpacked) Retrieving package ceph-10.2.1+git.1464009581.b8ee9e4-1.1.x86_64 (42/42), 79.1 KiB ( 0 B unpacked) Checking for file conflicts: .............................................................................[done] ( 1/42) Installing: python-Werkzeug-0.9.6-5.2 ............................................................[done] ( 2/42) Installing: python-apipkg-1.2-1.2 ................................................................[done] ( 3/42) Installing: python-ecdsa-0.13-2.1 ................................................................[done] ( 4/42) Installing: python-itsdangerous-0.24-2.1 .........................................................[done] ( 5/42) Installing: supportutils-plugin-ses-1.0+git.1463485157.269428c-4.1 ...............................[done] ( 6/42) Installing: FastCGI-2.4.0-167.1 ..................................................................[done] ( 7/42) Installing: libboost_random1_54_0-1.54.0-13.1 ....................................................[done] ( 8/42) Installing: libleveldb1-1.18-1.3 .................................................................[done] ( 9/42) Installing: python-MarkupSafe-0.18-7.1 ...........................................................[done] (10/42) Installing: python-pycrypto-2.6.1-4.1 ............................................................[done] (11/42) Installing: python-execnet-1.4.1-1.3 .............................................................[done] (12/42) Installing: python-Jinja2-2.7.3-17.1 .............................................................[done] (13/42) Installing: python-paramiko-1.15.2-1.1 ...........................................................[done] (14/42) Installing: python-remoto-0.0.27-1.1 .............................................................[done] (15/42) Installing: python-Flask-0.10.1-2.1 ..............................................................[done] (16/42) Installing: python-pushy-0.5.3-1.5 ...............................................................[done] (17/42) Installing: python-argparse-1.2.1-15.8 ...........................................................[done] (18/42) Installing: python-requests-2.3.0-6.5.2 ..........................................................[done] (19/42) Installing: babeltrace-1.2.4-3.2 .................................................................[done] (20/42) Installing: gperftools-2.2-2.2 ...................................................................[done] (21/42) Installing: libboost_iostreams1_54_0-1.54.0-13.1 .................................................[done] (22/42) Installing: libboost_program_options1_54_0-1.54.0-13.1 ...........................................[done] (23/42) Installing: libibverbs1-1.1.7-12.2 ...............................................................[done] (24/42) Installing: liburcu0-0.8.8-1.2 ...................................................................[done] (25/42) Installing: librdmacm1-1.0.18.1-1.14 .............................................................[done] (26/42) Installing: lttng-ust-2.7.0-1.2 ..................................................................[done] (27/42) Installing: ceph-deploy-1.5.32+git.1464082518.5aab689-1.1 ........................................[done] (28/42) Installing: libxio-1.6+git.1461071064.e0ed264-2.1 ................................................[done] (29/42) Installing: librados2-10.2.1+git.1464009581.b8ee9e4-1.1 ..........................................[done] (30/42) Installing: libcephfs1-10.2.1+git.1464009581.b8ee9e4-1.1 .........................................[done] (31/42) Installing: python-rados-10.2.1+git.1464009581.b8ee9e4-1.1 .......................................[done] (32/42) Installing: librgw2-10.2.1+git.1464009581.b8ee9e4-1.1 ............................................[done] (33/42) Installing: librbd1-10.2.1+git.1464009581.b8ee9e4-1.1 ............................................[done] (34/42) Installing: libradosstriper1-10.2.1+git.1464009581.b8ee9e4-1.1 ...................................[done] (35/42) Installing: python-cephfs-10.2.1+git.1464009581.b8ee9e4-1.1 ......................................[done] (36/42) Installing: python-rbd-10.2.1+git.1464009581.b8ee9e4-1.1 .........................................[done] (37/42) Installing: ceph-common-10.2.1+git.1464009581.b8ee9e4-1.1 ........................................[done] (38/42) Installing: ceph-base-10.2.1+git.1464009581.b8ee9e4-1.1 ..........................................[done] Additional rpm output: Updating /etc/sysconfig/ceph... (39/42) Installing: ceph-osd-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done] (40/42) Installing: ceph-mon-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done] (41/42) Installing: ceph-mds-10.2.1+git.1464009581.b8ee9e4-1.1 ...........................................[done] (42/42) Installing: ceph-10.2.1+git.1464009581.b8ee9e4-1.1 ...............................................[done] ses-admin:~ # su – cephadm su: user – does not exist ses-admin:~ # su - cephadm cephadm@ses-admin:~> ls bin public_html cephadm@ses-admin:~> ceph-deploy install ses-node1 ses-node2 ses-node3 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy install ses-node1 ses-node2 ses-node3 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] testing : None [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f12cbf6cef0> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] dev_commit : None [ceph_deploy.cli][INFO ] install_mds : False [ceph_deploy.cli][INFO ] stable : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] adjust_repos : True [ceph_deploy.cli][INFO ] func : <function install at 0x7f12cc1b0668> [ceph_deploy.cli][INFO ] install_all : False [ceph_deploy.cli][INFO ] repo : False [ceph_deploy.cli][INFO ] host : ['ses-node1', 'ses-node2', 'ses-node3'] [ceph_deploy.cli][INFO ] install_rgw : False [ceph_deploy.cli][INFO ] install_tests : False [ceph_deploy.cli][INFO ] repo_url : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] install_osd : False [ceph_deploy.cli][INFO ] version_kind : stable [ceph_deploy.cli][INFO ] install_common : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] dev : master [ceph_deploy.cli][INFO ] local_mirror : None [ceph_deploy.cli][INFO ] release : None [ceph_deploy.cli][INFO ] install_mon : False [ceph_deploy.cli][INFO ] gpg_url : None [ceph_deploy.install][DEBUG ] Installing stable version infernalis on cluster ceph hosts ses-node1 ses-node2 ses-node3 [ceph_deploy.install][DEBUG ] Detecting platform for host ses-node1 ... [ses-node1][DEBUG ] connection detected need for sudo [ses-node1][DEBUG ] connected to host: ses-node1 [ses-node1][DEBUG ] detect platform information from remote host [ses-node1][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [ses-node1][INFO ] installing Ceph on ses-node1 [ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [ses-node1][DEBUG ] [ses-node1][DEBUG ] The following 35 NEW packages are going to be installed: [ses-node1][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [ses-node1][DEBUG ] [ses-node1][DEBUG ] The following package is not supported by its vendor: [ses-node1][DEBUG ] libxio [ses-node1][DEBUG ] [ses-node1][DEBUG ] 35 new packages to install. [ses-node1][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [ses-node1][DEBUG ] Continue? [y/n/? shows all options] (y): y [ses-node1][INFO ] Running command: sudo ceph --version [ses-node1][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) [ceph_deploy.install][DEBUG ] Detecting platform for host ses-node2 ... [ses-node2][DEBUG ] connection detected need for sudo [ses-node2][DEBUG ] connected to host: ses-node2 [ses-node2][DEBUG ] detect platform information from remote host [ses-node2][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [ses-node2][INFO ] installing Ceph on ses-node2 [ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [ses-node2][DEBUG ] [ses-node2][DEBUG ] The following 35 NEW packages are going to be installed: [ses-node2][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [ses-node2][DEBUG ] [ses-node2][DEBUG ] The following package is not supported by its vendor: [ses-node2][DEBUG ] libxio [ses-node2][DEBUG ] [ses-node2][DEBUG ] 35 new packages to install. [ses-node2][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [ses-node2][DEBUG ] Continue? [y/n/? shows all options] (y): y [ses-node2][INFO ] Running command: sudo ceph --version [ses-node2][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) [ceph_deploy.install][DEBUG ] Detecting platform for host ses-node3 ... [ses-node3][DEBUG ] connection detected need for sudo [ses-node3][DEBUG ] connected to host: ses-node3 [ses-node3][DEBUG ] detect platform information from remote host [ses-node3][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [ses-node3][INFO ] installing Ceph on ses-node3 [ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [ses-node3][DEBUG ] [ses-node3][DEBUG ] The following 35 NEW packages are going to be installed: [ses-node3][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [ses-node3][DEBUG ] [ses-node3][DEBUG ] The following package is not supported by its vendor: [ses-node3][DEBUG ] libxio [ses-node3][DEBUG ] [ses-node3][DEBUG ] 35 new packages to install. [ses-node3][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [ses-node3][DEBUG ] Continue? [y/n/? shows all options] (y): y [ses-node3][INFO ] Running command: sudo ceph --version [ses-node3][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) cephadm@ses-admin:~> ls bin ceph-deploy-ceph.log public_html cephadm@ses-admin:~> cat ceph-deploy-ceph.log [2016-11-07 11:37:32,854][ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy install ses-node1 ses-node2 ses-node3 [2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] ceph-deploy options: [2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] verbose : False [2016-11-07 11:37:32,854][ceph_deploy.cli][INFO ] testing : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f12cbf6cef0> [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] cluster : ceph [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] dev_commit : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_mds : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] stable : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] default_release : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] username : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] adjust_repos : True [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] func : <function install at 0x7f12cc1b0668> [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_all : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] repo : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] host : ['ses-node1', 'ses-node2', 'ses-node3'] [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_rgw : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_tests : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] repo_url : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] ceph_conf : None [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] install_osd : False [2016-11-07 11:37:32,855][ceph_deploy.cli][INFO ] version_kind : stable [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] install_common : False [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] overwrite_conf : False [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] quiet : False [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] dev : master [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] local_mirror : None [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] release : None [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] install_mon : False [2016-11-07 11:37:32,856][ceph_deploy.cli][INFO ] gpg_url : None [2016-11-07 11:37:32,856][ceph_deploy.install][DEBUG ] Installing stable version infernalis on cluster ceph hosts ses-node1 ses-node2 ses-node3 [2016-11-07 11:37:32,856][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node1 ... [2016-11-07 11:37:33,075][ses-node1][DEBUG ] connection detected need for sudo [2016-11-07 11:37:33,221][ses-node1][DEBUG ] connected to host: ses-node1 [2016-11-07 11:37:33,221][ses-node1][DEBUG ] detect platform information from remote host [2016-11-07 11:37:33,256][ses-node1][DEBUG ] detect machine type [2016-11-07 11:37:33,260][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [2016-11-07 11:37:33,260][ses-node1][INFO ] installing Ceph on ses-node1 [2016-11-07 11:37:33,263][ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [2016-11-07 11:37:33,491][ses-node1][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [2016-11-07 11:37:33,758][ses-node1][DEBUG ] [2016-11-07 11:37:33,758][ses-node1][DEBUG ] The following 35 NEW packages are going to be installed: [2016-11-07 11:37:33,758][ses-node1][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [2016-11-07 11:37:33,758][ses-node1][DEBUG ] [2016-11-07 11:37:33,758][ses-node1][DEBUG ] The following package is not supported by its vendor: [2016-11-07 11:37:33,758][ses-node1][DEBUG ] libxio [2016-11-07 11:37:33,758][ses-node1][DEBUG ] [2016-11-07 11:37:33,759][ses-node1][DEBUG ] 35 new packages to install. [2016-11-07 11:37:33,759][ses-node1][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [2016-11-07 11:37:33,759][ses-node1][DEBUG ] Continue? [y/n/? shows all options] (y): y [2016-11-07 11:37:44,565][ses-node1][INFO ] Running command: sudo ceph --version [2016-11-07 11:37:44,683][ses-node1][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) [2016-11-07 11:37:44,683][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node2 ... [2016-11-07 11:37:44,876][ses-node2][DEBUG ] connection detected need for sudo [2016-11-07 11:37:45,016][ses-node2][DEBUG ] connected to host: ses-node2 [2016-11-07 11:37:45,017][ses-node2][DEBUG ] detect platform information from remote host [2016-11-07 11:37:45,055][ses-node2][DEBUG ] detect machine type [2016-11-07 11:37:45,060][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [2016-11-07 11:37:45,060][ses-node2][INFO ] installing Ceph on ses-node2 [2016-11-07 11:37:45,062][ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [2016-11-07 11:37:45,335][ses-node2][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [2016-11-07 11:37:45,602][ses-node2][DEBUG ] [2016-11-07 11:37:45,602][ses-node2][DEBUG ] The following 35 NEW packages are going to be installed: [2016-11-07 11:37:45,602][ses-node2][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [2016-11-07 11:37:45,602][ses-node2][DEBUG ] [2016-11-07 11:37:45,603][ses-node2][DEBUG ] The following package is not supported by its vendor: [2016-11-07 11:37:45,603][ses-node2][DEBUG ] libxio [2016-11-07 11:37:45,603][ses-node2][DEBUG ] [2016-11-07 11:37:45,603][ses-node2][DEBUG ] 35 new packages to install. [2016-11-07 11:37:45,603][ses-node2][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [2016-11-07 11:37:45,603][ses-node2][DEBUG ] Continue? [y/n/? shows all options] (y): y [2016-11-07 11:37:56,699][ses-node2][INFO ] Running command: sudo ceph --version [2016-11-07 11:37:56,819][ses-node2][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) [2016-11-07 11:37:56,820][ceph_deploy.install][DEBUG ] Detecting platform for host ses-node3 ... [2016-11-07 11:37:57,028][ses-node3][DEBUG ] connection detected need for sudo [2016-11-07 11:37:57,173][ses-node3][DEBUG ] connected to host: ses-node3 [2016-11-07 11:37:57,173][ses-node3][DEBUG ] detect platform information from remote host [2016-11-07 11:37:57,210][ses-node3][DEBUG ] detect machine type [2016-11-07 11:37:57,215][ceph_deploy.install][INFO ] Distro info: SUSE Linux Enterprise Server 12 x86_64 [2016-11-07 11:37:57,216][ses-node3][INFO ] installing Ceph on ses-node3 [2016-11-07 11:37:57,218][ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet refresh [2016-11-07 11:37:57,494][ses-node3][INFO ] Running command: sudo zypper --non-interactive --quiet install ceph ceph-radosgw [2016-11-07 11:37:57,762][ses-node3][DEBUG ] [2016-11-07 11:37:57,762][ses-node3][DEBUG ] The following 35 NEW packages are going to be installed: [2016-11-07 11:37:57,762][ses-node3][DEBUG ] babeltrace ceph ceph-base ceph-common ceph-mds ceph-mon ceph-osd ceph-radosgw FastCGI gperftools libboost_iostreams1_54_0 libboost_program_options1_54_0 libboost_random1_54_0 libcephfs1 libibverbs1 libleveldb1 librados2 libradosstriper1 librbd1 librdmacm1 librgw2 liburcu0 libxio lttng-ust python-cephfs python-Flask python-itsdangerous python-Jinja2 python-MarkupSafe python-rados python-rbd python-requests python-setuptools python-Werkzeug supportutils-plugin-ses [2016-11-07 11:37:57,762][ses-node3][DEBUG ] [2016-11-07 11:37:57,762][ses-node3][DEBUG ] The following package is not supported by its vendor: [2016-11-07 11:37:57,762][ses-node3][DEBUG ] libxio [2016-11-07 11:37:57,762][ses-node3][DEBUG ] [2016-11-07 11:37:57,762][ses-node3][DEBUG ] 35 new packages to install. [2016-11-07 11:37:57,762][ses-node3][DEBUG ] Overall download size: 47.3 MiB. Already cached: 0 B. After the operation, additional 204.1 MiB will be used. [2016-11-07 11:37:57,762][ses-node3][DEBUG ] Continue? [y/n/? shows all options] (y): y [2016-11-07 11:38:08,867][ses-node3][INFO ] Running command: sudo ceph --version [2016-11-07 11:38:08,987][ses-node3][DEBUG ] ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) cephadm@ses-admin:~> ceph -v ceph version 10.2.1-112-gb8ee9e4 (b8ee9e4007f0443ef50deb1a691c3ea8bd0a11fe) cephadm@ses-admin:~> ceph-deploy new ses-node1 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadm/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.32): /usr/bin/ceph-deploy new ses-node1 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fbd48a4df80> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] ssh_copykey : True [ceph_deploy.cli][INFO ] mon : ['ses-node1'] [ceph_deploy.cli][INFO ] func : <function new at 0x7fbd48a285f0> [ceph_deploy.cli][INFO ] public_network : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] cluster_network : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] fsid : None [ceph_deploy.new][DEBUG ] Creating new cluster named ceph [ceph_deploy.new][INFO ] making sure passwordless SSH succeeds [ses-node1][DEBUG ] connected to host: ses-admin [ses-node1][INFO ] Running command: ssh -CT -o BatchMode=yes ses-node1 [ses-node1][DEBUG ] connection detected need for sudo [ses-node1][DEBUG ] connected to host: ses-node1 [ses-node1][DEBUG ] detect platform information from remote host [ses-node1][DEBUG ] detect machine type [ses-node1][DEBUG ] find the location of an executable [ses-node1][INFO ] Running command: sudo /bin/ip link show [ses-node1][INFO ] Running command: sudo /bin/ip addr show [ses-node1][DEBUG ] IP addresses found: ['192.168.100.11', '192.168.200.11'] [ceph_deploy.new][DEBUG ] Resolving host ses-node1 [ceph_deploy.new][DEBUG ] Monitor ses-node1 at 192.168.100.11 [ceph_deploy.new][DEBUG ] Monitor initial members are ['ses-node1'] [ceph_deploy.new][DEBUG ] Monitor addrs are ['192.168.100.11'] [ceph_deploy.new][DEBUG ] Creating a random mon key... [ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring... [ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf... cephadm@ses-admin:~> Creation de pool pool name number of placement groups : (nber of OSD * 100 ) / nber of replicas pool size : 2,10 g take 20 g of storage and 20 g of netw transfert pool size : 3,10 g take 30 g of storage and 30 g of netw transfert => il vaut ;ieux bcp de petits data pool plutot qu un gros rados attaque en direct le pool sans passer par la couche osd default, replicate all datas => full replication, 3x stockage et bande passante RDB Storage pour avoir du block storage, il faut creer une image d abord puis ;appe en block device format1 or 2 => 2 support snapshots rbd create, puis map pour connect si besoin, faire le pool avant rbd create -p poolname rdbimagename --size xxx --image format 2 size en mb par defaut rbd list cephadm@ses-admin:~> ceph osd map test myobj osdmap e23 pool 'test' (1) object 'myobj' -> pg 1.c3ca3d86 (1.6) -> up ([2,1], p2) acting ([2,1], p2) cephadm@ses-admin:~> ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.04376 root default -2 0.01459 host ses-node1 0 0.01459 osd.0 up 1.00000 1.00000 -3 0.01459 host ses-node2 1 0.01459 osd.1 up 1.00000 1.00000 -4 0.01459 host ses-node3 2 0.01459 osd.2 up 1.00000 1.00000 cephadm@ses-admin:~> ceph osd pool set test size 3 set pool 1 size to 3 cephadm@ses-admin:~> ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.04376 root default -2 0.01459 host ses-node1 0 0.01459 osd.0 up 1.00000 1.00000 -3 0.01459 host ses-node2 1 0.01459 osd.1 up 1.00000 1.00000 -4 0.01459 host ses-node3 2 0.01459 osd.2 up 1.00000 1.00000 Mapping pour utilisation en direct persistence de mapping se fait dans fstab cephadm@ses-admin:~> rbd create -p block-pool block-storage --size 1024 cephadm@ses-admin:~> rbd -p block-pool info block-storage rbd image 'block-storage': size 1024 MB in 256 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.108f238e1f29 format: 2 features: layering flags: cephadm@ses-admin:~> sudo su - ses-admin:~ # cat /etc/fstab UUID=a8cecc6a-ed55-45ec-9092-08cccd1d1fa5 swap swap defaults 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb / btrfs defaults 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /boot/grub2/i386-pc btrfs subvol=@/boot/grub2/i386-pc 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /boot/grub2/x86_64-efi btrfs subvol=@/boot/grub2/x86_64-efi 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /home btrfs subvol=@/home 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /opt btrfs subvol=@/opt 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /srv btrfs subvol=@/srv 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /tmp btrfs subvol=@/tmp 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /usr/local btrfs subvol=@/usr/local 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/crash btrfs subvol=@/var/crash 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/libvirt/images btrfs subvol=@/var/lib/libvirt/images 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mailman btrfs subvol=@/var/lib/mailman 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/named btrfs subvol=@/var/lib/named 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/log btrfs subvol=@/var/log 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/opt btrfs subvol=@/var/opt 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/spool btrfs subvol=@/var/spool 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /var/tmp btrfs subvol=@/var/tmp 0 0 UUID=f809d9df-cc52-43e0-a42c-3f2c8ee680eb /.snapshots btrfs subvol=@/.snapshots 0 0 ses-admin:~ # rbd map -p block-pool --image block-storage /dev/rbd0 ses-admin:~ # rbd showmapped id pool image snap device 0 block-pool block-storage - /dev/rbd0 ses-admin:~ # ls /dev/rbd* /dev/rbd0 /dev/rbd: block-pool Snapshot sans overhead rbd snap (etc) Copy on write snapshot layering, i;age au for;at 2 obligatoirement mais les snapshots sont alors indispensables, i faut les proteger avec la fonction lock rfc3720 for iscsi json configurqtion quthenticqtion to iscsi authentication https://www.suse.com/docrep/documents/kgu61iyowz/suse_enterprise_storage_3_and_iscsi.pdf ses-node3:~ # targetcli ls o- / ..................................................................................................... [...] o- backstores .......................................................................................... [...] | o- fileio ............................................................................... [0 Storage Object] | o- iblock ............................................................................... [1 Storage Object] | | o- iscsi .................................................................. [/dev/rbd/rbd/iscsi activated] | o- pscsi ................................................................................ [0 Storage Object] | o- rbd .................................................................................. [0 Storage Object] | o- rd_mcp ............................................................................... [0 Storage Object] o- ib_srpt ....................................................................................... [0 Targets] o- iscsi .......................................................................................... [1 Target] | o- iqn.2003-01.org.linux-iscsi.ses-node3.x8664:sn.76494aa74ebc ..................................... [1 TPG] | o- tpg1 ........................................................................................ [enabled] | o- acls ....................................................................................... [0 ACLs] | o- luns ........................................................................................ [1 LUN] | | o- lun0 .......................................................... [iblock/iscsi (/dev/rbd/rbd/iscsi)] | o- portals .................................................................................. [1 Portal] | o- 192.168.100.13:3260 ........................................................... [OK, iser disabled] o- loopback ...................................................................................... [0 Targets] o- qla2xxx ....................................................................................... [0 Targets] o- tcm_fc ........................................................................................ [0 Targets] o- vhost ......................................................................................... [0 Targets] erasure code profile une sorte de qos pour econo;iser du reseau pour limiter les controles d integrite chaque erreur part dans une sorte de poubelle cephadm@ses-admin:~> ceph osd erasure-code-profile get default k=2 m=1 plugin=jerasure technique=reed_sol_van exemple k+m (10+6), on peut perdre 6 sur 16 disks (totalite des disques) 10 disks requis prend 60% de capacite attention aux perf, se rapproche d un raid6 si erasure profile fixe, ne peut pas etre change utile sur des donnes chaudes/donnees froides/cout de stockage ceph status : look at placement group ex : recovering en bas de page monitor : health, mon quorum, osd status, pg status, full disk, nodes status => ceph health => ceph status cephadm@ses-admin:~> ceph -w cluster 9a5b2edf-4d00-4b46-a010-3420d96450e5 health HEALTH_WARN too many PGs per OSD (408 > max 300) monmap e3: 3 mons at {ses-node1=192.168.100.11:6789/0,ses-node2=192.168.100.12:6789/0,ses-node3=192.168.100.13:6789/0} election epoch 6, quorum 0,1,2 ses-node1,ses-node2,ses-node3 osdmap e57: 3 osds: 3 up, 3 in flags sortbitwise pgmap v3256: 408 pgs, 15 pools, 70949 kB data, 505 objects 322 MB used, 45724 MB / 46046 MB avail 408 active+clean ceph calamari backend, romana frontend inclus avec suse enterprise storage install calamari puis deploy --master monitoring-mode alaister@suse.com
1- Blog-Notes
A la maison
Ailleurs
Cinoche
Lis tes râtures
Musiques
Vacances
Au boulot
2- Politique
3- Logiciels Libres
Expériences
Historique et analyses
Les Aventures de TuX
Téléchargement
4- BTS Biochimie
Biochimie
Biologie Cellulaire
Immunologie
Microbiologie
Physiologie
6- Webographie
Mots-clés
Sites favoris
Sur le Web
Connexion
Statistiques
Dernière mise à jour
samedi 11 septembre 2021
Publication
35819 Articles
Aucun album photo
Aucune brève
84 Sites Web
2336 Auteurs
Visites
660 aujourd’hui
1229 hier
676900 depuis le début
33 visiteurs actuellement connectés
Publications
Derniers articles publiés
Sans titre
Sans titre
Littératurefrançaise.net - Pierre de Ronsard
gilles-dhaussy/amis_communs.py — Python — NumWorks
Sans titre
Navigation
Mots-clés de l’article
linux
Articles de la rubrique
Notes Meetup Docker - chez GFI
Notes Configuration Management Camp 2017
Notes Oracle Cloud
Notes Susecon 2016 - Washington
Notes Susecon 2016 - Washington
Notes Sysadmin Days #6 - Paris
Notes Suse Expert Day 2016 Paris
Notes Configuration Management Camp 2016
Notes présentation RedHat Satellite 6
Notes présentation RedHat OpenShift
0
10
20
30
...
6