Re: [PVE-User] [ceph-users] Re: Ceph Usage web and terminal.

["Сергей Цаболов" <tsabolov@t8.ru>]

Hello to all.

In my case I have the 7 node cluster Proxmox and working Ceph (ceph 
version 15.2.15  octopus (stable)": 7)

Ceph HEALTH_OK

ceph -s
   cluster:
     id:     9662e3fa-4ce6-41df-8d74-5deaa41a8dde
     health: HEALTH_OK

   services:
     mon: 7 daemons, quorum 
pve-3105,pve-3107,pve-3108,pve-3103,pve-3101,pve-3111,pve-3109 (age 17h)
     mgr: pve-3107(active, since 41h), standbys: pve-3109, pve-3103, 
pve-3105, pve-3101, pve-3111, pve-3108
     mds: cephfs:1 {0=pve-3105=up:active} 6 up:standby
     osd: 22 osds: 22 up (since 17h), 22 in (since 17h)

   task status:

   data:
     pools:   4 pools, 1089 pgs
     objects: 1.09M objects, 4.1 TiB
     usage:   7.7 TiB used, 99 TiB / 106 TiB avail
     pgs:     1089 active+clean

---------------------------------------------------------------------------------------------------------------------

ceph osd tree

ID   CLASS  WEIGHT     TYPE NAME            STATUS  REWEIGHT PRI-AFF
  -1         106.43005  root default
-13          14.55478      host pve-3101
  10    hdd    7.27739          osd.10           up   1.00000 1.00000
  11    hdd    7.27739          osd.11           up   1.00000 1.00000
-11          14.55478      host pve-3103
   8    hdd    7.27739          osd.8            up   1.00000 1.00000
   9    hdd    7.27739          osd.9            up   1.00000 1.00000
  -3          14.55478      host pve-3105
   0    hdd    7.27739          osd.0            up   1.00000 1.00000
   1    hdd    7.27739          osd.1            up   1.00000 1.00000
  -5          14.55478      host pve-3107
   2    hdd    7.27739          osd.2            up   1.00000 1.00000
   3    hdd    7.27739          osd.3            up   1.00000 1.00000
  -9          14.55478      host pve-3108
   6    hdd    7.27739          osd.6            up   1.00000 1.00000
   7    hdd    7.27739          osd.7            up   1.00000 1.00000
  -7          14.55478      host pve-3109
   4    hdd    7.27739          osd.4            up   1.00000 1.00000
   5    hdd    7.27739          osd.5            up   1.00000 1.00000
-15          19.10138      host pve-3111
  12    hdd   10.91409          osd.12           up   1.00000 1.00000
  13    hdd    0.90970          osd.13           up   1.00000 1.00000
  14    hdd    0.90970          osd.14           up   1.00000 1.00000
  15    hdd    0.90970          osd.15           up   1.00000 1.00000
  16    hdd    0.90970          osd.16           up   1.00000 1.00000
  17    hdd    0.90970          osd.17           up   1.00000 1.00000
  18    hdd    0.90970          osd.18           up   1.00000 1.00000
  19    hdd    0.90970          osd.19           up   1.00000 1.00000
  20    hdd    0.90970          osd.20           up   1.00000 1.00000
  21    hdd    0.90970          osd.21           up   1.00000 1.00000

---------------------------------------------------------------------------------------------------------------

POOL                               ID  PGS   STORED   OBJECTS USED     
%USED  MAX AVAIL
vm.pool                            2  1024  3.0 TiB  863.31k  6.0 TiB   
6.38     44 TiB  (this pool have the all VM disk)

---------------------------------------------------------------------------------------------------------------

ceph osd map vm.pool vm.pool.object
osdmap e14319 pool 'vm.pool' (2) object 'vm.pool.object' -> pg 
2.196f68d5 (2.d5) -> up ([2,4], p2) acting ([2,4], p2)

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.143-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-helper: 6.4-8
pve-kernel-5.4: 6.4-7
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph: 15.2.15-pve1~bpo10
ceph-fuse: 15.2.15-pve1~bpo10
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve1~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.6-pve1~bpo10+1

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------

And now my problem:

For all VM I have one pool for VM disks

When  node/host pve-3111  is shutdown in many of other nodes/hosts 
pve-3107, pve-3105  VM not shutdown but not available in network.

After the node/host is up Ceph back to HEALTH_OK and the all VM back to 
access in Network (without reboot).

Can some one to suggest me what I can to check in Ceph ?

Thanks.

-- 
-------------------------
Best Regards