* [PVE-User] zfs raidz2 expansion @ 2025-05-20 1:24 Randy Bush 2025-05-20 7:05 ` Aaron Lauterer 0 siblings, 1 reply; 8+ messages in thread From: Randy Bush @ 2025-05-20 1:24 UTC (permalink / raw) To: ProxMox Users running 8.4.1 with images on 4x2tb raidz2 ssds (zfs-2.2.7-pve2). we want to double the space. the net of a million lies says all sorts of things about expanding raidz2. as we do full replication, we could just create a new pool for replocation and call it a day. but i am curious if we can just expand the current pool. my reading says doubtful, but am not deeply clued. randy --- https://forum.proxmox.com/threads/raidz-expansion.135413/ https://github.com/openzfs/zfs/pull/15022 _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-20 1:24 [PVE-User] zfs raidz2 expansion Randy Bush @ 2025-05-20 7:05 ` Aaron Lauterer 2025-05-20 17:21 ` Randy Bush 0 siblings, 1 reply; 8+ messages in thread From: Aaron Lauterer @ 2025-05-20 7:05 UTC (permalink / raw) To: Proxmox VE user list, Randy Bush You can always add a new VDEV to the pool (man zpool-add). Expanding a raidz2 VDEV itself is only possible with ZFS 2.3 AFAIU and we do not ship that yet [0]. But if this is for VM storage, you might want to consider a ZFS pool of multiple mirrored VDEVs. [1] has more details. While performance might not be much of an issue anymore with recent NVMEs that provide a ton of IOPS, the additional space usage might still be something to think about in a raidz pool used for VMs (ZVOL). [0] https://forum.proxmox.com/threads/proxmox-ve-8-4-released.164821/#post-762367 [1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_raid_considerations On 2025-05-20 03:24, Randy Bush wrote: > running 8.4.1 with images on 4x2tb raidz2 ssds (zfs-2.2.7-pve2). we > want to double the space. the net of a million lies says all sorts of > things about expanding raidz2. as we do full replication, we could just > create a new pool for replocation and call it a day. but i am curious > if we can just expand the current pool. my reading says doubtful, but > am not deeply clued. > > randy > > --- > > https://forum.proxmox.com/threads/raidz-expansion.135413/ > https://github.com/openzfs/zfs/pull/15022 > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-20 7:05 ` Aaron Lauterer @ 2025-05-20 17:21 ` Randy Bush 2025-05-22 13:23 ` Aaron Lauterer 0 siblings, 1 reply; 8+ messages in thread From: Randy Bush @ 2025-05-20 17:21 UTC (permalink / raw) To: Aaron Lauterer; +Cc: Proxmox VE user list ok, makes sense. thanks. as this is production, we probably would not want to be early adopters of zfs 2.3 anyway. so, can i specify a second pool to be replication target, as i suggested in >> as we do full replication, we could just create a new pool for >> replocation and call it a day. i do not see an attribute for relocation pool in storage.cfg randy _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-20 17:21 ` Randy Bush @ 2025-05-22 13:23 ` Aaron Lauterer 2025-05-22 21:03 ` Randy Bush 0 siblings, 1 reply; 8+ messages in thread From: Aaron Lauterer @ 2025-05-22 13:23 UTC (permalink / raw) To: Randy Bush; +Cc: Proxmox VE user list On 2025-05-20 19:21, Randy Bush wrote: > ok, makes sense. thanks. as this is production, we probably would not > want to be early adopters of zfs 2.3 anyway. > > so, can i specify a second pool to be replication target, as i suggested > in > >>> as we do full replication, we could just create a new pool for >>> replocation and call it a day. > > i do not see an attribute for relocation pool in storage.cfg Where does the quote here come from? Did I miss an email in this thread? > > randy > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-22 13:23 ` Aaron Lauterer @ 2025-05-22 21:03 ` Randy Bush 2025-05-23 8:50 ` Aaron Lauterer 0 siblings, 1 reply; 8+ messages in thread From: Randy Bush @ 2025-05-22 21:03 UTC (permalink / raw) To: Aaron Lauterer; +Cc: Proxmox VE user list >> so, can i specify a second pool to be replication target, as i >> suggested in >>>> as we do full replication, we could just create a new pool for >>>> replocation and call it a day. >> >> i do not see an attribute for relocation pool in storage.cfg > Where does the quote here come from? Did I miss an email in this thread? all the above are from my keyboard on this thread randy _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-22 21:03 ` Randy Bush @ 2025-05-23 8:50 ` Aaron Lauterer 2025-05-23 18:17 ` Randy Bush 0 siblings, 1 reply; 8+ messages in thread From: Aaron Lauterer @ 2025-05-23 8:50 UTC (permalink / raw) To: Randy Bush; +Cc: Proxmox VE user list On 2025-05-22 23:03, Randy Bush wrote: >>> so, can i specify a second pool to be replication target, as i >>> suggested in >>>>> as we do full replication, we could just create a new pool for >>>>> replocation and call it a day. >>> >>> i do not see an attribute for relocation pool in storage.cfg > >> Where does the quote here come from? Did I miss an email in this thread? > > all the above are from my keyboard on this thread ah sorry. I was too blind yesterday to see that in your first email. There is no option to replicate a full ZFS pool to another. So, you have a current pool with one raidz2 VDEV made up of 4x 2TB disks. What kind of pool layout/sitation do you want to have in the end? Because if you have another set of 4x 2TB disks, you can just expand the pool with another raidz2 VDEV, without expanding the current VDEV you have. Right now, `zpool status` will show you something like: pool raidz2-0 disk0 disk1 disk2 disk3 If you add another VDEV, the pool could have the following layout: pool raidz2-0 disk0 disk1 disk2 disk3 raidz2-1 disk4 disk5 disk6 disk7 If you want to create a new pool, then things will be a bit more complicated, as you would need to create a new storage config for it as well, Move-Disk all the disks over to it. If you have a cluster and use the VM replication feature, that new pool must be present on the other nodes as well and you will have to remove the replication jobs before you move the disks to the new pool and then re-create them once all VM disks are on the new pool. _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-23 8:50 ` Aaron Lauterer @ 2025-05-23 18:17 ` Randy Bush 2025-05-26 7:38 ` Aaron Lauterer 0 siblings, 1 reply; 8+ messages in thread From: Randy Bush @ 2025-05-23 18:17 UTC (permalink / raw) To: Aaron Lauterer; +Cc: Proxmox VE user list > There is no option to replicate a full ZFS pool to another. not exactly what i want to do. my bad in saying "full replication." what i meant was all vms are replicated. on other nodes. i was thinking that each node could have one pool for primary vm images and a second to receive replication from other nodes. > So, you have a current pool with one raidz2 VDEV made up of 4x 2TB > disks. yup > Because if you have another set of 4x 2TB disks, you can just expand > the pool with another raidz2 VDEV, without expanding the current VDEV > you have. yup. what are the performance implications? > If you add another VDEV, the pool could have the following layout: > > pool > raidz2-0 > disk0 > disk1 > disk2 > disk3 > raidz2-1 > disk4 > disk5 > disk6 > disk7 yup > If you want to create a new pool, then things will be a bit more > complicated, as you would need to create a new storage config for it > as well, Move-Disk all the disks over to it. If you have a cluster and > use the VM replication feature, that new pool must be present on the > other nodes as well and you will have to remove the replication jobs > before you move the disks to the new pool and then re-create them once > all VM disks are on the new pool. we would keep the nodes all symmetric, so that would not be an issue. and it's just a few hours of ops pain to de-repl and re-repl. but what i do not see is how to tell `/etc/pve/storage.cfg` that pool0 is for images and pool1 is for incoming replication. maybe i am just trying to do something too weird. randy _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PVE-User] zfs raidz2 expansion 2025-05-23 18:17 ` Randy Bush @ 2025-05-26 7:38 ` Aaron Lauterer 0 siblings, 0 replies; 8+ messages in thread From: Aaron Lauterer @ 2025-05-26 7:38 UTC (permalink / raw) To: Randy Bush; +Cc: Proxmox VE user list On 2025-05-23 20:17, Randy Bush wrote: >> There is no option to replicate a full ZFS pool to another. > > not exactly what i want to do. my bad in saying "full replication." > what i meant was all vms are replicated. on other nodes. > > i was thinking that each node could have one pool for primary vm images > and a second to receive replication from other nodes. Ah okay. No, the way it works is that you have ZFS pools in both nodes with the same name. Then when the replication is configured for a guest, its disk images are replicated to the other node to the pool with the same name. > >> So, you have a current pool with one raidz2 VDEV made up of 4x 2TB >> disks. > > yup > > >> Because if you have another set of 4x 2TB disks, you can just expand >> the pool with another raidz2 VDEV, without expanding the current VDEV >> you have. > > yup. what are the performance implications? As usually with ZFS, the change will only affect newly written data. It will be spread over both VDEVs, with likely a bias to the newer, much emptier VDEV. So if you are already happy with the current performance, you should see similar or better performance, depending if one or both VDEVs are used to read/write data. > >> If you add another VDEV, the pool could have the following layout: >> >> pool >> raidz2-0 >> disk0 >> disk1 >> disk2 >> disk3 >> raidz2-1 >> disk4 >> disk5 >> disk6 >> disk7 > > yup > >> If you want to create a new pool, then things will be a bit more >> complicated, as you would need to create a new storage config for it >> as well, Move-Disk all the disks over to it. If you have a cluster and >> use the VM replication feature, that new pool must be present on the >> other nodes as well and you will have to remove the replication jobs >> before you move the disks to the new pool and then re-create them once >> all VM disks are on the new pool. > > we would keep the nodes all symmetric, so that would not be an issue. > and it's just a few hours of ops pain to de-repl and re-repl. but what > i do not see is how to tell `/etc/pve/storage.cfg` that pool0 is for > images and pool1 is for incoming replication. maybe i am just trying to > do something too weird. Yeah, see my reply at the beginning. I think you have a more complicated view of the replication than it actually is. If you are okay with the current performance, I would just add the second VDEV to the pool with `zpool add {pool} raidz2 /dev/disk/by-id/nvme-… /dev/disk/by-id/nvme-…` Before you do it on a production system, you can test the procedure in a (virtual) test machine to make sure you get the CLI command correct. By extending the pool, you don't need to change anything in the storage config or replication settings. > > randy > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-05-26 7:38 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-05-20 1:24 [PVE-User] zfs raidz2 expansion Randy Bush 2025-05-20 7:05 ` Aaron Lauterer 2025-05-20 17:21 ` Randy Bush 2025-05-22 13:23 ` Aaron Lauterer 2025-05-22 21:03 ` Randy Bush 2025-05-23 8:50 ` Aaron Lauterer 2025-05-23 18:17 ` Randy Bush 2025-05-26 7:38 ` Aaron Lauterer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.