[PVE-User] zfs raidz2 expansion

[PVE-User] zfs raidz2 expansion

[Randy Bush]

running 8.4.1 with images on 4x2tb raidz2 ssds (zfs-2.2.7-pve2).  we
want to double the space.  the net of a million lies says all sorts of
things about expanding raidz2.  as we do full replication, we could just
create a new pool for replocation and call it a day.  but i am curious
if we can just expand the current pool.  my reading says doubtful, but
am not deeply clued.

randy

---

https://forum.proxmox.com/threads/raidz-expansion.135413/
https://github.com/openzfs/zfs/pull/15022

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Aaron Lauterer]

You can always add a new  VDEV to the pool (man zpool-add). Expanding a 
raidz2 VDEV itself is only possible with ZFS 2.3 AFAIU and we do not 
ship that yet [0].

But if this is for VM storage, you might want to consider a ZFS pool of 
multiple mirrored VDEVs. [1] has more details.

While performance might not be much of an issue anymore with recent 
NVMEs that provide a ton of IOPS, the additional space usage might still 
be something to think about in a raidz pool used for VMs (ZVOL).


[0] 
https://forum.proxmox.com/threads/proxmox-ve-8-4-released.164821/#post-762367
[1] 
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_raid_considerations

On  2025-05-20  03:24, Randy Bush wrote:
> running 8.4.1 with images on 4x2tb raidz2 ssds (zfs-2.2.7-pve2).  we
> want to double the space.  the net of a million lies says all sorts of
> things about expanding raidz2.  as we do full replication, we could just
> create a new pool for replocation and call it a day.  but i am curious
> if we can just expand the current pool.  my reading says doubtful, but
> am not deeply clued.
> 
> randy
> 
> ---
> 
> https://forum.proxmox.com/threads/raidz-expansion.135413/
> https://github.com/openzfs/zfs/pull/15022
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 
> 



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Randy Bush]

ok, makes sense.  thanks.  as this is production, we probably would not
want to be early adopters of zfs 2.3 anyway.

so, can i specify a second pool to be replication target, as i suggested
in

>> as we do full replication, we could just create a new pool for
>> replocation and call it a day.

i do not see an attribute for relocation pool in storage.cfg

randy

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Aaron Lauterer]

On  2025-05-20  19:21, Randy Bush wrote:
> ok, makes sense.  thanks.  as this is production, we probably would not
> want to be early adopters of zfs 2.3 anyway.
> 
> so, can i specify a second pool to be replication target, as i suggested
> in
> 
>>> as we do full replication, we could just create a new pool for
>>> replocation and call it a day.
> 
> i do not see an attribute for relocation pool in storage.cfg

Where does the quote here come from? Did I miss an email in this thread?

> 
> randy
> 



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Randy Bush]

>> so, can i specify a second pool to be replication target, as i
>> suggested in
>>>> as we do full replication, we could just create a new pool for
>>>> replocation and call it a day.
>> 
>> i do not see an attribute for relocation pool in storage.cfg

> Where does the quote here come from? Did I miss an email in this thread?

all the above are from my keyboard on this thread

randy

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Aaron Lauterer]

On  2025-05-22  23:03, Randy Bush wrote:
>>> so, can i specify a second pool to be replication target, as i
>>> suggested in
>>>>> as we do full replication, we could just create a new pool for
>>>>> replocation and call it a day.
>>>
>>> i do not see an attribute for relocation pool in storage.cfg
> 
>> Where does the quote here come from? Did I miss an email in this thread?
> 
> all the above are from my keyboard on this thread

ah sorry. I was too blind yesterday to see that in your first email.

There is no option to replicate a full ZFS pool to another.

So, you have a current pool with one raidz2 VDEV made up of 4x 2TB disks.

What kind of pool layout/sitation do you want to have in the end?

Because if you have another set of 4x 2TB disks, you can just expand the 
pool with another raidz2 VDEV, without expanding the current VDEV you have.

Right now, `zpool status` will show you something like:

pool
   raidz2-0
     disk0
     disk1
     disk2
     disk3

If you add another VDEV, the pool could have the following layout:

pool
   raidz2-0
     disk0
     disk1
     disk2
     disk3
   raidz2-1
     disk4
     disk5
     disk6
     disk7

If you want to create a new pool, then things will be a bit more 
complicated, as you would need to create a new storage config for it as 
well, Move-Disk all the disks over to it. If you have a cluster and use 
the VM replication feature, that new pool must be present on the other 
nodes as well and you will have to remove the replication jobs before 
you move the disks to the new pool and then re-create them once all VM 
disks are on the new pool.



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Randy Bush]

> There is no option to replicate a full ZFS pool to another.

not exactly what i want to do.  my bad in saying "full replication."
what i meant was all vms are replicated. on other nodes.

i was thinking that each node could have one pool for primary vm images
and a second to receive replication from other nodes.

> So, you have a current pool with one raidz2 VDEV made up of 4x 2TB
> disks.

yup


> Because if you have another set of 4x 2TB disks, you can just expand
> the pool with another raidz2 VDEV, without expanding the current VDEV
> you have.

yup.  what are the performance implications?

> If you add another VDEV, the pool could have the following layout:
> 
> pool
>   raidz2-0
>     disk0
>     disk1
>     disk2
>     disk3
>   raidz2-1
>     disk4
>     disk5
>     disk6
>     disk7

yup

> If you want to create a new pool, then things will be a bit more
> complicated, as you would need to create a new storage config for it
> as well, Move-Disk all the disks over to it. If you have a cluster and
> use the VM replication feature, that new pool must be present on the
> other nodes as well and you will have to remove the replication jobs
> before you move the disks to the new pool and then re-create them once
> all VM disks are on the new pool.

we would keep the nodes all symmetric, so that would not be an issue.
and it's just a few hours of ops pain to de-repl and re-repl.  but what
i do not see is how to tell `/etc/pve/storage.cfg` that pool0 is for
images and pool1 is for incoming replication.  maybe i am just trying to
do something too weird.

randy

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs raidz2 expansion

[Aaron Lauterer]

On  2025-05-23  20:17, Randy Bush wrote:
>> There is no option to replicate a full ZFS pool to another.
> 
> not exactly what i want to do.  my bad in saying "full replication."
> what i meant was all vms are replicated. on other nodes.
> 
> i was thinking that each node could have one pool for primary vm images
> and a second to receive replication from other nodes.

Ah okay. No, the way it works is that you have ZFS pools in both nodes 
with the same name. Then when the replication is configured for a guest, 
its disk images are replicated to the other node to the pool with the 
same name.

> 
>> So, you have a current pool with one raidz2 VDEV made up of 4x 2TB
>> disks.
> 
> yup
> 
> 
>> Because if you have another set of 4x 2TB disks, you can just expand
>> the pool with another raidz2 VDEV, without expanding the current VDEV
>> you have.
> 
> yup.  what are the performance implications?

As usually with ZFS, the change will only affect newly written data. It 
will be spread over both VDEVs, with likely a bias to the newer, much 
emptier VDEV. So if you are already happy with the current performance, 
you should see similar or better performance, depending if one or both 
VDEVs are used to read/write data.

> 
>> If you add another VDEV, the pool could have the following layout:
>>
>> pool
>>    raidz2-0
>>      disk0
>>      disk1
>>      disk2
>>      disk3
>>    raidz2-1
>>      disk4
>>      disk5
>>      disk6
>>      disk7
> 
> yup
> 
>> If you want to create a new pool, then things will be a bit more
>> complicated, as you would need to create a new storage config for it
>> as well, Move-Disk all the disks over to it. If you have a cluster and
>> use the VM replication feature, that new pool must be present on the
>> other nodes as well and you will have to remove the replication jobs
>> before you move the disks to the new pool and then re-create them once
>> all VM disks are on the new pool.
> 
> we would keep the nodes all symmetric, so that would not be an issue.
> and it's just a few hours of ops pain to de-repl and re-repl.  but what
> i do not see is how to tell `/etc/pve/storage.cfg` that pool0 is for
> images and pool1 is for incoming replication.  maybe i am just trying to
> do something too weird.

Yeah, see my reply at the beginning. I think you have a more complicated 
view of the replication than it actually is.

If you are okay with the current performance, I would just add the 
second VDEV to the pool with
`zpool add {pool} raidz2 /dev/disk/by-id/nvme-… /dev/disk/by-id/nvme-…`

Before you do it on a production system, you can test the procedure in a 
(virtual) test machine to make sure you get the CLI command correct.

By extending the pool, you don't need to change anything in the storage 
config or replication settings.

> 
> randy
> 



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user