[pve-devel] Cluster storage plugin assistant service

[pve-devel] Cluster storage plugin assistant service

[Andrei Perapiolkin via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 7530 bytes --]

From: Andrei Perapiolkin <andrei.perepiolkin@open-e.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: [pve-devel] Cluster storage plugin assistant service
Date: Wed, 28 May 2025 11:42:46 -0400
Message-ID: <59e66c97-2337-4717-8c8f-c7b31b47da6b@open-e.com>

Hi Proxmox Community,

I am working on implementing custom storage plugin and after evaluating 
various edge cases related to volume live migration, I have decided to 
create a 'storage assistant service'.
This service will be responsible for managing my storage plugin volumes 
activation and deactivation across all nodes in the cluster.
The idea is to run a service on every cluster node that conducts:
     Volume activation upon creation
     Volume deactivation upon deletion

To achieve this, the service must receive notifications from the storage 
plugin whenever a volume is created or deleted.

However, I am not yet familiar with the internal mechanisms of Proxmox 
clustering and am unsure what technology or approach would be most 
appropriate for such inter-node communication.

At present, I am considering using the Proxmox Cluster File System 
(pmxcfs)—specifically the '/etc/pve/priv' directory—as a means of 
transferring notifications through file creation and deletion.
Another option I’m exploring is executing remote commands over SSH.

I would appreciate any feedback on potential issues with this approach.
Additionally, if there are any existing services or projects that have 
solved similar design problems, I would be grateful to learn about them.


Best regards,

Andrei



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Cluster storage plugin assistant service

[Fabian Grünbichler]

> Andrei Perapiolkin via pve-devel <pve-devel@lists.proxmox.com> hat am 28.05.2025 17:42 CEST geschrieben:
> Hi Proxmox Community,
> 
> I am working on implementing custom storage plugin and after evaluating 
> various edge cases related to volume live migration, I have decided to 
> create a 'storage assistant service'.
> This service will be responsible for managing my storage plugin volumes 
> activation and deactivation across all nodes in the cluster.
> The idea is to run a service on every cluster node that conducts:
>      Volume activation upon creation

you shouldn't (need to) activate a volume on all nodes upon creation - before
a volume is used by PVE it must always be activated on the node where it is
used, anything else is a bug.

>      Volume deactivation upon deletion

that part I understand (given the context from the other thread:

https://lore.proxmox.com/pve-devel/1351543294.511.1748849575029@webmail.proxmox.com/T/#t

but it seems to me there is a far easier solution if you are worried about
the following sequence of events:

volume is activated
volume is freed (but somehow volume remains active, e.g. some partial error
handling)
=> dangling activation might confuse something, eat up resources, ..
same volume is created again
=> existing activation clashes with newly created volume

you could just
- regularly run some housekeeping task that compares local activation state
  against storage contents => any leftover activation where the corresponding
  volume doesn't exist must be bogus and can be removed/cleared
- before creating a volume, explicitly check whether an activation for it
  exists => that activation must be bogus and can be removed/cleared as well

no need to coordinate across nodes for any of that, you just need a single
call to your housekeeping service to tell it to check volume XYZ. of course
how that housekeeping looks like very much depends on how your storage works
and your plugin is implemented ;)

> To achieve this, the service must receive notifications from the storage 
> plugin whenever a volume is created or deleted.
> 
> However, I am not yet familiar with the internal mechanisms of Proxmox 
> clustering and am unsure what technology or approach would be most 
> appropriate for such inter-node communication.
> 
> At present, I am considering using the Proxmox Cluster File System 
> (pmxcfs)—specifically the '/etc/pve/priv' directory—as a means of 
> transferring notifications through file creation and deletion.
> Another option I’m exploring is executing remote commands over SSH.

storage plugins should not drop random files in /etc/pve - they might
clash with future additions on the PVE side, and there's quite a bit
of restrictions that you'd have to be aware of.

> I would appreciate any feedback on potential issues with this approach.
> Additionally, if there are any existing services or projects that have 
> solved similar design problems, I would be grateful to learn about them.
> 
> 
> Best regards,
> 
> Andrei


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel