[pve-devel] Adding new data storage

[pve-devel] Adding new data storage

[Andrei Perapiolkin]

Hi,


I would like to add new data storage. This storage would resemble ZFS 
over iSCSI but will use different API to access storage.

Should I create/provide anything besides 
pve-storage/PVE/Storage/<new-driver>.pm ?


Is there any documentation I should familiarize myself with beside 
coding style?


Best regards,

Andrei Perepiolkin

Re: [pve-devel] Adding new data storage

[Thomas Lamprecht]

Hi,

On 17/12/2020 16:06, Andrei Perapiolkin wrote:
> I would like to add new data storage. This storage would resemble ZFS over iSCSI but will use different API to access storage.
> 

What do you mean here, what API will it use? If you want to include this in our
repository some more details would be appreciated. Else, for storages with have
a external plugin system which can be used to include plugins which are proprietary
or otherwise unfit for inclusion.

> Should I create/provide anything besides pve-storage/PVE/Storage/<new-driver>.pm ?

Once above is cleared, then yes, and base it off the closed related
storage plugin - i.e., maybe ZFS, else the base plugin.

> 
> 
> Is there any documentation I should familiarize myself with beside coding style?

our Perl style guide is here:
https://pve.proxmox.com/wiki/Perl_Style_Guide

Besides that, check the developer documentation in general, especially those two
sections:

https://pve.proxmox.com/wiki/Developer_Documentation#Preparing_Patches
https://pve.proxmox.com/wiki/Developer_Documentation#Software_License_and_Copyright

regards,
Thomas

Re: [pve-devel] Adding new data storage

[Andrei Perapiolkin]

Hi Thomas, Dietmar,

Thanks for the quick reply.


Sorry for ambiguity with "API". By API I meant usage of SSH to send 
commands to Open-E JovianDSS.

The problem here is that this commands are different from the one used 
in "ZFS over iSCSI" plugin.

That is why Im thinking about adding separate plugin.


Also this plugin is going to be OpenSource and API is public, so there 
should not be any legal complications with adding it to Proxmox repo.

My concern is whether I should provide additional CI, Unit tests or 
something else?

Coding style goes without saying.


On 12/17/20 5:35 PM, Thomas Lamprecht wrote:
> Hi,
>
> On 17/12/2020 16:06, Andrei Perapiolkin wrote:
>> I would like to add new data storage. This storage would resemble ZFS over iSCSI but will use different API to access storage.
>>
> What do you mean here, what API will it use? If you want to include this in our
> repository some more details would be appreciated. Else, for storages with have
> a external plugin system which can be used to include plugins which are proprietary
> or otherwise unfit for inclusion.



Best regards,

Andrei Perepiolkin

Re: [pve-devel] Adding new data storage

[Thomas Lamprecht]

Hi again,

On 17/12/2020 17:16, Andrei Perapiolkin wrote:
> Hi Thomas, Dietmar,
> 
> Thanks for the quick reply.
> 
> 
> Sorry for ambiguity with "API". By API I meant usage of SSH to send commands to Open-E JovianDSS.
> 
> The problem here is that this commands are different from the one used in "ZFS over iSCSI" plugin.
> 
> That is why Im thinking about adding separate plugin.
> 
> 
> Also this plugin is going to be OpenSource and API is public, so there should not be any legal complications with adding it to Proxmox repo.

To be included into our git tree the backing implementation also needs
to have an open source license, ideally with an open development model (not
just some code dump download side) covering the storages API and the plugin
itself under an open license is not enough.

Taking in a plugin in our tree means for us to ensure it is maintained and safe
to use for our users and enterprise support, and for that access to source
code, among other things, is crucial.

But, as mentioned, we have a mechanism to support external Storages plugins,
with mechanisms to ensure it can stay compatible and adapt to API changes (using a
GNU libtool like /APIVER/sion and /APIAGE/ definition[1][2])

Such external plugins integrate fully in the backend, they are somewhat limited
in the user interface, can be selected for use everywhere once defined, but not
added or edited itself - for that the CLI needs to be used.

Boiled down, you'd need to provide a perl module which would get installed to
`/usr/share/perl5/PVE/Storage/Custom/`, it needs to have an api sub which
returns the API version the plugin supports, further, it then needs to have the
respective sub methods implementing that interface.

If unsure, check some external plugins, like the Linstore one[3], or also our
internal ones, for example the ceph RBD or ZFS Pool one.


[1]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=PVE/Storage.pm;h=bd6e15e9f1f9f05dec9a6cbcd88391705ff08212;hb=HEAD#l43
[2]: https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
[3]: https://github.com/LINBIT/linstor-proxmox


> 
> My concern is whether I should provide additional CI, Unit tests or something else?
> 

Correct me if I'm mistaken, but FWICT, the referred storage tech is not free (meaning
the definition of free software from FSF/GNU itself, that can probably not be stated in
a general, absolute, manner and needs to be looked at, at a case by case basis - GPL
compatible would be definitively remove most to all road blocks).

Independent of that, adding a new storage plugin would require at least some basic tests,
the more the better.

regards,
Thomas

> Coding style goes without saying.
> 
> 
> On 12/17/20 5:35 PM, Thomas Lamprecht wrote:
>> Hi,
>>
>> On 17/12/2020 16:06, Andrei Perapiolkin wrote:
>>> I would like to add new data storage. This storage would resemble ZFS over iSCSI but will use different API to access storage.
>>>
>> What do you mean here, what API will it use? If you want to include this in our
>> repository some more details would be appreciated. Else, for storages with have
>> a external plugin system which can be used to include plugins which are proprietary
>> or otherwise unfit for inclusion.
> 
> 
> 
> Best regards,
> 
> Andrei Perepiolkin
> 
> 

Re: [pve-devel] Adding new data storage

[Dietmar Maurer]

> I would like to add new data storage. This storage would resemble ZFS 
> over iSCSI but will use different API to access storage.

I am curios, what API exactly?