[pve-devel] Interest in a file manager interface within pve-manager?

[pve-devel] Interest in a file manager interface within pve-manager?

[Blythe, Nathan F. - US via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14367 bytes --]

From: "Blythe, Nathan F. - US" <nathaniel.blythe@caci.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Interest in a file manager interface within pve-manager?
Date: Thu, 11 Jul 2024 22:27:32 +0000
Message-ID: <SN5P110MB08631CBC92535BADBE9076B8E1A5A@SN5P110MB0863.NAMP110.PROD.OUTLOOK.COM>

Hello,

We have a need for a non-network based (i.e. hypercall / out-of-band) file browser for containers and VMs within the pve-manager web UI. I'd like to be able to select a VM or container, click "More", and choose "File manager" and then browse the VM or container's file systems and upload or download files from my local system, through the web-browser.

For VMs (qemu), the qemu-guest-agent provides the ability to read/write guest files, and the pve API exposes it (with strict file size limits) as nodes->{node}->qemu->{vmid}->agent->file-*. Presumably, a web UI file manager would use that API endpoint, plus maybe an enhancement for directory listing and chunked upload/download (to support arbitrary size files).

For containers (LXC), the pct application can read/write files by (I think) mounting the container root file system on the host and touching it directly. But there's no corresponding API endpoint. If there were, it could be used by a future web UI file manager.

So to implement this feature, I think there are 5 steps:

  1.  Enhance the qemu-guest-agent and qm to support reading directory structure and creating directories.
  2.  Enhance the pve API to expose these new features like file-read and file-write.
  3.  Enhance the pve API to support chunked file read and write, for large files. Maybe also requiring an enhancement to the qemu-guest-agent?
  4.  Enhance the pve API to support chunked file read/write, whole file read/write, and directory structure/creating directories (through direct filesystem interaction on the host).
  5.  Write a JS front-end file manager, invokable from the Other menu, which uses these new/enhanced API endpoints to implement a general purpose out-of-band file manager.

Is there interest from the proxmox developers for something like this? If we started working on some of these items, would there be interest in accepting patches? Does my general approach look sound?

(We also investigated using SPICE's folder-sharing capabilities and the spice-proxy provided by the host, but that doesn't work with the Windows virt-viewer client, only the Linux client. It looks like there was some interest in the past in enhancing xterm.js to support xmodem file transfer, but it didn't go far.)

Regards,
Nathan Blythe
CACI

________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Interest in a file manager interface within pve-manager?

[Dominik Csapak]

On 7/12/24 00:27, Blythe, Nathan F. - US via pve-devel wrote:
> 
> Hello,
> 

Hi, sorry for the late answer.

First thanks for a well written proposal/question. This is what we'd like to see if third-parties
are interested in making features/changes/etc.

I try to answer the question/proposal to the best of my knowledge, but maybe some of my colleagues
have additional things to say or want to correct me
(this could take a bit though, due to summer holiday season)

> We have a need for a non-network based (i.e. hypercall / out-of-band) file browser for containers and VMs within the pve-manager web UI. I'd like to be able to select a VM or container, click "More", and choose "File manager" and then browse the VM or container's file systems and upload or download files from my local system, through the web-browser.
> 
> For VMs (qemu), the qemu-guest-agent provides the ability to read/write guest files, and the pve API exposes it (with strict file size limits) as nodes->{node}->qemu->{vmid}->agent->file-*. Presumably, a web UI file manager would use that API endpoint, plus maybe an enhancement for directory listing and chunked upload/download (to support arbitrary size files).
> 
> For containers (LXC), the pct application can read/write files by (I think) mounting the container root file system on the host and touching it directly. But there's no corresponding API endpoint. If there were, it could be used by a future web UI file manager.
> 
> So to implement this feature, I think there are 5 steps:
> 
>    1.  Enhance the qemu-guest-agent and qm to support reading directory structure and creating directories.
>    2.  Enhance the pve API to expose these new features like file-read and file-write.
>    3.  Enhance the pve API to support chunked file read and write, for large files. Maybe also requiring an enhancement to the qemu-guest-agent?
>    4.  Enhance the pve API to support chunked file read/write, whole file read/write, and directory structure/creating directories (through direct filesystem interaction on the host).
>    5.  Write a JS front-end file manager, invokable from the Other menu, which uses these new/enhanced API endpoints to implement a general purpose out-of-band file manager.
> 
> Is there interest from the proxmox developers for something like this? If we started working on some of these items, would there be interest in accepting patches? Does my general approach look sound?

After a bit of internal discussion, the general tone is that we probably don't want to expose the
guest content/state by default this much now, but in the longer term it could be OK, but we see the
following issues:

* For now QEMU allows those guest calls by default, and there is at the moment not a really good
   way for configuring what's allowed and what not.
* Our privileges and ACLs are not really designed for this currently.
* Handling permissions (and other filesystem/guest "quirks") will be very hard
   (e.g. windows permissions are very different from linux/unix ones)

As breaking the host/guest barrier can have bad implications on security/data integrity/etc.
those problems would have to be solved before one could/should start with such a feature.

There is currently some work done on the QEMU side for limiting/configuring QGA calls:
https://lore.kernel.org/qemu-devel/20240604153242.251334-1-berrange@redhat.com/

When that's integrated, one could think about how our ACLs, privileges could be extended to allow
such things.

Only after those are solved, implementing this feature make sense.

What could be done in the meantime probably is to work with upstream QEMU to add
a directory listing to the qga and maybe some proper permission handling?

This is relatively independent from our side, since we don't actually ship
the guest agent ourselves, so this has to be solved upstream anyway.

So all in all, such a feature could work, but only in the longer term, this is nothing
for the short/midterm.

Hope this answers your questions. If not, just write again :)

> 
> (We also investigated using SPICE's folder-sharing capabilities and the spice-proxy provided by the host, but that doesn't work with the Windows virt-viewer client, only the Linux client. It looks like there was some interest in the past in enhancing xterm.js to support xmodem file transfer, but it didn't go far.)
> 
> Regards,
> Nathan Blythe
> CACI


Best regards
Dominik


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Interest in a file manager interface within pve-manager?

[Dominik Csapak]

sorry for the noise, it seems i forgot to put you in CC, so now again with that...

On 8/13/24 10:35, Dominik Csapak wrote:
> On 7/12/24 00:27, Blythe, Nathan F. - US via pve-devel wrote:
>>
>> Hello,
>>
> 
> Hi, sorry for the late answer.
> 
> First thanks for a well written proposal/question. This is what we'd like to see if third-parties
> are interested in making features/changes/etc.
> 
> I try to answer the question/proposal to the best of my knowledge, but maybe some of my colleagues
> have additional things to say or want to correct me
> (this could take a bit though, due to summer holiday season)
> 
>> We have a need for a non-network based (i.e. hypercall / out-of-band) file browser for containers 
>> and VMs within the pve-manager web UI. I'd like to be able to select a VM or container, click 
>> "More", and choose "File manager" and then browse the VM or container's file systems and upload or 
>> download files from my local system, through the web-browser.
>>
>> For VMs (qemu), the qemu-guest-agent provides the ability to read/write guest files, and the pve 
>> API exposes it (with strict file size limits) as nodes->{node}->qemu->{vmid}->agent->file-*. 
>> Presumably, a web UI file manager would use that API endpoint, plus maybe an enhancement for 
>> directory listing and chunked upload/download (to support arbitrary size files).
>>
>> For containers (LXC), the pct application can read/write files by (I think) mounting the container 
>> root file system on the host and touching it directly. But there's no corresponding API endpoint. 
>> If there were, it could be used by a future web UI file manager.
>>
>> So to implement this feature, I think there are 5 steps:
>>
>>    1.  Enhance the qemu-guest-agent and qm to support reading directory structure and creating 
>> directories.
>>    2.  Enhance the pve API to expose these new features like file-read and file-write.
>>    3.  Enhance the pve API to support chunked file read and write, for large files. Maybe also 
>> requiring an enhancement to the qemu-guest-agent?
>>    4.  Enhance the pve API to support chunked file read/write, whole file read/write, and 
>> directory structure/creating directories (through direct filesystem interaction on the host).
>>    5.  Write a JS front-end file manager, invokable from the Other menu, which uses these new/ 
>> enhanced API endpoints to implement a general purpose out-of-band file manager.
>>
>> Is there interest from the proxmox developers for something like this? If we started working on 
>> some of these items, would there be interest in accepting patches? Does my general approach look 
>> sound?
> 
> After a bit of internal discussion, the general tone is that we probably don't want to expose the
> guest content/state by default this much now, but in the longer term it could be OK, but we see the
> following issues:
> 
> * For now QEMU allows those guest calls by default, and there is at the moment not a really good
>    way for configuring what's allowed and what not.
> * Our privileges and ACLs are not really designed for this currently.
> * Handling permissions (and other filesystem/guest "quirks") will be very hard
>    (e.g. windows permissions are very different from linux/unix ones)
> 
> As breaking the host/guest barrier can have bad implications on security/data integrity/etc.
> those problems would have to be solved before one could/should start with such a feature.
> 
> There is currently some work done on the QEMU side for limiting/configuring QGA calls:
> https://lore.kernel.org/qemu-devel/20240604153242.251334-1-berrange@redhat.com/
> 
> When that's integrated, one could think about how our ACLs, privileges could be extended to allow
> such things.
> 
> Only after those are solved, implementing this feature make sense.
> 
> What could be done in the meantime probably is to work with upstream QEMU to add
> a directory listing to the qga and maybe some proper permission handling?
> 
> This is relatively independent from our side, since we don't actually ship
> the guest agent ourselves, so this has to be solved upstream anyway.
> 
> So all in all, such a feature could work, but only in the longer term, this is nothing
> for the short/midterm.
> 
> Hope this answers your questions. If not, just write again :)
> 
>>
>> (We also investigated using SPICE's folder-sharing capabilities and the spice-proxy provided by 
>> the host, but that doesn't work with the Windows virt-viewer client, only the Linux client. It 
>> looks like there was some interest in the past in enhancing xterm.js to support xmodem file 
>> transfer, but it didn't go far.)
>>
>> Regards,
>> Nathan Blythe
>> CACI
> 
> 
> Best regards
> Dominik
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Interest in a file manager interface within pve-manager?

[Blythe, Nathan F. - US via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 23264 bytes --]

From: "Blythe, Nathan F. - US" <nathaniel.blythe@caci.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: RE: [pve-devel] Interest in a file manager interface within pve-manager?
Date: Thu, 15 Aug 2024 16:38:55 +0000
Message-ID: <CY1P110MB0855CA241B6986FBED197150E180A@CY1P110MB0855.NAMP110.PROD.OUTLOOK.COM>

Dominik,

Thank you for the detailed response! I see what you mean about the limitations on controlling guest access with pve RBAC. In our specific situation that's mostly a non-issue: users are scripting the deployment, management, and shutdown of containers/VMs for parallelization of long-running tasks, so they completely own all their "subtask" VMs, which only exist transiently. But in general, it does seem like the upstream qemu tooling could use finer-grained permission of some kind, which could be mapped to pve RBAC.

For now we're going to use IP-based file sharing services between our VMs/containers. In the future, however, we may be interested in putting some effort towards some of those upstream "enabling" features that you described, both for this eventual purpose and for general improvements of the tooling that proxmox uses. We'll revisit this next year, as we move from prototype to production use.

Another question, though, what if we restricted this only to containers? Does that change the feasibility? Maybe new roles: ContainerGuestFileAudit and ContainerGuestFileAdmin. Owner/group would just be set to match the root user (already known to proxmox, at least during new container creation). File permissions could be set by UI or just defaulted to the typical 644/755.

I appreciate the team taking the time to discuss!

Regards,
Nathan


-----Original Message-----
From: Dominik Csapak <d.csapak@proxmox.com>
Sent: Tuesday, August 13, 2024 2:44 AM
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Cc: Blythe, Nathan F. - US <nathaniel.blythe@caci.com>
Subject: Re: [pve-devel] Interest in a file manager interface within pve-manager?

EXTERNAL EMAIL - This email originated from outside of CACI. Do not click any links or attachments unless you recognize and trust the sender.





sorry for the noise, it seems i forgot to put you in CC, so now again with that...

On 8/13/24 10:35, Dominik Csapak wrote:
> On 7/12/24 00:27, Blythe, Nathan F. - US via pve-devel wrote:
>>
>> Hello,
>>
>
> Hi, sorry for the late answer.
>
> First thanks for a well written proposal/question. This is what we'd
> like to see if third-parties are interested in making features/changes/etc.
>
> I try to answer the question/proposal to the best of my knowledge, but
> maybe some of my colleagues have additional things to say or want to
> correct me (this could take a bit though, due to summer holiday
> season)
>
>> We have a need for a non-network based (i.e. hypercall / out-of-band)
>> file browser for containers and VMs within the pve-manager web UI.
>> I'd like to be able to select a VM or container, click "More", and
>> choose "File manager" and then browse the VM or container's file systems and upload or download files from my local system, through the web-browser.
>>
>> For VMs (qemu), the qemu-guest-agent provides the ability to
>> read/write guest files, and the pve API exposes it (with strict file size limits) as nodes->{node}->qemu->{vmid}->agent->file-*.
>> Presumably, a web UI file manager would use that API endpoint, plus
>> maybe an enhancement for directory listing and chunked upload/download (to support arbitrary size files).
>>
>> For containers (LXC), the pct application can read/write files by (I
>> think) mounting the container root file system on the host and touching it directly. But there's no corresponding API endpoint.
>> If there were, it could be used by a future web UI file manager.
>>
>> So to implement this feature, I think there are 5 steps:
>>
>>    1.  Enhance the qemu-guest-agent and qm to support reading
>> directory structure and creating directories.
>>    2.  Enhance the pve API to expose these new features like file-read and file-write.
>>    3.  Enhance the pve API to support chunked file read and write,
>> for large files. Maybe also requiring an enhancement to the qemu-guest-agent?
>>    4.  Enhance the pve API to support chunked file read/write, whole
>> file read/write, and directory structure/creating directories (through direct filesystem interaction on the host).
>>    5.  Write a JS front-end file manager, invokable from the Other
>> menu, which uses these new/ enhanced API endpoints to implement a general purpose out-of-band file manager.
>>
>> Is there interest from the proxmox developers for something like
>> this? If we started working on some of these items, would there be
>> interest in accepting patches? Does my general approach look sound?
>
> After a bit of internal discussion, the general tone is that we
> probably don't want to expose the guest content/state by default this
> much now, but in the longer term it could be OK, but we see the following issues:
>
> * For now QEMU allows those guest calls by default, and there is at the moment not a really good
>    way for configuring what's allowed and what not.
> * Our privileges and ACLs are not really designed for this currently.
> * Handling permissions (and other filesystem/guest "quirks") will be very hard
>    (e.g. windows permissions are very different from linux/unix ones)
>
> As breaking the host/guest barrier can have bad implications on security/data integrity/etc.
> those problems would have to be solved before one could/should start with such a feature.
>
> There is currently some work done on the QEMU side for limiting/configuring QGA calls:
> https://lore.kernel.org/qemu-devel/20240604153242.251334-1-berrange@re
> dhat.com/
>
> When that's integrated, one could think about how our ACLs, privileges
> could be extended to allow such things.
>
> Only after those are solved, implementing this feature make sense.
>
> What could be done in the meantime probably is to work with upstream
> QEMU to add a directory listing to the qga and maybe some proper permission handling?
>
> This is relatively independent from our side, since we don't actually
> ship the guest agent ourselves, so this has to be solved upstream anyway.
>
> So all in all, such a feature could work, but only in the longer term,
> this is nothing for the short/midterm.
>
> Hope this answers your questions. If not, just write again :)
>
>>
>> (We also investigated using SPICE's folder-sharing capabilities and
>> the spice-proxy provided by the host, but that doesn't work with the
>> Windows virt-viewer client, only the Linux client. It looks like
>> there was some interest in the past in enhancing xterm.js to support
>> xmodem file transfer, but it didn't go far.)
>>
>> Regards,
>> Nathan Blythe
>> CACI
>
>
> Best regards
> Dominik
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>



________________________________

This electronic message contains information from CACI International Inc or subsidiary companies, which may be company sensitive, proprietary, privileged or otherwise protected from disclosure. The information is intended to be used solely by the recipient(s) named above. If you are not an intended recipient, be aware that any review, disclosure, copying, distribution or use of this transmission or its contents is prohibited. If you have received this transmission in error, please notify the sender immediately.

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel