From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id AEF591FF0B2 for ; Mon, 24 Aug 2026 12:23:46 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5347A21545; Mon, 24 Aug 2026 12:23:46 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; t=1787566082; cv=none; d=zohomail.com.au; s=zohoarc; b=cLVMdtPBkv9YnRV5fQhc3XmBhKGYHnlj6Vooyu0Ydf4edJEVDO6c7zBliFQfuvee0L2BjpFl/eKSft95nSVDJch35HKlna6FoarEnomEnQJS4vDcbUNoRdKkoVQmtaDPUddz3tdYc8iky9igHT5C4YI1QH0VPCh4vZHNTn9d+NM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com.au; s=zohoarc; t=1787566082; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:Reply-To:Subject:Subject:To:To:Message-Id; bh=LuNuc23q+Nli/svlMqlrGzjGf248BukfSe55Izo4Ot4=; b=e6j8NQ9kHcxDkgp8AS5MzW+qQFpQn+FwFFS4Wi3Ztdy29K4HYLvDyy+Qo5o96YkUGfSKJpfSvFFu4jYiqLr+pdwOzuW0yATaxyfgTDEIrsRu4/gGFTg6oKmhnBhKF2BEaiB+85gYEEaLYROAi1blpu2FiExDfQ6LrkFvgfeyV4g= ARC-Authentication-Results: i=1; mx.zohomail.com.au; dkim=pass header.i=mooresolutions.co.nz; spf=pass smtp.mailfrom=sales@mooresolutions.co.nz; dmarc=pass header.from= DKIM-Signature: a=rsa-sha256; b=hK7Fuvss3SFOU8lZGW3LKqHHyRQI5OF7PufqJ6VfAVRoOjhKIABxdMrshatrSzyjyZ3SJ0ierC9vE8dokSaGIJUK6I3CIcjRmHVK21rmiToZ2UzlWFeAAlLWkyRDvNcquVOhc8JtkmGt4b8+wpRm1ue7llokCpM8GuJKa+PYFUY=; c=relaxed/relaxed; s=zmail; d=mooresolutions.co.nz; v=1; bh=LuNuc23q+Nli/svlMqlrGzjGf248BukfSe55Izo4Ot4=; h=Content-Description:Content-ID:Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Owner:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Resent-Sender:Resent-Cc:Resent-Date:Resent-To:Reply-To:Resent-Message-ID:Resent-From:Sender:Subject:To:Feedback-ID:Date:From:Message-Id:Reply-To:Subject:To:Cc; Date: Mon, 24 Aug 2026 22:08:02 +1200 From: thomas moore To: "Christian Ebner" Message-Id: <1a0333dc86a.55d63485246212.4924095821986999565@mooresolutions.co.nz> In-Reply-To: <1186c9b2-f08e-4db1-9c8c-e415f256f835@proxmox.com> References: <1a026d2b773.4f0fd0cd237366.1462259800663760854@mooresolutions.co.nz> <1186c9b2-f08e-4db1-9c8c-e415f256f835@proxmox.com> Subject: Re: [RFC] add max-depth to the datastore snapshots API endpoint MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_901146_1882695093.1787566082154" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-SPAM-LEVEL: Spam detection results: 0 AWL -0.025 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KZLDMTZ6V3A2F6KR3HAVE4UKGHZV3JSF X-Message-ID-Hash: KZLDMTZ6V3A2F6KR3HAVE4UKGHZV3JSF X-MailFrom: sales@mooresolutions.co.nz X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pbs-devel X-Mailman-Version: 3.3.10 Precedence: list Reply-To: sales@mooresolutions.co.nz List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: ------=_Part_901146_1882695093.1787566082154 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > Yes, although there has already been some work on this [0], > implementing a streaming API which should already cover your case as > well. Thanks, that patch covers both halves of what I asked for, and better than what I proposed. I had not found it. > Nevertheless, may I ask what your goal is and what limitation which is > not already covered by the current API you are running into? Does > performing the additional requests cause to much latency for your > usecase? It is not latency. I have three namespaces, so the extra requests are not a problem at that scale. At larger namespace counts it clearly. would be, which is what the streaming call addresses. The problem was that a client calling the documented snapshots endpoint gets root-only results with nothing indicating other namespaces exist. I have since implemented enumerate then fetch per namespace, so this is about discoverability for third-party clients rather than something I am blocked on. Happy to drop my proposal and wait for the streaming content API to land. Regards, Thomas Moore From: Christian Ebner To: , "pbs-devel" Date: Mon, 24 Aug 2026 21:39:23 +1200 Subject: Re: [RFC] add max-depth to the datastore snapshots API endpoint Hi!=20 =20 On 8/22/26 2:30 AM, thomas moore wrote:=20 > Hi all=20 >=20 > This is a proposal for discussion, not a patch. The=20 > developer=C2=A0documentation asks that plans be discussed before=20 > starting work, so=C2=A0I'd like to check whether this would be=20 > acceptable before writing it.=20 >=20 > GET /api2/json/admin/datastore/{store}/snapshots returns only=20 > the=C2=A0snapshots in the requested namespace, so a client wanting=20 > a=C2=A0datastore-wide view has to call=20 > /admin/datastore/{store}/namespace=C2=A0first and then issue one=20 > snapshots request per namespace. I ran into=C2=A0this writing a=20 > third-party monitoring client.=20 >=20 > Tested against PBS 4.2:=20 >=20 > /admin/datastore/pbs/snapshots -> 485 snapshots (root=20 > only)=20 >=20 > /admin/datastore/pbs/snapshots?ns=3Dtest -> 1 snapshot=20 >=20 > /admin/datastore/pbs/snapshots?max-depth=3D7=C2=A0-> 400=20 > parameter verification failed - 'max-depth': schema does=C2=A0not=20 > allow additional properties=20 >=20 > Proposal: accept an optional max-depth using the=20 > existing=C2=A0NS_MAX_DEPTH_SCHEMA, defaulting to 0 so behaviour is=20 > unchanged for=C2=A0current callers.=20 >=20 > This looks anticipated in the code. As of=20 > 5d95fc20e,=C2=A0src/api2/admin/datastore.rs lines 571, 575 and 583=20 > each carry a=C2=A0"// FIXME: Recursion" on the group=20 > construction in=C2=A0list_snapshots_blocking, with a further one at=20 > 565 about filtering by=C2=A0owner before collecting.=20 > ListAccessibleBackupGroups already takes=C2=A0max_depth and applies=20 > the per-namespace privilege and owner checks=C2=A0during iteration,=20 > and get_snapshots_count uses it in roughly the shape=C2=A0list_snapshots= =20 > would need.=20 >=20 > I'd send it as two patches: the refactor=20 > onto=C2=A0ListAccessibleBackupGroups with no behavioural change, then=20 > max-depth=C2=A0on top. I can test on a 4.2 instance including with a=20 > token holding=C2=A0DatastoreAudit on the root namespace only, to=20 > confirm namespaces the=C2=A0token cannot see stay absent from a=20 > recursive listing.=20 >=20 >=20 >=20 >=20 >=20 > Questions for the list=20 >=20 > 1. Is adding max-depth to the snapshots endpoint acceptable in=20 > principle?=20 =20 Yes, although there has already been some work on this [0], implementing=20 a streaming API which should already cover your case as well. This=20 however is not applied yet, but might be picked up again.=20 =20 Nevertheless, may I ask what your goal is and what limitation which is=20 not already covered by the current API you are running into? Does=20 performing the additional requests cause to much latency for your usecase?= =20 =20 > 2. Is ListAccessibleBackupGroups the right vehicle here, or is=20 > there a reason list_snapshots =C2=A0=C2=A0=C2=A0=C2=A0does not already us= e it?=20 =20 list_snapshots_blocking() is rather heavy weight and performance=20 critical, especially for datastores with lots of groups/snapshots on=20 slow (spin rust) storage. So performance and memory analysis is a must=20 here. Further, current behavior must be retained also with respect to=20 group listing by type and backup id only, respectively. But again, I=20 think patch [0] already would cover what you are asking for.=20 =20 > 3. Snapshot entries carry no namespace field, so a recursive=20 > listing is ambiguous unless the =C2=A0=C2=A0=C2=A0=C2=A0client tracks it = per request. Adding=20 > one would mean a new field on SnapshotListItem, which =C2=A0=C2=A0=C2=A0= =C2=A0is also=20 > decoded from remote instances in server/sync.rs and server/push.rs=20 =20 While not that critical, adding the additional field does have=20 implications for the response payload size, especially for deeply nested=20 datastores with lots of snapshots.=20 =20 > If you'd rather this were solved another way or not at all I'm=20 > happy=C2=A0to hear it before writing anything.=20 >=20 > Regards=C2=A0Thomas Moore=20 =20 [0]=20 https://lore.proxmox.com/pbs-devel/20251008134344.3512958-8-d.csapak@proxmo= x.com/ =20 =20 Best regards,=20 Chris ------=_Part_901146_1882695093.1787566082154 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
> Yes, although there has already been some wor= k on this [0],
> implementing a streaming API which should= already cover your case as
> well.

Thanks, that patch covers both halves of what I asked for, and bette= r
than what I proposed. I had not found it.
> Nevertheless, may I ask what your goal is and what limitat= ion which is
> not already covered by the current API you = are running into? Does
> performing the additional request= s cause to much latency for your
> usecase?
=
It is not latency. I have three namespaces, so the extra req= uests are
not a problem at that scale. At larger namespace co= unts it clearly.
would be, which is what the streaming call a= ddresses. The problem was
that a client calling the documente= d snapshots endpoint gets root-only
results with nothing indi= cating other namespaces exist. I have since
implemented enume= rate then fetch per namespace, so this is about
discoverabili= ty for third-party clients rather than something I am
blocked= on.

Happy to drop my proposal and wait for th= e streaming content API to
land.

Regards,
Thomas Moore


From: Christian Ebner <c.ebner@proxmox.com>
To: <sales@= mooresolutions.co.nz>, "pbs-devel"<pbs-devel@lists.proxmox.com>Date: Mon, 24 Aug 2026 21:39:23 +1200
Subject: Re: [RFC] add max-depth = to the datastore snapshots API endpoint

Hi!

On 8/22/2= 6 2:30 AM, thomas moore wrote:
> Hi all
>
> This is a = proposal for discussion, not a patch. The
> developer documenta= tion asks that plans be discussed before
> starting work, so I'= d like to check whether this would be
> acceptable before writing it= .
>
> GET /api2/json/admin/datastore/{store}/snapshots return= s only
> the snapshots in the requested namespace, so a client = wanting
> a datastore-wide view has to call
> /admin/dat= astore/{store}/namespace first and then issue one
> snapshots r= equest per namespace. I ran into this writing a
> third-party m= onitoring client.
>
> Tested against PBS 4.2:
>
&g= t; /admin/datastore/pbs/snapshots -> 485 snapshots (root
&= gt; only)
>
> /admin/datastore/pbs/snapshots?ns=3Dtest ->= ; 1 snapshot
>
> /admin/datastore/pbs/snapshots?max-depth=3D7=  -> 400
> parameter verification failed - 'max-depth': schem= a does not
> allow additional properties
>
> Prop= osal: accept an optional max-depth using the
> existing NS_MAX_= DEPTH_SCHEMA, defaulting to 0 so behaviour is
> unchanged for c= urrent callers.
>
> This looks anticipated in the code. As of=
> 5d95fc20e, src/api2/admin/datastore.rs lines 571, 575 and 58= 3
> each carry a "// FIXME: Recursion" on the group
> co= nstruction in list_snapshots_blocking, with a further one at
> = 565 about filtering by owner before collecting.
> ListAccessibl= eBackupGroups already takes max_depth and applies
> the per-nam= espace privilege and owner checks during iteration,
> and get_s= napshots_count uses it in roughly the shape list_snapshots
> wo= uld need.
>
> I'd send it as two patches: the refactor
&g= t; onto ListAccessibleBackupGroups with no behavioural change, then > max-depth on top. I can test on a 4.2 instance including with a=
> token holding DatastoreAudit on the root namespace only, to =
> confirm namespaces the token cannot see stay absent from a > recursive listing.
>
>
>
>
>
= > Questions for the list
>
> 1. Is adding max-depth to the= snapshots endpoint acceptable in
> principle?

Yes, althoug= h there has already been some work on this [0], implementing
a streamin= g API which should already cover your case as well. This
however is not= applied yet, but might be picked up again.

Nevertheless, may I as= k what your goal is and what limitation which is
not already covered by= the current API you are running into? Does
performing the additional r= equests cause to much latency for your usecase?

> 2. Is ListAcc= essibleBackupGroups the right vehicle here, or is
> there a reason l= ist_snapshots     does not already use it?

lis= t_snapshots_blocking() is rather heavy weight and performance
critical,= especially for datastores with lots of groups/snapshots on
slow (spin = rust) storage. So performance and memory analysis is a must
here. Furth= er, current behavior must be retained also with respect to
group listin= g by type and backup id only, respectively. But again, I
think patch [0= ] already would cover what you are asking for.

> 3. Snapshot en= tries carry no namespace field, so a recursive
> listing is ambiguou= s unless the     client tracks it per request. Adding <= br>> one would mean a new field on SnapshotListItem, which   &= nbsp; is also
> decoded from remote instances in server/sync.rs= and server/push.rs

While not that critical, adding the additional= field does have
implications for the response payload size, especially= for deeply nested
datastores with lots of snapshots.

> If = you'd rather this were solved another way or not at all I'm
> happy&= nbsp;to hear it before writing anything.
>
> Regards Tho= mas Moore

[0]
https://lor= e.proxmox.com/pbs-devel/20251008134344.3512958-8-d.csapak@proxmox.com/ =

Best regards,
Chris





------=_Part_901146_1882695093.1787566082154--