public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
@ 2022-04-22 10:04 Dietmar Maurer
  2022-04-22 10:54 ` Roland Sturm
  0 siblings, 1 reply; 6+ messages in thread
From: Dietmar Maurer @ 2022-04-22 10:04 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Daniel Koch

> Using PBS for NTFS with the de-duplication feature enabled (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in
> corrupted file restores.

Not answering your questions, but using deduplication on the file system does not make sense, because PBS already do the deduplication...




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
  2022-04-22 10:04 [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files Dietmar Maurer
@ 2022-04-22 10:54 ` Roland Sturm
  0 siblings, 0 replies; 6+ messages in thread
From: Roland Sturm @ 2022-04-22 10:54 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion

Hello

Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".

We also use Win Dedup on File Servers


More Info see: Data Deduplication terminology 
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand

The chunk store is an organized series of container files in the System Volume Information folder that Data Deduplication uses to uniquely store chunks.


kr

Mit freundlichen Grüssen

Roland Sturm
Senior Systems Engineer

cibex gmbh
6406 Oberhofen, Krautfeldweg 11 | 6361 Hopfgarten, Brixentalerstr. 21

<tel> +43.50688.33 | <fax> +43.50688.50
<mail> r.sturm@cibex.net | <web>  www.cibex.net

Aktuelle Informationen auf Facebook: https://www.facebook.com/cibexgmbh

-----Ursprüngliche Nachricht-----
Von: pbs-devel <pbs-devel-bounces@lists.proxmox.com> Im Auftrag von Dietmar Maurer
Gesendet: Freitag, 22. April 2022 12:04
An: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>; Daniel Koch <koch@triple6.org>
Betreff: Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

> Using PBS for NTFS with the de-duplication feature enabled 
> (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in corrupted file restores.

Not answering your questions, but using deduplication on the file system does not make sense, because PBS already do the deduplication...


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
  2022-04-27  7:35 ` Fabian Grünbichler
@ 2022-04-27 12:16   ` Thomas Lamprecht
  0 siblings, 0 replies; 6+ messages in thread
From: Thomas Lamprecht @ 2022-04-27 12:16 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion,
	Fabian Grünbichler, Roland Sturm

On 27.04.22 09:35, Fabian Grünbichler wrote:
> users do report success with mapping + attaching the disk to an existing 
> (version-matching) windows VM for manual file-restoring, so we could 
> also think whether it's possible to streamline this approach somehow 
> ("attach disk RO from backup" on the PVE side?), since it's a lot more 
> flexible for other use cases as well (e.g., encrypted disks inside the 
> VM, storage management or FS we don't support in our file-restore VM, 
> FS features that are not available in our file-restore VM kernel (yet or 
> anymore ;)), ..). obviously an advanced use case, and for simple cases 
> the existing file-restore approach is far nicer UX-wise.

IMO the better idea than using some other NTFS implementation, as there's no
free one that can support all newest features and details 100%.
This can also be useful in general, i.e., for other, more esoteric FS.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
  2022-04-27  4:30 Dietmar Maurer
@ 2022-04-27  7:35 ` Fabian Grünbichler
  2022-04-27 12:16   ` Thomas Lamprecht
  0 siblings, 1 reply; 6+ messages in thread
From: Fabian Grünbichler @ 2022-04-27  7:35 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Roland Sturm

On April 27, 2022 6:30 am, Dietmar Maurer wrote:
>> Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
>> Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".
>> 
>> We also use Win Dedup on File Servers
> 
> And you can reproduce the problem?
> 

see (German) https://forum.proxmox.com/threads/file-restore-dateien-defekt.108257

we could see whether ntfs-3g works better, but I am afraid from what I 
read in their issues they are playing a bit of cat and mouse with 
changes on the windows side, and the feature seems to not be backwards 
compatible in the windows world either, so chances are it will work 
better the older the VM's version of windows is.

alternatively, we could try to detect deduplicated files in the 
file-restore VM (based on some metadata?) and hard-fail the restore 
instead of returning valid-looking garbage.

users do report success with mapping + attaching the disk to an existing 
(version-matching) windows VM for manual file-restoring, so we could 
also think whether it's possible to streamline this approach somehow 
("attach disk RO from backup" on the PVE side?), since it's a lot more 
flexible for other use cases as well (e.g., encrypted disks inside the 
VM, storage management or FS we don't support in our file-restore VM, 
FS features that are not available in our file-restore VM kernel (yet or 
anymore ;)), ..). obviously an advanced use case, and for simple cases 
the existing file-restore approach is far nicer UX-wise.




^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
@ 2022-04-27  4:30 Dietmar Maurer
  2022-04-27  7:35 ` Fabian Grünbichler
  0 siblings, 1 reply; 6+ messages in thread
From: Dietmar Maurer @ 2022-04-27  4:30 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Roland Sturm

> Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
> Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".
> 
> We also use Win Dedup on File Servers

And you can reproduce the problem?




^ permalink raw reply	[flat|nested] 6+ messages in thread

* [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files
@ 2022-04-19  9:11 Daniel Koch
  0 siblings, 0 replies; 6+ messages in thread
From: Daniel Koch @ 2022-04-19  9:11 UTC (permalink / raw)
  To: pbs-devel

Hello List,

Current situation:

Using PBS for NTFS with the de-duplication feature enabled (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in
corrupted file restores.

ntfs3 can't handle de-duplicated file systems atm.

ntfs-3g by default is not aware of ntfs'es de-duplication but provides plugins for those features:
https://jp-andre.pagesperso-orange.fr/advanced-ntfs-3g-old.html#download

Is there any chance that the restore kernel will use ntfs-3g over ntfs3 with dedup capabilities? The de-duplication feature of NTFS is widely used and people don't seam to be aware that their backups are actually corrupted.

Restoring the whole disk image will be fine tho.

Kind regards,
Daniel



^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-04-27 12:16 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-22 10:04 [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files Dietmar Maurer
2022-04-22 10:54 ` Roland Sturm
  -- strict thread matches above, loose matches on Subject: below --
2022-04-27  4:30 Dietmar Maurer
2022-04-27  7:35 ` Fabian Grünbichler
2022-04-27 12:16   ` Thomas Lamprecht
2022-04-19  9:11 Daniel Koch

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal