Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Dietmar Maurer]

> Using PBS for NTFS with the de-duplication feature enabled (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in
> corrupted file restores.

Not answering your questions, but using deduplication on the file system does not make sense, because PBS already do the deduplication...

Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Roland Sturm]

Hello

Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".

We also use Win Dedup on File Servers


More Info see: Data Deduplication terminology 
https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand

The chunk store is an organized series of container files in the System Volume Information folder that Data Deduplication uses to uniquely store chunks.


kr

Mit freundlichen Grüssen

Roland Sturm
Senior Systems Engineer

cibex gmbh
6406 Oberhofen, Krautfeldweg 11 | 6361 Hopfgarten, Brixentalerstr. 21

<tel> +43.50688.33 | <fax> +43.50688.50
<mail> r.sturm@cibex.net | <web>  www.cibex.net

Aktuelle Informationen auf Facebook: https://www.facebook.com/cibexgmbh

-----Ursprüngliche Nachricht-----
Von: pbs-devel <pbs-devel-bounces@lists.proxmox.com> Im Auftrag von Dietmar Maurer
Gesendet: Freitag, 22. April 2022 12:04
An: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>; Daniel Koch <koch@triple6.org>
Betreff: Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

> Using PBS for NTFS with the de-duplication feature enabled 
> (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in corrupted file restores.

Not answering your questions, but using deduplication on the file system does not make sense, because PBS already do the deduplication...


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Thomas Lamprecht]

On 27.04.22 09:35, Fabian Grünbichler wrote:
> users do report success with mapping + attaching the disk to an existing 
> (version-matching) windows VM for manual file-restoring, so we could 
> also think whether it's possible to streamline this approach somehow 
> ("attach disk RO from backup" on the PVE side?), since it's a lot more 
> flexible for other use cases as well (e.g., encrypted disks inside the 
> VM, storage management or FS we don't support in our file-restore VM, 
> FS features that are not available in our file-restore VM kernel (yet or 
> anymore ;)), ..). obviously an advanced use case, and for simple cases 
> the existing file-restore approach is far nicer UX-wise.

IMO the better idea than using some other NTFS implementation, as there's no
free one that can support all newest features and details 100%.
This can also be useful in general, i.e., for other, more esoteric FS.

Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Fabian Grünbichler]

On April 27, 2022 6:30 am, Dietmar Maurer wrote:
>> Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
>> Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".
>> 
>> We also use Win Dedup on File Servers
> 
> And you can reproduce the problem?
> 

see (German) https://forum.proxmox.com/threads/file-restore-dateien-defekt.108257

we could see whether ntfs-3g works better, but I am afraid from what I 
read in their issues they are playing a bit of cat and mouse with 
changes on the windows side, and the feature seems to not be backwards 
compatible in the windows world either, so chances are it will work 
better the older the VM's version of windows is.

alternatively, we could try to detect deduplicated files in the 
file-restore VM (based on some metadata?) and hard-fail the restore 
instead of returning valid-looking garbage.

users do report success with mapping + attaching the disk to an existing 
(version-matching) windows VM for manual file-restoring, so we could 
also think whether it's possible to streamline this approach somehow 
("attach disk RO from backup" on the PVE side?), since it's a lot more 
flexible for other use cases as well (e.g., encrypted disks inside the 
VM, storage management or FS we don't support in our file-restore VM, 
FS features that are not available in our file-restore VM kernel (yet or 
anymore ;)), ..). obviously an advanced use case, and for simple cases 
the existing file-restore approach is far nicer UX-wise.

Re: [pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Dietmar Maurer]

> Think he would like to restore files from a windows Server (like a fileserver) with "Windows dedup" enabled on a volume.
> Win Dedup works with chunk-files in "System Volume Information" Folder and metadatafiles to the "chunk-data".
> 
> We also use Win Dedup on File Servers

And you can reproduce the problem?

[pbs-devel] PBS File Retore for NTFS with deduplication enabled leads to corrupted files

[Daniel Koch]

Hello List,

Current situation:

Using PBS for NTFS with the de-duplication feature enabled (https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview) results in
corrupted file restores.

ntfs3 can't handle de-duplicated file systems atm.

ntfs-3g by default is not aware of ntfs'es de-duplication but provides plugins for those features:
https://jp-andre.pagesperso-orange.fr/advanced-ntfs-3g-old.html#download

Is there any chance that the restore kernel will use ntfs-3g over ntfs3 with dedup capabilities? The de-duplication feature of NTFS is widely used and people don't seam to be aware that their backups are actually corrupted.

Restoring the whole disk image will be fine tho.

Kind regards,
Daniel