public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH-SERIES v4 pxar proxmox-backup proxmox-widget-toolkit 00/26] fix #3174: improve file-level backup
@ 2023-11-09 18:45 Christian Ebner
  2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 1/26] fix #3174: decoder: factor out skip_bytes from skip_entry Christian Ebner
                   ` (26 more replies)
  0 siblings, 27 replies; 32+ messages in thread
From: Christian Ebner @ 2023-11-09 18:45 UTC (permalink / raw)
  To: pbs-devel

Changes to the patch series since version 3 are based on the feedback
obtained via internal communication channels. Many thanks to Thomas,
Fabian, Wolfgang and Dominik for their continuous feedback up until
now.

This series of patches implements an metadata based file change
detection mechanism for improved pxar file level backup creation speed
for unchanged files.

The chosen approach is to skip encoding of regular file payloads,
for which metadata (currently ctime and size) did not change as
compared to a previous backup run. Instead of re-encoding the files, a
reference to a newly introduced appendix section of the pxar archive
will be written. The appendix section will be created as concatenation
of indexed chunks from the previous backup run, thereby containing the
sequential file payload at a calculated offset with respect to the
starting point of the appendix section.

Metadata comparison and calculation of the chunks to be indexed for the
appendix section is performed using the catalog of a previous backup as
reference. In order to be able to calculate the offsets, an updated
catalog file format version 2 is introduced which extends the previous
version by including the file offset with respect to the pxar archive
byte stream, as well as the files ctime. This allows to find the required
chunks indexes and the start padding within the concatenated chunks.
The catalog reader remains backwards compatible to the catalog file
format version 1.

During encoding, the chunks needed for the appendix section are injected
in the backup upload stream after forcing a chunk boundary when regular
pxar encoding is finished. Finally, the pxar archive containing an
appendix section are marked as such by appending a final pxar goodbye
lookup table only containing the offset to the appendix section start and
total size of that section, needed for random access as e.g. to mount
the archive via the fuse filesystem implementation.

The following lists the most notable changes included in this series since
the version 3:

- count appendix chunks as reused chunks, as they are not re-encoded and
  re-uploaded.
- add a heuristic to reduce chunk fragmentation for appendix chunks for
  multiple consecutive backup runs with metadata based file change detection.
- refactor the appendix list generation during pxar encoding in the archiver.
- switch from Vec to BTreeMap for the restoring the appendix entries so
  entries are inserted in sorted order based on their offset, making it
  unnecessary to sort afterwards.
- fix issue with chunk injection code which lead to data corruption in some
  edge cases, add additional checks as fortification.

The following lists the most notable changes included in this series since
the version 2:

- avoid re-indexing the same chunks multiple times in the appendix
  section by looking them up in the already present appendix chunks list
  and calculate the appendix reference offset accordingly. This now
  requires to sort entries by their appendix start offset for sequential
  restore.
- reduce appendix reference chunk fragmentation by increasing the file
  size threshold to 1k.
- Fix the WebUIs file browser and single file restore, broken in the
  previous patch series.
- fixes previous catalog and/or dynamic index downloads when either the
  backup group was empty or no archive with the same name present in the
  backup

The following lists the most notable changes included in this series since
the version 1:

- fixes and refactors the missing chunk issue by modifying the logic to
  avoid re-appending the same chunks multiple times if referenced by
  multiple consecutive files.
- fixes a performance issue with catalog lookup being the bottleneck
  for cases with directories with many entries, resulting in the
  metadata based file change detection performing worse than the regular
  mode.
- fixes the creation of multi archive backups. All of the archives use
  the same reference catalog.
- the catalog file format is pushed to version 2, including the needed
  archive offsets as well as ctime for file change detection.
- the catalog is fully backward compatible to catalog file format
  version 1, so both can be decoded by the reader. However, the new
  version of the catalog file format will be used for all new backups
- it is now possible to perform multiple consecutive runs of the backup
  with metadata based file change detection, no more need to perform the
  regular run previous to the other one.
- change from `incremental` flag to enum based `BackupDetectionMode`
  parameter for command invocations.
- includes a new `proxmox-backup-test-suite` binary to create and run
  benchmarks to compare the performance of the different detection modes.

An invocation of a backup run with this patches now is:
```bash
proxmox-backup-client backup <label>.pxar:<source-path> --change-detection-mode=metadata
```
During the first run, no reference catalog can be used, the backup will
be a regular run. Following backups will however utilize the catalog and
index files of the previous run to perform file change detection.

As benchmarks, the linux source code as well as the coco dataset for
computer vision and pattern recognition can be used.
The benchmarks can be performed by running:
```bash
proxmox-backup-test-suite detection-mode-bench prepare --target /<path-to-bench-source-target>
proxmox-backup-test-suite detection-mode-bench run linux.pxar:/<path-to-bench-source-target>/linux
proxmox-backup-test-suite detection-mode-bench run coco.pxar:/<path-to-bench-source-target>/coco
```

Above command invocations assume the default repository and credentials
to be set as environment variables, they might however be passed as
additinal optional parameters instead.

Benchmark runs using these test data show a significant improvement in
the time needed for the backups:

For the linux source code backup:
    Completed benchmark with 5 runs for each tested mode.

    Completed regular backup with:
    Total runtime: 69.79 s
    Average: 13.96 ± 0.07 s
    Min: 13.88 s
    Max: 14.05 s

    Completed metadata detection mode backup with:
    Total runtime: 8.72 s
    Average: 1.74 ± 0.05 s
    Min: 1.65 s
    Max: 1.79 s

    Differences (metadata based - regular):
    Delta total runtime: -61.07 s (-87.50 %)
    Delta average: -12.21 ± 0.09 s (-87.50 %)
    Delta min: -12.23 s (-88.10 %)
    Delta max: -12.26 s (-87.26 %)

For the coco dataset backup:
    Completed benchmark with 5 runs for each tested mode.

    Completed regular backup with:
    Total runtime: 545.56 s
    Average: 109.11 ± 0.50 s
    Min: 108.26 s
    Max: 109.45 s

    Completed metadata detection mode backup with:
    Total runtime: 116.69 s
    Average: 23.34 ± 1.66 s
    Min: 22.16 s
    Max: 26.26 s

    Differences (metadata based - regular):
    Delta total runtime: -428.87 s (-78.61 %)
    Delta average: -85.77 ± 1.74 s (-78.61 %)
    Delta min: -86.10 s (-79.53 %)
    Delta max: -83.19 s (-76.00 %)

Above tests were performed inside a test VM backing up to a local
datastore.

pxar:

Christian Ebner (7):
  fix #3174: decoder: factor out skip_bytes from skip_entry
  fix #3174: decoder: impl skip_bytes for sync dec
  fix #3174: encoder: calc filename + metadata byte size
  fix #3174: enc/dec: impl PXAR_APPENDIX_REF entrytype
  fix #3174: enc/dec: impl PXAR_APPENDIX entrytype
  fix #3174: encoder: helper to add to encoder position
  fix #3174: enc/dec: impl PXAR_APPENDIX_TAIL entrytype

 examples/mk-format-hashes.rs |  11 +++
 examples/pxarcmd.rs          |   4 +-
 src/accessor/mod.rs          | 132 ++++++++++++++++++++++++-----
 src/decoder/mod.rs           |  50 +++++++++--
 src/decoder/sync.rs          |   5 ++
 src/encoder/aio.rs           |  38 ++++++++-
 src/encoder/mod.rs           | 156 ++++++++++++++++++++++++++++++++++-
 src/encoder/sync.rs          |  45 +++++++++-
 src/format/mod.rs            |  16 ++++
 src/lib.rs                   |  10 +++
 10 files changed, 429 insertions(+), 38 deletions(-)

proxmox-backup:

Christian Ebner (18):
  fix #3174: index: add fn index list from start/end-offsets
  fix #3174: index: add fn digest for DynamicEntry
  fix #3174: api: double catalog upload size
  fix #3174: catalog: introduce extended format v2
  fix #3174: archiver/extractor: impl appendix ref
  fix #3174: catalog: add specialized Archive entry
  fix #3174: extractor: impl seq restore from appendix
  fix #3174: archiver: store ref to previous backup
  fix #3174: upload stream: impl reused chunk injector
  fix #3174: chunker: add forced boundaries
  fix #3174: backup writer: inject queued chunk in upload steam
  fix #3174: archiver: reuse files with unchanged metadata
  fix #3174: specs: add backup detection mode specification
  fix #3174: client: Add detection mode to backup creation
  test-suite: add detection mode change benchmark
  test-suite: Add bin to deb, add shell completions
  catalog: fetch offset and size for files and refs
  pxar: add heuristic to reduce reused chunk fragmentation

 Cargo.toml                                    |   1 +
 Makefile                                      |  13 +-
 debian/proxmox-backup-client.bash-completion  |   1 +
 debian/proxmox-backup-client.install          |   2 +
 debian/proxmox-backup-test-suite.bc           |   8 +
 examples/test_chunk_speed2.rs                 |   9 +-
 pbs-client/src/backup_specification.rs        |  30 +
 pbs-client/src/backup_writer.rs               |  89 ++-
 pbs-client/src/catalog_shell.rs               |   3 +-
 pbs-client/src/chunk_stream.rs                |  42 +-
 pbs-client/src/inject_reused_chunks.rs        | 155 +++++
 pbs-client/src/lib.rs                         |   1 +
 pbs-client/src/pxar/create.rs                 | 373 +++++++++-
 pbs-client/src/pxar/extract.rs                | 151 +++-
 pbs-client/src/pxar/mod.rs                    |   2 +-
 pbs-client/src/pxar/tools.rs                  |   9 +
 pbs-client/src/pxar_backup_stream.rs          |   8 +-
 pbs-datastore/src/catalog.rs                  | 644 ++++++++++++++++--
 pbs-datastore/src/dynamic_index.rs            |  38 ++
 pbs-datastore/src/file_formats.rs             |   3 +
 proxmox-backup-client/src/main.rs             | 176 ++++-
 proxmox-backup-test-suite/Cargo.toml          |  18 +
 .../src/detection_mode_bench.rs               | 288 ++++++++
 proxmox-backup-test-suite/src/main.rs         |  17 +
 .../src/proxmox_restore_daemon/api.rs         |  21 +-
 pxar-bin/src/main.rs                          |  23 +-
 src/api2/backup/upload_chunk.rs               |   4 +-
 src/tape/file_formats/snapshot_archive.rs     |   2 +-
 tests/catar.rs                                |   3 +
 zsh-completions/_proxmox-backup-test-suite    |  13 +
 30 files changed, 1971 insertions(+), 176 deletions(-)
 create mode 100644 debian/proxmox-backup-test-suite.bc
 create mode 100644 pbs-client/src/inject_reused_chunks.rs
 create mode 100644 proxmox-backup-test-suite/Cargo.toml
 create mode 100644 proxmox-backup-test-suite/src/detection_mode_bench.rs
 create mode 100644 proxmox-backup-test-suite/src/main.rs
 create mode 100644 zsh-completions/_proxmox-backup-test-suite

proxmox-widget-toolkit:

Christian Ebner (1):
  file-browser: support pxar archive and fileref types

 src/Schema.js             |  2 ++
 src/window/FileBrowser.js | 16 ++++++++++++----
 2 files changed, 14 insertions(+), 4 deletions(-)

-- 
2.39.2





^ permalink raw reply	[flat|nested] 32+ messages in thread

end of thread, other threads:[~2023-11-13 15:45 UTC | newest]

Thread overview: 32+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-11-09 18:45 [pbs-devel] [PATCH-SERIES v4 pxar proxmox-backup proxmox-widget-toolkit 00/26] fix #3174: improve file-level backup Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 1/26] fix #3174: decoder: factor out skip_bytes from skip_entry Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 2/26] fix #3174: decoder: impl skip_bytes for sync dec Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 3/26] fix #3174: encoder: calc filename + metadata byte size Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 4/26] fix #3174: enc/dec: impl PXAR_APPENDIX_REF entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 5/26] fix #3174: enc/dec: impl PXAR_APPENDIX entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 6/26] fix #3174: encoder: helper to add to encoder position Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 7/26] fix #3174: enc/dec: impl PXAR_APPENDIX_TAIL entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 08/26] fix #3174: index: add fn index list from start/end-offsets Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 09/26] fix #3174: index: add fn digest for DynamicEntry Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 10/26] fix #3174: api: double catalog upload size Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 11/26] fix #3174: catalog: introduce extended format v2 Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 12/26] fix #3174: archiver/extractor: impl appendix ref Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 13/26] fix #3174: catalog: add specialized Archive entry Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 14/26] fix #3174: extractor: impl seq restore from appendix Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 15/26] fix #3174: archiver: store ref to previous backup Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 16/26] fix #3174: upload stream: impl reused chunk injector Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 17/26] fix #3174: chunker: add forced boundaries Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 18/26] fix #3174: backup writer: inject queued chunk in upload steam Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 19/26] fix #3174: archiver: reuse files with unchanged metadata Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 20/26] fix #3174: specs: add backup detection mode specification Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 21/26] fix #3174: client: Add detection mode to backup creation Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 22/26] test-suite: add detection mode change benchmark Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 23/26] test-suite: Add bin to deb, add shell completions Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 24/26] catalog: fetch offset and size for files and refs Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 25/26] pxar: add heuristic to reduce reused chunk fragmentation Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-widget-toolkit 26/26] file-browser: support pxar archive and fileref types Christian Ebner
2023-11-13 14:23 ` [pbs-devel] [PATCH-SERIES v4 pxar proxmox-backup proxmox-widget-toolkit 00/26] fix #3174: improve file-level backup Fabian Grünbichler
2023-11-13 15:14   ` Christian Ebner
2023-11-13 15:21     ` Christian Ebner
2023-11-13 15:35     ` Fabian Grünbichler
2023-11-13 15:45       ` Christian Ebner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal