[pve-devel] applied: [PATCH v2 0/9] Improve live-restore speed and replace AsyncIndexReader

[Wolfgang Bumiller <w.bumiller@proxmox.com>]

applied series

On Mon, Jun 07, 2021 at 05:35:23PM +0200, Stefan Reiter wrote:
> This series is the third/fourth attempt[0] at substantially improving
> live-restore performance. This time, a fully async- and concurrency safe LRU
> cache is implemented, and a new CachedChunkReader is used to provide lock-free
> reading from a remote chunk source. The big performance improvements come from
> removing the global Mutex synchronising all read_image_at requests in the QEMU
> library and the shared LRU cache.
> 
> The idea of using 'mmap' to alleviate memory pressure was dropped in favor of
> using a shared cache between drives. This provides about the same performance
> improvement in typical live-restore scenarios, but with a fixed memory
> requirement, independant of drives. Adding 'mmap' caching is doable, but the
> Rust code really doesn't look very nice with it, and I think it's not necessary
> in this version...
> 
> Anyway, I figured out how to better benchmark the changes too, and can now
> reproduce the performance gains very reliably. Below is a (gnu)plot of running
> 40 live-restores of a Windows 10 VM with a single 32GB disk, first with the
> currently shipping libproxmox-backup-qemu0 (A) and then with the patched one
> from this series (B). Testing was done with QEMU 6.0.
> 
> 
> v2:
> * address Wolfgang's review comments, mostly style stuff
> 
> 
>  seconds
>  240 +-----------------------------------------------------------------------+
>      |        +        +        +        +        +        +   A    +        |
>  230 |-+                               A        A A                        +-|
>      |   A  A A            AA       A                  A     A  A            |
>  220 |-+        A AA              A        A                               +-|
>      |    A          A   A    A A    A       AA       A             A        |
>  210 |-+               A                            A      A      A   A  A A-|
>      |                                                                  A    |
>  200 |-+                                 A                                 +-|
>      |                                                                       |
>  190 |-+                                                                   +-|
>      |                                                                       |
>  180 |-A                                                                   +-|
>      |                                                                       |
>  170 |-+                                                                   +-|
>      |                            B                             B   B        |
>  160 |-B                    B   B      B B          B    B                 +-|
>      |                   B    B               B                   B     B    |
>  150 |-+ BB   B   BB B     B        B      B          B    B          B    B-|
>      |      B + B      +        +    B   +   B  B +    B   + B B    +    B   |
>  140 +-----------------------------------------------------------------------+
>      0        5        10       15       20       25       30       35       40
>                                      iteration
>      1.0.3-1    A
>      patched    B
> 
> 
> [0] see: https://lists.proxmox.com/pipermail/pbs-devel/2021-April/002932.html
> 
> 
> proxmox-backup: Stefan Reiter (7):
>   tools/BroadcastFuture: add testcase for better understanding
>   tools: add AsyncLruCache as a wrapper around sync LruCache
>   backup: add CachedChunkReader utilizing AsyncLruCache
>   backup: add AsyncRead/Seek to CachedChunkReader
>   replace AsyncIndexReader with SeekableCachedChunkReader
>   backup: remove AsyncIndexReader
>   tools/lru_cache: make minimum capacity 1
> 
>  src/api2/admin/datastore.rs            |   4 +-
>  src/backup.rs                          |   4 +-
>  src/backup/async_index_reader.rs       | 215 -------------------------
>  src/backup/cached_chunk_reader.rs      | 189 ++++++++++++++++++++++
>  src/bin/proxmox_backup_client/mount.rs |   4 +-
>  src/tools.rs                           |   1 +
>  src/tools/async_lru_cache.rs           | 135 ++++++++++++++++
>  src/tools/broadcast_future.rs          |  11 ++
>  src/tools/lru_cache.rs                 |   1 +
>  9 files changed, 343 insertions(+), 221 deletions(-)
>  delete mode 100644 src/backup/async_index_reader.rs
>  create mode 100644 src/backup/cached_chunk_reader.rs
>  create mode 100644 src/tools/async_lru_cache.rs
> 
> proxmox-backup-qemu: Stefan Reiter (2):
>   add shared_cache module
>   access: use CachedChunkReader
> 
>  src/lib.rs          |  7 ++++++-
>  src/restore.rs      | 30 ++++++++----------------------
>  src/shared_cache.rs | 37 +++++++++++++++++++++++++++++++++++++
>  3 files changed, 51 insertions(+), 23 deletions(-)
>  create mode 100644 src/shared_cache.rs
> 
> -- 
> 2.30.2