From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 7336E1FF177 for ; Fri, 2 Aug 2024 15:28:17 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D8E12DE1A; Fri, 2 Aug 2024 15:28:13 +0200 (CEST) From: Max Carrara To: pve-devel@lists.proxmox.com Date: Fri, 2 Aug 2024 15:26:46 +0200 Message-Id: <20240802132656.270077-9-m.carrara@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240802132656.270077-1-m.carrara@proxmox.com> References: <20240802132656.270077-1-m.carrara@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -2.471 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SOMETLD_ARE_BAD_TLD 5 .bar, .beauty, .buzz, .cam, .casa, .cfd, .club, .date, .guru, .link, .live, .monster, .online, .press, .pw, .quest, .rest, .sbs, .shop, .stream, .top, .trade, .wiki, .work, .xyz TLD abuse SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH v1 pve-common 08/18] pbsclient: document package and its public functions & methods X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" This commit adds a brief overview for the `PVE::PBSClient` package and documents its public functions and methods. Examples are added where deemed appropriate. Signed-off-by: Max Carrara --- src/PVE/PBSClient.pm | 526 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 511 insertions(+), 15 deletions(-) diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm index 231406a..e0468d3 100644 --- a/src/PVE/PBSClient.pm +++ b/src/PVE/PBSClient.pm @@ -1,5 +1,4 @@ package PVE::PBSClient; -# utility functions for interaction with Proxmox Backup client CLI executable use strict; use warnings; @@ -13,14 +12,83 @@ use POSIX qw(mkfifo strftime ENOENT); use PVE::JSONSchema qw(get_standard_option); use PVE::Tools qw(run_command file_set_contents file_get_contents file_read_firstline $IPV6RE); -# returns a repository string suitable for proxmox-backup-client, pbs-restore, etc. -# $scfg must have the following structure: -# { -# datastore -# server -# port (optional defaults to 8007) -# username (optional defaults to 'root@pam') -# } +=pod + +=head1 NAME + +PVE::PBSClient - Proxmox Backup Client Library + +=head1 DESCRIPTION + +This package contains utilities that wrap common Proxmox Backup client CLI +operations. + +=head2 THE CLIENT OBJECT + +While the C> package contains regular L, +the majority is done via the C> object. This object represents +a client that is used to connect to a Proxmox Backup Server: + + use strict; + use warnings; + + use Data::Dumper; + + $Data::Dumper::Indent = 1; + $Data::Dumper::Quotekeys = 0; + $Data::Dumper::Sortkeys = 1; + $Data::Dumper::Terse = 1; + + use PVE::PBSClient; + + my $scfg = { + server => 'example.tld', + username => 'alice@pam', + datastore => 'foo-store', + fingerprint => '...', + }; + + my $client = PVE::PBSClient->new($scfg, "pbs-main"); + + my ($total, $free, $used) = $client->status(); + + print "Datastore has a total capacity of $total bytes, of which $used bytes are used" + . "and $free bytes are still available.\n"; + + my $snapshot = "vm/1337/2024-07-30T10:01:57Z"; + my $filepath = "/"; + + my $file_list = $client->file_restore_list($snapshot, $filepath); + + print "The snapshot '$snapshot' contains the following restorable files:\n"; + print Dumper($file_list); + print "\n"; + + +=head1 FUNCTIONS + +=cut + +=pod + +=head3 get_repository + + $repository = get_repository($scfg) + +Returns a repository string suitable for the C and +C executables. + +The C<$scfg> hash must have the following structure: + + { + datastore => 'my-datastore-name', + server => 'example.tld', + port => 8007, # optional, defaults to 8007 + username => 'user@realm', # optional, defaults to 'root@pam' + } + +=cut + sub get_repository { my ($scfg) = @_; @@ -41,6 +109,58 @@ sub get_repository { return "$username\@$server:$datastore"; } +=pod + +=head1 METHODS + +=cut + +=pod + +=head3 new + + $client = PVE::PBSClient->new($scfg, $storeid) + $client = PVE::PBSClient->new($scfg, $storeid, $secret_dir) + +Creates a new instance of a C>. + +Throws an exception if no C<$scfg> hash is provided or if C<$storeid> is C. + +=over + +=item C<$scfg> + +The I hash that the client should use. + +This hash is expected to have the following structure: + + { + datastore => 'my-datastore-name', + namespace => 'my-namespace', + server => 'example.tld', + fingerprint => '...', + port => 8007, # optional, defaults to 8007 + username => 'user@realm', # optional, defaults to 'root@pam' + } + +=item C<$storeid> + +The I of the storage corresponding to C<$scfg>. This ID is used for operations +concerning the I and I, such as C> and +C>. + +=item C<$secret_dir> (optional) + +The name of the I in which the I and I +files are stored. Defaults to C. + +Note that the I and I files are expected to be named +C and C respectively, if, for example, C<$storeid> is C<"foo">. + +=back + +=cut + sub new { my ($class, $scfg, $storeid, $secret_dir) = @_; @@ -63,6 +183,21 @@ my sub password_file_name { return "$self->{secret_dir}/$self->{storeid}.pw"; } +=pod + +=head3 set_password + + $client->set_password($password) + +Updates or creates the I file, storing the given C<$password>. + +If the I does not exist, it is created beforehand. + +If the I file does not exist, a new one with the permissions C<600> +is created. + +=cut + sub set_password { my ($self, $password) = @_; @@ -72,6 +207,19 @@ sub set_password { PVE::Tools::file_set_contents($pwfile, "$password\n", 0600); }; +=pod + +=head3 delete_password + + $client->delete_password() + +Deletes the I file inside the I. + +Will throw an exception if deleting the I file fails, but not +if the file doesn't exist. + +=cut + sub delete_password { my ($self) = @_; @@ -83,6 +231,16 @@ sub delete_password { } }; +=pod + +=head3 get_password + + $password = $client->get_password() + +Reads and returns the I from its file inside the I. + +=cut + sub get_password { my ($self) = @_; @@ -91,12 +249,38 @@ sub get_password { return PVE::Tools::file_read_firstline($pwfile); } +=pod + +=head3 encryption_key_file_name + + $file_name = $self->encryption_key_file_name() + +Returns the full name of the I file, including the path of the +I it is located in. + +=cut + sub encryption_key_file_name { my ($self) = @_; return "$self->{secret_dir}/$self->{storeid}.enc"; }; +=pod + +=head3 set_encryption_key + + $client->set_encryption_key($key) + +Updates or creates the I file, storing the given C<$key>. + +If the I does not exist, it is created beforehand. + +If the I file does not exist, a new one with the permissions C<600> +is created. + +=cut + sub set_encryption_key { my ($self, $key) = @_; @@ -106,6 +290,19 @@ sub set_encryption_key { PVE::Tools::file_set_contents($encfile, "$key\n", 0600); }; +=pod + +=head3 delete_encryption_key + + $client->delete_encryption_key() + +Deletes the I file inside the I. + +Will throw an exception if deleting the I file fails, but not +if the file doesn't exist. + +=cut + sub delete_encryption_key { my ($self) = @_; @@ -235,6 +432,21 @@ my sub run_client_cmd : prototype($$;$$$$) { return $res; } +=pod + +=head3 autogen_encryption_key + + $new_key = $client->autogen_encryption_key() + +Generates a new I and stores it as a file inside the I. +The raw contents of the key file, B, are +returned afterwards. + +If an I file already exists at its expected location, an +exception is thrown. + +=cut + sub autogen_encryption_key { my ($self) = @_; my $encfile = $self->encryption_key_file_name(); @@ -257,7 +469,61 @@ my sub split_namespaced_parameter : prototype($$) { return ($namespace, $snapshot); } -# lists all snapshots, optionally limited to a specific group +=pod + +=head3 get_snapshots + + $snapshots = $client->get_snapshots() + $snapshots = $client->get_snapshots($group) + +Returns all snapshots of the current client instance as a list of nesteded hashes. + +Optionally, the snapshots may be filtered by their C<$group>, such as C<"vm/100"> +or C<"ct/2000">, for example. + +The returned list has the following structure: + + [ + { + 'backup-id' => "100", + 'backup-time' => 1721901601, + 'backup-type' => "vm", + comment => "standalone node/example-host/100-example-vm", + files: [ + { + 'crypt-mode' => "encrypt", + filename => "qemu-server.conf.blob", + size => 428 + }, + { + 'crypt-mode' => "encrypt", + filename => "drive-scsi0.img.fidx", + size => 17179869184 + }, + { + 'crypt-mode' => "sign-only", + filename => "index.json.blob", + size => 651 + }, + { + filename => "client.log.blob" + }, + ... + ], + fingerprint => "...", + owner => "root@pam", + protected => false, + size => 17179870263, + verification => { + state => "ok", + upid => "..." + } + }, + ... + ] + +=cut + sub get_snapshots { my ($self, $group) = @_; @@ -274,8 +540,28 @@ sub get_snapshots { return run_client_cmd($self, "snapshots", $param, undef, undef, $namespace); }; -# create a new PXAR backup of a FS directory tree - doesn't cross FS boundary -# by default. +=pod + +=head3 backup_fs_tree + + $client->backup_fs_tree($root, $id, $pxarname) + $client->backup_fs_tree($root, $id, $pxarname, $cmd_opts) + +Create a new PXAR backup of a directory tree starting at C<$root>. + +C<$id> is the I of the stored backup and C<$pxarname> is the name of the +uploaded PXAR archive. + +Optionally, the C<$cmd_opts> hash may be supplied, which should contain +additional parameters to pass to C> that is used +under the hood. + +Raises an exception if either C<$root>, C<$id> or C<$pxarname> is C. + +B This does B cross filesystem boundaries. + +=cut + sub backup_fs_tree { my ($self, $root, $id, $pxarname, $cmd_opts) = @_; @@ -298,6 +584,32 @@ sub backup_fs_tree { return run_raw_client_cmd($self, 'backup', $param, %$cmd_opts); }; +=pod + +=head3 restore_pxar + + $client->restore_pxar($snapshot, $pxarname, $target) + $client->restore_pxar($snapshot, $pxarname, $target, $cmd_opts) + +Restore a PXAR backup of a directory tree from the given C<$snapshot>. + +C<$pxarname> is the name of the previously uploaded PXAR archive to restore and +C<$target> the directory to which the backed up tree will be restored to. + +Note that C<$snapshot> must be the snapshot's complete name in the format +C - for example C<"vm/100/2023-07-31T16:00:00Z"> or +C<"ct/2000/2024-08-01T09:54:08Z"> (like it's displayed in the PBS UI). + +Optionally, the C<$cmd_opts> hash may be supplied, which should contain +additional parameters to pass to C> that is used +under the hood. + +Raises an exception if either C<$snapshot>, C<$pxarname> or C<$target> is C, +or if a filesystem entry to be restored already exists inside the C<$target> +directory. + +=cut + sub restore_pxar { my ($self, $snapshot, $pxarname, $target, $cmd_opts) = @_; @@ -320,6 +632,22 @@ sub restore_pxar { return run_raw_client_cmd($self, 'restore', $param, %$cmd_opts); }; +=pod + +=head3 forget_snapshot + + $client->forget_snapshot($snapshot) + +Forgets the given C<$snapshot>. + +Note that C<$snapshot> must be the snapshot's complete name in the format +C - for example C<"vm/100/2023-07-31T16:00:00Z"> or +C<"ct/2000/2024-08-01T09:54:08Z"> (as displayed in the PBS UI). + +Raises an exception if C<$snapshot> is C. + +=cut + sub forget_snapshot { my ($self, $snapshot) = @_; @@ -330,6 +658,41 @@ sub forget_snapshot { return run_client_cmd($self, 'forget', [ "$snapshot" ], 1, undef, $namespace) }; +=pod + +=head3 prune_group + + $client->prune_group($opts, $prune_opts, $group) + +Prunes a backup C<$group>. The exact behaviour can be controlled using the +C<$opts> and C<$prune_opts> hashes. + +C<$group> must be in the format of C, like C<"vm/100"> or C<"ct/2000">, +for example (as displayed in the PBS UI). + +The C<$opts> hash supports the following options and may be left empty: + + { + 'dry-run' => 1, # perform a dry run + } + +The C<$prune_opts> hash supports the following options: + + { + 'keep-last' => 1, + 'keep-hourly' => 1, + 'keep-daily' => 1, + 'keep-weekly' => 1, + 'keep-monthly' => 1, + 'keep-yearly' => 1, + } + +Will do nothing if no C<$prune_opts> are supplied. + +Raises an exception if C<$group> is C. + +=cut + sub prune_group { my ($self, $opts, $prune_opts, $group) = @_; @@ -356,6 +719,19 @@ sub prune_group { return run_client_cmd($self, 'prune', $param, undef, undef, $namespace); }; +=pod + +=head3 status + + ($total, $free, $used, $active) = $client->status() + +Return the I of the client's repository as an array. + +The array contains the C<$total>, C<$free> and C<$used> size of the repository +as bytes, as well as whether the repository is C<$active> or not. + +=cut + sub status { my ($self) = @_; @@ -379,6 +755,59 @@ sub status { return ($total, $free, $used, $active); }; +=pod + +=head3 file_restore_list + + $restore_list = $client->($snapshot, $filepath) + $restore_list = $client->($snapshot, $filepath, $base64) + $restore_list = $client->($snapshot, $filepath, $base64, $extra_params) + +Return the list of entries from a directory C<$filepath> of a backup C<$snapshot>. + +Note that C<$snapshot> must be the snapshot's complete name in the format +C - for example C<"vm/100/2023-07-31T16:00:00Z"> or +C<"ct/2000/2024-08-01T09:54:08Z"> (as displayed in the PBS UI). + +C<$base64> may optionally be set to C<1> if the C<$filepath> is base64-encoded. + +The C<$extra_params> hash supports the following options and may be left empty: + + { + timeout => 5, # in seconds + } + +If successful, the returned list of hashes has the following structure: + + [ + { + filepath => "L2RyaXZlLXNjc2kwLmltZy5maWR4", + leaf => 0, + size => 34359738368, + text => "drive-scsi0.img.fidx", + type => "v" + }, + { + filepath => "L2RyaXZlLXNjc2kxLmltZy5maWR4", + leaf => 0, + size => 68719476736, + text => "drive-scsi1.img.fidx", + type => "v" + }, + ... + ] + +On error, the list of hashes will contain an error message, for example: + + [ + { + message => "wrong key - unable to verify signature since manifest's key [...] does not match provided key [...]" + }, + ... + ] + +=cut + sub file_restore_list { my ($self, $snapshot, $filepath, $base64, $extra_params) = @_; @@ -399,8 +828,32 @@ sub file_restore_list { ); } -# call sync from API, returns a fifo path for streaming data to clients, -# pass it to file_restore_extract to start transfering data +=pod + +=head3 file_restore_extract_prepare + + $fifo = $client->file_restore_extract_prepare() + +Create a I (FIFO) for streaming data and return its path. + +The returned path is usually passed to C> which will +stream the data to the I. A different process may then read from the +same path, receiving the streamed data. + +Raises an exception if: + +=over + +=item creating the I fails + +=item the call to C for the C user fails + +=item changing the permissions for the I or its directory fails + +=back + +=cut + sub file_restore_extract_prepare { my ($self) = @_; @@ -419,7 +872,50 @@ sub file_restore_extract_prepare { return "$tmpdir/fifo"; } -# this blocks while data is transfered, call this from a background worker +=pod + +=head3 file_restore_extract + + $client->file_restore_extract($output_file, $snapshot, $filepath, $base64, $tar) + +Restores and extracts a C<$filepath> from a C<$snapshot> to the given C<$output_file>. +By default, the C<$output_file> will be a ZIP archive. + +Because this method is mostly used for streaming purposes in conjunction with +C>, the C<$output_file> will be automatically +I once the extraction is complete. See below for an example on how to +use this method. + +C<$base64> may optionally be set to C<1> if the C<$filepath> is base64-encoded. + +C<$tar> may optionally be set to C<1> if the output written to C<$output_file> +should be a ZSTD-compressed TAR archive. B + +Usually used in conjunction with C>. + +B This method B while data is being transferred to C<$output_file>. +It is therefore best to call this within C> +or C>, for example: + + # [...] + my $fifo = $client->file_restore_extract_prepare(); + + $rpcenv->fork_worker('backup-download', undef, $user, sub { + print "Starting download of file: $filename\n"; + $client->file_restore_extract($fifo, $snapshot, $filepath, 0, $tar); + }); + + return { + download => { + path => $fifo, + stream => 1, + 'content-type' => 'application/octet-stream', + }, + }; + +=cut + sub file_restore_extract { my ($self, $output_file, $snapshot, $filepath, $base64, $tar) = @_; -- 2.39.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel