public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload
@ 2021-07-20 11:51 Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint Lorenz Stechauner
                   ` (7 more replies)
  0 siblings, 8 replies; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

this series overhauls the iso/vztmpl upload window completely:
* moved window to UploadToStorage.js
* initComponent() reduced to necessary - static definition instead
* added possibility to upload file to another name ("new-filename")
* added "Size" and "MIME type" (like Download from URL)
* added hash algorithm and checksum
* added a TaskViewer window after the upload (to see possible checksum errors and to mirror "Download from URL")
* removed file unlink from http-server -> endpoint has to unlink the files now
    AFAICT only this endpoint allowes file uploads, so this should not be a problem

note: ping for https://lists.proxmox.com/pipermail/pve-devel/2021-June/049049.html


pve-http-server:
Lorenz Stechauner (1):
  anyevent: move unlink from http-server to endpoint

 src/PVE/APIServer/AnyEvent.pm | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)


pve-storage:
Lorenz Stechauner (3):
  status: move unlink from http-server to endpoint
  status: add new-filename to upload
  status: add checksum and algorithm to file upload

 PVE/API2/Storage/Status.pm | 54 +++++++++++++++++++++++++++++++-------
 1 file changed, 44 insertions(+), 10 deletions(-)


pve-manager:
Lorenz Stechauner (4):
  ui: move upload window into UploadToStorage.js
  ui: refactor UploadToStorage.js
  ui/UploadToStorage.js: add checksum and algorithm
  ui/UploadToStorage.js: add TaskViewer

 www/manager6/Makefile                       |   1 +
 www/manager6/storage/ContentView.js         | 197 +-----------
 www/manager6/window/UploadToStorage.js      | 313 ++++++++++++++++++++
 4 files changed, 326 insertions(+), 207 deletions(-)
 create mode 100644 www/manager6/window/UploadToStorage.js

-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 13:23   ` Thomas Lamprecht
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 1/3] status: " Lorenz Stechauner
                   ` (6 subsequent siblings)
  7 siblings, 1 reply; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

any uploaded file has to be deleted by the corrosponding
endpoint. the file upload was only used by the 'upload to
storage' feature in pve.

this change allows the endpoint to delete the file itself,
making the old and racey`sleep 1` (waiting until the worker
has opened the file) obsolete.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 src/PVE/APIServer/AnyEvent.pm | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index cd77806..8d498c5 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -121,11 +121,7 @@ sub cleanup_reqstate {
     delete $reqstate->{proto};
     delete $reqstate->{accept_gzip};
     delete $reqstate->{starttime};
-
-    if ($reqstate->{tmpfilename}) {
-	unlink $reqstate->{tmpfilename};
-	delete $reqstate->{tmpfilename};
-    }
+    delete $reqstate->{tmpfilename};
 }
 
 sub client_do_disconnect {
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH storage 1/3] status: move unlink from http-server to endpoint
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 13:31   ` Thomas Lamprecht
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 2/3] status: add new-filename to upload Lorenz Stechauner
                   ` (5 subsequent siblings)
  7 siblings, 1 reply; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index 72fd851..b549d7d 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -478,6 +478,7 @@ __PACKAGE__->register_method ({
 	    print "command: " . join(' ', @$cmd) . "\n";
 
 	    eval { PVE::Tools::run_command($cmd, errmsg => 'import failed'); };
+	    unlink $tmpfilename;
 	    if (my $err = $@) {
 		unlink $dest;
 		die $err;
@@ -485,14 +486,7 @@ __PACKAGE__->register_method ({
 	    print "finished file import successfully\n";
 	};
 
-	my $upid = $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
-
-	# apache removes the temporary file on return, so we need
-	# to wait here to make sure the worker process starts and
-	# opens the file before it gets removed.
-	sleep(1);
-
-	return $upid;
+	return $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
    }});
 
 __PACKAGE__->register_method({
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH storage 2/3] status: add new-filename to upload
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 1/3] status: " Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 13:27   ` Thomas Lamprecht
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload Lorenz Stechauner
                   ` (4 subsequent siblings)
  7 siblings, 1 reply; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index b549d7d..eac5e13 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -378,9 +378,15 @@ __PACKAGE__->register_method ({
 	    content => {
 		description => "Content type.",
 		type => 'string', format => 'pve-storage-content',
+		enum => ['iso', 'vztmpl'],
 	    },
 	    filename => {
-		description => "The name of the file to create.",
+		description => "The original name of the file.",
+		type => 'string',
+	    },
+	    'new-filename' => {
+		description => "The name of the file to create. Caution: This will be normalized!",
+		maxLength => 255,
 		type => 'string',
 	    },
 	    tmpfilename => {
@@ -414,7 +420,7 @@ __PACKAGE__->register_method ({
 	my $size = -s $tmpfilename;
 	die "temporary file '$tmpfilename' does not exist\n" if !defined($size);
 
-	my $filename = PVE::Storage::normalize_content_filename($param->{filename});
+	my $filename = PVE::Storage::normalize_content_filename($param->{'new-filename'});
 
 	my $path;
 
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
                   ` (2 preceding siblings ...)
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 2/3] status: add new-filename to upload Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 13:40   ` Thomas Lamprecht
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 2/5] ui: move upload window into UploadToStorage.js Lorenz Stechauner
                   ` (3 subsequent siblings)
  7 siblings, 1 reply; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index eac5e13..e3bf758 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -389,6 +389,19 @@ __PACKAGE__->register_method ({
 		maxLength => 255,
 		type => 'string',
 	    },
+	    checksum => {
+		description => "The expected checksum of the file.",
+		type => 'string',
+		requires => 'checksum-algorithm',
+		optional => 1,
+	    },
+	    'checksum-algorithm' => {
+		description => "The algorithm to calculate the checksum of the file.",
+		type => 'string',
+		enum => ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'],
+		requires => 'checksum',
+		optional => 1,
+	    },
 	    tmpfilename => {
 		description => "The source file name. This parameter is usually set by the REST handler. You can only overwrite it when connecting to the trusted port on localhost.",
 		type => 'string',
@@ -478,6 +491,27 @@ __PACKAGE__->register_method ({
 	    my $upid = shift;
 
 	    print "starting file import from: $tmpfilename\n";
+
+	    eval {
+		my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
+		if ($checksum_algorithm) {
+		    print "calculating checksum...";
+
+		    my $checksum_got = PVE::Tools::get_file_hash($checksum_algorithm, $tmpfilename);
+
+		    if (lc($checksum_got) eq lc($checksum)) {
+			print "OK, checksum verified\n";
+		    } else {
+			print "\n";  # the front end expects the error to reside at the last line without any noise
+			die "checksum mismatch: got '$checksum_got' != expect '$checksum'\n";
+		    }
+		}
+	    };
+	    if (my $err = $@) {
+		unlink $tmpfilename;
+		die $err;
+	    }
+
 	    print "target node: $node\n";
 	    print "target file: $dest\n";
 	    print "file size is: $size\n";
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH manager 2/5] ui: move upload window into UploadToStorage.js
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
                   ` (3 preceding siblings ...)
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 3/5] ui: refactor UploadToStorage.js Lorenz Stechauner
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/Makefile                  |   1 +
 www/manager6/storage/ContentView.js    | 195 +------------------------
 www/manager6/window/UploadToStorage.js | 192 ++++++++++++++++++++++++
 3 files changed, 194 insertions(+), 194 deletions(-)
 create mode 100644 www/manager6/window/UploadToStorage.js

diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 75d355a5..552e842b 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -106,6 +106,7 @@ JSSRC= 							\
 	window/Snapshot.js				\
 	window/StartupEdit.js				\
 	window/DownloadUrlToStorage.js 			\
+	window/UploadToStorage.js 			\
 	window/Wizard.js				\
 	ha/Fencing.js					\
 	ha/GroupEdit.js					\
diff --git a/www/manager6/storage/ContentView.js b/www/manager6/storage/ContentView.js
index 3f5b686b..ca0ad664 100644
--- a/www/manager6/storage/ContentView.js
+++ b/www/manager6/storage/ContentView.js
@@ -1,196 +1,3 @@
-Ext.define('PVE.storage.Upload', {
-    extend: 'Ext.window.Window',
-    alias: 'widget.pveStorageUpload',
-
-    resizable: false,
-
-    modal: true,
-
-    initComponent: function() {
-        var me = this;
-
-	if (!me.nodename) {
-	    throw "no node name specified";
-	}
-	if (!me.storage) {
-	    throw "no storage ID specified";
-	}
-
-	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
-
-	let pbar = Ext.create('Ext.ProgressBar', {
-            text: 'Ready',
-	    hidden: true,
-	});
-
-	let acceptedExtensions = {
-	    iso: ".img, .iso",
-	    vztmpl: ".tar.gz, .tar.xz",
-	};
-
-	let defaultContent = me.contents[0] || '';
-
-	let fileField = Ext.create('Ext.form.field.File', {
-	    name: 'filename',
-	    buttonText: gettext('Select File...'),
-	    allowBlank: false,
-	    setAccept: function(content) {
-		let acceptString = acceptedExtensions[content] || '';
-		this.fileInputEl.set({
-		    accept: acceptString,
-		});
-	    },
-	    listeners: {
-		afterrender: function(cmp) {
-		    cmp.setAccept(defaultContent);
-		},
-	    },
-	});
-
-	me.formPanel = Ext.create('Ext.form.Panel', {
-	    method: 'POST',
-	    waitMsgTarget: true,
-	    bodyPadding: 10,
-	    border: false,
-	    width: 300,
-	    fieldDefaults: {
-		labelWidth: 100,
-		anchor: '100%',
-            },
-	    items: [
-		{
-		    xtype: 'pveContentTypeSelector',
-		    cts: me.contents,
-		    fieldLabel: gettext('Content'),
-		    name: 'content',
-		    value: defaultContent,
-		    allowBlank: false,
-		    listeners: {
-			change: function(cmp, newValue, oldValue) {
-			    fileField.setAccept(newValue);
-			},
-		    },
-		},
-		fileField,
-		pbar,
-	    ],
-	});
-
-	let form = me.formPanel.getForm();
-
-	let doStandardSubmit = function() {
-	    form.submit({
-		url: "/api2/htmljs" + baseurl,
-		waitMsg: gettext('Uploading file...'),
-		success: function(f, action) {
-		    me.close();
-		},
-		failure: function(f, action) {
-		    var msg = PVE.Utils.extractFormActionError(action);
-                    Ext.Msg.alert(gettext('Error'), msg);
-		},
-	    });
-	};
-
-	let updateProgress = function(per, bytes) {
-	    var text = (per * 100).toFixed(2) + '%';
-	    if (bytes) {
-		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
-	    }
-	    pbar.updateProgress(per, text);
-	};
-
-	let abortBtn = Ext.create('Ext.Button', {
-	    text: gettext('Abort'),
-	    disabled: true,
-	    handler: function() {
-		me.close();
-	    },
-	});
-
-	let submitBtn = Ext.create('Ext.Button', {
-	    text: gettext('Upload'),
-	    disabled: true,
-	    handler: function(button) {
-		var fd;
-		try {
-		    fd = new FormData();
-		} catch (err) {
-		    doStandardSubmit();
-		    return;
-		}
-
-		button.setDisabled(true);
-		abortBtn.setDisabled(false);
-
-		var field = form.findField('content');
-		fd.append("content", field.getValue());
-		field.setDisabled(true);
-
-		field = form.findField('filename');
-		var file = field.fileInputEl.dom;
-		fd.append("filename", file.files[0]);
-		field.setDisabled(true);
-
-		pbar.setVisible(true);
-		updateProgress(0);
-
-		let xhr = new XMLHttpRequest();
-		me.xhr = xhr;
-
-		xhr.addEventListener("load", function(e) {
-		    if (xhr.status === 200) {
-			me.close();
-			return;
-		    }
-		    let err = Ext.htmlEncode(xhr.statusText);
-		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
-		    if (xhr.responseText !== "") {
-			let result = Ext.decode(xhr.responseText);
-			result.message = msg;
-			msg = Proxmox.Utils.extractRequestError(result, true);
-		    }
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		}, false);
-
-		xhr.addEventListener("error", function(e) {
-		    let err = e.target.status.toString();
-		    let msg = `Error '${err}' occurred while receiving the document.`;
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		});
-
-		xhr.upload.addEventListener("progress", function(evt) {
-		    if (evt.lengthComputable) {
-			let percentComplete = evt.loaded / evt.total;
-			updateProgress(percentComplete, evt.loaded);
-		    }
-		}, false);
-
-		xhr.open("POST", `/api2/json${baseurl}`, true);
-		xhr.send(fd);
-	    },
-	});
-
-	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
-
-	Ext.apply(me, {
-	    title: gettext('Upload'),
-	    items: me.formPanel,
-	    buttons: [abortBtn, submitBtn],
-	    listeners: {
-		close: function() {
-		    if (me.xhr) {
-			me.xhr.abort();
-			delete me.xhr;
-		    }
-		},
-	    },
-	});
-
-        me.callParent();
-    },
-});
-
 Ext.define('PVE.storage.ContentView', {
     extend: 'Ext.grid.GridPanel',
 
@@ -259,7 +66,7 @@ Ext.define('PVE.storage.ContentView', {
 		    text: gettext('Upload'),
 		    disabled: !me.enableUploadButton,
 		    handler: function() {
-			Ext.create('PVE.storage.Upload', {
+			Ext.create('PVE.window.UploadToStorage', {
 			    nodename: nodename,
 			    storage: storage,
 			    contents: [content],
diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
new file mode 100644
index 00000000..3c35020a
--- /dev/null
+++ b/www/manager6/window/UploadToStorage.js
@@ -0,0 +1,192 @@
+Ext.define('PVE.window.UploadToStorage', {
+    extend: 'Ext.window.Window',
+    alias: 'widget.pveStorageUpload',
+
+    resizable: false,
+
+    modal: true,
+
+    initComponent: function() {
+        var me = this;
+
+	if (!me.nodename) {
+	    throw "no node name specified";
+	}
+	if (!me.storage) {
+	    throw "no storage ID specified";
+	}
+
+	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
+
+	let pbar = Ext.create('Ext.ProgressBar', {
+            text: 'Ready',
+	    hidden: true,
+	});
+
+	let acceptedExtensions = {
+	    iso: ".img, .iso",
+	    vztmpl: ".tar.gz, .tar.xz",
+	};
+
+	let defaultContent = me.contents[0] || '';
+
+	let fileField = Ext.create('Ext.form.field.File', {
+	    name: 'filename',
+	    buttonText: gettext('Select File...'),
+	    allowBlank: false,
+	    setAccept: function(content) {
+		let acceptString = acceptedExtensions[content] || '';
+		this.fileInputEl.set({
+		    accept: acceptString,
+		});
+	    },
+	    listeners: {
+		afterrender: function(cmp) {
+		    cmp.setAccept(defaultContent);
+		},
+	    },
+	});
+
+	me.formPanel = Ext.create('Ext.form.Panel', {
+	    method: 'POST',
+	    waitMsgTarget: true,
+	    bodyPadding: 10,
+	    border: false,
+	    width: 300,
+	    fieldDefaults: {
+		labelWidth: 100,
+		anchor: '100%',
+            },
+	    items: [
+		{
+		    xtype: 'pveContentTypeSelector',
+		    cts: me.contents,
+		    fieldLabel: gettext('Content'),
+		    name: 'content',
+		    value: defaultContent,
+		    allowBlank: false,
+		    listeners: {
+			change: function(cmp, newValue, oldValue) {
+			    fileField.setAccept(newValue);
+			},
+		    },
+		},
+		fileField,
+		pbar,
+	    ],
+	});
+
+	let form = me.formPanel.getForm();
+
+	let doStandardSubmit = function() {
+	    form.submit({
+		url: "/api2/htmljs" + baseurl,
+		waitMsg: gettext('Uploading file...'),
+		success: function(f, action) {
+		    me.close();
+		},
+		failure: function(f, action) {
+		    var msg = PVE.Utils.extractFormActionError(action);
+                    Ext.Msg.alert(gettext('Error'), msg);
+		},
+	    });
+	};
+
+	let updateProgress = function(per, bytes) {
+	    var text = (per * 100).toFixed(2) + '%';
+	    if (bytes) {
+		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
+	    }
+	    pbar.updateProgress(per, text);
+	};
+
+	let abortBtn = Ext.create('Ext.Button', {
+	    text: gettext('Abort'),
+	    disabled: true,
+	    handler: function() {
+		me.close();
+	    },
+	});
+
+	let submitBtn = Ext.create('Ext.Button', {
+	    text: gettext('Upload'),
+	    disabled: true,
+	    handler: function(button) {
+		var fd;
+		try {
+		    fd = new FormData();
+		} catch (err) {
+		    doStandardSubmit();
+		    return;
+		}
+
+		button.setDisabled(true);
+		abortBtn.setDisabled(false);
+
+		var field = form.findField('content');
+		fd.append("content", field.getValue());
+		field.setDisabled(true);
+
+		field = form.findField('filename');
+		var file = field.fileInputEl.dom;
+		fd.append("filename", file.files[0]);
+		field.setDisabled(true);
+
+		pbar.setVisible(true);
+		updateProgress(0);
+
+		let xhr = new XMLHttpRequest();
+		me.xhr = xhr;
+
+		xhr.addEventListener("load", function(e) {
+		    if (xhr.status === 200) {
+			me.close();
+			return;
+		    }
+		    let err = Ext.htmlEncode(xhr.statusText);
+		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
+		    if (xhr.responseText !== "") {
+			let result = Ext.decode(xhr.responseText);
+			result.message = msg;
+			msg = Proxmox.Utils.extractRequestError(result, true);
+		    }
+		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
+		}, false);
+
+		xhr.addEventListener("error", function(e) {
+		    let err = e.target.status.toString();
+		    let msg = `Error '${err}' occurred while receiving the document.`;
+		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
+		});
+
+		xhr.upload.addEventListener("progress", function(evt) {
+		    if (evt.lengthComputable) {
+			let percentComplete = evt.loaded / evt.total;
+			updateProgress(percentComplete, evt.loaded);
+		    }
+		}, false);
+
+		xhr.open("POST", `/api2/json${baseurl}`, true);
+		xhr.send(fd);
+	    },
+	});
+
+	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
+
+	Ext.apply(me, {
+	    title: gettext('Upload'),
+	    items: me.formPanel,
+	    buttons: [abortBtn, submitBtn],
+	    listeners: {
+		close: function() {
+		    if (me.xhr) {
+			me.xhr.abort();
+			delete me.xhr;
+		    }
+		},
+	    },
+	});
+
+        me.callParent();
+    },
+});
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH manager 3/5] ui: refactor UploadToStorage.js
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
                   ` (4 preceding siblings ...)
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 2/5] ui: move upload window into UploadToStorage.js Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 4/5] ui/UploadToStorage.js: add checksum and algorithm Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 5/5] ui/UploadToStorage.js: add TaskViewer Lorenz Stechauner
  7 siblings, 0 replies; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

this also removes the "content" selector from the window.
as far as it seems, this selector was never able to select
more than one entry, so it was useless.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/storage/ContentView.js    |   2 +-
 www/manager6/window/UploadToStorage.js | 359 +++++++++++++++----------
 2 files changed, 213 insertions(+), 148 deletions(-)

diff --git a/www/manager6/storage/ContentView.js b/www/manager6/storage/ContentView.js
index ca0ad664..00a94f3c 100644
--- a/www/manager6/storage/ContentView.js
+++ b/www/manager6/storage/ContentView.js
@@ -69,7 +69,7 @@ Ext.define('PVE.storage.ContentView', {
 			Ext.create('PVE.window.UploadToStorage', {
 			    nodename: nodename,
 			    storage: storage,
-			    contents: [content],
+			    content: content,
 			    autoShow: true,
 			    taskDone: () => reload(),
 			});
diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index 3c35020a..75423b0f 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -1,191 +1,256 @@
 Ext.define('PVE.window.UploadToStorage', {
     extend: 'Ext.window.Window',
     alias: 'widget.pveStorageUpload',
+    mixins: ['Proxmox.Mixin.CBind'],
 
     resizable: false,
-
     modal: true,
 
-    initComponent: function() {
-        var me = this;
+    title: gettext('Upload'),
 
-	if (!me.nodename) {
-	    throw "no node name specified";
-	}
-	if (!me.storage) {
-	    throw "no storage ID specified";
-	}
+    acceptedExtensions: {
+	iso: ['.img', '.iso'],
+	vztmpl: ['.tar.gz', '.tar.xz'],
+    },
 
-	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
+    cbindData: function(initialConfig) {
+	const me = this;
+	return {
+	    nodename: me.nodename,
+	    storage: me.storage,
+	    content: me.content,
+	    extensions: me.acceptedExtensions[me.content].join(', '),
+	};
+    },
 
-	let pbar = Ext.create('Ext.ProgressBar', {
-            text: 'Ready',
-	    hidden: true,
-	});
+    cbind: {
+	url: `/nodes/{nodename}/storage/{storage}/upload`,
+    },
 
-	let acceptedExtensions = {
-	    iso: ".img, .iso",
-	    vztmpl: ".tar.gz, .tar.xz",
-	};
+    viewModel: {
+	data: {
+	    size: '-',
+	    mimetype: '-',
+	    filename: '',
+	},
+    },
 
-	let defaultContent = me.contents[0] || '';
+    controller: {
+	submit: function(button) {
+	    const view = this.getView();
+	    const form = Ext.getCmp('formPanel').getForm();
+	    const abortBtn = Ext.getCmp('abortBtn');
+	    const pbar = Ext.getCmp('progressBar');
 
-	let fileField = Ext.create('Ext.form.field.File', {
-	    name: 'filename',
-	    buttonText: gettext('Select File...'),
-	    allowBlank: false,
-	    setAccept: function(content) {
-		let acceptString = acceptedExtensions[content] || '';
-		this.fileInputEl.set({
-		    accept: acceptString,
+	    const doStandardSubmit = function() {
+		form.submit({
+		    url: "/api2/htmljs" + view.url,
+		    waitMsg: gettext('Uploading file...'),
+		    success: function(f, action) {
+			view.close();
+		    },
+		    failure: function(f, action) {
+			const msg = PVE.Utils.extractFormActionError(action);
+			Ext.Msg.alert(gettext('Error'), msg);
+		    },
 		});
-	    },
-	    listeners: {
-		afterrender: function(cmp) {
-		    cmp.setAccept(defaultContent);
-		},
-	    },
-	});
+	    };
+
+	    const updateProgress = function(per, bytes) {
+		let text = (per * 100).toFixed(2) + '%';
+		if (bytes) {
+		    text += " (" + Proxmox.Utils.format_size(bytes) + ')';
+		}
+		pbar.updateProgress(per, text);
+	    };
+
+	    let fd;
+	    try {
+		fd = new FormData();
+	    } catch (err) {
+		doStandardSubmit();
+		return;
+	    }
+
+	    button.setDisabled(true);
+	    abortBtn.setDisabled(false);
+
+	    let field = form.findField('content');
+	    fd.append("content", field.getValue());
+	    field.setDisabled(true);
+
+	    field = form.findField('new-filename');
+	    fd.append("new-filename", field.getValue());
+	    field.setDisabled(true);
+
+	    field = form.findField('filename');
+	    const file = field.fileInputEl.dom;
+	    fd.append("filename", file.files[0]);
+	    field.setDisabled(true);
+
+	    pbar.setVisible(true);
+	    updateProgress(0);
+
+	    const xhr = new XMLHttpRequest();
+	    view.xhr = xhr;
+
+	    xhr.addEventListener("load", function(e) {
+		if (xhr.status === 200) {
+		    view.close();
+		    return;
+		}
+		const err = Ext.htmlEncode(xhr.statusText);
+		let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
+		if (xhr.responseText !== "") {
+		    const result = Ext.decode(xhr.responseText);
+		    result.message = msg;
+		    msg = Proxmox.Utils.extractRequestError(result, true);
+		}
+		Ext.Msg.alert(gettext('Error'), msg, btn => view.close());
+	    }, false);
+
+	    xhr.addEventListener("error", function(e) {
+		const err = e.target.status.toString();
+		const msg = `Error '${err}' occurred while receiving the document.`;
+		Ext.Msg.alert(gettext('Error'), msg, btn => view.close());
+	    });
+
+	    xhr.upload.addEventListener("progress", function(evt) {
+		if (evt.lengthComputable) {
+		    const percentComplete = evt.loaded / evt.total;
+		    updateProgress(percentComplete, evt.loaded);
+		}
+	    }, false);
+
+	    xhr.open("POST", `/api2/json${view.url}`, true);
+	    xhr.send(fd);
+	},
+
+	validitychange: function(f, valid) {
+	    const submitBtn = Ext.getCmp('submitBtn');
+	    submitBtn.setDisabled(!valid);
+	},
+
+	fileChange: function(input) {
+	    const vm = this.getViewModel();
+	    const name = input.value.replace(/^.*(\/|\\)/, '');
+	    const fileInput = input.fileInputEl.dom;
+	    vm.set('filename', name);
+	    vm.set('size', (fileInput.files[0] && Proxmox.Utils.format_size(fileInput.files[0].size)) || '-');
+	    vm.set('mimetype', (fileInput.files[0] && fileInput.files[0].type) || '-');
+	},
+    },
 
-	me.formPanel = Ext.create('Ext.form.Panel', {
+    items: [
+	{
+	    xtype: 'form',
+	    id: 'formPanel',
 	    method: 'POST',
 	    waitMsgTarget: true,
 	    bodyPadding: 10,
 	    border: false,
-	    width: 300,
+	    width: 400,
 	    fieldDefaults: {
 		labelWidth: 100,
 		anchor: '100%',
             },
 	    items: [
 		{
-		    xtype: 'pveContentTypeSelector',
-		    cts: me.contents,
-		    fieldLabel: gettext('Content'),
-		    name: 'content',
-		    value: defaultContent,
+		    xtype: 'filefield',
+		    name: 'filename',
+		    buttonText: gettext('Select File'),
 		    allowBlank: false,
+		    fieldLabel: gettext('File'),
+		    cbind: {
+			accept: '{extensions}',
+		    },
 		    listeners: {
-			change: function(cmp, newValue, oldValue) {
-			    fileField.setAccept(newValue);
-			},
+			change: 'fileChange',
 		    },
 		},
-		fileField,
-		pbar,
-	    ],
-	});
-
-	let form = me.formPanel.getForm();
-
-	let doStandardSubmit = function() {
-	    form.submit({
-		url: "/api2/htmljs" + baseurl,
-		waitMsg: gettext('Uploading file...'),
-		success: function(f, action) {
-		    me.close();
+		{
+		    xtype: 'textfield',
+		    name: 'new-filename',
+		    allowBlank: false,
+		    fieldLabel: gettext('File name'),
+		    bind: {
+			value: '{filename}',
+		    },
 		},
-		failure: function(f, action) {
-		    var msg = PVE.Utils.extractFormActionError(action);
-                    Ext.Msg.alert(gettext('Error'), msg);
+		{
+		    xtype: 'displayfield',
+		    name: 'size',
+		    fieldLabel: gettext('File size'),
+		    bind: {
+			value: '{size}',
+		    },
 		},
-	    });
-	};
-
-	let updateProgress = function(per, bytes) {
-	    var text = (per * 100).toFixed(2) + '%';
-	    if (bytes) {
-		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
-	    }
-	    pbar.updateProgress(per, text);
-	};
+		{
+		    xtype: 'displayfield',
+		    name: 'mimetype',
+		    fieldLabel: gettext('MIME type'),
+		    bind: {
+			value: '{mimetype}',
+		    },
+		},
+		{
+		    xtype: 'progressbar',
+		    text: 'Ready',
+		    hidden: true,
+		    id: 'progressBar',
+		},
+		{
+		    xtype: 'hiddenfield',
+		    name: 'content',
+		    cbind: {
+			value: '{content}',
+		    },
+		},
+	    ],
+	   listeners: {
+		validitychange: 'validitychange',
+	   },
+	},
+    ],
 
-	let abortBtn = Ext.create('Ext.Button', {
+    buttons: [
+	{
+	    xtype: 'button',
 	    text: gettext('Abort'),
+	    id: 'abortBtn',
 	    disabled: true,
 	    handler: function() {
-		me.close();
+		const me = this;
+		me.up('pveStorageUpload').close();
 	    },
-	});
-
-	let submitBtn = Ext.create('Ext.Button', {
+	},
+	{
 	    text: gettext('Upload'),
+	    id: 'submitBtn',
 	    disabled: true,
-	    handler: function(button) {
-		var fd;
-		try {
-		    fd = new FormData();
-		} catch (err) {
-		    doStandardSubmit();
-		    return;
-		}
+	    handler: 'submit',
+	},
+    ],
 
-		button.setDisabled(true);
-		abortBtn.setDisabled(false);
-
-		var field = form.findField('content');
-		fd.append("content", field.getValue());
-		field.setDisabled(true);
-
-		field = form.findField('filename');
-		var file = field.fileInputEl.dom;
-		fd.append("filename", file.files[0]);
-		field.setDisabled(true);
-
-		pbar.setVisible(true);
-		updateProgress(0);
-
-		let xhr = new XMLHttpRequest();
-		me.xhr = xhr;
-
-		xhr.addEventListener("load", function(e) {
-		    if (xhr.status === 200) {
-			me.close();
-			return;
-		    }
-		    let err = Ext.htmlEncode(xhr.statusText);
-		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
-		    if (xhr.responseText !== "") {
-			let result = Ext.decode(xhr.responseText);
-			result.message = msg;
-			msg = Proxmox.Utils.extractRequestError(result, true);
-		    }
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		}, false);
-
-		xhr.addEventListener("error", function(e) {
-		    let err = e.target.status.toString();
-		    let msg = `Error '${err}' occurred while receiving the document.`;
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		});
+    listeners: {
+	close: function() {
+	    const me = this;
+	    if (me.xhr) {
+		me.xhr.abort();
+		delete me.xhr;
+	    }
+	},
+    },
 
-		xhr.upload.addEventListener("progress", function(evt) {
-		    if (evt.lengthComputable) {
-			let percentComplete = evt.loaded / evt.total;
-			updateProgress(percentComplete, evt.loaded);
-		    }
-		}, false);
+    initComponent: function() {
+        const me = this;
 
-		xhr.open("POST", `/api2/json${baseurl}`, true);
-		xhr.send(fd);
-	    },
-	});
-
-	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
-
-	Ext.apply(me, {
-	    title: gettext('Upload'),
-	    items: me.formPanel,
-	    buttons: [abortBtn, submitBtn],
-	    listeners: {
-		close: function() {
-		    if (me.xhr) {
-			me.xhr.abort();
-			delete me.xhr;
-		    }
-		},
-	    },
-	});
+	if (!me.nodename) {
+	    throw "no node name specified";
+	}
+	if (!me.storage) {
+	    throw "no storage ID specified";
+	}
 
         me.callParent();
     },
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH manager 4/5] ui/UploadToStorage.js: add checksum and algorithm
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
                   ` (5 preceding siblings ...)
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 3/5] ui: refactor UploadToStorage.js Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 5/5] ui/UploadToStorage.js: add TaskViewer Lorenz Stechauner
  7 siblings, 0 replies; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/window/UploadToStorage.js | 42 ++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index 75423b0f..ca89692e 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -83,6 +83,16 @@ Ext.define('PVE.window.UploadToStorage', {
 	    fd.append("new-filename", field.getValue());
 	    field.setDisabled(true);
 
+	    field = form.findField('checksum-algorithm');
+	    field.setDisabled(true);
+	    if (field.getValue() !== '__default__') {
+		fd.append("checksum-algorithm", field.getValue());
+
+		field = form.findField('checksum');
+		fd.append("checksum", field.getValue());
+		field.setDisabled(true);
+	    }
+
 	    field = form.findField('filename');
 	    const file = field.fileInputEl.dom;
 	    fd.append("filename", file.files[0]);
@@ -139,6 +149,18 @@ Ext.define('PVE.window.UploadToStorage', {
 	    vm.set('size', (fileInput.files[0] && Proxmox.Utils.format_size(fileInput.files[0].size)) || '-');
 	    vm.set('mimetype', (fileInput.files[0] && fileInput.files[0].type) || '-');
 	},
+
+	hashChange: function(field) {
+	    const checksum = Ext.getCmp('downloadUrlChecksum');
+	    if (field.getValue() === '__default__') {
+		checksum.setDisabled(true);
+		checksum.setValue("");
+		checksum.allowBlank = true;
+	    } else {
+		checksum.setDisabled(false);
+		checksum.allowBlank = false;
+	    }
+	},
     },
 
     items: [
@@ -193,6 +215,26 @@ Ext.define('PVE.window.UploadToStorage', {
 			value: '{mimetype}',
 		    },
 		},
+		{
+		    xtype: 'pveHashAlgorithmSelector',
+		    name: 'checksum-algorithm',
+		    fieldLabel: gettext('Hash algorithm'),
+		    allowBlank: true,
+		    hasNoneOption: true,
+		    value: '__default__',
+		    listeners: {
+			change: 'hashChange',
+		    },
+		},
+		{
+		    xtype: 'textfield',
+		    name: 'checksum',
+		    fieldLabel: gettext('Checksum'),
+		    allowBlank: true,
+		    disabled: true,
+		    emptyText: gettext('none'),
+		    id: 'downloadUrlChecksum',
+		},
 		{
 		    xtype: 'progressbar',
 		    text: 'Ready',
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* [pve-devel] [PATCH manager 5/5] ui/UploadToStorage.js: add TaskViewer
  2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
                   ` (6 preceding siblings ...)
  2021-07-20 11:51 ` [pve-devel] [PATCH manager 4/5] ui/UploadToStorage.js: add checksum and algorithm Lorenz Stechauner
@ 2021-07-20 11:51 ` Lorenz Stechauner
  7 siblings, 0 replies; 14+ messages in thread
From: Lorenz Stechauner @ 2021-07-20 11:51 UTC (permalink / raw)
  To: pve-devel

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/window/UploadToStorage.js | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index ca89692e..6a01821f 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -106,7 +106,21 @@ Ext.define('PVE.window.UploadToStorage', {
 
 	    xhr.addEventListener("load", function(e) {
 		if (xhr.status === 200) {
-		    view.close();
+		    view.hide();
+
+		    const result = JSON.parse(xhr.response);
+		    const upid = result.data;
+		    Ext.create('Proxmox.window.TaskViewer', {
+			autoShow: true,
+			upid: upid,
+			taskDone: view.taskDone,
+			listeners: {
+			    destroy: function() {
+				view.close();
+			    },
+			},
+		    });
+
 		    return;
 		}
 		const err = Ext.htmlEncode(xhr.statusText);
-- 
2.30.2





^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint
  2021-07-20 11:51 ` [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint Lorenz Stechauner
@ 2021-07-20 13:23   ` Thomas Lamprecht
  0 siblings, 0 replies; 14+ messages in thread
From: Thomas Lamprecht @ 2021-07-20 13:23 UTC (permalink / raw)
  To: Proxmox VE development discussion, Lorenz Stechauner

On 20.07.21 13:51, Lorenz Stechauner wrote:
> any uploaded file has to be deleted by the corrosponding
> endpoint. the file upload was only used by the 'upload to
> storage' feature in pve.
> 
> this change allows the endpoint to delete the file itself,
> making the old and racey`sleep 1` (waiting until the worker
> has opened the file) obsolete.
> 

that was mostly from PVE 1.x times which used indeed an apache in front of the
API, bur if this is a breaking change it needs to be recorded as such with a
d/control `Breaks: libpve-storage-perl (<< x.y)`, else one can install a version
where PVE does not cleanup any uploaded temporary file.

As x.y is not yet known at time of sending this (there could be intermediate
bumps between this getting applied) it makes sense to not include that directly,
but it'd be definitively good to mention that the one applying this needs to do
record the Breaks.

> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  src/PVE/APIServer/AnyEvent.pm | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
> index cd77806..8d498c5 100644
> --- a/src/PVE/APIServer/AnyEvent.pm
> +++ b/src/PVE/APIServer/AnyEvent.pm
> @@ -121,11 +121,7 @@ sub cleanup_reqstate {
>      delete $reqstate->{proto};
>      delete $reqstate->{accept_gzip};
>      delete $reqstate->{starttime};
> -
> -    if ($reqstate->{tmpfilename}) {
> -	unlink $reqstate->{tmpfilename};
> -	delete $reqstate->{tmpfilename};
> -    }
> +    delete $reqstate->{tmpfilename};
>  }
>  
>  sub client_do_disconnect {
> 





^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [pve-devel] [PATCH storage 2/3] status: add new-filename to upload
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 2/3] status: add new-filename to upload Lorenz Stechauner
@ 2021-07-20 13:27   ` Thomas Lamprecht
  2021-07-20 19:15     ` Thomas Lamprecht
  0 siblings, 1 reply; 14+ messages in thread
From: Thomas Lamprecht @ 2021-07-20 13:27 UTC (permalink / raw)
  To: Proxmox VE development discussion, Lorenz Stechauner

On 20.07.21 13:51, Lorenz Stechauner wrote:
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  PVE/API2/Storage/Status.pm | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index b549d7d..eac5e13 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -378,9 +378,15 @@ __PACKAGE__->register_method ({
>  	    content => {
>  		description => "Content type.",
>  		type => 'string', format => 'pve-storage-content',
> +		enum => ['iso', 'vztmpl'],

unrelated change? That could be send as its own patch, does not even needs to be a
part of this series.

>  	    },
>  	    filename => {
> -		description => "The name of the file to create.",
> +		description => "The original name of the file.",
> +		type => 'string',
> +	    },
> +	    'new-filename' => {
> +		description => "The name of the file to create. Caution: This will be normalized!",
> +		maxLength => 255,

new non optional API parameter would be an ABI break, 7.0 is released, that won't
fly anymore for ~1.9 years ;-)

Rather, make it optional and fallback to the filename (or whatever makes it
actually backward compatible).

>  		type => 'string',
>  	    },
>  	    tmpfilename => {
> @@ -414,7 +420,7 @@ __PACKAGE__->register_method ({
>  	my $size = -s $tmpfilename;
>  	die "temporary file '$tmpfilename' does not exist\n" if !defined($size);
>  
> -	my $filename = PVE::Storage::normalize_content_filename($param->{filename});
> +	my $filename = PVE::Storage::normalize_content_filename($param->{'new-filename'});
>  
>  	my $path;
>  
> 





^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [pve-devel] [PATCH storage 1/3] status: move unlink from http-server to endpoint
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 1/3] status: " Lorenz Stechauner
@ 2021-07-20 13:31   ` Thomas Lamprecht
  0 siblings, 0 replies; 14+ messages in thread
From: Thomas Lamprecht @ 2021-07-20 13:31 UTC (permalink / raw)
  To: Proxmox VE development discussion, Lorenz Stechauner

On 20.07.21 13:51, Lorenz Stechauner wrote:

for some odd stuff like here I'd like to see a bit more in the commit message,
when did this come in, since when can it get done in another way (and why wasn't
that way then chosen then already?). Stuff is there for a reason, dropping
something fishy like this without any (commit) comment whatsoever is not too ideal.

Besides that, would need a versioned bump on pve-http-server's package, but as we
do not depend on that directly here, IIRC, we would break old manager and bump
the versioned dependency for http-server in there (manager).

Again, not really doable in a "for sure way" on sending, but should be noted ;-)

> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  PVE/API2/Storage/Status.pm | 10 ++--------
>  1 file changed, 2 insertions(+), 8 deletions(-)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index 72fd851..b549d7d 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -478,6 +478,7 @@ __PACKAGE__->register_method ({
>  	    print "command: " . join(' ', @$cmd) . "\n";
>  
>  	    eval { PVE::Tools::run_command($cmd, errmsg => 'import failed'); };
> +	    unlink $tmpfilename;
>  	    if (my $err = $@) {
>  		unlink $dest;
>  		die $err;
> @@ -485,14 +486,7 @@ __PACKAGE__->register_method ({
>  	    print "finished file import successfully\n";
>  	};
>  
> -	my $upid = $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
> -
> -	# apache removes the temporary file on return, so we need
> -	# to wait here to make sure the worker process starts and
> -	# opens the file before it gets removed.
> -	sleep(1);
> -
> -	return $upid;
> +	return $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
>     }});
>  
>  __PACKAGE__->register_method({
> 





^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload
  2021-07-20 11:51 ` [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload Lorenz Stechauner
@ 2021-07-20 13:40   ` Thomas Lamprecht
  0 siblings, 0 replies; 14+ messages in thread
From: Thomas Lamprecht @ 2021-07-20 13:40 UTC (permalink / raw)
  To: Proxmox VE development discussion, Lorenz Stechauner

On 20.07.21 13:51, Lorenz Stechauner wrote:
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  PVE/API2/Storage/Status.pm | 34 ++++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index eac5e13..e3bf758 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -389,6 +389,19 @@ __PACKAGE__->register_method ({
>  		maxLength => 255,
>  		type => 'string',
>  	    },
> +	    checksum => {
> +		description => "The expected checksum of the file.",
> +		type => 'string',
> +		requires => 'checksum-algorithm',
> +		optional => 1,
> +	    },
> +	    'checksum-algorithm' => {
> +		description => "The algorithm to calculate the checksum of the file.",
> +		type => 'string',
> +		enum => ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'],
> +		requires => 'checksum',
> +		optional => 1,
> +	    },
>  	    tmpfilename => {
>  		description => "The source file name. This parameter is usually set by the REST handler. You can only overwrite it when connecting to the trusted port on localhost.",
>  		type => 'string',
> @@ -478,6 +491,27 @@ __PACKAGE__->register_method ({
>  	    my $upid = shift;
>  
>  	    print "starting file import from: $tmpfilename\n";
> +
> +	    eval {
> +		my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
> +		if ($checksum_algorithm) {
> +		    print "calculating checksum...";
> +
> +		    my $checksum_got = PVE::Tools::get_file_hash($checksum_algorithm, $tmpfilename);
> +
> +		    if (lc($checksum_got) eq lc($checksum)) {
> +			print "OK, checksum verified\n";
> +		    } else {
> +			print "\n";  # the front end expects the error to reside at the last line without any noise
> +			die "checksum mismatch: got '$checksum_got' != expect '$checksum'\n";
> +		    }
> +		}
> +	    };
> +	    if (my $err = $@) {
> +		unlink $tmpfilename;

missing error handling: `unlink $tmpfilename or warn "error on clean up of temporary file '$tmpfilename' - $!";

> +		die $err;
> +	    }
> +
>  	    print "target node: $node\n";
>  	    print "target file: $dest\n";
>  	    print "file size is: $size\n";
> 





^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [pve-devel] [PATCH storage 2/3] status: add new-filename to upload
  2021-07-20 13:27   ` Thomas Lamprecht
@ 2021-07-20 19:15     ` Thomas Lamprecht
  0 siblings, 0 replies; 14+ messages in thread
From: Thomas Lamprecht @ 2021-07-20 19:15 UTC (permalink / raw)
  To: Proxmox VE development discussion, Lorenz Stechauner

On 20.07.21 15:27, Thomas Lamprecht wrote:
> On 20.07.21 13:51, Lorenz Stechauner wrote:
>> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
>> ---
>>  PVE/API2/Storage/Status.pm | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
>> index b549d7d..eac5e13 100644
>> --- a/PVE/API2/Storage/Status.pm
>> +++ b/PVE/API2/Storage/Status.pm
>> @@ -378,9 +378,15 @@ __PACKAGE__->register_method ({
>>  	    content => {
>>  		description => "Content type.",
>>  		type => 'string', format => 'pve-storage-content',
>> +		enum => ['iso', 'vztmpl'],
> 
> unrelated change? That could be send as its own patch, does not even needs to be a
> part of this series.
> 
>>  	    },
>>  	    filename => {
>> -		description => "The name of the file to create.",
>> +		description => "The original name of the file.",
>> +		type => 'string',
>> +	    },
>> +	    'new-filename' => {
>> +		description => "The name of the file to create. Caution: This will be normalized!",
>> +		maxLength => 255,
> 
> new non optional API parameter would be an ABI break, 7.0 is released, that won't
> fly anymore for ~1.9 years ;-)
> 
> Rather, make it optional and fallback to the filename (or whatever makes it
> actually backward compatible).

Intially I just saw the new non-optional param and ABI break was enough to comment
that, but after I took another glance here the new param seems rather bogus in general?

So, what's it actual use? Why is the existing `filename` one not enough?
It'd be good if such things would be stated in the commit message already..

> 
>>  		type => 'string',
>>  	    },
>>  	    tmpfilename => {
>> @@ -414,7 +420,7 @@ __PACKAGE__->register_method ({
>>  	my $size = -s $tmpfilename;
>>  	die "temporary file '$tmpfilename' does not exist\n" if !defined($size);
>>  
>> -	my $filename = PVE::Storage::normalize_content_filename($param->{filename});
>> +	my $filename = PVE::Storage::normalize_content_filename($param->{'new-filename'});
>>  
>>  	my $path;
>>  
>>
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 





^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2021-07-20 19:15 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-20 11:51 [pve-devel] [PATCH-SERIES http-server/storage/manager] add checksum and algorithm to iso upload Lorenz Stechauner
2021-07-20 11:51 ` [pve-devel] [PATCH http-server] anyevent: move unlink from http-server to endpoint Lorenz Stechauner
2021-07-20 13:23   ` Thomas Lamprecht
2021-07-20 11:51 ` [pve-devel] [PATCH storage 1/3] status: " Lorenz Stechauner
2021-07-20 13:31   ` Thomas Lamprecht
2021-07-20 11:51 ` [pve-devel] [PATCH storage 2/3] status: add new-filename to upload Lorenz Stechauner
2021-07-20 13:27   ` Thomas Lamprecht
2021-07-20 19:15     ` Thomas Lamprecht
2021-07-20 11:51 ` [pve-devel] [PATCH storage 3/3] status: add checksum and algorithm to file upload Lorenz Stechauner
2021-07-20 13:40   ` Thomas Lamprecht
2021-07-20 11:51 ` [pve-devel] [PATCH manager 2/5] ui: move upload window into UploadToStorage.js Lorenz Stechauner
2021-07-20 11:51 ` [pve-devel] [PATCH manager 3/5] ui: refactor UploadToStorage.js Lorenz Stechauner
2021-07-20 11:51 ` [pve-devel] [PATCH manager 4/5] ui/UploadToStorage.js: add checksum and algorithm Lorenz Stechauner
2021-07-20 11:51 ` [pve-devel] [PATCH manager 5/5] ui/UploadToStorage.js: add TaskViewer Lorenz Stechauner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal