[pve-devel] [PATCH-SERIES v3 http-server/storage/manager] fix #3505: add checksum and algorithm to iso upload

[pve-devel] [PATCH-SERIES v3 http-server/storage/manager] fix #3505: add checksum and algorithm to iso upload

[Lorenz Stechauner]

changes to v2:
* fixed typo
* prefixed suitable patches with 'fix #3505'
* removed hidden content input field
* moved some part of a patch into the next - did't belong there
* replaced all usages of 'id' with 'reference'
* using a regex instead of the filenameChange function

note: two commits in total state that they break/require some 
versions of other repos. please do not forget to bump those and
create the accoring 'breaks' or 'requires'.


pve-http-server:
Lorenz Stechauner (1):
  anyevent: move unlink from http-server to endpoint

 src/PVE/APIServer/AnyEvent.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)


pve-storage:
Lorenz Stechauner (3):
  status: move unlink from http-server to enpoint
  status: remove sleep(1) in file upload
  fix #3505: status: add checksum and algorithm to file upload

 PVE/API2/Storage/Status.pm | 44 +++++++++++++++++++++++++++++++-------
 1 file changed, 36 insertions(+), 8 deletions(-)


pve-manager:
Lorenz Stechauner (5):
  ui: move upload window into UploadToStorage.js
  ui: refactor UploadToStorage.js
  fix #3505: ui/UploadToStorage: add checksum and algorithm
  ui/UploadToStorage: add TaskViewer
  ui/UplaodToStorage: check file extension

 www/manager6/Makefile                  |   1 +
 www/manager6/storage/ContentView.js    | 197 +----------------
 www/manager6/window/UploadToStorage.js | 295 +++++++++++++++++++++++++
 3 files changed, 298 insertions(+), 195 deletions(-)
 create mode 100644 www/manager6/window/UploadToStorage.js

-- 
2.30.2

[pve-devel] [PATCH v3 http-server 1/1] anyevent: move unlink from http-server to endpoint

[Lorenz Stechauner]

any uploaded file has to be deleted by the corrosponding
endpoint. the file upload was only used by the 'upload to
storage' feature in pve.

this change allows the endpoint to delete the file itself,
making the old and racey`sleep 1` (waiting until the worker
has opened the file) obsolete.

this change breaks all pve-manager versions, in which the
worker does not unlink the temp file itself.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 src/PVE/APIServer/AnyEvent.pm | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm
index cd77806..23b8d10 100644
--- a/src/PVE/APIServer/AnyEvent.pm
+++ b/src/PVE/APIServer/AnyEvent.pm
@@ -114,7 +114,7 @@ sub log_aborted_request {
 }
 
 sub cleanup_reqstate {
-    my ($reqstate) = @_;
+    my ($reqstate, $deletetmpfile) = @_;
 
     delete $reqstate->{log};
     delete $reqstate->{request};
@@ -123,7 +123,7 @@ sub cleanup_reqstate {
     delete $reqstate->{starttime};
 
     if ($reqstate->{tmpfilename}) {
-	unlink $reqstate->{tmpfilename};
+	unlink $reqstate->{tmpfilename} if $deletetmpfile;
 	delete $reqstate->{tmpfilename};
     }
 }
@@ -131,7 +131,7 @@ sub cleanup_reqstate {
 sub client_do_disconnect {
     my ($self, $reqstate) = @_;
 
-    cleanup_reqstate($reqstate);
+    cleanup_reqstate($reqstate, 1);
 
     my $shutdown_hdl = sub {
 	my $hdl = shift;
-- 
2.30.2

[pve-devel] [PATCH v3 storage 1/3] status: move unlink from http-server to enpoint

[Lorenz Stechauner]

this is the first step in which not the http server removes the
temporary file, but the worker itself.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index b838461..a5ac372 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -486,6 +486,7 @@ __PACKAGE__->register_method ({
 	    print "command: " . join(' ', @$cmd) . "\n";
 
 	    eval { run_command($cmd, errmsg => 'import failed'); };
+	    unlink $tmpfilename or warn "unable to clean up temporary file '$tmpfilename' - $!";
 	    if (my $err = $@) {
 		eval { $err_cleanup->() };
 		warn "$@" if $@;
-- 
2.30.2

[pve-devel] [PATCH v3 storage 2/3] status: remove sleep(1) in file upload

[Lorenz Stechauner]

this racey sleep(1) is only there for legacy reasons: because
we don't use apache anymore and only emulate its behabiour
regarding removing temp files, this is under our own control
now and so we can improve this whole situation.

this change requires a pve-http-server version, in which the
tmpfile gets not automatically removed anymore.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index a5ac372..db5a76e 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -495,14 +495,7 @@ __PACKAGE__->register_method ({
 	    print "finished file import successfully\n";
 	};
 
-	my $upid = $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
-
-	# apache removes the temporary file on return, so we need
-	# to wait here to make sure the worker process starts and
-	# opens the file before it gets removed.
-	sleep(1);
-
-	return $upid;
+	return $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
    }});
 
 __PACKAGE__->register_method({
-- 
2.30.2

[pve-devel] [PATCH v3 storage 3/3] fix #3505: status: add checksum and algorithm to file upload

[Lorenz Stechauner]

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 PVE/API2/Storage/Status.pm | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
index db5a76e..07366cc 100644
--- a/PVE/API2/Storage/Status.pm
+++ b/PVE/API2/Storage/Status.pm
@@ -388,6 +388,19 @@ __PACKAGE__->register_method ({
 		maxLength => 255,
 		type => 'string',
 	    },
+	    checksum => {
+		description => "The expected checksum of the file.",
+		type => 'string',
+		requires => 'checksum-algorithm',
+		optional => 1,
+	    },
+	    'checksum-algorithm' => {
+		description => "The algorithm to calculate the checksum of the file.",
+		type => 'string',
+		enum => ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'],
+		requires => 'checksum',
+		optional => 1,
+	    },
 	    tmpfilename => {
 		description => "The source file name. This parameter is usually set by the REST handler. You can only overwrite it when connecting to the trusted port on localhost.",
 		type => 'string',
@@ -480,6 +493,27 @@ __PACKAGE__->register_method ({
 	    my $upid = shift;
 
 	    print "starting file import from: $tmpfilename\n";
+
+	    eval {
+		my ($checksum, $checksum_algorithm) = $param->@{'checksum', 'checksum-algorithm'};
+		if ($checksum_algorithm) {
+		    print "calculating checksum...";
+
+		    my $checksum_got = PVE::Tools::get_file_hash($checksum_algorithm, $tmpfilename);
+
+		    if (lc($checksum_got) eq lc($checksum)) {
+			print "OK, checksum verified\n";
+		    } else {
+			print "\n";  # the front end expects the error to reside at the last line without any noise
+			die "checksum mismatch: got '$checksum_got' != expect '$checksum'\n";
+		    }
+		}
+	    };
+	    if (my $err = $@) {
+		unlink $tmpfilename or warn "unable to clean up temporory file '$tmpfilename' - $!";
+		die $err;
+	    }
+
 	    print "target node: $node\n";
 	    print "target file: $dest\n";
 	    print "file size is: $size\n";
-- 
2.30.2

[pve-devel] [PATCH v3 manager 1/5] ui: move upload window into UploadToStorage.js

[Lorenz Stechauner]

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/Makefile                  |   1 +
 www/manager6/storage/ContentView.js    | 195 +------------------------
 www/manager6/window/UploadToStorage.js | 192 ++++++++++++++++++++++++
 3 files changed, 194 insertions(+), 194 deletions(-)
 create mode 100644 www/manager6/window/UploadToStorage.js

diff --git a/www/manager6/Makefile b/www/manager6/Makefile
index 75d355a5..552e842b 100644
--- a/www/manager6/Makefile
+++ b/www/manager6/Makefile
@@ -106,6 +106,7 @@ JSSRC= 							\
 	window/Snapshot.js				\
 	window/StartupEdit.js				\
 	window/DownloadUrlToStorage.js 			\
+	window/UploadToStorage.js 			\
 	window/Wizard.js				\
 	ha/Fencing.js					\
 	ha/GroupEdit.js					\
diff --git a/www/manager6/storage/ContentView.js b/www/manager6/storage/ContentView.js
index 3f5b686b..ca0ad664 100644
--- a/www/manager6/storage/ContentView.js
+++ b/www/manager6/storage/ContentView.js
@@ -1,196 +1,3 @@
-Ext.define('PVE.storage.Upload', {
-    extend: 'Ext.window.Window',
-    alias: 'widget.pveStorageUpload',
-
-    resizable: false,
-
-    modal: true,
-
-    initComponent: function() {
-        var me = this;
-
-	if (!me.nodename) {
-	    throw "no node name specified";
-	}
-	if (!me.storage) {
-	    throw "no storage ID specified";
-	}
-
-	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
-
-	let pbar = Ext.create('Ext.ProgressBar', {
-            text: 'Ready',
-	    hidden: true,
-	});
-
-	let acceptedExtensions = {
-	    iso: ".img, .iso",
-	    vztmpl: ".tar.gz, .tar.xz",
-	};
-
-	let defaultContent = me.contents[0] || '';
-
-	let fileField = Ext.create('Ext.form.field.File', {
-	    name: 'filename',
-	    buttonText: gettext('Select File...'),
-	    allowBlank: false,
-	    setAccept: function(content) {
-		let acceptString = acceptedExtensions[content] || '';
-		this.fileInputEl.set({
-		    accept: acceptString,
-		});
-	    },
-	    listeners: {
-		afterrender: function(cmp) {
-		    cmp.setAccept(defaultContent);
-		},
-	    },
-	});
-
-	me.formPanel = Ext.create('Ext.form.Panel', {
-	    method: 'POST',
-	    waitMsgTarget: true,
-	    bodyPadding: 10,
-	    border: false,
-	    width: 300,
-	    fieldDefaults: {
-		labelWidth: 100,
-		anchor: '100%',
-            },
-	    items: [
-		{
-		    xtype: 'pveContentTypeSelector',
-		    cts: me.contents,
-		    fieldLabel: gettext('Content'),
-		    name: 'content',
-		    value: defaultContent,
-		    allowBlank: false,
-		    listeners: {
-			change: function(cmp, newValue, oldValue) {
-			    fileField.setAccept(newValue);
-			},
-		    },
-		},
-		fileField,
-		pbar,
-	    ],
-	});
-
-	let form = me.formPanel.getForm();
-
-	let doStandardSubmit = function() {
-	    form.submit({
-		url: "/api2/htmljs" + baseurl,
-		waitMsg: gettext('Uploading file...'),
-		success: function(f, action) {
-		    me.close();
-		},
-		failure: function(f, action) {
-		    var msg = PVE.Utils.extractFormActionError(action);
-                    Ext.Msg.alert(gettext('Error'), msg);
-		},
-	    });
-	};
-
-	let updateProgress = function(per, bytes) {
-	    var text = (per * 100).toFixed(2) + '%';
-	    if (bytes) {
-		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
-	    }
-	    pbar.updateProgress(per, text);
-	};
-
-	let abortBtn = Ext.create('Ext.Button', {
-	    text: gettext('Abort'),
-	    disabled: true,
-	    handler: function() {
-		me.close();
-	    },
-	});
-
-	let submitBtn = Ext.create('Ext.Button', {
-	    text: gettext('Upload'),
-	    disabled: true,
-	    handler: function(button) {
-		var fd;
-		try {
-		    fd = new FormData();
-		} catch (err) {
-		    doStandardSubmit();
-		    return;
-		}
-
-		button.setDisabled(true);
-		abortBtn.setDisabled(false);
-
-		var field = form.findField('content');
-		fd.append("content", field.getValue());
-		field.setDisabled(true);
-
-		field = form.findField('filename');
-		var file = field.fileInputEl.dom;
-		fd.append("filename", file.files[0]);
-		field.setDisabled(true);
-
-		pbar.setVisible(true);
-		updateProgress(0);
-
-		let xhr = new XMLHttpRequest();
-		me.xhr = xhr;
-
-		xhr.addEventListener("load", function(e) {
-		    if (xhr.status === 200) {
-			me.close();
-			return;
-		    }
-		    let err = Ext.htmlEncode(xhr.statusText);
-		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
-		    if (xhr.responseText !== "") {
-			let result = Ext.decode(xhr.responseText);
-			result.message = msg;
-			msg = Proxmox.Utils.extractRequestError(result, true);
-		    }
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		}, false);
-
-		xhr.addEventListener("error", function(e) {
-		    let err = e.target.status.toString();
-		    let msg = `Error '${err}' occurred while receiving the document.`;
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		});
-
-		xhr.upload.addEventListener("progress", function(evt) {
-		    if (evt.lengthComputable) {
-			let percentComplete = evt.loaded / evt.total;
-			updateProgress(percentComplete, evt.loaded);
-		    }
-		}, false);
-
-		xhr.open("POST", `/api2/json${baseurl}`, true);
-		xhr.send(fd);
-	    },
-	});
-
-	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
-
-	Ext.apply(me, {
-	    title: gettext('Upload'),
-	    items: me.formPanel,
-	    buttons: [abortBtn, submitBtn],
-	    listeners: {
-		close: function() {
-		    if (me.xhr) {
-			me.xhr.abort();
-			delete me.xhr;
-		    }
-		},
-	    },
-	});
-
-        me.callParent();
-    },
-});
-
 Ext.define('PVE.storage.ContentView', {
     extend: 'Ext.grid.GridPanel',
 
@@ -259,7 +66,7 @@ Ext.define('PVE.storage.ContentView', {
 		    text: gettext('Upload'),
 		    disabled: !me.enableUploadButton,
 		    handler: function() {
-			Ext.create('PVE.storage.Upload', {
+			Ext.create('PVE.window.UploadToStorage', {
 			    nodename: nodename,
 			    storage: storage,
 			    contents: [content],
diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
new file mode 100644
index 00000000..3c35020a
--- /dev/null
+++ b/www/manager6/window/UploadToStorage.js
@@ -0,0 +1,192 @@
+Ext.define('PVE.window.UploadToStorage', {
+    extend: 'Ext.window.Window',
+    alias: 'widget.pveStorageUpload',
+
+    resizable: false,
+
+    modal: true,
+
+    initComponent: function() {
+        var me = this;
+
+	if (!me.nodename) {
+	    throw "no node name specified";
+	}
+	if (!me.storage) {
+	    throw "no storage ID specified";
+	}
+
+	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
+
+	let pbar = Ext.create('Ext.ProgressBar', {
+            text: 'Ready',
+	    hidden: true,
+	});
+
+	let acceptedExtensions = {
+	    iso: ".img, .iso",
+	    vztmpl: ".tar.gz, .tar.xz",
+	};
+
+	let defaultContent = me.contents[0] || '';
+
+	let fileField = Ext.create('Ext.form.field.File', {
+	    name: 'filename',
+	    buttonText: gettext('Select File...'),
+	    allowBlank: false,
+	    setAccept: function(content) {
+		let acceptString = acceptedExtensions[content] || '';
+		this.fileInputEl.set({
+		    accept: acceptString,
+		});
+	    },
+	    listeners: {
+		afterrender: function(cmp) {
+		    cmp.setAccept(defaultContent);
+		},
+	    },
+	});
+
+	me.formPanel = Ext.create('Ext.form.Panel', {
+	    method: 'POST',
+	    waitMsgTarget: true,
+	    bodyPadding: 10,
+	    border: false,
+	    width: 300,
+	    fieldDefaults: {
+		labelWidth: 100,
+		anchor: '100%',
+            },
+	    items: [
+		{
+		    xtype: 'pveContentTypeSelector',
+		    cts: me.contents,
+		    fieldLabel: gettext('Content'),
+		    name: 'content',
+		    value: defaultContent,
+		    allowBlank: false,
+		    listeners: {
+			change: function(cmp, newValue, oldValue) {
+			    fileField.setAccept(newValue);
+			},
+		    },
+		},
+		fileField,
+		pbar,
+	    ],
+	});
+
+	let form = me.formPanel.getForm();
+
+	let doStandardSubmit = function() {
+	    form.submit({
+		url: "/api2/htmljs" + baseurl,
+		waitMsg: gettext('Uploading file...'),
+		success: function(f, action) {
+		    me.close();
+		},
+		failure: function(f, action) {
+		    var msg = PVE.Utils.extractFormActionError(action);
+                    Ext.Msg.alert(gettext('Error'), msg);
+		},
+	    });
+	};
+
+	let updateProgress = function(per, bytes) {
+	    var text = (per * 100).toFixed(2) + '%';
+	    if (bytes) {
+		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
+	    }
+	    pbar.updateProgress(per, text);
+	};
+
+	let abortBtn = Ext.create('Ext.Button', {
+	    text: gettext('Abort'),
+	    disabled: true,
+	    handler: function() {
+		me.close();
+	    },
+	});
+
+	let submitBtn = Ext.create('Ext.Button', {
+	    text: gettext('Upload'),
+	    disabled: true,
+	    handler: function(button) {
+		var fd;
+		try {
+		    fd = new FormData();
+		} catch (err) {
+		    doStandardSubmit();
+		    return;
+		}
+
+		button.setDisabled(true);
+		abortBtn.setDisabled(false);
+
+		var field = form.findField('content');
+		fd.append("content", field.getValue());
+		field.setDisabled(true);
+
+		field = form.findField('filename');
+		var file = field.fileInputEl.dom;
+		fd.append("filename", file.files[0]);
+		field.setDisabled(true);
+
+		pbar.setVisible(true);
+		updateProgress(0);
+
+		let xhr = new XMLHttpRequest();
+		me.xhr = xhr;
+
+		xhr.addEventListener("load", function(e) {
+		    if (xhr.status === 200) {
+			me.close();
+			return;
+		    }
+		    let err = Ext.htmlEncode(xhr.statusText);
+		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
+		    if (xhr.responseText !== "") {
+			let result = Ext.decode(xhr.responseText);
+			result.message = msg;
+			msg = Proxmox.Utils.extractRequestError(result, true);
+		    }
+		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
+		}, false);
+
+		xhr.addEventListener("error", function(e) {
+		    let err = e.target.status.toString();
+		    let msg = `Error '${err}' occurred while receiving the document.`;
+		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
+		});
+
+		xhr.upload.addEventListener("progress", function(evt) {
+		    if (evt.lengthComputable) {
+			let percentComplete = evt.loaded / evt.total;
+			updateProgress(percentComplete, evt.loaded);
+		    }
+		}, false);
+
+		xhr.open("POST", `/api2/json${baseurl}`, true);
+		xhr.send(fd);
+	    },
+	});
+
+	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
+
+	Ext.apply(me, {
+	    title: gettext('Upload'),
+	    items: me.formPanel,
+	    buttons: [abortBtn, submitBtn],
+	    listeners: {
+		close: function() {
+		    if (me.xhr) {
+			me.xhr.abort();
+			delete me.xhr;
+		    }
+		},
+	    },
+	});
+
+        me.callParent();
+    },
+});
-- 
2.30.2

[pve-devel] [PATCH v3 manager 2/5] ui: refactor UploadToStorage.js

[Lorenz Stechauner]

this also removes the "content" selector from the window.
as far as it seems, this selector was never able to select
more than one entry, so it was useless.

the check for FormData() is also removed, because this is
supported by all major browsers for a long time. therefore
doStandardSubmit() is also not necessary.

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/storage/ContentView.js    |   2 +-
 www/manager6/window/UploadToStorage.js | 346 ++++++++++++++-----------
 2 files changed, 196 insertions(+), 152 deletions(-)

diff --git a/www/manager6/storage/ContentView.js b/www/manager6/storage/ContentView.js
index ca0ad664..00a94f3c 100644
--- a/www/manager6/storage/ContentView.js
+++ b/www/manager6/storage/ContentView.js
@@ -69,7 +69,7 @@ Ext.define('PVE.storage.ContentView', {
 			Ext.create('PVE.window.UploadToStorage', {
 			    nodename: nodename,
 			    storage: storage,
-			    contents: [content],
+			    content: content,
 			    autoShow: true,
 			    taskDone: () => reload(),
 			});
diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index 3c35020a..e7db41fb 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -1,191 +1,235 @@
 Ext.define('PVE.window.UploadToStorage', {
     extend: 'Ext.window.Window',
     alias: 'widget.pveStorageUpload',
+    mixins: ['Proxmox.Mixin.CBind'],
 
     resizable: false,
-
     modal: true,
 
-    initComponent: function() {
-        var me = this;
+    title: gettext('Upload'),
 
-	if (!me.nodename) {
-	    throw "no node name specified";
-	}
-	if (!me.storage) {
-	    throw "no storage ID specified";
-	}
+    acceptedExtensions: {
+	iso: ['.img', '.iso'],
+	vztmpl: ['.tar.gz', '.tar.xz'],
+    },
 
-	let baseurl = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
+    cbindData: function(initialConfig) {
+	const me = this;
+	const ext = me.acceptedExtensions[me.content];
 
-	let pbar = Ext.create('Ext.ProgressBar', {
-            text: 'Ready',
-	    hidden: true,
-	});
+	me.url = `/nodes/${me.nodename}/storage/${me.storage}/upload`;
 
-	let acceptedExtensions = {
-	    iso: ".img, .iso",
-	    vztmpl: ".tar.gz, .tar.xz",
+	return {
+	    extensions: ext.join(', '),
 	};
+    },
 
-	let defaultContent = me.contents[0] || '';
-
-	let fileField = Ext.create('Ext.form.field.File', {
-	    name: 'filename',
-	    buttonText: gettext('Select File...'),
-	    allowBlank: false,
-	    setAccept: function(content) {
-		let acceptString = acceptedExtensions[content] || '';
-		this.fileInputEl.set({
-		    accept: acceptString,
-		});
-	    },
-	    listeners: {
-		afterrender: function(cmp) {
-		    cmp.setAccept(defaultContent);
-		},
-	    },
-	});
+    viewModel: {
+	data: {
+	    size: '-',
+	    mimetype: '-',
+	    filename: '',
+	},
+    },
+
+    controller: {
+	submit: function(button) {
+	    const view = this.getView();
+	    const form = this.lookup('formPanel').getForm();
+	    const abortBtn = this.lookup('abortBtn');
+	    const pbar = this.lookup('progressBar');
+
+	    const updateProgress = function(per, bytes) {
+		let text = (per * 100).toFixed(2) + '%';
+		if (bytes) {
+		    text += " (" + Proxmox.Utils.format_size(bytes) + ')';
+		}
+		pbar.updateProgress(per, text);
+	    };
+
+	    const fd = new FormData();
+
+	    button.setDisabled(true);
+	    abortBtn.setDisabled(false);
+
+	    fd.append("content", view.content);
 
-	me.formPanel = Ext.create('Ext.form.Panel', {
+	    const fileField = form.findField('file');
+	    const file = fileField.fileInputEl.dom.files[0];
+	    fileField.setDisabled(true);
+
+	    const filenameField = form.findField('filename');
+	    const filename = filenameField.getValue();
+	    filenameField.setDisabled(true);
+
+	    fd.append("filename", file, filename);
+
+	    pbar.setVisible(true);
+	    updateProgress(0);
+
+	    const xhr = new XMLHttpRequest();
+	    view.xhr = xhr;
+
+	    xhr.addEventListener("load", function(e) {
+		if (xhr.status === 200) {
+		    view.close();
+		    return;
+		}
+		const err = Ext.htmlEncode(xhr.statusText);
+		let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
+		if (xhr.responseText !== "") {
+		    const result = Ext.decode(xhr.responseText);
+		    result.message = msg;
+		    msg = Proxmox.Utils.extractRequestError(result, true);
+		}
+		Ext.Msg.alert(gettext('Error'), msg, btn => view.close());
+	    }, false);
+
+	    xhr.addEventListener("error", function(e) {
+		const err = e.target.status.toString();
+		const msg = `Error '${err}' occurred while receiving the document.`;
+		Ext.Msg.alert(gettext('Error'), msg, btn => view.close());
+	    });
+
+	    xhr.upload.addEventListener("progress", function(evt) {
+		if (evt.lengthComputable) {
+		    const percentComplete = evt.loaded / evt.total;
+		    updateProgress(percentComplete, evt.loaded);
+		}
+	    }, false);
+
+	    xhr.open("POST", `/api2/json${view.url}`, true);
+	    xhr.send(fd);
+	},
+
+	validitychange: function(f, valid) {
+	    const submitBtn = this.lookup('submitBtn');
+	    submitBtn.setDisabled(!valid);
+	},
+
+	fileChange: function(input) {
+	    const vm = this.getViewModel();
+	    const name = input.value.replace(/^.*(\/|\\)/, '');
+	    const fileInput = input.fileInputEl.dom;
+	    vm.set('filename', name);
+	    vm.set('size', (fileInput.files[0] && Proxmox.Utils.format_size(fileInput.files[0].size)) || '-');
+	    vm.set('mimetype', (fileInput.files[0] && fileInput.files[0].type) || '-');
+	},
+    },
+
+    items: [
+	{
+	    xtype: 'form',
+	    reference: 'formPanel',
 	    method: 'POST',
 	    waitMsgTarget: true,
 	    bodyPadding: 10,
 	    border: false,
-	    width: 300,
+	    width: 400,
 	    fieldDefaults: {
 		labelWidth: 100,
 		anchor: '100%',
             },
 	    items: [
 		{
-		    xtype: 'pveContentTypeSelector',
-		    cts: me.contents,
-		    fieldLabel: gettext('Content'),
-		    name: 'content',
-		    value: defaultContent,
+		    xtype: 'filefield',
+		    name: 'file',
+		    buttonText: gettext('Select File'),
 		    allowBlank: false,
+		    fieldLabel: gettext('File'),
+		    cbind: {
+			accept: '{extensions}',
+		    },
 		    listeners: {
-			change: function(cmp, newValue, oldValue) {
-			    fileField.setAccept(newValue);
-			},
+			change: 'fileChange',
 		    },
 		},
-		fileField,
-		pbar,
-	    ],
-	});
-
-	let form = me.formPanel.getForm();
-
-	let doStandardSubmit = function() {
-	    form.submit({
-		url: "/api2/htmljs" + baseurl,
-		waitMsg: gettext('Uploading file...'),
-		success: function(f, action) {
-		    me.close();
+		{
+		    xtype: 'textfield',
+		    name: 'filename',
+		    allowBlank: false,
+		    fieldLabel: gettext('File name'),
+		    bind: {
+			value: '{filename}',
+		    },
 		},
-		failure: function(f, action) {
-		    var msg = PVE.Utils.extractFormActionError(action);
-                    Ext.Msg.alert(gettext('Error'), msg);
+		{
+		    xtype: 'displayfield',
+		    name: 'size',
+		    fieldLabel: gettext('File size'),
+		    bind: {
+			value: '{size}',
+		    },
 		},
-	    });
-	};
-
-	let updateProgress = function(per, bytes) {
-	    var text = (per * 100).toFixed(2) + '%';
-	    if (bytes) {
-		text += " (" + Proxmox.Utils.format_size(bytes) + ')';
-	    }
-	    pbar.updateProgress(per, text);
-	};
-
-	let abortBtn = Ext.create('Ext.Button', {
+		{
+		    xtype: 'displayfield',
+		    name: 'mimetype',
+		    fieldLabel: gettext('MIME type'),
+		    bind: {
+			value: '{mimetype}',
+		    },
+		},
+		{
+		    xtype: 'progressbar',
+		    text: 'Ready',
+		    hidden: true,
+		    reference: 'progressBar',
+		},
+		{
+		    xtype: 'hiddenfield',
+		    name: 'content',
+		    cbind: {
+			value: '{content}',
+		    },
+		},
+	    ],
+	   listeners: {
+		validitychange: 'validitychange',
+	   },
+	},
+    ],
+
+    buttons: [
+	{
+	    xtype: 'button',
 	    text: gettext('Abort'),
+	    reference: 'abortBtn',
 	    disabled: true,
 	    handler: function() {
-		me.close();
+		const me = this;
+		me.up('pveStorageUpload').close();
 	    },
-	});
-
-	let submitBtn = Ext.create('Ext.Button', {
+	},
+	{
 	    text: gettext('Upload'),
+	    reference: 'submitBtn',
 	    disabled: true,
-	    handler: function(button) {
-		var fd;
-		try {
-		    fd = new FormData();
-		} catch (err) {
-		    doStandardSubmit();
-		    return;
-		}
+	    handler: 'submit',
+	},
+    ],
+
+    listeners: {
+	close: function() {
+	    const me = this;
+	    if (me.xhr) {
+		me.xhr.abort();
+		delete me.xhr;
+	    }
+	},
+    },
 
-		button.setDisabled(true);
-		abortBtn.setDisabled(false);
-
-		var field = form.findField('content');
-		fd.append("content", field.getValue());
-		field.setDisabled(true);
-
-		field = form.findField('filename');
-		var file = field.fileInputEl.dom;
-		fd.append("filename", file.files[0]);
-		field.setDisabled(true);
-
-		pbar.setVisible(true);
-		updateProgress(0);
-
-		let xhr = new XMLHttpRequest();
-		me.xhr = xhr;
-
-		xhr.addEventListener("load", function(e) {
-		    if (xhr.status === 200) {
-			me.close();
-			return;
-		    }
-		    let err = Ext.htmlEncode(xhr.statusText);
-		    let msg = `${gettext('Error')} ${xhr.status.toString()}: ${err}`;
-		    if (xhr.responseText !== "") {
-			let result = Ext.decode(xhr.responseText);
-			result.message = msg;
-			msg = Proxmox.Utils.extractRequestError(result, true);
-		    }
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		}, false);
-
-		xhr.addEventListener("error", function(e) {
-		    let err = e.target.status.toString();
-		    let msg = `Error '${err}' occurred while receiving the document.`;
-		    Ext.Msg.alert(gettext('Error'), msg, btn => me.close());
-		});
-
-		xhr.upload.addEventListener("progress", function(evt) {
-		    if (evt.lengthComputable) {
-			let percentComplete = evt.loaded / evt.total;
-			updateProgress(percentComplete, evt.loaded);
-		    }
-		}, false);
-
-		xhr.open("POST", `/api2/json${baseurl}`, true);
-		xhr.send(fd);
-	    },
-	});
-
-	form.on('validitychange', (f, valid) => submitBtn.setDisabled(!valid));
-
-	Ext.apply(me, {
-	    title: gettext('Upload'),
-	    items: me.formPanel,
-	    buttons: [abortBtn, submitBtn],
-	    listeners: {
-		close: function() {
-		    if (me.xhr) {
-			me.xhr.abort();
-			delete me.xhr;
-		    }
-		},
-	    },
-	});
+    initComponent: function() {
+        const me = this;
+
+	if (!me.nodename) {
+	    throw "no node name specified";
+	}
+	if (!me.storage) {
+	    throw "no storage ID specified";
+	}
+	if (!(me.content in me.acceptedExtensions)) {
+	    throw "content type not supported";
+	}
 
         me.callParent();
     },
-- 
2.30.2

[pve-devel] [PATCH v3 manager 3/5] fix #3505: ui/UploadToStorage: add checksum and algorithm

[Lorenz Stechauner]

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/window/UploadToStorage.js | 40 ++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index e7db41fb..489be9dc 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -62,6 +62,16 @@ Ext.define('PVE.window.UploadToStorage', {
 	    const filename = filenameField.getValue();
 	    filenameField.setDisabled(true);
 
+	    const algorithmField = form.findField('checksum-algorithm');
+	    algorithmField.setDisabled(true);
+	    if (algorithmField.getValue() !== '__default__') {
+		fd.append("checksum-algorithm", algorithmField.getValue());
+
+		const checksumField = form.findField('checksum');
+		fd.append("checksum", checksumField.getValue());
+		checksumField.setDisabled(true);
+	    }
+
 	    fd.append("filename", file, filename);
 
 	    pbar.setVisible(true);
@@ -115,6 +125,16 @@ Ext.define('PVE.window.UploadToStorage', {
 	    vm.set('size', (fileInput.files[0] && Proxmox.Utils.format_size(fileInput.files[0].size)) || '-');
 	    vm.set('mimetype', (fileInput.files[0] && fileInput.files[0].type) || '-');
 	},
+
+	hashChange: function(field, value) {
+	    const checksum = this.lookup('downloadUrlChecksum');
+	    if (value === '__default__') {
+		checksum.setDisabled(true);
+		checksum.setValue("");
+	    } else {
+		checksum.setDisabled(false);
+	    }
+	},
     },
 
     items: [
@@ -169,6 +189,26 @@ Ext.define('PVE.window.UploadToStorage', {
 			value: '{mimetype}',
 		    },
 		},
+		{
+		    xtype: 'pveHashAlgorithmSelector',
+		    name: 'checksum-algorithm',
+		    fieldLabel: gettext('Hash algorithm'),
+		    allowBlank: true,
+		    hasNoneOption: true,
+		    value: '__default__',
+		    listeners: {
+			change: 'hashChange',
+		    },
+		},
+		{
+		    xtype: 'textfield',
+		    name: 'checksum',
+		    fieldLabel: gettext('Checksum'),
+		    allowBlank: false,
+		    disabled: true,
+		    emptyText: gettext('none'),
+		    reference: 'downloadUrlChecksum',
+		},
 		{
 		    xtype: 'progressbar',
 		    text: 'Ready',
-- 
2.30.2

[pve-devel] [PATCH v3 manager 4/5] ui/UploadToStorage: add TaskViewer

[Lorenz Stechauner]

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/window/UploadToStorage.js | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index 489be9dc..f0c6186f 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -82,7 +82,21 @@ Ext.define('PVE.window.UploadToStorage', {
 
 	    xhr.addEventListener("load", function(e) {
 		if (xhr.status === 200) {
-		    view.close();
+		    view.hide();
+
+		    const result = JSON.parse(xhr.response);
+		    const upid = result.data;
+		    Ext.create('Proxmox.window.TaskViewer', {
+			autoShow: true,
+			upid: upid,
+			taskDone: view.taskDone,
+			listeners: {
+			    destroy: function() {
+				view.close();
+			    },
+			},
+		    });
+
 		    return;
 		}
 		const err = Ext.htmlEncode(xhr.statusText);
-- 
2.30.2

[pve-devel] [PATCH v3 manager 5/5] ui/UplaodToStorage: check file extension

[Lorenz Stechauner]

Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
---
 www/manager6/window/UploadToStorage.js | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/www/manager6/window/UploadToStorage.js b/www/manager6/window/UploadToStorage.js
index f0c6186f..ec83c581 100644
--- a/www/manager6/window/UploadToStorage.js
+++ b/www/manager6/window/UploadToStorage.js
@@ -21,6 +21,7 @@ Ext.define('PVE.window.UploadToStorage', {
 
 	return {
 	    extensions: ext.join(', '),
+	    filenameRegex: RegExp('^.*(?:' + ext.join('|').replaceAll('.', '\\.') + ')$', 'i'),
 	};
     },
 
@@ -186,6 +187,10 @@ Ext.define('PVE.window.UploadToStorage', {
 		    bind: {
 			value: '{filename}',
 		    },
+		    cbind: {
+			regex: '{filenameRegex}',
+		    },
+		    regexText: "wrong file extension",
 		},
 		{
 		    xtype: 'displayfield',
-- 
2.30.2

Re: [pve-devel] [PATCH v3 storage 1/3] status: move unlink from http-server to enpoint

[Thomas Lamprecht]

On 03/08/2021 14:16, Lorenz Stechauner wrote:
> this is the first step in which not the http server removes the
> temporary file, but the worker itself.
> 
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  PVE/API2/Storage/Status.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index b838461..a5ac372 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -486,6 +486,7 @@ __PACKAGE__->register_method ({
>  	    print "command: " . join(' ', @$cmd) . "\n";
>  
>  	    eval { run_command($cmd, errmsg => 'import failed'); };
> +	    unlink $tmpfilename or warn "unable to clean up temporary file '$tmpfilename' - $!";

nit as it may not happen often in practice but it could be good to check if unlink
only failed because it is already gone (or was never produced):

unlink $tmpfilename;
warn "unable to clean up temporary file '$tmpfilename' - $!" if $! && $! != ENOENT;

(just wrote that from top of my mind, so not tested, you may need to add a
`use POSIX qw(ENOENT)` stanza in the module.)

>  	    if (my $err = $@) {
>  		eval { $err_cleanup->() };
>  		warn "$@" if $@;
> 

Re: [pve-devel] [PATCH v3 storage 1/3] status: move unlink from http-server to enpoint

[Thomas Lamprecht]

On 03/08/2021 14:16, Lorenz Stechauner wrote:
> this is the first step in which not the http server removes the
> temporary file, but the worker itself.
> 
> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
> ---
>  PVE/API2/Storage/Status.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index b838461..a5ac372 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -486,6 +486,7 @@ __PACKAGE__->register_method ({
>  	    print "command: " . join(' ', @$cmd) . "\n";
>  
>  	    eval { run_command($cmd, errmsg => 'import failed'); };
> +	    unlink $tmpfilename or warn "unable to clean up temporary file '$tmpfilename' - $!";

and doesn't this have the same issues as you recently fixed in regards to the node
not being the local one and thus ssh/scp being used?

>  	    if (my $err = $@) {
>  		eval { $err_cleanup->() };
>  		warn "$@" if $@;
> 

Re: [pve-devel] [PATCH v3 storage 1/3] status: move unlink from http-server to enpoint

[Lorenz Stechauner]

On 26.08.21 18:30, Thomas Lamprecht wrote:
> On 03/08/2021 14:16, Lorenz Stechauner wrote:
>> this is the first step in which not the http server removes the
>> temporary file, but the worker itself.
>>
>> Signed-off-by: Lorenz Stechauner <l.stechauner@proxmox.com>
>> ---
>>   PVE/API2/Storage/Status.pm | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
>> index b838461..a5ac372 100644
>> --- a/PVE/API2/Storage/Status.pm
>> +++ b/PVE/API2/Storage/Status.pm
>> @@ -486,6 +486,7 @@ __PACKAGE__->register_method ({
>>   	    print "command: " . join(' ', @$cmd) . "\n";
>>   
>>   	    eval { run_command($cmd, errmsg => 'import failed'); };
>> +	    unlink $tmpfilename or warn "unable to clean up temporary file '$tmpfilename' - $!";
> and doesn't this have the same issues as you recently fixed in regards to the node
> not being the local one and thus ssh/scp being used?
in this case, the local temp file is deleted and this is intended. 
previously, this was done by the http server.
but it would be better to clean up the remote file as well and therefore 
a ssh command has to be used - could be another commit
>
>>   	    if (my $err = $@) {
>>   		eval { $err_cleanup->() };
>>   		warn "$@" if $@;
>>

Re: [pve-devel] [PATCH v3 storage 1/3] status: move unlink from http-server to enpoint

[Thomas Lamprecht]

On 30.08.21 13:54, Lorenz Stechauner wrote:
> On 26.08.21 18:30, Thomas Lamprecht wrote:
>> On 03/08/2021 14:16, Lorenz Stechauner wrote:
>>> @@ -486,6 +486,7 @@ __PACKAGE__->register_method ({
>>>           print "command: " . join(' ', @$cmd) . "\n";
>>>             eval { run_command($cmd, errmsg => 'import failed'); };
>>> +        unlink $tmpfilename or warn "unable to clean up temporary file '$tmpfilename' - $!";
>> and doesn't this have the same issues as you recently fixed in regards to the node
>> not being the local one and thus ssh/scp being used?
> in this case, the local temp file is deleted and this is intended. previously, this was done by the http server.
> but it would be better to clean up the remote file as well and therefore a ssh command has to be used - could be another commit

Oh, you're right, this is only the intermediate uploaded file which is always local.
A comment could be great for that detail, but I can add that one as follow up.