From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2D081B50D for ; Mon, 11 Sep 2023 14:50:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0CF2B882A for ; Mon, 11 Sep 2023 14:50:24 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 11 Sep 2023 14:50:21 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 3A8D244645 for ; Mon, 11 Sep 2023 14:50:21 +0200 (CEST) Message-ID: Date: Mon, 11 Sep 2023 14:50:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Content-Language: en-US To: Proxmox VE development discussion , Lukas Wagner References: <20230830123754.606260-1-l.wagner@proxmox.com> From: Fiona Ebner In-Reply-To: <20230830123754.606260-1-l.wagner@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.080 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] applied-series: [PATCH common/cluster 0/3] fix #4937: fix utf8 encoding issues while saving notification config X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2023 12:50:54 -0000 Am 30.08.23 um 14:37 schrieb Lukas Wagner: > These patches should fix issues with certain special characters > (e.g ü) in the notification configuration [1]. > Before, when setting a comment for an endpoint to certain values > (e.g. 'für admins'), the resulting saved configuration would > contain invalid UTF8 text. > > The reason for that was the configuration was returned from > Rust as a proper utf8 string, however it was saved to the file > system without specifying an encoding. > > > [1] https://bugzilla.proxmox.com/show_bug.cgi?id=4937 > applied-series, after discussing with Fabian, thanks! We decided to make it a $force_utf8 flag instead of passing the encoding as a string, and use the strict 'UTF-8' encoding rather than 'utf8'. See the "UTF-8 vs. utf8 vs. UTF8" section in 'perldoc Encode' for more info. I also moved the binmode() call to after the loop, where $fh is known to be set. We also found that file_get_contents() doesn't set Perl's UTF-8 flag for strings even if the input is UTF-8, because it uses sysread(). It's fine for the notification configs apparently, but going forward, we'll need to be careful. One idea was an opt-in parameter for detecting UTF-8 and setting the Perl string flag and using that for (most) cluster filesystem files.