From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 349981FF0C0 for ; Wed, 18 Nov 2026 11:32:14 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2F59A21868; Wed, 18 Nov 2026 11:31:35 +0100 (CET) From: Thomas Ellmenreich To: pve-devel@lists.proxmox.com Subject: [PATCH common v2 00/11] pem parser: refactor parsing to cover more edge cases Date: Wed, 18 Nov 2026 11:31:14 +0100 Message-ID: <20261118103127.165100-1-t.ellmenreich@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794997889020 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.338 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RIW7AWC3L2OJVFLUTACLLI3XRUPUSYQY X-Message-ID-Hash: RIW7AWC3L2OJVFLUTACLLI3XRUPUSYQY X-MailFrom: t.ellmenreich@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Thomas Ellmenreich X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: After the previous series: [0], was already applied, I realized that the parsing ignores a specific edgecase. After that, I was also pointed to the following thread [1], which includes another edgecase. Trying to fix these edgecases, I have decided to rework the PEM parsing so that it cleans incoming input, then parses it in a stricter manner, and finally outputs a more standardized PEM at the end. Change Overview: --------------- I have tried to split up all of the new changes into separate commits to make them more easily reversible but also better understandable. But here is a summary of said changes: * Introduction of a ANY_LABEL constant instead of recreating 'qr/.*?/' everywhere. This new constant also avoids matching a next header. * 'pem-string' format calls 'check_pem' with multiple, since PEM chains should also be valid. * Force BEGIN and END lines to start at the beginning of the line and either end with the end of the line or the string. * Make the validation of single PEM entries stricter so that it adheres more to RFC 7468 [2]. * Add a utility to not only strip leading, but also following text, so that subroutines that are expected to return a single item, but are handed a chain, can strip everything except the first item. * Parsing a chain now adds the cleaned and validated entries to a list, before joining them with newlines instead of adding them to a string. * Decoding a PEM to a DER is quicker to avoid parsing invalid PEM's. * Refactored and expanded the tests to cover more cases. * Added CRLF support by simply replacing them all with newlines at the beginning of the 'check_pem', 'split_pem' and 'der_to_pem' subroutines. Changelog: --------- * v1 -> v2: The first version only contained the usage of the ANY_LABEL for the 'pem_to_der' subroutine, as well as the check that BEGIN and END should be on their own lines. Through the further edge case mentioned in the forum post [1], I decided to take a more complete but more invasive approach. Open Questions: -------------- * Integration tests with openssl I started working on some integration tests that work directly with openssl to generate PEM's, which are then validated by our library. In the end, I decided to drop the patch in favour of gathering some opinions first. My approach would have been to generate some edgecase PEM's with openssl and commit them, the Perl code could then be run against these PEM's to validate the stability of its functionality. Otherwise, we could also make the tests depend on openssl directly, and thus generate a way greater number of PEM's on the fly, to then test against. * Normalisation of content: One more improvement that could be made is to pass the base64 content through the decoder and encoder, which would lead it to be normalized. I personally find this unnecessary, but it could be one more thing to make the output of the 'check_pem' function more consistent. Notes for the Maintainer: ------------------------ I have split up the series into a number of different commits to make them easier to understand, but I would recommend to doublecheck before dropping an inbetween commit, as not every commit makes sense by themselves. That said, I have made sure that all commits pass all tests and can be built. [0]: https://lore.proxmox.com/all/20260805141114.190477-1-t.ellmenreich@proxmox.com/T/#u [1]: https://forum.proxmox.com/threads/custom-certificate-pem-with-comments-or-key-pem-with-a-section-ec-parameters-fail-to-upload.187526/#post-875517 [2]: https://www.rfc-editor.org/info/rfc7468/ pve-common: Thomas Ellmenreich (11): certificate tests: use q{} for multiline strings. pem parser: add constant to match any label pem parser: refactor check_pem to use list instead of string pem parser: stricter handling of newlines certificate formats: enable multiple parsing for pem-string format pem parsing: strip all surrounding text from chain entries pem parsing: make validation of pem entry stricter pem parsing: default splitting to any label pem to der: create clearer error boundary certificate tests: refactor and exapand testsuite pom parsing: allow for CRLF line ending src/PVE/Certificate.pm | 81 ++++-- test/Makefile | 2 + test/certificate_utils_test.pl | 355 ++++++++++++++++++++++++++ test/check_pem_test.pl | 444 +++++++++++++-------------------- test/pem_to_der_test.pl | 172 +++++++++++++ test/split_pem_test.pl | 260 +++++++++---------- 6 files changed, 891 insertions(+), 423 deletions(-) create mode 100755 test/certificate_utils_test.pl create mode 100755 test/pem_to_der_test.pl Summary over all repositories: 6 files changed, 891 insertions(+), 423 deletions(-) -- Generated by murpp 0.12.0