[pve-devel] [TurnKey Linux] Looking to update our signing key... Advice?

[pve-devel] [TurnKey Linux] Looking to update our signing key... Advice?

[Jeremy Davis]

[-- Attachment #1.1: Type: text/plain, Size: 1287 bytes --]

Hi,

Apologies in advance if this is not the right place to post this. Please 
redirect me to the appropriate forum if not. I'm also happy to discuss 
off list if that is deemed more appropriate.

My name is Jeremy and I work with TurnKey Linux.

As a housekeeping matter, we're looking to update our GPG signing key - 
that we sign the index file we provide for downloading our LXC templates 
via the PVE UI (which includes hashes of our templates).

The current key recently expired (caught us a bit unawares). We updated 
the expiry to keep it alive. And it doesn't seem to have caused any 
issues (at least not in my local PVE servers).

However, the key is quite old and doesn't have current best practice 
size (RSA-4098 AFAIK?). So I'd like to rotate it.

I was hoping that someone with some authoritative knowledge of the 
relevant PVE components would be willing to give me some guidance on the 
process (not generating the key itself, just the PVE integration 
specific bits). Hopefully that can ensure that key rotation causes 
minimal disruptions to users.

Also if there are any specific PVE recommendations/requirements re the 
new GPG keypair to generate, that would also be great.

Thanks in advance.

Regards,
Jeremy Davis
TurnKey Linux

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [pve-devel] [TurnKey Linux] Looking to update our signing key... Advice?

[Thomas Lamprecht]

Hello!

Am 22/11/2023 um 05:50 schrieb Jeremy Davis:
> Apologies in advance if this is not the right place to post this. Please 
> redirect me to the appropriate forum if not. I'm also happy to discuss 
> off list if that is deemed more appropriate.

It's fine here, thanks for reaching out.

> My name is Jeremy and I work with TurnKey Linux.
> 
> As a housekeeping matter, we're looking to update our GPG signing key - 
> that we sign the index file we provide for downloading our LXC templates 
> via the PVE UI (which includes hashes of our templates).

That would be indeed great, we switched to generating a new key for
every new major release quite a bit ago.

> The current key recently expired (caught us a bit unawares). We updated 
> the expiry to keep it alive. And it doesn't seem to have caused any 
> issues (at least not in my local PVE servers).
> 
> However, the key is quite old and doesn't have current best practice 
> size (RSA-4098 AFAIK?). So I'd like to rotate it.

Yes, our release keys use RSA 4096 (not 6 not 8 at the end):

# sq inspect proxmox-release-bookworm.gpg   
proxmox-release-bookworm.gpg: OpenPGP Certificate.

    Fingerprint: F4E136C67CDCE41AE6DE6FC81140AF8F639E0C39
Public-key algo: RSA
Public-key size: 4096 bits
  Creation time: 2022-11-27 13:26:52 UTC
Expiration time: 2032-11-24 13:26:52 UTC (creation time + P3650D)
      Key flags: certification, signing

         UserID: Proxmox Bookworm Release Key <proxmox-release@proxmox.com>

> I was hoping that someone with some authoritative knowledge of the 
> relevant PVE components would be willing to give me some guidance on the 
> process (not generating the key itself, just the PVE integration 
> specific bits). Hopefully that can ensure that key rotation causes 
> minimal disruptions to users.

Currently the public keys we use are tracked in the pve-manager repo,
inside the aplinfo directory:

https://git.proxmox.com/?p=pve-manager.git;a=tree;f=aplinfo;h=9dbe1f31f712bb537168bf11e052d5117c62e1f6;hb=ad1278fae8e6e678219a702eea960c746551c635

The build-system then concatenates all the trusted keys, i.e., our ans
your current (old) one to a joined keyring that we use on checking the
appliance index.

So, you would just need to send us your new public key in a secure
manner and we'd add that key to the keyring.  Secure manner here would
be to have it available on a TLS secured domain of your via HTTP and
send it to us via email with a signature from the old (current) key.

The one question is how you plan the upgrade, i.e., it might be nice to
not have a hard switch between index signed with old to index signed
with new key.

For example, since doing a new GPG key per-release we also use a index
that can be associated with the release, e.g. see:

http://download.proxmox.com/images/

For example, the plain & compressed indexes, and the signature of the
plain one, used for the Proxmox VE 8 series are:

aplinfo-pve-8.dat
aplinfo-pve-8.dat.asc
aplinfo-pve-8.dat.gz


It could be also good for TurnKey to provide the new templates under a
new index so that older installation can still use them.
Even if you want to consciously break support for systems using the old
key, it might be more pleasant to do a phased switch  even then.
Especially as one could test the new index URL and signature without
impacting production systems, you could still drop the signature with
the ancient key in a few weeks or so.

Any how, I'm asking the latter because that might need some extra
adaption in our code, but not much, and if you give us the new URL to
the new index we could integrate that too. But if you want to sent
patches, then we'd also be happy about that, most of the code is also in
pve-manager, in the PVE::APLInfo module (PVE/APLInfo.pm file).

For how to contribute patches to our project see:
https://pve.proxmox.com/wiki/Developer_Documentation

> Also if there are any specific PVE recommendations/requirements re the 
> new GPG keypair to generate, that would also be great.

Nothing technical,  RSA 4096-bit key with a identity (mail email) that
matches your org would be the baseline. Having a expiry of about 10y
could be nice too, but not to hard-feelings there.

cheers,
 Thomas

Re: [pve-devel] [TurnKey Linux] Looking to update our signing key... Advice?

[Jeremy Davis]

[-- Attachment #1.1: Type: text/plain, Size: 4317 bytes --]

Thanks for your quick and comprehensive response! You guys rock! :)

On 22/11/23 19:19, Thomas Lamprecht wrote:
> It's fine here, thanks for reaching out.
> 

:)

>> As a housekeeping matter, we're looking to update our GPG signing key -
>> that we sign the index file we provide for downloading our LXC templates
>> via the PVE UI (which includes hashes of our templates).
> 
> That would be indeed great, we switched to generating a new key for
> every new major release quite a bit ago.
> 

Ok great, thanks.

>> The current key recently expired (caught us a bit unawares). We updated
>> the expiry to keep it alive. And it doesn't seem to have caused any
>> issues (at least not in my local PVE servers).
>>
>> However, the key is quite old and doesn't have current best practice
>> size (RSA-4098 AFAIK?). So I'd like to rotate it.
> 
> Yes, our release keys use RSA 4096 (not 6 not 8 at the end):

Oops. That's what I meant... ;)

> 
> Currently the public keys we use are tracked in the pve-manager repo,
> inside the aplinfo directory:
> 
> https://git.proxmox.com/?p=pve-manager.git;a=tree;f=aplinfo;h=9dbe1f31f712bb537168bf11e052d5117c62e1f6;hb=ad1278fae8e6e678219a702eea960c746551c635
> 
> The build-system then concatenates all the trusted keys, i.e., our ans
> your current (old) one to a joined keyring that we use on checking the
> appliance index.
> 
> So, you would just need to send us your new public key in a secure
> manner and we'd add that key to the keyring.  Secure manner here would
> be to have it available on a TLS secured domain of your via HTTP and
> send it to us via email with a signature from the old (current) key.
> 

Ok, brilliant

> The one question is how you plan the upgrade, i.e., it might be nice to
> not have a hard switch between index signed with old to index signed
> with new key.
> 
> For example, since doing a new GPG key per-release we also use a index
> that can be associated with the release, e.g. see:
> 
> http://download.proxmox.com/images/
> 
> For example, the plain & compressed indexes, and the signature of the
> plain one, used for the Proxmox VE 8 series are:
> 
> aplinfo-pve-8.dat
> aplinfo-pve-8.dat.asc
> aplinfo-pve-8.dat.gz
> 

Thanks for sharing that info. That's really useful.

> 
> It could be also good for TurnKey to provide the new templates under a
> new index so that older installation can still use them.
> Even if you want to consciously break support for systems using the old
> key, it might be more pleasant to do a phased switch  even then.
> Especially as one could test the new index URL and signature without
> impacting production systems, you could still drop the signature with
> the ancient key in a few weeks or so.

That makes tons of sense.

> 
> Any how, I'm asking the latter because that might need some extra
> adaption in our code, but not much, and if you give us the new URL to
> the new index we could integrate that too. But if you want to sent
> patches, then we'd also be happy about that, most of the code is also in
> pve-manager, in the PVE::APLInfo module (PVE/APLInfo.pm file).
> 
> For how to contribute patches to our project see:
> https://pve.proxmox.com/wiki/Developer_Documentation

I'll digest all this a little more and confer with my colleague Alon and 
we'll decide exactly how we approach this.

> 
>> Also if there are any specific PVE recommendations/requirements re the
>> new GPG keypair to generate, that would also be great.
> 
> Nothing technical,  RSA 4096-bit key with a identity (mail email) that
> matches your org would be the baseline. Having a expiry of about 10y
> could be nice too, but not to hard-feelings there.

That sound fair to me.

Thanks again for your comprehensive guidance and advice.

Considering that we're already a bit overwhelmed with a backlog a mile 
long and xmas/new year just around the corner, I'm not sure we'll get 
this done this year or not. But hopefully sooner rather than later.

Regardless, I'll be back at some point with patches and/or further 
questions and/or ... once we have some progress on our end.

Please don't hesitate to reach out if you're wondering where we're up to...

Take care and thanks again.

Cheers,
Jeremy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]