From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 0E86CBA7F5 for ; Wed, 20 Mar 2024 11:08:43 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E3B9DDD27 for ; Wed, 20 Mar 2024 11:08:12 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 20 Mar 2024 11:08:12 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DE86548AF9 for ; Wed, 20 Mar 2024 11:08:11 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Wed, 20 Mar 2024 11:08:10 +0100 Message-Id: From: "Max Carrara" To: "Lukas Wagner" , "Proxmox VE development discussion" Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: aerc 0.17.0-72-g6a84f1331f1c References: <20240319153250.629369-1-m.carrara@proxmox.com> <20240319153250.629369-4-m.carrara@proxmox.com> In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL 0.026 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [python.org, listvms.py, vm.name] Subject: Re: [pve-devel] [PATCH v1 pve-esxi-import-tools 3/5] listvms: improve typing and add dataclasses to represent dicts X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2024 10:08:43 -0000 On Wed Mar 20, 2024 at 10:38 AM CET, Lukas Wagner wrote: > > > On 2024-03-19 16:32, Max Carrara wrote: > > This commit replaces some of the explicitly imported types from the > > `typing` module with their inbuilt counterparts, e.g. `typing.List` > > becomes `list`. This is supported since Python 3.9 [0]. > >=20 > > Additionally, file paths are now represented as `pathlib.Path` [1], > > which also checks whether the given string is actually a valid path > > when constructed. > >=20 > > Furthermore, the `dict`s with values of mixed types are now > > represented as dataclasses [2] instead, in order to make them more > > type-safe (--> allow for better linting). > >=20 > > Because dataclasses and `pathlib.Path`s are not JSON-serializable by > > default however, a helper function is added, which allows for more > > fine-grained control regarding how those objects are serialized. > >=20 > > [0]: https://docs.python.org/3.9/whatsnew/3.9.html#type-hinting-generic= s-in-standard-collections > > [1]: https://docs.python.org/3.11/library/pathlib.html > > [2]: https://docs.python.org/3.11/library/dataclasses.html > >=20 > > Signed-off-by: Max Carrara > > --- > > listvms.py | 99 ++++++++++++++++++++++++++++++++++++++++-------------- > > 1 file changed, 73 insertions(+), 26 deletions(-) > >=20 > > diff --git a/listvms.py b/listvms.py > > index cc3209f..cdea95a 100755 > > --- a/listvms.py > > +++ b/listvms.py > > @@ -1,26 +1,69 @@ > > #!/usr/bin/python3 > > =20 > > +import dataclasses > > import json > > import ssl > > import sys > > =20 > > -from typing import List, Dict, Optional, Tuple > > +from dataclasses import dataclass > > +from pathlib import Path > > +from typing import Any > > =20 > > from pyVim.connect import SmartConnect, Disconnect > > from pyVmomi import vim > > =20 > > =20 > > -def get_datacenter_of_vm(vm: vim.VirtualMachine) -> Optional[vim.Datac= enter]: > > +@dataclass > > +class VmVmxInfo: > > + datastore: str > > + path: Path > > + checksum: str > > + > > + > > +@dataclass > > +class VmDiskInfo: > > + datastore: str > > + path: Path > > + capacity: int > > + > > + > > +@dataclass > > +class VmInfo: > > + config: VmVmxInfo > > + disks: list[VmDiskInfo] > > + power: str > > + > > + > > +def json_dump_helper(obj: Any) -> Any: > > + """Converts otherwise unserializable objects to types that can be > > + serialized as JSON. > > + > > + Raises: > > + TypeError: If the conversion of the object is not supported. > > + """ > > + if dataclasses.is_dataclass(obj): > > + return dataclasses.asdict(obj) > > + > > + match obj: > > + case Path(): > > + return str(obj) > > + > > + raise TypeError( > > + f"Can't make object of type {type(obj)} JSON-serializable: {re= pr(obj)}" > > + ) > > + > > + > > +def get_datacenter_of_vm(vm: vim.VirtualMachine) -> vim.Datacenter | N= one: > > """Find the Datacenter object a VM belongs to.""" > > current =3D vm.parent > > while current: > > if isinstance(current, vim.Datacenter): > > return current > > current =3D current.parent > > - return None > > + return > > mypy does not seem to like this change :) > > listvms.py:157: error: Return value expected [return-value] Interesting! I got `rufflsp`, which doesn't seem to warn me here. Thanks for pointing that out, will add that in v2. > > > > =20 > > =20 > > -def list_vms(service_instance: vim.ServiceInstance) -> List[vim.Virtua= lMachine]: > > +def list_vms(service_instance: vim.ServiceInstance) -> list[vim.Virtua= lMachine]: > > """List all VMs on the ESXi/vCenter server.""" > > content =3D service_instance.content > > vm_view =3D content.viewManager.CreateContainerView( > > @@ -32,39 +75,36 @@ def list_vms(service_instance: vim.ServiceInstance)= -> List[vim.VirtualMachine]: > > vm_view.Destroy() > > return vms > > =20 > > -def parse_file_path(path): > > +def parse_file_path(path) -> tuple[str, Path]: > > """Parse a path of the form '[datastore] file/path'""" > > datastore_name, relative_path =3D path.split('] ', 1) > > datastore_name =3D datastore_name.strip('[') > > - return (datastore_name, relative_path) > > + return (datastore_name, Path(relative_path)) > > =20 > > -def get_vm_vmx_info(vm: vim.VirtualMachine) -> Dict[str, str]: > > +def get_vm_vmx_info(vm: vim.VirtualMachine) -> VmVmxInfo: > > """Extract VMX file path and checksum from a VM object.""" > > datastore_name, relative_vmx_path =3D parse_file_path(vm.config.fi= les.vmPathName) > > - return { > > - 'datastore': datastore_name, > > - 'path': relative_vmx_path, > > - 'checksum': vm.config.vmxConfigChecksum.hex() if vm.config.vmx= ConfigChecksum else 'N/A' > > - } > > =20 > > -def get_vm_disk_info(vm: vim.VirtualMachine) -> Dict[str, int]: > > + return VmVmxInfo( > > + datastore=3Ddatastore_name, > > + path=3Drelative_vmx_path, > > + checksum=3Dvm.config.vmxConfigChecksum.hex() if vm.config.vmxC= onfigChecksum else 'N/A' > > + ) > > + > > +def get_vm_disk_info(vm: vim.VirtualMachine) -> list[VmDiskInfo]: > > disks =3D [] > > for device in vm.config.hardware.device: > > - if type(device).__name__ =3D=3D 'vim.vm.device.VirtualDisk': > > + if isinstance(device, vim.vm.device.VirtualDisk): > > try: > > (datastore, path) =3D parse_file_path(device.backing.f= ileName) > > capacity =3D device.capacityInBytes > > - disks.append({ > > - 'datastore': datastore, > > - 'path': path, > > - 'capacity': capacity, > > - }) > > + disks.append(VmDiskInfo(datastore, path, capacity)) > > except Exception as err: > > # if we can't figure out the disk stuff that's fine... > > print("failed to get disk information for esxi vm: ", = err, file=3Dsys.stderr) > > return disks > > =20 > > -def get_all_datacenters(service_instance: vim.ServiceInstance) -> List= [vim.Datacenter]: > > +def get_all_datacenters(service_instance: vim.ServiceInstance) -> list= [vim.Datacenter]: > > """Retrieve all datacenters from the ESXi/vCenter server.""" > > content =3D service_instance.content > > dc_view =3D content.viewManager.CreateContainerView(content.rootFo= lder, [vim.Datacenter], True) > > @@ -107,18 +147,25 @@ def main(): > > name =3D 'vm ' + vm.name > > try: > > dc =3D get_datacenter_of_vm(vm) > > - vm_info =3D { > > - 'config': get_vm_vmx_info(vm), > > - 'disks': get_vm_disk_info(vm), > > - 'power': vm.runtime.powerState, > > - } > > + if dc is None: > > + print( > > + f"Failed to get datacenter for {name}", > > + file=3Dsys.stderr > > + ) > > + > > + vm_info =3D VmInfo( > > + config=3Dget_vm_vmx_info(vm), > > + disks=3Dget_vm_disk_info(vm), > > + power=3Dvm.runtime.powerState, > > + ) > > + > > datastore_info =3D {ds.name: ds.url for ds in vm.confi= g.datastoreUrl} > > data.setdefault(dc.name, {}).setdefault('vms', {})[vm.= name] =3D vm_info > > data.setdefault(dc.name, {}).setdefault('datastores', = {}).update(datastore_info) > > except Exception as err: > > print("failed to get info for", name, ':', err, file= =3Dsys.stderr) > > =20 > > - print(json.dumps(data, indent=3D2)) > > + print(json.dumps(data, indent=3D2, default=3Djson_dump_helper)= ) > > I guess this could be=20 > > `json.dump(data, sys.stdout, ...)`=20 > > That'd not add a newline in the end, not sure if that is important here. > But it really does not matter that much, I guess. ;) I guess it does not in this case, but since you mentioned it, I'll just include it in v2 as well. I don't think the newline is important, but will test nevertheless. Thanks for the tip! > > > finally: > > Disconnect(si) > > =20