From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D6D571FF0E1 for ; Mon, 16 Nov 2026 02:25:58 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E17E2218D7; Mon, 16 Nov 2026 02:24:39 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH v5 cluster,storage,docs,manager 00/36] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Date: Mon, 16 Nov 2026 02:23:09 +0100 Message-ID: <20261116012355.3187417-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794792241127 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.060 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QCJEHKVN6RA76LIKNRGAXVFZAQAV6DP2 X-Message-ID-Hash: QCJEHKVN6RA76LIKNRGAXVFZAQAV6DP2 X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This is v5 of the combined and reduced series of the cluster-wide multipath work [0][1] and Dietmar's RFC for guided remote storage setup and SAN visibility [2]. Most changes since v4 [3] come from feedback from Blockbridge – a third-party storage plugin vendor. Their plugin attaches one LUN per guest disk, so a plain upgrade hits nodes with several hundred multipath maps that are not ours: - The iSCSI storage option is named 'managed-multipath' now. Their plugin defines a 'multipath' property, and a property that two plugins define stops PVE::Storage from loading. Per-plugin property isolation would be the real answer, but a single plugin that opts in switches the storage API to a schema in which updates need the type, so 'pvesm set' and the edit dialogs fail for all storages of that node. We (I and/or Max, Wolfgang) will look into completing that separately. - A node reports the maps that Proxmox VE manages or has a storage on, and counts the rest. Before, every map got a row, and beyond about 185 of them the report no longer fit into the 32 KiB of a value in the node key-value store, which cost the node the state of our LUNs as well. - The LUN scan no longer queries udev for the paths of a map. With 1000 additional maps on a node that was one udevadm call for each of 2000 path devices and 13 seconds per scan, for views that poll it every few seconds. It is 2 seconds now. - Writing to a LUN, for a new volume group or a wipe, is refused while a process of the node has it open. A guest that got the LUN from a plugin or by its device path is neither a holder nor a storage reference, and such a node lists hundreds of them. v3 had a broader census of device users, which I dropped in v4 with the rest of that machinery; this is the cheap part of it. The alleged long multipathd query time on every status pass I could not reproduce as a problem, as it took 30 ms for those 1000 maps on a small nested node and the status loop stayed in its normal cycle. Overview of what an admin gets with this: - Datacenter > Storage > SAN, a new entry below Storage, shows the multipath state of every node and of the LUNs that Proxmox VE manages or has a storage on, and edits the multipath entries and array-specific overrides. I explicitly chose this over the client/plugin side of Storage, as we do not have e.g. Ceph or ZFS/LVM/... creation in there either. We might want to rethink and consolidate our storage interface vs. management story in the future, but IMO a partial switch now is not worth it, that would add even more potential for confusion than the new SAN view. - Its Add Storage menu guides through shared LVM on a LUN: for iSCSI it connects or reuses a target, with the initiator names of all nodes at hand, for Fibre Channel it takes a LUN the nodes already see. Both set up multipath for the LUN and end with its state on every node of the new storage. - Datacenter > Storage > Add keeps its dialogs. The iSCSI one gets a Multipath option and the LVM one enables Shared for a base storage, so the same works there in two steps. Both name the SAN view in a note whose link opens it, as admins coming from other platforms look for their SAN there first. - Node > SAN lists the LUNs of a node with their paths and usage, can wipe a LUN that no storage uses, and shows the iSCSI sessions. The rules the series follows since v4, in short: nodes add WWIDs and maps but never remove them, a write is checked where it happens, lists do not probe devices, there are no new iSCSI operations, the guided setup makes the API calls an admin would make by hand, and multipath management is turned on by the admin. The v4 cover [3] has the reasons and what each of them costs. Two things an admin will likely miss: there is no notification when a LUN loses path redundancy, the state only shows on the SAN pages and in the API. A node with fewer paths than its iSCSI portals or the expected paths of the LUN's entry at least shows as degraded there, and a LUN that only has one path reads "OK (1 path)" instead of optimal. And Fibre Channel starts at a LUN the nodes already see, the web UI neither lists the WWPNs of the nodes nor rescans their HBAs; the chapter gives the commands for both, and the setup points to the rescan. How it works: - /etc/pve/multipath.cfg holds explicit entries per WWID (nodes, alias, no-path-retry, expected paths), /etc/pve/multipath-overrides.conf array settings in multipath.conf syntax; patch 01 registers both. An iSCSI storage with the new 'managed-multipath' option supplies its LUNs without entries, a Fibre Channel LUN gets an explicit entry unless it has a map already. - pvestatd calls one function of pve-storage per pass. It adds the WWIDs the node needs to /etc/multipath/wwids, writes the drop-ins /etc/multipath/conf.d/pve-*.conf (entries, overrides and one generated default, auto_resize grow_only) and reloads multipathd when something changed. The same call publishes the state of the maps that the node manages or that hold the volume group of a storage, and the number of its other maps, through the node key-value store (key 'multipath'). - /cluster/multipath/status aggregates these reports, derives 'missing' from the configuration and names the storage that requests each map; /cluster/multipath/{wwid,overrides} edit the two files. - The volumes of an iSCSI storage with the option end in dm-uuid-mpath- instead of the name of one path, so an LVM storage uses them through the existing 'base' mechanism. - /nodes/{node}/scan/san-luns and 'pvesm scan san-luns' return one row per LUN with a WWID, /nodes/{node}/iscsi/{status,target} the read-only session view. - Volume group creation and wipe run under a cluster lock per LUN (cfs_lock_domain) and do not hold the storage configuration lock while they write. Adding an LVM storage with a new base volume still does, as before. Both refuse a LUN that a process of the node has open. - multipath-tools becomes a Recommends of libpve-storage-perl. Nodes without it report that; plain iSCSI and LVM are unaffected. Patches: - cluster 01: register the two configuration files - storage 02-22: helpers for the disk and iSCSI session inventory (mostly Dietmar's), the multipath library and configuration, the iSCSI option, the per-node apply and status with their API, the LUN inventory, the read-only iSCSI status, volume groups on LUNs and the wipe, in that order - docs 23: the multipath chapter, rewritten around tasks - manager 24-36: the pvestatd call, report and API mounts, then the views, the two dialog changes, the two guided setups and the note in the add dialogs Changes since v4 [3]: - renamed the iSCSI storage option 'multipath' to 'managed-multipath' in storage.cfg, API and CLI; the web UI keeps the label Multipath - the node report has rows for maps with an explicit entry for the node, of an iSCSI storage with the option, that are the base volume of an LVM storage or that hold the volume group of a storage, and the new 'other-maps' count for the rest. The SAN panel shows that count in a column of the node list once a node has such maps. The volume groups come from LVM's hint file; a node that leaves maps out without one says so, and counts as unknown instead of missing for a LUN that only a volume group relates to a storage there. A report over the size limit keeps the initiator name, and the chapter names the limit - the LUN scan leaves the paths of maps out before the disk enumeration - volume group creation on a LUN and the wipe check for processes that have the LUN open. That reads /proc//fd, so it does not see an open through another device node, for example one of a container - both also stop if a device name they use no longer means the LUN they hold open, the wipe also for the partitions it clears and for its exclusive descriptor. Creating a volume group checked and initialized the kernel name it was called with, but cleared the first sector through the by-id name, and a map can get another name while the lock is awaited - a volume group on a partition of a multipath map counts for the map - the Fibre Channel setup offers SCSI LUNs only. It also listed NVMe/FC namespaces, which the cluster-wide status does not cover, so its closing window stayed empty for them - the LUN step only asks to wipe data that is no longer needed, and the chapter says how to keep a volume group that a LUN already has - from an automated review of v4: the multipath values of the LUN scan are described in the API, the portal label is translatable, imports are grouped, the new iSCSI topology module uses signatures, WWID and map UUID patterns use ASCII digit classes, three statement modifiers after multi-line calls became blocks, some comments and commit messages got reworded - rebased onto current master in pve-storage, pve-manager and pve-cluster When applying, versioned dependencies have to be set: pve-manager on libpve-storage-perl, and libpve-storage-perl (also as build dependency) on the libpve-cluster-perl with patch 01, as PVE::Storage registers the two files when it is loaded. If the bound for LVM status commands [4] gets applied too, whichever is applied second has to pass the status cache to the two lvm_vgs() calls of the base volume activation, they can block a status pass otherwise. The per-node target and portal series [5][6] are independent of this one. It takes the LUNs of an iSCSI storage from the sessions of its one target, so if a storage can have several targets per node, whichever is applied second has to cover that. For v4 I tested this on a six-node nested cluster attached to a NetApp AFF A20 with three iSCSI portals and to Linux LIO targets with two: the guided iSCSI setup with a new and with a reused connection storage, the same through the plain dialogs and from the CLI as documented, a LUN that one node is not allowed to see and gets access later, the Fibre Channel flow through the web UI, also as an administrator other than root@pam, a guest on the resulting storages (snapshot, rollback, online resize, live migration, one path cut under I/O), growing a LUN, wipe and its refusals, one path and all paths down with queued I/O, a stopped multipathd, a node reboot and the documented LUN removal. The lab has no FC hardware, so I emulated it with software FCoE in VN2VN mode and an LIO tcm_fc target on two fabrics: the nodes get real FC host and remote port objects, and the documented WWPN listing, rescan and LUN growth work there as written. The guided Fibre Channel setup ran on such a LUN, with one node on one fabric only and one node that saw the LUN only after the rescan. The setup records the paths it sees in the LUN's entry, so the node on one fabric shows as degraded, and a fabric cut stays degraded also after the kernel removed the lost paths (66 seconds dev_loss_tmo of the LIO profile). Real HBA drivers were not tested. Most of that ran with the NVMe over Fabrics series on top and with the LVM status patch [4] applied, the series alone ran on one node of that cluster for the guided setups, the CLI procedure and the views. Without [4] a stuck LVM command blocks pvestatd as on master. For v5 I upgraded that cluster from the v4 build, renamed the option in its storage.cfg and checked that the storages on the NetApp and LIO LUNs stay active with their guests running. I emulated the plugin vendor's case with scsi_debug, two hosts with the same LUN identities, so 1000 additional maps with two paths each on one node: report, status loop and scan before and after the changes, both SAN views after them. The wipe refusal for an open LUN ran on such a map, with a process holding it open, the one of volume group creation on a LIO LUN, followed by the creation without the process. A hand-made map under a shared LVM storage on two nodes covered the case of a node without LVM hints and the one of a volume group on a partition of the map. The guided iSCSI setup ran once more on a NetApp LUN, from wiping its old volume group to the documented removal. The rest of the list above was not repeated for v5, although the changes touch what those flows use: the LUN scan, the status report, the write checks and the LUN selection of the Fibre Channel setup. [0]: https://lore.proxmox.com/pve-devel/20260626121000.2095591-1-t.lamprecht@proxmox.com/ [1]: https://lore.proxmox.com/pve-devel/20260703124707.1172980-2-t.lamprecht@proxmox.com/ [2]: https://lore.proxmox.com/pve-devel/20260731102156.3947857-1-dietmar@proxmox.com/ [3]: https://lore.proxmox.com/pve-devel/20261113030146.1559030-1-t.lamprecht@proxmox.com/ [4]: https://lore.proxmox.com/pve-devel/20261111015217.941220-1-t.lamprecht@proxmox.com/ [5]: https://lore.proxmox.com/pve-devel/20260430173220.441001-1-m.limbeck@proxmox.com/ [6]: https://lore.proxmox.com/pve-devel/20260804090819.2136483-1-dietmar@proxmox.com/ pve-cluster: Thomas Lamprecht (1): cfs: observe the multipath configuration files src/PVE/Cluster.pm | 2 ++ src/pmxcfs/status.c | 2 ++ 2 files changed, 4 insertions(+) pve-storage: Dietmar Maurer (8): diskmanage: collect disk transport type from lsblk diskmanage: qualify NVMe over fabrics transport diskmanage: optionally include iSCSI session devices in disk enumeration diskmanage: collect native NVMe multipath path state iscsi: factor out session device map from device list diskmanage: add helper to list multipath devices pvesm: expose the SAN LUN inventory api: add node-level iSCSI status Thomas Lamprecht (13): multipath: add a library around multipathd multipath: add a cluster-wide configuration iscsi: add an option to manage storage LUNs with multipath lvm: activate a missing base volume before using the volume group multipath: apply the configuration on every node multipath: publish per-node status api: add endpoints for the cluster-wide multipath configuration api: multipath: add the cluster-wide health status api: scan: add a SAN LUN inventory disks: lvm: drop unused imports disks: lvm: allow volume groups on SAN LUNs and multipath maps disks: lvm: refuse to destroy the volume group of a shared storage disks: allow wiping a SAN LUN that no storage uses debian/control | 3 +- src/PVE/API2/Disks.pm | 36 +- src/PVE/API2/Disks/LVM.pm | 64 ++- src/PVE/API2/ISCSI.pm | 166 +++++++ src/PVE/API2/Makefile | 2 + src/PVE/API2/Multipath.pm | 454 ++++++++++++++++++ src/PVE/API2/Storage/Scan.pm | 202 ++++++++ src/PVE/CLI/pvesm.pm | 14 + src/PVE/Diskmanage.pm | 277 +++++++++-- src/PVE/ISCSI/Topology.pm | 86 ++++ src/PVE/Makefile | 6 + src/PVE/Multipath.pm | 213 ++++++++ src/PVE/Multipath/Config.pm | 205 ++++++++ src/PVE/Multipath/Generator.pm | 114 +++++ src/PVE/Multipath/Identity.pm | 44 ++ src/PVE/Multipath/Status.pm | 220 +++++++++ src/PVE/Storage/ISCSIPlugin.pm | 212 ++++---- src/PVE/Storage/LVMPlugin.pm | 146 +++++- src/PVE/Storage/Makefile | 3 +- src/PVE/Storage/SAN.pm | 352 ++++++++++++++ src/test/Makefile | 34 +- src/test/disk_api_test.pm | 256 ++++++++++ src/test/disk_tests/iscsi/disklist | 1 + .../disk_tests/iscsi/disklist_expected.json | 19 + src/test/disk_tests/iscsi/lsblk | 5 + src/test/disk_tests/iscsi/sda/device/model | 1 + src/test/disk_tests/iscsi/sda/device/vendor | 1 + .../disk_tests/iscsi/sda/queue/rotational | 1 + src/test/disk_tests/iscsi/sda/size | 1 + src/test/disk_tests/iscsi/sda_udevadm | 21 + src/test/disk_tests/multipath/disklist | 11 + .../multipath/disklist_expected.json | 107 +++++ src/test/disk_tests/multipath/dm-0/dm/name | 1 + src/test/disk_tests/multipath/dm-0/dm/uuid | 1 + src/test/disk_tests/multipath/dm-0/size | 1 + src/test/disk_tests/multipath/dm-0_slaves | 4 + src/test/disk_tests/multipath/dm-1/dm/name | 1 + src/test/disk_tests/multipath/dm-1/dm/uuid | 1 + src/test/disk_tests/multipath/dm-1/size | 1 + src/test/disk_tests/multipath/dm-1_slaves | 4 + src/test/disk_tests/multipath/dm-2/dm/name | 1 + src/test/disk_tests/multipath/dm-2/dm/uuid | 1 + src/test/disk_tests/multipath/dm-2/size | 1 + src/test/disk_tests/multipath/dm-2_slaves | 0 src/test/disk_tests/multipath/dm-3/dm/name | 1 + src/test/disk_tests/multipath/dm-3/dm/uuid | 1 + src/test/disk_tests/multipath/dm-3/size | 1 + src/test/disk_tests/multipath/dm-3_slaves | 0 src/test/disk_tests/multipath/dm-4/dm/name | 1 + src/test/disk_tests/multipath/dm-4/dm/uuid | 1 + src/test/disk_tests/multipath/dm-4/size | 1 + src/test/disk_tests/multipath/dm-4_slaves | 0 src/test/disk_tests/multipath/lsblk | 77 +++ src/test/disk_tests/multipath/lvs | 0 src/test/disk_tests/multipath/mounts | 1 + .../multipath/multipath_expected.json | 59 +++ src/test/disk_tests/multipath/multipathd | 124 +++++ .../multipath/multipathd_status_expected.json | 40 ++ .../nvme-subsystem/nvme-subsys0/nvme3/address | 1 + .../nvme-subsys0/nvme3/nvme2c3n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme3/state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/address | 1 + .../nvme-subsys0/nvme4/nvme2c4n1/ana_state | 1 + .../nvme-subsystem/nvme-subsys0/nvme4/state | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/address | 1 + .../nvme-subsystem/nvme-subsys1/nvme0/state | 1 + .../disk_tests/multipath/nvme8n1/device/model | 1 + .../multipath/nvme8n1/device/transport | 1 + .../multipath/nvme8n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme8n1/size | 1 + src/test/disk_tests/multipath/nvme8n1_udevadm | 14 + .../disk_tests/multipath/nvme9n1/device/model | 1 + .../multipath/nvme9n1/device/transport | 1 + .../multipath/nvme9n1/device/vendor | 1 + .../multipath/nvme9n1/queue/rotational | 1 + src/test/disk_tests/multipath/nvme9n1/size | 1 + src/test/disk_tests/multipath/nvme9n1_udevadm | 12 + .../multipath/nvme_path_status_expected.json | 19 + src/test/disk_tests/multipath/partlist | 0 src/test/disk_tests/multipath/pvs | 1 + .../disk_tests/multipath/sda/device/model | 1 + .../disk_tests/multipath/sda/device/vendor | 1 + .../disk_tests/multipath/sda/holders/holder | 1 + .../disk_tests/multipath/sda/queue/rotational | 1 + src/test/disk_tests/multipath/sda/size | 1 + src/test/disk_tests/multipath/sda_udevadm | 18 + .../disk_tests/multipath/sdb/device/model | 1 + .../disk_tests/multipath/sdb/device/vendor | 1 + .../disk_tests/multipath/sdb/holders/holder | 1 + .../disk_tests/multipath/sdb/queue/rotational | 1 + src/test/disk_tests/multipath/sdb/size | 1 + src/test/disk_tests/multipath/sdb_udevadm | 18 + .../disk_tests/multipath/sdc/device/model | 1 + .../disk_tests/multipath/sdc/device/vendor | 1 + .../disk_tests/multipath/sdc/holders/holder | 1 + .../disk_tests/multipath/sdc/queue/rotational | 1 + src/test/disk_tests/multipath/sdc/size | 1 + src/test/disk_tests/multipath/sdc_udevadm | 18 + .../disk_tests/multipath/sdd/device/model | 1 + .../disk_tests/multipath/sdd/device/vendor | 1 + .../disk_tests/multipath/sdd/holders/holder | 1 + .../disk_tests/multipath/sdd/queue/rotational | 1 + src/test/disk_tests/multipath/sdd/size | 1 + src/test/disk_tests/multipath/sdd_udevadm | 18 + src/test/disk_tests/multipath/zpool | 1 + .../disk_tests/sas/disklist_expected.json | 1 + src/test/disk_tests/sas/lsblk | 5 + src/test/disklist_test.pm | 107 ++++- src/test/multipath-three-states.json | 52 ++ src/test/run_disk_api_tests.pl | 11 + src/test/run_iscsi_activation_tests.pl | 86 ++++ src/test/run_lun_identity_tests.pl | 48 ++ src/test/run_multipath_api_tests.pl | 444 +++++++++++++++++ src/test/run_multipath_tests.pl | 169 +++++++ src/test/run_san_api_tests.pl | 94 ++++ src/test/run_san_library_tests.pl | 135 ++++++ src/test/run_san_lun_scan_tests.pl | 197 ++++++++ src/test/run_san_nvme_tests.pl | 108 +++++ src/test/run_san_taint_tests.pl | 60 +++ src/test/run_san_wipe_tests.pl | 190 ++++++++ src/test/run_test_lvm_base_activation.pl | 118 +++++ src/test/run_test_lvm_base_conflicts.pl | 233 ++++++--- src/test/run_test_lvm_create.pl | 41 ++ 123 files changed, 5622 insertions(+), 234 deletions(-) pve-docs: Thomas Lamprecht (1): storage: document the cluster-wide multipath management pmxcfs.adoc | 2 + pve-storage-iscsi.adoc | 11 ++ pve-storage-iscsidirect.adoc | 2 +- pve-storage-lvm.adoc | 3 + pve-storage-multipath.adoc | 282 +++++++++++++++++++++++++++++++++++ pvesm.adoc | 18 ++- 6 files changed, 312 insertions(+), 6 deletions(-) pve-manager: Dietmar Maurer (2): api: nodes: expose the iSCSI status ui: node: san: show iSCSI targets and sessions Thomas Lamprecht (11): pvestatd: apply the multipath configuration and report the node status report: add the managed multipath configuration api: cluster: add the multipath endpoints ui: dc: add a SAN view ui: storage: iscsi: add the multipath option ui: lvm: enable shared when a base storage is selected ui: node: add SAN LUNs panel ui: node: san: wipe a LUN that no storage uses ui: dc: san: add a guided setup for shared LVM on iSCSI ui: dc: san: add a guided setup for shared LVM on Fibre Channel ui: storage: point the iSCSI and LVM add dialogs to the SAN view PVE/API2/Cluster.pm | 7 + PVE/API2/Nodes.pm | 7 + PVE/Report.pm | 3 + PVE/Service/pvestatd.pm | 7 + www/manager6/Makefile | 12 + www/manager6/SANUtils.js | 161 ++++++++ www/manager6/Utils.js | 14 + www/manager6/dc/Config.js | 23 +- www/manager6/dc/MultipathWwidEdit.js | 81 ++++ www/manager6/dc/SAN.js | 347 ++++++++++++++++++ www/manager6/node/Config.js | 24 ++ www/manager6/node/ISCSITargets.js | 122 ++++++ www/manager6/node/SANLuns.js | 184 ++++++++++ www/manager6/storage/Base.js | 5 + www/manager6/storage/IScsiEdit.js | 46 +++ www/manager6/storage/LVMEdit.js | 6 + www/manager6/storage/wizard/FibreChannel.js | 162 ++++++++ www/manager6/storage/wizard/IScsi.js | 272 ++++++++++++++ www/manager6/storage/wizard/LunSelect.js | 179 +++++++++ www/manager6/storage/wizard/NodeIdentities.js | 83 +++++ www/manager6/storage/wizard/Wizard.js | 258 +++++++++++++ www/manager6/window/SANLunState.js | 115 ++++++ www/manager6/window/SANLunWipe.js | 53 +++ 23 files changed, 2165 insertions(+), 6 deletions(-)