From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 2A3C71FF139 for ; Mon, 26 Jan 2026 04:04:34 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C94A515AE; Mon, 26 Jan 2026 04:04:53 +0100 (CET) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769396655; x=1770001455; h=to:subject:message-id:date:from:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sXlbAMictu4AHn4MDTDdDImmG81bZtjiAjoSaHh75nU=; b=ArQaI0zWUBWNdQGIe14ApCF0lpCn7Le0mHSp9lDgA7RdbF2qbFQIM+F9eAvOG/IUEo KioIq3Wa/sNbKLUhDUo26wsIv23T7dnF3I6P9tWPmMe6uGmjAVFtUDqmwK6MhvBLKhVv aN5mS+f90kfqeMQFewpWWVa3u359j0qw6OKSbRQgkZULurD+NBB6hmXOv412KumPbvka 4wQN7Tuoa4AkdU2UugSpn7EvO4v7+7H1IA5eIRB9flOAPGXVvJ7blk6J3QkOalB4cUup xMQrq0A0rFQXeYl/6XZYVybltXcOoL6TyzUowAdpTYNxgW8Kyn7si0WR5XGxdUAbE+B9 x/Rg== X-Gm-Message-State: AOJu0YwNTq0d+a4MfFf9iCEY5m+t5Yt4H2tuYWku9N6IjHw+r67qkbYb bfPbN6fcz0wICjQR1slo4uKbqElZnTSpHXJ8yeuPjnEMEH2N3dGVFz328M6RpktC X-Gm-Gg: AZuq6aLzUBARkCAzuxMKh+nT1C0f58mCCYvLcP+hF77Brrr0PWPYDzkWgnkRkdWi602 54Xu8SP29w9ncdaAibr0/1QGwDyacflc/01lr4KWPWn8yetQkvC0VzM27FHmh9LKkIvxB2QA6YF Bdj3WCIG1tdTC2heP527HahsLBTzJZEyu0tIkv5Z1NIvP1ajXitFSIuT3Dpz6pYCPDGPB3tiAk6 K3wvyeSA3HG690pruOGvHa+m1aCA0yqAMUXjp899SsuC8JPiv7Chrs1+7rdzGgBSEufc+0uMPee dFjcmK6DQgQIgMIHctAUxxERwr1iALlzK8wqiZIVGsGzxzlgGVdD0vD5TXV8+cDM/fmm1ZyqbOu /2uiyTKKXGDZWufII6KpzzqaL8f4LWhult2Ast14DxszPeEEBrV95quI6pMOx5+cQcQKO4Kc+EL PP9wT/QU/ZvmrQTfyvmod7TxOkvrwU91VmY/8xg17lVHCzY/JI X-Received: by 2002:a17:907:1b10:b0:b87:17df:4d65 with SMTP id a640c23a62f3a-b8d2e6f02c5mr222379966b.51.1769396654432; Sun, 25 Jan 2026 19:04:14 -0800 (PST) X-Received: by 2002:a05:6000:2912:b0:430:f742:fbc7 with SMTP id ffacd0b85a97d-435ca0ecea4mr4594237f8f.14.1769396653742; Sun, 25 Jan 2026 19:04:13 -0800 (PST) MIME-Version: 1.0 From: Thomas Skinner Date: Sun, 25 Jan 2026 21:03:46 -0600 X-Gmail-Original-Message-ID: X-Gm-Features: AZwV_QhTgThKqYB-avMK9Emv3_M7iilQbLbmwvSAYGT2bA6ngxNph29j0fBnRvw Message-ID: To: Proxmox VE development discussion X-SPAM-LEVEL: Spam detection results: 0 AWL 0.007 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy FREEMAIL_FORGED_FROMDOMAIN 0.001 2nd level domains in From and EnvelopeFrom freemail headers are different FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] PVE Auditing System X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Hello! I'm looking to implement an auditing system for PVE to help organizations better understand the actions performed by users via the API. In reference to the conversation in bug #4244, it seems that there's not currently any development on an auditing system. I am with Fabian in that we need to flesh out a design before going too far into development. Below are Thomas Lamprecht's thoughts from a couple years ago with my comments inline. ---- > - [ ] Auditing Framework > - [ ] Explore some auditing projects and possibly some (security) standard > requirements about what could be a good feature set and design, and > about what is a requirement to have to help users with strict > requirements/rules on such things (e.g., gov agencies) A lot of what I've seen have been requirements to be able to adjust/configure success/failure auditing for elevated privileges, access control CRUD (user/group/domain/ACL), and other organizationally defined requirements (a catch-all for subjectively _important_ happening in the application). The logs must be in a standardized format including entities associated with the event and an accurate representation of what/when/where actions occurred. Some of the requirements in the US stem from PCI, NIST (specifically 800r53), and HIPAA. > - [ ] Probably add some (root only) log format on disk that can be > filtered, rotated and allows configuring some guarantees for how long > stuff is saved I would say leave the size, rotation, and retention up to the user. The way that logrotate is already used in the pveproxy logs should already be sufficiently configurable for this. In some fiddling so far, I found it easiest to spit out a least privilege log file for each of pveproxy, pvedaemon, and spiceproxy all in JSON format, which is highly ingestible and extendable later. Ultimately, I think the output should be to a file that some log ingester could read for aggregation on another system if required. > - [ ] Then, one probably wants hook/trace on every config change of guests > and node relevant stuff with an signature like: `($type, $id, > $change-key, $old, $new)` where `$type` and `$change-key` to be > considered API (no arbitrary changes of existing ones) and `$old` and > `$new` are arbitrary (scalar, hash/array ref). A good hook spot looks like in the `rest_handler` function in `PVE/HTTPServer.pm` of the pve-manager package. I'd propose having a call outside of the eval so that it can handle both success and failures in the logs. In my experience so far, it's necessary to log from each proxy endpoint because of the way that validation/permission check occurs: e.g. pvedaemon won't ever see a failure due to a permissions check from pveproxy because the code returns before it's ever proxied over. Putting the logging function here means there is risk of the function potentially not returning a valid response on an audit log failure, but I have seen some requirements where stopping the application when it cannot audit is a requirement (need to make this configurable and safe). Another option that I've thought of is having another daemon run (call it pveauditd), which receives messages from different PVE processes with audit logging information. This requires some interprocess communication, but could potentially reduce any kind of lag because the daemon could buffer messages and it has the I/O wait instead of the calling process. Permissions lockdown on files is easier here, too. Hook would probably still be in the same spot, but audit failures would be handled differently. A nice pro would be that other PVE processes could communicate log messages this way, too. I could use a little guidance/example of existing code if the developers want to go this route. As far as fields go, I think there are some steadfast required fields: - Datetime in UTC - Subject (who performed) - Object (identifier of object modified and its type) - Action (what did the subject attempt to/successfully do to the object) - Status (success/failure) - Node name or IP (where it was performed, could be implied by where the log resides) - Source IP (where the request originated) - Process name and ID (name could be implied in log file name) Some other interesting fields would be: - Before/after whole objects (mentioned specifically in bug #4244): I'd recommend this to be configurable because I think it would require two extra calls: one to retrieve the object before and another after it's modified. - Considerations: sensitive fields (especially credentials) would need to be redacted (character replacement or hashed) - What changed on the object: this would be a breakdown of the differences of the above objects (either fields that changed or the actual changes that were made) - Considerations: requires at least the information above and may be redundant if the above is included - API parameters: this would include the parameters passed to the API endpoint; development side, this is easy to include and would be useful for determining how an object was changed and is less costly than the before/after model. - Considerations: sensitive fields (especially credentials) would need to be redacted (character replacement or hashed) - Event ID: some auditing systems (particularly Microsoft's) use a unique ID for every different type of audit event. - Pros: translations of and custom formatted messages, easily processed/filtered by automated systems, ID mapping could be used across multiple products (e.g. pve and datacenter manager); - Cons: uniqueness of use must be guaranteed, not always human friendly. - Considerations: having this field could potentially eliminate the need for action/object-type fields if each action/object-type combo has its own ID; format of the ID For inclusion of auditing into each API endpoint, I think an addition to the `method_info` construct for each method would be appropriate. Some advanced validations could be done during build-time to ensure uniqueness of action/object type or event ID. > - [ ] Allow one to enable or disable auditing on some/all guests/nodes, and > disable it by default due to cost I completely agree on this one, and I'd argue that it should be built-in from the start. The default could easily be to not audit anything explicitly, which should have minimal impact on runtime. It would be nice to have configuration of the auditing performed in the API and synced across the cluster filesystem. I'd recommend node-level overrides to cluster-level settings. If done in the API, this should have a separate permission or API path. My idea for implementation here is to have a hash built during application startup to load all of the action/object types or event IDs and lookup their status in said hash. The hash could either be updated or reloaded on changes in auditing settings. > - [ ] Add interface to view and filter audit events Consistent format in the log files should make this easier on the dev side. Event IDs or action/object combos could be used to determine translatable message formats to be shown to the user. Interfaces should have either a separate permission or API path (restricting this even to administrators is a common requirement). Cluster-level view would be nice with a reasonable default for how many messages to retrieve per node. An option to retrieve more for each node would be nice. Node-level view could also be adapted, similar to how the VM events are currently shown. > - [ ] Allow to produce notification's for an audit filter I'd use the same logic as for the view/filter interface above. Event IDs might make this easier. ---- I appreciate you all taking the time to review and reply to this thread. An auditing system would be a great addition to the PVE project that makes it even more enterprise friendly. In the implementation that I'm thinking of, this would be an addition to current logging files, not changing any existing formats, which should make it a non-breaking change. -- Thomas Skinner _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel