[RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

Every daemon that cares about /etc/pve learns about changes by calling
cfs_update on each loop iteration and comparing the version vector it
gets over the libqb IPC. inotify cannot replace that, since remote
changes arrive through corosync and never touch the local VFS, so
pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
timers and re-read what usually has not changed.

This series adds a push path. pmxcfs gets a Unix stream socket at
/run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
daemon as a static library with a small C ABI. A client subscribes with
named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
receives one JSON line per matching mutation, carrying the memdb version
as sequence number, the event type, the path and the values the
placeholders captured. Connections authorized by group membership never
see private paths, as on the IPC and FUSE side.

The daemon keeps the last mutations in a fixed size ring and each
connection a cursor into it, so the mutating thread only appends and
never waits for a client. A slow client is caught up from the ring, one
that fell off it gets a single resync event, and a reconnecting client
resumes at the last sequence number it saw, also across a restart of
pmxcfs when nothing changed meanwhile. A node that takes the whole
state from the cluster after a membership change hands its clients a
resync event, since no sequence of mutations describes that. A panic in
the Rust code only disables the notifier for the rest of the daemon's
life. A sequence number identifies a state within one line of history
only. A client that missed a resync while disconnected and returns at
the same version after a pmxcfs restart resumes as if nothing changed.
Carrying the root entry's mtime next to the version would close that.

On top of the socket, pve-cluster ships a Perl client with reconnect
and resync handling and a hook registry, where a hook is registered
like an API method with a path template whose placeholders become the
parameters of a run. pve-manager gets the runner that executes those
runs in children of a listener hosted by pvescheduler, one run per hook
and parameter set in flight and later events for the same set collapsed
into a single rerun, so a config update through the API costs one extra
run rather than one per save. The listener records the position it has
processed up to under /run and resumes there after a reload. No hook
ships in this series. A consumer registers one as the example below
shows.

pve-manager depends on the pve-cluster packages of this series for the
new modules and the socket, at build time as well, since its tests run
through the check target.

Example usage:

    package PVE::Network::Hooks;

    use base qw(PVE::Cluster::Hooks);

    __PACKAGE__->register_hook({
        name => 'guest-firewall',
        path => 'firewall/{vmid}.fw',
        code => sub {
            my ($param, $event) = @_;

            my $vmids = $event->{type} eq 'resync'
                ? [] # every guest with a firewall config
                : [ $param->{vmid} ];

            for my $vmid ($vmids->@*) {
                # reload and apply the firewall for guest $vmid
            }
        },
    });

    1;


pve-cluster:

Hannes Laimer (8):
  buildsys: add rust workspace under src/rust
  rust: notify: add change notification socket server
  rust: ffi: add C ABI staticlib for pmxcfs
  pmxcfs: memdb: add change notification hook
  buildsys: link pmxcfs against the rust notify staticlib
  pmxcfs: notify: emit change events over the notification socket
  cfs: add perl client for the change notification socket
  cfs: add hook registry for change notification consumers

 .gitignore                                  |   2 +
 Makefile                                    |   1 +
 debian/control                              |  16 +-
 debian/pve-cluster.install                  |   2 +
 debian/rules                                |  16 +
 src/Makefile                                |   2 +-
 src/PVE/Cluster/Hooks.pm                    | 134 ++++
 src/PVE/Cluster/Makefile                    |   2 +-
 src/PVE/Cluster/Watch.pm                    | 371 ++++++++++
 src/pmxcfs/Makefile                         |  15 +-
 src/pmxcfs/cfs-utils.h                      |   4 +
 src/pmxcfs/database.c                       |   2 +
 src/pmxcfs/memdb.c                          |  24 +
 src/pmxcfs/memdb.h                          |  11 +
 src/pmxcfs/pmxcfs.c                         |  25 +
 src/rust/.cargo/config.toml                 |   8 +
 src/rust/Cargo.toml                         |  23 +
 src/rust/Makefile                           |  18 +
 src/rust/pmxcfs-ffi/Cargo.toml              |  17 +
 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h |  29 +
 src/rust/pmxcfs-ffi/src/lib.rs              | 309 ++++++++
 src/rust/pmxcfs-notify/Cargo.toml           |  16 +
 src/rust/pmxcfs-notify/src/conn.rs          | 317 ++++++++
 src/rust/pmxcfs-notify/src/lib.rs           |  43 ++
 src/rust/pmxcfs-notify/src/protocol.rs      | 237 ++++++
 src/rust/pmxcfs-notify/src/registry.rs      | 405 ++++++++++
 src/rust/pmxcfs-notify/src/ring.rs          | 171 +++++
 src/rust/pmxcfs-notify/src/server.rs        | 780 ++++++++++++++++++++
 src/rust/pmxcfs-notify/src/template.rs      | 254 +++++++
 src/rust/rustfmt.toml                       |   2 +
 src/test/Makefile                           |  10 +-
 src/test/hooks_test.pl                      | 219 ++++++
 src/test/watch_client_test.pl               | 294 ++++++++
 33 files changed, 3773 insertions(+), 6 deletions(-)
 create mode 100644 src/PVE/Cluster/Hooks.pm
 create mode 100644 src/PVE/Cluster/Watch.pm
 create mode 100644 src/rust/.cargo/config.toml
 create mode 100644 src/rust/Cargo.toml
 create mode 100644 src/rust/Makefile
 create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml
 create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h
 create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs
 create mode 100644 src/rust/pmxcfs-notify/Cargo.toml
 create mode 100644 src/rust/pmxcfs-notify/src/conn.rs
 create mode 100644 src/rust/pmxcfs-notify/src/lib.rs
 create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs
 create mode 100644 src/rust/pmxcfs-notify/src/registry.rs
 create mode 100644 src/rust/pmxcfs-notify/src/ring.rs
 create mode 100644 src/rust/pmxcfs-notify/src/server.rs
 create mode 100644 src/rust/pmxcfs-notify/src/template.rs
 create mode 100644 src/rust/rustfmt.toml
 create mode 100644 src/test/hooks_test.pl
 create mode 100644 src/test/watch_client_test.pl


pve-manager:

Hannes Laimer (2):
  hooks: add runner executing cluster change hooks in children
  pvescheduler: run cluster change hooks from a listener child

 PVE/HookRunner.pm           | 368 ++++++++++++++++++++++++++++++++
 PVE/Makefile                |   1 +
 PVE/Service/pvescheduler.pm |  26 ++-
 test/Makefile               |   6 +-
 test/hook_runner_test.pl    | 405 ++++++++++++++++++++++++++++++++++++
 5 files changed, 801 insertions(+), 5 deletions(-)
 create mode 100644 PVE/HookRunner.pm
 create mode 100755 test/hook_runner_test.pl


Summary over all repositories:
  38 files changed, 4574 insertions(+), 11 deletions(-)

-- 
Generated by murpp 0.12.0