[PATCH cluster 00/12] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

pmxcfs gets a Unix stream socket at /run/pve-cluster/pmxcfs.sock. It
pushes one JSON line per change to /etc/pve to every client whose path
template matches the changed path. A client subscribes with named
templates such as nodes/{node}/qemu-server/{vmid}.conf. An event
carries the type of the change, the path, the rename target and the
values the placeholders captured. So a client reads the one file that
changed. Today a daemon that cares about /etc/pve compares the version
vector on every loop pass and re-reads what usually has not changed. A
connection admitted by its group id never sees private paths, as on
the IPC and FUSE side.

The socket is served by a Rust thread linked into the C daemon as a
static library with a small C ABI. The daemon never waits on a client.
A client that falls too far behind gets a resync event in place of
what it missed, and one that stopped reading is cut off. A node that
takes its whole state from the cluster sends a resync as well. Nothing
is kept across connections, so every subscription starts with a resync
and the client reads everything once. A panic in the Rust code never
takes pmxcfs down, at worst it switches the notifier off.

On top of the socket, pve-cluster ships three Perl modules. A client
handles the connection and its reconnects. A hook registry takes hooks
registered like API methods, with a path template whose placeholders
become the parameters of a run. And a runner executes those runs in
children. It keeps one run per hook and parameter set in flight and
collapses later events for the same set into a single rerun. The
series ships no hook and nothing that runs the runner.

Example, hook registration:
```
    package PVE::Example::Hooks;

    use base qw(PVE::Cluster::Hooks);

    __PACKAGE__->register_hook({
        name => 'guest-config',
        path => 'nodes/{node}/qemu-server/{vmid}.conf',
        code => sub {
            my ($param, $event) = @_;

            if ($event->{type} eq 'resync') {
                # reread every guest config
                return;
            }
            # reread guest $param->{vmid} on $param->{node}
        },
    });

    1;
```

Example, in a child of the daemon:
```
    use PVE::Example::Hooks;
    use PVE::Cluster::HookRunner;

    my $pid = fork() // die "fork failed: $!\n";
    if (!$pid) {
        my $ok = eval { PVE::Cluster::HookRunner->new()->run(); 1 };
        warn "hook runner failed: $@" if !$ok;
        POSIX::_exit($ok ? 0 : 1);
    }
```

Example, in the daemon's own loop:
```
    use PVE::Example::Hooks;
    use PVE::Cluster;
    use PVE::Cluster::Hooks;
    use PVE::Cluster::Watch;

    my $watch = PVE::Cluster::Watch->new();
    $watch->subscribe(PVE::Cluster::Hooks->subscription()->%*);

    # handle events for ten seconds. Events that arrive while the daemon
    # does other work wait in the socket until the next pass. If the
    # socket is full for 30s the server drops the connection, the next
    # wait reconnects and starts with a resync.
    my $deadline = time() + 10;
    while ((my $left = $deadline - time()) > 0) {
        my @events = $watch->wait($left);
        # so hooks reading through the cluster cache see the change
        PVE::Cluster::cfs_update() if @events;
        for my $event (@events) {
            for my $run (PVE::Cluster::Hooks->dispatch($event)->@*) {
                $run->{hook}->{code}->($run->{param}, $run->{event});
            }
        }
    }
```

Changes since the RFC, thanks @Wolfgang!
* resuming is gone, every subscribe is answered with a resync, so
  there are no sequence numbers, no ring and no saved positions
* the server runs on tokio with one task per connection and one
  bounded channel
* the protocol is parsed by serde alone
* the ffi takes the event enum directly and lost the version argument,
  the memdb hook lost it as well
* the build goes through cargo's dependency file with the build mode
  from d/rules
* the runner moved to pve-cluster and the pve-manager part is gone.
  The series adds the machinery only, the examples above show how to
  use it


pve-cluster:

Hannes Laimer (12):
  buildsys: add rust workspace under src/rust
  rust: notify: add path templates with placeholder capture
  rust: notify: add the wire protocol
  rust: notify: add subscription matching
  rust: notify: add the socket server
  rust: ffi: add C ABI staticlib for pmxcfs
  pmxcfs: memdb: add change notification hook
  buildsys: link pmxcfs against the rust notify staticlib
  pmxcfs: notify: emit change events over the notification socket
  cfs: add perl client for the change notification socket
  cfs: add hook registry for change notification consumers
  cfs: add runner executing cluster change hooks in children

 .gitignore                                  |   2 +
 Makefile                                    |   1 +
 debian/control                              |  22 +-
 debian/pve-cluster.install                  |   3 +
 debian/rules                                |  17 +
 src/Makefile                                |   2 +-
 src/PVE/Cluster/HookRunner.pm               | 268 ++++++++++
 src/PVE/Cluster/Hooks.pm                    | 135 +++++
 src/PVE/Cluster/Makefile                    |   2 +-
 src/PVE/Cluster/Watch.pm                    | 325 ++++++++++++
 src/defines.mk                              |  11 +
 src/pmxcfs/Makefile                         |  20 +-
 src/pmxcfs/cfs-utils.h                      |   4 +
 src/pmxcfs/database.c                       |   2 +
 src/pmxcfs/memdb.c                          |  22 +
 src/pmxcfs/memdb.h                          |   8 +
 src/pmxcfs/pmxcfs.c                         |  24 +
 src/rust/.cargo/config.toml                 |   8 +
 src/rust/Cargo.toml                         |  24 +
 src/rust/Makefile                           |  20 +
 src/rust/pmxcfs-ffi/Cargo.toml              |  17 +
 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h |  24 +
 src/rust/pmxcfs-ffi/src/lib.rs              | 306 +++++++++++
 src/rust/pmxcfs-notify/Cargo.toml           |  17 +
 src/rust/pmxcfs-notify/src/conn.rs          | 334 ++++++++++++
 src/rust/pmxcfs-notify/src/lib.rs           |  32 ++
 src/rust/pmxcfs-notify/src/protocol.rs      | 189 +++++++
 src/rust/pmxcfs-notify/src/server.rs        | 542 ++++++++++++++++++++
 src/rust/pmxcfs-notify/src/template.rs      | 285 ++++++++++
 src/rust/rustfmt.toml                       |   2 +
 src/test/Makefile                           |  14 +-
 src/test/hook_runner_test.pl                | 302 +++++++++++
 src/test/hooks_test.pl                      | 219 ++++++++
 src/test/watch_client_test.pl               | 229 +++++++++
 34 files changed, 3426 insertions(+), 6 deletions(-)
 create mode 100644 src/PVE/Cluster/HookRunner.pm
 create mode 100644 src/PVE/Cluster/Hooks.pm
 create mode 100644 src/PVE/Cluster/Watch.pm
 create mode 100644 src/defines.mk
 create mode 100644 src/rust/.cargo/config.toml
 create mode 100644 src/rust/Cargo.toml
 create mode 100644 src/rust/Makefile
 create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml
 create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h
 create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs
 create mode 100644 src/rust/pmxcfs-notify/Cargo.toml
 create mode 100644 src/rust/pmxcfs-notify/src/conn.rs
 create mode 100644 src/rust/pmxcfs-notify/src/lib.rs
 create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs
 create mode 100644 src/rust/pmxcfs-notify/src/server.rs
 create mode 100644 src/rust/pmxcfs-notify/src/template.rs
 create mode 100644 src/rust/rustfmt.toml
 create mode 100644 src/test/hook_runner_test.pl
 create mode 100644 src/test/hooks_test.pl
 create mode 100644 src/test/watch_client_test.pl


Summary over all repositories:
  34 files changed, 3426 insertions(+), 6 deletions(-)

-- 
Generated by murpp 0.12.0