[PATCH 7/7] docs: client: document scheduled backup jobs

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 docs/backup-client.rst | 274 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 274 insertions(+)

diff --git a/docs/backup-client.rst b/docs/backup-client.rst
index 39b1f9be0..c224cc19c 100644
--- a/docs/backup-client.rst
+++ b/docs/backup-client.rst
@@ -210,6 +210,280 @@ Additionally, system credentials (e.g. passed down from the hypervisor to a
 virtual machine via SMBIOS type 11) can be loaded on a service via
 `LoadCredential=` as described in the manual page ``systemd.exec(5)``.
 
+.. _client_scheduled_backups:
+
+Scheduled Backups
+-----------------
+
+The backup client has a built-in scheduler that runs backup jobs at regular
+intervals, without :manpage:`cron(8)` or a systemd timer for each job. A job is
+a named configuration holding everything a backup run needs: the target
+repository, the backup sources, the encryption key, and optionally a script
+that decides whether a run should proceed.
+
+Jobs are configured per user and stored in that user's XDG directories. Manage
+the jobs as the user that should run them, for example as ``root`` for a backup
+that has to read every file on the system.
+
+``$XDG_CONFIG_HOME/proxmox-backup/jobs.cfg``
+  The job definitions, in section configuration format. The format has no
+  comment lines and cannot store values with leading or trailing spaces, so the
+  ``job`` subcommands reject such values. If you edit the file by hand, do not
+  change jobs with the ``job`` subcommands at the same time, as they rewrite
+  the whole file.
+
+  If the file does not parse, the ``job`` subcommands report the error and
+  the scheduler refuses to start, until you fix the file.
+
+``$XDG_STATE_HOME/proxmox-backup/jobstates/backup-job-<id>.json``
+  The outcome of the job's most recent run, see `Job State and Concurrency`_.
+
+``$XDG_STATE_HOME/proxmox-backup/scheduler.pid``
+  The process ID of the running scheduler.
+
+Without these environment variables, the paths default to
+``~/.config/proxmox-backup/`` and ``~/.local/state/proxmox-backup/``. The
+configuration and state files are created with mode ``0600``, as they record
+where the secrets of a job come from.
+
+Managing Jobs
+~~~~~~~~~~~~~
+
+The ``proxmox-backup-client job`` subcommands create, list, show, update,
+remove, and run jobs. For example, the following command creates a job:
+
+.. code-block:: console
+
+  # proxmox-backup-client job create nightly \
+      --schedule '02:30' \
+      --server pbs.example.com --datastore store1 --auth-id 'backup@pbs!nightly' \
+      --backupspec root.pxar:/ --backupspec home.pxar:/home \
+      --exclude /var/cache --exclude /var/tmp \
+      --keyfile /root/.config/proxmox-backup/key.json \
+      --password-file /run/secrets/pbs-token \
+      --pre-run-hook /usr/local/bin/pbs-check-battery.sh \
+      --comment 'nightly laptop backup'
+
+As with ``proxmox-backup-client backup``, you can name the target repository
+with the separate ``--server``, ``--port``, ``--datastore``, and ``--auth-id``
+options, as above, or as a single URL with ``--repository``.
+
+All paths in a job must be absolute: the backup sources, the key file, the
+files holding secrets, the ``--include-dev`` mount points, and the pre-run
+hook. The scheduler does not share the working directory and ``$PATH`` of your
+shell, so a relative path could resolve to a different file. The patterns of
+``--exclude`` are the exception, as they are matched relative to the backup
+source.
+
+The other subcommands inspect and change existing jobs:
+
+.. code-block:: console
+
+  # proxmox-backup-client job list
+  # proxmox-backup-client job show nightly
+  # proxmox-backup-client job update nightly --schedule 'mon..fri 02:30'
+  # proxmox-backup-client job update nightly --delete comment --delete pre-run-hook
+  # proxmox-backup-client job disable nightly
+  # proxmox-backup-client job enable nightly
+  # proxmox-backup-client job run nightly
+  # proxmox-backup-client job run nightly --dry-run
+  # proxmox-backup-client job remove nightly
+
+A dry run shows what the job would back up without uploading anything. It
+still runs the pre-run hook and connects to the server, but it records no run,
+so the last result and the next run of the job stay as they were.
+
+``update`` accepts the same options as ``create`` and changes only the options
+you give. ``--delete <property>`` clears an optional setting. Setting
+``--repository`` on a job that uses the separate options clears them, and the
+other way around, so you can switch between both forms in one call.
+
+A disabled job keeps its configuration, but the scheduler does not start it.
+You can still run it with ``job run``.
+
+The ``--rate`` and ``--burst`` options limit the bandwidth of a job, and take
+the same values as for ``proxmox-backup-client backup``:
+
+.. code-block:: console
+
+  # proxmox-backup-client job update nightly --rate 10MB --burst 20MB
+
+Unattended Authentication
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A scheduled run cannot ask for a password, so a job names the source of each
+secret it needs with the following options. A secret configured in the job
+takes precedence over the corresponding ``PBS_*`` environment variables.
+
+``--password-file <path>`` / ``--password-cmd <command>``
+  The source of the repository password or API token secret: the first line of
+  the file, or the first line that the command prints. Use only one of them.
+  The command runs without a shell, so give it as an absolute path, and use
+  ``/bin/sh -c '...'`` for shell syntax. If the command fails, the run fails
+  with its error. An empty password is not caught early, and shows up as an
+  authentication failure. For example, read a file that systemd decrypted with
+  ``--password-file /run/secrets/pbs-token``, or ask a password manager with
+  ``--password-cmd '/usr/bin/pass show pbs/nightly'``.
+
+  The client waits for the command without a time limit, and a hanging command
+  blocks one of the two backups that the scheduler runs at a time. If the
+  command can hang, give it a timeout of its own, for example
+  ``--password-cmd '/usr/bin/timeout 30 /usr/bin/pass show pbs/nightly'``.
+
+``--encryption-password-file <path>`` / ``--encryption-password-cmd <command>``
+  The same for the passphrase of the key file, which the client otherwise reads
+  from :envvar:`PBS_ENCRYPTION_PASSWORD`. A job needs one of them if its key
+  file is protected by a passphrase.
+
+``--fingerprint <sha256>``
+  Pins the fingerprint of the server certificate, in the colon-separated
+  hexadecimal form that the web interface shows. Use it if the system does not
+  trust the certificate of the server, for example a self-signed one. A
+  scheduled run cannot ask you to confirm an unknown fingerprint, so without
+  this option it fails, unless you already accepted the fingerprint in an
+  interactive run as the same user, for example with ``job run``.
+
+Schedules
+~~~~~~~~~
+
+The ``--schedule`` option accepts the same calendar events as the jobs of the
+server, see :ref:`calendar-event-scheduling` for the syntax. Some common
+schedules:
+
+======================= ==============================================
+Schedule                Meaning
+======================= ==============================================
+``hourly``              At the top of every hour.
+``daily``               At 00:00 every day.
+``02:30``               Every day at 02:30.
+``mon..fri 02:30``      Weekdays at 02:30.
+``*/15``                Every 15 minutes.
+``sat *-*-1..7 03:00``  First Saturday of each month at 03:00.
+======================= ==============================================
+
+The next run is computed from the end of the previous one. A run missed while
+the machine was switched off therefore starts shortly after the scheduler comes
+back up, rather than at the next occurrence. The scheduler checks the schedules
+at least once a minute, so a run may start up to a minute late.
+
+A run that fails or is interrupted, for example because the machine was
+switched off, is not retried. The job runs again at its next scheduled time,
+or right away with ``job run``.
+
+``create`` and ``update`` reject a schedule without future events, such as a
+date in the past.
+
+The Scheduler Daemon
+~~~~~~~~~~~~~~~~~~~~
+
+Jobs only run while a scheduler is running, and creating a job does not start
+one. The recommended way to run it is the systemd user service that the client
+package ships. The service is not enabled by default, as a backup daemon should
+only run for users that configured a job:
+
+.. code-block:: console
+
+  # systemctl --user enable --now proxmox-backup-client-scheduler
+
+The scheduler rereads ``jobs.cfg`` at least once a minute. To apply a change at
+once, run ``systemctl --user reload proxmox-backup-client-scheduler``. If no
+scheduler is running when you create or enable a job, the subcommand prints how
+to start one.
+
+If ``jobs.cfg`` stops parsing while the scheduler runs, the scheduler keeps
+running but starts no new runs until you fix the file. Runs already in progress
+finish.
+
+A run started by the scheduler first checks that the job is still due, and does
+nothing if the job was disabled, removed, or has already run in the meantime. A
+manual ``job run`` always runs the job, regardless of its schedule.
+
+On ``SIGTERM``, the scheduler starts no new runs and waits for the running ones
+to finish. A second ``SIGTERM`` makes it exit without waiting. ``systemctl
+--user stop`` therefore lets a running backup finish, up to the
+``TimeoutStopSec`` of the service.
+
+Each run is a separate ``proxmox-backup-client job run <id>`` process with its
+own environment, so that the password of one job cannot leak into another. The
+runs log to the journal of the user service. At most two backups run at the
+same time. Jobs that have to wait start as soon as another run finishes,
+longest waiting first.
+
+Only one scheduler can run per user. A second one refuses to start.
+
+.. Note:: A systemd user service only runs while its user is logged in, unless
+   `lingering <https://www.freedesktop.org/software/systemd/man/loginctl.html>`_
+   is enabled for that user with ``loginctl enable-linger <username>``.
+
+Job State and Concurrency
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Each run records its outcome in the state file of the job. ``job list`` and
+``job show`` display it in the ``last-run`` and ``last-result`` columns, and the
+time the job is due next in the ``next-run`` column. In ``job list``,
+``next-run`` reads ``disabled`` for a disabled job and ``never`` for a schedule
+without future events. ``last-result`` is one of the following:
+
+``created``
+  The job has not run yet.
+
+``running``
+  A run is in progress.
+
+``ok``
+  The last run succeeded.
+
+``failed``
+  The last run failed. ``job show`` prints the error.
+
+``skipped``
+  The pre-run hook skipped the last run, so no backup was made.
+
+``interrupted``
+  The last run never finished, for example because the machine was switched
+  off or the process was killed.
+
+Without a readable state file, ``last-run`` and ``last-result`` stay empty, and
+the job is due right away.
+
+A job never runs twice at the same time. Starting ``job run`` for a job that is
+already running fails after a few seconds, and ``job remove`` refuses to remove
+a running job.
+
+Pre-Run Hooks
+~~~~~~~~~~~~~
+
+The ``--pre-run-hook <path>`` option names an executable, by its absolute path,
+that decides by its exit status whether a run proceeds:
+
+- ``0``: The backup proceeds.
+- ``1``: This run is skipped and recorded as ``skipped``. The schedule continues
+  as usual.
+- Any other status, or a hook killed by a signal, fails the run, so that a
+  broken hook is not mistaken for a deliberate skip.
+
+The hook gets the job ID in the ``PBS_JOB_ID`` environment variable, runs in
+``/`` without standard input, and does not see the ``PBS_PASSWORD*``,
+``PBS_ENCRYPTION_PASSWORD*``, and ``CREDENTIALS_DIRECTORY`` variables, as it
+does not need the secrets of the backup. Its output goes to the log of the run.
+
+Hooks typically skip a run under conditions that a schedule cannot express, for
+example:
+
+- while the machine runs on battery, or the battery is low
+- while the backup server is not reachable
+- during working hours
+
+A hook may take up to one minute. After that, the client stops it along with
+any processes it started, and the run fails.
+
+The client packages ship an example hook that skips a backup when the battery
+is low. It is ``examples/pbs-check-battery.sh`` in the documentation directory
+of the installed package, ``/usr/share/doc/proxmox-backup-client/`` or
+``/usr/share/doc/proxmox-backup-client-static/``. Copy it to a stable path such
+as ``/usr/local/bin``, adjust the thresholds, and point ``--pre-run-hook`` at
+the copy.
+
 Output Format
 -------------
 
-- 
2.47.3