[pmg-devel] [PATCH pmg-api/docs] improve documentation

[pmg-devel] [PATCH pmg-api/docs] improve documentation

[Dominik Csapak]

by expanding a bit on some options, adding references to postfix options
and explaining the statistics panels

screenshots are missing yet from the statistics panels

pmg-api:

Dominik Csapak (1):
  config: extend documentation for options a bit

 src/PMG/Config.pm | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)

pmg-docs:

Dominik Csapak (2):
  config: add reference to postfix postconf
  administration: add section for statistics

 pmg-administration.adoc | 117 ++++++++++++++++++++++++++++++++++++++++
 pmgconfig.adoc          |  14 +++--
 2 files changed, 127 insertions(+), 4 deletions(-)

-- 
2.30.2

[pmg-devel] [PATCH pmg-api 1/1] config: extend documentation for options a bit

[Dominik Csapak]

add an explanation what the 'advanced statistic filter' are.
add the reference to the postfix options if we have a direct one

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/PMG/Config.pm | 36 +++++++++++++++++++++++++-----------
 1 file changed, 25 insertions(+), 11 deletions(-)

diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm
index 386df18..ca9b5dd 100755
--- a/src/PMG/Config.pm
+++ b/src/PMG/Config.pm
@@ -63,7 +63,14 @@ sub type {
 sub properties {
     return {
 	advfilter => {
-	    description => "Use advanced filters for statistic.",
+	    description => "Enable advanced filters for statistic.",
+	    verbose_description => <<EODESC,
+Enable advanced filters for statistic.
+
+If this is enabled, the receiver statistic are limited to active ones
+(receivers which also sent out mail in the 90 days before), and the contact
+statistic will not contain these active receivers.
+EODESC
 	    type => 'boolean',
 	    default => 1,
 	},
@@ -519,11 +526,13 @@ sub properties {
 	    default => 0,
 	},
 	smarthost => {
-	    description => "When set, all outgoing mails are deliverd to the specified smarthost.",
+	    description => "When set, all outgoing mails are deliverd to the specified smarthost.".
+		"(postfix option `default_transport`)",
 	    type => 'string', format => 'address',
 	},
 	smarthostport => {
-	    description => "SMTP port number for smarthost.",
+	    description => "SMTP port number for smarthost.".
+		"(postfix option `default_transport`)",
 	    type => 'integer',
 	    minimum => 1,
 	    maximum => 65535,
@@ -587,13 +596,13 @@ sub properties {
 	    default => 0,
 	},
 	maxsize => {
-	    description => "Maximum email size. Larger mails are rejected.",
+	    description => "Maximum email size. Larger mails are rejected. (postfix option `message_size_limit`)",
 	    type => 'integer',
 	    minimum => 1024,
 	    default => 1024*1024*10,
 	},
 	dwarning => {
-	    description => "SMTP delay warning time (in hours).",
+	    description => "SMTP delay warning time (in hours). (postfix option `delay_warning_time`)",
 	    type => 'integer',
 	    minimum => 0,
 	    default => 4,
@@ -643,31 +652,36 @@ sub properties {
 	    default => 64,
 	},
 	helotests => {
-	    description => "Use SMTP HELO tests.",
+	    description => "Use SMTP HELO tests. (postfix option `smtpd_helo_restrictions)",
 	    type => 'boolean',
 	    default => 0,
 	},
 	rejectunknown => {
-	    description => "Reject unknown clients.",
+	    description => "Reject unknown clients. (postfix option `reject_unknown_client_hostname`)",
 	    type => 'boolean',
 	    default => 0,
 	},
 	rejectunknownsender => {
-	    description => "Reject unknown senders.",
+	    description => "Reject unknown senders. (postfix option `reject_unknown_sender_domain`)",
 	    type => 'boolean',
 	    default => 0,
 	},
 	verifyreceivers => {
-	    description => "Enable receiver verification. The value spefifies the numerical reply code when the Postfix SMTP server rejects a recipient address.",
+	    description => "Enable receiver verification. The value spefifies the numerical reply ".
+		"code when the Postfix SMTP server rejects a recipient address.".
+		"(postfix options `reject_unknown_recipient_domain`, `reject_unverified_recipient`,".
+		" and `unverified_recipient_reject_code`)",
 	    type => 'string',
 	    enum => ['450', '550'],
 	},
 	dnsbl_sites => {
-	    description => "Optional list of DNS white/blacklist domains (see postscreen_dnsbl_sites parameter).",
+	    description => "Optional list of DNS white/blacklist domains (postfix option ".
+		"`postscreen_dnsbl_sites`).",
 	    type => 'string', format => 'dnsbl-entry-list',
 	},
 	dnsbl_threshold => {
-	    description => "The inclusive lower bound for blocking a remote SMTP client, based on its combined DNSBL score (see postscreen_dnsbl_threshold parameter).",
+	    description => "The inclusive lower bound for blocking a remote SMTP client, based on".
+		"its combined DNSBL score (postfix option `postscreen_dnsbl_threshold`).",
 	    type => 'integer',
 	    minimum => 0,
 	    default => 1
-- 
2.30.2

[pmg-devel] applied: [PATCH pmg-api 1/1] config: extend documentation for options a bit

[Thomas Lamprecht]

Am 24/03/2023 um 14:30 schrieb Dominik Csapak:
> add an explanation what the 'advanced statistic filter' are.
> add the reference to the postfix options if we have a direct one
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  src/PMG/Config.pm | 36 +++++++++++++++++++++++++-----------
>  1 file changed, 25 insertions(+), 11 deletions(-)
> 
>

applied, with a follow up that adds a few missing whitespace in the descriptions
after line breaks, and used the newer style where concatenation points and whitespace
are handled at the beginning of a code line, not the end. thanks!

[pmg-devel] [PATCH pmg-docs 1/2] config: add reference to postfix postconf

[Dominik Csapak]

so that the users can find the place where the options are described

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pmgconfig.adoc | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/pmgconfig.adoc b/pmgconfig.adoc
index 564b87c..87e3192 100644
--- a/pmgconfig.adoc
+++ b/pmgconfig.adoc
@@ -302,8 +302,11 @@ ifndef::manvolnum[]
 [thumbnail="pmg-gui-mailproxy-relaying.png", big=1]
 endif::manvolnum[]
 
-These settings are saved to the 'mail' subsection in `/etc/pmg/pmg.conf`,
-using the following configuration keys:
+These settings are saved to the 'mail' subsection in `/etc/pmg/pmg.conf`. Some of these correspond
+to postfix options in the `main.cf` (see the
+https://www.postfix.org/postconf.5.html[postconf documentation]).
+
+They use the following configuration keys:
 
 include::pmg.mail-relaying-conf-opts.adoc[]
 
@@ -328,8 +331,11 @@ ifndef::manvolnum[]
 [thumbnail="pmg-gui-mailproxy-ports.png", big=1]
 endif::manvolnum[]
 
-These settings are saved to the 'mail' subsection in `/etc/pmg/pmg.conf`,
-using the following configuration keys:
+These settings are saved to the 'mail' subsection in `/etc/pmg/pmg.conf`. Many of these correspond
+to postfix options in the `main.cf` (see the
+https://www.postfix.org/postconf.5.html[postconf documentation]).
+
+They use the following configuration keys:
 
 include::pmg.mail-ports-conf-opts.adoc[]
 
-- 
2.30.2

[pmg-devel] [PATCH pmg-docs 2/2] administration: add section for statistics

[Dominik Csapak]

by explaining each panel and it's options

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pmg-administration.adoc | 117 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)

diff --git a/pmg-administration.adoc b/pmg-administration.adoc
index 05f4589..c4b83c4 100644
--- a/pmg-administration.adoc
+++ b/pmg-administration.adoc
@@ -238,3 +238,120 @@ You can view the complete headers and filter by sender or receiver of
 queued emails.
 
 Here, you can also flush or delete each deferred email independently.
+
+
+[[pmg_statistics]]
+Statistics
+----------
+
+{pmg} offers a useful and feature-rich statistics interface, which allows the
+admins to quickly see the general workload and makes it easy to identify
+problems.
+
+The statistics are shown for the selected time range, which defaults to the
+current day, but can be changed to any other day, a whole month, or even a
+whole year.
+
+On the man statistics page there are three graphs with additional data:
+
+'Total Mail Count'::
+
+Shows the total mail flow as a graph and the following details:
+
+* Total Mails
+* Incoming/Outgoing Mails (as count and percentage)
+* Virus Outbreaks (the amount of outgoing virus mails)
+* Avg. Mail Processing Time
+* Incoming/Outgoing Mail Traffic
+
+'Incoming Mails'::
+
+Shows the incoming mail counts as a graph and the following details
+(as counts and percentages of the incoming mail volume):
+
+* Incoming Mails
+* Junk Mails (Virus + Spam + Greylisted + SPF rejects + RBL rejects)
+* Greylisted Mails
+* Spam Mails (Mails with Spamscore >= 3 and no virus)
+* SPF rejects
+* Bounces (mails without a sender)
+* Virus Mails
+
+'Outgoing Mails'::
+
+Shows the outgoing mail counts as a graph and the following details
+(as counts and percentages of the incoming mail volume):
+
+* Outgoing Mails
+* Bounces (mails without a sender)
+* Virus Mails
+
+Spam Scores
+~~~~~~~~~~~
+
+This panel shows the mail count and percentage of the different spam scores
+for the selected time frame.
+
+Virus Charts
+~~~~~~~~~~~~
+
+This panel shows which and how many viruses were detected in the selected
+time frame.
+
+Hourly Distribution
+~~~~~~~~~~~~~~~~~~~
+
+Shows the amount of incoming and outgoing mail per hour for the selected
+time frame.
+
+Postscreen
+~~~~~~~~~~
+
+Contains a chart with the RBL and pregreet rejects for the selected time frame.
+For more info about postscreen and pregreet tests, see the
+https://www.postfix.org/POSTSCREEN_README.html[postfix postscreen readme].
+
+Domain
+~~~~~~
+
+Here is a summary for the recipient domain of the incoming and outgoing mails
+of the selected time frame, with traffic, count, how many viruses were detected,
+and how much of it was spam.
+
+Sender
+~~~~~~
+
+This panel contains a list of e-mail addresses that sent mail out in the
+selected time frame, with a total count, how many viruses were detected and how
+big these mails were.
+
+If you click on one of these e-mail addresses, you see a detailed list of
+recipients, complete with size, date and time.
+
+Receiver
+~~~~~~~~
+
+Analog to the `Sender` panel, this contains a list of e-mail addresses that
+received e-mails from outside, with a total, spam and virus count, as well
+as the total mail size.
+
+If you click an entry, it shows a detailed list of mails with size, date,
+time, virus and spam score info.
+
+If the `advanced statistics filters` are active, only active accounts will be
+listed. Active accounts are those which sent mail in the selected time frame
+or up to 90 days before.
+
+Contact
+~~~~~~~
+
+This contains the reverse list to the `Receiver` panel, namely a list of
+external recipients that received mail from this {pmg}, with total count,
+virus count and size.
+
+If you click an entry, it shows a list of mails with size, date and time.
+
+If the `advanced statistics filters` are active, active accounts will be
+filtered out, since they can already be seen in the `Receiver` panel. Active
+accounts are those which sent mail in the selected time frame or up to 90 days
+before.
-- 
2.30.2

Re: [pmg-devel] [PATCH pmg-docs 2/2] administration: add section for statistics

[Stoiko Ivanov]

Thanks for addressing this - This should make the experience for our users
much better, and reduce quite a few support-requests!

In general it looks good to me and could be merged as is

One high-level suggestion and a few nits inline (could be a follow-up as
well):

Since Statistics is its own entry in the GUI's navigation tree - why not
have it as a dedicated chapter of the docs? (quickly tried it here - just
putting it in a file of its own, and raising the heading levels by one
(`-` -> `=`, `~` -> `-`) and including it in pmg-adminstration-guide.adoc
seems to work fine)

It's not a strict necessity, and the docs to GUI matching is not 100%
currently either (Backup/Restore is its own chapter, but under
Configuration on the GUI for example)

Let me know what you think (and if you'll find the time soon to address it
or if I should send the suggestions as follow-up)

On Fri, 24 Mar 2023 14:30:13 +0100
Dominik Csapak <d.csapak@proxmox.com> wrote:

> by explaining each panel and it's options
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  pmg-administration.adoc | 117 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 117 insertions(+)
> 
> diff --git a/pmg-administration.adoc b/pmg-administration.adoc
> index 05f4589..c4b83c4 100644
> --- a/pmg-administration.adoc
> +++ b/pmg-administration.adoc
> @@ -238,3 +238,120 @@ You can view the complete headers and filter by sender or receiver of
>  queued emails.
>  
>  Here, you can also flush or delete each deferred email independently.
> +
> +
> +[[pmg_statistics]]
> +Statistics
> +----------
> +
> +{pmg} offers a useful and feature-rich statistics interface, which allows the
> +admins to quickly see the general workload and makes it easy to identify
s/admins/administrators/ ?
> +problems.
> +
> +The statistics are shown for the selected time range, which defaults to the
> +current day, but can be changed to any other day, a whole month, or even a
> +whole year.
> +
> +On the man statistics page there are three graphs with additional data:
s/man/main/

> +
> +'Total Mail Count'::
> +
> +Shows the total mail flow as a graph and the following details:
> +
> +* Total Mails
> +* Incoming/Outgoing Mails (as count and percentage)
> +* Virus Outbreaks (the amount of outgoing virus mails)
> +* Avg. Mail Processing Time
> +* Incoming/Outgoing Mail Traffic
> +
> +'Incoming Mails'::
> +
> +Shows the incoming mail counts as a graph and the following details
> +(as counts and percentages of the incoming mail volume):
> +
> +* Incoming Mails
> +* Junk Mails (Virus + Spam + Greylisted + SPF rejects + RBL rejects)
> +* Greylisted Mails
> +* Spam Mails (Mails with Spamscore >= 3 and no virus)
s/and no virus/and not containing a virus/?

> +* SPF rejects
> +* Bounces (mails without a sender)
s/without a sender/with an empty envelope-sender address/ ?
> +* Virus Mails
> +
> +'Outgoing Mails'::
> +
> +Shows the outgoing mail counts as a graph and the following details
> +(as counts and percentages of the incoming mail volume):
s/incoming/outgoing/ 

> +
> +* Outgoing Mails
> +* Bounces (mails without a sender)
as above - s/without a sender/with an empty envelope-sender address/ ?

> +* Virus Mails
> +
> +Spam Scores
> +~~~~~~~~~~~
> +
> +This panel shows the mail count and percentage of the different spam scores
> +for the selected time frame.
s/mail count and percentage/distribution/ ?

> +
> +Virus Charts
> +~~~~~~~~~~~~
> +
> +This panel shows which and how many viruses were detected in the selected
> +time frame.
> +
> +Hourly Distribution
> +~~~~~~~~~~~~~~~~~~~
> +
> +Shows the amount of incoming and outgoing mail per hour for the selected
> +time frame.
> +
> +Postscreen
> +~~~~~~~~~~
> +
> +Contains a chart with the RBL and pregreet rejects for the selected time frame.
> +For more info about postscreen and pregreet tests, see the
> +https://www.postfix.org/POSTSCREEN_README.html[postfix postscreen readme].
s/postfix// ?

> +
> +Domain
> +~~~~~~
> +
> +Here is a summary for the recipient domain of the incoming and outgoing mails
> +of the selected time frame, with traffic, count, how many viruses were detected,
> +and how much of it was spam.
s/Here is/Contains/ ?

> +
> +Sender
> +~~~~~~
> +
> +This panel contains a list of e-mail addresses that sent mail out in the
> +selected time frame, with a total count, how many viruses were detected and how
> +big these mails were.
> +
> +If you click on one of these e-mail addresses, you see a detailed list of
> +recipients, complete with size, date and time.
> +
> +Receiver
> +~~~~~~~~
> +
> +Analog to the `Sender` panel, this contains a list of e-mail addresses that
> +received e-mails from outside, with a total, spam and virus count, as well
> +as the total mail size.
s/Analog/Similar/ ?

> +
> +If you click an entry, it shows a detailed list of mails with size, date,
> +time, virus and spam score info.
> +
> +If the `advanced statistics filters` are active, only active accounts will be
> +listed. Active accounts are those which sent mail in the selected time frame
> +or up to 90 days before.
> +
> +Contact
> +~~~~~~~
> +
> +This contains the reverse list to the `Receiver` panel, namely a list of
> +external recipients that received mail from this {pmg}, with total count,
> +virus count and size.
Not sure if 'the reverse list' is easily understandable here?
Maybe:
This contains the list of external recipients that received mail from this
{pmg}, coming in on the internal port, with total count, virus count and
size. ?

> +
> +If you click an entry, it shows a list of mails with size, date and time.
> +
> +If the `advanced statistics filters` are active, active accounts will be
> +filtered out, since they can already be seen in the `Receiver` panel. Active
> +accounts are those which sent mail in the selected time frame or up to 90 days
> +before.

Re: [pmg-devel] [PATCH pmg-docs 2/2] administration: add section for statistics

[Dominik Csapak]

On 3/27/23 13:32, Stoiko Ivanov wrote:
> Thanks for addressing this - This should make the experience for our users
> much better, and reduce quite a few support-requests!
> 
> In general it looks good to me and could be merged as is
> 
> One high-level suggestion and a few nits inline (could be a follow-up as
> well):
> 
> Since Statistics is its own entry in the GUI's navigation tree - why not
> have it as a dedicated chapter of the docs? (quickly tried it here - just
> putting it in a file of its own, and raising the heading levels by one
> (`-` -> `=`, `~` -> `-`) and including it in pmg-adminstration-guide.adoc
> seems to work fine)
> 
> It's not a strict necessity, and the docs to GUI matching is not 100%
> currently either (Backup/Restore is its own chapter, but under
> Configuration on the GUI for example)
> 
> Let me know what you think (and if you'll find the time soon to address it
> or if I should send the suggestions as follow-up)
> 


thanks for the review, the inline changes all make sense and i'll send a v2 shortly

as for the own chapter, it don't really have a preference here. i put
it into administration as i did not have the impression that
the amount of information would warrant it's own chapter.

from a user perspective though, it would probably better as it's own
chapter, since it's also separate in the gui as you mentioned

so i'll do that in the v2