From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <t.lamprecht@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 9BE176474C
 for <pbs-devel@lists.proxmox.com>; Mon, 20 Jul 2020 16:47:00 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 8E505F1BD
 for <pbs-devel@lists.proxmox.com>; Mon, 20 Jul 2020 16:47:00 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [212.186.127.180])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id 94BDEF1AE
 for <pbs-devel@lists.proxmox.com>; Mon, 20 Jul 2020 16:46:58 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 59EA1430DA
 for <pbs-devel@lists.proxmox.com>; Mon, 20 Jul 2020 16:46:58 +0200 (CEST)
To: Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>, Oguz Bektas <o.bektas@proxmox.com>
References: <20200720140357.654551-1-o.bektas@proxmox.com>
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
Message-ID: <2f552422-6e3a-4bb4-5a5a-08b6d542f27a@proxmox.com>
Date: Mon, 20 Jul 2020 16:46:56 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101
 Thunderbird/79.0
MIME-Version: 1.0
In-Reply-To: <20200720140357.654551-1-o.bektas@proxmox.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL 0.010 Adjusted score from AWL reputation of From: address
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 NICE_REPLY_A           -0.001 Looks like a legit reply (A)
 RCVD_IN_DNSWL_MED        -2.3 Sender listed at https://www.dnswl.org/,
 medium trust
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: Re: [pbs-devel] [PATCH proxmox-backup] docs: add prototype sphinx
 extension for explicitly defined section labels
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Mon, 20 Jul 2020 14:47:00 -0000

On 20.07.20 16:03, Oguz Bektas wrote:
> goes through the sections in the documents and creates an output.txt
> file with the following contents:
> filename, line, title, refname, labelid
> 
> for example:
> 
> filename: /home/oguz/Git/rust/proxmox-backup/docs/package-repositories.rst
> line: 4
> title: <title>Debian Package Repositories</title>
> refname: Debian Package Repositories
> labelid: sysadmin-package-repositories
> 
> should be enough information to generate the OnlineHelp map. (anchor is
> same as labelid)

so from a quick glance it looks like it goes in the right direction, nice!

We now just need to output this in a JSON structured format, maybe an object
with the labelid as key and title (=refname?), and link - which would be
something like the /docs/ + file basename with s/\.rst$/.html/ + #labelid

That way we could easily build a small helper script (rust, or perl) which
replicates the /usr/bin/asciidoc-pve scan-extjs from PVE.

> ---
>  docs/Makefile                 |  4 ++
>  docs/_ext/proxmox-scanrefs.py | 83 +++++++++++++++++++++++++++++++++++
>  docs/conf.py                  |  7 ++-
>  3 files changed, 92 insertions(+), 2 deletions(-)
>  create mode 100644 docs/_ext/proxmox-scanrefs.py
> 
> diff --git a/docs/Makefile b/docs/Makefile
> index 34ec8809..796861b1 100644
> --- a/docs/Makefile
> +++ b/docs/Makefile
> @@ -68,6 +68,10 @@ proxmox-backup-manager.1: proxmox-backup-manager/man1.rst  proxmox-backup-manage
>  proxmox-backup-proxy.1: proxmox-backup-proxy/man1.rst  proxmox-backup-proxy/description.rst
>  	rst2man $< >$@
>  
> +.PHONY: scanrefs
> +scanrefs:
> +	$(SPHINXBUILD) -b proxmox-scanrefs $(ALLSPHINXOPTS) $(BUILDDIR)/scanrefs
> +
>  .PHONY: html
>  html: ${GENERATED_SYNOPSIS}
>  	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
> diff --git a/docs/_ext/proxmox-scanrefs.py b/docs/_ext/proxmox-scanrefs.py
> new file mode 100644
> index 00000000..c45de14f
> --- /dev/null
> +++ b/docs/_ext/proxmox-scanrefs.py
> @@ -0,0 +1,83 @@
> +#!/usr/bin/env python3
> +
> +# debugging stuff
> +from pprint import pprint
> +
> +from typing import cast
> +
> +import os
> +import io
> +from docutils import nodes
> +
> +from sphinx.builders import Builder
> +from sphinx.util import logging
> +
> +logger = logging.getLogger(__name__)
> +
> +# refs are added in the following manner before the title of a section (note underscore and newline before title):
> +# .. _my-label:
> +#
> +# Section to ref
> +# --------------
> +#
> +#
> +# then referred to like (note missing underscore):
> +# "see :ref:`my-label`"
> +#
> +# the benefit of using this is if a label is explicitly set for a section,
> +# we can refer to it with this anchor even if the section name changes.
> +#
> +# see https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref
> +
> +def setup(app):
> +    logger.info('Mapping reference labels...')
> +    app.add_builder(ReflabelMapper)
> +    return {
> +        'version': '0.1',
> +        'parallel_read_safe': True,
> +        'parallel_write_safe': True,
> +    }
> +
> +class ReflabelMapper(Builder):
> +    name = 'proxmox-scanrefs'
> +
> +    def init(self):
> +        self.docnames = []
> +
> +        if not os.path.isdir(self.outdir):
> +            os.mkdir(self.outdir)
> +
> +        self.output_filename = os.path.join(self.outdir, 'output.txt')
> +        self.output = io.open(self.output_filename, 'w', encoding='UTF-8')
> +
> +    def write_doc(self, docname, doctree):
> +            for node in doctree.traverse(nodes.section):
> +                #pprint(vars(node))
> +
> +                if hasattr(node, 'expect_referenced_by_id'): # only explicitly defined labels
> +                    labelid = node['ids'][1]
> +                    title = cast(nodes.title, node[0])
> +                    logger.info(f'traversing section {title}')
> +                    ref_name = getattr(title, 'rawsource', title.astext())
> +
> +                    self.output.write(u"\n\nfilename: %s\nline: %s\ntitle: %s\nrefname: %s\nlabelid: %s\n" % (
> +                        self.env.doc2path(docname),
> +                        node.line,
> +                        title,
> +                        ref_name,
> +                        labelid
> +                    ))
> +            return
> +
> +    def get_outdated_docs(self):
> +        return 'all documents'
> +
> +    def prepare_writing(self, docnames):
> +        return
> +
> +    def get_target_uri(self, docname, typ=None):
> +        return ''
> +
> +    def finish(self):
> +        self.output.close()
> +        return
> diff --git a/docs/conf.py b/docs/conf.py
> index 0c40ffb5..a9d45c2e 100644
> --- a/docs/conf.py
> +++ b/docs/conf.py
> @@ -18,9 +18,12 @@
>  # documentation root, use os.path.abspath to make it absolute, like shown here.
>  #
>  import os
> -# import sys
> +import sys
>  # sys.path.insert(0, os.path.abspath('.'))
>  
> +# custom extensions
> +sys.path.append(os.path.abspath("./_ext"))
> +
>  # -- Implement custom formatter for code-blocks ---------------------------
>  #
>  # * use smaller font
> @@ -46,7 +49,7 @@ PygmentsBridge.latex_formatter = CustomLatexFormatter
>  # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
>  # ones.
>  
> -extensions = ["sphinx.ext.graphviz", "sphinx.ext.todo"]
> +extensions = ["sphinx.ext.graphviz", "sphinx.ext.todo", "proxmox-scanrefs"]
>  
>  todo_link_only = True
>  
>