From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C0931603AE for ; Tue, 1 Sep 2020 18:11:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B623BAB3D for ; Tue, 1 Sep 2020 18:11:44 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 5073FAB33 for ; Tue, 1 Sep 2020 18:11:43 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id EC1E74496E for ; Tue, 1 Sep 2020 18:11:42 +0200 (CEST) To: Proxmox Backup Server development discussion , Oguz Bektas References: <20200825125359.4176466-1-o.bektas@proxmox.com> From: Thomas Lamprecht Message-ID: Date: Tue, 1 Sep 2020 18:11:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:81.0) Gecko/20100101 Thunderbird/81.0 MIME-Version: 1.0 In-Reply-To: <20200825125359.4176466-1-o.bektas@proxmox.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.943 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -2.13 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox-scanrefs.py, sphinx-doc.org, sphinx.builders, conf.py] Subject: Re: [pbs-devel] [PATCH v2 proxmox-backup] docs: add prototype sphinx extension for online help X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Sep 2020 16:11:44 -0000 On 25.08.20 14:53, Oguz Bektas wrote: > goes through the sections in the documents and creates the > OnlineHelpInfo.js file from the explicitly defined section labels which= > are used in the js files with 'onlineHelp' variable. >=20 > these anchors can then be used to refer to the sections even if the > section name changes. >=20 > --- much better as I can now actually test if it works without patching aroun= d myself :-) > docs/Makefile | 7 +- > docs/_ext/proxmox-scanrefs.py | 127 ++++++++++++++++++++++++++++++++++= > docs/conf.py | 7 +- > docs/local-zfs.rst | 3 + > www/Makefile | 4 ++ > www/OnlineHelpInfo.js | 10 +-- > 6 files changed, 150 insertions(+), 8 deletions(-) > create mode 100644 docs/_ext/proxmox-scanrefs.py >=20 > diff --git a/docs/Makefile b/docs/Makefile > index 34ec8809..8d1e8a1b 100644 > --- a/docs/Makefile > +++ b/docs/Makefile > @@ -28,7 +28,6 @@ COMPILEDIR :=3D ../target/debug > SPHINXOPTS +=3D -t devbuild > endif > =20 > - > # Sphinx internal variables. > ALLSPHINXOPTS =3D -d $(BUILDDIR)/doctrees $(SPHINXOPTS) . > =20 > @@ -68,6 +67,12 @@ proxmox-backup-manager.1: proxmox-backup-manager/man= 1.rst proxmox-backup-manage > proxmox-backup-proxy.1: proxmox-backup-proxy/man1.rst proxmox-backup-= proxy/description.rst > rst2man $< >$@ > =20 > +.PHONY: onlinehelpinfo > +onlinehelpinfo: > + @echo "Generating OnlineHelpInfo.js..." > + $(SPHINXBUILD) -b proxmox-scanrefs $(ALLSPHINXOPTS) $(BUILDDIR)/scanr= efs > + @echo "Build finished. OnlineHelpInfo.js is in $(BUILDDIR)/scanrefs."= > + > .PHONY: html > html: ${GENERATED_SYNOPSIS} > $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html > diff --git a/docs/_ext/proxmox-scanrefs.py b/docs/_ext/proxmox-scanrefs= =2Epy > new file mode 100644 > index 00000000..b021c6e9 > --- /dev/null > +++ b/docs/_ext/proxmox-scanrefs.py > @@ -0,0 +1,127 @@ > +#!/usr/bin/env python3 > + > +# debugging stuff > +from pprint import pprint > + > +from typing import cast > + > +import json > +import re > + > +import os > +import io > +from docutils import nodes > + > +from sphinx.builders import Builder > +from sphinx.util import logging > + > +logger =3D logging.getLogger(__name__) > + > +# refs are added in the following manner before the title of a section= (note underscore and newline before title): > +# .. _my-label: > +# > +# Section to ref > +# -------------- > +# > +# > +# then referred to like (note missing underscore): > +# "see :ref:`my-label`" > +# > +# the benefit of using this is if a label is explicitly set for a sect= ion, > +# we can refer to it with this anchor #my-label in the html, > +# even if the section name changes. > +# > +# see https://www.sphinx-doc.org/en/master/usage/restructuredtext/role= s.html#role-ref > + > +def scan_extjs_files(wwwdir=3D"../www"): # a bit rough i know, but we = can optimize later > + js_files =3D [] > + used_anchors =3D [] > + logger.info("scanning extjs files for onlineHelp definitions") > + for root, dirs, files in os.walk("{}".format(wwwdir)): > + #print(root, dirs, files) > + for filename in files: > + if filename.endswith('.js'): > + js_files.append(os.path.join(root, filename)) > + for js_file in js_files: > + fd =3D open(js_file).read() > + match =3D re.search("onlineHelp:\s*[\'\"](.*?)[\'\"]", fd) # m= atch object is tuple > + if match: > + anchor =3D match.groups()[0] > + anchor =3D re.sub('_', '-', anchor) # normalize labels > + logger.info("found onlineHelp: {} in {}".format(anchor, js= _file)) > + used_anchors.append(anchor) > + return used_anchors > + > + > +def setup(app): > + logger.info('Mapping reference labels...') > + app.add_builder(ReflabelMapper) > + return { > + 'version': '0.1', > + 'parallel_read_safe': True, > + 'parallel_write_safe': True, > + } > + > +class ReflabelMapper(Builder): > + name =3D 'proxmox-scanrefs' > + > + def init(self): > + self.docnames =3D [] > + self.env.online_help =3D {} > + self.env.used_anchors =3D scan_extjs_files() > + > + if not os.path.isdir(self.outdir): > + os.mkdir(self.outdir) > + > + self.output_filename =3D os.path.join(self.outdir, 'OnlineHelp= Info.js') > + self.output =3D io.open(self.output_filename, 'w', encoding=3D= 'UTF-8') > + > + def write_doc(self, docname, doctree): > + for node in doctree.traverse(nodes.section): > + #pprint(vars(node)) > + > + if hasattr(node, 'expect_referenced_by_id') and len(no= de['ids']) > 1: # explicit labels > + filename =3D self.env.doc2path(docname) > + filename_html =3D re.sub('.rst', '.html', filename= ) > + labelid =3D node['ids'][1] # [0] is predefined by = sphinx, we need [1] for explicit ones > + title =3D cast(nodes.title, node[0]) > + logger.info('traversing section {}'.format(title.a= stext())) > + ref_name =3D getattr(title, 'rawsource', title.ast= ext()) > + > + self.env.online_help[labelid] =3D {'link': '', 'ti= tle': ''} > + self.env.online_help[labelid]['link'] =3D "/docs/"= + os.path.basename(filename_html) + "#{}".format(labelid) > + self.env.online_help[labelid]['title'] =3D ref_nam= e > + > + return > + > + > + def get_outdated_docs(self): > + return 'all documents' > + > + def prepare_writing(self, docnames): > + return > + > + def get_target_uri(self, docname, typ=3DNone): > + return '' > + > + def validate_anchors(self): > + #pprint(self.env.online_help) > + to_remove =3D [] > + for anchor in self.env.online_help: > + if anchor in self.env.used_anchors: shouldn't this get switched? I.e., loop over used_anchors and error out i= f you find one which isn't in online_help (like we do now for asciidoc/pve)? > + logger.info("[+] anchor {} found. leaving in onlineHel= p.".format(anchor)) > + else: > + logger.info("[-] anchor {} not used! removing...".form= at(anchor)) > + to_remove.append(anchor) > + for anchor in to_remove: > + self.env.online_help.pop(anchor, None) > + return > + > + def finish(self): > + # generate OnlineHelpInfo.js output > + self.validate_anchors() > + self.output.write("const proxmoxOnlineHelpInfo =3D ") > + self.output.write(json.dumps(self.env.online_help, indent=3D2)= ) > + self.output.write(";\n") > + self.output.close() > + return > diff --git a/docs/conf.py b/docs/conf.py > index 0c40ffb5..a9d45c2e 100644 > --- a/docs/conf.py > +++ b/docs/conf.py > @@ -18,9 +18,12 @@ > # documentation root, use os.path.abspath to make it absolute, like sh= own here. > # > import os > -# import sys > +import sys > # sys.path.insert(0, os.path.abspath('.')) > =20 > +# custom extensions > +sys.path.append(os.path.abspath("./_ext")) > + > # -- Implement custom formatter for code-blocks ----------------------= ----- > # > # * use smaller font > @@ -46,7 +49,7 @@ PygmentsBridge.latex_formatter =3D CustomLatexFormatt= er > # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom > # ones. > =20 > -extensions =3D ["sphinx.ext.graphviz", "sphinx.ext.todo"] > +extensions =3D ["sphinx.ext.graphviz", "sphinx.ext.todo", "proxmox-sca= nrefs"] > =20 > todo_link_only =3D True > =20 > diff --git a/docs/local-zfs.rst b/docs/local-zfs.rst > index 41633fd8..cd9d5315 100644 > --- a/docs/local-zfs.rst > +++ b/docs/local-zfs.rst > @@ -1,3 +1,6 @@ > + > +.. _chapter-zfs: > + > ZFS on Linux > ------------ > =20 > diff --git a/www/Makefile b/www/Makefile > index edce8cb3..2aaf1e7b 100644 > --- a/www/Makefile > +++ b/www/Makefile > @@ -54,6 +54,10 @@ all: js/proxmox-backup-gui.js css/ext6-pbs.css > js: > mkdir js > =20 > +OnlineHelpInfo.js: > + make -C ../docs onlinehelpinfo It's good practice to use $(MAKE) for the make executable as else you may= lose some definitions the debian buildsystem or other parts may set. > + mv ../docs/output/scanrefs/OnlineHelpInfo.js . > + > js/proxmox-backup-gui.js: js OnlineHelpInfo.js ${JSSRC} > cat OnlineHelpInfo.js ${JSSRC} >$@.tmp > mv $@.tmp $@ > diff --git a/www/OnlineHelpInfo.js b/www/OnlineHelpInfo.js > index bb48bab6..b05c5950 100644 > --- a/www/OnlineHelpInfo.js > +++ b/www/OnlineHelpInfo.js > @@ -1,6 +1,6 @@ > -var proxmoxOnlineHelpInfo =3D { > - "pbs_documentation_index" : { > - "link" : "/pbs-docs/index.html", > - "title" : "Proxmox Backup Server Documentation Index" > - } Above is always required, as it's either the fall-back or used for the "D= ocumentation" button at the top-right. So your plugin should probably just output that too. > +const proxmoxOnlineHelpInfo =3D { > + "chapter-zfs": { > + "link": "/docs/sysadmin.html#chapter-zfs", > + "title": "ZFS on Linux" > + } > }; >=20