docs(build): unified output subtree under docs/build/#4081
docs(build): unified output subtree under docs/build/#4081grandixximo wants to merge 12 commits into
Conversation
All rendered artefacts (HTML, PDF, po4a-translated .adoc) move under
docs/build/. Layout:
docs/build/html/index.html static redirect -> en/
docs/build/html/{shared assets, CSS}
docs/build/html/<lang>/index.html
docs/build/html/<lang>/<topic>/<page>.html
docs/build/html/<lang>/man/manN/<page>.html
docs/build/html/<lang>/pdf/LinuxCNC_*[_<lang>].pdf
docs/build/adoc/<lang>/ po4a-translated source
English now has its own en/ subdir so every language is symmetric
and the top-level redirect lands users on the right tree.
Manpage troff stays at docs/man/ (halcompile output, install-man /
mandb / rip-environment read it).
Static assets in the former output dir docs/html/ go to docs/src/
(gcode.html, index.css, logo); asciidoc-py-era leftovers are dropped.
debian/*.docs and po4a.cfg are repointed at the new tree. Install
destinations under usr/share/doc/ are unchanged.
Also moves three build-time artefacts that previously leaked into
docs/src/ to the new build tree, so the source tree stays clean and
no per-build .gitignore entries are needed:
docs/src/man/man1/linuxcnc.1.adoc (config.status output)
-> docs/build/adoc/en/man/man1/linuxcnc.1.adoc
The manpage HTML rule grows a build-tree fallback so the
English manpage build still finds it; the explicit troff rule
is duplicated from the generic pattern (sources under
docs/src/man/) since the build-tree path no longer matches.
docs/src/drivers/mb2hal_HOWTO.ini (.include-stamp copy)
-> docs/build/adoc/en/drivers/mb2hal_HOWTO.ini
mb2hal.adoc's include now uses {fixturedir}, set per-language
by the asciidoctor HTML rule. The link: target still copies
to docs/build/html/<lang>/drivers/ for runtime download.
po files repointed at the new include path.
docs/src/gui/gmoccapy_release_notes.txt (.include-stamp copy)
Dead code under the new layout: gmoccapy.adoc only link:s the
file (HTML render time), and the recipe already copies it
next to the rendered HTML.
|
reverting to draft till it builds on sid |
…ionally CI on PR LinuxCNC#4081 failed for two reasons exposed by --enable-build-documentation=pdf: * The asciidoctor PDF rule did not set the fixturedir attribute, so the mb2hal {fixturedir}/drivers/mb2hal_HOWTO.ini include resolved literally in every Master_Documentation*.pdf and asciidoctor reported it missing. Pass fixturedir=$(DOC_OUT_ADOC)/<lang> from the rule, where <lang> is the literal "en" for the English call site and $(firstword $(subst /, ,$*)) for the per-language call site (so the same recipe serves both). * docs/build/html/en/gcode.html only got copied as a side effect of .copy-asciidoc-stamp, which lives under htmldocs. debian/configure sets --enable-build-documentation=pdf for CI, so htmldocs never fires and the en/ copy never lands, breaking dh_installdocs --package= linuxcnc-doc-en. Promote the copy to its own rule, depended on unconditionally by `docs:`.
|
I tested your branch just for the english language.
Another thing we may could do in this rush: there is a folder |
…extractor
The mb2hal.adoc source contains `include::{fixturedir}/drivers/mb2hal_HOWTO.ini[]`,
where `{fixturedir}` is an asciidoctor attribute resolved per-language at
render time. asciideps does literal sed extraction and was emitting the
`{fixturedir}` placeholder as a phantom prereq path; the subsequent
recursive sed on that path could not find the file and produced
sed: can't read .../{fixturedir}/drivers/mb2hal_HOWTO.ini: No such file or directory
four times per build. Make went on (sed in a pipe does not propagate
non-zero exit), so the build still succeeded, but the noise was
confusing.
Wire FIXTUREDIR via env from the depends/%.d rules (one per language)
and let asciideps sed-substitute it in extracted include paths. Mirrors
the attribute asciidoctor itself receives in the HTML and PDF rules.
Reported by @hansu on PR LinuxCNC#4081.
Per @hansu's review of PR LinuxCNC#4081. Moving PDFs out from under the html subtree keeps the html zip free of PDF bloat and matches the rest of the docs/build/ layout convention (one top-level dir per format). `PDF_TARGETS_*` rules now build into $(DOC_OUT_PDF)/<lang>/, where $(DOC_OUT_PDF) := $(DOC_BUILD)/pdf. Per-page hardlink rules and the Manual_Pages.pdf rule follow. The pdf-index helper now lands docs/build/pdf/index.html (was docs/build/html/pdf-index.html), enumerated from the new tree with relative `<lang>/X.pdf` links. install-doc-pdf still flattens $(PDF_TARGETS) into the docsdir, so the debian package layout (single flat usr/share/doc/linuxcnc/X.pdf per .docs file) is unchanged.
Per @hansu's review of PR LinuxCNC#4081. gen_complist.py runs early (its output is a po4a master), well before the HTML manpage targets are rendered. Its broken-link check against $(DOC_OUT_HTML)/en/hal/<link> always failed at that stage, emitting gen_complist: Broken link: ../man/man1/axis.1.html once per linked component (~100 warnings per build) even though the links resolved correctly in the final tree. Match the link href against the `manN/<name>` capture and check membership in the man_files set (built from listdir of ../docs/man/man{1,9}). man_files is populated before gen_complist runs from its $(MAN_SRCS) prereq, so the check now reflects whether a manpage source actually exists. Real broken-link validation between rendered HTML pages is checkref's job; this script only needs to know whether a target manpage exists in the source set.
|
Just discovered: Running make docs the second time invokes Anyway, this script needs to be changed when the reorganisation moves stuff to other places. |
Aligns with the rest of the documentation source tree (docs/src/<topic>/ for everything po4a-translated and consumed by the docs build). Refs adjusted: - docs/po4a.cfg masters: help/rtfaults.adoc -> src/help/rtfaults.adoc (same for tklinuxcnc.adoc); translated outputs still land at docs/build/adoc/<lang>/help/. - src/Makefile install-docs DOCS_HELP glob. Debian package layout unchanged (install-docs flattens the .adoc into usr/share/doc/linuxcnc/). Suggested by @hansu on PR LinuxCNC#4081.
pdfdocs was phony with `../scripts/make-docs-pdf-index` in its recipe, so the script re-ran on every invocation and rewrote $(DOC_OUT_PDF)/index.html even when no PDF input changed. Make the pdf-index a real file target gated on $(PDF_TARGETS); the script fires only when a PDF changes or the script itself changes. Second `make pdfdocs` is now a no-op. Reported by @BsAtHome on PR LinuxCNC#4081.
|
Build is stable now. Next up is to review it all... |
Per @BsAtHome's review on PR LinuxCNC#4086. The old COMP_MANPAGES / COMP_DRIVER_MANPAGES rules ran halcompile in --document mode (troff straight out) and sed-edited the troff to escape .als / .URL groff directives the old dblatex-based PDF pipeline could not parse. Asciidoctor now reads the component adoc directly, so the sed hacks are obsolete. New chain: - halcompile --adoc -o objects/man/man9/<comp>.9.adoc <comp>.comp - asciidoctor --doctype=manpage --backend=manpage -> ../docs/man/man9/<comp>.9 The adoc extraction only needs Python + halcompile.py (yapps-generated, not the C build), so a fresh-tree docs-only build (make htmldocs / pdfdocs / docs) extracts and renders component manpages without first compiling the rest of LinuxCNC. Previously a docs-only build silently omitted them. Also adds homecomp.9 to COMP_MANPAGES alongside tpcomp.9 (both .comp files are filtered out of COMPS because they do not compile to .o modules, but their manpages are still worth shipping). Surfaces in @hansu's review of PR LinuxCNC#4081 as a 'gen_complist: Broken link: ../man/man9/homecomp.9.html' warning. .9.adoc suffix matches the convention the docs HTML manpage rule already expects under objects/man/. Tested: clean-tree make docs (English-only) builds 207 troff + 154 extracted component adocs, link checker clean. Standalone make ../docs/man/man9/abs.9 from a tree with no halcompile binary rebuilds halcompile and produces the manpage without touching C code.
|
I think there is some whitespace problem here. Seems to happen for each language: |
|
There are PDF build artifacts left in the docs/src directory. The languages at the top of the master index.html does not match the list made in the po4a.cfg. The list must be generated but the The PDF tree is outside the HTML tree. That means we cannot link the PDFs in one tree. The index.html generated needs to be rewritten. We need one index.html per language and it must be linked in the master HTML index to find the PDFs. A lot of the .gitignore files in the docs tree have become obsolete and should be removed entirely. The master .gitignore needs to be cleaned up too. I suggest that the ignores for the docs are actually in The man page tree still contains the cat[139] parts. I think we can get rid of the (old) catalog stuff. We only really need the man-page tree. That would also clean up some stuff in the makefiles again. FWIW, I don't really understand why Another issue seen is: This only seems to happen for the Ukranian (uk) translation. There must be something in there that trips it and it is a consistent error. (and probably more...) |
|
Thanks for the thorough review. Most of it I can just fix, one item needs a decision from you first. PDF tree location: On the master index language list not matching po4a.cfg: that one is handled in the topbar/lang-switcher PR, which drops the hardcoded |
For consistency, all PDFs should IMO be called I think the target directory
Check, great! |
da91abf expanded {fixturedir} but left `$DIR/` in the sed pattern, so an absolute expansion got doubled (`$DIR//abs/.../mb2hal_HOWTO.ini`), firing once per language. Reported by @BsAtHome on PR LinuxCNC#4081. Capture the bare target, expand attrs, then prepend DIR only when the result is relative. For includes this runs in the command substitution, before the loop, since DIR is global and the recursive call clobbers it.
Running `mandb docs/man` on every `make docs` produced index.db and the cat[139] dirs, which nothing in the build or install consumes; dpkg rebuilds the whatis database at package install anyway. It also spewed per-language `whatis parse failed` warnings. Remove the step, the clean-mandb rule, and the now-dead cat/index.db ignores. The configure AC_PATH_PROG(MANDB) detection is now unused; dropping it needs a configure regen, left to a separate commit.
The English asciidoctor-pdf rule used one directory for both input and output, so Master_*.pdf (and the gs .raw intermediates) were written into the source tree. Give the rule a separate output dir and point the English instance at objects/ (build scratch); translated PDFs stay next to their po4a-generated source. The pdf-tree hardlink and .SECONDARY list follow the new path.
The per-directory ignore files under docs/src/ listed dblatex-era intermediates (.tex and extensionless outputs) that the asciidoctor toolchain no longer produces. Remove them and the docs/man one, move the still-live patterns (build/, beside-source .html, components_gen.adoc, the .dot-rendered SVGs, generated man trees) into a single docs/.gitignore, and drop the docs entries from the top-level .gitignore.
Per review, the PDF output moves from the sibling docs/build/pdf/ into the per-language trees docs/build/html/<lang>/pdf/, so the html subtree is self-contained and can link its PDFs with relative paths. Every PDF now carries a _<lang> suffix (English gains _en) so a downloaded file keeps its language and languages coexist side by side. A flat aggregate is kept at docs/build/html/pdf/ (hardlinks of every language's PDF, regenerated by make-docs-pdf-index) so the legacy flat /pdf/ URL keeps working; now it lists all languages, not just English. install-doc-pdf strips the _en suffix for English so the package filenames (referenced by the desktop launchers and doc-base) are unchanged. PDF_TARGETS_LinuxCNC_* were unused; removed.
|
Pushed a batch addressing the review. PDF layout (your call): PDFs now build into Also in this push: PDF artifacts out of Deferred to the topbar PR: the lang-list/ |
|
The real problem here is that you updated the .pot/.po outside weblate. That will give serious trouble. The way is to change the source file, then goto weblate and fix all the translations. Then, probably @smoe, should perform a pot/po update to make things consistent. It sucks this way, but otherwise we'll have a merge conflict on translations. Maybe @smoe can chime in here on the preferred sequence? |
|
You have not merged anything against master yet Weblate, right? Weblate is not in a good state at the moment. I suggest I bring this back to normal and independently you remove anything on the .pot/.po from your branch. My biggest concern is that we lose externally contributed translations. Everything else is fine with me. |
| # source-highlight local cache. | ||
| /src/source-highlight/local | ||
|
|
There was a problem hiding this comment.
Source-highlight was removed and you are using rouge. All the source-highlight stuff is obsolete.
| # Generated component manpage list (gen_complist.py). | ||
| /src/hal/components_gen.adoc | ||
|
|
There was a problem hiding this comment.
Wouldn't it be better to run po4a on the build/adoc/en tree and not have to generate the components_get.adoc in the src tree?
| /src/code/homing.svg | ||
| /src/code/homing_es.svg | ||
| /src/code/hss.svg | ||
| /src/code/hss_es.svg | ||
| /src/code/task-state-transitions.svg | ||
| /src/code/task-state-transitions_es.svg | ||
|
|
There was a problem hiding this comment.
These should be generated into the build tree.
We should keep the docs/src tree clean from anything generated for as much as possible.
| # po4a-translated man trees. | ||
| /man/ar/* | ||
| /man/bg/* | ||
| /man/ckb/* | ||
| /man/cs/* | ||
| /man/da/* | ||
| /man/de/* | ||
| /man/es/* | ||
| /man/fr/* | ||
| /man/hu/* | ||
| /man/it/* | ||
| /man/ka/* | ||
| /man/nb/* | ||
| /man/pl/* | ||
| /man/pt/* | ||
| /man/pt_BR/* | ||
| /man/ro/* | ||
| /man/ru/* | ||
| /man/sai/* | ||
| /man/sk/* | ||
| /man/sv/* | ||
| /man/tr/* | ||
| /man/uk/* | ||
| /man/vi/* | ||
| /man/zh_CN/* |
There was a problem hiding this comment.
This should now all be under build.
| /build/ | ||
|
|
||
| # Intermediate .html asciidoctor writes beside each source .adoc before the | ||
| # copy step moves it into build/html/. | ||
| /src/*/*.html | ||
|
|
There was a problem hiding this comment.
Why have these a leading /? Using relative paths is a thing ;-)
(other places too)
| # Troff manpages generated from .adoc by the asciidoctor manpage backend. | ||
| /man/man1/ | ||
| /man/man3/ | ||
| /man/man9/ |
There was a problem hiding this comment.
Aren't they be generated into the right place now?
| subst_attrs() { | ||
| if [ -n "$FIXTUREDIR" ]; then | ||
| sed -e "s|{fixturedir}|$FIXTUREDIR|g" | ||
| else | ||
| cat | ||
| fi | ||
| } |
There was a problem hiding this comment.
I'm not sure that using the {fixture} way is the right fix.
| [source,ini] | ||
| ---- | ||
| include::mb2hal_HOWTO.ini[] | ||
| include::{fixturedir}/drivers/mb2hal_HOWTO.ini[] | ||
| ---- |
There was a problem hiding this comment.
This document is never translated. It is a verbatim version from the sources. It is doubtful that you want it to be translated. If you want translations, then the document including this should do a better job explaining things in a more granular way instead of dropping in a big INI file. It is also in the wrong place and the docs/src/Submakefile hacks the document (and also gmoccapy release notes) to be copied somewhere else.
It is IMO an abomination to copy sources from the code-tree into the docs tree. It is either documentation or source, not both.
Placing this stuff the right place inlined will also make it easier and you don't need to "{fixture}" hacks and the pot/po problem vanishes instantly.
(Also, the release notes are not translatable)
No, and if it is up to me we shouldn't touch the pot/po in normal PRs at all. And in this case here it is actually not necessary.
My PR comments already suggest to remove all these pot/po changes so we keep the status quo. |
|
Hmm I get this, but doesn't seem to be a problem on the CI: |
We can move the |
Summary
Move all rendered docs artefacts (HTML, PDF, po4a-translated
.adoc) under a singledocs/build/tree, drop asciidoc-py-era leftovers indocs/html/, and stop leaking build-time artefacts intodocs/src/.Builds on top of #4053 (asciidoctor migration). Implements Bertho's layout proposal agreed in the #4053 discussion.
Layout
English now lives under
en/like every other language, so the top-level redirect lands users on the right tree.Manpage troff stays at
docs/man/because halcompile output,install-man,mandb, andrip-environmentread it.Source-tree cleanup
Three build-time artefacts no longer leak into
docs/src/:docs/src/man/man1/linuxcnc.1.adoc(config.status output)docs/build/adoc/en/man/man1/linuxcnc.1.adocdocs/src/drivers/mb2hal_HOWTO.ini(include fixture)docs/build/adoc/en/drivers/mb2hal_HOWTO.inidocs/src/gui/gmoccapy_release_notes.txtmb2hal.adoc'sinclude::now uses the{fixturedir}attribute, set per-language by the asciidoctor HTML rule. The 8.pofiles are repointed at the new include path (mechanical msgid edit, no po4a regen).The manpage HTML rule grows a build-tree fallback so the English manpage build finds
linuxcnc.1.adoc; theLinuxCNC_Manual_Pages.pdfmaster generator grows the same fallback. An explicit troff rule forlinuxcnc.1replaces the generic pattern (which expects sources underdocs/src/man/).Static assets in the former output dir
docs/html/move todocs/src/(gcode.html,index.css, logo). asciidoc-py-era files (asciidoc.css,linuxcnc.css,minus.png,plus.png) are deleted.debian/*.docsanddocs/po4a.cfgare repointed at the new tree. Install destinations underusr/share/doc/are unchanged.Test plan
cd src && make -j$(nproc)clean full buildmb2hal.htmlcontains the example INI fixture in all 8 languagesLinuxCNC_Documentation.pdfcontains the mb2hal INI fixturelinuxcnc(1)troff renders todocs/man/man1/linuxcnc.1linuxcnc(1)HTML and PDF (inLinuxCNC_Manual_Pages.pdf) both contain the manpage bodymake doccleanleaves no generated.adoc/.ini/.txtindocs/src/git clean -dxn docs/src/reports onlyhal/components_gen.adoc(pre-existing, generated bygen_complist.py, not from this PR)