🐛 Accurate directive line mapping and source-attributed nested renders

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## Unreleased
+
+### ✨ New Features
+
+- ✨ Add a `source=` argument to `DocutilsRenderer.nested_render_text`, so extensions rendering generated content (an included file, a template's output) can attribute its nodes and warnings to the originating file or template, in both docutils and Sphinx builds
+- ✨ Add `MockStateMachine.insert_input`, a docutils-compatible way for a directive to render generated MyST at its position
+- ✨ Expose the active renderer to directives via read-only `MockState.renderer` and `MockStateMachine.renderer` properties
+
+### 🐛 Bug Fixes
+
+- 🐛 Make a directive's `content_offset` document-relative (the 0-based absolute line of its first content line), matching docutils' reStructuredText parser, so extensions computing `content_offset + N` (e.g. sphinx-jinja2) report the correct line; `MockState.nested_parse` correspondingly treats its `input_offset` as an absolute document line. Also corrects `{epigraph}` attribution and `{parsed-literal}` node lines to their true document lines
+  - _Migration note for extension authors:_ the standard `nested_parse(self.content, self.content_offset, node)` idiom is unchanged, but a directive passing a literal *relative* offset (e.g. `0`) now gets docutils-identical placement. Under Sphinx, autodoc/autosummary's default `offset=0` relocates rendered content to the top of the document — rST-consistent in the offset value, though rST additionally attributes such content via per-line `StringList` source items (which the MyST mock does not model), so the attribution is not byte-identical. Arithmetic such as `self.lineno + self.content_offset` now double-counts the position (as it always did under reStructuredText)
+- 🐛 Attribute warnings raised inside an `{include}`d file to that file rather than the parent document (in Sphinx), and fix line reporting to match the included file: an off-by-one in the base case, and a `:start-after:` bug that derived the line from a *character* index (so warnings landed past the file's end). MyST reports true file lines here, unlike docutils' `:start-after:`, which reports lines relative to the retained segment
+
+### 📚 Documentation
+
+- 📚 Add a developer guide for rendering generated content from a directive, covering the renderer accessors, `nested_render_text`, `insert_input` and `content_offset` semantics
+
 ## 5.1.0 - 2026-05-13
 
 ### ✨ New Features

docs/develop/extending.md

@@ -0,0 +1,176 @@
+(develop/extending)=
+
+# Extending the parser: rendering generated content
+
+Some Sphinx/docutils directives do not just wrap their literal content, but
+*generate* new content to render -- for example an included file, or markup
+produced from a template.  This page describes the supported API for doing that
+from a directive under MyST, so that the generated content is rendered as MyST
+Markdown and its warnings and nodes are attributed to the right source.
+
+:::{important}
+Content rendered through these APIs is parsed as **MyST Markdown**, not
+reStructuredText.  A directive that generates rST should wrap it in an
+[`{eval-rst}`](#syntax/directives/parsing) block instead.
+:::
+
+## Stability
+
+The APIs documented on this page are **supported, public API** as of the release
+that includes this page: ``DocutilsRenderer.nested_render_text`` (including its
+``source`` parameter), ``MockStateMachine.insert_input``, the
+``MockState.renderer`` / ``MockStateMachine.renderer`` properties, and the
+document-relative ``content_offset`` contract.  Third-party extensions may rely
+on them: any breaking change will be announced in the changelog and, where
+practicable, go through a deprecation cycle rather than changing silently.
+
+## Reaching the renderer from a directive
+
+When MyST runs a directive it passes mock ``state`` and ``state_machine``
+objects (rather than the docutils reStructuredText ones).  Both expose the
+active MyST renderer through a read-only ``renderer`` property, which is the
+supported entry point for the APIs below:
+
+```python
+from docutils.parsers.rst import Directive
+
+
+class MyDirective(Directive):
+    has_content = False
+
+    def run(self):
+        renderer = self.state.renderer  # or self.state_machine.renderer
+        ...
+        return []
+```
+
+See {py:obj}`myst_parser.mocking.MockState.renderer` and
+{py:obj}`myst_parser.mocking.MockStateMachine.renderer`.
+
+## Rendering generated text
+
+{py:meth}`myst_parser.mdit_to_docutils.base.DocutilsRenderer.nested_render_text`
+renders a string of MyST at the current position (appending to the current
+node):
+
+```python
+self.state.renderer.nested_render_text(text, lineno)
+```
+
+The ``lineno`` argument is a **0-based line shift** added to every rendered
+node's line: it establishes the *line-space* of ``text``.  There are two common
+choices:
+
+- **Document-relative content** -- pass the directive's ``content_offset`` (see
+  [below](content-offset)), so the generated lines map onto the document.
+- **File- or template-relative content** -- pass ``0``, so line ``N`` of the
+  generated text is reported as line ``N`` (of that file/template, when combined
+  with ``source`` below).
+
+For example, generated text whose warnings should read as lines ``1, 2, 3, …``
+of a template is rendered with ``lineno=0``.
+
+### Attributing to a source
+
+Pass ``source`` to attribute both the rendered nodes' ``source`` and any
+warnings emitted during the render to a specific path (typically the absolute
+path of the file or template the text came from), rather than the containing
+document:
+
+```python
+self.state.renderer.nested_render_text(template_output, 0, source="/abs/template.j2")
+```
+
+A warning on the first line of ``template_output`` is then reported as
+``/abs/template.j2:1`` in both docutils and Sphinx builds, and the resulting
+nodes carry ``source="/abs/template.j2"``.  The override is restored when the
+render finishes (even on error) and nests correctly, so a source render may
+itself contain further source renders -- this is exactly how nested
+``{include}`` directives attribute each file.
+
+:::{note}
+In Sphinx the location is passed to the logger as a ``"source:line"`` *string*.
+This is deliberate: Sphinx passes a colon-containing string through verbatim,
+whereas a plain path would be resolved against the project via ``doc2path``.
+:::
+
+## Inserting text at the directive's position
+
+{py:meth}`myst_parser.mocking.MockStateMachine.insert_input` mirrors the
+signature of docutils' ``RSTStateMachine.insert_input``, so directives written
+for docutils keep working:
+
+```python
+self.state_machine.insert_input(generated_lines, source="/abs/gen.txt")
+```
+
+Several behavioural differences are worth knowing:
+
+- **``source`` is optional.**  docutils' ``insert_input(input_lines, source)``
+  takes ``source`` as a *required* positional argument; here it is optional (see
+  the last paragraph).
+- the lines are parsed as **MyST Markdown**, not reStructuredText;
+- they are rendered *immediately* into the current node -- appearing **before**
+  any nodes the directive itself returns -- whereas docutils splices the input
+  back into the state machine, so it is processed **after** the directive's
+  returned nodes.  The two match only for the common ``return []`` pattern;
+- if you pass a docutils ``StringList``, its per-line ``(source, offset)`` items
+  are **not** preserved: the lines are joined and attributed uniformly to
+  ``source``.
+
+Without ``source`` the text is rendered in the document's own line-space, just
+after the directive.
+
+(content-offset)=
+## What ``content_offset`` means
+
+A directive's ``self.content_offset`` is **document-relative**: it is the
+0-based absolute line of the directive's first content line (equivalently, its
+1-based document line minus one), exactly as docutils' own reStructuredText
+parser provides.  The standard idiom therefore reports true document lines:
+
+```python
+# the Nth (1-based) content line is document line content_offset + N
+self.state_machine.reporter.warning("...", line=self.content_offset + n)
+```
+
+and each entry of ``self.content.items`` carries the absolute document line of
+its line.  A plain nested parse of the content needs nothing special:
+
+```python
+self.state.nested_parse(self.content, self.content_offset, node)
+```
+
+### Directives inside `{eval-rst}` blocks
+
+Directives written in an [`{eval-rst}`](#syntax/directives/parsing) block run
+under the **real** docutils reStructuredText state machine, not the mocks used
+for MyST-native directives.  Their ``lineno`` and ``content_offset`` are still
+document-absolute with respect to the containing ``.md`` file -- the same
+convention as MyST-native fences -- so the ``content_offset + N`` idiom reports
+true document lines identically in both.  This equivalence is pinned by
+``tests/test_directive_line_mapping.py::test_eval_rst_content_offset_pinned``.
+
+## Documented limitations
+
+- **Deferred, document-level warnings** are *not* covered by a render-scoped
+  ``source``.  For instance, a duplicate Markdown reference definition is
+  reported at the end of the whole-document render, so it attributes to the
+  containing document (with the in-file line) even when the definition came from
+  an included file.
+- **Inline tokens carry no per-line maps**, so a warning about inline content is
+  attributed to the line of its containing block, not the exact inline position.
+- **``document["source"]`` is repointed during a source-attributed render.**
+  While a ``source`` render is in progress, ``document["source"]`` and the
+  reporter are temporarily pointed at the override, so an extension that resolves
+  filesystem paths against ``document["source"]`` *mid-render* will see the
+  override rather than the containing document -- the same behaviour the
+  ``{include}`` directive has always had.
+
+## API reference
+
+- {py:obj}`myst_parser.mocking.MockState.renderer` /
+  {py:obj}`myst_parser.mocking.MockStateMachine.renderer`
+- {py:meth}`myst_parser.mdit_to_docutils.base.DocutilsRenderer.nested_render_text`
+- {py:meth}`myst_parser.mocking.MockStateMachine.insert_input`
+- {py:func}`myst_parser.warnings_.create_warning`

docs/develop/index.md

@@ -6,6 +6,7 @@ codebase, and some guidelines for how you can contribute.
 ```{toctree}
 contributing.md
 architecture.md
+extending.md
 test_infrastructure.md
 ```
 

myst_parser/_docs.py

@@ -161,7 +161,7 @@ def run(self):
 
         text.append("```````")
         node = nodes.Element()
-        self.state.nested_parse(text, 0, node)
+        self.state.nested_parse(text, self.content_offset, node)
         return node.children
 
 
@@ -219,7 +219,7 @@ def run(self):
             for key, func in klass.option_spec.items():
                 text += f"  {key} | {convert_opt(name, func)}\n"
         node = nodes.Element()
-        self.state.nested_parse(text.splitlines(), 0, node)
+        self.state.nested_parse(text.splitlines(), self.content_offset, node)
         return node.children
 
 
@@ -276,7 +276,7 @@ def run(self):
         ]
         text = [f"- `myst.{name}`: {' '.join(doc)}" for name, doc in warning_names]
         node = nodes.Element()
-        self.state.nested_parse(text, 0, node)
+        self.state.nested_parse(text, self.content_offset, node)
         return node.children
 
 

myst_parser/mdit_to_docutils/base.py

@@ -113,6 +113,10 @@ def __init__(self, parser: MarkdownIt) -> None:
         }
         # these are lazy loaded, when needed
         self._inventories: None | dict[str, inventory.InventoryType] = None
+        # a stack of source paths that nested renders attribute their warnings to
+        # (the top entry, if any, overrides the logged location of a warning);
+        # see ``nested_render_text``'s ``source`` parameter
+        self._attribution_sources: list[str] = []
 
     def __getattr__(self, name: str):
         """Warn when the renderer has not been setup yet."""
@@ -178,6 +182,10 @@ def create_warning(
 
         If the warning type is listed in the ``suppress_warnings`` configuration,
         then ``None`` will be returned and no warning logged.
+
+        While a source-attributed nested render is active (see
+        :meth:`nested_render_text`), the warning is attributed to that source
+        path rather than the containing document.
         """
         return create_warning(
             self.document,
@@ -186,6 +194,7 @@ def create_warning(
             wtype=wtype,
             line=line,
             append_to=append_to,
+            source=self._attribution_sources[-1] if self._attribution_sources else None,
         )
 
     def _render_tokens(self, tokens: list[Token]) -> None:
@@ -302,14 +311,37 @@ def nested_render_text(
         inline: bool = False,
         temp_root_node: None | nodes.Element = None,
         heading_offset: int = 0,
+        source: str | None = None,
     ) -> None:
-        """Render unparsed text (appending to the current node).
-
-        :param text: the text to render
-        :param lineno: the starting line number of the text, within the full source
-        :param inline: whether the text is inline or block
-        :param temp_root_node: If set, allow sections to be created as children of this node
-        :param heading_offset: offset heading levels by this amount
+        """Render a string of unparsed text, appending to the current node.
+
+        This is the supported mechanism for extensions to render generated
+        content (for example an included file, or the output of a template) at
+        the current position in the document.
+
+        .. important::
+
+            The text is parsed as **MyST Markdown**, not reStructuredText.  A
+            directive whose content is written in rST (rather than MyST) should
+            wrap it in an ``{eval-rst}`` block instead of rendering it here.
+
+        :param text: the text to render.
+        :param lineno: a 0-based line shift added to every rendered node's line,
+            i.e. it establishes the line-space of ``text``.  A directive rendering
+            document-relative content passes its ``content_offset``; content that
+            is relative to an external file passes ``0`` (so the file's 1-based
+            line ``N`` is reported as line ``N``).
+        :param inline: whether to parse the text as inline or block content.
+        :param temp_root_node: if set, allow sections to be created as children
+            of this node (used when parsing content that may contain headings).
+        :param heading_offset: offset heading levels by this amount.
+        :param source: if given, attribute both the rendered nodes' ``source``
+            and any warnings emitted during the render to this path (rather than
+            the containing document), for the duration of the render.  Typically
+            an absolute path to the file or template the ``text`` originates from.
+            The override is restored afterwards and nests correctly, so a source
+            render may itself contain further source renders (e.g. nested
+            includes).
         """
         tokens = (
             self.md.parseInline(text, self.md_env)
@@ -344,9 +376,46 @@ def _restore():
                 self.md_env["temp_root_node"] = current_root_node
                 self._level_to_section = current_level_to_section
 
-        with _restore():
+        with self._attribute_to_source(source), _restore():
             self._render_tokens(tokens)
 
+    @contextmanager
+    def _attribute_to_source(self, source: str | None) -> Iterator[None]:
+        """Temporarily attribute rendered nodes and warnings to ``source``.
+
+        For the duration of the context the document/reporter are pointed at
+        ``source`` (so node ``source`` stamping and docutils reporter warnings
+        attribute to it) and ``source`` is pushed onto the attribution stack
+        consulted by :meth:`create_warning` (so the sphinx logger attributes to
+        it too).  Everything is restored on exit, including after an exception,
+        and nests re-entrantly.  A ``source`` of ``None`` (or an empty string)
+        is a no-op.
+        """
+        if not source:
+            yield
+            return
+        reporter = self.reporter
+        prev_document_source = self.document["source"]
+        prev_reporter_source = reporter.source
+        # ``get_source_and_line`` may not pre-exist (it is normally installed by
+        # the rST state machine), so record whether to restore or delete it.
+        had_get_source_and_line = hasattr(reporter, "get_source_and_line")
+        prev_get_source_and_line = getattr(reporter, "get_source_and_line", None)
+        self.document["source"] = source
+        reporter.source = source
+        reporter.get_source_and_line = lambda li=None: (source, li)
+        self._attribution_sources.append(source)
+        try:
+            yield
+        finally:
+            self._attribution_sources.pop()
+            self.document["source"] = prev_document_source
+            reporter.source = prev_reporter_source
+            if had_get_source_and_line:
+                reporter.get_source_and_line = prev_get_source_and_line
+            else:
+                del reporter.get_source_and_line
+
     @contextmanager
     def current_node_context(
         self, node: nodes.Element, append: bool = False
@@ -1897,6 +1966,15 @@ def run_directive(
                 lineno=position,
             )
         else:
+            source = self.document["source"]
+            # ``position`` is the 1-based document line of the opening fence, and
+            # ``parsed.body_offset`` is relative to the line after it, so their sum
+            # is the 0-based document line of the first content line, i.e. the
+            # ``content_offset`` docutils' rST parser provides (== 1-based first
+            # content line minus 1). The per-line ``items`` likewise carry the
+            # 0-based *absolute* document line of each content line, matching
+            # docutils rather than the historical 0, 1, 2, ... relative offsets.
+            content_offset = position + parsed.body_offset
             state_machine = MockStateMachine(self, position)
             state = MockState(self, state_machine, position)
             directive_instance = directive_class(
@@ -1906,11 +1984,18 @@ def run_directive(
                 # a dictionary mapping option names to values
                 options=parsed.options,
                 # the directive content line by line
-                content=StringList(parsed.body, self.document["source"]),
+                content=StringList(
+                    parsed.body,
+                    source,
+                    items=[
+                        (source, content_offset + i) for i in range(len(parsed.body))
+                    ],
+                ),
                 # the absolute line number of the first line of the directive
                 lineno=position,
-                # the line offset of the first line of the content
-                content_offset=parsed.body_offset,
+                # the 0-based document line of the first content line
+                # (docutils convention, document-relative)
+                content_offset=content_offset,
                 # a string containing the entire directive
                 block_text=content if block_text is None else block_text,
                 state=state,