Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ docs/**/.media
docs/**/_static
docs/**/assets

# Section directories - aggregated content is ignored, index.md is tracked
# Section directories - aggregated content is ignored, index.md and selected files are tracked
docs/contributing/**
!docs/contributing/index.md
!docs/contributing/documentation/
!docs/contributing/documentation/**
!docs/contributing/documentation_workflow.md
docs/explanation/**
!docs/explanation/index.md
!docs/explanation/documentation/
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
---
title: "Adding Repositories to documentation aggregation"
title: "Adding Repositories to Documentation Aggregation"
description: "Guide for adding new repositories to the documentation aggregation system"
---

# Adding Repositories to documentation aggregation
# Adding Repositories to Documentation Aggregation

This guide explains how to add a new repository to the documentation aggregation system.
This guide explains how to add a new repository to the documentation aggregation
system.

> **Source Repository:** [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)
> **Source Repository:**
> [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)

## Prerequisites

Expand Down Expand Up @@ -41,9 +43,11 @@ Edit `repos-config.json` and add a new entry to the `repos` array:
### Optional Fields

- **`commit`**: Lock to a specific commit hash for reproducibility
- **`root_files`**: List of root-level files to copy (e.g., `["README.md", "CONTRIBUTING.md"]`)
- **`root_files`**: List of root-level files to copy (e.g.,
`["README.md", "CONTRIBUTING.md"]`)
- **`structure`**: Directory mapping strategy (see below)
- **`media_directories`**: List of media directories to copy (e.g., `[".media", "_static", "assets"]`)
- **`media_directories`**: List of media directories to copy (e.g.,
`[".media", "_static", "assets"]`)

## Step 2: Choose a Structure Strategy

Expand Down Expand Up @@ -78,7 +82,8 @@ Map source directories to Diataxis categories:

## Step 3: Configure Targeted Documentation

To have files automatically placed into the main Diataxis structure, add `github_target_path` frontmatter to markdown files in the source repository:
To have files automatically placed into the main Diataxis structure, add
`github_target_path` front-matter to markdown files in the source repository:

```markdown
---
Expand All @@ -91,7 +96,8 @@ github_target_path: "docs/how-to/example-guide.md"
Content here...
```

Files with `github_target_path` will be copied to that exact location, not to `projects/<repo-name>/`.
Files with `github_target_path` will be copied to that exact location, not to
`projects/<repo-name>/`.

## Step 4: Test with Local Configuration

Expand Down Expand Up @@ -172,7 +178,8 @@ The system will:

- Find all instances of these directories recursively
- Copy nested media dirs (e.g., `tutorials/assets/`) to the same relative path
- Copy root-level media dirs (e.g., `_static/`) to the common ancestor of targeted files
- Copy root-level media dirs (e.g., `_static/`) to the common ancestor of
targeted files

### Root Files

Expand All @@ -185,7 +192,8 @@ Copy root-level files (like README.md or CONTRIBUTING.md):
}
```

These files can also have `github_target_path` frontmatter for targeted placement.
These files can also have `github_target_path` front-matter for targeted
placement.

### Special Files

Expand Down Expand Up @@ -233,7 +241,7 @@ Here's a complete configuration:

- Verify `docs_path` points to the correct directory
- Check that the repository has a `docs-ng` branch or adjust `ref`
- Ensure `github_target_path` frontmatter is correct
- Ensure `github_target_path` front-matter is correct

### Media Not Copied

Expand All @@ -246,9 +254,13 @@ Here's a complete configuration:
- Check that relative links in source docs are correct
- Review `src/aggregation/transformer.py` for link rewriting logic

## See Also
## Related Topics

- [Adding Repositories](../../how-to/documentation/adding-repos.md) — How to add new repositories to the aggregation
- [Technical Reference](../../reference/documentation/technical.md) — Source code and API documentation
- [Configuration Reference](../../reference/documentation/configuration.md) — Complete configuration options
- [Architecture Explanation](../../explanation/documentation/aggregation-architecture.md) — Deep dive into how the documentation aggregation system works
- [Documentation Workflow](./documentation_workflow.md)
- [Documentation Quality Markers](./writing_good_docs.md)
- [Documentation Aggregator Architecture](./aggregation-architecture.md)
- [How to Documentation - Adding Repos to Aggregate](./adding-repos.md)
- [How to Documentation - Working With the Aggregator Locally](./working-locally.md)
- [Documentation Aggregator Technical Reference](./technical.md)
- [Documentation Aggregator Local Testing Guide](./testing.md)
- [Working with the Documentation Hub on Your Machine](./working-locally.md)
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,16 @@ description: "Deep dive into how the documentation aggregation system works"

# Documentation Aggregation Architecture

Deep dive into the design and implementation of the documentation aggregation system.
Deep dive into the design and implementation of the documentation aggregation
system.

> **Source Repository:** [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)
> **Source Repository:**
> [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)

## System Overview

We use a documentation aggregation pipeline that combines content from multiple source repositories into a unified VitePress documentation site.
We use a documentation aggregation pipeline that combines content from multiple
source repositories into a unified VitePress documentation site.

```
┌─────────────────┐
Expand All @@ -33,7 +36,7 @@ We use a documentation aggregation pipeline that combines content from multiple
┌─────────────────┐
│ Transform Stage │
│ Rewrite links │
│ Fix frontmatter
│ Fix front-matter
└────────┬────────┘
Expand All @@ -58,8 +61,10 @@ We use a documentation aggregation pipeline that combines content from multiple

**Mechanisms:**

- **Git Sparse Checkout:** For remote repositories, uses sparse checkout to fetch only the `docs/` directory, minimizing clone size
- **Local Copy:** For `file://` URLs, performs direct filesystem copy without git operations
- **Git Sparse Checkout:** For remote repositories, uses sparse checkout to
fetch only the `docs/` directory, minimizing clone size
- **Local Copy:** For `file://` URLs, performs direct filesystem copy without
git operations
- **Commit Resolution:** Records the resolved commit hash for locking

**Key Features:**
Expand All @@ -74,15 +79,16 @@ We use a documentation aggregation pipeline that combines content from multiple

**Transformations:**

1. **Link Rewriting:** Transform relative links to work across repository boundaries
1. **Link Rewriting:** Transform relative links to work across repository
boundaries

- Intra-repo links: Maintained relative to project mirror
- Cross-repo links: Rewritten to absolute paths
- External links: Preserved as-is

2. **Frontmatter Handling:** Ensure all documents have proper frontmatter
2. **Front-matter Handling:** Ensure all documents have proper front-matter

- Add missing frontmatter blocks
- Add missing front-matter blocks
- Quote YAML values safely
- Preserve existing metadata

Expand All @@ -94,8 +100,10 @@ We use a documentation aggregation pipeline that combines content from multiple

**Operations:**

1. **Targeted Documentation:** Copy files with `github_target_path` to specified locations
2. **Directory Mapping:** Transform source directories according to `structure` config
1. **Targeted Documentation:** Copy files with `github_target_path` to specified
locations
2. **Directory Mapping:** Transform source directories according to `structure`
config
3. **Media Copying:** Discover and copy media directories
4. **Markdown Processing:** Apply transformations to all markdown files

Expand All @@ -109,7 +117,8 @@ We use a documentation aggregation pipeline that combines content from multiple

### Targeted Documentation

Files with `github_target_path` frontmatter are copied directly to their specified location:
Files with `github_target_path` front-matter are copied directly to their
specified location:

```yaml
---
Expand All @@ -128,7 +137,8 @@ This allows fine-grained control over where content appears in the final site.

### Project Mirrors

In addition to targeted docs, the entire `docs/` directory from each repo is mirrored under `docs/projects/<repo-name>/`:
In addition to targeted docs, the entire `docs/` directory from each repo is
mirrored under `docs/projects/<repo-name>/`:

**Purpose:**

Expand Down Expand Up @@ -158,9 +168,9 @@ For reproducible builds, commits can be locked:

```json
{
"name": "repo",
"ref": "main",
"commit": "abc123..."
"name": "repo",
"ref": "main",
"commit": "abc123..."
}
```

Expand All @@ -186,7 +196,7 @@ This fetches the latest from `ref` and updates commit locks.
- **Speed:** Faster than full clone, especially for large repos
- **Minimal Disk Usage:** Reduces storage requirements

### Why Frontmatter-Based Targeting?
### Why Front-Matter-Based Targeting?

- **Flexibility:** Authors control where their docs appear
- **Decentralization:** No central mapping file to maintain
Expand Down Expand Up @@ -243,7 +253,7 @@ Temp Directory Docs Output
### Transform Stage

- **Link rewriting:** O(n \* m) where n = files, m = avg file size
- **Frontmatter:** O(n) single pass through files
- **Front-matter:** O(n) single pass through files

### Structure Stage

Expand All @@ -267,7 +277,7 @@ Temp Directory Docs Output

### Transform Failures

- Invalid frontmatter → Add default frontmatter, log warning
- Invalid front-matter → Add default front-matter, log warning
- Broken links → Log warning, preserve original link
- Invalid markdown → Process as best-effort, log error

Expand All @@ -277,9 +287,13 @@ Temp Directory Docs Output
- Conflicting file paths → Error with clear message
- Media directory not found → Log warning, continue

## See Also
## Related Topics

- [Adding Repositories](../../how-to/documentation/adding-repos.md) — How to add new repositories to the aggregation
- [Technical Reference](../../reference/documentation/technical.md) — Source code and API documentation
- [Configuration Reference](../../reference/documentation/configuration.md) — Complete configuration options
- [Architecture Explanation](../../explanation/documentation/aggregation-architecture.md) — Deep dive into how the documentation aggregation system works
- [Documentation Workflow](./documentation_workflow.md)
- [Documentation Quality Markers](./writing_good_docs.md)
- [Documentation Aggregator Architecture](./aggregation-architecture.md)
- [How to Documentation - Adding Repos to Aggregate](./adding-repos.md)
- [How to Documentation - Working With the Aggregator Locally](./working-locally.md)
- [Documentation Aggregator Technical Reference](./technical.md)
- [Documentation Aggregator Local Testing Guide](./testing.md)
- [Working with the Documentation Hub on Your Machine](./working-locally.md)
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,22 @@ description: "Complete reference for repos-config.json and repos-config.local.js

Complete reference for configuring the documentation aggregation system.

> **Source Repository:** [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)
> **Source Repository:**
> [gardenlinux/docs-ng](https://github.com/gardenlinux/docs-ng)

## Configuration Files

### `repos-config.json`

Main configuration file for production aggregation. Uses git URLs and commit locks.
Main configuration file for production aggregation. Uses git URLs and commit
locks.

**Location:** Project root

### `repos-config.local.json`

Development configuration file for local testing. Uses `file://` URLs to avoid git operations.
Development configuration file for local testing. Uses `file://` URLs to avoid
git operations.

**Location:** Project root

Expand Down Expand Up @@ -64,7 +67,8 @@ Development configuration file for local testing. Uses `file://` URLs to avoid g
- **Examples:**
- Git: `"https://github.com/gardenlinux/gardenlinux"`
- Local: `"file://../gardenlinux"`
- **Notes:** For local development, use `file://` URLs in `repos-config.local.json`
- **Notes:** For local development, use `file://` URLs in
`repos-config.local.json`

#### `docs_path`

Expand Down Expand Up @@ -103,7 +107,8 @@ Development configuration file for local testing. Uses `file://` URLs to avoid g
- **Description:** Root-level files to copy (e.g., README.md, CONTRIBUTING.md)
- **Example:** `["README.md", "CONTRIBUTING.md", "LICENSE"]`
- **Default:** `[]` (no root files copied)
- **Notes:** Files can have `github_target_path` frontmatter for targeted placement
- **Notes:** Files can have `github_target_path` front-matter for targeted
placement

#### `structure`

Expand Down Expand Up @@ -137,7 +142,8 @@ This maps source directories to Diataxis categories.
- **Notes:**
- Searched recursively in source repository
- Nested media dirs (e.g., `tutorials/assets/`) copied to same relative path
- Root-level media dirs (e.g., `_static/`) copied to common ancestor of targeted files
- Root-level media dirs (e.g., `_static/`) copied to common ancestor of
targeted files

#### `special_files`

Expand Down Expand Up @@ -257,7 +263,7 @@ Simplest configuration for a flat repository:

### With Targeted Documentation

Repository using `github_target_path` frontmatter:
Repository using `github_target_path` front-matter:

```json
{
Expand All @@ -280,27 +286,37 @@ github_target_path: "docs/tutorials/my-tutorial.md"
---
```

## Frontmatter Fields
## Front-Matter Fields

When using `github_target_path`, you can include additional metadata:

- **`github_org`**: Organization name (e.g., `"gardenlinux"`)
- **`github_repo`**: Repository name (e.g., `"docs-ng"`)
- **`github_source_path`**: Original file path in source repo (e.g., `"docs/tutorial.md"`)
- **`github_source_path`**: Original file path in source repo (e.g.,
`"docs/tutorial.md"`)

These help create source links in the documentation.

### Page Display Fields

- **`description`**: A short summary of the page. Used as the `<meta>` description by VitePress and shown in section index listings when `overviewDescriptions` is enabled on the parent page.
- **`overviewDescriptions`**: Boolean controlling whether the `<SectionIndex />` component displays child page descriptions. Defaults to `true`. Set to `false` on an index page to hide descriptions for its listing.
- **`order`**: Numeric value for controlling sort order in the sidebar and section listings. Lower values appear first.
- **`migration_status`**: Status for content migration (e.g., `"new"`, `"adapt"`, `"aggregate"`).
- **`description`**: A short summary of the page. Used as the `<meta>`
description by VitePress and shown in section index listings when
`overviewDescriptions` is enabled on the parent page.
- **`overviewDescriptions`**: Boolean controlling whether the `<SectionIndex />`
component displays child page descriptions. Defaults to `true`. Set to `false`
on an index page to hide descriptions for its listing.
- **`order`**: Numeric value for controlling sort order in the sidebar and
section listings. Lower values appear first.
- **`migration_status`**: Status for content migration (e.g., `"new"`,
`"adapt"`, `"aggregate"`).

## See Also

- [Working with the Documentation System Locally](../../how-to/documentation/working-locally.md)
- [Testing Reference](./testing.md) — Test suite documentation
- [Adding Repositories](../../how-to/documentation/adding-repos.md) — How to add new repos
- [Technical Reference](./technical.md) — Source code documentation
- [Architecture](../../explanation/documentation/aggregation-architecture.md) — System design
- [Documentation Workflow](./documentation_workflow.md)
- [Documentation Quality Markers](./writing_good_docs.md)
- [Documentation Aggregator Architecture](./aggregation-architecture.md)
- [How to Documentation - Adding Repos to Aggregate](./adding-repos.md)
- [How to Documentation - Working With the Aggregator Locally](./working-locally.md)
- [Documentation Aggregator Technical Reference](./technical.md)
- [Documentation Aggregator Local Testing Guide](./testing.md)
- [Working with the Documentation Hub on Your Machine](./working-locally.md)
Loading
Loading