Skip to content

Commit 06aeff3

Browse files
ascorbicascorbic
committed
Add live content loaders RFC
1 parent dc3e5c8 commit 06aeff3

File tree

1 file changed

+383
-0
lines changed

1 file changed

+383
-0
lines changed

‎proposals/0055-live-content-loaders.md‎

Lines changed: 383 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,383 @@
1+
<!--
2+
Note: You are probably looking for `stage-1--discussion-template.md`!
3+
This template is reserved for anyone championing an already-approved proposal.
4+
5+
Community members who would like to propose an idea or feature should begin
6+
by creating a GitHub Discussion. See the repo README.md for more info.
7+
8+
To use this template: create a new, empty file in the repo under `proposals/${ID}.md`.
9+
Replace `${ID}` with the official accepted proposal ID, found in the GitHub Issue
10+
of the accepted proposal.
11+
-->
12+
13+
**If you have feedback and the feature is released as experimental, please leave it on the Stage 3 PR. Otherwise, comment on the Stage 2 issue (links below).**
14+
15+
- Start Date: 2025-05-02
16+
- Reference Issues: <!-- related issues, otherwise leave empty -->
17+
- Implementation PR: https://github.com/withastro/astro/pull/13685
18+
- Stage 1 Discussion: https://github.com/withastro/roadmap/discussions/1137
19+
- Stage 2 Issue: https://github.com/withastro/roadmap/issues/1151
20+
- Stage 3 PR: <!-- related roadmap PR, leave it empty if you don't have a PR yet -->
21+
22+
# Summary
23+
24+
Adds support for live data to content collections. Defines a new type of content loader that fetches data at runtime rather than build time, allowing users to get the data with a similar API.
25+
26+
# Example
27+
28+
Defining a live loader for a store API:
29+
30+
```ts
31+
// storeloader.ts
32+
import { type Product, loadStoreData } from "./lib/api.ts";
33+
34+
interface StoreCollectionFilter {
35+
category?: string;
36+
}
37+
38+
interface StoreEntryFilter {
39+
slug?: string;
40+
}
41+
42+
export function storeLoader({
43+
field,
44+
key,
45+
}): LiveLoader<Product, StoreEntryFilter, StoreCollectionFilter> {
46+
return {
47+
name: "store-loader",
48+
loadCollection: async ({ logger, filter }) => {
49+
logger.info(`Loading collection from ${field}`);
50+
// load from API
51+
const products = await loadStoreData({ field, key, filter });
52+
const entries = products.map((product) => ({
53+
id: product.id,
54+
data: product,
55+
}));
56+
return {
57+
entries,
58+
};
59+
},
60+
loadEntry: async ({ logger, filter }) => {
61+
logger.info(`Loading entry from ${field}`);
62+
// load from API
63+
const product = await loadStoreData({
64+
field,
65+
key,
66+
filter,
67+
});
68+
69+
if (!product) {
70+
logger.error(`Product not found`);
71+
return;
72+
}
73+
return {
74+
id: filter.id,
75+
data: product,
76+
};
77+
},
78+
};
79+
}
80+
```
81+
82+
A new `src/live.config.ts` file is introduced that uses the same syntax as the `src/content.config.ts` file:
83+
84+
```ts
85+
// src/live.config.ts
86+
import { defineCollection } from "astro:content";
87+
88+
import { storeLoader } from "@mystore/astro-loader";
89+
90+
const products = defineCollection({
91+
type: "live",
92+
loader: storeLoader({ field: "products", key: process.env.STORE_KEY }),
93+
});
94+
95+
export const collections = { products };
96+
```
97+
98+
The loader can be used in the same way as a normal content collection:
99+
100+
```astro
101+
---
102+
import { getCollection, getEntry } from "astro:content";
103+
104+
// Get all entries in a collection, like other collections
105+
const allProducts = await getCollection("products");
106+
107+
// Live collections optionally allow extra filters to be passed in, defined by the loader
108+
const clothes = await getCollection("products", { category: "clothes" });
109+
110+
// Get entrey by ID like other collections
111+
const productById = await getEntry("products", Astro.params.id);
112+
113+
// Query a single entry using the object syntax
114+
const productBySlug = await getEntry("products", { slug: Astro.params.slug });
115+
---
116+
```
117+
118+
# Background & Motivation
119+
120+
In Astro 5, the content layer API added support for adding diverse content sources to content collections. Users can create loaders that fetch data from any source at build time, and then access it inside a page via `getEntry` and `getCollection`. The data is cached between builds, giving fast access and updates. However there is no method for updating the data store between builds, meaning any updates to the data need a full site deploy, even if the pages are rendered on-demand.
121+
122+
This means that content collections are not suitable for pages that update frequently. Instead, today these pages tend to access the APIs directly in the frontmatter. This works, but leads to a lot of boilerplate, and means users don't benefit from the simple, unified API that content loaders offer. In most cases users tend to individually create loader libraries that they share between pages.
123+
124+
This proposal introduces a new kind of loader that fetches data from an API at runtime, rather than build time. As with other content loaders, these loaders abstract the loading logic, meaning users don't need to understand the details of how data is loaded. These loaders can be distributed as node modules, or injected by integrations.
125+
126+
# Goals
127+
128+
- a new type of **live content loader** that is executed at runtime
129+
- integration with user-facing `getEntry` and `getCollection` functions, allowing developers to use **a familiar, common API** to fetch data
130+
- loader-specific **query and filters**, which a loader can define and pass to the API
131+
- **type-safe** data and query options, defined by the loader as generic types
132+
- support for user-defined **Zod schemas**, executed at runtime, to validate or transform the data returned by the loader.
133+
- support for runtime **markdown rendering**, using a helper function provided in the loader context.
134+
- optional **integration with [route caching](https://github.com/withastro/roadmap/issues/1140)**, allowing loaders to define cache tags and expiry times associated with the data which are then available to the user
135+
136+
# Non-Goals
137+
138+
- server-side caching of the data. Instead it would integrate with the route cache and HTTP caches to cache the full page response, or individual loaders could implement their own API caching.
139+
- rendering of MDX or other content-like code. This isn't something that can be done at runtime.
140+
- support for image processing, either in the Zod schema or Markdown. This is not something that can be done at runtime.
141+
- loader-defined Zod schemas. Instead, loaders define types using TypeScript generics. Users can define their own Zod schemas to validate or transform the data returned by the loader, which Astro will execute at runtime.
142+
- updating the content layer data store. Live loaders return data directly and do not update the store.
143+
- support for existing loaders. They will have a different API. Developers could in theory use shared logic, but the loader API will be different
144+
145+
# Detailed Design
146+
147+
While the user-facing API is similar to the existing content loaders, the implementation is significantly different.
148+
149+
## Loader API
150+
151+
A live loader is an object with two methods: `loadCollection` and `loadEntry`. For libraries that distribute a loader, the convention for these will be for users to call a function that returns a loader object, which is then passed to the `defineCollection` function. This allows the user to pass in any configuration options they need. The loader object is then passed to the `defineCollection` function.
152+
153+
The `loadCollection` and `loadEntry` methods are called when the user calls `getCollection` or `getEntry`. They return the requested data from the function, unlike existing loaders which are responsible for storing the data in the content layer data store.
154+
155+
```ts
156+
// storeloader.ts
157+
158+
export function storeLoader({ field, key }): LiveLoader {
159+
return {
160+
name: "store-loader",
161+
loadCollection: async ({ filter }) => {
162+
// ...
163+
return {
164+
entries: products.map((product) => ({
165+
id: product.id,
166+
data: product,
167+
})),
168+
};
169+
},
170+
loadEntry: async ({ filter }) => {
171+
// ...
172+
return {
173+
id: filter.id,
174+
data: product,
175+
};
176+
},
177+
};
178+
}
179+
```
180+
181+
## Loader execution
182+
183+
Existing content loaders are executed at build time, and the data is stored in the content layer data store, which is then available during rendering. The new live loaders are executed at runtime, and the data is returned directly.
184+
185+
The new `live.config.ts` file has similar syntax to the existing `content.config.ts` file, but it is compiled as part of the build process and included in the build so that it can be called at runtime.
186+
187+
## Filters
188+
189+
For existing collections, `getCollection` accepts an optional function to filter the collection. This filtering is performed in-memory on the data returned from the store. This is not an efficient approach for live loaders, which are likely to be making network requests for the data at request time. Loading all of the entries and then filtering them on the client would cause over-fetching, so it is preferable to filter the data natively in the API.
190+
191+
For this reason, the `getCollection` and `getEntry` methods accept a query object, which is passed to the loader `loadEntry` and `loadCollection` functions. This is an arbitrary object, the type of which is defined by the loader. The loader can then use this filter to fetch the data from the API, according to the API's query syntax. The `getEntry` function also has a shorthand syntax for querying a single entry by ID by passing a string that matches the existing `getEntry` syntax. This is passed to the loader as an object with a single `id` property.
192+
193+
## Type Safety
194+
195+
The `LiveLoader` type is a generic type that takes three parameters:
196+
197+
- `TData`: the type of the data returned by the loader
198+
- `TEntryFilter`: the type of the filter object passed to `getEntry`
199+
- `TCollectionFilter`: the type of the filter object passed to `getCollection`
200+
201+
These types will be used to type the `loadCollection` and `loadEntry` methods.
202+
203+
```ts
204+
// storeloader.ts
205+
import type { LiveLoader } from "astro/loaders";
206+
import { type Product, loadStoreData } from "./lib/api.ts";
207+
208+
interface StoreCollectionFilter {
209+
category?: string;
210+
}
211+
212+
interface StoreEntryFilter {
213+
slug?: string;
214+
}
215+
216+
export function storeLoader({
217+
field,
218+
key,
219+
}): LiveLoader<Product, StoreEntryFilter, StoreCollectionFilter> {
220+
return {
221+
name: "store-loader",
222+
// `filter` is typed as `StoreCollectionFilter`
223+
loadCollection: async ({ filter }) => {
224+
// ...
225+
},
226+
// `filter` is typed as `StoreEntryFilter`
227+
loadEntry: async ({ filter }) => {
228+
// ...
229+
},
230+
};
231+
}
232+
```
233+
234+
The `LiveLoader` type is defined as follows:
235+
236+
```ts
237+
export interface LiveDataEntry<
238+
TData extends Record<string, unknown> = Record<string, unknown>
239+
> {
240+
/** The ID of the entry. Unique per collection. */
241+
id: string;
242+
/** The entry data */
243+
data: TData;
244+
/** Optional cache hints */
245+
cache?: {
246+
/** Cache tags */
247+
tags?: string[];
248+
/** Maximum age of the response in seconds */
249+
maxAge?: number;
250+
};
251+
}
252+
253+
export interface LiveDataCollection<
254+
TData extends Record<string, unknown> = Record<string, unknown>
255+
> {
256+
entries: Array<LiveDataEntry<TData>>;
257+
/** Optional cache hints */
258+
cache?: {
259+
/** Cache tags */
260+
tags?: string[];
261+
/** Maximum age of the response in seconds */
262+
maxAge?: number;
263+
};
264+
}
265+
266+
export interface LoadEntryContext<TEntryFilter = never> {
267+
filter: TEntryFilter extends never ? { id: string } : TEntryFilter;
268+
}
269+
270+
export interface LoadCollectionContext<TCollectionFilter = unknown> {
271+
filter?: TCollectionFilter;
272+
}
273+
274+
export interface LiveLoader<
275+
TData extends Record<string, unknown> = Record<string, unknown>,
276+
TEntryFilter extends Record<string, unknown> | never = never,
277+
TCollectionFilter extends Record<string, unknown> | never = never
278+
> {
279+
/** Unique name of the loader, e.g. the npm package name */
280+
name: string;
281+
/** Load a single entry */
282+
loadEntry: (
283+
context: LoadEntryContext<TEntryFilter>
284+
) => Promise<LiveDataEntry<TData> | undefined>;
285+
/** Load a collection of entries */
286+
loadCollection: (
287+
context: LoadCollectionContext<TCollectionFilter>
288+
) => Promise<LiveDataCollection<TData>>;
289+
}
290+
```
291+
292+
The user-facing `getCollection` and `getEntry` methods exported from `astro:content` will also be typed with these types, so that the user can call them in a type-safe way.
293+
294+
Users will still be able to define a Zod schema inside `defineCollection` to validate the data returned by the loader. If provided, this schema will also be used to infer the returned type of `getCollection` and `getEntry` for the collection, taking precedence over the loader type. This means that users can use the loader to fetch data from an API, and then use Zod to validate or transform the data before it is returned.
295+
296+
## Caching
297+
298+
The returned data is not cached by Astro, but a loader can provide hints to assist in caching the response. This would be designed to integrate with the proposed [route caching API](https://github.com/withastro/roadmap/issues/1140), but could also be used to manually set response headers. The scope of this RFC does not include details on the route cache integration, but will illustrate how the loader can provide hints that can then be used by the route cache or other caching mechanisms.
299+
300+
Loader responses can include a `cache` object that contains the following properties:
301+
302+
- `tags`: an array of strings that can be used to tag the response. This is useful for cache invalidation.
303+
- `maxAge`: a number that specifies the maximum age of the response in seconds. This is useful for setting the cache expiry time.
304+
305+
The loader does not define how these should be used, and the user is free to use them in any way they like.
306+
307+
For example, a loader could return the following object for a collection:
308+
309+
```ts
310+
return {
311+
entries: products.map((product) => ({
312+
id: product.id,
313+
data: product,
314+
})),
315+
cache: {
316+
tags: ["products", "clothes"],
317+
maxAge: 60 * 60, // 1 hour
318+
},
319+
};
320+
```
321+
322+
This would allow the user to tag the response with the `products` and `clothes` tags, and set the expiry time to 1 hour. The user could then use these tags to invalidate the cache when the data changes.
323+
324+
The loader can also provide a `cache` object for an individual entry, allowing fine-grained cache control:
325+
326+
```ts
327+
return {
328+
id: filter.id,
329+
data: product,
330+
cache: {
331+
tags: ["products", "clothes", `product-${filter.id}`],
332+
maxAge: 60 * 60, // 1 hour
333+
},
334+
};
335+
```
336+
337+
When the user calls `getCollection` or `getEntry`, the response will include a cache object that contains the tags and expiry time. The user can then use this information to cache the response in their own caching layer, or pass it to the route cache.
338+
339+
This example shows how cache tags and expiry time in the response headers:
340+
341+
```astro
342+
---
343+
import { getEntry } from "astro:content";
344+
import Product from "../components/Product.astro";
345+
346+
const product = await getEntry("products", Astro.params.id);
347+
348+
Astro.response.headers.set("Cache-Tag", product.cache.tags.join(","));
349+
Astro.response.headers.set("CDN-Cache-Control", `s-maxage=${product.cache.maxAge}`);
350+
---
351+
<Product product={product.data} />
352+
```
353+
354+
# Testing Strategy
355+
356+
Much of the testing strategy will be similar to the existing content loaders, as integration tests work in the same way. It will also be easier to test the loaders in isolation, as they are not dependent on the content layer data store.
357+
358+
End-to-end tests will be added to test the bundling and runtime execution of the loaders.
359+
360+
Type tests will be added to ensure that the generated types are correct, and that the user can call the `getCollection` and `getEntry` methods with the correct types.
361+
362+
# Drawbacks
363+
364+
- This is a significant addition to the content collections API, and will work to implement and document.
365+
- This is a new API for loader developers to learn, and existing loaders cannot be trivially converted. While the API and mental model are simpler than the existing content loaders, it is still a new API that will require some work for developers to implement.
366+
- Unlike the content layer APIs, there will not be any built-in loaders for this API, so it will be up to the community to implement them. There will be limited value to this featured until there are a number of loaders available.
367+
- The user-facing API is similar but not identical to the existing content loaders, which may cause confusion for users. The behavior is also different, as the data is not stored in the content layer data store. This means that users will need to understand the difference between the two APIs.
368+
369+
# Alternatives
370+
371+
- **Do nothing**: For regularly-updated data they will need to use any APIs directly in the frontmatter. This is the current approach, and while it works, it is not ideal. It means that users need to implement their own loading logic. This tends to involve a lot of boilerplate, and there is no common API for accessing the data.
372+
- **Add support for updating the content layer data store at runtime**: This would allow users to update the data in the content layer data store, for example via a webhook. This would be significantly more complex and would require a lot of work to implement. It would also require users provision third-party database services to support this in a serverless environment.
373+
374+
# Adoption strategy
375+
376+
- This would be released as an experimental feature, with a flag to enable it.
377+
- As live collections are defined in a new file, existing sites will not be affected by this change unless they add new collections that use it.
378+
379+
# Unresolved Questions
380+
381+
- The **proposed name of the file** is potentially confusing. While the name is analogous to the existing `content.config.ts` file, it is not really a configuration file, and has more in common with actions or middleware files. Would it better to not use `config` in the name, or would that be confusing when compared to the existing file? Possible alternatives: `src/live.ts`, `src/live-content.ts`, `src/live-content.config.ts`, `src/content.live.ts`...
382+
383+
- The way to handle **rendering Markdown**, or even whether to support it. The most likely approach is to add a `renderMarkdown` helper function to the loader context that can be used to render Markdown to HTML, which would be stored in the `entry.rendered.html` field, similar to existing collections. This is essentially a pre-configured instance of the renderer from the `@astrojs/markdown-remark` integration, using the user's Markdown config. This helper would be also likely be added to existing loaders, as users have complained that is hard to do manually. This may be extending the scope too far, but it is a common use case for loaders. We may not want to encourage rendering Markdown inside loaders, as it could lead to performance issues. It may be confusing that image processing, and nor is MDX.

0 commit comments

Comments
 (0)