astro-slop

Astro integration for LLMs: markdown versions of every page, llms.txt, and alternate links.

Astro
JavaScript
AI

Content slop for LLMs to slurp. The most complete and flexible Astro integration for LLMs and generating markdown versions of pages.

.md siblings for every page. Flexible, bottom-up, fully customizable approach: you create simple endpoints with the integration’s helpers by example.
HTML → Markdown conversion for pages that don’t have a markdown source from collections. MDX cleaning for MDX sources.
<link rel="alternate"> injection on every HTML page.
Auto-generated—and customizable!—llms.txt and llms-full.txt.
HTTP content negotiation for text/markdown in dev and SSR.

Rationale¶

There are a number of LLM/llms.txt/Markdown-exporting integrations for Astro. Most promise a simple config or even zero-configuration, but are no good for real-world cases.

Some only generate llms.txt; not super useful. Some generate markdown, but by converting all HTML automatically—can result in a mess. Some try to use collections for export, but are not flexible enough.

The top-down approach does not work. We need to go for the bottom-up one. The “correct” approach already exists in Astro: create an endpoint; the same approach used in @astrojs/rss. The integration, or Astro’s role, is to help you do that by giving examples and helper functions, and automating as much as possible.

astro-slop does exactly that: it gives you enough tooling to create your own simple, short endpoints (posts.md.ts next to posts.astro) where you can export markdown by yourself; next, it collects that markdown for llms.txt, auto-injects alternate links and does a ton of other stuff to reduce manual changes.

Features¶

The md tagged template for writing .md.ts endpoints. Returns a Response with text/markdown content-type. Real newlines, no \n escapes; arrays auto-join with newlines so a posts.map(...) interpolation becomes a bullet list directly; template indentation dedented based on template strings only, so interpolated values aren’t affected. Plus inline helpers—md.link, md.heading, md.section—and a md.string variant for sub-composition or custom Response shapes.
Tiny user-facing surface—one integration call (slop()), one tagged template (md), one HTML-fallback helper (fromHtml), one MDX cleaner (cleanMdx), and two “default content” hooks (defaultLlmsTxt, defaultLlmsFullTxt) for users who want to extend the auto-generated indexes. Six exports, all documented below.
<link rel="alternate" type="text/markdown"> auto-injected into every HTML page that has a .md sibling. Two mutually-exclusive paths so it works across Astro output modes: runtime middleware handles dev / SSR responses, the astro:build:done HTML rewriter handles static prerender output. No double-injection, no version-dependent fallback.
cleanMdx(body)—strips MDX-specific syntax (top-level import/export statements, JSX components) from raw collection entry bodies so the result is plain Markdown. HTML is preserved as-is (it’s valid Markdown, LLMs parse it fine, and user-written HTML often carries semantic intent). Use it whenever you interpolate entry.body into a markdown response — per-entry endpoints, composed pages, anywhere .mdx content might leak JSX chrome.
fromHtml(url, options)—for static .astro pages without a markdown source. Fetches the rendered HTML and runs it through Defuddle for main-content extraction, chrome stripping (nav, footer, sidebars, share buttons), and Markdown conversion in a single pass. Lazy-loaded with @vite-ignore dynamic imports so Defuddle and its DOM dependency don’t appear in your bundle if fromHtml is never called.
/llms.txt auto-assembled from your .md routes per the llmstxt.org spec. H1 site name, blockquote summary, then a flat bullet list of every route with title + description—title and description extracted automatically from each endpoint’s emitted frontmatter or first H1. Override by writing your own src/pages/llms.txt.ts and calling defaultLlmsTxt(context) to splice the auto-gen body into your own structure (sections, recommended reading, references, etc.).
/llms-full.txt—same site header plus each route’s full body content inlined as ### Title\n\n${body} sections. One document an LLM can ingest end-to-end. Same override pattern via defaultLlmsFullTxt(context).
HTTP content negotiation—when a request’s Accept header explicitly ranks text/markdown above text/html, the middleware redirects to the .md sibling. RFC 9110-compliant, integration-internal—users add no config. LLM-aware clients (curl scripts, RSS readers, well-behaved crawlers) get markdown automatically; browsers—whose default Accept never lists markdown—keep getting HTML.

Install¶

npm install astro-slop
# or: bun add astro-slop
# or: pnpm add astro-slop

Configure¶

// astro.config.mjs
import { defineConfig } from "astro/config";
import slop from "astro-slop";

export default defineConfig({
  integrations: [
    // ...
    slop({
      siteName: "Your Site",
      siteDescription: "One-sentence summary used in /llms.txt blockquote.",
    }),
  ],
});

Option	Type	Default	Effect
`siteName`	`string`	`"Site"`	H1 of `/llms.txt` and `/llms-full.txt`
`siteDescription`	`string`	`undefined`	Blockquote line under the H1
`injectAlternateLink`	`boolean`	`true`	Inject `<link rel="alternate">` into HTML head
`llmsFullTxt`	`boolean`	`true`	Generate `/llms-full.txt`
`contentNegotiation`	`boolean`	`true`	Redirect HTML→`.md` when `Accept` prefers markdown (request-time only—dev / SSR)

Authoring `.md` endpoints¶

The integration doesn’t “auto-generate” markdown content. Instead, it goes for flexibility and custom-fitted approach.

You write a small .md.ts endpoint per page that should have a markdown view, the same way you’d write rss.xml.ts for an RSS feed. The integration handles the response wrapper, alt-link injection, and /llms.txt indexing.

Per-entry route (collection-driven)¶

For dynamic routes backed by a content collection ([slug].astro), the markdown sibling re-exports getStaticPaths from the HTML page and emits the entry’s frontmatter + body. Wrap the body in cleanMdx() to strip JSX/imports for .mdx collections:

// src/pages/posts/[slug].md.ts
import type { APIRoute } from "astro";
import type { CollectionEntry } from "astro:content";
import { cleanMdx, md } from "astro-slop";

export { getStaticPaths } from "./[slug].astro";

interface Props {
  post: CollectionEntry<"posts">;
}

export const GET: APIRoute<Props> = ({ props }) => md`---
title: ${props.post.data.title}
description: ${props.post.data.description}
---

${cleanMdx(props.post.body ?? "")}`;

The getStaticPaths re-export means the .md.ts shares Astro’s built path resolution with the HTML page—no duplicate filtering or sorting. Astro’s module cache means the underlying getCollection query runs once per build.

Listing route¶

For listing pages backed by a collection, the .astro exports its data; the .md.ts imports it:

---
// src/pages/posts/index.astro
import { getCollection } from "astro:content";

export const posts = (await getCollection("posts", ({ data }) => !data.draft))
  .sort((a, b) => /* however you sort */);
---

<Layout>
  {posts.map((p) => <article>{p.data.title}</article>)}
</Layout>

// src/pages/posts.md.ts
import type { APIRoute } from "astro";
import { md } from "astro-slop";
import { posts } from "./posts/index.astro";

export const GET: APIRoute = () => md`
# Posts

${posts.map((p) =>
  `- ${md.link(p.data.title, `/posts/${p.id}.md`)} — ${p.data.description}`
)}
`;

The export keyword is the only refactor needed—your filter/sort logic stays in the .astro, the HTML render reads the same posts constant via closure.

Composed page (homepage with intro + dynamic content, for instance)¶

---
// src/pages/index.astro
import { getEntry, getCollection } from "astro:content";

export const home = await getEntry("pages", "home");
export const recent = (await getCollection("posts")).slice(0, 5);
---

<Layout>
  <Content>{home!.body}</Content>
  <PostCardList posts={recent} />
</Layout>

// src/pages/index.md.ts
import type { APIRoute } from "astro";
import { cleanMdx, md } from "astro-slop";
import { home, recent } from "./index.astro";

export const GET: APIRoute = () => md`
${cleanMdx(home!.body)}

## Recent posts

${recent.map((p) =>
  `- ${md.link(p.data.title, `/posts/${p.id}.md`)} — ${p.data.description}`
)}
`;

home!.body is the raw markdown from your collection entry—interpolated as-is, no HTML round-trip. Wrap it in cleanMdx() if your homepage is a .mdx file with import statements or JSX components you don’t want in the markdown output (HTML is left alone—valid Markdown, semantic intent preserved). recent.map(...) becomes a bullet list because the md tag joins arrays with newlines automatically.

HTML → Markdown for static `.astro` pages without a markdown source¶

For pages whose content lives in HTML (an about.astro with prose mixed into JSX, for example), use fromHtml:

// src/pages/about.md.ts
import { fromHtml } from "astro-slop";

export const GET = fromHtml("/about");

fromHtml fetches the HTML at the given URL and runs it through Defuddle, which detects the main content (stripping nav, footer, sidebars), cleans it up, and emits Markdown. No selector or exclude config needed—Defuddle’s heuristics handle the common cases. Pass defuddleOptions to forward extra Defuddle settings if you need them.

Works in astro dev and SSR modes. For pure static prerender, prefer the source-driven patterns above—HTML → MD round-trip is lossy for code blocks (Shiki) and embedded components.

Customizing `/llms.txt`¶

The integration auto-generates /llms.txt as a flat bullet list of every .md route. To group, reorder, add sections, or replace entirely, write your own endpoint at src/pages/llms.txt.ts—when present, the plugin’s auto-injection skips and yours wins.

defaultLlmsTxt(context) splices the auto-generated body into your wrapping content. Pass the endpoint’s APIContext so it works in dev and SSR (where it fetches each .md route at request time); in static builds it returns a sentinel that the integration’s astro:build:done hook replaces with the disk-derived body. Simplest case—append a static “About” section:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { md, defaultLlmsTxt } from "astro-slop";

export const GET: APIRoute = async (context) => {
  const body = md.string`
${await defaultLlmsTxt(context)}

## About this site

Custom site description, hours, contact info, or anything else you want under a top-level heading.
`;

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

md.string is the variant of the md tag that returns a string instead of a Response—useful when you need to construct the Response yourself (here, with text/plain for the .txt extension).

If you want the extra section sourced from a content collection (e.g., a list of recommended external resources), the same pattern works—load the collection inside the handler:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { md, defaultLlmsTxt } from "astro-slop";
import { getCollection } from "astro:content";

export const GET: APIRoute = async (context) => {
  const links = await getCollection("links");

  const body = md.string`
${await defaultLlmsTxt(context)}

## Recommended reading

${links.map((l) => `- ${md.link(l.data.title, l.data.url)} — ${l.data.summary}`)}
`;

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

Customizing `/llms-full.txt`¶

/llms-full.txt reuses /llms.txt’s site header (# Site plus the > description blockquote) and then inlines every route’s full body content as a flat list of ### Title sections instead of bullet links. Same source data, different density.

# My Site

> Site description

### Welcome to My Site

[full body of /index.md, frontmatter stripped]

### About

[full body of /about.md]

### My First Entry

[full body of /entries/first-entry.md]

### Hello, World

[full body of /entries/hello-world.md]

Override by writing src/pages/llms-full.txt.ts:

// src/pages/llms-full.txt.ts
import type { APIRoute } from "astro";
import { defaultLlmsFullTxt } from "astro-slop";

export const GET: APIRoute = async (context) => {
  const body = await defaultLlmsFullTxt(context);
  return new Response(`${body}\n## Notes\n\nExtra context here.\n`, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

Each section uses an ### heading with the route’s title (frontmatter title: or first H1) followed by the body with leading frontmatter stripped—so titles aren’t repeated and the document reads top-to-bottom.

API reference¶

import {
  default as slop,        // the integration
  md,                     // tagged template, returns Response
  fromHtml,               // HTML → Markdown helper, returns GET handler
  cleanMdx,               // strip MDX-specific syntax from a body string
  defaultLlmsTxt,         // returns auto-gen llms.txt body as string
  defaultLlmsFullTxt,     // returns auto-gen llms-full.txt body as string
} from "astro-slop";

The `md` tag¶

Tagged template that returns a Response with Content-Type: text/markdown; charset=utf-8.

md`# Hello`
// → Response { body: "# Hello\n", headers: { "Content-Type": "text/markdown; charset=utf-8" } }

Behavior:

Real newlines work—no \n escapes needed
Arrays in interpolation are joined with \n automatically
Leading whitespace from template indentation is dedented (computed from template strings only—interpolated values aren’t affected)
Output has \n-terminated lines and a single trailing newline

const items = ["- a", "- b", "- c"];

md`
  # Heading

  ${items}
`
// → Response with body:
// "# Heading\n\n- a\n- b\n- c\n"

`md.string`¶

Same template behavior as md, returns a string instead of a Response. Use for sub-composition or when you need a custom Response shape (e.g., text/plain for .txt URLs):

const intro = md.string`
  # Intro

  Welcome.
`;

const fullDoc = md.string`
  ${intro}

  ## Section 2

  More content.
`;

return new Response(fullDoc, { headers: { "Content-Type": "text/plain" } });

Inline helpers¶

These return strings—call them inside template interpolations:

Helper	Returns	Example
`md.link(text, url)`	`[text](https://github.com/yaroslav/astro-slop/blob/main/url)` with `[`/`]` in `text` escaped	`md.link("Hello [world]", "/foo")` → `"[Hello \\[world\\]](/foo)"`
`md.heading(depth, text)`	Markdown heading	`md.heading(2, "Posts")` → `"## Posts"`
`md.section(heading, body)`	`${heading}\n\n${body}`	`md.section("## Section", "- a")` → `"## Section\n\n- a"`

`fromHtml(url, options)`¶

Returns an Astro GET handler that fetches HTML from url and runs it through Defuddle for content extraction + Markdown conversion in one pass.

fromHtml("/about");

// Optionally forward extra options to Defuddle:
fromHtml("/about", {
  defuddleOptions: { /* ... */ },
});

Option	Type	Default	Effect
`defuddleOptions`	object	`undefined`	Pass-through to Defuddle. Whatever you set is merged with a forced `markdown: true` (the integration always emits markdown — your `markdown: false` is ignored). See Defuddle’s options for the full set.

Defuddle handles main-content detection (<main> / <article> / smart fallbacks) and chrome removal automatically—no selector or exclude knobs to tune. The integration lazy-loads Defuddle and linkedom (its DOM dependency) on first request so they don’t appear in the bundle if fromHtml is never called.

`cleanMdx(body)`¶

Strips MDX-specific syntax from a content collection entry’s raw body, so the result is plain Markdown suitable for .md output.

What gets stripped:

Top-level import and export statements
Self-closing JSX components: <Component prop="..." />
Paired JSX components: <Component>...content...</Component>—content kept

What’s preserved:

All HTML—valid in Markdown, LLMs handle it, often carries semantic intent (<address>, <time>, <cite>, etc.). If you want HTML scrubbed, run a separate sanitizer pass.
All actual prose, lists, code blocks, links, emphasis

import { cleanMdx } from "astro-slop";

const cleaned = cleanMdx(`
import Note from '../components/Note.astro';

# Title

<Note type="tip">Just a tip.</Note>

<span class="badge">Featured</span> introduction here.
`);
// → "# Title\n\nJust a tip.\n\n<span class=\"badge\">Featured</span> introduction here.\n"

The <span> is preserved (HTML is valid Markdown). The MDX import is stripped, and the <Note> JSX component is unwrapped to keep its content.

Use it on entry.body for collection entries that may use MDX:

md`...${cleanMdx(entry.body ?? "")}`

`defaultLlmsTxt(context?: APIContext): Promise<string>`¶

Returns the auto-generated /llms.txt body. Use inside a custom src/pages/llms.txt.ts to extend.

Live request (dev / SSR): fetches each .md route over HTTP using context.request.url as the base, crawls response bodies for links to per-entry siblings, and returns the assembled body. Pass context from your endpoint to get this path.
Static prerender, or no context: returns a sentinel string. The integration’s astro:build:done hook reads dist/llms.txt, replaceAlls the sentinel with the disk-walked auto-gen body, and writes it back. Your wrapping content (custom sections, etc.) is preserved; the auto-gen content lands in place of the sentinel.

`defaultLlmsFullTxt(context?: APIContext): Promise<string>`¶

Returns the auto-generated /llms-full.txt body—site header followed by each route’s full content inlined as ### Title\n\n${body} sections. Same prerender / runtime split as defaultLlmsTxt; pass context from your endpoint.

Examples¶

Collection-backed routes¶

A common case: a content collection of .md/.mdx files (the example uses posts, but any name works), plus a listing page and per-entry detail pages. Two endpoint files cover both.

src/pages/posts/index.astro—the listing. Hoist the posts array to export const so the .md.ts sibling can import it:

---
import { getCollection } from "astro:content";

export const posts = (await getCollection("posts"))
  .sort((a, b) => b.data.date.valueOf() - a.data.date.valueOf());
---

<html>
  <body>
    <h1>Posts</h1>
    <ul>
      {posts.map((p) => <li><a href={`/posts/${p.id}`}>{p.data.title}</a></li>)}
    </ul>
  </body>
</html>

src/pages/posts/index.md.ts—the listing’s markdown sibling:

import type { APIRoute } from "astro";
import { md } from "astro-slop";
import { posts } from "./index.astro";

export const GET: APIRoute = () => md`
# Posts

${posts.map((p) => `- ${md.link(p.data.title, `/posts/${p.id}.md`)} — ${p.data.description}`)}
`;

src/pages/posts/[slug].md.ts—per-post markdown. Re-export getStaticPaths from the HTML page so both routes resolve from the same source:

import type { APIRoute } from "astro";
import type { CollectionEntry } from "astro:content";
import { cleanMdx, md } from "astro-slop";

export { getStaticPaths } from "./[slug].astro";

export const GET: APIRoute<{ post: CollectionEntry<"posts"> }> = ({ props }) =>
  md`---
title: ${props.post.data.title}
description: ${props.post.data.description}
---

${cleanMdx(props.post.body ?? "")}`;

/llms.txt and /llms-full.txt are auto-generated from these endpoints. <link rel="alternate" type="text/markdown"> is auto-injected into /posts and every /posts/<slug> HTML page.

Single page from a markdown collection¶

A single page sourced from one entry in a content collection (e.g., a pages collection containing home.md, about.md, etc.). The .astro page renders the entry’s content; the .md.ts sibling emits the raw body.

src/pages/about.astro—exports the entry so the sibling can read its body:

---
import { getEntry, render } from "astro:content";

export const aboutEntry = await getEntry("pages", "about");
const { Content } = await render(aboutEntry!);
---

<html>
  <body>
    <Content />
  </body>
</html>

src/pages/about.md.ts—interpolates the raw body, runs through cleanMdx to strip MDX/styling tags:

import type { APIRoute } from "astro";
import { cleanMdx, md } from "astro-slop";
import { aboutEntry } from "./about.astro";

export const GET: APIRoute = () => md`${cleanMdx(aboutEntry!.body ?? "")}`;

That’s a single .md route at /about.md. The integration auto-injects the alt-link and includes the page in /llms.txt.

Single HTML page that needs conversion¶

For pages whose content is hard-coded into JSX in an .astro file (no markdown source available), use fromHtml to fetch the rendered HTML and convert via Defuddle:

src/pages/contact.astro (illustrative—typical handwritten page with prose mixed into JSX):

---
const email = "hello@example.com";
---

<html>
  <body>
    <main>
      <h1>Contact</h1>
      <p>Reach out at <a href={`mailto:${email}`}>{email}</a>.</p>
      <p>Office hours: Mon-Fri 9-5 PT.</p>
    </main>
  </body>
</html>

src/pages/contact.md.ts—one line:

import { fromHtml } from "astro-slop";

export const GET = fromHtml("/contact");

fromHtml fetches /contact, runs Defuddle (main-content extraction + chrome stripping + Markdown conversion), and returns the result. The integration handles routing, alt-link injection, and /llms.txt inclusion automatically.

Mix and match freely: use the md tag for collection-backed routes (lossless, source-driven) and fromHtml for static .astro content (HTML→MD via Defuddle). The integration’s manifest treats them uniformly—alt-links and /llms.txt entries work the same regardless of which shape produced the .md URL.

Customizing `/llms.txt`¶

The auto-generated /llms.txt is a good default. To extend it (or replace it entirely), write your own endpoint at src/pages/llms.txt.ts—Astro’s routing prefers user-defined files over integration-injected ones, so yours wins.

Append a static “About this site” section to the auto-gen:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { md, defaultLlmsTxt } from "astro-slop";

export const GET: APIRoute = async (context) => {
  const body = md.string`
${await defaultLlmsTxt(context)}

## About this site

Custom site description, hours, contact info, or whatever else you want under a top-level heading.
`;

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

Or pull the extra section from a content collection—useful for adding curated external links the integration can’t auto-discover:

// src/pages/llms.txt.ts
import type { APIRoute } from "astro";
import { md, defaultLlmsTxt } from "astro-slop";
import { getCollection } from "astro:content";

export const GET: APIRoute = async (context) => {
  const links = await getCollection("links");

  const body = md.string`
${await defaultLlmsTxt(context)}

## Recommended reading

${links.map((l) => `- ${md.link(l.data.title, l.data.url)} — ${l.data.summary}`)}
`;

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

Customizing `/llms-full.txt`¶

Same pattern as /llms.txt. Write src/pages/llms-full.txt.ts and call defaultLlmsFullTxt(context) to wrap or extend.

Append a static “Notes for AI agents” section:

// src/pages/llms-full.txt.ts
import type { APIRoute } from "astro";
import { defaultLlmsFullTxt } from "astro-slop";

export const GET: APIRoute = async (context) => {
  const body = await defaultLlmsFullTxt(context);
  return new Response(
    `${body}\n## Notes for agents\n\nThis content is licensed CC BY 4.0. Attribution: https://example.com.\n`,
    { headers: { "Content-Type": "text/plain; charset=utf-8" } },
  );
};

Or pre-process the auto-gen—e.g., add a license preamble at the top:

// src/pages/llms-full.txt.ts
import type { APIRoute } from "astro";
import { defaultLlmsFullTxt } from "astro-slop";

export const GET: APIRoute = async (context) => {
  const preamble = `<!--\nThis document is licensed CC BY 4.0.\nGenerated ${new Date().toISOString()}.\n-->\n\n`;
  const body = await defaultLlmsFullTxt(context);

  return new Response(preamble + body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
};

How content negotiation works¶

When a request comes in to a route that has a .md sibling, the middleware checks the request’s Accept header. If text/markdown (or text/x-markdown) is ranked strictly higher than text/html, the middleware returns a 302 redirect to the .md sibling.

GET /posts/hello-world
Accept: text/markdown

→ 302 Location: /posts/hello-world.md

Browsers never trigger this—their default Accept is text/html, application/xhtml+xml, ..., */* with no markdown listed, so HTML always wins. Clients that explicitly request markdown (curl scripts, RSS readers, well-behaved LLM crawlers) get redirected automatically.

The redirect carries a Vary: Accept header so downstream caches (Cloudflare, Vercel edge, nginx, etc.) treat it as Accept-conditional. Without Vary, a CDN could serve a redirect cached for an HTML-preferring client to a markdown-preferring one (or vice versa).

Test it locally with bun run dev:

curl -L -H "Accept: text/markdown" http://localhost:4321/posts/hello-world
# ↑ -L follows the 302; you get the .md content

curl http://localhost:4321/posts/hello-world
# ↑ default Accept; you get HTML as before

Important limitation: content negotiation only works at request time—astro dev and SSR mode. Static prerender bakes one file per URL into dist/, and the deployed file server has no awareness of Accept. Static-mode users would configure server-level negotiation (Apache mod_negotiation, nginx try_files + Accept matching, Cloudflare Workers, etc.) themselves; the integration can’t substitute for that.

To disable: slop({ contentNegotiation: false }).

How alt-link injection works¶

The integration handles <link rel="alternate" type="text/markdown"> injection through two complementary paths, one per execution context:

Runtime middleware—registered via addMiddleware in astro:config:setup, fires on every dev/SSR response. The middleware skips prerendered routes entirely (those have no live request and would just trigger a headers access warning). For HTML responses with a registered .md sibling, it injects the <link> tag before </head>.
Build-time HTML rewriting—runs in astro:build:done, walks the emitted dist/**/*.html files, derives each one’s URL pathname back from its file path, looks up the manifest, and injects the link tag inline. Handles all static prerender output.

The two paths are mutually exclusive: middleware handles request-time output, build:done handles prerender output. There’s no overlap, no double-injection.

Both paths share the same insertion logic (in src/alt-link.ts)—same regex for finding the </head> tag, same already-injected detection, same tag format. The duplication of execution context (request-time vs. build-time) doesn’t translate into duplicated logic.

The route manifest is built in astro:routes:resolved from Astro’s resolved route table—every project route ending in .md becomes a manifest entry paired with its HTML sibling URL pattern. Static and dynamic patterns are handled identically.

To disable: slop({ injectAlternateLink: false }).

How `/llms.txt` and `/llms-full.txt` are assembled¶

Two paths produce the same logical output from different sources:

Static build — at astro:build:done, the integration walks dist/*.md, matches each file to a route in the manifest, parses title (frontmatter title: or first H1) and description (frontmatter description:), strips frontmatter from the body, and writes dist/llms.txt and dist/llms-full.txt. If the user’s src/pages/llms.txt.ts produced a sentinel-bearing file at prerender, the build:done hook splices the auto-gen body in place of the sentinel — preserving the user’s wrapping content.

Dev / SSR — at request time, the injected /llms.txt and /llms-full.txt route handlers fetch every static .md route over HTTP using the incoming request’s URL as the base, then crawl response bodies for markdown links ([…](https://github.com/yaroslav/astro-slop/blob/main/foo.md)) that match per-entry patterns from the manifest, recursively fetching those. Per-entry routes that aren’t linked from any static parent listing won’t be enumerated at request time (they still appear in static builds, since the disk walk doesn’t depend on link structure).

/llms.txt formats each record as a - [title](https://github.com/yaroslav/astro-slop/blob/main/mdUrl) — description bullet, in URL-sorted order. /llms-full.txt formats each record as a ### title\n\n${body} section, in URL-sorted order, with frontmatter stripped from the body.

Auto-gen output is a single flat list — no ## Pages / ## Posts bucketing. To group, reorder, or add custom sections, write your own src/pages/llms.txt.ts and arrange your output around defaultLlmsTxt(context).

Verify your setup¶

Two third-party tools are useful for sanity-checking that your site exposes markdown the way LLM-aware clients expect:

acceptmarkdown.com—sends a request with Accept: text/markdown and checks that your site responds with markdown (via content negotiation or the .md sibling).
isitagentready.com—broader audit: checks for /llms.txt, <link rel="alternate" type="text/markdown"> injection, content negotiation, and other agent-readiness signals.
contentsignals.org—content-signals standard for declaring how your content can be used by AI agents and crawlers; useful complement to the structural checks above.

Run your site through both after deploying. With astro-slop configured per this README, you should pass cleanly: /llms.txt is auto-generated, <link rel="alternate"> is injected on every HTML page that has a .md sibling, and content negotiation redirects markdown-preferring clients automatically.

Caveats¶

fromHtml in static prerender: Pre-rendering may run before the corresponding HTML file exists, so fromHtml can fail for purely static builds. Use astro dev to verify, or rely on the md tag’s source-driven patterns above.
.mdx raw bodies: entry.body for .mdx collection entries is the raw source—including import statements and JSX components like <Note> or <Hero />. Wrap it in cleanMdx() whenever you interpolate entry.body into a markdown response (per-entry endpoints AND composed pages like a homepage). Without it, the LLM sees raw MDX syntax alongside the content.
HTML is preserved: cleanMdx deliberately leaves HTML alone. HTML is valid Markdown, LLMs handle it, and user-written HTML often carries semantic intent (<address>, <time>, <cite>). If you want HTML scrubbed, run a separate sanitizer pass after cleanMdx.
MDX cleanup is regex-based: cleanMdx handles common cases (top-level imports/exports, simple JSX components) but isn’t a full MDX parser. For complex MDX (nested JSX expressions, fragments, conditional rendering), inspect the output and extend if needed.
Override detection extensions: the integration checks .ts, .js, .mts, .mjs, and .astro for user override files at src/pages/llms.txt.* and src/pages/llms-full.txt.*. Other extensions aren’t checked—the integration’s auto-gen will collide and Astro will emit a warning.
md.link escape coverage: only escapes [ and ] in link text. Most other markdown-special chars don’t break links—but if your link text contains parentheses, ampersands, or HTML entities and you need strict correctness, post-process or write the link by hand.
State sharing: the integration shares state across module realms via globalThis[Symbol.for("astro-slop:state")]. This works for single-process Astro builds (the standard case). Multi-process setups (workers, custom isolation) may need a different mechanism—file an issue if you hit this.
Per-entry route enumeration in dev / SSR: at request time, the injected /llms.txt and /llms-full.txt handlers fetch every static .md route and crawl response bodies for […](https://github.com/yaroslav/astro-slop/blob/main/foo.md) links to discover dynamic per-entry routes. A per-entry route that isn’t linked from any static parent listing won’t be enumerated in dev / SSR responses (it still appears in static builds, since the disk walk doesn’t depend on link structure). The README’s listing-route pattern (/posts.md linking each /posts/<slug>.md) is the canonical way to make per-entry routes discoverable.

Acknowledgements¶

Thanks to Steph Ango for Defuddle, the library fromHtml uses for content extraction and HTML → Markdown conversion.

Shout out to Pavel Grinchenko and Margarita Klubochkina.

License¶

MIT

Rationale¶

Features¶

Install¶

Configure¶

Authoring .md endpoints¶

Per-entry route (collection-driven)¶

Listing route¶

Composed page (homepage with intro + dynamic content, for instance)¶

HTML → Markdown for static .astro pages without a markdown source¶

Customizing /llms.txt¶

Customizing /llms-full.txt¶

API reference¶

The md tag¶

md.string¶

Inline helpers¶

fromHtml(url, options)¶

cleanMdx(body)¶

defaultLlmsTxt(context?: APIContext): Promise<string>¶

defaultLlmsFullTxt(context?: APIContext): Promise<string>¶

Examples¶

Collection-backed routes¶

Single page from a markdown collection¶

Single HTML page that needs conversion¶

Customizing /llms.txt¶

Customizing /llms-full.txt¶

How content negotiation works¶

How alt-link injection works¶

How /llms.txt and /llms-full.txt are assembled¶

Verify your setup¶

Caveats¶

Acknowledgements¶

License¶

Authoring `.md` endpoints¶

HTML → Markdown for static `.astro` pages without a markdown source¶

Customizing `/llms.txt`¶

Customizing `/llms-full.txt`¶

The `md` tag¶

`md.string`¶

`fromHtml(url, options)`¶

`cleanMdx(body)`¶

`defaultLlmsTxt(context?: APIContext): Promise<string>`¶

`defaultLlmsFullTxt(context?: APIContext): Promise<string>`¶

Customizing `/llms.txt`¶

Customizing `/llms-full.txt`¶

How `/llms.txt` and `/llms-full.txt` are assembled¶