Routes as data

A proposal, not built yet. Every page on the site gets a row in the content database, including the ones written as TSX in the repo, so the nav, breadcrumbs, sitemap and feeds can all come from one query instead of from the file system and hand-edited lists.

The problem

The site has two kinds of pages, and only one of them is data. Pages stored in the database have a title, a parent, tags and a publish date, so breadcrumbs and feeds already work for them. Pages written as page.tsx files in the repo have none of that. They exist only as folders.

  • The nav is a hand-edited array, so every new section means editing code.
  • Breadcrumbs can't label /docs/code-stress, because nothing says it's called “Code stress test”.
  • Nothing at runtime can ask “what pages exist?”, so a sitemap, a search index or “recently updated” can only see half the site.

The idea

A TSX page keeps rendering from the repo, exactly as it does now. What changes is that it also gets a row in the database holding its metadata: title, parent, tags, visibility, position in the nav. The code is the body; the row is everything about it. A new source column says which kind of page a row is.

sourceBody lives inThe row exists becauseRendered by
dbThe row's bodySomeone created it in goatboatThe catch-all route
codeA page.tsx in the repoThe file exists, and the sync registered itNext.js, as today

One tree per site

A path like /docs only means something on a particular site, and the plan is for one database to serve several sites. So a new sites table, a site_id on every content row, and paths that are unique per site instead of globally. Every existing row becomes part of the jaygriff site.

tsschema.ts (additions)
export const sites = sqliteTable("sites", {
  id: id(),
  slug: text().notNull().unique(),   // "jaygriff"
  name: text().notNull(),
  baseUrl: text("base_url").notNull(),
});

// New columns on content
siteId: text("site_id").notNull().references(() => sites.id),
source: text({ enum: ["db", "code"] }).notNull().default("db"),
file: text(),                         // set when source = 'code', e.g. app/tests/feed/page.tsx
missingSince: text("missing_since"),  // set by the sync when a code route disappears

// statuses gains "unlisted"; path becomes unique per site
uniqueIndex("uq_content_site_path").on(t.siteId, t.path)

The sync

1. Discover

A script in the site walks src/app and turns every page.tsx into a path, following Next's own rules.

textdiscovery
src/app/
  page.tsx                      →  /
  docs/page.tsx                 →  /docs
  docs/code-blocks/page.tsx     →  /docs/code-blocks
  tests/page.tsx                →  /tests
  tests/feed/page.tsx           →  /tests/feed
  tests/content/[...path]/…     →  skipped: dynamic, a template rather than a page
  (marketing)/about/page.tsx    →  /about   (route groups don't add a segment)
  _components/…                 →  skipped: private folder

2. Read metadata from code

Each page's export const metadata = { title, description } is Next's own convention, so it also gives the page a real <title> and description for search engines. The script reads it from the source text without running the page.

3. Send the full list to goatboat

httprequest.http
POST /api/sites/jaygriff/routes/sync
Authorization: Bearer <token>

{
  "routes": [
    { "path": "/tests", "file": "app/tests/page.tsx", "title": "Tests", "description": "Pages that read from the content API." },
    { "path": "/tests/feed", "file": "app/tests/feed/page.tsx", "title": "Content feed", "description": null },
    { "path": "/docs/code-blocks", "file": "app/docs/code-blocks/page.tsx", "title": "Code blocks", "description": "…" }
  ]
}

4. Goatboat reconciles

CaseWhat happens
A path in the list with no rowCreate a code row. Parent comes from the path; title and description from metadata
A path in the list that already has a rowUpdate only the file pointer. Everything else is left alone
A code row whose path is no longer in the listSet missing_since. Never deleted automatically
A path that's back after going missingClear missing_since

Who owns what

This is the rule that keeps the sync safe to run any number of times: code decides whether a page exists; the database decides everything said about it.

FieldOwnerWhy
path, fileCodeThe file system is the truth about which pages exist
Initial title and descriptionCode, onceSensible defaults when the row is created
Title and description afterwardsDatabaseEditable in goatboat; the sync never overwrites them
Nav placement and orderDatabaseReorder the nav without touching code
Tags, visibility, datesDatabaseSame as every other piece of content

What the site gets

tsusage.ts
// Everything the nav needs, from one request
const nav = await listContent({ site: "jaygriff", inNav: true }); // top-level rows, by sort_order

// Breadcrumbs for any URL, code page or database page
const { breadcrumbs } = await getContentByPath("/tests/feed");
// → [{ title: "Tests", path: "/tests" }, { title: "Content feed", path: "/tests/feed" }]
  • Nav from top-level rows marked for the nav, ordered in goatboat.
  • Breadcrumbs for every URL with one call, because every URL has a row.
  • Sitemap, search, “recently updated” covering both kinds of page.
  • Unlisted pages: a new status for pages that are reachable by URL but left out of feeds and the sitemap and marked noindex. That's what /tests needs.

Open questions

Things worth deciding before building, and where the design could be wrong.

  • Title drift. After creation the database owns the title, so renaming a page in its metadata won't change the row. The page's <title> and its breadcrumb label can then disagree, unless pages read their title from the row too.
  • When the sync runs. On production deploys only, a page you just created locally has no row until you deploy, so dev needs a fallback label. Running it during dev too means writing to the shared database from your laptop.
  • Builds depending on goatboat. If the sync runs in the build and goatboat is down, the deploy should still succeed. The sync has to be allowed to fail without failing the build.
  • Missing pages pile up. Flagging instead of deleting is safe, but something has to show them to you so you can clean up.
  • Dynamic routes are invisible. [...path] and similar are templates, so they get no row. The pages they render are database rows already, so that's probably fine.
  • One site per row. site_id means a post lives on exactly one site. Cross-posting later would need a join table instead.

Alternatives considered

AlternativeWhy not
A hand-maintained route map in the siteThe same manual editing as the nav, just moved to another file
Derive everything from URL segmentsGives parents but not labels, tags or visibility
Put every page in the database, no app routesBespoke pages still need code; the component registry covers some, not all
  • Page identity: an ID on the first line

    Why code pages carry a goatboat ID on their first line, and the alternatives weighed for following pages across folder moves.

  • What the MCP create_page call actually sends

    What a goatboat MCP create_page call actually sends, hop by hop from the agent's arguments to the proposal request.

  • Tags

    Every tag with its count. Each links to the feed filtered by that tag.

  • Projects

    Every project and experiment, any status, with kind and status shown.