Routes as data
A proposal, not built yet. Every page on the site gets a row in the content database, including the ones written as TSX in the repo, so the nav, breadcrumbs, sitemap and feeds can all come from one query instead of from the file system and hand-edited lists.
The problem
The site has two kinds of pages, and only one of them is data. Pages stored in the database have a title, a parent, tags and a publish date, so breadcrumbs and feeds already work for them. Pages written as page.tsx files in the repo have none of that. They exist only as folders.
- The nav is a hand-edited array, so every new section means editing code.
- Breadcrumbs can't label
/docs/code-stress, because nothing says it's called “Code stress test”. - Nothing at runtime can ask “what pages exist?”, so a sitemap, a search index or “recently updated” can only see half the site.
The idea
A TSX page keeps rendering from the repo, exactly as it does now. What changes is that it also gets a row in the database holding its metadata: title, parent, tags, visibility, position in the nav. The code is the body; the row is everything about it. A new source column says which kind of page a row is.
One tree per site
A path like /docs only means something on a particular site, and the plan is for one database to serve several sites. So a new sites table, a site_id on every content row, and paths that are unique per site instead of globally. Every existing row becomes part of the jaygriff site.
export const sites = sqliteTable("sites", {
id: id(),
slug: text().notNull().unique(), // "jaygriff"
name: text().notNull(),
baseUrl: text("base_url").notNull(),
});
// New columns on content
siteId: text("site_id").notNull().references(() => sites.id),
source: text({ enum: ["db", "code"] }).notNull().default("db"),
file: text(), // set when source = 'code', e.g. app/tests/feed/page.tsx
missingSince: text("missing_since"), // set by the sync when a code route disappears
// statuses gains "unlisted"; path becomes unique per site
uniqueIndex("uq_content_site_path").on(t.siteId, t.path)The sync
1. Discover
A script in the site walks src/app and turns every page.tsx into a path, following Next's own rules.
src/app/
page.tsx → /
docs/page.tsx → /docs
docs/code-blocks/page.tsx → /docs/code-blocks
tests/page.tsx → /tests
tests/feed/page.tsx → /tests/feed
tests/content/[...path]/… → skipped: dynamic, a template rather than a page
(marketing)/about/page.tsx → /about (route groups don't add a segment)
_components/… → skipped: private folder2. Read metadata from code
Each page's export const metadata = { title, description } is Next's own convention, so it also gives the page a real <title> and description for search engines. The script reads it from the source text without running the page.
3. Send the full list to goatboat
POST /api/sites/jaygriff/routes/sync
Authorization: Bearer <token>
{
"routes": [
{ "path": "/tests", "file": "app/tests/page.tsx", "title": "Tests", "description": "Pages that read from the content API." },
{ "path": "/tests/feed", "file": "app/tests/feed/page.tsx", "title": "Content feed", "description": null },
{ "path": "/docs/code-blocks", "file": "app/docs/code-blocks/page.tsx", "title": "Code blocks", "description": "…" }
]
}4. Goatboat reconciles
Who owns what
This is the rule that keeps the sync safe to run any number of times: code decides whether a page exists; the database decides everything said about it.
What the site gets
// Everything the nav needs, from one request
const nav = await listContent({ site: "jaygriff", inNav: true }); // top-level rows, by sort_order
// Breadcrumbs for any URL, code page or database page
const { breadcrumbs } = await getContentByPath("/tests/feed");
// → [{ title: "Tests", path: "/tests" }, { title: "Content feed", path: "/tests/feed" }]- Nav from top-level rows marked for the nav, ordered in goatboat.
- Breadcrumbs for every URL with one call, because every URL has a row.
- Sitemap, search, “recently updated” covering both kinds of page.
- Unlisted pages: a new status for pages that are reachable by URL but left out of feeds and the sitemap and marked
noindex. That's what/testsneeds.
Open questions
Things worth deciding before building, and where the design could be wrong.
- Title drift. After creation the database owns the title, so renaming a page in its
metadatawon't change the row. The page's<title>and its breadcrumb label can then disagree, unless pages read their title from the row too. - When the sync runs. On production deploys only, a page you just created locally has no row until you deploy, so dev needs a fallback label. Running it during dev too means writing to the shared database from your laptop.
- Builds depending on goatboat. If the sync runs in the build and goatboat is down, the deploy should still succeed. The sync has to be allowed to fail without failing the build.
- Missing pages pile up. Flagging instead of deleting is safe, but something has to show them to you so you can clean up.
- Dynamic routes are invisible.
[...path]and similar are templates, so they get no row. The pages they render are database rows already, so that's probably fine. - One site per row.
site_idmeans a post lives on exactly one site. Cross-posting later would need a join table instead.
Alternatives considered
Related
- Page identity: an ID on the first line
Why code pages carry a goatboat ID on their first line, and the alternatives weighed for following pages across folder moves.
- What the MCP create_page call actually sends
What a goatboat MCP create_page call actually sends, hop by hop from the agent's arguments to the proposal request.
- Tags
Every tag with its count. Each links to the feed filtered by that tag.
- Projects
Every project and experiment, any status, with kind and status shown.