Dev sync: new pages appear while I work
A new code page should show up in goatboat (and so in /tests, feeds and the tree) as soon as I save the file, with no commit or push. This doc is the design. It's a plan to review, not built yet.
The problem
Goatboat knows about code pages (src/app/**/page.tsx in the site) only through the routes sync. Today that runs in a GitHub Action on pushes to main. So the loop for a new page is: write it, commit it (the hook stamps its id), push, wait for the Action, then look. For a page I'm only trying out, that's far too slow.
Running the existing sync by hand from a dev checkout isn't safe. The sync does more than add pages, and the database is shared and live:
The idea
A second, much smaller sync for development: additive only.
The dev sync can create rows, and it can only touch rows that are still drafts.
Everything else (moves, redirects, flagging missing, anything on a published page) stays with the full sync in the GitHub Action on main. The Action doesn't change at all.
That one rule is what keeps the two from conflicting. The dev sync never writes anything the Action relies on for live pages, so neither can undo or fight the other on a published page.
What stays the same
- The GitHub Action, its workflow file and its trigger (push to
maintouchingsrc/app/**). - The full sync's behaviour, and the
/api/syncendpoint for callers who don't ask for the new mode. - The
// goatboat: <uuid>first line, and the pre-commit hook that stamps it. - Goatboat owns everything about a page except its existence and its place in the code. A sync never overwrites a title, description or any other metadata.
Goatboat: an additive mode on /api/sync
Request
The body gets an optional mode. Without it, the request is a full sync exactly as today.
{ "mode": "additive", "approutes": [ { "id": "…", "path": "/tests/x", "file": "src/app/tests/x/page.tsx", "toc": [] } ] }mode is "full" (the default) or "additive". The route list has the same shape and the same validation in both modes.
What additive does, route by route
Routes are handled shallowest first, as the full sync does.
Two points about conflicts:
- The full sync fails the whole run on a conflict. Additive reports it and carries on, so one clash doesn't stop other new pages from appearing. The Action still fails loudly on the same clash at push time, which is where it should block.
- Placeholders count as creates and are drafts, so they follow the same rule.
Response
The same JSON as today, plus the new groups. A full sync never returns skipped, wouldMove or conflicts. Every group is a list of paths, and empty groups are fine, as now.
Safety properties
- All changes in one run happen in one transaction, as now.
- Nothing is ever deleted.
- A published row is never written to. This is checked in the code that writes, not just by the route's logic, so a future bug can't quietly widen it.
- The same token and the same
Authorizationcheck as the full sync. The mode is something my own script asks for, so a separate token would add no real protection for a single-owner setup.
Tests (in tests/, one thing per test)
Each has the usual // Checks: and // Expected: comment.
- Additive creates a draft for a new route.
- Additive creates placeholder drafts for missing parents.
- Additive doesn't flag a code page as missing when it isn't listed.
- Additive refreshes headings on a draft code page.
- Additive clears the missing flag on a draft code page that is listed again.
- Additive leaves a published code page untouched, even when its headings differ.
- Additive doesn't move a row whose folder changed, and reports
wouldMove. - Additive reports a path conflict and still creates the other routes.
- Additive adopts a draft db row that has the same id.
- A full sync still behaves as before (the existing sync tests keep passing unchanged).
The site: one command, then automatic
npm run routes
One command that does both steps:
- Stamp any page that has no id (the same script the pre-commit hook uses, without staging).
- Send the route list to goatboat with
mode: "additive"and print the result.
It uses the same Bitwarden token lookup as routes:sync. It's safe to run any time, as often as I like. Running it twice in a row changes nothing the second time.
routes:sync stays as it is, for a full sync by hand when I really want one.
Automatic while npm run dev is running
A small watcher runs alongside the dev server:
- On start, it does one
routesrun, so goatboat is current when I sit down. - It watches
src/appforpage.tsxfiles being created, deleted, renamed or edited (headings change the list), waits half a second after the last event so editors can finish writing, then runs the same stamp-and-send. - It remembers a fingerprint of the last list it sent and does nothing when the list hasn't changed. This also stops it from reacting to its own stamping.
- Runs never overlap: an event during a run queues one more run after it.
- If goatboat is down, or the request fails, it prints one short line and keeps going. A sync failure must never stop the dev server.
- It needs the sync token, which
npm run devfetches from Bitwarden once at start, the same way it fetches the read token. The token is never printed.
Why stamping from a watcher is OK
Writing an id into a file I might have open could clash with an unsaved buffer. It doesn't in practice: only pages with no id are stamped, and every existing page has one. A new file either starts empty (nothing typed yet) or arrives with content already saved, so the editor reloads it cleanly. I'd still wait the half second before stamping.
How it works with the GitHub Action
The Action runs a full sync on every push to main from a clean checkout. The two syncs touch different things:
The stamped id is what ties them together: the dev sync uses the id in the file, and the commit includes that file, so the Action finds the same row.
Edge cases
- A page I never commit. Its row stays a hidden draft. The next full sync flags it missing (it isn't in
main). The next dev run clears that flag again if the file is still in my tree. The flapping only ever involves drafts, and nobody but me sees them. It disappears when I commit the page or delete the row in goatboat. - Rename before the first commit. The row keeps its old path until the commit, then the full sync moves it. In the meantime
wouldMoveshows in the output, and the new path doesn't appear in/testsuntil then. This is the one place where "pops up instantly" doesn't hold. I'd rather accept that than let the dev sync create redirects. - A copied page. The stamp script already gives copies a new id, so a copy shows up as its own new row.
- A path already taken in goatboat (for example a page I made by hand in goatboat, with a different id). It's reported as a conflict and left alone. The full sync would also refuse it, so the fix is the same either way: delete the goatboat row, or use its id.
- Two machines. Both send additive requests, which only create or touch drafts, so they can't corrupt each other.
What I'm deliberately not doing
- No moves, redirects or missing flags from dev. That's the Action's job.
- No new auth or token scope.
- No change to the Action, the hook, the stamp format or the database schema. There's no migration.
- No sync from the production site or from Vercel builds.
- No syncing of metadata. Goatboat stays the owner of titles, descriptions and the rest.
Build order
Each step works on its own and ends with something I can use.
- Goatboat additive mode. Schema change to the request, the mode in
syncAppRoutes, the tests above, and the sync endpoint's description on goatboat's home page. Done when a request withmode: "additive"behaves as the table says and the old sync tests still pass. npm run routesand--additiveon the sync script. Done when I add a page, run one command, and see it in/tests.- The dev watcher. Done when saving a new
page.tsxmakes it appear in/testswithin a couple of seconds, with no command.
Questions I'd still like answered
- Is half a second the right wait, or do I want it faster even if it syncs mid-typing now and then? (Syncing early is harmless: the list is re-sent on the next change and nothing duplicates.)
- Should the watcher log every run, or only when it created or changed something? My pick: only when something happened, plus errors.
Related
- Proposing complete, published pages
Agents can now propose a new page with its description, tags, project and section, published on approval. What changed, and the security questions that come with it.
- Link render
- Page identity: an ID on the first line
Why code pages carry a goatboat ID on their first line, and the alternatives weighed for following pages across folder moves.
- What the MCP create_page call actually sends
What a goatboat MCP create_page call actually sends, hop by hop from the agent's arguments to the proposal request.