Serialising/Parsing Markdown

There was some serialisation/parsing code for markdown already in the history code. I decided to re-use that, then later restructure it. I prompted the LLM:

read this @README_markdown_redesign.md as context.
I don't want to throw away the code in @src/dockb/infrastructure/history/ but it will need changes and
restructure. The whole markdown redesign will require quite a number of sections of work, and I don't
want to consider them all yet.
Firstly the format used when serialising/parsing the markdown format will need to be changed so that it
is in line with what is described in the @README_markdown_redesign.md document.
Discuss with me how you will change this code.

That gave it something to think about. It asked a couple of intelligent questions, which I answered, and it also asked about the spans (README_markdown_redesign.md section 6 gets into that.) I told it the spans are for later because we don't have BE support yet for SPO Triples.

I checked its plan, and then told it to do the three planned sections of work with TDD. This is the first time I've tried my revised AIDLC on this project, and I expect to be revising it.

By the way, I added a couple of tools to my toolkit:

Codegraph and Meld . (Meld works best on Linux and Windows (via mingw), but there is also a Meld for OSX project.) Meld is great for viewing the changes just made by the LLM.

I looked at the code it had produced, and was happy with the result. However, it got me thinking:

The initial NLP sentence splitting process will be fast. A custom Markdown Editor could be set up to re-read the file after saving, and after the processing by this (currently) "history" package...

I then realised I needed to embed the UUIDs into the markdown, so I prompted it:

@src/dockb/infrastructure/history and refer to @README_markdown_redesign.md  Last coding session I told you
that the UUIDs of sentences and paragraphs would not need to be put in markdown as metadata in the markdown
into span elements. The redesign document also says the same. I am changing that. The UUIDs will need to be
embedded in the markdown. Make changes to the serialiser and parser to write and read the UUIDs

I then gave it a large prompt, which I lost because I did not save my edits to this html file before I rebooted my PC to fix a USB driver issue. Basically, I told it to work on implementing the function that will find changes in the markdown... See the files in this directory: src/dockb/infrastructure/changes/

I also did some work on the AGENTS.md file so that the TDD process will use AI reviews rather than human reviews.

The instructions and, effectively, the way you scaffold the LLM, are key to good development using AI.

The "changes" package it created obviously won't work for a new chapter in a new document, and it calls nothing yet, so the new data has no way to get in to the KG database. I also have concerns about performance.

I then prompted it as follows (which will help you to grasp how much I now rely upon chatting with the LLM to help me think about the project:

look at @src/dockb/infrastructure/changes/ and discuss with me how I could present it with a completely new
chapter file for a new document - will it perform well? It will presumably produce a set of changes which
will be all new paragraphs.

It replied:

What happens on an all-new chapter

detect_changes(old_chapter, new_markdown, nlp) walks the new file's blocks (split("\n\n")) and classifies each against the hydrated old chapter. With an empty old chapter:

• every block lands in diff.new (NewParagraph), because par_id is None or par_id not in old_texts (detect_changes.py:62),
• changed stays empty, deleted stays empty (no old ids to miss → line 69),
• the model is exactly "all new paragraphs". The bool(ChapterDiff) lets you short-circuit. This path is well-covered by tests (test_new_*).

So structurally: yes, it handles a brand-new chapter cleanly.

Caveats for the new-document case

1. It's not wired anywhere yet. Grep shows detect_changes has no call site in services/ or controllers/ — only __init__ re-export + tests. The sync save-and-rehydrate engine (README_markdown_redesign §7 step 2) that's supposed to call it isn't built. "Presenting" a new chapter means writing that engine.
2. A new document ≠ a new chapter. This module only diffs paragraphs. Creating the Document, the empty Chapter, and the PART_OF links is orchestration outside detect_changes (the repositories MERGE chapter ids, so the loop must create the skeleton first, then diff, then rehydrate).
3. Front matter id must be agreed beforehand. _strip_front_matter requires the file's id to equal old_chapter.id (detect_changes.py:145). For a brand-new chapter there's no DB row, so the loop has to generate the chapter id, write it in the file, and diff against a skeleton Chapter(id=...) carrying that same id. The id cannot be discovered from the file.

Performance

• Canonical/span-bearing file (writer output): any block that carries data-par-id runs zero spaCy — pure regex per block. Near-instant regardless of chapter size.
• Span-free (hand-typed) new file: every paragraph block triggers a full spaCy nlp() call. Nothing is batched; one en_core_web_sm parse per paragraph. Model loads once per process (fine), but parse throughput is roughly thousands of words/sec — page-thousands of ms per paragraph. A long manuscript chapter lands in seconds — acceptable for a synchronous save, but it's the only real cost, and it's unavoidable by design (span-free text has to be NLP-split to find sentences).
• Memory: the whole file string plus the full diff (every sentence text) in memory at once. Fine for chapter sizes.
• Minimal-write/byte-for-byte guarantee is trivially satisfied — there's nothing to preserve on an all-new chapter.

Verdict: functionally it performs well and produces exactly the "all new paragraphs" diff you expect; the only soft spot is that a span-free new file pays a per-paragraph spaCy sentence-split (seconds, not real-time), and the new-chapter id + document/chapter creation must be orchestrated around detect_changes, since it only sees paragraphs.

Want me to sketch a plan for the calling engine (create doc+chapter, diff, rehydrate) as a new cycle?