Lesson validation

Every tool that writes a lesson — create_lesson, create_lesson_file, update_lesson and patch_lesson — checks it against the authoring standard first. Errors reject the write; warnings ride along with a successful one.

The point of validating rather than only documenting is that the standard then holds even when the model never read it. Server instructions are optional in the MCP spec and some clients drop them (claude.ai's connector UI is the notable one), and a tool description is advice the model may or may not follow. Validation does not depend on either.

The split between the two halves of the standard lives in two files:

FileHolds
apps/mcp/src/standards.jsThe rules that need judgement — tone, difficulty, what makes a tight open easy. Sent as MCP instructions and embedded in create_lesson's description.
apps/mcp/src/validate.jsThe rules a script can decide. Enforced on write, whatever the client showed the model.

Keep them in step: a rule stated in one that the other also covers should describe the same thing.

Errors — the write is rejected

CodeWhat tripped it
E_GROUNDING_SINGLEA green (single) answer does not appear, word for word, in its own section's passage.
E_GROUNDING_MULTIPLEA multi-word orange (multiple) answer is not in its own section's passage.
E_ORANGE_PARAPHRASEDA single-word orange answer is not in the passage — usually paraphrase ("HOT" for "superheated") or general knowledge, which belongs in a background question.
E_GROUNDING_NUMBER_FILLA fill-in-the-blank number answer (one with no steps) is not in the passage.
E_BACKGROUND_IN_TEXTA blue (background) answer does appear in its own passage, defeating the point of the type.
E_BACKGROUND_NO_CONTEXTA background question has no background field.
E_SPELLING_LENGTHA spelling word is outside 6–9 letters.
E_SPELLING_DUPLICATEA spelling word is used in two sections (or twice in one).
E_SPELLING_COLLISIONA spelling word appears inside an answer anywhere in the lesson — PRISON within "the prisoner's dilemma". Matched as a raw substring, which is the point.
E_ANSWER_WORD_REUSEDThe same answer word answers two different questions, anywhere in the lesson and at any length. Also fires when a one-word answer reappears inside a longer answer.
E_NUMBER_DUPLICATETwo questions resolve to the same number.
E_OPEN_HAS_ANSWERAn open question carries answer, answers or exampleAnswer.
E_RETIRED_STEMA pink question uses the retired "…one word that comes to mind…" stem.

A rejection names the section, the offending value and the fix, because the model reads it and resubmits — "validation failed" buys a guess, a specific message buys a correction in one round trip. Up to 25 are listed at a time.

Warnings — saved, and reported back

Returned as a warnings array on the successful result:

{
  "id": "…",
  "url": "…",
  "warnings": [
    {
      "code": "W_NUMBER_NO_STEPS",
      "section": 3,
      "message": "Section 3 \"Deserts\": no purple question carries `steps`. …"
    }
  ]
}
CodeWhat it flags
W_SECTION_COUNTThe lesson isn't 6 sections. Lesson-wide, so it carries no section.
W_QUESTION_SHAPEA section's question types or order differ from 3 single, 2 number, 2 multiple, 1 background, 7 open.
W_NO_QUESTIONA section has no questions at all.
W_OPEN_SPLITA section's 7 pink questions don't read as 4 tight opens followed by 3 extended ones.
W_ORANGE_MULTIWORDAn orange accepted answer is more than one word.
W_ORANGE_ANSWER_COUNTAn orange question accepts fewer than 2 or more than 4 answers.
W_SPELLING_COUNTA section doesn't have exactly 4 spelling words.
W_NUMBER_NO_STEPSA section's word problem has no steps.
W_SPELLING_IN_CAPSA spelling word is also ALL-CAPS learning vocabulary in the same passage. A warning rather than an error because acronyms trip it legitimately.

These are warnings and not errors because a legitimate lesson can trip each one: a user who asks for four sections gets W_SECTION_COUNT and should not be blocked by it.

skipValidation

Every writing tool takes skipValidation: true, which turns the errors off (and with them the warnings — nothing is checked). It exists for the user who deliberately wants a lesson the standard forbids, not as a way around a defect that should be fixed.

Patching an existing lesson

patch_lesson validates the lesson before and after the edit and holds the caller only to the difference. Without that, a one-line tweak to a lesson written in the web editor — or written before these rules existed — would be blocked by defects the patch never touched and the assistant may have no mandate to change. The filter applies to warnings as well as errors, so a patch reports only what its own edit introduced.

Findings are matched on the defect's identity rather than its message, which has to hold two properties at once:

  • Section numbers can't be part of it. Moving or inserting a section would otherwise make every later finding look new. The identity uses the section's and block's ids, which survive move_section, move_block and replace_block.
  • Block identity has to be part of it. Code plus offending value alone is not enough: a patch can add a genuinely new question carrying the same defect on the same word, and it would be written off as pre-existing. Including the block's id separates them.

For a collision, which names two parties (two spelling words, two questions), the pair is sorted before it becomes a key — otherwise reordering the sections swaps which end the walk reaches first and rewrites the identity of a defect nobody touched.

update_lesson replaces the whole document, so it gets no such exemption: whatever the result contains, the caller sent. That is a reason to prefer patch_lesson for small edits.

Comparison rules

Text is normalised before any comparison — uppercased, punctuation dropped, whitespace collapsed. Two details matter and both caused false failures before they were handled:

  • Thousands separators. The passage says 3,776 and the answer field holds 3776. Both normalise to 3776.
  • Decimal points. 112.5 has to survive the punctuation strip as one token, while the full stop in MAGMA. must not.

Grounding uses whole-word matching, so ASH is not found inside WASHED. The spelling collision check deliberately uses raw substring matching instead, because PRISON really is inside PRISONER'S.

Passages are flattened out of rich text first, so a lesson round-tripped through the web editor (which stores HTML) is compared on its words rather than its markup.