Compaction Enrichment Pipeline

Every one of the steps below has a real function — skip them and the work product falls apart.

Every step below exists because skipping it produces garbage. In one real project, an agent once read this skill file, skipped every sub-skill invocation, and wrote the output from its own head. The person reviewing it audited it, found zero skills were actually invoked, and called it out. That is why every step below says INVOKE — not "produce output like the skill would." INVOKE means use the Skill tool. If you write the output yourself instead of invoking the skill, you are hallucinating — you are hallucinating that your output is equivalent to the skill's output. It is not.

What This Is NOT

This is NOT a mechanical bulk field-extraction pass. This skill runs a deep enrichment using the actual reasoning skills — you MUST INVOKE /reflect, /align, /compact-agentic-session, and /incomplete-tasks-get-from-session via the Skill tool.

Writing output that resembles what these skills would produce, without invoking them, is a HALLUCINATION. Do not do it.

Where the enriched compaction lives (read this before Step 0)

Everything below is written against a compaction record — a stored document with the fields named throughout (alignmentNote, leverageScore, incompleteItems, and so on). Where that record lives depends on your setup:

• If a compaction database or API is set up (see /alignment-harness:harness-setup) — one reference setup is a MongoDB collection reached through a backend API — read and write the record there, using the concrete code shown in each step as a worked example of the pattern (adjust the connection/model details to your own schema).
• If nothing like that is set up — run alignment-harness records compactions to get a local folder path, and treat the compaction as one JSON file in that folder, named by its short id (e.g. <that folder>/<shortId>.json). Every "write to the DB" instruction below becomes "read the JSON file, merge in the new fields, write it back" — same fields, same idempotency rule, same immediate-write-after-each-step discipline. There is no admin URL to print in this case; instead, tell the person the exact file path so they can open it.

Nothing about the reasoning pipeline below changes based on which of these you're using — only where the record is read from and written to.

When To Use

• Enriching compactions that have no leverage score
• Enriching compactions that have leverage scores but no reflection or alignment analysis
• Enriching the most significant compactions with deeper insight than a quick mechanical pass can provide
• Batch enrichment of all unenriched compactions (dispatch as parallel subagents on an interactive subscription seat, not a metered API key — see the cost rule below)

COST RULE — NEVER USE A METERED API KEY FOR BULK WORK

NEVER write scripts that import @anthropic-ai/sdk and call a metered API key for this. Interactive subscription seats (what this pipeline is meant to run on) are a fixed monthly cost; a metered API key bills per token and can run up real, unauthorized spend on someone else's account if a bulk job is pointed at it by mistake. All enrichment work should run through subagents dispatched from Claude Code on the person's own interactive session, not through a separately-billed API key.

Pipeline Steps (MANDATORY — in this order, skip none)

Step 0: Read the compaction and all attached items

Before any reasoning, load the full compaction record — via DB query/API if you have one set up, or by reading its JSON file from the local records folder otherwise (see above):

• compactedSession (the raw session text)
• All existing enrichment fields (don't overwrite what's already good)
• principles, uxStoriesBroughtToReality, incompleteItems (existing content to build on)
• enrichmentHistory (what's already been done)

Idempotency check: If enrichmentHistory already contains an entry with enrichedBy: 'sonnet-enrichment-pipeline', skip this compaction unless --force is specified.

PRINT what you loaded. Show the shortId, title, content length, and which fields are already populated vs empty. This is auditable evidence that Step 0 happened.

Step 1: INVOKE /reflect — Structured self-reflection on the session

YOU MUST USE THE SKILL TOOL TO INVOKE /reflect. Do not write reflect output yourself.

Pass the compaction content to the reflect skill. Treat the compaction text as if you are reflecting on work that was done.

The reflect skill produces these sections — you do not produce them, the skill does:

• Target: What was the specific intent in UX terms the human was trying to get?
• Location: Where did the session end up relative to that intent?
• Gap: What gap remains between intent and reality?
• Initiative: What overarching intent does this connect to? What would full realization look like?
• Leverage: What hasn't been considered? What UX problems or gaps could help or harm?
• Questions to keep in mind: What questions matter for staying connected to the intent?

Output destination: The synthesis goes into the alignmentNote field as markdown. Also informs agentReasoning fields.

WRITE IMMEDIATELY AFTER THIS STEP. Do not wait for all steps to complete. Write alignmentNote and agentReasoning to the record NOW — a mongoose updateOne if you have a database, or the local JSON file otherwise. This prevents data loss if you run out of context or crash before Step 5. Every subsequent step also writes immediately after completing.

PRINT the reflect output. This is auditable evidence that Step 1 happened.

Step 2: INVOKE /align — Derive nested intent from session content AND attach full markdown

YOU MUST USE THE SKILL TOOL TO INVOKE /align. Do not write align output yourself.

The align skill derives nested intent layers:

• Overarching intent: What company-level objective was this session serving? NOT "what did the session do" but "what business goal does this serve?" (goes into overarchingIntentTitle + overarchingIntentDescription)
• Target UX intent: The specific testable UX promise. Written as "When X, then Y" conditional. (goes into targetUxIntentTitle + targetUxIntentDescription)
• Scope declaration: Intent, certainty, decomposed UX outcomes, boundary, assumptions (goes into scopeDeclaration)

CRITICAL — ATTACH THE FULL /ALIGN OUTPUT AS MARKDOWN: The complete /align output — the nested intent map with certainty scores, the full human-legible breakdown, every layer — MUST be stored in the reflectionOnScope field as markdown. This is not a summary. This is the FULL align document. The reflectionOnScope field exists specifically for this purpose (schema line 426: "full /reflect output text" — it holds the full reasoning document from both reflect and align).

The reflectionOnScope field should contain:

1. The full /reflect synthesis (from Step 1)
2. The full /align nested intent map (from this step)
3. Both as readable markdown, not JSON, not field values — the actual human-legible analysis

Voice rules from /align apply: Non-technical, human-legible, no variable names, no jargon.

Certainty scoring: Each intent statement gets a certainty dot per /align format. Max certainty 85% for post-hoc enrichment — never claim 100% without human validation.

PRINT the full align output. This is auditable evidence that Step 2 happened.

WRITE IMMEDIATELY AFTER THIS STEP — THIS IS THE MOST COMMONLY SKIPPED WRITE. Agents skip writing reflectionOnScope far more often than any other field. Do NOT be one of them.

If you have a database set up, here is the pattern from a reference setup (a MongoDB collection reached from backend code) — adjust the connection and model require to whatever your own schema actually is:

node -e "
require('dotenv').config();
const mongoose = require('mongoose');
const SC = require('./models/SessionCompaction'); // your own compaction model
mongoose.connect(process.env.MONGODB_URI).then(async () => {
  const reflectOutput = \`YOUR FULL REFLECT OUTPUT HERE\`;
  const alignOutput = \`YOUR FULL ALIGN OUTPUT HERE\`;
  const fullMarkdown = reflectOutput + '\\n\\n---\\n\\n' + alignOutput;
  
  const result = await SC.updateOne(
    { shortId: 'TARGET_SHORT_ID' },
    { \$set: {
      reflectionOnScope: fullMarkdown,
      overarchingIntentTitle: 'YOUR VALUE',
      overarchingIntentDescription: 'YOUR VALUE',
      targetUxIntentTitle: 'YOUR VALUE',
      targetUxIntentDescription: 'YOUR VALUE',
      scopeDeclaration: {
        intent: 'YOUR VALUE',
        certainty: 80,
        decomposed: ['When X then Y'],
        boundary: 'YOUR VALUE',
        assumptions: ['YOUR VALUE'],
        updatedAt: new Date()
      }
    }}
  );
  console.log('Step 2 write:', JSON.stringify(result));
  mongoose.disconnect();
});
"

Verify the write succeeded by checking result.modifiedCount === 1. If it's 0, something went wrong — print the error and retry.

If you don't have a database set up, do the equivalent against the local JSON file: read <records folder>/<shortId>.json, merge in the same fields (reflectionOnScope, overarchingIntentTitle, overarchingIntentDescription, targetUxIntentTitle, targetUxIntentDescription, scopeDeclaration), and write the file back. Verify by reading it back and confirming the fields are actually there.

Either way, the reflectionOnScope field is the MOST IMPORTANT field in the entire enrichment — it's the full human-readable analysis the person reviewing this reads. If this field is empty, the enrichment failed.

ESCAPING FIX (database path only): If your node -e command fails due to template literal escaping (backticks inside backticks), write the script to a temp file inside whichever directory your compaction model's code actually lives in (not /tmp — a relative require() won't resolve from there):

cat > <path to your backend project>/tmp-enrich-SHORTID.js << 'SCRIPT'
require('dotenv').config();
const mongoose = require('mongoose');
const SC = require('./models/SessionCompaction'); // your own compaction model
// ... your update code here with no escaping issues ...
SCRIPT
cd <path to your backend project> && node tmp-enrich-SHORTID.js && rm tmp-enrich-SHORTID.js

CRITICAL: The script MUST run from the directory that your require('./models/...') path is relative to, or the require will fail.

Step 3: INVOKE /compact-agentic-session field structure — Fill remaining fields

YOU MUST USE THE SKILL TOOL TO INVOKE /compact-agentic-session. Do not fill fields yourself.

The skill defines mandatory execution order and field requirements:

1. agentReasoning (informed by Step 1 reflect output)
2. Leverage score (0-100):
   • 80-100: Revenue-critical (payments, checkout, conversion, pricing, paid user experience)
   • 60-79: Core product (quality of your product's central experience, auth, onboarding, key UX flows — for a coaching app, coaching quality itself)
   • 40-59: Infrastructure (agent systems, admin tooling, observability, deployment)
   • 20-39: Maintenance (error triage, test fixes, cleanup, documentation)
   • 0-19: Trivial (auto-compactions, config changes, one-off lookups)
3. uxCriticality — criticalityScore (0-10), isUserFacing, dimensions, reasoning
4. preExecutionReflection — targetIntent, failureModes, uxRequirements, harmPotential
5. successMeasure — 1-2 actual KPIs
6. howToConsumeThisWorkProduct — conversational framing ("Ok so this session was about..."), sanity check steps, full clickable admin URL

Voice rules: Conversational framing MANDATORY. No jargon-first formal language. All URLs must be full clickable links.

Phase assignment (MANDATORY, only if your project tracks roadmap phases at all): Match this compaction to the most similar roadmap phase in your own roadmap. If you don't have a defined list of phases, skip this and leave roadmapPhaseId null — don't invent phases to fill the field.

Example from a real project (opt-in illustration of the pattern, not a list to reuse): roadmap phases named as outcome statements a person outside engineering could read and understand at a glance — things like "get the signup funnel to convert twice as well," "clean up failed payments and block access if unpaid," "get the marketing budget fully optimized," rather than engineering-shaped names like "Q3 infra work." The pattern worth copying is the shape of a phase name (a plain-language outcome, not a project-management label) — your own project's actual phases will be specific to what you're building.

Pick the phase whose intent most closely matches this compaction's work. Set roadmapPhaseId to that phase's id in your own system. If none fit, leave null.

Category assignment (MANDATORY, only if your project has defined categories): Match to the most fitting category in your own category list — set categoryIds to the matching category id(s) from your own system. If you don't maintain a category list, skip this and leave categoryIds empty rather than inventing categories.

Example from a real project (opt-in illustration, not literal ids to reuse): categories like "self-healing code" (agents auto-healing code without hallucination), "personal experiments" (research, custom model training), "bug fixes," "product quality improvements," "roadmap-level initiatives." Again, the pattern is the shape — broad, recurring buckets that make sense of a compaction at a glance — not these specific names.

Parent compaction linking: If this compaction's work is clearly a sub-task or continuation of another compaction that already exists, set parentCompactionIds to link them. Query by title similarity or sessionId references in the content. If no clear parent, leave empty.

PRINT each field value including phase, category, and parent decisions with reasoning. This is auditable evidence that Step 3 happened.

WRITE IMMEDIATELY AFTER THIS STEP. Write leverageScore, leverageAssignedBy, leverageAssignedAt, uxCriticality, preExecutionReflection, successMeasure, howToConsumeThisWorkProduct, roadmapPhaseId, and categoryIds to the DB NOW. Do not wait.

Step 4: INVOKE /incomplete-tasks-get-from-session — Extract unresolved work

YOU MUST USE THE SKILL TOOL TO INVOKE /incomplete-tasks-get-from-session. Do not extract items yourself.

Adaptation for post-hoc enrichment: Extract incomplete work from:

1. The compactedSession text — look for "next step", "TODO", "planned but not built", "needs to be", "hasn't been", "not yet", "will need to"
2. Existing incompleteItems array — preserve and enrich, don't overwrite
3. The reflect output (Step 1) — gap and leverage sections surface what's unresolved

For each incomplete item, produce the full required shape:

• title — UX-first, under 80 chars. The human must instantly know the specific UX impact and its constraints. Lead with what changes for a real person, not what code to write. Example: "Paying users see full access even when entitlement check fails" not "Add fallback to getAccessTier()"
• intentStatement — "When {actor} does {action}, they experience {outcome}" MANDATORY
• status — pending (default)
• priority — 1-100 integer
• domain — searchable tag
• humanImpact — who is affected, current experience, fixed experience

UX-FIRST WRITING RULE (applies to ALL written objects — incompleteItems, completedItems, proposedActions, principles, uxStoriesBroughtToReality, comments):

Every object written to any array on a compaction MUST lead with human UX intent before any technical description. The person you're working with reads these. They must instantly understand:

1. Who is affected — which specific person in which specific moment
2. What they experience now — the current reality in their words
3. What they should experience — the intended reality
4. What constrains it — what condition or edge case makes this non-trivial

If an item says "Refactor accessTiers.js to emit LEGACY_TRIAL_ACCESS" — that is WRONG. Nobody knows what that means without reading the code.

If an item says "Free trial users currently look identical to paying subscribers to every downstream system, which means when we build trial-specific UX (upgrade prompts, usage limits, trial expiry warnings) nothing can distinguish them — so the entitlement system needs a separate state for legacy trial users" — that is RIGHT. The person reading it knows exactly what's broken, who it affects, and why it matters, without opening a single file.

Test before writing any item: "If the person you're working with reads only this title and intentStatement, do they know the specific UX impact and its constraints?" If no — rewrite until yes.

Merge rule: Don't duplicate or overwrite existing items.

PRINT new items extracted. This is auditable evidence that Step 4 happened.

WRITE IMMEDIATELY AFTER THIS STEP. Write any new incompleteItems to the DB NOW via $push to the incompleteItems array. Also push the enrichmentHistory entry at this point. Do not wait.

Step 5: Update the compaction

With a database: write via a direct MongoDB update, as in a reference setup (the compaction API's own POST endpoint requires a real session UUID, which post-hoc enrichment won't have, so this step goes straight to the database instead):

// Use mongoose updateOne with $set and $push
// NEVER use the Anthropic API key — this is a DB write, not an LLM call
await SessionCompaction.updateOne(
  { shortId: TARGET_SHORT_ID },
  {
    $set: {
      reflectionOnScope,       // FULL /reflect + /align markdown document (Step 1 + Step 2)
      alignmentNote,           // from reflect (Step 1)
      overarchingIntentTitle,   // from align (Step 2) — ONLY if currently empty
      overarchingIntentDescription,
      targetUxIntentTitle,      // from align (Step 2) — ONLY if currently empty
      targetUxIntentDescription,
      scopeDeclaration,         // from align (Step 2)
      agentReasoning,           // from reflect (Step 1)
      leverageScore,            // from compact-agentic-session (Step 3)
      leverageAssignedBy: 'sonnet-enrichment-pipeline',
      leverageAssignedAt: new Date(),
      uxCriticality,            // from compact-agentic-session (Step 3)
      preExecutionReflection,   // from compact-agentic-session (Step 3)
      successMeasure,           // from compact-agentic-session (Step 3)
      howToConsumeThisWorkProduct, // from compact-agentic-session (Step 3)
      roadmapPhaseId,           // from phase matching (Step 3) — match to most similar phase
      categoryIds,              // from category matching (Step 3) — match to parent category
    },
    $push: {
      enrichmentHistory: {
        enrichedBy: 'sonnet-enrichment-pipeline',
        enrichedAt: new Date(),
        enrichedFields: [/* list every field you touched */],
        confidenceLevel: 'high',
        modelUsed: 'claude-opus-4-6' // or whatever model this agent is
      }
    }
  }
);

Without a database: apply the same $set/$push shape to the local JSON file instead — merge these same fields into <records folder>/<shortId>.json and write it back.

Only update fields that are empty or clearly inferior. If a field already has rich content, don't overwrite it.

PRINT the update result (acknowledged, modifiedCount). This is auditable evidence that Step 5 happened.

Step 6: Verify and print audit report

After update, read back the updated record and print ALL of these:

1. shortId and title
2. Where to see it: if you have an admin URL configured (see /alignment-harness:harness-setup), print the full link, e.g. <your-admin-base>/admin2/session-compactions/{shortId} (one reference path shape). Otherwise, print the local JSON file's path.
3. alignmentNote — full content
4. leverageScore with reasoning
5. overarchingIntentTitle — the company-level goal
6. targetUxIntentTitle — the testable UX promise
7. scopeDeclaration.intent and scopeDeclaration.decomposed
8. uxCriticality.criticalityScore and reasoning
9. successMeasure
10. howToConsumeThisWorkProduct — full text
11. incompleteItems count and titles of any NEW items added
12. enrichmentHistory — latest entry showing what was touched

THEN print this audit checklist:

AUDIT CHECKLIST:
[ ] Step 0: Read compaction — printed shortId, title, content length, field status
[ ] Step 1: /reflect INVOKED via Skill tool — printed reflect output
[ ] Step 2: /align INVOKED via Skill tool — printed align output
[ ] Step 3: /compact-agentic-session INVOKED via Skill tool — printed field values
[ ] Step 4: /incomplete-tasks-get-from-session INVOKED via Skill tool — printed extracted items
[ ] Step 5: DB update — printed update result (acknowledged: true, modifiedCount: 1)
[ ] Step 6: Verification — printed all 12 items above

Mark each with [x] if actually done, [ ] if skipped. If ANY are [ ], the enrichment FAILED and must be redone.

Batch Mode

For processing multiple compactions, dispatch parallel subagents running on an interactive subscription seat, not a metered API key (see the cost rule above). Each agent receives:

• The compaction shortId
• The full compactedSession text (pre-loaded — subagents can't access MCP)
• The full text of this skill file
• Explicit instruction: "INVOKE each skill via the Skill tool. Do not approximate."

Max 5 concurrent subagents. Rate limit 1.2s between DB writes.

Quality Audit

After batch enrichment, verify quality on a sample:

1. Pick 3 compactions at random from the enriched set
2. For each, check:
   • Does the alignmentNote actually reflect on intent and unresolved risk?
   • Is the overarchingIntentTitle about the COMPANY goal, not the task?
   • Are incompleteItems real gaps, not padding?
   • Is howToConsumeThisWorkProduct conversational, with a clickable URL?
   • Does the leverageScore match the tier definitions?
   • Does the enrichmentHistory entry exist with all touched fields listed?
3. If 2+ fail quality check, stop and recalibrate before continuing

Skills This Pipeline Depends On

Skill	What it provides	How used	MUST INVOKE?
/reflect	Structured self-reflection (target, gap, initiative, leverage)	Applied to stored session text	YES — Skill tool
/align	Nested intent derivation, certainty scoring, human-legible output	Applied to compaction content	YES — Skill tool
/compact-agentic-session	Field structure, execution order, voice rules	Canonical field specification	YES — Skill tool
/incomplete-tasks-get-from-session	Incomplete work extraction with full item shape	Extract from compaction text	YES — Skill tool