Skip to main content

Knowledge Layer — product specification (draft)

Status: APPROVED — owner approval 2026-10-10, with all four open decisions (§12) approved as recommended. Author: maverick-muse-planner-001, on the chief of staff's dispatch, 2026-10-09; finalized by the author on the owner's approval. Owner rulings of 2026-10-09 and the chief's rulings on them are incorporated as decided. Goal: goal_04c4bd036675.

1. Purpose and problem statement​

Justin's state and memory are scattered across the surfaces he works on: each frontier vendor's harness keeps its own memory, in its own form, behind its own account. Moving from one AI to another means moving context by hand, or losing it. His words: "I need a better way to keep my data mine no matter which frontier vendor I'm using."

The fleet already runs an implicit, file-based version of the answer: MEMORY.md and daily notes on this machine, a docsite, and a textual ontology the sweeper derives daily (21 concepts, 30 triples at drafting). It works, but it is bound to one harness's file conventions and one machine's directory tree.

The Knowledge Layer formalizes that practice into a service inside the agent orchestration platform: one store, owned by Justin, on his infrastructure, that every agent he uses reads and writes through a vendor-neutral protocol. Harnesses become replaceable front-ends to memory that is not theirs.

Success criterion: switching vendor or harness loses nothing. An agent that has never run before comes online, connects, pulls its brief, and works with the same facts, decisions, and project state the previous agent had — because none of it lived in the previous agent.

Clients (owner ruling 1): Muse, ChatGPT, Gemini, Claude, and Hermes — in every case the agent as grounded on a machine (local app, CLI, or host), not a web chat surface. Machine grounding is what makes a single protocol surface possible (§6) and keeps the estate off third-party servers.

2. The three-axis model​

Every item in the layer sits in a three-axis space.

  • North–South: lifecycle tiers. Working, short-term, long-term, archive. North is hot: working items form the default context every agent starts from. South is cold: archived items are compressed, out of default context, and fully retained. Items drift south as they go unaccessed and are pulled north by use (§5). Tiers govern default context assembly and storage temperature — never findability (§5.1).
  • East–West: surface versus substance. West is the governing side: the primitives, the rules, the permissions, the curation loop — what may be done, by whom, and what happens automatically. East is the AI surface: an MCP server (§6) through which agents exercise the read and write primitives. Vendor neutrality lives here: the east surface speaks a protocol every named harness already speaks, so no client is written against a vendor's memory API.
  • Z: compartments. Access boundaries stacked through both other axes. Every item has exactly one home compartment; every session is scoped to the compartments it may see (§7). A brief is the working tier, across the session's permitted compartments, on the north face of the stack.
N — working
· freshness ring (7d) at the core
S — short-term
W primitives S — long-term (canon, chief-gated)
and rules ←→ S — archive E — MCP surface
Z: compartments as parallel planes
through all of it (one home per item)

3. Item model​

One adaptable item model (owner ruling 2). An item is a fixed envelope plus a typed payload; the envelope is what the machinery (tiers, search, gates, cascades) operates on, the payload is what agents read.

The envelope (fixed for all types): id; type; title; summary (one or two lines, the form briefs and search results carry); body (the full content, or a reference to it for document-type items); compartment (exactly one, §7); tier; tags (free topics — tags are not compartments); links (to ontology concepts and to other items); provenance (below); the three clocks (below).

Initial type list (closed): fact, preference, decision, project, document, episode, person, commitment. The list is closed because the ladder, the gate, and the brief composer reason per type; a new type is a spec amendment with rationale, not an improvisation. Payload expectations per type (stored as JSONB, validated loosely at store time — the envelope is strict, the payload is advisory):

  • fact — a statement that is true, with its source.
  • preference — what Justin likes or wants by default.
  • decision — what was decided, when, and what it supersedes.
  • project — an initiative with a state (active, completed, dormant); the link-state driver of §5 reads these.
  • document — a pointer to a document (docsite path, repo path, URL) with its summary; the body lives where it lives.
  • episode — something that happened: a meeting, an event, a completed piece of work.
  • person — a person and Justin's relationship to them.
  • commitment — a promise, deadline, or recurring obligation, with its due state.

The three clocks (chief's ruling): every item carries system time (created, updated, last-accessed — drives the tier ladder), domain time (when the thing occurred, was completed, or is due — drives historical and time-bound queries), and tier state (current tier, plus the gate state for canon candidates). Keeping domain time separate from system time is what lets "what projects did we complete this year" and "what does Justin have going on this week" resolve as filters over stored facts rather than as guesses about recency (§5.4).

Provenance: every item records who stored it (agent identity and class), when, from what (a conversation, a file, a document, another item), and its derivation chain where it was derived from other items (a summary names its sources). Provenance is append-only and is what the delete cascade (§4) walks to find derivatives.

4. Primitives and rules (west)​

The west side defines what can be done. The east surface (§6) exposes these; it invents nothing of its own.

  • store — create or update an item (envelope + payload). Updates supersede: the prior version is retained in the item's history, and a decision that replaces another links to it as superseded.
  • get — fetch one item's full content by id. A get is an access (§5.3) and resurrects (§5.3).
  • search — full-text and semantic search across all tiers (§5.1), scoped to the session's compartments, with filters (type, compartment, tags, domain-time range, tier).
  • ask — a question-shaped search: domain-time and type filters applied from the question's terms ("completed in the last 30 days" becomes a domain-time completion filter over project/episode). Ask returns evidence items; the calling agent composes the answer. The layer runs no model of its own.
  • brief — the default context assembly (§6.3).
  • link — connect an item to an ontology concept or another item. Links drive the link-state tier driver (§5.2) and are subject to the firewall (§7).
  • promote — move an item north explicitly (any agent, for items it can see). Touch also promotes (§5.3); this is the deliberate form.
  • demote / curate — the sweeper's southward moves, executed under §5's rules, not by hand.
  • gate — the chief's long-term admission decision (§5.5). Exercised by the chief (or escalated to Justin); not available to ordinary sessions.
  • delete — hard deletion (owner ruling 6): Justin-only, per item or named batch. Deletion is a primitive, not an accident of curation; nothing else in the system deletes.
  • forget-cascade — deletion's reach: deleting an item cascades to its body, its embeddings, and its derived items and summaries, found by walking provenance; derivatives that mix sources are flagged for Justin's decision rather than silently kept or silently deleted (the forget-cascade problem, named in the ruling). Deletion leaves a content-free tombstone in the provenance log — item id, type, and deletion date, nothing else — so the audit trail stays unbroken (owner decision, 2026-10-10).

Write permissions per agent class. Classes: owner (Justin's own sessions), chief, sweeper, fleet desk (other fleet agents), external harness (Hermes and other non-fleet harnesses Justin connects). All classes may get, search, ask, brief, and promote within their compartment grants. Fleet desks may store and link within their grants — the layer is the fleet's shared memory, not a read-only publication. External harnesses start read-only (brief, search, get, promote); store and link are enabled per harness once earned (owner decision, 2026-10-10). The sweeper additionally executes demote/curate under rule. The gate belongs to the chief (with escalation to the owner). Delete belongs to the owner alone: no other class's session is even offered the tool (§6.2).

Conflicts (the rule — owner decision, 2026-10-10). When two agents store contradictory items: in the working and short-term tiers, both items stand, linked as contradicting, each with its provenance; retrieval surfaces the contradiction rather than silently picking a winner, except that an item stored by Justin himself outranks agent-stored items in brief ordering. At the long-term gate, a contradiction with canon is exactly what the gate exists to catch: the chief resolves it (one item admitted, the other superseded or rejected) or escalates to Justin. Automated resolution is deferred (§11). This was the spec's proposal; the owner adopted it as the rule on 2026-10-10 (§12).

5. Tier lifecycle​

5.1 The governing rule​

Tiers govern default context assembly and storage temperature, never findability. Search and ask are always tier-transparent: a query returns what matches, from archive as readily as from working, with the tier shown. Archiving compresses and summarizes an item and removes it from default briefs; it deletes nothing, hides nothing, and demotes no fact. (This generalizes the fleet's archive-first protocol: south is a temperature, not a disposal.)

5.2 The ladder and its two drivers​

Southward movement is driven by days since last access (chief's ladder reconciliation of owner ruling 3), with one boundary per step:

  • Working → short-term at 14 days unaccessed — for items whose only claim on working was recency. Items linked to an active project, initiative, or data point are pinned in working while the link lives and they are accessed within the window; inside working, the freshness ring (accessed within 7 days) is the hot core a brief leads with, and days 8–14 remain working outside the ring.
  • Short-term → long-term at 28 days unaccessed — through the chief's gate (§5.5). Long-term is the canon tier: what future agents treat as settled truth. Admission is a decision, not a drift.
  • Long-term → archive at 60+ days unaccessed, by rule. Canon that nobody has touched in two months cools to archive; it remains canon in standing — an archived decision still outranks a working-tier rumor — and resurrects on touch like everything else.

The second driver is link-state: when a project completes, its linked items lose their working pin immediately (they then age by the clock alone); when a project reactivates, its linked items are pulled north to working. The sweeper knows all active, current projects (owner ruling 3) and applies link-state changes in its daily pass (§10).

5.3 Access, and resurrection​

An item is accessed only when an agent deliberately fetches its full content through the east surface (a get, or inclusion in a brief the agent actually consumed), or when Justin reads it. List scans, search-result impressions, sweeper curation reads, and indexing never heat an item — otherwise the machinery's own housekeeping would keep everything warm forever, and being findable would be indistinguishable from being used.

Touch is resurrection: fetching an item from any southern tier promotes it to working immediately and resets its clock. Moving an item back from archive must be easy (owner ruling 3), and this is the mechanism: no restore procedure, no special request — the first real use is the restore. The explicit promote primitive (§4) covers the deliberate case (an agent preparing context it has not yet fetched).

5.4 The owner's example queries​

  • "What does Justin have going on this week?" — resolves in working: commitments and episodes filtered by domain time to the current week, from the freshness ring outward.
  • "What projects did we complete this year?" — a historical query: project items with domain completion time in range, searched across all tiers (§5.1), temperature irrelevant.
  • "Completed in the last 30 days" — the same shape: a domain-time filter, not a tier or a recency guess.

5.5 The long-term gate​

The sweeper proposes admissions in batches — candidates at 28 days cold, or link-dead (their project completed and them unaccessed since) — through the fleet's decision-queue pattern. The chief approves, returns, or holds each batch; anything he cannot decide escalates to Justin (owner ruling 4). Admission is where canonical summarization happens: the admitted item's summary is rewritten to stand alone as canon (the sources remain linked in provenance), so long-term holds the settled form, not the raw accumulation. A candidate the gate declines stays in short-term and re-ages from there; declining is not deleting.

5.6 Summarization at each southward step​

  • Working → short-term: none. The item leaves default briefs; nothing about it changes.
  • Short-term → long-term: canonical summarization at the gate (§5.5). The full body is retained; the summary becomes the canonical statement.
  • Long-term → archive: compression — the item is carried by its canonical summary plus a compressed body reference, out of default briefs, full content retained and searchable (§5.1). Resurrection restores the working form from the retained content.

6. East surface: the MCP server​

6.1 Form​

An MCP server on Justin's own infrastructure (chief's ruling). Every named harness — Muse, ChatGPT (desktop), Gemini, Claude, Hermes — speaks MCP, and the clients are machine-grounded (owner ruling 1), so there is no public API, no hosted auth problem, and no vendor in the middle. The server binds to Justin's infrastructure only (his tailnet / local network); it is not a public endpoint.

6.2 Tool set​

One tool per primitive, mapping 1:1 to §4: kl_store, kl_get, kl_search, kl_ask, kl_brief, kl_link, kl_promote. Rule-side primitives are exposed by class: kl_gate is registered only for chief-class sessions; kl_delete (with its cascade report) only for owner-class sessions. A session is configured per harness with an agent class and compartment grants (§7); the server enforces both — a tool a class may not use is not merely refused, it is not offered.

6.3 The brief​

kl_brief returns the session's starting context, composed as: the freshness ring core (working items accessed within 7 days), then the working remainder (working items at 8–14 days, plus link-pinned items), then the active-project list (the project items currently active, with their states) — all within the session's permitted compartments. Any agent coming online pulls this and is current (owner ruling 3); reaching farther back is a search away (§5.1), and southern items it actually fetches resurrect (§5.3).

7. Compartments​

Start set (chief's ruling): personal (personal/family), finance, work (X-Centric), fleet (fleet/projects), divorce. Few on purpose: splitting a compartment later is cheap; merging is hard.

One home. Every item lives in exactly one compartment. Topics are tags, applied freely; a compartment is an access boundary, not a filing label, and an item that seems to belong in two places is stored in the more restrictive one and tagged for the other.

Access model. Grants are explicit per harness/desk configuration, default-deny. Indicative defaults, for the owner's confirmation at build time: fleet desks are granted fleet and work; the finance desk finance (and fleet); the chief's sessions are granted broadly, as Justin routes cross-cutting work through the chief; Hermes and other external harnesses are granted what Justin names for them, starting narrow. divorce is granted only to sessions Justin names for divorce work.

Firewall. divorce is firewalled, generalizing the existing precedent (divorce material never enters the derived ontology): no link crosses the firewall boundary in either direction, the ontology derivation never proposes one, and no summarization carries content across it. The firewall binds the machinery regardless of session grants — a session granted both sides still receives no cross-boundary links.

Cross-compartment behavior. A session searches the union of its granted compartments, filtered before ranking: results from ungranted compartments do not appear, and neither do their counts, titles, or any existence signal. A brief composes across the session's grants (§6.3) and no further.

8. Storage design​

One self-hosted Postgres is the system of record (chief's ruling). No third-party database service — hosted storage would hand the estate to a vendor, which is the condition this project exists to end. The server runs in Docker on Justin's infrastructure, consistent with the runtime's Docker direction — hosted on the owner's Azure host (owner decision, as amended 2026-10-10), and the MCP server of §6 runs beside it there. The host is created by fleet-built provisioning automation that is part of the build (lane story KL-01a): the fleet does not hand-provision the layer's home. The sequence of record — the approval first named jkdev001; an amendment superseded it with an owner-provisioned Azure host; a second amendment the same day made the host's creation fleet-built automation — stands in §12, and the host of record is the automation-built Azure host.

Host amendment (2026-10-10, owner, final): the HYBRID. The Azure VM host is superseded. Supabase hosts the database only — Postgres + pgvector + JSONB — starting on the free tier (the inactivity pause mitigated by the fleet's daily touch cadence) and moving to Pro when warranted. Azure Container Apps hosts everything that runs: the MCP server with a Tailscale sidecar (agent access stays tailnet-only), and the tiering/promotion jobs as scheduled Container Apps Jobs. Supabase credentials are held only by the Azure-side workloads, in Azure-side secret storage — never issued to agent callers; per-caller MCP tokens and the compartment gate at the MCP layer are unchanged. The MCP-to-database leg crosses the public internet over TLS via the Supabase pooler, allowlist-hardened to the deployment's pinned Azure egress — the one deliberate exception to the tailnet-only posture, accepted by the owner knowingly (no Tailscale path into hosted Supabase exists). Exit is preserved: the database is plain Postgres, so repointing the Azure side at an Azure Flexible Server moves the data home without touching the agent-facing layer. Section 12 decision 1 is amended accordingly; the provisioning automation re-cuts as lane story KL-01d, and KL-01a/KL-01c stand done on their build legs with this supersession recorded.

Schema sketch (tables, not DDL):

  • items — the envelope: id, type, title, summary, body (or body reference), compartment, tier, tags, the three clocks' fields, gate state.
  • item_versions — superseded bodies and summaries, retained per §4's update rule.
  • item_links — item↔concept and item↔item links, with kind (including contradicts and supersedes).
  • provenance_events — append-only: who stored/derived/ promoted/gated/deleted what, when, from what.
  • access_events — deliberate fetches only (§5.3); the ladder's input. Curation and indexing write nothing here.
  • embeddings — per item (and per body chunk for long items): the vector, the model identifier, and its version.
  • ontology_concepts, ontology_triples — the sweeper's derived ontology as tables in the same database, so links join items to concepts without a second store.
  • compartments, agent_grants — the compartment registry and per-class/per-harness grants (§7).
  • gate_queue — admission batches and their dispositions (the decision-queue pattern, in-database).

Search. Postgres full-text search (tsvector) for lexical retrieval and pgvector for semantic retrieval, merged at query time; both run over every tier (§5.1) and inside the session's compartment filter (§7). JSONB payloads per item type (§3) ride on items.

Embeddings are local. Vectors are produced by a local embedding model running on Justin's infrastructure. An embedding API would send the estate's content to a third party — the exact exposure the layer exists to remove — so it is not a configuration option. The model identifier and version are stored with each vector; changing models is a migration event (re-embed the estate), never a silent swap.

Backup posture. Nightly database dumps, plus a human-readable export mirror (items as files, in the estate's existing file conventions) so the knowledge survives even the database: the data is Justin's in a form he can read without any of this machinery running. Archiving in the tier sense (§5) is unrelated to backup; both exist, neither substitutes for the other.

9. Seeding and migration​

The layer starts from the estate that already exists; it is a formalization, not a greenfield (§1). Initial load imports, as items with provenance naming their source:

  • MEMORY.md and the daily memory notes (facts, preferences, decisions, episodes, people) — the curated core.
  • The docsite ontology (concepts and triples) into the ontology tables, and the links it asserts.
  • Docsite project records (docs/03-projects/*/work-record.md) as project items with their states — the link-state driver's initial truth about what is active.
  • People and relationship pages as person items.

Seeds enter at working or short-term by their freshness; canonical material is chief-gated into long-term through the normal gate (§5.5) rather than bulk-admitted — the gate's first batches are the seed's canon, reviewed like anything else. Divorce material seeds only into the divorce compartment, with its firewall intact from the first row. Import is a one-time program with per-source verification (counts and spot checks against the files), not a sync: after seeding, the layer is the store and the files are history.

10. The sweeper curation loop​

The sweeper's daily pass extends to own the ladder, as it already owns ontology derivation and archive-first curation:

  • Tier transitions: compute days-since-last-access from access_events; execute the rule-driven moves (working → short-term at 14 unpinned days; long-term → archive at 60+); apply link-state changes (project completed → pins stripped; project reactivated → items pulled north).
  • Gate proposals: assemble admission batches for short-term items at 28 days cold or link-dead, with draft canonical summaries, and file them through the decision queue for the chief (§5.5).
  • Ontology derivation: the existing daily derivation runs against the layer — new concepts/triples proposed from the day's items, link proposals for items (never across the firewall, §7), and the derived ontology stored in the layer's tables rather than only as files.
  • What does not change: the sweeper's reads for curation never heat items (§5.3); its destructive-action protocol is untouched and mostly moot here — the layer gives it no delete primitive at all (§4), and tier movement is not destruction.

Runbook changes: the sweeper's runbook gains a knowledge-layer section naming the pass above (ladder arithmetic, gate batch format, firewall rule), and its existing ontology section is amended to derive into the layer. The docsite project record for this initiative tracks the runbook change as build work, not as part of this spec.

11. Non-goals and deferred capabilities​

Each deferral carries the trigger that ends it (build for the now; expand on evidence — owner ruling 8).

  • Neo4j / a graph database. Deferred. The ontology is tables in Postgres until multi-hop graph traversal becomes a primary query pattern — that day, a graph store earns its place as a second database beside Postgres, not instead of it.
  • Web surfaces. Out of scope: clients are machine-grounded (owner ruling 1). Trigger: a named need to reach the layer from a device with no local harness.
  • Multi-user. The estate is Justin's. Trigger: a second person with their own compartments and grants.
  • Automated conflict resolution. Deferred (§4's proposal stands meanwhile). Trigger: contradiction volume at the gate exceeding what the chief can resolve by hand.
  • Hosted anything. Not deferred — rejected. Storage, embeddings, and the MCP surface run on Justin's infrastructure; a hosted component would reintroduce the dependency the layer removes. Amended in part (2026-10-10, owner): the database leg alone is hosted (Supabase), under the hybrid of §8 — the owner's knowing exception, with the exit preserved; every other component runs on the owner's Azure side, and the rejection stands for the rest.
  • A model inside the layer. The layer stores, retrieves, and assembles; it does not summarize by model except at the gate, where summarization is the chief's reviewed act (§5.5), nor answer questions beyond returning evidence (§4, ask). Trigger for revisiting: none named; this is a shape decision, not a capacity one.

12. Decisions (owner, 2026-10-10)​

The draft's open questions were four, all genuine owner decisions. On 2026-10-10 the owner approved all four of the chief's recommendations as proposed ("Approve all four recommendations"). They are recorded here verbatim in substance, and their consequences are flowed into the sections named. No open questions remain in this spec.

  1. Host = the owner's Azure host, created by fleet-built automation (amended twice, 2026-10-10). The approval first named jkdev001; minutes later the owner amended it ("I'm going to add another host in azure for this") to an Azure host he would provision; later the same day he amended it again — "Add the azure automation for creating the host to the build process" — and the host of record is the Azure host created by fleet-built provisioning automation, part of the build (lane story KL-01a). The Postgres and the MCP server run there (§8). The machine is on when any agent is. A third amendment the same day (2026-10-10, owner, final) superseded the VM with the HYBRID of §8: Supabase hosts the database only; Azure Container Apps hosts everything that runs; the host of record is the hybrid deployment, created by fleet-built provisioning automation (lane story KL-01d; KL-01a and KL-01c stand as the superseded VM automation, retained). Decisions 2–4 below are untouched by this amendment.
  2. Hermes write access = read-only start. External harnesses begin with brief, search, get, and promote; store and link are enabled per harness once earned (§4).
  3. Deletion tombstones = kept. A hard deletion leaves a content-free tombstone in the provenance log — item id, type, deletion date — so the audit trail stays unbroken (§4).
  4. Conflict rule = the §4 rule as proposed. Contradictions stand linked in the hot tiers and surface in retrieval; canon conflicts resolve at the chief's long-term gate (§4).

13. Proposed build breakdown​

An epic for the AO board, KL — Knowledge Layer, with story sketches for grooming after the owner approves this spec. Sizes are rough Fibonacci sketches, not commitments.

Phase 1 (the chief's slice: the retrofit-painful pieces — item model, storage, surface, ladder, compartments, gate — are in slice 1 by design):

  • KL-01a — Azure host provisioning automation. The enabler, added 2026-10-10 when the owner put the host's creation in the build: creates the Azure host KL-01's stack deploys on (KL-01's enabler; its on-host verification depends on this). ~8. No dependencies. Superseded (2026-10-10) by the hybrid (§8): KL-01a and its companion KL-01c stand done on their build legs, their automation retained as evidence; the provisioning of record is KL-01d — Hybrid provisioning automation (Container Apps + Supabase), and KL-01's verification remainder (KL-01b) re-cuts against the hybrid deployment.
  • KL-01 — Item model and Postgres store. Envelope, types, provenance, three clocks; store/get against one Postgres in Docker. ~8. No dependencies.
  • KL-02 — Search and ask. Full-text + pgvector over a local embedding model; tier-transparent, compartment-filtered; domain-time filters implementing ask. ~8. Depends: KL-01.
  • KL-03 — MCP east surface and session scoping. The §6.2 tool set with per-class tool registration and compartment grants enforced. ~8. Depends: KL-01, KL-02.
  • KL-04 — Brief composition. Freshness ring core, working remainder, active-project list (§6.3). ~5. Depends: KL-03.
  • KL-05 — Tier ladder and the sweeper pass. Access counting (§5.3), rule-driven transitions, link-state driver, resurrection on touch. ~8. Depends: KL-01.
  • KL-06 — The long-term gate. Admission batches via the decision queue, chief disposition, canonical summarization step, escalation to the owner. ~5. Depends: KL-05.
  • KL-07 — Compartments and the firewall. Grant enforcement end-to-end, pre-ranking filtering, the divorce firewall across links and derivation. ~5. Depends: KL-01, KL-03.
  • KL-08 — Seeding. Importers for §9's sources with provenance, verification counts, gate-first canon batches. ~5. Depends: KL-01, KL-06.

Phase 2 (sketches, groomed after Phase 1 lands):

  • KL-09 — Delete and forget-cascade (owner-only, §4). ~5. Depends: KL-01.
  • KL-10 — Ontology derivation into the layer (§10). ~5. Depends: KL-05, KL-07.
  • KL-11 — Contradiction surfacing (§4's proposal: linked contradictions in retrieval and at the gate). ~3. Depends: KL-02, KL-06.
  • KL-12 — Export mirror and backup verification (§8). ~3. Depends: KL-01.

Phase 1 totals 52 points across eight stories; every story is independently reviewable, and the slice ends with the success criterion (§1) testable end-to-end: a harness not used during the build connects, pulls a brief, and works from the seeded estate.


Specification by maverick-muse-planner-001 · drafted 2026-10-09; approved by the owner 2026-10-10 and finalized the same day.