Skip to content
Anurag Raina
Back to projects

Case study

Learner Memory System

An AI biology tutor that remembers what each learner struggles with - and shows why it thinks so.

Role
Solo - architecture and implementation
Year
2026
Status
Live
Live demo(opens in new tab)
The RocketMan AP Biology tutor walking a learner through the Calvin cycle, personalized from the learner memory.

At a glance

AP Biology concepts
12
mastery levels
4
AI prompts, total
2
database tables
5

Overview

Learner Memory System is an AI tutor for AP Biology that keeps notes on the person it is teaching. As a learner works through a topic, it records what they have shown they understand and which wrong ideas keep coming back, then uses those notes to answer differently next time.

The notes are not hidden. A dashboard beside the chat shows what the tutor currently believes about the learner, and every belief links back to the message that produced it. Underneath: one Next.js app over Postgres, and two prompts - a Memory Extractor and a Tutor. No vector search, no agent framework.

Why I Built It

A chatbot that answers biology questions is one prompt and an afternoon of work. What it cannot do is notice that a learner keeps making the same mistake, or that something they got wrong last week has finally clicked.

Most chat memory is the transcript itself, which nobody can point at, correct, or show to the learner. So the design question was what a learner model should be if it has to be displayed on screen and defended: a small set of typed records, each traceable to the message that caused it.

The Approach

The memory is a short list of facts, not a pile of past conversation. Each fact is either a signal that the learner is getting a concept right or wrong, or a named misconception that stays active until they show they have understood it.

Two prompts, one per turn: an Extractor that returns a validated structured object, and a Tutor that reads a scoped slice of memory plus a short window of raw turns. Durable memory and the ephemeral window stay separate, which is what stops the design collapsing back into memory-as-transcript.

Deliberately not built, each for a stated reason:

  • No vectors or embeddings - the concept space is closed, so extraction is classification into an enum.
  • No BKT or IRT - mastery is a simple additive heuristic shown as buckets, not a decimal.
  • No auth or multi-tenancy - one learner, with the seam named in future work rather than half-built.

Architecture

Learner Memory System architecture: the learning feedback loopA learner turn flows down the left column: the message goes to the Extract step, one LLM call that returns a Zod-validated structured object and never blocks the answer. Extraction writes to the memory on the right - an append-only log of mastery events and misconceptions with a lifecycle. Learner state is derived on read from that log as mastery buckets and active misconceptions. The Tutor, the second and only other LLM call, reads a scoped slice of that state plus a short raw window, and replies. The reply shapes the next message, closing the loop. Every belief carries the id of the message that produced it, so the memory is inspectable and each belief traces back to its source. The state also renders live on an inspectable dashboard.provenance · every belief cites its messagememory shapes the next turnLearner messageExtract · LLMstructured object (Zod)non-fatal writeTutor · LLMreads scoped memory+ short raw windowTutor replyMemoryappend-only log:events + misconceptionsLearner statemastery buckets +active misconceptionsDashboardinspectable model, livewritesreads
A closed loop: memory is extracted, made inspectable, and shapes the next answer.

Five tables; deliberately no snapshot table for current mastery. memory_events is append-only and the source of truth - one row per observation, a signed delta on a concept. concepts is a fixed 12-slug taxonomy from the AP Biology units, and the Extractor may only emit those slugs, enforced as a Zod enum.

Mastery is computed on read as a clamped sum of a concept's deltas around a 0.5 baseline. Zero events reads as UNKNOWN, distinct from events that net to 0.5. The tutor and dashboard never see the raw number - only four ordinal buckets, a view over the score, so changing a threshold changes the view and never the data.

Every event and misconception carries the id of the message that produced it. That one column powers both the "recent updates" feed and the "why does the AI believe this?" view - no beliefs from nowhere. Vitest covers the pure functions - derive, apply, scope, the extractor schema, the tutor's prompt assembly - without a live provider.

Decisions & Tradeoffs

Derive mastery on read instead of storing a snapshot

Tradeoff accepted

Every read sums a concept's events - irrelevant at this scale, and it buys one source of truth with trivial provenance. Because it is a sum, a replayed extraction would double-count, so idempotency is schema-level: a unique constraint on (source_message_id, concept_id) with ON CONFLICT DO NOTHING.

Alternatives considered (2)
  • A learner_memory table updated on every write
  • A materialized view refreshed on a schedule

A closed 12-concept taxonomy enforced as a Zod enum - no embeddings, no vector store

Tradeoff accepted

The system cannot learn a concept it was not given, so a new subject means editing the taxonomy. In return extraction is classification into an enum - deterministic, debuggable, no index, and no fuzzy near-miss filing a belief under the wrong concept.

Alternatives considered (2)
  • A vector database with semantic retrieval over free-form concepts
  • Free-text concept labels emitted by the model

Tune the extractor for precision over recall

Tradeoff accepted

Most turns record nothing and real signals are sometimes missed - the cheap error, since the next turn can catch it. An invented belief renders on the dashboard for the learner to read, and a product claiming "here is what we know about you" falsifies itself the moment it is wrong on screen.

Alternatives considered (2)
  • Capture every possible signal and let the dashboard filter
  • Attach a confidence score to each belief and dim the weak ones

Make extraction non-fatal and keep it off the tutor's critical path

Tradeoff accepted

A failed extraction costs one observation. That is fine because extracting the current message cannot improve the current answer - the tutor already reads it directly; memory's value is historical. So a provider error or a bad concept logs, skips the write, and still answers.

Alternatives considered (2)
  • Fail the turn when extraction fails
  • Retry extraction inline before answering

At most one active misconception per concept, enforced by a partial unique index

Tradeoff accepted

Two distinct false beliefs on one concept cannot coexist (recorded as future work). In exchange resolution is unambiguous - one row to flip - and a repeated mistake cannot spawn duplicate beliefs on the dashboard.

Alternatives considered (1)
  • Multiple active beliefs per concept, matched on resolution by text similarity

Engineering Challenges

Detecting that a learner has actually understood something.

How it was solved

Resolution is the headline behavior, and its trigger messages are the least informative - "got it", "I was wrong". The Extractor gets two reference-only inputs, the previous tutor turn and the active misconceptions, and precision holds by construction: a resolution may only name a concept on the active list, and the Extractor never derives a signal from the tutor's words.

Making sure a retry cannot count the same evidence twice.

How it was solved

Derive-on-read means mastery is addition - exactly what a retry corrupts by silently double-counting. The guarantee lives in a unique constraint rather than application bookkeeping, so the pipeline can run any number of times over one message and produce the same events.

Behaving well on day one, when the tutor knows nothing about the learner yet.

How it was solved

With every concept UNKNOWN the tutor must be a good ordinary tutor and must not invent a history to sound personalized. A seed script populates one learner through the same tables and apply path as live data, so cold start is visible but never the reviewer's first impression.

Features

  • Structured, inspectable learner model: per-concept mastery and named misconceptions
  • Every belief traceable to the message that produced it
  • Misconceptions with a lifecycle - active until the learner demonstrates understanding, then resolved on screen
  • Tutor reads a scoped slice of memory, not the whole profile
  • Dashboard feed of model changes, next to the chat

Tech Stack

  • Next.js 16 (App Router)
  • React 19
  • TypeScript
  • Tailwind CSS 4
  • shadcn/ui
  • Supabase (Postgres)
  • Vercel AI SDK
  • Google Gemini
  • Zod
  • Vitest
  • Vercel

Lessons Learned

  • The invariants that mattered ended up in the schema, not the prompt - a unique constraint made extraction idempotent, a partial index made resolution unambiguous.
  • Provenance is a feature, not bookkeeping. Once every belief points at its cause, "why does the AI believe this?" becomes a query.
  • Deciding what memory was not - transcript, embeddings, a snapshot - removed more work than any feature added.

Screenshots

Learner Memory System landing page - headline 'A tutor that remembers how you learn', with actions to open the tutor or view the memory.
The learner memory dashboard - concept mastery grouped into strengths, developing areas, weak spots and misconceptions, shaped by observed messages.

Contact

If you need an engineer who can own it end to end, from the interface to the infrastructure, let's talk.

Email me