Can a website's voice agent remember returning visitors — and should it?

Hypothesis

The voice guide on this site (bottom-right corner, every page) already knows the site. It does not know you. Every conversation starts from zero: same greeting, no memory that you asked about the ecosystem graph last Tuesday, no way to pick up where you left off.

A good host remembers. The hypothesis: give the agent a persistent memory — a knowledge graph built from past conversations — and the experience shifts from "voice-shaped FAQ" to something closer to a returning conversation. Our graph memory showdown showed entity extraction beats flat retrieval on people-shaped questions; this experiment points the same machinery outward, at the site's own visitors.

What we built

We built the whole thing, twice over one day — first the full vision, then a deliberately smaller version of it.

The full loop (built, then parked). A cognee instance on our own server behind a small FastAPI wrapper. Your browser minted an anonymous random ID on your first voice session; when a conversation ended, ElevenLabs posted the transcript back (HMAC-signed), and it was summarised and fed into a knowledge graph belonging to that visitor alone. Come back and the agent opened with "Hey, welcome back — want to pick up where we left off?", carried summaries of your past conversations in its context, and could answer "where did we leave off?". We verified the loop end to end with simulated conversations.

What's live now. The same memory service, minus you: a knowledge graph of the entire site — every page and lab entry, rebuilt automatically after each deploy — that the agent searches mid-conversation through a recall_memory tool. Deeper answers about what's on the site, no data about who's asking.

Learnings

The lawyer decides what ships, and that's the finding. Before real traffic we ran the design through a GDPR review. Verdict: the architecture was sound, but memory across visits legally requires opt-in consent — the browser identifier falls under the ePrivacy terminal-equipment rule, where "legitimate interest" cannot substitute for a checkbox. Verbatim transcripts kept forever, no deletion path, no notice: each a finding on its own. The review priced the fix at about a day (consent UI, retention, a "forget me" that also deletes the processor's copies). We chose to park recognition instead and ship the site-knowledge half, which needs none of it. Memory for anonymous visitors is mostly a consent-architecture problem with a graph database attached.

The plumbing is the product. The cognee part was hours; the trust boundaries took the day. Browsers can't hold API tokens, so greetings went through a server-side proxy; webhooks need HMAC verification; visitor IDs are capability tokens and get strict format checks. A security review mid-build flagged the ID validation as too loose — correctly.

Embedded databases want one owner. Kuzu and LanceDB are single-writer: the moment a second process touched the store we got lock conflicts and one genuinely puzzling segfault. The fix was architectural, not a workaround — every read and write goes through one long-lived service, and the site re-ingest became an API endpoint instead of a script.

Hardware is part of the stack. We resized the server mid-build (80 → 160GB disk, Intel → AMD), and cognee's multi-tenant auth layer started segfaulting in native code on the new CPU. Isolation moved up a layer — per-visitor datasets enforced by the wrapper — and the crash disappeared. Debugging plans rarely survive a silicon swap.

Log

• 2026-07-30 — Built the full returning-visitor loop end to end in one session: cognee + FastAPI memory service, per-visitor datasets, HMAC-verified post-call webhook, welcome-back greeting with past-context injection, recall_memory tool, deploy-time re-ingest. Then ran the GDPR review, which drew the consent line exactly where we'd built past it — stripped visitor recognition the same afternoon and shipped the site-knowledge graph alone. Live now: the agent semantically searches the whole site mid-conversation; nothing about you is stored anywhere. Next, if it earns its keep: a consent checkbox, retention windows, and a "forget me" voice command — recognition done right, or not at all.