Does a knowledge graph remember a community better than flat search does?

Hypothesis

Our AI OS already has a memory. One SQLite index holds everything we know about the communities we coordinate: group chats, CRM records, notes, network data, the internal wiki. Questions go in through a hybrid retrieval layer that runs keyword and vector search side by side, fuses the rankings, and hands the top dozen chunks to a model that writes a short answer. It works, and it has been the backbone of most of what we build.

But the questions we actually ask it are not really search questions. They are questions about a network. Who is working on what. Who shows up in three different rooms and plays a different part in each. Who should meet whom. Flat retrieval answers those by finding passages that sound like the question, which is a different operation from following a relationship.

So the hypothesis: if you extract entities and relationships up front and store them as an actual graph, questions shaped like a graph should get better answers. GraphRAG is not a new idea, and there is now a mature open-source implementation of it. Time to find out whether the idea survives contact with our own data.

What we built

We put cognee (Apache-2.0, around 29.5k stars) up against our incumbent memory, on the same corpus, with the same questions.

The corpus is the tracked community group chats described in Unified Group Memory: 33,882 messages across 125 groups, roughly 7MB of text, read-only, with personal and family conversations excluded by rule. Same scoping as before.

The challenger ran in an isolated virtual environment on our own server, fully embedded and self-hosted: Kuzu for the graph, LanceDB for the vectors, SQLite underneath, with gpt-4o-mini doing entity extraction. We bucketed the corpus one document per chat per day, which is the same chunking decision that worked for the incumbent, and ran a pilot on the 150 most recent buckets. That produced a knowledge graph of 1,329 nodes and 3,223 edges in about 90 seconds, for well under a dollar of extraction cost.

Then 20 identical questions to both systems, transcripts saved side by side. Five warm-ups, then fifteen deliberately hard ones: multi-hop chains, contradiction detection, how a topic shifted over weeks, single buried facts, open asks nobody answered, and matchmaking.

The incumbent went in with a real advantage. It had the full multi-source history, including the wiki and the CRM. The challenger had 150 days of chat and nothing else.

Learnings

The graph won, and not narrowly. Roughly 12 of 20 questions went to cognee outright, one to the incumbent, the rest split or too close to call. It won despite seeing a fraction of the data.

The wins cluster exactly where the hypothesis predicted. Anything people-shaped or enumeration-shaped went to the graph, often lopsidedly. Asked which people appear in more than one group chat and what role they play in each, our unified memory returned names with the roles missing and said so honestly; cognee returned the people, both chats, and a role in each. Asked to trace a chain from a report to its presenter to the organisations behind it to others connected to those organisations, the incumbent named the organisations and then suggested we go read the chat history ourselves. Asked for concrete open asks across all groups, the incumbent said the context did not contain any; cognee listed five, with who, what, and where. Same corpus. The difference is that one system had already done the work of deciding what a person is.

Its best moment was about us, not the communities. Asked to find contradictions across the chats, cognee noticed that one of our own workspace chats reported "all decisions logged and implemented" on 10 July, and then on 16 July reported a seventh consecutive day with zero new kanban tasks and drafts sitting unapproved for 168 hours. That is not chat trivia. That is an operational problem we had not noticed, surfaced by a memory system that was asked a completely generic question.

It has no taste. Asked what numbers had been shared recently, cognee dutifully reported a scheme promising 5 to 25 percent daily returns on a $500 minimum, listed under "Trading Investment Opportunity", presented with the same neutral confidence as everything else. It is an obvious scam post in a group chat. The graph extracted it as a fact about the world because that is what extraction does. Anything built on this needs a layer that knows the difference between something that was said and something that is true.

The comparison was not entirely fair, in the incumbent's favour. Some of what looks like architecture is parameters. Our retrieval layer fetches twelve chunks and is instructed to answer in 150 words or less. Several of its losses were enumeration questions, where a twelve-chunk ceiling and a word limit make a complete list structurally impossible. We are not going to claim a clean architectural verdict until we relax those and re-run.

The one thing it still does better is combine sources. The incumbent's single clear win came from a question that needed the wiki and the CRM and the chats at once. cognee only had chats. That is not a limitation of graphs, it is a limitation of what we ingested, and it points at the obvious next experiment.

Running the test audited the incumbent for free. Asked who is currently looking for funding, our unified memory answered with five company names that came from the inbox source and are plainly junk. That is not a retrieval failure. That is spam sitting in the memory, which we would not have found without watching the system answer a question we already knew the shape of. Also, both systems inherit unresolved sender identities from the messaging bridge, so some answers still name people by raw account identifiers rather than by name.

The verdict held up at full scale, and then some. Once the full 33,882-message backfill finished and we re-ran all 20 questions on equal footing — cognee with complete history instead of a 150-day sliver — the graph won roughly 15 of 20, up from 12. More data did not narrow the gap. It widened it.

Then we tested the one thing it was missing: breadth. The incumbent's only categorical advantage was combining sources — wiki, CRM, chats in one answer. So we fed cognee the same sources: the Twenty CRM contact list, the LinkedIn network and message history, and recent meeting analyses, as three more datasets in the same graph. Then asked the exact kind of cross-source question that used to be the incumbent's strength. "Which people from your LinkedIn network have you also talked with in chats or meetings?" Our unified memory answered "I cannot answer your question based on the available information." Cognee named six people with the actual context of each conversation. The breadth advantage was never architectural. It was just data we hadn't given the other system yet.

It also produced its single worst answer of the whole experiment, and the fix was two sentences of prompt. Asked to recommend speakers for a festival we organise, cognee suggested we invite ourselves, and separately suggested inviting "participants of the speakers chat" as though a chat room were a person. Not a retrieval failure — a missing frame. We added a short system prompt: you are Robin's assistant, never recommend Robin to himself, a group of people is not one person, unresolved IDs are not names, prefer recent information. Same question, same graph, re-run: five real people, each with a source and a date, self-recommendation gone. The lesson generalises further than this experiment. A retrieval system without domain framing will confidently produce nonsense that better retrieval alone cannot fix.

The other fixable failure was time, not relevance. Asked for "upcoming" events, cognee cheerfully listed a 2023 conference. Its time-aware search mode parses dates out of the query text itself and ignores the system prompt entirely, so the fix wasn't instruction — it was making our query router rewrite "upcoming" into an explicit "after 2026-07-30" before the request ever reaches that mode. Two lessons stacked on top of each other: know which part of the system reads your instructions, and vague temporal language is a bug magnet regardless of which memory system you're steering.

The infrastructure kept fighting back, and it kept losing on repeat. The full backfill needed eight separate attempts across two failure classes — an OOM kill, and an upstream graph-database bug that corrupts its write-ahead log when a subprocess dies mid-write and then refuses to reopen. Once diagnosed, both got permanent workarounds: one slice per process so a clean exit checkpoints the log before it can rot, a hard memory cap so the worst case is a dead lab job and not a dead server, and a two-line recovery recipe — quarantine the corrupted WAL files, the graph reopens with everything already checkpointed intact. The multi-source ingest hit the same OOM ceiling at a larger per-slice size and needed smaller slices to clear it. None of this is exotic ops discipline. It is the kind of thing a boring SQLite file has simply never required from us.

Storage grew in a way that needed a second look. The vector store alone swelled to 23GB carrying old table versions from every crashed and retried slice — for a knowledge graph that fits in a few million tokens of source text. A one-time compaction pass brought it back to 4GB with no data loss. Worth knowing before deciding to run this as a permanent background process rather than a lab experiment.

The verdict is "add", not "replace" — now with the shape of the addition specified. Entity extraction wins decisively on relationship-shaped questions and, once the same sources are ingested, closes the breadth gap that used to be the flat retriever's whole reason to exist. What it costs is real: fragile infrastructure that needs babysitting at this scale, storage that needs periodic compaction, and answers that need a framing layer or they will recommend you invite yourself to your own event. The two systems earn their keep in different registers — one for daily breadth with near-zero upkeep, one as a specialist you consult for network questions, with real maintenance cost attached. The interesting system is still both, and now we know what running the graph half of it actually requires.

Log

• 2026-07-30 — Full backfill completed: 35,052 nodes, 108,134 edges from all 33,882 messages. Re-ran all 20 questions on equal footing — cognee took roughly 15, up from 12 on the partial pilot. Added three more datasets to the same graph (Twenty CRM contacts, LinkedIn network and messages, meeting analyses — 51,653 nodes, 150,556 edges combined) and re-tested the cross-source questions that used to favour the incumbent; cognee won two of three, including the one the incumbent flatly couldn't answer. Found and fixed two real failures: a self-recommendation bug fixed by a two-sentence system prompt, and a stale-events bug fixed by rewriting vague time language into explicit dates before it reaches the time-aware search mode. Compacted the vector store from 23GB to 4GB. Concluding the experiment here — next step, if it graduates past lab status, is cognee's MCP server as a daily read-only tool inside our coding agent, sitting next to the unified memory rather than replacing it.
• 2026-07-29 — The full backfill fought back, twice. A one-shot ingest of all 5,798 chat-day buckets was killed by the OOM reaper at 5.7GB on a 7.6GB box. The memory-capped retry ran eight hours and died on something more interesting: the embedded graph database crashes with an assertion failure when it reopens and replays a write-ahead log left behind by a killed subprocess — and a clean rebuild hit the same crash with no OOM involved, so this is an upstream bug at our scale, not our residue. Current workaround: every 500-document slice runs as its own process, so each clean exit checkpoints the log away before it can grow teeth, all inside a memory-capped cgroup so the worst case is a dead lab process instead of a dead server.
• 2026-07-28 — Set up cognee in an isolated environment on our own server (embedded Kuzu, LanceDB, SQLite, gpt-4o-mini extraction). Piloted 150 recent chat-day buckets into a 1,329-node, 3,223-edge graph in about 90 seconds. Ran 20 identical questions against both systems and saved the transcripts. The graph took roughly 12 outright, the incumbent 1. Started the full 33,882 message backfill. Next: relax the incumbent's retrieval ceiling and re-run the losses to separate architecture from parameters, and try cognee's MCP server as a read-only network-graph tool inside our coding agent.