Enterprise agents miss tacit knowledge

Every company has context that never makes it into a doc. Who owns migrations. Why per-seat pricing lost. What a partner said in last Monday’s sync. Frontier agents don’t know any of it, and connector search only gets you partway there.

We wanted to know whether a knowledge graph built from how people actually work could change that. So we ran the same agent on 24 real questions about our own company, under four setups: no context, SaaS connectors, a knowledge graph, and both together.

Agent + Knowledge Graph scored 96.4 out of 100. Agent + SaaS Connectors scored 53.6. That is a 92.3% drop in error, at 3.1× lower cost and 2.5× lower latency. On the 13 questions whose answers live only in captured work context, connectors scored 26.1. The knowledge graph scored 98.8. And on the metric that matters most for trust: Agent + SaaS Connectors made up institutional facts on 20.8% of tasks. Agent + Knowledge Graph made up none.

The problem

Most of what makes a company work is tacit knowledge: the judgment, taste, and institutional memory that people carry but rarely write down. Who actually owns a migration. Why per-seat pricing lost to a flat pilot fee. What a partner said in last Monday’s sync that changed the roadmap. A new hire picks this up in hallway conversations and six months of context. An agent gets nothing.

Enterprises run on this kind of knowledge more than they admit. Onboarding takes months because the docs describe the system, not how the team actually uses it. Decisions get re-litigated because the rationale never left the room where it was made. When someone leaves, the workaround they invented walks out with them. The org keeps running, but the reasoning behind how it runs thins out.

Connector search was supposed to fix this. Point an agent at Slack, Notion, GitHub, email, and let it retrieve what it needs. That works when the answer was written down somewhere searchable. It fails when the knowledge was never captured at all, or when it was captured in a form the search never surfaces: a half sentence in a call, a rejected option in a decision that never became a doc, a release process that lives in one engineer’s head because CI broke three months ago and everyone adapted without updating the runbook.

That gap is what we mean by enterprise context. Not more files. The tacit layer underneath them: who decided what, under what constraints, with what tradeoffs rejected, and what the team would actually do today. Without it, a frontier agent either hedges or guesses. For a consumer chatbot, guessing is annoying. For an enterprise assistant, it is poison.

Civitas captures that tacit layer from how people work at their laptops and turns it into a structured knowledge graph. The question we wanted to answer: does the graph actually make a real agent better on real company tasks where tacit knowledge is the whole point?

We ran the experiment on customer zero: ourselves. Ten people, pre-revenue. One where tacit knowledge outruns documentation by a wide margin. If the graph helps anywhere, it should help here.

Benchmark design

The question: given the same agent and the same tasks, does a knowledge graph from captured work beat connector search alone? Not just on accuracy. On fabrication, cost, and speed too.

Four setups, one variable

Same model (claude-sonnet-5). Same system prompt. Same read-only tools wherever tools were allowed. The only thing we changed was what each setup gave the agent access to.

24 questions, each with a verified answer

We wrote 24 tasks across five categories: decision rationale (6), current state (5), judgment (7), architecture (4), and procedure (2). Every task has a reference answer verified by the team, two to eight key facts the judge checks one by one, and a label for where the answer actually lives:

• Findable via connectors (10): written down somewhere in SaaS or the codebase. These test whether the knowledge graph is faster and cheaper, not whether it is the only way in.
• Knowledge graph only (13): the answer never got checked in. A partner quote from a sync. A rejected pricing model. The incident runbook. The team roster. These are the tasks that carry the thesis.
• Public (1): a sanity check any model should pass. Every setup scored 95. The harness does not handicap anyone.

Seven tasks are deliberately adversarial. Each one poses a proposal that sounds reasonable if you do not know this company: blur PII instead of blacking it out, hide the capture indicator in full-screen, .unwrap() on the capture hot path, rename an enum in one shot. A good engineer without company context might say yes. Our logged standards say no. Those tasks test whether the agent can apply the company’s taste, not just recall facts.

The knowledge graph

Fifteen structured notes, shaped like what Civitas’s Scribe pipeline produces from laptop context: a decision log with rejected options, procedures for release and incidents and code review, taste rubrics for engineering and product and security, partner state with sync history, a team roster with review routing, and the current sprint with blockers.

One important caveat: for this experiment, the notes were hand-written, not mined from live capture. They reflect our real current state (ten people, pre-revenue, actual decisions and blockers checked against the codebase) with realistic operational detail added on top. That lets us isolate one question: does a context layer of this shape help? The separate question, how well does the mining pipeline produce it, comes next. See Limitations.

How we scored it

For every task and every setup we measured correctness (0–100, graded by claude-opus-4-8 against the reference, blind to which setup produced the answer), key-fact recall, and fabrication (stating something that contradicts the reference, scored separately from honest “I don’t know”). We also logged cost, tokens, latency, and turns.

Each setup ran in an isolated directory, rebuilt fresh before every run. No setup saw the task file, the reference answers, another setup’s context, or prior results. Read-only tools only. No shell, no network. The judge never knew which setup it was grading.

We also admission-checked the “knowledge graph only” labels. Agent + SaaS Connectors ran on every task. If it got a knowledge-graph-only question right, we flagged it. Two tasks did (both judgment tasks; more below).

Results

96 runs total: 24 tasks × 4 setups, one run each.

Where the answer lives

Two results stand out. First, the moat: on knowledge that never made it into a connector, search is almost as useless as no context at all. Second, the surprise: the knowledge graph beats connectors even when the answer is written down somewhere. A curated summary of what the code means outperforms grepping the code, at a third of the cost.

By category

What the transcripts show

Numbers tell you the gap. Transcripts tell you what it feels like.

When search finds nothing, it still costs you. On pilot pricing, Agent + SaaS Connectors searched three repositories, concluded honestly that no evidence exists, and burned 30 turns and $0.83 doing it. Agent + Knowledge Graph answered in ten turns with the decision, the date, and the options the team rejected.

When search finds something, it can be worse. Asked how releases actually ship while CI notarization is blocked, Agent + SaaS Connectors pulled real facts from different parts of the codebase (Sparkle config here, notarytool mentions there) and stitched them into a confident, coherent, wrong answer. The judge’s note: “describes a Sparkle/DMG/notarytool-apple-id pipeline that contradicts the reference.” The actual recipe lives in operational knowledge, not in the repo. Partial evidence plus fluent writing equals plausible fiction. That is what a new hire gets from a stale wiki too.

Taste transfers too. On the adversarial judgment tasks, Agent + Knowledge Graph scored 98.9. It rejected blur over blackout because blur is reversible. It rejected hiding the capture indicator because trust is non-negotiable. It rejected hot-path .unwrap() because we do not panic in production. It routed the review to the right person.Agent + SaaS Connectors scored 66.1. It often reached the right verdict, but grounded it in invented or generic policy. Judgment is not just the call. It is knowing why.

Limitations

• The graph was hand-built. The notes reflect our real state, and the knowledge-graph-only reference answers match them by construction. This benchmark tests retrieval and application, not whether live mining produces notes this clean. Production mining will be messier. The ten connector-findable tasks, verified independently against the codebase, do not share that circularity. The knowledge graph setup’s lead holds on them too.
• One run per cell. One agent model. One judge. No variance estimates. The gaps are forty points, not four, so the direction is clear even if the error bars are wide.
• One company, one week, 24 tasks. Current-state answers go stale fast. That is by design. The benchmark has to be rebuilt for each organization, which is also the product.
• Agent only mostly hedged. Told to ground claims in evidence, it said “I don’t know” more than it guessed. A looser prompt would likely raise its 4.2% fabrication rate.
• Total cost: about $50 in agent spend, $12 in judging, four hours wall clock.

Agent: claude-sonnet-5. Judge: claude-opus-4-8, blind to setup. 96 cells, July 1, 2026. Harness, tasks, knowledge graph, and raw and graded results live in the benchmark workspace.

@techreport{civitas2026ccb,
  title   = {Enterprise Agents Miss Tacit Knowledge:
             Benchmarking a Laptop-Derived Knowledge Graph
             on Real Company Tasks},
  author  = {Subconscious Intelligence Research Team},
  year    = {2026},
  month   = {July},
  note    = {Civitas Context Benchmark (CCB), Experiment 1}
}