Over the past six or so months, I’ve been doing some variation of workshops on “building and owning agent infrastructure, context and capabilities”. People and organisations understand the importance, but it’s concerning how few act on it. In my opinion, it’s likely to be the difference between individuals and teams who unlock real agent capabilities and those who don’t.

I offer Commons as a starting template, but I recommend building your own. Doing the work yourself helps you understand what context and capabilities actually mean, and build an intuition for why they matter.

The video below is my best attempt at capturing the workshop: why this matters, what context and capabilities mean in practice, and how to build a system around your own work. Watch the full workshop first, then use this article and the references below to explore the ideas further.

How to build a system to unlock real agent work

Watch the full workshop: How to build a system to unlock real agent work

If I had to guess why so few people act on this, based on my limited enterprise experience and the workshops I’ve done:

  • Misleading intuitions about model capabilities: models and agents are increasingly capable, but still struggle in real-world scenarios.
  • Confusion around the model, harness and interface, and what different agent services actually package.
  • Not wanting to do the boring, gnarly work of “looking at the data”: consolidating logs and traces, memory, skills and knowledge, and shaping this around a specific business.

Those are the three things I unpack in the video and explore below: how agents handle messy working environments, what the services actually package, and what it means to build and maintain context and capabilities around your own work.

Agents inside a system you own

Most of us (myself included) still treat AI like a chatbot. We know agents can already do increasingly complex, autonomous work, but the real world is messy. We haven’t built the systems that let them work more like digital labour.

The labs are already showing us where they think this is going. Long-running, packaged agents with memory, tools and their own computers. I use these products and I think they’re fantastic. My bet is that the best organisations will use these agents as one component inside a system they own and actively shape. A service built for hundreds of millions of people can’t decide the shape of your knowledge or how it should accumulate.

I think about that system in terms of context and capabilities. Context is memory, logs and traces. Capabilities are skills, tools and workflows. It’s any information worth collecting and connecting so the agent can reason over it and do better work. So much of it is trapped inside people’s heads, fragmented across departments, conversations and individual tools. Now it’s also trapped inside threads and interactions with agents.

The real world is messy

Imagine you had unlimited tokens. How would you set up an agent to build a million-dollar business?

Jared Zoneraich and Sandra Yang gave Devin a card, a phone number and an email address and called it BugBasher. It looked for restaurants with pest violations, called them, and tried to sell referrals to exterminators. It had a shared “brain” of files containing skills, instructions, transcripts and logs. Each new session could pick up where the last one left off.

It made calls, reviewed transcripts and updated the phone agent’s instructions. When it realised it was hanging up on people who didn’t understand English, it added an offer to speak Spanish or Chinese. Sometimes the phone agent read its own instructions out loud mid-call. BugBasher tried changing the prompt, checked the next calls and eventually escalated when the fixes didn’t work. The operators report roughly $3,000 spent and $75 earned over four weeks. The interesting bit is the loop of doing the work, looking at what happened and changing what the next run gets. Zoneraich and Yang’s write-up

Cognition’s post describing Devin’s BugBasher business experiment

Cognition’s post about the BugBasher experiment. Original post; full account by Jared Zoneraich and Sandra Yang.

Watching popular benchmarks saturate can make it feel like we’re approaching a threshold where models will suddenly be able to do anything. SemiAnalysis worked with specialist data partners to test agents on realistic work across eleven domains. Claude Opus 5.5 scored 12% overall on the GDPx leaderboard shown in the video. In the public Bellwood finance example, the agent had to pick up an existing diligence workbook and work through the deal materials, using the conventions and definitions in the purchase agreement. It had to work out the adjustments to earnings, working capital and net debt and explain where its figures came from. SemiAnalysis reports that none of the models fully solved Bellwood in the 25 attempts it describes. Fifteen runs counted the banker’s success fee in net debt even though the purchase agreement deducted it separately. GDPx leaderboard · Bellwood task

GDPx overall leaderboard shown in the video, led by Claude Opus 5.5 at 12 percent

SemiAnalysis’s GDPx leaderboard as shown in the video. The 12% figure is the overall score across all eleven domains. Explore GDPx.

For more on interpreting these scores, Florian Brand and Jean-Stanislas Denain’s “Why benchmarking is hard” explains how scaffolds and API providers can change measured performance. Epoch AI’s benchmark-review methodology sets out checks for reviewability, scoring defects, consistency and capability elicitation.

Models will keep improving, but they’ll still be jagged and janky. Every business has its own quirks and processes. The people who do well will get good at theory of mind for agents, understanding how and where they fail and what they can do. A lot of that is gnarly data work. Sift through what the agent was given, what tools it used and what it actually did. Then you can decide whether you need a better model, better-curated knowledge or a change to the system.

The model, the harness and the system

The model is the neural network. By itself it’s tokens in and tokens out. The harness is the software that wraps it. The model can decide to call a tool, the harness runs it and passes the result back, and the model decides what to do next. The model and harness together make up the agent. The interface is how you use it.

Agent loop showing the interface, model, response, tool execution and returned tool result

The model chooses a response or tool call; the harness executes the call and returns its result.

Say you ask, “What’s the weather?” The model can call a weather tool rather than answer from its static weights. The harness runs the tool and passes the result back. The model can then answer or choose another tool. Better autonomous work means making useful decisions through more of these loops, with the right information available at each step.

I borrow and build on Theo’s chef-and-kitchen analogy in the video. The model is a knowledgeable chef; the harness supplies the kitchen, ingredients and utensils. Together they make the agent. The interface is the restaurant through which you use it.

Chef as model, kitchen as harness, both as agent, and restaurant as interface

Model = chef; harness = kitchen; agent = both; interface = restaurant. Adapted from Theo’s analogy.

Drew Breunig’s “The State of Context Engineering” traces the changing language around this work, from prompts to context and harnesses. His account of context failures is useful here: the information an agent sees, and how the harness assembles it, can change what it does.

Drew Breunig’s timeline from prompt engineering to context engineering and harness engineering

Drew Breunig’s timeline of prompt, context and harness engineering, from “The State of Context Engineering”.

Keep your context and capabilities separate

My bet is that the harness will get increasingly tied to the model. I’d leave that loop to the agent and put the work into the system it plugs into. Think about what you’d want to take with you if you switched agents tomorrow. The useful bits are the knowledge, processes and capabilities you’ve accumulated doing the work.

Every agent interaction should be able to draw from that shared context and write back what’s useful so the next interaction can pick it up. Most people have skills, workflows and tools saved across different agents and projects, with memory in a wiki or inside an individual agent. I’m suggesting you pull the useful bits into a system you can actively shape.

To extend the restaurant analogy, I want to own the accumulating knowledge and processes from the work, so I can use different chefs, kitchens and restaurants without starting again.

Different agents reading and writing shared context, skills, tools and workflows

The idea: different agents can draw on—and contribute to—the context and capabilities you maintain.

Introducing Commons

I’ve spent a lot of time over-engineering this. The agents have gotten good enough that you only need a very basic system with the core primitives and an easy way for them to read from it and write back. Commons is my bare-bones template for the shared knowledge part.

  • The database stores your work and the context around it so agents can reason across it. I’m using Postgres, which also stores the vectors for semantic search. How you organise and connect that information is your ontology, and it should reflect what you’re trying to achieve.

  • The API gives agents and the UI a consistent way to search, read and write. You define the kinds of records, how they can connect and the rules the agent works within.

  • The MCP tools let the agent use the API. The agent can decide what to search for, read what it finds, follow connections and create or refine records as it works.

Commons architecture: agent and interface, MCP tools, Rust API and PostgreSQL

Commons connects agents through MCP tools and a Rust API to shared knowledge in PostgreSQL.

The database, API and search service run together through Docker. In the packaged setup, the embedding model runs locally through Ollama so semantic search doesn’t need an external embedding API. The retrieval tool combines keyword and semantic search; the agent decides what to read next.

What it looks like in my setup

Animated preview moving through agent services, GDPx, shared context and capabilities, and Pi

A short animated preview of the video: agent services, GDPx, shared context and capabilities, and Pi.

In the video I ask Pi to find my GDPx research and explain how it connects to the launch video. It finds the article, podcast and finance note and pulls in the connection to the launch task. I can open the source in Commons and see the records and traces connected to it.

Then I ask which data partner SemiAnalysis used for finance and ask Pi to create an entity and connect it to the source article. Pi identifies Living Context. I open the source and you can see the new entity in its connections.

The agent could just look up the answer on the web. What I’m interested in is building up this cluster of connected information so the next agent I work with can reason over it. I treat sources, people, organisations and tasks as individual units of information, with links back to the work they came from. Your schema should represent your business and how you work.

There’s a lot I’ve skipped here, especially around search, schema design and evals. This is a work in progress and one particular way to build the shared knowledge part. I’d love to hear from people working on the same problem or interested in collaborating.

If you want to see the whole argument and demonstration, watch the video. You can use Commons as a starting point, follow the setup documentation, or build your own version. I’m also happy to talk through questions in the ENYU Discord.

Video and setup

Research and examples