---
title: "Agent Harness: 6 Critical Parts Worth Checking First"
url: https://sanjayshankar.me/agent-harness/
description: "An agent harness is everything around the model: the loop, tools, memory, permissions and budget. Six parts decide whether your SEO agent works or just reports."
author: Sanjay Shankar
date_published: 2026-10-18T15:26:56+00:00
date_modified: 2026-10-18T15:29:02+00:00
categories: [Agentic SEO]
tags: ["agent harness", "Agentic SEO", "ai agents for seo", "AI Visibility", "harness engineering"]
reading_time_minutes: 11
---

# Agent Harness: 6 Critical Parts Worth Checking First

**An agent harness is everything around the model: the loop that runs it, the tools it can reach, the context it is handed, the memory it keeps, the budget and permissions that bound it, and the check that decides whether its work counts. In my own tracking, those six parts matter more than which model sits in the middle. Build the agent harness well, and a modest model does useful work. Skip them, and the best model in the market reports numbers nobody can act on.**

Most SEO agents I have looked at are sold as a model plus a set of prompts. That is the cheap half. The expensive half is the scaffolding nobody screenshots: what the thing remembers between runs, which data it is allowed to touch, what happens when it runs out of credits halfway through, and who has to approve a change before it goes live.

I run one of these against my own site every week. It reads a query report, watches a fixed set of prompts across four assistants, and hands me a list of what moved. Seven jobs it does better than I do by hand, which I wrote about separately. This post is about the part that decides whether any of that is trustworthy.

## What an agent harness actually is

The word arrived before most of us had a use for it. [Wikipedia defines an agent harness](https://en.wikipedia.org/wiki/Agent_harness) as the software infrastructure surrounding a large language model that enables it to operate as an AI agent. True, and close to useless, because it describes everything and therefore excludes nothing.

LangChain’s version is tighter. In March 2026, Vivek Trivedy [put the boundary where it belongs](https://www.langchain.com/blog/the-anatomy-of-an-agent-harness):

> A harness is every piece of code, configuration, and execution logic that isn’t the model itself. If you are not the model, you are the harness.

That gives you the shortest test available. Subtract the model. Whatever is left is the agent harness. Prompts, tool definitions, the loop that decides when to stop, the file the agent writes its notes into, the permission list, the retry logic. All of it.

It also explains why the term spread so fast. Mitchell Hashimoto [described the practice](https://mitchellh.com/writing/my-ai-adoption-journey) in February 2026 as harness engineering, and his rule is the one I keep coming back to: any time an agent makes a mistake, take the time to engineer a solution so that the agent never makes that mistake again. He was not selling anything when he wrote that. Neither was Anthropic when it published [its notes on harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) the previous November.

Three different things get called a harness, and conflating them is where most arguments about agent performance go wrong.

| Term | What it is | Typical example | What it produces |
| --- | --- | --- | --- |
| Agent harness | The system that runs a model as an agent on real work | A coding agent, or a scheduled SEO agent | Finished work and a trace of how it got there |
| Eval harness | A benchmark runner that scores models against fixed tasks | A standard LLM evaluation suite | A score |
| Execution harness | The plumbing that applies a model’s actions in a sandbox | A terminal benchmark environment | State changes and observations |

The middle column is the older usage, and the third is narrower still. An agent harness usually contains an execution harness, and an eval harness has to wrap an agent harness before it can score an agent. They answer different questions.

There is a second split worth knowing. Birgitta Böckeler [distinguished the inner harness from the outer](https://martinfowler.com/articles/harness-engineering.html): the inner one shipped by whoever built the model, the outer one assembled on top: instruction files, connectors, your own skills. Most teams can only change the outer one, and the outer one is where SEO work actually happens.

## The harness can outweigh the model

Two measurements from this year make the case better than any argument I could write.

The first is uncomfortable if you have been buying models. Researchers [held the weights fixed](https://arxiv.org/abs/2609.01437) and changed only the surrounding system: GPT-5 solved 35.2 percent of Terminal-Bench 2.1 inside one harness and 49.6 percent inside another. Same model. A fourteen-point swing in capability from the software wrapped around it.

The second is about money, which is the argument that gets budget approved. A [controlled swap](https://arxiv.org/abs/2607.06906) across 22 locked tasks and six foundation models, changing nothing but the orchestration layer:

| Measure | Conventional loop | Harness at the core | Change |
| --- | --- | --- | --- |
| Cost per task | 0.21 | 0.12 | down 41 percent |
| Median wall clock | 48 seconds | 27 seconds | down 44 percent |
| Tokens per task | 14.2 thousand | 8.8 thousand | down 38 percent |
| Task quality | 0.78 | 0.81 | flat to slightly better |

Both papers are vendor-adjacent, and each measures its own harness, so treat the exact percentages as theirs rather than as a law. The direction is what matters, and the two studies point the same way. Efficiency gains showed up on every model tested, between 33 and 61 percent. Quality gains were uneven and tracked how strong the model already was.

Which is the sentence worth taking into a budget meeting: a better harness makes any model cheaper, and it makes a strong model better. It does less for a weak one.

## Where agentic SEO runs into harness limits

I have written before about [the three things my own agent cannot see](https://sanjayshankar.me/seo-agent/). Read them again as harness problems, and they stop being mysterious.

**It cannot tell me which question triggered an answer.** Google’s generative AI performance report gives impressions, not the queries behind them. An agent that waits for query data to appear will produce a confident number with no explanation attached. The fix lives in the loop: log the prompt and the assistant beside every result, so the run itself becomes the record.

**It cannot count every AI visit.** Assistant traffic arrives looking like a direct hit or a stray referral. An agent reading one analytics channel will undercount what its own work produced. The fix is to measure the prompt set on a schedule instead of waiting for a session to show up in a dashboard.

**It cannot tell a person from a machine inside my own data.** In one export I looked at, 16 queries were unambiguously machine-issued, including a ten-clause exclusion chain nobody types. Thirty-seven percent of queries ran eight words or longer against a human baseline nearer twenty percent. An agent that aggregates that file produces a corrupted report and no warning. Someone has to classify before summing, and that someone is part of the harness. I wrote about [why scores move between runs](https://sanjayshankar.me/geo-visibility-score-drops/) for the same reason: the measurement layer is where these errors either get caught or get published.

None of those three are model failures. A larger model reads the same corrupted file and reports the same wrong total with more confidence.

> **Quick question: would a better model fix any of this?**
>
>
>
> Partly, and not the parts that hurt. A stronger model writes better recommendations and follows instructions more reliably. It cannot see a query that Google never published, count a visit the analytics tool filed under direct, or spot a machine issued query on its own. Those are structural gaps, and structure is what a harness is made of.

## The 6 parts of an SEO agent harness

The research literature lists six components. Translated into SEO work, they line up almost one-to-one, and this is the list I would hand to anyone evaluating a tool.

### 1. The loop and the schedule

What runs, how often, and what happens when a run is skipped. An agent that only acts when someone asks is a search box with opinions. An agent harness that runs weekly, unattended, on a fixed day, is the difference between a habit and a project. Check whether the schedule survives a failed run, and whether you are told about it.

### 2. The tools it can reach

Agent harness quality is bounded by what it can actually touch. Search performance data, backlink data, the crawl of your own site, AI answer samples. A tool that reads two of those four will produce recommendations that ignore the other two, and it will sound just as sure. Ask for the source list before you ask for the price.

### 3. The context it is handed

This is the part SEO teams most often get wrong. If the set of questions changes between runs, the numbers are not comparable, and the trend line is noise. A fixed question set, run against a fixed list of assistants, is what makes last week comparable to this one. Agents that let you add a question mid-flight without flagging it will silently break your own history.

### 4. The state it keeps

Memory is where an agent either gets useful or starts repeating itself. Prior runs, decisions already taken, pages already fixed. Without state, every run rediscovers the same twenty problems and hands you the same list. With it, the second run can tell you what changed. Ask what the agent remembers and how long it remembers it.

### 5. The lifecycle: budget, permissions, failure

An agent that can spend money and change live pages needs a boundary drawn around it before it starts, not after the invoice. Three questions settle most of it. What is the spend cap, and per what unit? What may it change without asking? What happens at half a run when the budget or the data source fails? The honest answer from most vendors is that failure handling was not built. I have written about [budget control for agents in more detail](https://sanjayshankar.me/ai-agent-budget-control/) because it is the part that gets skipped until it costs someone money.

### 6. The verification layer

What counts as done. This is the part I care about most and the part most tools skip, because it is invisible in a demo. A recommendation is not a result. A task that an agent marked complete is not a fix. The verification layer decides which claims get to leave the system, and if it does not exist, everything the agent says arrives with equal confidence, including the wrong things.

![Six-part agent harness diagram labelled loop, tools, context, memory, budget and verification](https://sanjayshankar.me/wp-content/uploads/2026/10/agent-harness-six-parts-1024x576.webp)

| Research component | SEO translation | The question to ask |
| --- | --- | --- |
| Execution | The loop and schedule | Does it run without me? |
| Tools | Data sources | What can it actually read? |
| Context | The fixed question set | Will this week compare to last week? |
| State | Memory across runs | Does it know what it already told me? |
| Lifecycle | Budget, permissions, failure | What stops it, and who is told? |
| Verification | What counts as done | Why should I believe the output? |

## How to score an agent harness before you buy

Eight checks. None need a trial account, and a vendor who can answer all eight quickly is telling you something.

1. Ask which data sources the agent reads, by name, and what happens when one of them fails mid run.
2. Ask whether the question set is fixed and versioned, and what the tool does when someone edits it.
3. Ask what the agent remembers between runs, and whether you can inspect that memory.
4. Ask for the spend cap and its unit. Credits, dollars, calls, per what.
5. Ask what the agent may change without approval, and where approvals are logged.
6. Ask how a completed task is verified, and request one example of a task the agent flagged as failed.
7. Ask how you would export everything it has collected if you left.
8. Run the same question twice and see whether the tool reports the difference or hides it.

The sixth question is the one that separates tools. Any system can report success. A system that can show you its failures has a verification layer, and a verification layer is the only thing standing between an agent’s output and your reputation.

![Agent Harness: 6 Critical Parts Worth Checking First](https://sanjayshankar.me/wp-content/uploads/2026/10/agent-harness-scorecard-1024x576.webp)

## Where the human stays

An agent harness decides a lot, and it should not decide everything. The split I have settled on is stable: agents run the measurement, the comparison and the first pass at prioritisation, and a person owns the judgement about what to change, because that judgement depends on context no tool holds.

That is also the shape we built into visibility.so. The parts we care about are the ones this post keeps returning to: what an agent runs and when, which sources it can reach, what it is measured against, what it carries between runs, what it is permitted to spend and change, and who signs off before a change goes live. If any of those six are missing, the agent is a demo with a login.

It is worth saying plainly what an operating system for a hybrid team means here. The org chart lives in the harness. Roles, budgets, approvals, memory, and the record of what was decided do not sit in a chat window. They sit in the layer around the model, which is why the layer is the product and the model is a setting.

## Frequently Asked Questions

**What is an agent harness in simple terms?**

It is everything around the language model that turns it into something that can do work: the loop, the tools, the instructions, the memory, the permissions and the checks. Take the model out and what remains is the agent harness.

**Is an agent harness the same as an agent framework?**

Not quite. A framework is a library you build with. An agent harness is the running system, including configuration, prompts, permissions and the parts a library does not cover. You can build a harness on a framework, and most teams do.

**Does the agent harness matter more than the model?**

For cost and reliability, often yes. Identical model weights have scored fourteen points apart on the same benchmark depending on the surrounding system. For raw reasoning quality the model still leads, which is why harness gains are largest on models that are already capable.

**Do I need to build my own agent harness for SEO?**

No, and most teams should not. Buying one means accepting someone else’s six decisions, so the useful work is asking what those decisions were. Build your own only when the workflow is specific enough that no vendor has the pieces you need.

**How does agent harness relate to agentic SEO?**

Agentic SEO describes the operating model, where tracking, prioritising and fixing run continuously rather than in monthly bursts. An agent harness is the machinery that makes that operating model possible. I covered [what agentic SEO actually covers](https://sanjayshankar.me/what-is-agentic-seo-complete-guide-2026/) separately if the distinction is still fuzzy.

**What does an SEO agent harness cost to run?**

It depends on how many questions you ask, how many assistants you ask them of, and how often. Most of the cost is queries against AI answer data, not model tokens, which surprises people. Start with one fixed question set and a weekly cadence, then expand only the part that produced a decision you acted on.

**How do I tell an agent harness from a wrapper?**

Ask what happens between runs, and whether the tool can show you a task it failed. A wrapper passes your prompt to a model and returns text. A harness remembers state, respects a budget, records what it did and can prove it did not do the rest.

## What I would do next

If you are evaluating an SEO agent this quarter, run the eight checks before you look at the price. It takes an afternoon, and it changes which tools are still on the list.

Then pick one fixed question set, run it weekly, and read the diff rather than the score. That single habit is what turned my own agent from a novelty into something I would not go back from. The seven jobs it does well are worth having. The six parts above are what make them worth trusting.

If you want the longer version of the operating model behind all of this, [how to deploy AI agents on a team](https://sanjayshankar.me/how-to-deploy-ai-agents/) is the practical layer, and [running several agents on the same job](https://sanjayshankar.me/multi-agent-engineering-team-paperclip/) covers what changes when one is not enough.