The Model Evaluation Covers the Model. Your Team Built the Application Around It.

The evaluation on the table answers a real question

AI red team engagements come in two shapes. Some are already scoped at the deployed system and cover injection carried in retrieved content, abuse of connected tools, and exfiltration paths. Others evaluate the model. Read the proposal for which one you are being offered, because everything below concerns the half that a model-scoped engagement leaves with you.

Take the model-scoped version. Adversarial inputs go in, a report comes back on how the model handled them, and you end up with a record you can show a customer or an auditor. That work has value and it is worth buying. It examines the one component that arrived from outside your organization and stays opaque from the inside.

The system your users touch is larger than that component. Between the person typing and the model sits a retrieval step, a set of connected systems, a credential that reaches them, and a path from the model's output back into something that changes state. Each of those was written or configured by your organization, and each settles a question the model is never consulted about.

So a system can handle every improper request put to it correctly and still return a document to somebody who should not have had it, because the retrieval step ran under an identity broader than the person asking. The model was never asked that question. The application answered it weeks earlier, in a configuration file.

The decisions on the application side of that line are few enough to list, and a person made each one.

• Which identity the retrieval runs as, and whether that identity is the person asking or something with wider reach
• Which systems the deployment holds a credential into, and what it can write as well as read
• What content reaches the model that no user typed: documents, tickets, mail, pages fetched from outside
• Whether an output can trigger an action directly, and which actions have a person standing in front of the effect
• Who finds out when an answer was wrong, and who can reverse what it caused

The retrieval step decides what comes back before the model sees any of it, and that decision belongs to your application.

Most of the second scope is questions you have asked before

Write the application-side questions down without the AI vocabulary and they turn familiar. Which identity does this process run as. What does its service credential reach. Can input from outside arrive at a code path that writes. Does the audit trail attribute an action to a named person. Your application security and identity teams hold working practice on all of those.

One thing here is new. The component making these calls takes its instruction from text, and text arrives from more places than the box a user types into. Somebody can spend a conversation persuading the system to work outside its intended purpose, and content the system retrieves can carry instruction of its own. The line between the data a system processes and the instruction it follows sits in a different place here, and a review that assumes the older line will pass a design it should have questioned.

That is the argument for scoping the application on its own terms. How the model behaves under pressure is one question. What its answer can reach and what it can move is a separate one, and the second is where a bound on the damage comes from.

• Whether the retrieval path enforces the asking user's permissions at the moment of the query
• How wide the deployment's credential is, measured against the narrowest one the feature would run on
• Which downstream actions can fire without a person confirming, and what each costs to undo
• What the system does when asked for something outside its purpose, and whether that response was designed or inherited
• Which parts of an input the system treats as content, and which parts it can end up following

Where the line between the two scopes goes

Buy the two engagements separately and the gap between them becomes a third deliverable that nobody scheduled. Drawing the line is a short exercise and it belongs at the front, before either scope is agreed. It comes out as a list of questions with a scope name written against each one.

Some of them clearly belong to the model. How it responds to inputs designed to push it past its instructions. Whether its behavior moved between versions. Whether it produces content you would not want attributed to your organization. An evaluation is the instrument built for those.

Others clearly belong to the application. What it retrieves and on whose authority. What it holds a credential into. What it can trigger. What it records. Those get answered by reading configuration and code inside your own estate, and by testing the deployed system against them. A model-scoped evaluation leaves them where it found them.

The third group goes missing, because it has no natural owner in either engagement. These questions turn on both halves at once, and the first to ask is which protections your application has come to depend on the model to provide. Where an action is prevented only by the model's own judgment about what it should do, that protection sits outside your change control and outside anything your test suite exercises. Naming the protections your application has come to depend on the model to supply is worth an hour of the scoping meeting, and no framework will prompt you for it. That claim is CWS's, offered as an argument you can push back on.

A protection that exists only because the model chooses to apply it is a protection your change control has never seen.

Who is equipped to scope the application half

The application scope gets read by people who work in authorization code, permission models and service credentials, and it gets structured by the AI governance frameworks so what comes out files somewhere a program can act on. Those are two different skill sets, and the boundary question needs somebody holding both at once.

CWS comes at this from both sides. The application half draws on one estate discovery across roughly 6,300 repositories, where most of the effort went into establishing what existed and who owned it before any finding meant anything. The governance half runs on SAIL 2.0, the Secure AI Lifecycle Framework CWS contributed to, which orders the problem by lifecycle phase so a finding lands on the team working in that phase, with the NIST AI Risk Management Framework and ISO/IEC 42001 carrying the parts that have to be shown to somebody outside.

What holding both produces is the boundary itself: a written statement of which questions each scope answers, and a name against the ones that fall between. A retrieval index inheriting a decade of sharing decisions and an agent credential scoped to a deadline are both entries in that statement.

The statement itself stays short. It has to carry this much.

• Which stores the deployment queries or indexes, and the identity each query runs as
• Every system the deployment can write into, and what a write can do once it lands there
• Every route by which text reaches the model, including content the system fetches that no user typed
• Where a person stands between an output and its effect, each point recorded as a decision somebody made
• What the log holds, and whether it answers why the system did a particular thing on a particular day
• What sits outside the scope deliberately, with the reason and the name of whoever accepted it

Settle the boundary before you agree either scope

The order that works puts the application scope statement first, because it also sharpens the evaluation. An evaluation pointed at what the system is used for in your estate tells you more than one pointed at general capability, and the statement is where that purpose gets written down.

Everything below is cheap to agree while both engagements are still proposals. It gets expensive once one report has landed, a steering group has read it, and the other half has no scope yet.

• The list of questions, with the scope that answers each one written beside it
• A named individual who owns the questions sitting between the two scopes
• What the application is permitted to do, written down while it is still a decision
• Which protections the application provides on its own, independent of how the model behaves on the day
• The trigger for running the application work again: a change to the retrieval scope, the credential, or the systems it reaches
• Where the two results get read together, by whom, and what happens to a finding that belongs to neither

What the system is permitted to do should be written down while it is still a decision, before it becomes something a team reconstructs from the code.