The Chatbot Answers From Documents Nobody Checked It Could Read

First, check whose identity the retrieval runs as

Before anything else, establish one fact about your own build. When a user asks a question, does the retrieval step run under that user's identity, or under a service identity that reaches everything in the index? Permission trimming at query time is a property of particular products and particular configurations. A custom retrieval build can index and retrieve under one broad service credential, with the access check happening nowhere in the request path, and from the user's side the two look identical.

If the second describes your deployment, this is the wrong article for you today. Fix the retrieval identity first, because a service credential answering questions for anybody who can reach the interface is an authorization defect, and it belongs to your application team this week.

The rest of this assumes the first case: a deployment that scopes its search to the identity of the person asking. Somebody types a question, the system retrieves from the content that identity is permitted to reach, and it composes an answer out of what came back. That is the design, and it is the right one. The assistant returns what the requester was already entitled to see, and what the requester is entitled to see was settled inside your organization long before the assistant arrived.

So the answer that alarms somebody in month two is a correct answer. The compensation spreadsheet sitting in a site nobody has opened since a reorganization was reachable by that person the day before the assistant existed. They could have opened it themselves. No permission was granted and no boundary was crossed.

What moved is the effort. Reaching that spreadsheet used to require knowing it existed, knowing roughly where it sat, and caring enough to go looking. Now it takes one sentence, typed by somebody who was not looking for it at all. The assistant reads across everything reachable on every question and returns whatever passage answers best. The organization's intent about who should see what is written down nowhere in the estate, so the permission is the only instruction available.

What ten years of sharing decisions leaves behind

Any organization that has run a collaboration platform at scale is carrying a permission estate it never designed. It accumulated, one grant at a time, mostly during weeks when somebody needed a project to move by Friday.

Every grant was reasonable at the moment it happened. Each was a person clearing an obstacle in front of work that needed doing, and each was defensible on the day it was made. Ownership stops at the individual grant, and the total sits outside anybody's job description.

Each shape below has a mechanism holding it in place, which is why closing one by hand changes little.

• A site inherited through a reorganization, where the group granted access three org charts ago still exists in the directory and still resolves to people
• A folder opened to everyone in the tenant in 2019 so a project could move, never narrowed once the project shipped
• An export somebody pulled out of a system of record to build a board deck, parked in a personal drive that syncs and was shared once with a manager who has since changed departments
• Inherited permission on a parent container that quietly governs thousands of items nobody has enumerated
• Guest accounts retained long past the engagement that justified them, still resolving against content that has been reclassified since
• Sharing links that outlived their reason, where the record of who holds them lives in whichever mail archive they were forwarded through

Ten years of people cleared ten years of obstacles. The obstacles are gone and the grants are still resolving.

The cleanup is a data security posture program

Describe the work that has to happen before that assistant reaches everybody, and leave the assistant out of the description. Somebody has to find the stores, establish what is in them, put a name against each one, close the sharing that should have lapsed years ago, and set up a rhythm that stops it drifting back. That is a data security posture program, written down in AI project language.

The evidence here is narrow and worth naming as such: three data security posture engagements delivered by CWS on Microsoft Purview and Securiti, each running quick wins first, then a roadmap, then operationalize. Very little of the roadmap phase in any of them was engineering effort. It was decision effort, and decision effort moves at the speed of the organization's governance.

A retrieval rollout supplies the thing a posture program has the hardest time generating for itself, which is a date. Without one, the program competes for attention with everything else on the security calendar, and attention is the scarce thing. With a launch behind it, it gets a sponsor, a deadline, and a reason for a business unit to answer the phone. That is a genuine advantage, and the sequence decides whether the organization collects it.

The parts are the same parts either way.

• Discovery establishes which stores exist, what they hold, and which of them the retrieval index reaches
• Classification puts labels on those stores, defined by the consequence each label carries and applied automatically wherever the content allows it
• Ownership puts one individual's name against each class of store, somebody whose manager knows they hold it
• Remediation closes the oversharing that classification surfaces, which is where most of the calendar goes
• An exception register records what the business decided to accept, carrying an approver, a rationale and an expiry date
• A cadence brings newly created sites and workspaces into the register, because the estate keeps growing after the assistant goes live

The order that happens, and the order that works

The order that happens is understandable. A pilot goes to a small group over a narrow slice of content, a set of documentation somebody had already curated. It answers well. Everyone who sees it wants it, so the content scope widens, the user population widens, and the permission estate arrives all at once.

By then the permission work carries an executive sponsor and a launch date, which is the worst set of conditions for the parts of it that turn on decisions. Reconciling retention against a records schedule, getting legal to ratify a label taxonomy, telling a business unit that a site it has shared openly for six years is about to be locked down: each of those moves at the speed of the person who has to agree to it, whatever the launch date says.

The order that works puts a deliberate content scope in front of the pilot and treats that scope as a security decision with a named owner. Run the posture work against the scope, close or accept what it finds, then widen the scope and repeat. Widening becomes a small step with a gate in front of it, which is a manageable thing to do four times a year.

Treat that sequencing as a CWS position. It rests on which parts of the work turn on a decision somebody has to be entitled to make: naming an owner for a store, and ratifying a classification scheme. Those are the parts a launch date cannot compress.

The permission estate does not arrive during the pilot. It arrives on the day the pilot widens.

What to settle before the rollout date

One body of evidence carries two readings, which is the argument for building it once and building it properly. SAIL 2.0, the Secure AI Lifecycle Framework, which CWS contributed to, organizes the assistant by lifecycle phase and puts the data scope early in that sequence. ISO/IEC 42001 certification requires a stated scope for the management system, and naming the content a retrieval assistant reaches is how CWS writes that scope down. The register that answers the posture question answers both of those.

The list below is cheap to agree while the rollout is still a plan and expensive to agree once a date is public. Everything on it should exist in writing before general availability.

• A named content scope for the index, recorded with the stores deliberately excluded and the reason for each exclusion
• An owner for every store inside that scope, named as an individual
• The oversharing findings inside scope closed or exceptioned before go-live
• A gate in the change process, so that a new site enters the index by a decision somebody made, with permission inherited from a parent container checked first
• A set of adversarial questions written by legal, HR and finance, run against the assistant before launch and rerun every time the scope widens
• A named individual who owns the answer when somebody asks why the assistant could see a particular document

Settle the content scope while the rollout is still a plan. Once a launch date is public, these questions get answered by whoever happens to be in the room.