Six Months After Cutover, Nobody Can Say What Improved

The question arrives from outside the project

The question comes from outside the build, which is why it is hard to answer. A finance review, a new CISO reading two years of spend, a board member who approved the business case and wants to see the other end of it. The wording varies and the request underneath is the same: show me what changed.

The people in the room can describe the platform in detail. Sources onboarded through collectors and a Broker VM, cases handled, coverage across the Analytics BIOCs that ship with the platform and the IOC, BIOC and correlation rules the team wrote for itself, the shape of the current week. Every one of those describes the state now, and the question asked for a difference.

The reason is ordinary. The migration was scoped as a technical delivery, correctly, because that is what it was. The plan covered data, content, cutover and acceptance, and it was worked to. Recording what the old arrangement cost and produced sat outside every workstream on that plan, and that is the whole explanation.

Cortex XSIAM has been running and observed the whole time, and across three deployments it did what it was bought to do. The record of the state it replaced lived in a system your organization decommissioned, and that record ended where the decommission ticket closed. What can still be said about the difference is bounded by what other systems happened to keep.

Nobody's deliverable was to write down what it was like before.

What to record while the old system is still up

If you are reading this before a cutover, the whole problem is a short piece of work on the plan with a name and a date against it. The capture happens while the old system is licensed, running and staffed by people who understand its output. Give it to whoever will be in the room when the value question gets asked, which points at the security leader, because the build team will have moved on by then.

What to take is narrower than it sounds. Export records, since a record can be recounted later under whatever definition the question turns out to need. Then store the export where retention outlasts the argument it will settle.

• Case and alert records as data, covering a run of weeks long enough that one bad night does not set the tone.
• The log source list as it really stood, with what each source was relied on for, and which ones had stopped arriving without anyone noticing.
• Ticket history for security work, with created and resolved timestamps and a category on each. That system belongs to another team, which is why it survives a security migration.
• Out of hours paging: how many, at what hours, who acknowledged them, and how often a page turned out to be nothing.
• Where the team's hours went. Content maintenance, chasing broken data sources, storage and license administration, the standing meeting about the platform.
• The questions the team was unable to answer, written as questions, dated, with a name against each. That list carries no numbers and it outlives everything else on this page.

The window closes on a date somebody else controls

Capture is easy while the old system is up. Then a set of dates arrives, each chosen by someone outside security, and each one takes a piece of the evidence with it.

So the capture belongs early on the plan, ahead of the change freeze, while somebody's job still includes answering questions about the old platform. Once the decommission ticket closes, what remains available is the reconstruction work in the rest of this article.

• The license lapses, because the business case counted on it lapsing.
• Infrastructure gets reclaimed by the team that has been waiting for it.
• Retention on the old archive runs out on a schedule set years ago by someone who has left.
• The contractor who wrote the monthly reports finishes at the end of the transition, and the report definitions leave with them.

A license lapses on a date the business case chose, and the evidence goes with it.

Reconstructing a baseline after the moment has passed

Standing in the after with a customer who has no before is where CWS has spent real time: three Cortex XSIAM deployments taken through post-deployment work, and in each one the value question landed after go-live because capturing the before had sat outside every workstream on a plan that was otherwise complete. A partial answer could be assembled in each, because other systems in the organization had been recording what the security team did all along.

That evidence sits in systems owned by other teams, which is why it survived the decommission. Each of those systems kept its records for a reason of its own, and read together they make a baseline.

• The ticketing system. Security cases carry creation and resolution timestamps, categories and assignees, and service management retention is set to satisfy audit, which makes it long. It is the richest of these, and it sits outside the scope of a security migration.
• The paging tool. Out of hours volume, acknowledgment times and who carried the phone, held by whoever runs on-call, on a contract that renews on its own cycle.
• Chat archives. Incident channels and shift handovers are dated, searchable, and written by the people doing the work, which makes them a usable record of what a bad week involved.
• The migration's own business case. Somebody wrote down what the old arrangement cost, what it could not do, and what the replacement was expected to change. That is a dated statement of the before, made by the organization, before anyone had a stake in the answer.
• The cold archive kept for a retention obligation. Where compliance took an export of the old platform's data before the decommission, that export outlives the platform, and it is worth asking for by name.
• The people. Analysts who worked both systems can describe the difference precisely, for as long as they stay. Take it down while they are here, dated, with a name against each statement.

Read the ticket queue, the paging records and the business case as three partial baselines, and say in writing that is what you are doing.

What a reconstructed baseline is allowed to say

The reason a reconstructed baseline is partial matters, and the grouping model is the clearest case of it. The old platform counted alerts one way. Cortex XSIAM records each alert as an issue, groups the related ones into a case by tracing the causality behind them, and gives the case a score describing how urgent it is, so one number on the new side can stand for a dozen on the old. A comparison built on that pairing gets taken apart by the first person who checks it.

So the comparison runs on what both sides recorded identically, and that evidence lives outside either platform. Tickets were raised into the same service management system before and after, pages went through the same on-call tool, and staff hours were charged the same way. Those cross a cutover intact.

Everything else gets reported as description with its evidence attached: what the team could not see then, what it can see now, and the dated artifact behind each claim. An account that states its gap in the first line survives scrutiny. A clean percentage assembled from two incompatible sources invites one question and fails it.

• State the gap up front. A reader who discovers it later stops trusting the parts that were sound.
• Compare only on the systems that spanned the cutover: tickets, pages, on-call load, staff time.
• Put a date and a source against every figure, including the ones lifted from the business case.
• Keep the qualitative claims as questions answered, with a named person willing to attest to the then and the now.
• Write down what you are choosing not to claim and why. That sentence is what makes the rest credible.

The account you give, and the one you will be able to give next time

The answer in that meeting is shorter than the work behind it. Here is what changed, here is the evidence, here is the part we cannot evidence and why, and here is the capture now running so the question has a full answer next time.

Say the reason plainly, because an unsaid reason gets filled in badly by whoever is listening. The migration plan covered the platform and covered it well. A record of the previous state sat outside the deliverable list. That is a scope gap in the program, ordinary enough that CWS has now watched it three times, and it describes how the work was divided between a project and an organization.

Pre-cutover baseline capture, and reconstructing a partial baseline afterward from ticketing, paging and business case records, are CWS recommendations. Both are argued here, and the argument is all the authority they carry.

The baseline that got away from the migration is still available for everything that comes after it. A capture taken this month becomes the before for the tuning, the detection content and the automation work still ahead, and each of those arrives with a value question of its own behind it.

Whatever the platform changed, the account of it is bookkeeping: one owner, one date, one export taken before a license expires. It is the cheapest line on a migration plan and the most expensive one to add late.