TLDR: MSSPs running security awareness training and phishing simulations across many tenants can usually show a calendar. They often cannot show what was scheduled this month versus what actually ran. That planned-vs-actuals gap shows up in QBRs, audits, and renewals. This post defines the operator questions that should be easy, the four ledgers that make a program reconcilable, and a 90-day plan to close the gap.
Jordan runs security awareness for a regional MSSP. Dozens of client tenants sit behind one console. A vCISO quarterly review starts in twelve minutes. The client asks a reasonable question: How many phishing simulations actually went out this quarter, and did every in-scope user get this month’s training?
The dashboard looks fine. January has a simulation. February has another. Training is assigned on the first of each month. Then the exceptions start. One tenant paused the program during a merger. A second had the campaign built but never sent. A third went to a stale list that still included departed users. A fourth looks complete until someone notices due dates that drifted because of timezone handling.
The calendar still says the program ran. The operator cannot say what actually ran. That is the gap.
The Calendar Is Not the Program
A schedule is a promise. Actuals are evidence. Security awareness platforms are very good at the first and uneven at the second, especially when one team is operating many logically isolated tenants.
Fortinet’s 2024 Security Awareness and Training Global Research Report, based on interviews with 1,850 executive and management-level professionals across 29 countries, found that 75% of security awareness campaigns are planned in advance, and that those campaigns are delivered monthly (34%) or quarterly (47%). Eighty-one percent of organizations run awareness training monthly or quarterly.1
Planning is the industry default. The operational problem for MSSPs is not “do we have a calendar?” It is “can we prove the calendar executed, for every tenant, against the right population, this month?”
That proof matters because the human layer is still the path in. Verizon’s 2026 Data Breach Investigations Report finds the human element present in 62% of breaches, a slight increase from 60% the prior year.2 IBM’s Cost of a Data Breach Report 2026 puts the global average breach cost at $4.99 million, a 12% increase over the prior year, and finds that organizations using AI and automation extensively in security save $1.93 million compared with those using none.3
KnowBe4’s 2026 Phishing by Industry Benchmarking Report shows why consistency is not optional. Before training, the global average Phish-prone Percentage is 33.2%. Organizations that commit to a full year of continuous training and simulated phishing reduce that figure by 87%, to 4.2%. The largest gains do not arrive in the first 90 days; they arrive between month three and month twelve.4
If the program did not actually run in months four through nine, the twelve-month outcome is a story, not a result. MSSPs that cannot reconcile planned versus actuals cannot honestly claim the benefit of a continuous program.
For an MSSP or vCISO practice, a missed month is a service-quality defect that multiplies by tenant count. Each tenant has its own program state, user population, timezone, and hold requests. Reconstructing the month tenant by tenant does not scale, and it steals the advisory conversation: the QBR becomes an argument about whether February’s simulation went out instead of a discussion of what to change next quarter. Insurance questionnaires and compliance reviews are less patient. They expect an artifact. A screenshot of a schedule is not one.
The Operator Questions That Should Be Easy
A mature managed service should answer these questions on the first business day of the month, without a hero analyst rebuilding the month from screenshots.
| Question | What the calendar shows | What actuals must show |
|---|---|---|
| Which tenants were in-program this month? | Tenants with a schedule row | Tenants whose program was active, not paused, expired, or leftover |
| What training was scheduled? | Module names on a date | Assignments created, for which users, with which due dates |
| What training actually completed? | Sometimes a completion percentage | Completions, incompletes, and overdues against the in-scope population |
| What phishing was scheduled? | Campaigns on a calendar | Campaigns that were configured to send |
| What phishing actually sent? | Often the same calendar | Sends, holds, failures, and cancelled launches |
| Who was in scope? | Last known user list | Active users after status filters; joiners, leavers, and exclusions |
| What evidence exists? | A dashboard screenshot | An export or audit record that would survive a QBR and an auditor |
If any row on the right requires opening a tenant, clicking around, and copying numbers into a spreadsheet, the service is still operating at “calendar plus memory.” That model does not survive a busy month, a vacation, or a client who asks a precise question.
Four Ledgers of a Reconcilable Program
Reconciling planned versus actuals is bookkeeping. Four ledgers cover almost every operator failure mode.
1. The Schedule Ledger (Planned)
This is the intended work: which tenants should receive which training and which simulations in which window. A useful schedule ledger excludes inactive program rows. A paused or never-activated program is not a miss against the plan. Mixing those rows with live programs inflates “scheduled” activity and makes every variance look worse than it is.
2. The Activity Ledger (Actuals)
This is what ran: assignments created, reminders sent, simulations launched, policies presented, completions recorded. A campaign that was built and never sent is planned, not actual. Operators need an activity audit they can pull by month, across tenants, without reconstructing the story by hand.
3. The Population Ledger (Who Counted)
Actuals without a population denominator are theater. Completion rates are meaningless if the denominator includes deactivated users or excludes new hires. Status filters on the user list are how the denominator stays honest.
4. The Evidence Ledger (What You Can Show)
QBRs and audits do not accept “it is on the calendar.” They accept artifacts: activity exports, signed policy records, completion files that identify the company, and due dates that mean the same thing in every timezone.
| Ledger | Records | Failure if missing |
|---|---|---|
| Schedule | Intended campaigns and assignments for active programs | Operators defend a plan that includes dead rows |
| Activity | What actually launched, sent, completed, or failed | Calendars get presented as proof of delivery |
| Population | In-scope users by status at the time of the work | Completion and click rates are computed on the wrong people |
| Evidence | Exports, audit records, signed policies, branded reports | QBRs become slideware; reviews become a scramble |
Where Planned and Actuals Diverge
The gap is rarely one dramatic outage. It is a pile of small, reasonable exceptions that never get rolled up.
| Divergence | What it looks like in operations | How it shows up to the client |
|---|---|---|
| Inactive program still on the calendar | Paused or unused tenant still has monthly rows | “You said you would train us” when the program was on hold |
| Campaign built, not sent | Draft simulation sitting in a queue | Calendar green, mailbox empty |
| Stale population | Departed users still in the send list; new hires missing | Click rates and completion rates that nobody trusts |
| Timezone due dates | On time in one region, late in another | False overdue flags and noisy reminders |
| Partial send | One group got the test; another did not | “We ran phishing” is true for some people and false for others |
| Missing company identity on exports | Users’ activity file with no tenant name | Analysts cannot assemble a multi-tenant month without hand tagging |
| Assumed notifications | Operators believe users were warned | Alerts were never enabled; nothing fired |
None of these is exotic. The difference between a fragile service and a defensible one is whether the exceptions are visible in a monthly close, or only when a client asks.
What Good Actuals Look Like in a QBR
Clients buy security awareness for compliance, insurance, and risk reduction. They renew when the MSSP can explain what changed. A reconcilable monthly pack is short and specific:
- Plan for the month. Training modules and phishing simulations that were supposed to run, limited to active programs.
- Actuals for the month. What assigned, what sent, what completed, what failed, with counts.
- Population snapshot. Active versus inactive users; notable joiners and leavers.
- Variance. Planned items that did not run, and actual items that were not on the original plan, such as off-cycle coaching.
- Exceptions and owners. Who is closing each miss, and by when.
- Trend, not trivia. Click rate, report rate, and completion rate against the same denominator as last month.
That pack is how a vCISO walks into a board pre-read without hoping the dashboard still matches memory. Keep the narrative on the variance. A client can live with a month that slipped if the miss is named and owned. A client cannot live with a month that looked green and was not.
A 90-Day Plan to Close the Gap
Do not rebuild the entire stack in a quarter. Install a monthly close.
Days 1–30: Name the Ledgers and Freeze Definitions
Write down, in one page, what “scheduled,” “sent,” “completed,” and “in scope” mean. Pick a monthly window. Decide which tenant states count as in-program. If a program is paused, it leaves the schedule ledger until it is reactivated.
Then pull last month for a handful of tenants using dashboards, CSVs, or API exports. Typical findings: inactive rows mixed into schedules, no send confirmation distinct from “campaign exists,” user lists that cannot be filtered by status, and activity files that omit the company name.
| Days 1–30 output | Owner |
|---|---|
| Written definitions for scheduled, sent, completed, in-scope | Program lead |
| Tenant-state rules (active, paused, offboarded) | Operations |
| Baseline reconstruction of last month for a sample of tenants | Analyst |
| List of missing artifacts | Program lead |
Days 31–60: Instrument the Monthly Close
Turn the reconstruction into a repeatable pull. Prefer APIs and structured exports over screenshots. At minimum, the close should produce:
- A schedule extract for the month that excludes inactive program rows.
- An activity audit of what actually ran.
- A user extract with status filters so the population ledger is exact.
- A completion or users-activity file that identifies the company, so multi-tenant rolls do not require hand labeling.
- Policy evidence that can be retrieved on demand, including signed-policy records.
Assign a close date. Many MSSPs use the second business day. The close is the internal packet that makes every later QBR cheap. Add exception codes: not sent, partial send, paused tenant, population mismatch, timezone due-date issue.
| Days 31–60 output | Owner |
|---|---|
| Automated or scripted monthly extracts | Operations |
| Exception taxonomy | Program lead |
| First real monthly close | Analyst |
| Variance review meeting (30 minutes) | Program lead |
Days 61–90: Put Actuals in Front of Clients
Fold the close into client reporting. The QBR slide should show planned versus actuals before it shows click rates. Click rates on a month that did not run are a distraction. Use the same pack internally as a service-quality metric:
| Metric | Why it belongs in the close |
|---|---|
| Schedule attainment | Distinguishes a delivery miss from a paused program |
| Population integrity | Stops bad denominators from entering QBRs |
| Close completeness | Makes the pack a service standard, not a heroic extra |
| Time to explain a miss | Measures whether evidence is actually retrievable |
By day 90, a new operator should be able to answer “what was scheduled versus what ran last month?” without asking the person who “knows the calendar.” Do not treat the calendar as the system of record. Do not leave inactive programs in attainment math, compute rates on the wrong people, or assume notifications ran. If a warning channel is opt-in, it is not evidence unless enrollment is confirmed.
Closing the Evidence Gap in the Platform
The operator gap is a process problem first. It is also a platform problem. If the product cannot emit schedule, activity, population, and evidence as data, the MSSP will keep paying analysts to reconstruct the month.
Symbol Security’s 2026.08.9 release, in late August, added MSP-oriented APIs and exports aimed at this reconciliation: a users list with status filters for the population ledger; monthly schedules that exclude inactive program rows; a monthly activity audit of what actually ran; signed-policy download via API; company name on the Users’ Activity CSV so multi-tenant rollups do not require hand tagging; and a due-date timezone fix that removes false overdues.
Email Threat notifications, shipped in 2026.08.8, remain opt-in and off by default. Operators should not treat those alerts as something that went out for every tenant unless the tenant is actually enrolled.
Partners who want this wired into their own PSA, QBR pack, or client portal can start with the Symbol API. Providers who want the operating model without building it from scratch can become a partner through the MSSP program, or look at the vCISO Partner Program and Managed Program Services.
The Bottom Line
A calendar is a good start and a bad audit trail. MSSPs that run security awareness across many tenants will keep losing QBR time, and eventually renewals, if they cannot separate what was supposed to happen from what happened.
The fix is not a prettier dashboard. It is four ledgers, a monthly close, and a platform that emits planned versus actuals as data. Start with last month. Reconcile a sample of tenants. Then make the close boring.
References
- Fortinet, 2024 Security Awareness and Training Global Research Report. Survey of 1,850 leaders across 29 countries. Finds that 75% of campaigns are planned in advance and delivered monthly (34%) or quarterly (47%); 81% of organizations train monthly or quarterly.
- Verizon, 2026 Data Breach Investigations Report. Finds the human element present in 62% of breaches, up from 60% the prior year.
- IBM, Cost of a Data Breach Report 2026. Reports a global average breach cost of $4.99 million, a 12% increase over the prior year, and $1.93 million in average savings for organizations using AI and automation extensively in security.
- KnowBe4, 2026 Phishing by Industry Benchmarking Report. Reports a global baseline Phish-prone Percentage of 33.2%, reduced by 87% to 4.2% after one year of continuous training and simulated phishing. The largest gains arrive between month three and month twelve.