To measure whether AI training worked, look past attendance and exit surveys to two things: whether people changed what they do on the job, and whether that change produced a result worth more than the training cost. Practically, that means picking a few before-and-after signals, capturing the baseline before you train, and re-checking at 30, 60, and 90 days. Here's how to do it without a research department.

The default outcome is a measurement gap. Training gets approved, it happens, everyone shows up, the exit survey is positive, and three months later nobody can say whether it made a difference. When you can't tell whether it worked, you can't decide whether to do it again.

Why attendance and a happy survey aren't enough

Attendance measures logistics. A positive exit survey measures mood. Neither tells you anything changed.

This is built into how training evaluation has worked for over 60 years. The Kirkpatrick Model, developed by Donald Kirkpatrick in 1959 and still the most widely used training evaluation framework, defines four levels:

  1. Reaction: did they like it?
  2. Learning: did they gain knowledge or skill?
  3. Behavior: did they change what they do on the job?
  4. Results: did it affect business outcomes?

Each level is harder to measure than the one before, and more meaningful. Attendance doesn't even reach Level 1. The exit survey is Level 1, reaction, the weakest of the four. Most organizations stop there because it's easy, then wonder why they can't prove impact. The value lives at Levels 3 and 4.

There's a fifth level worth knowing. The Phillips ROI Methodology, from Jack Phillips, extends Kirkpatrick with Level 5: return on investment, translating results into monetary value against program cost. You don't need a formal ROI calculation to use the idea. It keeps the point in view: the training should produce a change worth more than it cost.

The problem you're really fighting

Here's why this matters for the money. Research on training transfer consistently finds that only a small fraction of what's taught gets applied on the job, commonly cited around 10 to 20%, decaying further in the months afterward without reinforcement.

Most training, by default, doesn't stick. So if you don't measure behavior over time, the likely outcome is that your AI training quietly joined the majority that faded, and you'd never know. Measuring is how you catch the fade before you pay for it twice.

Before/after signals you should track

You don't need Level 5 econometrics. You need honest before-and-after signals. (This is my practical approach: mapping the rigorous frameworks above onto things a normal team can observe.)

Pick a handful before training and capture a baseline:

  • A specific task's time or quality. If training targets drafting reports with AI, how long does that take now, and how good is the output now? Measure the same thing after.
  • Tool usage. Are people using the AI tools on real work, or did the licenses go quiet? Usage is a rough Level 3 signal.
  • Self-rated confidence on specific tasks, not "do you feel good about AI" but "can you confidently do this with it," before versus after.
  • A small skill check. A quick, real task attempted before and after. That's Level 2, and it's more honest than a survey.

The discipline is capturing the before. Almost nobody does, which is exactly why almost nobody can prove impact.

What "it stuck" looks like at 30, 60, and 90 days

Behavior change, Kirkpatrick Level 3, shows up over weeks, not on exit day. A rough timeline of what to watch for:

  • 30 days: people use the skill on real work without being prompted. Unprompted use is the earliest real signal, because it means the skill crossed from training exercise into how they work.
  • 60 days: the behavior is normal, not novel. New questions and new use cases appear, a sign people are extending the skill instead of repeating the demo.
  • 90 days: it's part of the workflow, and ideally you can point to a downstream effect: time saved, fewer errors, faster turnaround. That's Level 4 territory.

If usage has quietly returned to zero at 90 days, the training didn't stick, and now you know to invest differently next time instead of repeating it blindly.

Build measurement in from day one

The biggest reason training goes unmeasured is that nobody set it up beforehand. You can't capture a before after the fact. Trying to measure impact once training is over means reconstructing a baseline you never took, which is guesswork wearing a chart.

So measurement belongs in the design:

  1. Before design: decide what change you're after, in observable terms.
  2. Before delivery: capture the baseline, the times, quality, usage, and confidence you'll compare against.
  3. During design: build the training to move those specific signals, not a generic notion of "AI skills."
  4. After, on a schedule: re-check at 30, 60, and 90 days rather than one exit survey.

(That build-it-in-first sequence is my process, and it's the whole difference between training you can evaluate and training you can only hope about.)

The short version

Measuring whether AI training worked means getting past attendance and exit surveys, the weakest signals, to behavior and results, where the Kirkpatrick and Phillips frameworks put the value. Practically: pick a few before/after signals, capture the baseline before you train, and re-check at 30, 60, and 90 days. Do that and you'll know whether it worked.

Building measurement into a program from the start is part of how I approach this work, closer to learning operations than one-off training. If you want training you can evaluate, the free 15-minute call is the place to start.

Sources

Notes marked as my process or practical approach are exactly that, not external data. Transfer-of-training figures describe training generally and vary by study.