What should be decided before the session?
Start from the result you want, which is also where the Kirkpatrick Model, a training evaluation framework in use for decades, says evaluation should begin. Write two or three changes per role that a manager could observe. For a finance team: variance commentary drafted with the approved tool and checked against the ledger before it goes out. For client services: first drafts of routine replies produced with the tool and edited, never sent unread. Goals such as better AI skills cannot be measured. Observable behaviours can, and they also tell the facilitator what the exercises need to practise.
What can be measured on the day?
The day itself covers the first two Kirkpatrick levels, reaction and learning. A feedback form captures reaction, which is useful for improving the session and weak evidence of anything else. Learning is better measured from what participants produce. If each person completes an exercise on a fictional task, review the outputs against criteria agreed beforehand: did they give the tool enough context, did they catch the invented detail, would the draft meet your standard? The share of participants who catch a planted error tells you more than an average satisfaction score.
How do you check behaviour back at work?
Behaviour is the third level, and the Kirkpatrick Model warns against waiting too long: measurement cannot begin 90 days after learning without losing the chance to adjust. Plan short check-ins in the first weeks. The model also stresses support and accountability, such as manager check-ins, observations and job aids, because applying learning depends on the work environment as much as on the training. Ask managers to look at one or two real pieces of work per person that used the tool, and to note what was checked before it was used.
- A manager check-in in the first fortnight: which tasks have you tried, and what went wrong?
- A sample of real work produced with the tool, reviewed against the source.
- Usage data from the tool, where it exists.
- A short follow-up survey on confidence and use.
Which numbers can you trust, and which mislead?
Usage data is objective but narrow. Microsoft's Copilot usage report, for example, shows enabled users, active users, the active users rate and prompts submitted over periods from 7 to 180 days. A rise after training shows people are trying the tool. It does not show whether the outputs were good, and a high prompt count can mean people are struggling to get a usable answer. Self-reported time savings are estimates and should be labelled that way. Small samples mislead in both directions, so avoid reporting a percentage change from a handful of users.
How do you look at results without inventing savings?
Results, the fourth level, are the outcomes the organisation cares about, and the Kirkpatrick Model notes that most useful indicators are already tracked somewhere in the organisation. Pick one or two that the target tasks feed into, such as turnaround time for a routine report or the number of drafts a manager sends back. Time a few real instances of each task before and after training. Report the measured instances as measured and everything else as estimated. If the measures do not move, look at access, rules and workload before concluding the training failed.
What do you do with what you find?
Feed the findings into the next step instead of a report that sits in a folder. Tasks where people struggled become the next practice session. Tasks nobody uses the tool for may be poor candidates, which is a useful finding in itself. Questions about what is allowed go to the policy owner, and access problems go to IT. The National AI Centre suggests regular check-ins to track how confidence and understanding change over time, which also shows where people need more support.