Measure Whether a Training Video Improved Understanding
Use a scenario quiz, delayed check, and safe task demo to measure training video learning. Includes a worksheet, worked example, and edge cases.
To look for improved understanding, compare what learners can do before and after a video. A new scenario helps you check whether they can use what they learned. They should do more than repeat words from the video. Views and completion rates show delivery. They do not show learning. Start with a baseline. Then give a short scenario quiz after the video and a delayed check at a chosen interval. Add a safe task demonstration when the task allows it.
This guide gives you a practical method. You will find a worksheet, a worked example with clear math, edge cases, and three short FAQs. It builds on the delivery and creation topics covered in How to Measure AI Video Performance, AI Video for Corporate Trainers, and Turn SOPs & Handbooks into Training Videos.
Why views and completion are not enough
A view means the video played. Completion means the player reached the end. Neither proves that someone understood the content. A learner can leave a video running while doing other work. A learner can watch every second and still fail to apply the steps.
The What Works Clearinghouse practice guide on organizing instruction recommends using quizzes to promote learning. It also recommends using delayed judgments of learning to find content that needs more study. The guide suggests spacing learning over time and asking deep explanatory questions. This guide concerns education. The method below adapts its ideas for a workplace pilot; it does not establish the same effects for workplace videos. WWC Practice Guide: Organizing Instruction and Study to Improve Student Learning
So treat views and completion as delivery signals. Use them to track access to the video. Then measure understanding on its own.
A baseline and three follow-up checks
Before showing the video, test the target skill with a baseline scenario. Keep the result for each person who also completes later checks. Then use three follow-up checks, each with its own purpose.
- Scenario quiz. Give a short quiz right after the video. Use scenarios that require applying the content, not recalling exact words. This checks immediate understanding.
- Delayed check. Ask fresh questions that test the same skills at a similar level. Choose the delay to fit when learners need the skill. Ten working days is an example for a pilot, not a proven best interval.
- Safe task demonstration. Ask learners to perform the task or explain the decision in a safe setting. This checks transfer. Use this only when the task is safe to practice and easy to observe.
Keep records minimal. A learner code may still identify a person, so restrict access and set a deletion date. Explain the purpose and use results for feedback, not a public staff ranking.
Worksheet: plan your measurement
Fill this out before you launch the video.
| Field | Your answer |
|---|---|
| Training topic | |
| One key outcome learners should achieve | |
| Scenario question 1 | |
| Scenario question 2 | |
| Scenario question 3 | |
| Correct answer for each question | |
| Baseline timing and matched learner codes | |
| Quiz timing after video | |
| Delayed check timing | |
| Safe task to demonstrate (if any) | |
| Who will score the task | |
| Expected participants and missing-response plan | |
| How you will record results | |
| How you will protect learner privacy |
Choose questions that cover the decisions you need to assess. The three-question example below is a starting design, not a rule for every video. Pilot the questions for clarity and difficulty.
Worked example: support ticket triage video
This example is fictional. It shows the method with clear numbers.
A support team of 24 people watches a seven-minute video on triaging incoming support tickets. The video covers three priority levels, how to spot a billing issue, and when to escalate to a senior agent. The trainer wants to know whether the video improved understanding.
Step 1: Baseline check. Before the video, each learner answers three scenario questions. One question asks how to classify a ticket about a failed payment. Another asks which tickets need escalation. The third asks what to do when a ticket has two possible priority levels. The baseline shows what learners already knew.
Step 2: Scenario quiz. Right after the video, each learner answers three new scenario questions. The questions use different customer names and ticket details. They test the same three decisions.
Step 3: Delayed check. Ten working days later, three new scenarios test the same decisions at a similar level in a short online form. Learners answer without rewatching the video.
Step 4: Safe task demonstration. A team lead watches each learner triage five sample tickets in a sandbox system. The lead uses a simple three-point checklist: correct priority, correct escalation call, correct customer note.
Results. These are invented figures for the same 24 learners at each check. The team chose the example pass rules before the pilot. They are not validated standards for other tasks.
| Measure | Result |
|---|---|
| Learners who completed the baseline | 24 of 24 |
| Learners who passed the baseline (2 of 3 or better) | 10 of 24 |
| Learners who watched the video | 24 of 24 |
| Learners who completed the video | 23 of 24 |
| Learners who passed the immediate quiz (2 of 3 or better) | 19 of 24 |
| Learners who passed the delayed check (2 of 3 or better) | 17 of 24 |
| Learners who passed the task demonstration (3 of 3 checklist items) | 18 of 24 |
Reading the numbers. The baseline pass rate was 41.7 percent. The immediate quiz pass rate was 79.2 percent. That is a 37.5 percentage point improvement. The delayed check pass rate was 70.8 percent. The task pass rate was 75.0 percent.
Scores rose after the video, but that alone does not show why. The later drop could reflect recall, question difficulty, or other changes. The trainer might add a short refresher question one week after the video. The task results show that most learners can triage sample tickets. But six of 24 missed at least one checklist item. The trainer can review those specific steps with those learners.
What this example does not prove. The before-after improvement alone cannot prove the video caused the learning. Learners may have learned from a coworker. They may have seen the questions before. The sample is small. Treat the results as a useful signal, not a controlled experiment.
Edge cases
Small teams. With five learners, one person changes the percentage by 20 points. Report counts, not just percentages. Say "four of five passed" instead of "80 percent passed." Look for patterns in which questions people miss.
High-risk tasks. For safety, legal, or clinical content, a quiz is not enough. Require a supervised demonstration or a sign-off from a qualified reviewer. Keep the video as one input among several. Do not let a passing quiz replace required certification.
Learners with disabilities. Provide the quiz in an accessible format. Offer extra time where your policy allows. Make sure the task demonstration does not measure physical ability unless the job requires it. The W3C Web Accessibility Initiative explains that accessible media may need captions, a transcript, and description of key visual information. For this pilot, also check that learners can access and answer the questions. W3C WAI: Making Audio and Video Media Accessible
Question familiarity. If learners see the same questions twice, the delayed check may measure recognition, not retention. Use slightly reworded scenarios for the delayed check. Change names, numbers, or settings while keeping the same underlying decision.
Other support happening at the same time. If a supervisor also coaches the team that week, you cannot separate the video's effect from the coaching. Note any other support in your records. Then interpret results with that context.
Voluntary participation. If only some learners complete the quiz, the results may be biased. The people who finish may be the most motivated. Report the response rate. Treat low response rates as a signal to simplify the check or make time for it.
Missing data. If a learner misses the delayed check, do not count them as a fail. Report the number who completed each check. For example, if 22 of 24 complete the delayed check, report the pass rate out of 22. Then note the two missing responses separately. If you compare scores over time, also report results for the people who completed both checks. Different groups can have different starting skills.
Frequently asked questions
How many questions should a training video quiz have?
Use enough questions to cover the skill and its important exceptions. Ask learners to apply the content in new situations. Pilot a small set, then add or revise questions where the coverage is weak.
When should I run the delayed check?
Choose a delay that suits when the task will be used. You might pilot a check after ten working days, then review that choice. The educational guide does not establish an ideal delay for your workplace video.
Can I prove the video caused the improvement?
Not with a simple before-after check. A well-designed study with an appropriate comparison can support a stronger causal claim. A larger sample alone cannot remove bias. For most teams, a practical signal is enough. Record counts, note other support, and use the results to improve the video and the follow-up.
Start with one video and one outcome
Pick one training video. Define one key outcome. Write three scenario questions. Run a baseline check before the video. Run the quiz right after the video. Run a delayed check one to two weeks later. Add a safe task demonstration if the task allows it. Record counts, not just percentages. Use the results to revise the video, add a refresher, or coach the people who missed specific steps.
This method is simple enough to repeat. That repetition is what turns a one-time check into a useful habit.
Disclaimer
This article is for general information only. It offers a practical pilot method, not a validated test or certification standard. Privacy and employment rules vary by location and industry. Check your organization's policies before collecting learner data or using assessment results.
More from Kyndrify
Make your first video without filming.
Say what you need and the studio makes it: video, images, voices. Start free, no credit聽card.