Could anyone share how you pre-test your e-learning products before deploying them?
We would appreciate knowing how you conduct the testing, what you look for, and what you ask -- does anyone have a list of questions that they could share?
I am wondering how formal or informal the process should be -- to what degree should we ask specific questions, and to what degree should we simply observe our learners and collect their comments?
We are first-timers (for e-learning, but not as trainers) and very much appreciate any tips we receive.

There are two levels that I have used: Usability Testing and Functional Testing.

Usability Testing focuses on the effectiveness of the human interface such that it enhances the learning experience instead of hampers it. For example, the test would use a checklist to collect feedback on how clear it was to navigate through the course, or if it was clear how to do a prescribed action for an exercise. Some example checklist items: "Animation timing is appropriate for ease of understanding" and "Feedback given in interactions encourage reaching the learning goal as opposed to negatively focusing on failure to achieve the goal." This would basically help tune in the design of the user interface so that the focus remains on effective learning rather than trying to figure out how to use it. This is typically done once to design a common user interface or template that applies cognitive loading, causal principle, active engagement, assessment and other instructional strategies. Usability testing is repeated whenever the user interface design is altered.

The second is Functionality Testing which would also utilize a checklist on ensuring that each component of the e-learning functions properly, such as interaction exercises, audio, video, and animations. The checklist would include such items as "Animation replay button operates as expected" or "Mouse over windows appear when cursor is on and dissappears when cursor is moved off".

I would recommend a formal process that is documented to make sure nothing is lost or that the process can be modified when enhancements are introduced in the development process. In the long run a formal process can actually save time and expense.

Hi Kathy,

You have raised an excellent point. A common issue in eLearning testing is that most of the testing happens on the content/navigation. However, the focus should be on the quality and effectiveness of the eLearning courses developed.

My company, Kern Communications is doing a research to analyze how effective are these courses and do these actually meet the desired objectives. This research is being done using Kern’s Evaluation Methodology. Kern has formulated an evaluation mechanism to evaluate the effectiveness of eLearning courses. The evaluation mechanisms help discover whether the courses are effective, learnable, and meet their objectives.

Expert evaluation of eLearning courses is based on the proven evaluation technique of Heuristic Evaluation or Cognitive Walkthrough used by usability experts. In this evaluation technique, whether the course is effective or not is evaluated based on how effective the learning will be for the learners.

Training managers and eLearning companies can participate in this research by sending their courses (strictly under NDA) for evaluation. Kern shall share research findings of the expert evaluation with the respective companies. Kern shall also share the consolidated report with all the participants. The company names will be strictly kept confidential. To know more about Kern’s Evaluation Methodology, visit: http://elearning.kern-comm.com/

Kern shall be happy to do a sample expert review for your existing or on-going training program free of cost and send you a report. You can send me your request at rashmi@kern-comm.com.

Rashmi Varma
Director - Learning Solutions

Usability testing of eLearning programs is a little different from the usability testing of software products/applications. Whereas in software products/applications, we test the usability of the product with respect to the tasks performed by the user, in eLearning, the testing is a little more complex.

In eLearning, testing only the functionality of the eLearning program is not enough. We need to test the effectiveness of the program to see whether the program is learnable or not – whether the program meets its learning objectives or not. As Pat has mentioned, each program does go through functionality testing. But this is not usability testing and is done independent of learners. This is a QA round to check for bugs and functionality issues.

In eLearning, we typically have a formal learner testing round with the Alpha version of the course. This is how we go about testing our eLearning programs.

1) Profile the learners to be tested. The learners to be tested are the learners for whom the program is designed. This is the profile that was used to create the persona . The number of learners used for testing should be sufficient to indicate clear trends. A minimum of 7 is recommended.

2) We put the learner in an environment that is closest to a real life learning environment. There is a moderator who conducts the session and there can be either observers or the session can be recorded.

a. After introductions, the moderator sits a little behind the learner and observes him/her go through the course.
b. The moderator makes a declaration to the learner stating the duration of the session, and emphasizing clearly that the program is being tested and NOT the learner. The learner is helping the moderator find problems in the program that will help make the program more effective. If at any point, the learner wants to quit the session, he/she is free to do so.
c. Request the learner to think aloud while going through the program. For example, if the learner is looking for a button or some information, the learner can state this aloud.
d. The moderator must observe each and every action of the learner and note these down. At this stage, the moderator must focus on observing the learner without interrupting him/her until the learner closes a logical chunk of activity. Then the moderator can chip in gently to question a particular action of the learner during that activity. Ideally, do not ask for opinions from learners. Ask them questions related to the actions that they performed or did not perform.
e. Post observation, the moderator sits with the learner with a questionnaire. The questionnaire can pre-decided questions about the learning program, navigation, graphics/text elements in the program, and also have questions about the learners’ behaviour during the testing. These questionnaires do not have standard questions, you need to create them specifically for the program that you are testing.
f. Finally, the moderator thanks the learner and rewards him/her for participating in the testing activity.

3) The observations from these sessions are noted down by the moderators and compiled later by the instructional designers (instructional designers can also be the moderators). Finally, this data is analysed. The findings are categorized based on issues such as design issues, navigation issues, graphics issues, affordance issues etc. The team can also rank these issues as per priority and go ahead to fix them.

This concludes the learner testing round. However, the goal of this round is to test whether the learning experience meets the learners’ needs and requirements. However, this testing is not conclusive in terms of testing the effectiveness of the program. The effectiveness of the program can be tested only after the learners go through the course in real life and apply it at their workplace. This round of evaluation is done post launch of the program. Depending on the goal of the program, we decide the time of evaluation and the metrics for evaluation.

You can contact me offline for any specific queries that you may have about the process or the tools used during the process.

Cheers,
Geeta Bose

This message has been edited. Last edited by: Geeta,