We developed a draft of our first in a series of short e-learning modules. We will soon be offered the chance to observe a few prospective learners try it out, one at a time, before we finish and publish the product for the larger group. What should we do? Should we be quiet and watch? Or are there specific questions we should ask?

Hi Kathy -- have you already done QA testing? (i.e., have you already tested everything to get the bugs out?)

We've done this ourselves, yes. We made sure everything works.
Still, our testers are a computer-savvy group (web designers, etc.) whereas our trainees are not.
Among other things, we want to know whether the product works as our users expect.
One of my questions is whether it is best to approach testing in an unstructured, informal way and ask them to think aloud, or whether it is best for us to use a structured set of questions -- and if so, what should we ask.
As always, thanks for any tips you that you share.

Hi Kathy -- Gotcha. I asked because I wanted to get a feel for where you are in the testing cycle. At this point, it sounds like you're talking about user acceptance testing. If you want useful results, I believe it's important to keep it structured and framed appropriately. This should not be about allowing users to change what they don't like - rather, the focus should be on whether or not the system flows as users expect it to flow and as it was designed to flow. Thus, the need for structure. Otherwise, you're going to end up with a list of subjective "I want"s rather than objective data that can be prioritized. Now, you can have informality within the specific structure - it really depends just how much detail you want to address (or avoid) with this cycle of testing. In any case, I would definitely do something that involves observing what the users are doing.

For future reference, the next time you do this, before you even develop the interface, do usability testing on paper prototypes. It takes time up front, but saves a lot of headaches later on.

Thank you for your very helpful comments!