Another example of a question with dubious value:

A company for which I worked had a primary revenue stream we'll call A. It had begun to work on a new one we'll call B. After some time it asked employees whether they approved or disapproved of the work on B. The group with whom I worked said they disapproved: not enough time was spent on B, they believed. The group that got its money exclusively from A disapproved: too much time and money was being spent on B. It looked as though everyone disliked the work on B so it was mostly discoutinued. Had the question been "should we spend more or less time and money on B?" the results would have been more reflective of the true feelings of the group.

This applies to any case where you ask "do you approve or disapprove" and are not explicit about what that means. For instance, "do you approve or disapprove of the direction training is going?" Clearly if some think it is too much one way (too much spending on training for example), and others think it is too much the other (not enough spending on training), the results will be meaningless.

I have seen this in Level 1 evals where the questions were of the ilk, "Did this course have the correct level of detail?" and the answer block was a scale of 1-5 rather than +2 to -2 (for too detailed, not detailed enough).

Jim et al.

I think the problem here is poor definition of the term validity. Oddly enough I think the definition LL and you, perhaps are using, has to do with statistical validity. Statistical validity has to do with the ability to take the data, and then extrapolate that data to draw conclusions about the entire workforce (that's a bit simplistic as you probably know).

That's a very important concept, and IF one is concerned with that kind of validity, and using the data to do functions that require statistically based validity, one must have "it". I'm sure we agree.

However, while that's the research bent, the real world does not often afford us the luxury of perfecting our instruments, computing validity measures (which in themselves are subjective in terms of which measures are chosen), etc.

But, luckily, for many practical real life purposes, information we get on surveys, although not sufficiently robust or "clean" to use in strict quantitative terms, IS often useful in informing us what went right, what went wrong, etc., so we can fix it.

Here's an example. I use(d) level one evaluations for all my seminars. The instruments are those I created. They have NO research backing to establish reliability (which is required for quantitative validity to exist. We don't know test-retest. We don't do factor analysis on them. We can't really do much at all with the numbers that result except do frequency distributions.

However, when I look at them quantiatively, I use them to confirm or disconfirm my sense of how the sessions went. Even if I got only 50% of the evaluations back, I can still look at them and see if there's any pattern. Hey, three of the 10 returned said it was too slow, or too much group work, or whatever, too basic, too advanced.

That's incredibly valuable information for me to have. I know how to use it. I know its limits.

Now, you, or LL comes along and says garbage in, garbage out, and I would tell you that you don't have a clue about doing research in real life, or some such thing. Your criticism would be misplaced since you clearly do NOT understand how I use the data, and you clearly are NOT making the effort before launching a criticsm.

Note here, I'm talking on the basis of LL's posts, not yours.

You cannot criticize in general terms, and I'd suggest that the criticisms in this thread and the "questions asked" (they aren't questions, they are statements) are not sincere.

There's a basic tenet, and that is that you can't always get the data you want, but provided you understand the limitations of the data you have, and act accordingly, you can learn a great deal. THis applies both in the practitioner's world AND in the research world where "soft" data is often used to good effect.

Now, I'd suggest if people are interested in validity issues, they look at the different types of validity, the myriad of ways to conclude instrument validity, etc, BEFORE they start complaining, or asking questions to which they don't really want answers.

Now, as to asking questions, I'm with you in principle. The basic rule for any instrument item (and this is taught early on) is this:

1) Anticipate all possible responses to the question.
2) For each response, or each pattern of responses, can you provide a logical and fairly coherent interpretation>?
3) If not, throw out the item(s).

This is a very practical step for developing surveys and questionnaires in any setting, whether stat. validity is being evaluated or not.

The training resource center
http://thetrainingworld.com

quote:

However, when I look at them quantiatively, I use them to confirm or disconfirm my sense of how the sessions went. Even if I got only 50% of the evaluations back, I can still look at them and see if there's any pattern. Hey, three of the 10 returned said it was too slow, or too much group work, or whatever, too basic, too advanced.

Robert,

First, I was addressing both aspects, and I admit it may have been unclear.

Regarding the quote above. This is part of what I have found difficult. I generally teach classes arranged by a third party. That party creates the forms I use. I have no control over the questions, so issues like the ones I brought up before are real.

However. I have tried looking at trends. Consider the "too slow" and "too advanced" comments you note. Unfortunately, I do not teach a large enough group of even the same type of participants to get useful information for this. Even if I taught Java to two different groups of C programmers, my experience is that the exact same material may be too advanced for one and not advanced enough for another. A few years ago I looked at the "too advanced" issue for a course I wrote. I have about 15 instructors worldwide. I looked at the results from 1000+ evals and found that that as a whole, people thought the course was about the right amount of advanced-ness. However, some groups (public courses where there is no control over who attends) found it very advanced, others too basic. This did not correlate to instructor or country (although there is a slight tendency for UK audiences to think the course could be more advanced).

My point is that we teach groups of individuals and level 1 evals tell us how Jim or Mary felt, and maybe a few loose ideas about how one particular group felt, but I have very seldom, in my personal experience, found enough trend information to make significant changes in content or delivery based on them. Little things like "we need more coverage of foozle" help if there are a lot of such comments, but again, in the environment in which I teach, it is not consistent enough to help. I have found venue quality issues, issues with computers (mouse type preferences and so forth), but almost nothing related to the actual learning process.

I have, however, used level one evals to help novice instructors. On multiple occasions I have used them to help point out delivery issues. One potential instructor I worked with repeatedly said "It's obvious that..." enough that participants caught it.

YMMV, of course, but this has been my experience.

That was my point, LL.

quote:

Too slow or too advanced as compared to what? What is too slow for me might be just right for someone else. The construct validity of questions like that is highly suspect.

What you are missing here is that the simple statement of the PERCEPTION IS very important, and provides information a competent trainer or designer can use.

...once again you are confusing statistical validity with usefulness. Qualitative data, which is what we're talking about is useful. As for construct validity, I suspect you haven't worked much with these concepts in a real world setting.

I have no problem taking participant perceptions to items on a survey as having high face validity, and using them to make decisions.

This is not only defensible academically and research wise, but it's practical, since we can continue to improve our work without the expediture of large sums of money which we don't have.

The conflict resource center
http://conflict911.com