The ebb and flow of this discussion reminds me just how important the client is in the equation. Client meaning the trainer, the developer, the learner, the sponsor, et al.

Robert's last post about perceptions gets at a piece of this. The surveys I administer for clients generate lots of quantitative data. It does not become qualitative until the client reacts,understands, and makes sense of it.

We spend more time talking about local validation of items.....recognizing that improving validity grows over time and through trial and error.

I love this quote:
"You never reach a place called the land of valid measurements. Instead, you gradually strengthen the evidence for validity, but there is no threshold that you cross where you can say, 'We can now conclude that the measure is valid.' Similarly, there is no region we can point to where we can say with confidence 'We have not yet reached the point where we can say that the measure is valid.' (Steve Simon, Ph.D ,Research Biostatistician)

Have enjoyed watching this conversation unfold.

quote:

Robert's last post about perceptions gets at a piece of this. The surveys I administer for clients generate lots of quantitative data. It does not become qualitative until the client reacts,understands, and makes sense of it.

I would use different terms, I think. I'm not sure if they would mean what you mean. I'd say that DATA does not become INFORMATION until the client reacts, understands and makes sense of it (if we are talking about consulting).

If not then I would remove the client portion, and say, "Data does not become INFORMATION until the data is interpreted, etc.

Is that the same as what you are saying, Entropy?

To extent this a bit, INFORMATION can come in many ways and according to many criteria to evaluate it. Numerical information, for example, statistically anaysed is but ONE method, and as with all methods of "knowing" have pros and cons. Linguistic comments, and or perceptions expressed via language provides another window. Obviously you choose the tools you CAN use, and accept that if you use tools properly, you can gain from them, even if flawed (since all are flawed or problematic).


In any event, I'm writing to thank you for your great quote and contribution.

Small Business Resource Center
http://smallbusiness411.org

quote:

Glad someone's gained something out of it. Wish I could say same.

What would have helped you "gain something out of it"?

quote:

This applies to any case where you ask "do you approve or disapprove" and are not explicit about what that means. For instance, "do you approve or disapprove of the direction training is going?" Clearly if some think it is too much one way (too much spending on training for example), and others think it is too much the other (not enough spending on training), the results will be meaningless.

Just catching up here. Ok. Here's the issue. I understand the desire for clarity, but you can NEVER provide enough information to respondents in a survey to GUARANTEE they are anwering according to the meaning you INTENDED.

That is why extensive statistical analysis is done, actually. For example, factor analysis tells us something about how people are responding to a set of questions.

So, what I'm saying is you CANNOT achieve clarity, and the cost of ensuring you are as good as you can be is monumental. Virtually no company will undertake the validation of an in-house instrument.

quote:

I have seen this in Level 1 evals where the questions were of the ilk, "Did this course have the correct level of detail?" and the answer block was a scale of 1-5 rather than +2 to -2 (for too detailed, not detailed enough).

Even though these things are flawed, ALL measures of the type you are talking about are flawed. For each level of "flaw removal" you either introduce other flaws overall, OR you have to spend huge amounts of money and resources.

Now, if we want to talk establishing causal relationships in the workplace (such as ROI for training), the task becomes absolutely and ridiculously impossible to do "properly".

Thanks for the comments Robert.

Actually, I do use the term "information" interchangeably when making the distinction. For me that came from time in military when we would talk about how "intelligence" doesn't become information (actionable) until people make sense out of it.