Someone said..."How the world looks depends on where you are sitting."

I truly do find that most folks want to get it right....but are operating in such a time-poor environment that googlizing (literally and figuratively)is far too often seen as the only viable option.

In the end, the profession becomes somewhat marginalized as the depth and distribution of knowledge across the industry is diminished.

quote:

Originally posted by LoveLearning:
Perhaps I'm asking the wrong questions/addressing the wrong issue... How about this...

That is often one of my biggest complaints about surveys; they ask the wrong questions...

quote:

To what extent do people care about the validity of the results they get from surveys, smiley sheets, etc.?

I am not a psychometrician (I am not even sure I can spell it correctly!).

I am often concerned about the validity of the results. That said, I often have to deal with the fact that the survey may be invalid due to sample size or some other issue byond my control.

Perhaps my biggest gripe is when someone takes data from a survey/evaluation/whatever and abuses it. For instance, a class has 12 participants. A level one form asks a question such as "how good was the instructor" on a scale of 0 to 4. The orngaization compares that event's results with others where the population varies in size from 8 to 30 and makes quality decisions for individual events based on hundredths of a point. Or they say things like "all events must score above average". Really.

Another gripe is when a survey is when users are not given a choice that that applies. For instance:
How long have you been a member of ASTD?
1 month to one year
one year to five years
over five years.

If I am not a member, thee is no correct answer. Now, of course, that is an extreme example, but I too a survey recently that had:
What is the reason you use our product:
Military Compliance
Size of our company
Price
Reputation of our company

Well, I used it because a collegue used it and I wanted to be compatible. That was not a chice so the survey was not in any way valid.

In cases such as the above, it matters to me. The first example matters because it directly impacts my income.

[quote]Perhaps I'm asking the wrong questions/addressing the wrong issue... How about this...[quote]

[quote]That is often one of my biggest complaints about surveys; they ask the wrong questions...[quote]

Okay - here we have proof that it does indeed make sense to occasionally say someone is "asking the wrong question," especially when you follow that up by an explanation as to why it is the wrong question. Thanks for comfirming that for the moderator.

quote:

On a related venting note -- what on earth is the point of keeping and tracking level 1 data in online databases if its external validity is highly questionable in the first place?

One of the exercises commonly used in a number of communication, negotiation, thinking learning situations is to ask the person to take the position of "the other side", and respond.

I think you should try that. Perhaps if the answers came from you, you'd give them more credence.

So, what good reasons might exist to track level 1 data...as you asked in your question?

Small Business Resource Center
http://smallbusiness411.org

PS. If you can't think of any good reasons, my bet is you don't want to find any.

PPS. I'm a former statistician and educational researcher, and formal peer reviewed journal editor (in ed. research).

Robert,
I, too, have wondered about this. You say:

quote:

If you can't think of any good reasons, my bet is you don't want to find any.

Well, I want some, but the reasons I have found are not what I'd consider good. Maybe you think one or more of them are good. If so, I'd appreciate your opinion.

A. The people doing the tracking do not understand statistics, psychometrics or training. This seems likely, but maybe I am just being cynical. Even when those who do know those topics explain it to the trackers, they do not change.

B. The people doing the tracking have MBAs or know about project management, but they don't understand the way metrics work. This seems highly likely. They have learned (or at least been told) the value of measurement and level one data is easy to get and manipulate. The value of the information component (or what the information component actaually is) of the data is less important than tracking something. This seems pretty likely to me.

C. A combination of A & B.

D. The people taking the data are clueless and are keeping the data because they read in magazines that others do that. I strongly hope this is not the case. Sometines training pros want to think it is the case, but don't buy it is the only reason.

How did I do?