N/A

This message has been edited. Last edited by: LoveLearning,

LL,

Important question. I hope others join the conversation.

My perspective is tempered by experience in the area of 360 surveys. A common project might include doing pre-and post training assessments of observable behaviors. The end game is to improve individual performance in the skills, competencies and practices a person needs to accomplish performance objectives.

Having said that, the validity is established locally and is not a particularly complicated process. In essence you are looking directly at the behaviors, not inferred dimensions (which require a different approach more closely associated with assessments of things that cannot be directly observed.)

For pre-post training assessments we focus on two basic validity questions:

• Do the items describe important aspects of the work of the people receiving feedback?

• Is the feedback desired by the organization?

This validity is established when stakeholders within the organization have reviewed and revised the assessment and ultimately declared it valid.

To your more Zen-like question about "does it really matter" I think the model does not matter as much as the conversation about measurment itself. When organizations take the time to engage in a constructive dialogue about measurement, performance and development everybody wins.

quote:

I'm curious about the realities of other organizations -- is anyone out there confident in the validity of their surveys and results analysis?

No, but it's good to keep in mind that psychometric validity and usefulness are two separate things. A survey can be useful without being valid quantitatively, which is why people without the skills you mention use them.

The trick of course is for those using these things to know what they can conclude and what they can't.

One of the most interesting discussions on the board- thanks for this.

Unfortunately, my reply is based in two areas I studied, marketing and statistics:

"Perception is the reality" - unfortunately, many times surveys are designed to collect things that the executives/sponsors want, but not necessarily reflect what truly needs to be measured, is important to measure (a symptom of "tell me what I want to hear") Surveys are often designed to serve these needs.

"Liars can figure, figures can lie"- alternately, any information can truly be parsed to tell about whatever story they "need to tell" or the right metrics can be showcased whereas less flattering metrics can be whispered.

I think there are many things that CAN be measured with validity, but in practice, I think the validity of the true measure and psychometric and quantitative statistics to validate measures quickly take a back seat to convenience and "gathering the story we need".

Don't mean to be pessimistic, but I've seen the pattern far too many times (wish I had a firm metric for you, but I'll confess this isn't a "true measure", but a "historically formulated opinion", nothing more).

LL,

I suspect that some of the confusion around the topic flows from the difference between evaluating an event and evaluating its impact. Both have a role to play in the learning process and both need thoughtfully constructed tools.