Our contracting team and our customer are trying to come up with an effective quality control plan to ensure adherence to the ISD process. My initial attempt was to evaluate our process i.e. checksheets for each phase. However, our customer wants to use Level-1 surveys; I believe these are too subjective and can contradict the customer's own acceptence of quality course curriculum. I need something that evaluates each process, not necessarily the end product. Any suggestions would be appreciated.

Have you considered using the ADDIE model as an indicator of phases?

Analysis: you could determine whether the appropriate stakeholders were involved; whether the benchmark measurements are in place etc.

Design:
Development:
Implementation:
Evaluation:

If you had a defined process for assessing accurate completion of each 'gate' with both the designer and the audience stakeholder, you could avoid the 'but that's not what I wanted' hell that we sometimes fall into.

Mike,

I posed your question to Dan Raymond, an experienced pro with 35 years experience in the planning and execution of all aspects of the ISD process.
------------------------
Be sure to contact Dan directly to continue the conversation: Raymondppi@aol.com
------------------------
Forgive the length of this. Just thought your question deserves a thorough response:

THE QUICK ANSWER: You cannot just look at just the ISD process. The results (end product) are more important. If you do not know where you are going, any path will get you there. Level-1 surveys will not give you good objective data on which to make sound decisions. Level-1 surveys will always contain more opinions than facts. Some opinions are okay but I wouldn’t use them as the sole input to my decision-making process.

THE LONG ANSWER: The focus of any quality control plan for instruction needs to be on results; on whether or not the instruction provided the performers with the necessary skills and knowledge to accomplish the desired job-performance objectives. Developers should also not be subject to a quality control process that takes on a life of its own. With that said, you should examine the development effort and determine if it conforms to an acceptable ISD process. Quality control needs to be built into every phase of the project.

You first determine the ISD process requirements for a particular project and then how the inputs and outputs of each activity will be measured against established criteria. Your evaluation plan should address the degree to which the process was followed, the quality of the products of each phase, student performance results, and student and instructor input on the effectiveness and efficiency of the instruction. That should be followed by external evaluation involving looking at actual job performance, job performance results, and the return-on-investment. You can use surveys to gather some of this information but the surveys need to be very objective with little room for subjective input except in the case of opinions on the efficiency of the instructional development process and the conduct of the instruction.

For example, let’s say a project required job tasks to be defined. You would want to plan the job analysis so that the effort resulted in clearly stated job outcomes/accomplishments, job tasks, and performance criteria. There should be job analysis process criteria to make sure it is done correctly. You would want to plan to access relevant sources of information such as missions and objectives of the organization, accomplished performers, subject matter experts, supervisors, documents, direct observations of performers, etc. In addition to the sources of information, you should consider the sample sizes, locations, data gathering methods, and the questions to ask. You would want to make sure you identify job accomplishments and tasks associated with the people, products, procedures, and equipment associated with the job. You would want to make sure the accomplishments and tasks were directly linked to the business imperatives of the organization. They shouldn’t be doing them if they cannot link to the desired results of the organization. So there would be criteria for planning the job analysis.

You then want to make sure the results of the actual job analysis conform to acceptable ISD standards. The analysis should result in a clearly stated listing of job accomplishments/outcomes, tasks and the cues, conditions, and performance criteria for each task. You should also document the importance, difficulty, frequency, and the percentage of people performing each task. Your job analysis plan will need to make sure all this relevant information is gathered for each task.

An effective quality control plan to ensure adherence to the ISD process should include criteria for completing each phase of the process and the outputs of each phase. Surveys can be used to gather some of the data but not all of it. The plan should identify appropriate data gathering methods for each type of information desired.

Thanks Mitch, I am using ADDIE as the baseline for each of the areas to be evaluated. The problem is that the college has set no criteria for measurement within the phases ie. What percent of the audience needs to surveyed for an effective audience analysis? I have done much of what you and Dan have recommended. Just stating that we evaluate each phase and have a quality control plan doesn't do it for the customer. The problem is that the feedback needs to be monthly per our contract requirements. Which is very difficult to capture midway through a phase. We don't have the luxury or opportunity to wait for the final product. In fact, the final product will go through its normal rigor of level 1-3 evaluations. We're looking for a quality control plan that measures work performed on a monthly basis, regardless of where we are in the process. Thanks again for your feedback. This may be a tough one to crack.

Mike