Cognitive Interviewing Process in Instrument Design and Evaluation

This report delves into the cognitive interviewing process as a crucial element in instrument design and evaluation. It begins with an introduction to cognitive interviews, highlighting their role in refining questionnaires and assessments. A literature review explores the evolution of cognitive interviewing techniques, emphasizing the importance of understanding how respondents interpret questions. The methodology section details techniques like thinking aloud and verbal probing, both concurrent and retrospective, and their advantages and disadvantages. The report then presents a case study of the National Survey of Student Engagement (NSSE), demonstrating how cognitive interviews were used to improve the survey instrument. It outlines the preparation, pre-testing process, and documentation involved, including the use of focus groups and phone interviews. A critical analysis of the findings and the changes made to the instrument are also included, summarizing the process of refining the survey instrument. The report concludes by emphasizing the significance of cognitive interviewing in ensuring the accuracy and effectiveness of survey instruments.

COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
Name of the Student
Name of the University
Author’s Note

1COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
Table of Contents
Introduction................................................................................................................................2
Literature review....................................................................................................................3
Methodology of cognitive interviewing process........................................................................4
Qualitative instrument for evaluation.........................................................................................5
Preparation of the instrument.....................................................................................................6
Documenting the pre-testing process.....................................................................................7
Critical Analysis.....................................................................................................................8
Conclusion..................................................................................................................................9
References................................................................................................................................11

2COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
Introduction
Cognitive interview is a protocol for interviewing and probing individuals in order to
make them reveal precise details of an event even if they are unwilling to answer or having
difficulties in remembering (Willis, 2005). The primary motive of cognitive interview is
making the individual focus on the particular event and makes them aware of the transpired.
The cognitive interview helps to reduce misinterpretation along with the uncertainty in the
answers by following an established methodology while questioning the individual in an
interview.
The cognitive interview’s reliability lies in the enhancement of the memory retrieval
process without generating confabulations or inaccurate accounts. Due to the reported success
of incorporating cognitive interviewing in questionnaires and assessment, amplified usage of
it is seen during police investigations, training programs and surveys. As assessment of
educational practices is increasing with time, it is essential to make sure that the instruments
used for the evaluation must be able to accurately compute and evaluate the educational
practices along with the experiences (Collins, 2003). In particular, it is vital that the
questionnaires must quantify what they are intended to in addition to the fact that the
respondents must be able to deduce the items accurately.
The National Survey of Student Engagement or the NSSE refers to the survey
analysis instrument regularly used by the colleges and the universities to evaluate the effort
and the time taken by the students in order to dedicate to the educational activities (Fosnacht
& Gonyea, 2018). The study provides evaluation of such an instrument for data collection in
order to measure the effectiveness cognitive interviewing process.

3COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
Literature review
Since time immemorial, the researchers have faced problems regarding the wordings
of the questions in a survey. However, in the recent few decades, protocols to assess
effectiveness of questions in a survey along with researching on the evaluation of the
instrumentation has been taken up in a systematic manner with assistance of the theoretical
frameworks along with the cognitive and social psychological methods (Collins, 2003;
Pressser et al., 2004). Cognitive interview process has been established as an imperative
protocol to ensure accuracy of the survey instruments along with identifying and analyzing
the sources of errors in the responses in the survey questionnaires (Willis, 2005).
The cognitive interview based questionnaire is supposed to identify in addition with
analyzing the sources of the error in the response present in the survey instrument’s
questionnaires by highlighting and focusing at the cognitive process of the respondents that is
used in order to respond to the questions in the questionnaire. Particularly, the rationale of the
cognitive method is to understand if the subject understands the question, consistently
through the subjects along with the manner that is intended by the experts (Collins, 2003).
Study of both apparent along with the hidden messages in the response of the subjects are
studied (Willis, 2009; 2005).
Cognitive interview focuses on the questionnaire of the survey and not on the person
responding to the question to promote the soundness of the questionnaires in the survey.
Theoretically, the four actions involved in answering a question in the survey involves
enacting while answering the question; capability in comprehending the question accurately,
retrieving relevant information to the question, making a judgment followed by accurately
responding and answering the question (Daugherty et al., 2001; Collins, 2003; Jobe, 2003;
Willis, 2005).

4COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
Methodology of cognitive interviewing process
Cognitive interviewing practice includes techniques of thinking aloud along with
verbal probing (Daugherty et al., 2001). The thinking aloud methodology refers to the explicit
act when the subject is instructed along with giving encouragement to think out loud and put
into words the thought process while the individual is answering (Redline, Smiley & Lee,
2001).
The benefits of the think aloud method includes the following.
 Independence from the bias that is obligated by recurrent interjections of the
interviewer
 Nominal training requirements of the interviewer
 It is a design which is open ended
The disadvantages of the thinking aloud methodology is summarized below.
 The subject usually requires training prior to this methodology of questioning.
 The subject might resists the method
 The subject might stray away from topic
 The subject might bias the description of the processing
Another methodology incorporated into the questioning process based on the
cognitive interview process is the verbal probing protocol which could either be retrospective
or concurrent (Collins, 2003). The retrospective probing techniques is when the subject is
asked after the official interviewing process to verbalize the precise thoughts about the
questions which they had answered earlier while responding to the questionnaire. The
concurrent verbal probing refers to the additional specific questions made in order to extract
further information from the subject after a certain response has been provided already by the

5COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
subject. The probes could either be spontaneous or scripted. Whether the verbal probing is
retrospective or concurrent, spontaneous or scripted, it is an imperative way to elicit more
information from the subject during the question and answer process (Satin & Fisher, 2019).
The advantages of verbal probing technique include the following.
 The interviewer controls the flow of the interview being conducted
 Training the subject is relatively easy
The disadvantages of verbal probing methodology consist of the points below.
 The technique is not reflective and might disintegrate the thought process of
the subject
 Poor probe selection might set bias during the interview
Even though the cognitive interviewing process has been acknowledged as an integral
and essential part to design a survey, the researchers are characteristically concerned with the
normalizing the standard questionnaires for the survey along with modifying the procedure
without giving sufficient time to ensure whether the subjects understand questions asked
(Collins, 2003). Addressing the design of the questionnaire or pilot testing or pre testing the
questionnaire is insufficient to make certain the accuracy and quality of the survey instrument
designed. The incorporation of the cognitive interviews is significant and essential
component of the survey research as the survey instrument is being developed (Kuh et al.,
2006)
Qualitative instrument for evaluation
Since the inception of cognitive interview process in 1999, comprehensive testing of
The National Survey of Student Engagement or the NSSE survey instrument along with the
aid of cognitive research has been performed considerable number of times (Ouimet et al.,

6COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
2004). The survey instrument of NSSE was launched in the year 2000 and last updated in the
year 2013 and it attempts in extensively assessing the extent at which the students engage
themselves in the educational activities with the high quality of development and learning
that is desired. The NSSE questionnaire gathers the data of responses in five different
categories.
1. The involvement in various education based activities
2. The requirements of the institution along with the nature of the coursework
3. The individual awareness of institute’s environment
4. The estimation of the educational along with growth at the personal level
5. The information on the background and the demography.
The cognitive testing was conducted in order to explore cognitive processes used by
the students in order to answer survey questionnaire and to recognize the items which are not
much understood by the respondents (Willis, 2005). The techniques and instruments used to
evaluate the students who have precise understanding of the questions which were used
which includes the cognitive interviews, while utilizing amalgamation of the thinking aloud
protocol along with the verbal probing methodology. The approaches were utilized so as to
verify whether the students understood the survey topics along with analyzing the
effectiveness of the newly modified survey matters along with the varied response options.
Preparation of the instrument
Thorough extensive cognitive interviewing protocols were conducted in the midst of
120 students along with 10 focus groups with total of 79 students over 12 different campuses.
The campuses which were selected for the purpose of testing the instruments featured
multiple characteristics namely the size, the religious affiliation along with the Carnegie
classification (Fosnacht & Gonyea, 2018). The first slot of interviews conducted in the year

7COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
2010, numerous techniques were utilized while interviewing the students. Few individuals
were enquired to respond to a particular survey question followed by asking the student to
describe the interpretation of that question given (McCormick, Gonyea & Kinzie, 2013). This
specific interviewing process was intended to measure and quantify development of that
specific questionnaire of the survey. The concurrent verbal probing instrument was generally
used for the initial rounds of the interviews followed by retrospective technique in the final
round of surveying.
Documenting the pre-testing process
The method of documenting the pre-testing process comprises of the question being
asked by the interviewer, then after the subject replies to the question, analyzing the answers
followed by the interviewer posing a most specific and precise question in order to extract
further information from the subject to collect a comprehensive accurate response (Willis,
2005). The probes used to elicit better response could be either spontaneous or scripted. The
retrospective probing used on the subjects was introduced during the end of interview on
order to verbalize the thoughts of the subjects on the questions faced by them while taking the
questionnaire.
The next set of data generation was intended at examining definite items along with
evaluating the survey instrument used holistically. The retrospective questioning, concurrent
inquiring in addition to thinking aloud protocol were used over again in order to check the
specific items, however the cognitive interview participants’ were requested to complete full,
online survey along with providing feedback regarding definite questions recognized in
interview protocol. The questions in the survey were the newly altered and customized
questions of NSSE survey instrument (Fosnacht & Gonyea, 2018). Students were asked to
highlight questions which they found confusing or ambiguous. The participants taking part in

8COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
the survey were instructed to pause after completion of a set of questions in online survey
then asked to answer specific follow-up questions on particular items. The participants were
requested to follow thinking out loud procedure as they responded to questions to
comprehend if the students could comprehend the questions accurately.
For the duration of the focus groups, the participants were supplied with hard copy of
the survey questionnaire and asked to read out loud and then reply the specific questions. The
simultaneous probing instrument was utilized to inquire definite follow up questions
associated with the survey items. The item that must be addressed in the focus groups was
resolute for specific campus. The items which were more demanding for the students were
tested additionally in focus groups. Beyond the aid of the face-to-face interviewing protocol
along with focus groups, final set of interviewing protocol included phone based interviews
with the students who were studying in the institute on a distance learning course
(McCormick, Gonyea & Kinzie, 2013). The participants undertook survey online while
talking to the interviewer over the phone. The online enrolled apprentices were also enquired
the identical set of questionnaire and was also asked to highlight items which seemed
ambiguous and complicated for answering. During these online cognitive interviews, an
interviewer was present along with a note taker instead of focus groups.
Critical Analysis
The focus groups and interviews were recorded and saved for analysis. The analysis
was based on these digital recordings along with the notes in the process of cognitive
interviews in addition to the focus groups (Pike, 2013). The final memorandum consisted of
the comprehensive summary of the response of every student to precise items along with the
investigators’ impression on quality of the questions in the survey on the basis of the
responses received from the subjects (Haeger et al., 2012).

9COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
The critically analyzed memorandums were categorized as the following
a) The no-problem marked point,
b) Negligible misinterpretation or inconsequential problem
c) Noteworthy problems observed
The assessment with these codes was conducted autonomously with the scrutiny of
two reviewers to ensure reliability of the evaluation (Willis, 2015). A third reviewer
investigated the coding prepared by the researchers along with addressing the differences in
the coding. The summary of diverse ways in which the students had interpreted questions was
produced to notify and assess the question writing for the items which the students had
understood, however had responded in varied ways (BrckaLorenz et al., 2012). The
researcher team conducting cognitive testing jointly had analyzed the findings along with
thoroughly vetting the interpretations in addition to the conclusions across the items (Pike,
2013). The findings collected from the final wave of collection of information and evaluation,
along with the psychometric examination, then were utilized to update the survey and
develop a newly modified questionnaire with enhanced response options.
Conclusion
Therefore, in conclusion, it can be asserted that incorporation of cognitive interview
process has enhanced the technique along with modification of NSSE survey instrument. In
addition, these modified approaches could be adapted by institutions which want to
comprehend survey items taken by the student and have a better understanding of their
responses. The modifications could also aid the institutional research staff along with others
to better contextualize the results post the NSSE survey (Fosnacht & Gonyea, 2018).
Advanced instrument could be important and highly preferred for the institutions willing to

10COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
carry out its own comprehensive survey along with exploring the responses to the survey by
means of enquiring the students regarding the campus context and syllabus. As a result,
sharing of this contextualized data could aid in advancement of the campus based on the
survey results by gathering tangible examples of behaviors amongst students along with
institutional practices.

11COGNITIVE INTERVIEWING PROCESS IN INSTRUMENT DESIGN
References
BrckaLorenz, A., Gonyea, R., Miller, A., Lambert, A. D., Laird, T. N., Qi, W., ... &
Tukibayeva, M. (2012). Updating the national survey of student engagement:
Analyses of the NSSE 2.0 pilots. In Association for Institutional Research 2012
Annual Conference, New Orleans, LA.
Collins, D. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality
of Life Research, 12, 229-238.
Conrad, F. & Blair, J. (2006). From impressions to data: Increasing the objectivity of
cognitive interviews. In Proceedings of the Survey Research Methods Section of the
American Statistical Association, Alexandria, VA.
Daugherty, S. D., Harris-Kojetin, L., Squire, C., & Jael, E. (2001, August 5-9). Maximizing
the quality of cognitive interviewing data: An exploration of three approaches and
their informational contributions. Proceedings of the Annual Meeting of the American
Statistical Association.
Fosnacht, K., & Gonyea, R. M. (2018). The Dependability of the Updated NSSE: A
Generalizability Study. Research & Practice in Assessment, 13, 62-73
Haeger, H., Lambert, A. D., Kinzie, J., & Gieser, J. (2012). Using cognitive interviews to
improve survey instruments. Association for Institutional Research, New Orleans.
Jobe, J. B. (2003). Cognitive psychology and self-reports: Models and methods. Quality of
Life Research, 12, 219-227.
Kuh, G. D., Kinzie, J., Cruce, T., Shoup, R., & Gonyea, R. M. (2006, July). Connecting the
dots: Multi-faceted analyses of the relationships between student engagement results