Critical Analysis: Wagner's Critical Thinking Test Manual Evaluation

This assignment critically evaluates the Wagner's Critical Thinking Test (WCTT) manual, assessing its reliability and validity based on the provided information and studies. The analysis begins by defining reliability and examines the test's consistency through test-retest correlations, noting a high reliability score. The assignment then delves into the crucial aspects of validity, including content validity, which is found to be lacking due to insufficient expert consultation and evidence. Criterion validity is explored, revealing moderate correlations with related tests like the Watson-Glaser Critical Thinking Appraisal, but weaker correlations with Need for Achievement. Construct validity is also assessed, highlighting limitations in the factor analysis and the absence of key measures. The assignment emphasizes the need for further validation studies, including a thorough description of validation procedures, criterion-referenced validity testing, and evidence supporting the test's ability to predict academic success. The conclusion stresses the importance of both reliability and validity for a test's acceptance, and suggests that, based on the presented evidence, the WCTT falls short of being a fully valid instrument for evaluating critical thinking in undergraduate and graduate courses. The test's potential use as a preliminary measure is also discussed, along with the need for caution and additional analysis.

1. Reliability of a test implies that the test produces consistent results when taken over
different periods of time, across different researchers, and across items. In the present
case of the test manual of WCTT, reliability can be checked through the results of
Study 1. The same group of participants took the test twice at an interval of 16 days.
Here, the correlation was found to be 0.89. As a norm, correlation of +0.80 or higher
is a good indicator of reliability. Reliability in WCTT implies that the students who
scored high on critical thinking in week one would be tested high on critical thinking
after two weeks or two months as well. Critical thinking as a competency is largely a
variable like personality factors or intelligence that would be consistent over time
unlike moods or emotions that may vary across the time dimension. Hence, a high
reliability is to be expected in such a test. A study conducted at a different institute in
Study 4 yielded Cronbach alpha coefficient of 0.78 which is a good indicator of
internal consistency and reliability. In social science research, such as the present
one, a Cronbach alpha coefficient of 0.70 or higher is considered acceptable, when
there are dichotomous questions that is questions with two acceptable answers. This
measure of reliability indicates that the scale measures what it is supposed to measure,
which is critical thinking in this case. This measure is also useful in ensuring
reliability of latent or hidden variables.
2. Along with reliability, it is important to measure the validity of a test which is a
measure of whether the test measures what it truly represents. Content validity
measures the extent to which the tool includes the concept in its entirety. WCTT
includes four indicators to include critical thinking. Content validity is assessed by
measuring the tool against the conceptual definition. In the case of WCTT, there is
not much evidence suggesting that Wagner did a content validity of his tool. A good
way to measure content validity of WCTT would have been to consult several subject
matter experts to gauge whether the test covered all essential measures of critical
thinking. In the present case, Wagner based his test manual on a review of extant
literature and his judgment, but the efforts taken for ensuring content validity have not
been mentioned in the case. Content validity is an essential component in
determining the validity of an instrument since it ensures that the test measures what it
intends to inn its entirety. For example, a panel of experts may review the test and
find that a component such as maybe ‘problem solving’ is missing in the test manual
and that would make it difficult to attain generalizability of the WCTT. Content
validity is also necessary to ensure that there is no bias in the measurement and that

all indicators are measuring the relevant aspects of critical thinking. It can be
understood as a more formal measure than that of face validity.
3. Criterion validity is a measure of well the test scores are correlated with other
measures that may be known to be related to measure. For example, one who scores
higher on critical thinking may be expected to score high on motivation to learn. In
the case of WCTT, Study 2 measured the scores of critical thinking along with several
other related criteria or measures. It can be inferred that WCTT has a moderate
correlation with the Watson-Glaser Critical Thinking Appraisal (r=.61, p<.001) and
the Peak Verbal Ability Test (r=.43, p<.001) while a low or weak correlation with no
statistical significance with Need for Achievement (r=0.6, ns). This indicates that
score on critical thinking can be used to some extent to predict performance on
Watson-Glaser Critical thinking. It also can be interpreted that someone with a high
score on WCTT would have a high score on the test of their verbal ability. If critical
thinking measured through WCTT was being used as a predictor for one’s need for
achievement, the score and absence of statistical significance would indicate that it
would not be valid. In criterion related validity, it is important to test for those
measures or criteria that would be expected to be correlated to the construct of
interest. For example, critical thinking score may have a high correlation with high
IQ but it may not necessarily imply that one is predictive of the other. This measure
indicates how well the construct of interest has been operationalized by the researcher
by measuring it against already established criteria.
4. Construct related validity of WCTT would essentially indicate the extent to which the
test can be generalized that is how much the sample selected would be an accurate
representation of the population. The WCTT is supposed to measure Critical
Thinking. Through the measurement of different types of validity such as face and
content validity, and criterion-related validity (measured through predictive,
concurrent, convergent, and divergent validity), construct validity can be established.
In addition to the measures stated above, the factor analysis shows weak loadings of
factors 3 and 4 that is IEA and AIC. There could be various reasons attributed to this.
Additionally, Wagner used simple language structures in his instrument which may
appeal to taking the test accessible across cultures. The evidence is incomplete to
measure construct validity. The data from confirmatory factor analysis is a good
indicator of the nature of the construct being measured. The present evidence lacks
important measures of goodness of fit, comparative fit index, and Chi-squared test.

The results of studies 3 and 5 are insufficient to prove construct validity as well since
we do not know whether critical thinking is an accurate predictor of college grade
point scores. The sample varies across streams of education and there is more data
needed about the descriptive statistics of the various respondents to make any
conclusive claims about the validity of the measure of critical thinking as measured
by WCTT.
5. One of the possible uses of WCTT can be to have a preliminary understanding of
critical thinking abilities of college students owing to its simplicity, brevity, and low
cost. This is however to be used with caution and only as a source of additional
information rather than to predict any other behavioural outcomes. The test has a
good reliability score and hence it can be used as an initial measure of the critical
thinking abilities of the students. There would be additional analysis and evidence
needed to understand whether the test can be applied to understand the
appropriateness of academic courses meant to enhance critical thinking in the
students. The WCTT can be used as a pre-test for a control group of students who
would then be administered the same test after a period of 3 months or when specific
courses related to critical thinking have been completed. This would give an
indication of the usefulness of the test. The instructors of the courses related to
critical thinking can give their understanding of elements of critical thinking and
assess whether the WCTT was covering all of them. This would also give an
indication of the content and face validity of the test manual.
6. In the present case as mentioned in the case, Wagner has managed to create a reliable
instrument in the form of the WCTT to measure Critical Thinking. The test manual in
its present form however has not been tested for its validity. The test can therefore, in
my opinion, not be used to make valid predictions of students’ success in
undergraduate and graduate academic programs. I would need to have the contents of
the courses meant to develop critical thinking skills to know whether the four
elements as designed by Wagner namely, understanding meaning, recognizing
assumptions, identifying and evaluating arguments, and assessing inferences and
conclusions, are being explained and evaluated through these courses. I would also
need evidence of criterion related validity of the test manual, including concurrent and
predictive validity, to make it more robust and acceptable for evaluation of critical
thinking abilities. I would also need a criterion referenced validity testing between
WCTT and performance on the tests and overall grade scores of the students. The

presence of correlation does not imply causation or appropriate operationalization of
the construct. Reliability indicates the trustworthiness of the tool while validity is
essential to ensure its applicability in a certain situation. A good test needs to have
both reliability and validity to become an acceptable instrument for measurement of
the desired construct. Although the various studies mentioned in the manual indicate
a strong reliability score, they are not enough to prove the validity of the instrument
and hence in its present form and based on the evidence presented in the manual, the
WCTT falls short to live up to the desired expectations of being a valid tool to
measure critical thinking in the students of undergraduate and graduate courses. The
test manual needs to include a thorough description of the validation procedures used,
too.