Reliability Report: Assessment, Test Development, and Item Analysis

This report provides a comprehensive overview of test reliability, beginning with an introduction to the concept and its importance in assessment. It delves into Classical Test Theory, explaining how observed scores, true scores, and error are related. The report explores the sources of measurement error and its impact on test consistency. It then examines the Standard Error of Measurement and the calculation and interpretation of reliability coefficients. Different types of reliability, including test-retest, parallel forms, split-half, and inter-rater reliability, are defined and discussed. The report also covers the item development process, emphasizing the creation of multiple-choice questions for assessing reliability. The report concludes with recommendations for administering tests effectively and ensuring the validity of assessment results. The provided appendix includes a content matrix, questions, and an answer key.

Running Header: RELIABILITY 1
REPORT ON RELIABILITY
NAME
COURSE
INSTITUTION

RELIABILITY 2
Introduction and purpose of the test development
“Tests are measuring device which is used to numerically present the degree of knowledge,
skills, and/or abilities corresponding to a domain of interest” (Abdellatif & Al-Shahrani, 2019).
Tests are always done to evaluate the progress of a learner, it can be done in junior levels, middle
and also senior levels of studies. They can also be administered so that the examiner or a teacher
can access the understanding of his/her subjects, they are graded according to a set up criteria.
This criterion categorizes the subjects into different groups according to their scores.
Tests helps the examiner to evaluate knowledge, skills and ability of an examiner and hence
gauging his/her performance on a certain content. The content which is commonly tested by the
examiner is within the field of the taught. According to this report, the tests which are concern
are on the topic; reliability (Abdellatif & Al-Shahrani, 2019).
Background on Reliability
“Reliability is the degree to which a test produces stable and steady results” (Duke et al., 2020).
It is usually evaluated after administering a test to a group of examinees within an interval of
some period of time and thereafter compare the consistencies of the results. Reliability is a very
vital aspect during evaluation of their knowledge and skills on a certain domain of content the
examinees obtained.
“Reliability is classified into six types, these are test-retest reliability, parallel forms reliability,
split-half reliability, internal consistency reliability, inter-rater reliability and finally difference
score reliability” (Duke et al., 2020). Each types of reliability is calculated differently but they all
gives the reliability of a test. “Test-Retest reliability measures reliability by administering a test
to a group of examinees twice within a period of time and their results correlated to obtain

RELIABILITY 3
reliability” (Noble et al., 2019). On the other hand parallel forms reliability is obtained through
administering different versions of a test probing the same knowledge, skills and ability to an
individual group and the results correlated to obtain reliability. Also, there is inter-rater
reliability which is administered to raters and the results compared. Besides that, internal
consistency reliability measures reliability by administering different tests with similar probe but
different items and gives the same results. Finally, there are other two types of reliability,
average inter-item correlation which administer same test with similar probe and obtain the
results and thereafter, find their average while split-half reliability splits the all the items of the
tests with similar probe in an area and both administered to the examinee and both results
obtained are correlated to determine reliability. All the above types of reliability can be used to
obtain reliability of a test.
Item development process
Item development process is the general procedure which an examiner can use when formulating
a test. A test has to undergo some stages before it is administered to an examinee. This process
ensures that the examinees get the best out of the subject examined.
When an item is being developed, the examiner should write down the question to be included in
the test. This is the first step whereby the examiner writes down all the questions related to a
content to be tested and these questions should be more than the required number in the test.
After the questions have been written down, they undergo a vigorous review process (Taufiq et
al., 2019). This will help the examiner to identify and eliminate errors which might have been
made on technicality, bias and language structure of the each item in the test. The examiner will
then draft all the questions without any errors and organize them into a required number ready to
be given to the examinee(s) (O'Sullivan, 2019).

RELIABILITY 4
Proposed method of administration
The examiner has to carefully analyze and determine the best method to administer the test. This
will help in getting the best out of the intended purpose of the exam. The most appropriate
method to administer the test will be a multiple choice questions. The examinee ought to answer
the questions in the test by selecting only one choice in each question. The multiple choice
questions are the best method to administer the reliability test so that the examiner can test on the
knowledge, skills and ability of the subjected examinee. It is the easiest method and gives the
best outcomes (McDaniel & Little, 2019)
Conclusion
Reliability is regarded as the most vital aspect to consider whenever an examiner is setting a test
for a pilot sample. It ensure that the examinee gets the best out of a subject content. Reliability
can be tested under several circumstances as given in the types of reliability. Every test has
different types of reliability to be tested depending on the method it was administered to the
examinee.
Recommendation
Every test should be administered to the examinee carefully depending on the intended
outcomes. Each test has its own intended results and therefore the method of administering
should be carefully chosen. However, the best and convenient method of administering the
reliability test is through multiple choice questions to the pilot sample.

RELIABILITY 5
References
Abdellatif, H., & Al-Shahrani, A. M. (2019). Effect of blueprinting methods on test difficulty,
discrimination, and reliability indices: cross-sectional study in an integrated learning
program. Advances in medical education and practice, 10, 23.
Duke, C., Hamidi, S., & Ewing, R. (2020). 6 Validity and Reliability. Basic Quantitative
Research Methods for Urban Planners.
McDaniel, M. A., & Little, J. L. (2019). Multiple-choice and short-answer quizzing on equal
footing in the classroom: Potential indirect effects of testing.
Noble, S., Scheinost, D., & Constable, R. T. (2019). A decade of test-retest reliability of
functional connectivity: A systematic review and meta-analysis. Neuroimage, 203,
116157.
O'Sullivan, B. (2019). Redefining specific purpose tests. Developments in Language Education:
A Memorial Volume in Honour of Sauli Takala, 250.
Taufiq, M. A., Putri, R. E., Agustina, I., Zaim, M., & Jasmienti, S. (2019). Item Analysis and
Teachers’ Factors in Designing a Test.

RELIABILITY 6
Appendix
Process by content matrix
Source Knowledge Application Principle Total
Psychometrics 1 1 1 3
Mathematics 1 0 1 2
Item Analysis 3 1 1 5
Total 5 2 3 10
Test and answer key
Question 1
What is reliability?
A. the qualitative measure if the content of a test meets the intended purpose.
B. the degree of originality of content of a test.
C. The ability of test produces stable, steady results and exempting errors.
Answer: C
Question 2
There are five types of reliability, which ONE of the below is NOT?
A. Internal consistency
B. criterion-Related
C. Test-retest

RELIABILITY 7
Answer: B
Question 3
According to Classical Test Theory, an error of a test can be evaluated. What is its formula?
A. Error = observed score- true score.
B. Error = true score- observed score.
C. Error = observed score.
Answer: A
Question 4
All statements are true regarding reliability, except?
A. Reliability coefficient between 0 and 1.
B. A test is more reliable when it has less measurement error.
C. Reliability is measured the same way as content validity of a test.
Answer: C
Question 5
Identify one factor which does NOT influence reliability?
A. number of items
B. standardized instructions
C. unclear items

RELIABILITY 8
Answer: A
Question 6
James wanted to calculate SEM, how will he find it?
A. evaluate the product of coefficient reliability and its standard deviation.
B. evaluate the standard deviation * (square root of 1 – reliability coefficient).
C. evaluate the reliability coefficient.
Answer: B
Question 7
Classical Test Error has three assumptions. Which of the following statements is NOT among the
assumptions?
A. The correlation between true scores and error scores is zero
B. The absolute mean of an error must be zero.
C. The mean of an error ranges between 0 and 1.
Answer: C
Question 8
A test on reliability was administered to a group of ten students on Monday morning. Another
test on the same content was administered to the same group of students two weeks later. When
the results of both tests were released, it was realized that the students who had scored higher

RELIABILITY 9
marks in the first test managed to score higher again in the second test. What was the reliability
of the tests?
A. The tests were highly reliable.
B. The tests were not testing reliability.
C. The tests showed low reliability.
Answer: A
Question 9
Prof. Arthur once measured his weight three times and these were the results;
First time = 70.9kgs, Second time = 75.2kgs and the third time = 74.0kgs. What does this
illustrate on the reliability of the machine?
A. The weighing machine had high reliability.
B. The machine had low reliability.
C. The machine indicated nothing about reliability.
Answer: B
Question 10
Which of the following statements define inter-rater reliability?
A. The measurement of a degree which two or more different raters agree in their decision over
an assessment.

RELIABILITY 10
B. The measurement of degree to which different tests under the same content produce similar
results.
C. The measurement of the degree to which different raters or judges disagree over an
assessment.
Answer: A