Developing a Reliable and Valid Measurement Instrument for Psychological Research

This research paper describes a process for developing a reliable and valid measurement instrument for psychological research, including levels of measurement, reliability, and types of measures. It also covers the seven steps necessary for producing a reliable and user-friendly scale. The paper focuses on the use of scaling, which involves both qualitative constructs and quantitative metric units, and the four scales of measurement: nominal, ordinal, interval, and ratio. The paper also discusses the importance of content adequacy assessment, questionnaire administration, factor analysis, internal consistency assessment, construct validation, and replication.

TESTS AND MEASUREMENTS
PART - 1
A PSYCHOLOGICAL CONSTRUCT

OVERVIEW
The presentation of this research paper has been done with a purpose of describing a process which would be
suitable for developing a reliable and valid measurement instrument which can be used in the psychological
research field. There have been instances when a researcher is not able to find an adequate or appropriate
scale for measuring an important construct. Under these circumstances, it becomes necessary to evolve a new
scale as the failure to develop a measurement instrument can result in invalid data. In order to build-up this
new measurement instrument, we need to understand some of the terminologies which are to be used for this
construct.
Measurement
Measurement, in a construct, is the process of recording the observations which are collected as part of the
research. There are two issues which need to be considered. The two major measurement concepts of
measuring are Levels of Measurement and Reliability of Measurement. Under Levels we have four major
segments: nominal, ordinal, interval and ratio. Under Reliability we include consideration and estimators
related to the true score theory.
We also need to understand the various types of measures which a researcher may use in social research. We
shall only consider four broad categories of measurements.
1. Survey Research includes designing and implementing interviews and questionnaires.
2. Scaling involves certain methods of developing and implementing the scale.
3. Qualitative Research involves an overview of non-numerical measurement methods.
4. Unobtrusive measures include different measurement methods which do not interfere with the research’s
context.

OVERVIEW
In this research paper we will adopt and work on developing a method based on Scaling. In a research process,
Scaling consists of a measurement method which involves the making of a construct which includes both
qualitative constructs and quantitative metric units. Scaling has found extensive use in the field of psychology
education for measuring the ‘unmeasurable’ constructs such as authoritarianism and self-esteem.
In Scaling, we make use of four scales of measurement: nominal, ordinal, interval and ratio. Basically, these are just
ways of categorizing variables of different type. Quantitative metric units, also referred to as numerical data, are
used where the observations are countable or measureable. In this context, we place 'Age' and 'income' as
continuous numeric variables. An 'age group' is considered as an ordinal qualitative variable and 'sex' is placed
under nominal qualitative variables.
Ordinal Measure
In ordinal measure, the interval between the values cannot be interpreted. However, in interval measurement, the
interval between two attributes can be interpreted. Let us take an example of measuring the temperature. Here,
the interval of 10 Units between 30 & 40 is the same as interval of 10 Units between 70 & 80. Here, the interval
between two values can be interpreted.
Ordinal Scale
This is a scale where data is placed just in order of its magnitude and there is no method or standard of measuring
the intervals between two quantities. As an example, consider the game of squash ladder which is measured on an
ordinal scale and from the results we can say that one person is better than the other, but we cannot quantify the
margin.

PART – 1: SCALE DEVELOPMENT AND
ANALYSIS
The purpose of this paper is to describe a process for developing a construct using multi-item, multi-subscale and
interval-level scales. The essential seven steps necessary for producing a reliable and user-friendly scales have been
followed.
The seven sections discussed hereunder cover the seven steps of a scale development. The process of scale
development begins with the first step which is creation of items for assessing the construct under consideration. This
step can be constructed either by allowing the process to be conducted inductively, where the items are generated
first and then the scales are derived; OR by allowing the process to be conducted deductively, where the theoretical
definition is created first and then the items are generated.
Step 1. Item Generation
The process of Deductive Scale Development is based on the theoretical definition of a construct and this has been
used as a guide to create the items. This approach is based on an understanding of the relevant theory of the case
study, of the phenomenon which is to be investigated and helps the researcher in ensuring that there is content
adequacy in the final scale constructs. This case study also provides a good example of the deductive approach to be
followed for item development as it also creates a measure for lodging quality.

PART – 1: SCALE DEVELOPMENT AND
ANALYSIS
Step 2. Content Adequacy Assessment
Pre-testing the construct items for their content adequacy is a step that is often overlooked although it is a very
essential step in the process of scale development. This pre-testing helps the researcher and assures him the
adequacy of the content, prior to his finalizing the questionnaire. This also provides support for the validity of
the construct as it allows the researcher to delete those items which he may find to be conceptually
inconsistent with his final construct. In has happened on many occasions where researchers, after investing
substantial time and effort in the collection of large amount of data sets, finds that one of the important
measure is flawed and needs to be corrected.
This paper has also found a recently developed method which can be used for conducting a content
assessment. This methodology uses both sorting and factor analysis technique for assessing on a quantitative
scale the adequacy of the content of a set of the newly developed items. In this new method, the respondents
are asked to rate the extent to which the items enlisted in the questionnaire corresponded with definitions
provided in the construct. The responses given by the respondents are then factor-analyzed and only those
items are selected which can be retained for subsequent admission into an additional sample.
The items analyzed are meant to represent four constructs. But if we do not know how many separate
dimensions are in the data, then there are three alternative procedure to address this question.
1 Principal Components Analysis; 2 Factor Analysis; 3 Cluster Analysis
All the three procedures can attempt to approximate the nvar * nvar correlation matrix R with a matrix of lesser
rank, one that is nvar * nf.

PART – 1: SCALE DEVELOPMENT AND ANALYSIS
Step 3. Questionnaire Administration
The retained items, taken from Step-2 are then presented through an appropriate sample construct
and the objective of this exercise is to examine how well these items can confirm the expectations
with regard to the psychometric properties required from the new construct. The new items are also
required to be administered in relation to the other established measures adopted for the construct
for a later assessment of any overlap among the existing and proposed scales of the construct.
In addition to this, the data from the existing constructs can be later used for a preliminary
examination of another construct and its criterion-related validity of new scales. After the data
gathering has been completed, it is essential for the researcher to evaluate the performance of the
items included in the construct so as to validate whether they have adequately constituted the
construct scale. In the opinion of this paper, item evaluation through factor analysis should be
retained as one of the most essential steps for determining the viability of the construct scale.

PART – 1: SCALE DEVELOPMENT AND ANALYSIS
Step 4. Factor Analysis
Scale Development process can be executed with the help of two basic types of factor analyses. The first one is
known as exploratory factor analyses. This finds use in reducing the set of observed variables to smaller, yet
more parsimonious set of variables. The second type, known as confirmatory factor analyses is used for
assessing the quality of the factor structure and this is done by statistically testing the significance of the overall
model, and also by testing the relationship among the items and scales in the construct. It must however be
pointed out that the researcher, prior to conducting the factor analyses, must examine the inter-item
correlations among the variables and any other variable that may correlate at less than 0.4 with all other
variables then this variable may be deleted from the analysis. Low correlations of these types indicate about
items which have not been drawn from an appropriate domain and which may be responsible for producing
errors and unreliability.
Step 5. Internal Consistency Assessment
Although reliability of the items can be calculated in a number of different ways, the most common measure
used in field studies for the purpose of assessing a scale’s internal consistency is Cronbach’s alpha. This can tell
the researcher how well the items are measuring in the same construct. But the researcher has to first
establish the uni-dimensionality of each scale. Once the researcher has conducted the exploratory or
confirmatory factor analyses and has deleted all the bad items, he can successfully calculate the internal
consistency reliabilities for each of the scales. A large coefficient alpha, of above 0.70 for an exploratory
measure, indicates a strong item covariance and suggests that the researcher has adequately captured the
sampling domain.

PART – 1: SCALE DEVELOPMENT AND ANALYSIS
Step 6. Construct Validation
After completing Step-5, the new scales will reflect the content validity (refer to Step-2) and the internal consistency
reliability (refer to Step-5). Both these factors provide the required supportive evidence of the construct’s validity.
The researcher can find more evidence of the construct’s validity by examining the extent to which these scales are
correlating with the other measures which have been designed for assessing similar constructs ( for convergent
validity) and to those they do not correlate with the dissimilar measures (for discriminant validity). The researcher
can also find these useful for examining the relationships with those variables which have been theorized to be the
outcomes of the focal measure (for criterion-related validity).
Step 7. Replication
The last and final step in completing this construct scale development process is to collect another set of data from
another appropriate sample and then repeat the above mentioned scale-testing process with these new set of
scales. In case the researcher had a large enough initial sample data, he can split the original sample data randomly
into two sets and conduct a parallel analyses for his scale development. Researcher can also avoid the common
source problem, by collecting data from other sources, such as performance appraisals can be collected wherever
possible. However, the researcher must keep in view that replication includes confirmatory factor analyses,
assessment of internal consistency reliability and construct validation for all collected data. These analyses will
provide the researcher with the confidence and he can finalise the measures which possess reliability and validity
and which will be suitable for use by him or any other researcher in future research.

TESTS AND MEASUREMENTS
PART - 2
A PSYCHOLOGICAL TEST

A PSYCHOLOGICAL TEST
The Theory of Planned behavior (TPB) has been acknowledged widely as a health behavior theory which
has been found to give robust and parsimonious account of those factors which predict health behavior of
respondents in a test. In a simple yet effective way, TPB works on the assumption that behavior can be
predicted by intentions of the respondents of engaging in that behavior. TPB further links intentions with
attitude and norms and states that these have direct effect on the behavior and the perceived behavioral
control over behavior by respondents. In this paper, we are working on the assumption that attitude of the
respondents reflects the positive and negative feelings of the respondents towards behavior and norms will
refer to the subjective perception about the beliefs which the respondents hold towards their behavior.
Our studies have shown that TPB has been successfully predicting the abuse of a number of substances
through social criteria. Also, recent meta-analytic works have been successful in demonstrating the
experimental manipulation of attitudes, norms and self-efficacy in theoretically predicting the changes in
intentions and behavior of the respondents. However, little research, so far, has been undertaken about
identifying any important results through experiments and studies of testing the TPB in the domain of
marijuana abuse. With the increasing inclination in the US towards abuse of marijuana and its legalization,
it is the view of this paper that it is time to examine the validity of TPB as the base for possible intervention
in preventing the abuse of marijuana in respondents.

A PSYCHOLOGICAL TEST
We are of the opinion that perceived behavioral control has reference to the subjective perceptions of
the respondents about their ease of performance behavior. Where the context of substance abuse is
concerned, attitudes and norms can usually be measured with respect to the respondents’ engaging in
the behavior and the perceived behavioral control will be operationalized as efficacy to resist the abuse of
the substance. Although the experimental manipulations regarding TPB constructs have not been carried
out in the context of marijuana, various studies have been conducted for examining the degree to which
TPB can account for variability in marijuana abuse under correlational designs. These studies have also
been able to confirm that more positive attitudes, more positive perceived norms and lower efficacy to
refuse marijuana use lead to the prediction of greater intentions of using marijuana.
Most of these studies, however, have been limited by their reliance on those samples which had low
absolute levels of marijuana use or were lacking in information about the overall levels of use within the
sample. These studies were also not able to reveal the connection between operations of the model
within the broader cultural context of favourability towards marijuana, which is generally referred to as
the pro-marijuana community. These features of the past studies have made it difficult in determining the
extent to which TPB can account for marijuana-related intentions and behavior across the larger cultural
context of relative favourability towards its use. Both aspects are important for consideration under the
given increases in the use of marijuana which are likely to accompany its legalization in the US.

TESTS AND MEASUREMENTS
PART - 3
Construct of Interest

CONSTRUCT OF INTEREST
METHOD
Given the superior efficiency of theory-based interventions in comparison to those which are not theory-
based, the study by Ito et al. (2015) successfully examined marijuana abuse within the framework of the
theory of planned behavior. It is a dominant model which explains health behavior and determines
whether components of the TPB theory have associated with both the intention to engage in a behavior
and behavioral engagements as such. The general area of interest were young college students and their
responses to marijuana legalization in the United States. Specifically, the TPB reveals that the predictor
variables concerning the psychosocial influence of the constructs of initial attitudes, norms, and efficacy
were related to the outcomes reported, resisting use on initial intentions and examined the effects of the
original plans on the use of marijuana.
Moreover, Ito et al. (2015) have also reported that these relationships get moderated by following the
construct variables such as normative influences, both descriptive and injunctive, along with two kinds of
intentions, namely use and proximity intentions. However, recent research has also revealed the following
limitations of the study as its reliance is on self-reported measures. So, as per the rationale of this report,
use of quantitative survey method with the young college students should be used to address this
limitation as it is significant for the constructs of self-reported attitudes, intentions and refusal self-efficacy
which account for considerable modification in marijuana use (Ito et al. 2015 p.57).

CONSTRUCT OF INTEREST
PARTICIPANTS
Participants were 370 University of Colorado students taking part in a three-year longitudinal study. As
evidence of marijuana favorability in this context, students from the university from which we recruited
reported using marijuana within the past 30 days at more than double the rate (38.2%) of other US college
students (15.8%) (American College Health Association, 2011). In order to ensure that the sampling was of a
wide range of use, participants were recruited as never users (those who never tried marijuana), infrequent
marijuana users (those who used marijuana four times or less per month for less than three years) and
frequent marijuana users (those who used marijuana an average of 5 days a week or more for at least the
past year). The respondent’s eligibility was determined through phone interviews prior to enrollment.
Although we planned to sample the full range of recreational marijuana users, we did not intentionally focus
on recruiting dependent users. Examination of responses was with reference to the Marijuana Dependence
Scale, a scale which is based on Diagnostic and Statistical Manual of Mental Disorders (4th ed.) criteria for
determining dependence (such as “When I smoked marijuana, I often smoked more or for longer periods of
time than I intended”; “I need to smoke more marijuana to achieve the same ‘high’”) and this was
completed during first session and which showed that even though marijuana use was high among the
frequent marijuana users in this sample, frequent users on average endorsed only 4.30 of 10 symptoms of
dependence. We described in details only those measures of interest relevant to our current hypotheses.

CONSTRUCT OF INTEREST
PROCEDURE
Potential participants, recruited through email and phone invitations using their university account and
advertisements on campus. Those interested in the study were initially interviewed on the phone by the study
personnel for determining their eligibility. Participants who successfully met the selection criteria for inclusion
were then invited to participate in two sessions a year for a total of three years. Data for the present analyses
came from the first four sessions. Because our interest was in knowing how the psychosocial factors will help in
predicting actual use, our primary analyses were to use attitudes, norms, RSE, and intentions which were
measured in Year 1 for predicting use in Year 2.
We focused specifically on data obtained during the first two years of assessment as we found that this period
had captured the first two years of college for majority of the participating students and it was the appropriate
time when marijuana use risk is presumed to have increased and students are potentially getting exposed to
new social influences. It is also the appropriate period where young adults begin taking more personal
responsibilities for their life choices. The first two laboratory sessions, during which the baseline data were
obtained, occurred on an average within 5.77 days of each other (SD = 5.12). The first Year 2 session occurred
approximately 12 months after the first Year 1 session (M = 362.77 days, SD = 19.30 days). The second Year 2
session occurred on average 5.79 days later (SD = 6.53). We had instructed our participants to abstain from
alcohol for at least 24 hours, recreational drugs (including marijuana) for at least 6 hours and caffeine and
cigarettes for at least 1 hour prior to each laboratory session.