Programme Evaluation

This document discusses programme evaluation and its application in research. It explores the steps involved in identifying and retrieving relevant studies, as well as the threats to internal validity. Additionally, it provides an overview of the process of conducting a summative evaluation. The document is relevant for students studying research methods or programme evaluation.

Running head: PROGRAMME EVALUATION 1
Programme evaluation
Student’s Name
Institutional Affiliation
Date

PROGRAMME EVALUATION 2
Programme evaluation
Question 1 (a): Application of PICO format
The PICO format can be used in framing research problems or questions in evidence-
based models through locating, assessing, and evaluation of the necessary elements that are
pertinent to the research such as the population or problem, indicators or interventions,
comparison, and finally the outcomes. The approach would be used in evaluating the systematic
review done by Maynard, Solis, Miller, and Brendel (2017).
Patient/ problem/ Population
The population of study entails the pre-school, primary, and secondary school children
from any country that are in special or regular education who are expected to improve their
academic standards and accountability through a positive response to emotional and behavioural
needs. Additionally, the participants or selected population was made of both male and female
children from all the races. The Centre for Disease Control and Prevention (2013) claimed that
nearly 13 to 20 per cent of the children in the United States exhibit at least one or more mental
disorders such as social anxiety and stress which reflects heavily on the academic performance
among the students. Therefore, there was a need for an increase in the emphasis on the social,
behavioural, and emotional needs of the students in various schools through the following
intervention.
Intervention of interest
The intervention undertaken were conducted in schools in the course of the school days
or during the school-based at the end of the daily program. Mindfulness-based intervention
(MBIs) was deployed which majorly is centred on purposeful paying attention during certain
procedures such as group discussions, debates, answering of questions as well as following the

PROGRAMME EVALUATION 3
lessons taught in class appropriately (Jiga, Kaunhoven, and Dorjee, 2019).). Moreover, Dunning
et al. (2019) suggested that the intervention of the MBIs has positive effects on individuals with
chronic conditions, psychiatric disorders, mental health, and stress among other conditions.
Therefore, the tool will help the identified population of study to enhance their social and
emotional states and focus on their studies. An example of the MBIs that was an effective
intervention was the Mindfulness-Based Stress Reduction (MBSR) suggested by Raj and Kumar
(2019), Meditation of the Soles of the Feet (SOF) acknowledged by Semple and Burke (2019),
and the Mindfulness-Based Cognitive Therapy for children (MBCT-C) that is recommended by
Cotton et al. (2016).
Outcomes
The review reported the following outcomes among the studies including cognition
outcome, for example, attention, memory, and executive functions. Another finding from the
study was the academic performance that was evaluated using the measures of content mastery,
reading, grades, and standardized achievement tests. On the other hand, the review revealed
behavioural outcomes such as attendance, compliance, the timely accomplishment of tasks,
aggression, and disciplinary referrals. The socio-emotional and the psychological outcomes were
also evident such as social skills, emotional regulation, engagement, stress, and anxiety.
Question One (b): The level of the hierarchy
The evidence in the study was based on level one, which was a systematic review of all
the relevant Randomized Controlled Trials of good quality with similar results. The RCT study
design was used due to its provision of assigning the population into experimental groups or
control groups, which enable the researcher to establish the differences in the outcome variable
being studied. The findings of the study at this level of study are important, as they are evidence-

PROGRAMME EVALUATION 4
based on clinical practices and experiments thus possess a high quality that can be relied upon or
replicated to larger populations. Furthermore, Thomas et al. (2015) alluded that the Randomized
Control Trial study design is used majorly for three reasons. First, the author acknowledged that
the design reduces the biases in research, effective in approximating a controlled experiment, and
finally they are statistically efficient concerning the implications of the study. Concisely, the
survey of Maynard, Solis, Miller, and Brendel (2017) utilised the RCT in conducting the
systematic review that enables the authors to identify the relevant literature that could help in
answering the identified research problem.
In research, it is common to find many scholars who have studied similar researches
using different approaches. A systematic review is used to help researchers identify the most
suitable literature that fits the requirements of the PICO strategies inclined to their area of study
(Ajimsha and Shenoy, 2019). This is done using appropriate inclusion and exclusion criteria
selected by the author, which may include the use of certain terms, phrases, and years of
publication among other factors in sampling the pieces of literature in a systematic review. For
instance, it was essential to conduct the review in this study helped in the understanding of the
mindfulness-based approaches among the youth and the children before performing the
Randomized Controlled Trials.
Question Two (a): Steps in identification and retrieval of published and unpublished
studies that meet the inclusion criteria
The inclusion criteria for the research was that the studies published or conducted from
1990 to present considered as a cut off for implementing MBIs within the school settings.
However, the investigation was not restricted to geographical locations, publication status, and
the language used in the studies. The steps involved in the identification and retrieval of the most

PROGRAMME EVALUATION 5
suitable literature involved comprehensive electronic searches that entailed the electronic
databases, grey literature sources, research registers, and another relevant reference list of
reviews and studies. An example of the electronic databases was the British Education council,
the Australian Education Index among others while examples of research registers, and websites
included the National Technical Information Services and the System for information on grey
literature. The use of search terms and keywords was utilised where the combination of the
names associated with the study design, the population of interest, and intervention was keyed in
the electronic databases or other source searches. At this stage, the use of database-specified
frameworks was explored to each database that included truncating, consultation of the database
limiters were employed to precise the search within every database.
Question Two (b): Steps for selection of studies and data management and extraction
undertaken in the research
Study selection process
The selection of the studies involved screening of the abstracts and titles that were found
using the search procedures with an in-depth consideration of the CBCA Education database, the
British Education Index, and the Australian Education Index databases. A search specialist was
contracted to conduct searches in these databases, as the author had no access to the databases.
Concisely, one review screened for the titles and the abstract that was relevant to the
interventions, population and the study design. At this stage, the articles were deemed relevant or
irrelevant depending on whether the titles/ abstracts were appropriate or inappropriate; in
essence, those studies that did not involve the target population or the mentioning of the
intervention in their abstract were classified as inappropriate and thus excluded. However, those
whose title and abstract had information that was pointed to the mentioned aspects were included

PROGRAMME EVALUATION 6
and subjected to two reviewers for further independent review through detailed screening to
determine their final inclusion. During the review by the two independent processes, the arising
discrepancies were evaluated, discussed, and resolved through consensus (Riley et al., 2019).
The study selection process was summarised in the table below.
Figure 1: Flow chart for study selection process

PROGRAMME EVALUATION 7
Extraction of data and management
After the studies had qualified for inclusion in the eligible screening process of study
selection, they were independently coded by the two reviewers using a structured form of data
extraction as acknowledged by Maynard et al. (2017). Concisely, the data extraction form that
underwent pilot testing by the coding team entailed items related to information of bibliographic
content and the source descriptors such as the methods and procedures, intervention
implementation, nature, and context. The items also included the characteristics within the
samples, the outcome of data and the calculation of the sample size effects. However, because of
the massive number of the studies, the review team members responsible for coding of the
primary studies were Kristen Brendel, Veronica Miller, Michael Solis, and Brandy Maynard. At
this stage, the coders carried a comparison in their codes to identify the potential disparities,
which were discussed in details and resolved after the independent coding of the studies. In cases
where the attempts to resolve the discrepancies failed between the two coders, a third party was
brought in to analyse, assess, and help in settling the disparity between the two coders.
Question Three: Some of the threats to internal validity
Validity in research is an indication of how sound the research design and the method
used that truly represents the phenomenon or claims being measured or under investigation
(Goodwin and Goodwin, 2016). On the other hand, internal validity indicates how well an
experiment is conducted which dependents on the degree of confounding in the study. It is
concerned with the degree of control or rigour over the extraneous variables in the study. Some
of the potential threats to internal validity in the study are maturation, history, statistical
regression, selection, and the experimental mortality as explained in the subsequent paragraphs.

PROGRAMME EVALUATION 8
Maturation threat is likely to occur in the study because of physiological processes
among the participants that could cause various changes in behaviour (Flannelly and Jankowski,
2018).). The selected participants could change between the experimental test sessions that could
affect the results. Such changes could be fatigue, hunger, motivation, and the span of attention
among the participants.
Another threat is the statistical regression, which defines the tendency of participants
receiving extreme score when tested to incur less extreme scores throughout the subsequent
retests even in the absence of any treatment in the RCTs (Hill et al., 2017). This implicates that
differences in the formed groups based on the extreme scores become progressively smaller thus
affecting the validity of the experiment.
Experimental mortality is also a threat to the internal validity of the research, which
defines the differential dropping out of the experiment by some participants from the groups of
comparison in the course of the trial. This affects results in differences between the groups since
the subjects that dropped could be different from those completing the study thus inflating and
obscuring the treatment effects of interest causing bias.
Selection is a threat to internal validity where bias could arise in assigning participants to
the groups that lead to systematic indifferences among the participants in each group. However,
the differences existing within a group are exposed to the experimental treatment.
History is regarded as another threat to internal validity whereby the extraneous events
that occur in the course of the experiment which might affect the responses on the dependent
measure such as the practices occurring in the society like social upheaval or minor events
during the investigations such as malfunctioning of the equipment. Halperin, Pyne, and Martin

PROGRAMME EVALUATION 9
(2015) acknowledged that these events, in essence, history accounting for the events, could affect
the internal validity of an experiment.
Question Four: A summative evaluation for the MBI in Singapore
Research method
The approach of classic experimental design would be used with a RCT where a control
group and an experimental group would be used to examine different variables and the
corresponding differences in the outcomes as shown below.
Figure 2: A RCT flow diagram
Target respondents
The target respondent would be the primary and the secondary students including both
male and female from all the races that would be in the study. Additionally, the participants
would be from both regular and special education schools.
Sampling and recruitment method

PROGRAMME EVALUATION 10
Random sampling would be conducted to recruit the respondents in the study. Three
districts in Singapore with be selected randomly where the participants would be sampled (Sen
and Singer, 2017). The researcher would shortlist all the special and regular secondary and
primary schools upon which three students would be randomly selected for the participation of
voluntary basis who would be exposed to the designed inclusion and exclusion criteria.
Data collection procedures
Data collection would be categorised into two, for the control group and the experimental
group. For instance, qualitative data would be collected through semi-structured interviews for
the participants under the MBIs strategies such as Mindfulness-Based Stress Reduction within a
specified period and that of the counterpart group for comparison of data (Khoury, Sharma,
Rush, and Fournier, 2015).

PROGRAMME EVALUATION 11
References
Ajimsha, M. S., & Shenoy, P. D. (2019). Improving the quality of myofascial release research–A
critical appraisal of systematic reviews. Journal of Bodywork and Movement Therapies,
6(9), 40-67.
Center for Disease Control and Prevention . (2013, May 17). Mental Health Surveillance Among
Children — United States, 2005–2011. Retrieved from Center for Disease Control and
Prevention: https://www.cdc.gov/mmwr/preview/mmwrhtml/su6202a1.html
Cotton, S., Luberto, C. M., Sears, R. W., Strawn, J. R., Stahl, L., Wasson, R. S., & Delbello, M.
P. (2016). Mindfulness‐based cognitive therapy for youth with anxiety disorders at risk
for bipolar disorder: a pilot trial. Early Intervention in Psychiatry, 10(5), 426-434.
Dunning, D., Griffiths, K., Kuyken, W., Crane, C., Foulkes, L., Parker, J., & Dalgleish, T.
(2019). The Effects of Mindfulness-Based Interventions on Cognition and Mental Health
in Children and Adolescents: A Meta-Analysis of Randomised Controlled Trials.
Flannelly, K., & Jankowski, K. R. (2018). Threats to the Internal Validity of Experimental and
Quasi-Experimental Research in Healthcare. Journal of health care chaplaincy, 24(3),
107-130.
Goodwin, C. J., & Goodwin, K. A. (2016). Research in psychology methods and design. New
Jersey: John Wiley & Sons.
Halperin, I., Pyne, D. B., & Martin, D. T. (2015). Threats to internal validity in exercise science:
a review of overlooked confounding variables. International Journal of Sports
Physiology and Performance, 10(7), 823-829.

PROGRAMME EVALUATION 12
Hill, R. C., Fomby, T. B., Escanciano, J. C., Hillebrand, E., & Jeliazkov, I. (2017). Regression
discontinuity designs: Theory and applications. Bingley: Emerald Group Publishing.
Jiga, K., Kaunhoven, R. J., & Dorjee, D. (2019). Feasibility and Efficacy of an Adapted
Mindfulness-Based Intervention (MBI) in Areas of Socioeconomic Deprivation
(SED). Mindfulness, 10(2), 325-338.
Khoury, B., Sharma, M., Rush, S. E., & Fournier, C. (2015). Mindfulness-based stress reduction
for healthy individuals: A meta-analysis. Journal of psychosomatic research, 78(6), 519-
528.
Maynard, B. R., Solis, M. R., Miller, V. L., & Brendel, K. E. (2017). Mindfulness-Based
Interventions for Improving Cognition, Academic Achievement, Behavior, and Socio-
emotional Functioning of Primary and Secondary School Students. Campbell Systematic
Reviews 2017: 5. Campbell Collaboration.
Raj, A., & Kumar, P. (2019). Efficacy of Mindfulness Based Stress Reduction (MBSR): A Brief
Overview. Journal of Disability Management and Rehabilitation, 4(1), 73-81.
Riley, R. D., Moons, K. G., Snell, K. I., Ensor, J., Hooft, L., Altman, D. G., ... & Debray, T. P.
(2019). A guide to systematic review and meta-analysis of prognostic factor
studies. bmj, 364, k4597.
Semple, R., & Burke, C. (2019). State of the research: Physical and mental health benefits of
mindfulness‐based interventions for children and adolescents. OBM Integrative and
Complementary Medicine, 4(1), 31.
Sen, P. K., & Singer, J. M. (2017). Large Sample Methods in Statistics (1994): An Introduction
with Applications. Florida: CRC Press.

PROGRAMME EVALUATION 13
Thomas, D. G., Apps, J. N., Hoffmann, R. G., McCrea, M., & Hammeke, T. (2015). Benefits of
strict rest after acute concussion: a randomized controlled trial. Pediatrics, 135(2), 213-
223.