Program Evaluation: A Critique of Two Evaluation Reports

This report critiques two program evaluation reports and compares the methodology used to comment on transparency of reporting and quality of claims given. The evaluation reports focus on blended learning approach for teacher training program and community-based intervention program for left-behind children in rural China.

Running head: PROGRAM EVALUATION
Program evaluation
Name of the student:
Name of the University:
Author’s note

1PROGRAM EVALUATION
Introduction:
Program evaluation is an approach to understand the effectiveness of an implemented
program by establishing criteria of work, constructing appropriate standards, measuring
performance and integrating evidence to make judgment on the program. This report aims to
critique two evaluation report by Kocoglu, Ozek and Kesli (2011) and Zhao et al. (2017) and
compare the methodology used to comment on transparency of reporting and quality of claims
given.
Paper 1 (Kocoglu, Ozek and
Kesli 2011)
Paper 2 (Zhao et al. 2017)
Clarity of aim and focus The paper has a clear aim
which is to evaluate impact of
a blended learning approach
for teacher training program
on effectiveness of English
language teaching in a face to
face MA program
The aim of the article has been
to explore the feasibility,
acceptance, preliminary
outcomes and potential
sustainability of a community
based intervention program to
offer care and support to the
children. Hence, the aim is
clear and detailed.
Clarity in PE approach The clarity in PE approach is
understood from the fact
focus was on identifying the
type of instruction which
would yield higher content
The PE approach in this case
had been Context- Mechanism
Outcome (CMO)
configuration using the realist
evaluation methodology,

2PROGRAM EVALUATION
learning in a MA program which was clearly mentioned
and explained.
Rationalisation of PE
approach
The rationalisation of the
approach has been provided in
the paper by giving evidence
regarding the benefits of the
blended learning in different
educational setting.
The authors have stated that
the realist evaluation approach
explores the different program
inputs that can be presented to
reconfigure or initiate the
mechanisms in a different
manner, to facilitate
transformation.
Acknowledgment of
stakeholders
Although teachers and English
foreign language students are
the main research participants,
however no detail has been
given regarding other
important stakeholders of the
program.
Community stakeholders, the
children and the primary
caregivers were identified and
their roles and contribution
have been acknowledged.
Research questions The research question is clear
and understood from the aim
of the paper
To explore the efficiency and
feasibility of the Children’s
Club program to enhance the
emotional and behavioral
wellbeing of the children,
along with enhancing the

3PROGRAM EVALUATION
community support networks
for these children
Consideration of Program
theory
Theory of action has been
applied by identifying linkage
between different components
of implementation and
outcome of the program
The program theory had been
incorporated in the program
implementation and evaluation
as well
Use of logic frameworks No logic framework used. No logic framework applied
Program implementation
details
Internet based program was
used as a method of
implementation
Implementation mechanisms
for the program involved
assimilating existing
resources, engaging native
volunteers, and providing
various Club activities.
Evidence backing for claims Reliability of the study is
understood from discussion on
consistency of outcome with
other research study
Very little evidence cited to
support the claims made by
the project
The two papers presented over here used very different programs and areas of focus to frame a
program evaluation report. While Kocoglu, Ozek and Kesli (2011) used blended learning
approach to understand its impact on effectiveness of teaching English language, Zhao et al.
(2017) focused on investigation of the impact of ‘Children’s Club’ on care and support for

4PROGRAM EVALUATION
children who lag behind. Hence, the context and problem for each program was different. The
critique for each paper is done separately to understand strength and weakness in each report.
Kocoglu, Ozek and Kesli (2011) had clear aim and purpose which is accurately written
both in abstract and introduction part of the report. Furthermore, to proceed with a program
evaluation, the first important component is to establish criteria and characteristics of the
program. The author has discussed in detail about the characteristics of the program by first
giving a short overview of the benefits of the blended learning approach in teacher education and
then giving details on the program in the methodology part. Blended learning is an approach
where e-learning and traditional learning environment is combined to create a new learning
environment (Kiviniemi, 2014).Kocoglu, Ozek and Kesli (2011) implemented blending learning
approach by providing internet based English language teaching for each class in Anadolu
University. The internet program was designed in such a way that interactive and student-
centered session could take place during the learning process.
Another critical aspect of program evaluation reporting is to develop appropriate logical
framework that defines the planning process and vital components of the program. The
advantage of using a logical framework is that it establishes clarity regarding the nature and
function of the program. However, no specific framework was used to define the components of
the program and the planning process of the program. This could be interpreted only by reading
the whole section of the methodology part. However, the research design is the main strength of
the study as all strategies has been taken to increase the alignment of the process to research aim.
This is understood from the fact that proper blended learning program was developed and use of
teaching knowledge test, final course grades and focus group interviews provided accurate
analytical variable to understand the efficacy of the program as per research aim. By the use of

5PROGRAM EVALUATION
this approach, both qualitative and quantitative data related to the program was obtained. The
reliability of the data collected is also understood from the fact that recognized test was used to
assess English language competency. Furthermore, use of open-ended question during the focus
group interviews favour getting detailed ideas regarding the project. Open ended question
enhance the reliability of research data (Bryman, 2017).
The analysis of the study findings from paper 1 revealed that participants were satisfied
with the program and course objective was achieved. The suitability of the claim is understood
from low risk of bias in the data collection method. This is because only self-rate data were not
taken instead objective data was also used to increase the credibility of the work. Good
comparison was done with past research paper and consistent outcomes were achieved in that
respect too (Cope, 2014, January).
The article by Zhao et al. (2017) is the second article chosen for this review report. The
author in this research report have emphasized on the care of the left behind children in the
China and addressing the issues that they are facing, it has to be mentioned that there is mounting
evidence that suggests that large number of left behind children or LBC in many countries grow
up encountering a prolonged period of separation from their worker parents which has a
significant impact on their psychological development, these children are extremely vulnerable
to various psychological and developmental issues and there is a gaping lack of enough research
on interventions that can be applied to these children. The primary aim or objective of this
research study is on exploring the acceptance, feasibility, introductory outcomes and possible
sustainability of the community based intervention program designed to provide care and support
to the children. Hence, it has to be mentioned that the aims and objectives had been very clearly
identified in the paper by the authors (Zhao et al., 2017).

6PROGRAM EVALUATION
The program chosen for the article had been the establishment of a children’s club
involving the community interventions, play and education activities for the left behind children
in the rural china. In this case, the program evaluation technique utilized had been the Context-
Mechanism Outcome (CMO) configuration technique taking the aid of the realist evaluation
theory (Zhao et al., 2017). As discussed by Lindsay, Howard and Owen (2015), the realist
evaluation is based the theoretical methods that are resulting from the social sciences that proves
the distinctive explanation of different programs and how they can function. This when
combined with the reinvented program theory focuses on the critical realism coined by Pawson
and Tiley. Hence, it can be mentioned that the program evaluation approach is a detailed social
approach and will help in identifying the different aspects of the children club program
identified. With the accompanying programs theory, it has also has potential to explore the exact
benefits and pitfalls of the intervention program chosen (Dalkin et al., 2015). However,
considering the rationalism of the approach chosen, the authors have stated that the realist
evaluation approach explores the different program inputs that can be presented to reconfigure
the mechanisms differently, to bring about transformation.
The stakeholder acknowledgment has been carried out effectively by the authors,
community stakeholders, the children and the primary caregivers were identified and their roles
and contribution have been acknowledged; which is commendable. Although, there is no clear
mention of the use of a logistic framework in the article, which can be considered a notable
limitation or con (Punton, Vogel & Lloyd, 2016). The research question had been very clear, “To
explore the efficiency and feasibility of the Children’s Club program to enhance the emotional
and behavioral wellbeing of the children, along with enhancing the community support networks
for these children?”, which is linked to the data findings and the discussion and was efficiently

7PROGRAM EVALUATION
summarized in the conclusion. Considering program implementation mechanisms, the authors
mentioned assimilating the existing resources, engaging native volunteers, and supplying
different Club activities. Lastly, there had been very little evidence backing the claims made by
the authors. Another limitation is the lack of any mentioning reliability or the validity of the data
and transparency of the report (Zhao et al., 2017)
References:
Bryman, A. (2017). Quantitative and qualitative research: further reflections on their integration.
In Mixing methods: Qualitative and quantitative research (pp. 57-78). Routledge.
Cope, D. G. (2014, January). Methods and meanings: credibility and trustworthiness of
qualitative research. In Oncology nursing forum (Vol. 41, No. 1).
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a
mechanism? Development of a key concept in realist evaluation. Implementation
Science, 10(1), 49.
Kiviniemi, M. T. (2014). Effects of a blended learning approach on student outcomes in a
graduate-level public health course. BMC medical education, 14(1), 47.
Kocoglu, Z., Ozek, Y., & Kesli, Y. (2011). Blended learning: Investigating its potential in an
English language teacher training program. Australasian Journal of Educational
Technology, 27(7).

8PROGRAM EVALUATION
Linsley, P., Howard, D., & Owen, S. (2015). The construction of context-mechanisms-outcomes
in realistic evaluation. Nurse Researcher (2014+), 22(3), 28.
Punton, M., Vogel, I., & Lloyd, R. (2016). Reflections from a realist evaluation in progress:
scaling ladders and stitching theory.
Zhao, C., Zhou, X., Wang, F., Jiang, M., & Hesketh, T. (2017). Care for left-behind children in
rural China: a realist evaluation of a community-based intervention. Children and Youth
Services Review, 82, 239-245.