Policy Research and Evaluation

This study material provides an in-depth understanding of policy research and evaluation. It covers the purpose of policy evaluation, the conceptual approach to evaluation, evaluation questions, data gathering methods, and strategies to enhance the use of evaluation findings. The content is relevant for students studying policy research and evaluation in various subjects and courses.

Running head: POLICY RESEARCH AND EVALUATION
POLICY RESEARCH AND EVALUATION
Name of the Student
Name of the university
Author’s note

1POLICY RESEARCH AND EVALUATION
V. Purpose of the policy evaluation
The policy evaluation is mainly done to look at the reason for which the policy has been
adopted in the first place and to determine whether the policy planning is a success or a failure. If
the policy is a success, then it will always be supervised and appraised as the legal environment.
A policy which has failed should keep in to consideration, whether or not the foundation of the
policy was based on correct, incorrect or unanticipated data. If the data used for the creation of
the policies in the first place is improper then a well-planned program is also bound to fail. An
environmental change in workplace also lead to a failure of the policy in a short period of time.
The rationale behind this policy is the high smoking rate in the society that has led to
poor health among the young people and the pregnant women. It has been found that the
smoking rates have dramatically increased with time, 8% of the people still smoke risking a
lifetime of health. About 10 % of the women still smoke, with all the attendant risk of
miscarriage, still births and neonatal complexities (International Agency for Research on Cancer,
& World Health Organization, 2008). Although the government’s tobacco control plan for
England had set out an ambition for achieving a generation that is smoke free, meaning a
smoking prevalence of 5 % of the population, However, the evidence is quite clear that for this to
be delivered needs comprehensive and integrated government and public sector action for
funding and ensuring an effective tobacco control. There had been positive proposals in the
tobacco control plan, but the public health funding was not sufficient enough for supporting the
plan, and hence the support has declined within a year (International Agency for Research on
Cancer, & World Health Organization, 2008). The over reliant plan became so unrealistic during
the time of budget restraints in the local authorities. The commitment in the long term plan of the
NHS is for the funding to help the smokers being treated in the NHS, but this is too only a part of

2POLICY RESEARCH AND EVALUATION
he solution and not only one solution. It has already been pointed out that the public spending
would become constrained due to the fact and new ways have to be found in order to increase the
funds (International Agency for Research on Cancer, & World Health Organization, 2008). The
plan should therefore concentrate on what can be done by the central government for supporting
the delivery of a smoke free future throughout the communities.
United Kingdom had been trying their best to reduce the crime related to smoking. As
claimed by the government of United Kingdom, the smoking prevalence has reduce substantially
with the previous smoking plan, but as per the sources, there had been not much progress. It is
due to the failure of the plan and the increased rate of the smoking among the adolescent group
and pregnant women provides a strong rational for the evaluation of the policy.
VI. Description of the conceptual approach to the taken for the policy evaluation and its
appropriateness
When the policies are not implemented as per the anticipated aims and the objectives, a
common policy response is the introduction of the reforms aiming at the control of the tobacco
installation plan. I such a condition, evaluation might involve a top down approach that is
principally concerned for promoting responsibility and compliance with the implementation
targets. System methodology can be an appropriate way for the evaluation of the policy (Caffrey
& Munro, 2017).
The systems approach conceptual framework consists of a broad church that spans a
multiple discipline such that General systems theory that has been prepared by the biologist, von
Bertalanffy, Cybernetics, pioneered by Ashby, a psychiatrist and Bateson, who is an
anthropologist; System Dynamics, developed by Forrester, a computer engineer. Thee diverse

3POLICY RESEARCH AND EVALUATION
school are collaborated and united by their focus of understandings. It is shown how the different
factors are connected to each other in a system. It should be mentioned that they are also unified
by the idea of emergence as a whole, which states that connections between parts of a system can
generate developing properties which cannot be comprehended by investigating each of the parts
in isolation (World Health Organization, 2013). A systems approach hence emphasizes on the
functional abstraction instead of structural decomposition. The systems application in the sector
of assessment has emphasized on the general ideas of interrelations, boundaries and perspectives.
In the context of the evaluation, the concepts actually associate with the understanding of the
inter relationships linked with the situation. It also involves engagement with contrasting
perspective related to a condition and reflection on the limitations of such a representation and
the interactions (Weimer & Vining, 2017).
A systems approach in the evaluation of the policy is of use as they appoint authors from
the other schools of thought. However, there had been arguments over the fact that the systems
approach is “too wordy” and abstracted. As argued by Caffrey & Munro, (2017), a systems
approach fits well with analysis at the micro level. It has also been claimed that the systems
approaches can result in risky attribution of all the accountability to the system. It is necessary to
determine how the interventions had been working and diagnosing a problem that either prevent
them from working in the desired way . In order to do this, one should consider the totality of the
effects both incidental and intentional. For an effective evaluation, we need to consider what
effects might occur, design studies, allowing the detection of the effects in the variable of
interests and making of valid casual inferences regarding the contribution of the interventions.
An evaluation should start with a theoretical evaluation of how an intervention should
actually work. Often there might be only one clear theoretical mechanism that is generally

4POLICY RESEARCH AND EVALUATION
provided as a part of justification for the construction of the policy, whereas sometimes there are
alternative modes of effects that can be postulated (Caffrey & Munro, 2017). This occurs when
the head of power whom the policy has been constructed in limited. Hence, the tobacco control
plan meant for the smokers from the exposure to passive smoking cannot consider the possible
benefits of smoking free zones for the reduction of cigarette consumption. A good evaluation
policy for the plan should be able to consider all the other outcomes and not just the ones used to
provide any legal basis of the policy.
Again evaluation requires a proper explanation of the policies, the problems, the context and how
they had been changing. The goal here would be to provide a population estimate of what people
think and do, by focusing on the main outcomes. Data has to be collected from these four
domains-
1) People who uses tobacco, type used, amount, when and where do they use them, beliefs and
attitudes of both the ex-smokers and the non-smokers 2) The behavior and the attitude of the
Tobacco industry, including the features of their product 3) The various tobacco control
activities to which people are exposed to and 4) Aspects of the broader ambience which can
affect the tobacco use or the tobacco harm outcomes (like some cultural regulations
regarding alcohol consumption or smoking.
VII. List of the policy evaluation questions to be answered
Key evaluation questions are similar to that of the research questions developed in
academic research projects, guide the tools and the methods for understanding the issues under
examination. Evaluation question might seem to be instinctive and thus can be rapidly developed

5POLICY RESEARCH AND EVALUATION
for designing a more detailed program planning. Systematically, evaluation questions focus on
varied assessment types.
Needs assessment- Identification of the associated social conditions and an emerging need for a
program. The questions identifying and supporting problems that the interventions assess. If the
stakeholder’s communities are not identifiable by the stakeholders, then the plan/project will not
be fruitful. The Evaluation questionnaire should focus on the implication of the problems,
drawing on literary evidences, previous assessments and the baseline data. Some of the sample
questions that can be used for the evaluation are as follows-
 What are the characteristics of the target population?
 What are the needs of the target population?
 What percentage of the population smokes?
 Does the data collection method employed, were free from bias and truly represented the
target population?
 Did the policy mention about the health care and cost burden of the illness?
 Did the policy plan mention about the important areas like smoke free pregnancy?
 What are the fundamental conventions of the project / program?
 Are the objectives addressed? If so, how and if not, why not?
 Are there any other population that the program should be involving?
 Is the target audience adequately approached and involved in the activities?
 Did the target population communicate proactively in the program?
 What are the objectives, outcomes and goals of the project?
 Is the plan beneficial for the target population?
 Do the plan have any negative effects?

6POLICY RESEARCH AND EVALUATION
 Has the plan alleviated the problem addressed?
VII. Description of the evaluation participants
A participatory evaluation is required for the evaluation of the policy, where the
stakeholders dynamically involve in the development of the evaluation and all the phases of its
implementation. Those having the most stake in this plan such as the program beneficiaries, the
partners, the government, the policy makers and public. In this tobacco control plan, some of the
important stakeholders are Public health England. Participation takes place throughout the entire
evaluation process (Zukoski & Luluquisen, 2012). The evaluation process incudes identification
of the relevant questions, planning of the evaluation design, selection of appropriate measures
and data collection methods, gathering and analyzing the data. Finally, reaching the consensus
about the findings conclusions and the recommendations, dissemination of the results and
preparation of an action plan for the improvement of the program performance (Zukoski &
Luluquisen, 2012). A participatory evaluation ensures that the evaluation focuses on some
questions that are locally relevant, meeting the needs of the policy planners. A participatory the
evaluation is normally the effective and action, and provides the stakeholders such as the
beneficiaries with the chance to reflect on the processes of the project and develop knowledge
resulting facilitating the application of the lessons that has been learned (Zukoski & Luluquisen,
2012).
The evidences to be used in the evaluation of the participants are the use of observational
evidences, the views of the policy makers and the epidemiologists, results of the needs
assessment. A participatory evaluation build teams and participants commitment via

7POLICY RESEARCH AND EVALUATION
collaborative injury. Inviting a wide number of stakeholders for participating and leading
different parts of the process can form local leadership skills.
IX. Data gathering method used
Data has to be collected to understand the percentage of male and female indulging in
smoking, the age of onset of smoking, the reason behind the smoking, the frequency if smoking.
High quality of data collection like regular cross sectional surveys are important to describe the
nature of the problem. Cross sectional surveys has been found to be much helpful in
understanding the trends in the tobacco and the key determinants of using them (Vedung, 2017).
The tobacco industries might get motivated to moderate the effects of the policy, hence it is
important to conduct a continuous surveillance of possible counteractions to the policies.
Again, surveys are conducted in order to evaluate the impact of the policy plan on the
common people and the others, to check whether everything has been conducted as per the
planning made. Interviews and questionnaires helps further help the stakeholders to identify the
gaps and the loops still left in the policy (APPG, 2013). Rutter, (2012) have stated several
objectives of data quality control. For controlling the data, it should accurately reflect the reality
of the population. It should be he representative of the whole target population. For an effective
data quality control, it is necessary that the policy makers and the analysts make real time
decisions. It is also necessary to control the sampling and non-sampling error. Biases like
measurement errors, selection bias and non-response can be used in case of sampling and non-
sampling errors (Dunn, 2015).
Another crucial step is the total survey design (TSD). It is necessary to develop a suitable
sample design, like the sample size and the geographical distribution. The sampling weights and

8POLICY RESEARCH AND EVALUATION
the sampling errors needs to be calculated (APPG, 2013). It is necessary that reliable and valid
instruments are used .While using a questionnaire for the survey, it is necessary that often open
ended questions are used to combat the frequencies of undesirable behavior. Long questions, but
short instruments has to be used. The data can be captured by using Self-administered
questionnaire, Telephone, Paper and pencil (PAPI), computer assisted personal interview (CAPI)
or audio computer assisted self-interview (ACASI) (APPG, 2013). There are five outcomes that
is related to the individuals: improvement in the knowledge, changing the attitude and the
normative beliefs, alterations in the behavior patterns, changes in the exposure, health outcomes
to be detected soon after the implementation of the policy.
X. Description of the strategies to enhance the use of the evaluation findings
As pet the findings of the evaluation a written report is the prime output of the process of
evaluation. The draft evaluation report had to be structured logically and should be backed up by
evidence based findings, conclusions, lessons got and suitable recommendations (O'Donoughue
Jenkins et al., 2016). A proper draft evaluation report should be able to describe what is being
evaluated and why, explain the steps and the procedures used for answering the questions
(O'Donoughue Jenkins et al., 2016). Some of the important strategies that can be used for a better
tobacco control plan are as follows-
 Getting a valid percentage and demography of the smokers.
 Identifying whether proper participatory engagement of stakeholders are going on
 The findings has to be presented clearly and fairly
 All the findings has to be arranged in an order understandable to the decision makers and the
stakeholders. Any commendations should be clearly mentioned along with the findings.

9POLICY RESEARCH AND EVALUATION
 The findings and the commendations should preferably be represented by using a framework
indicating towards their relative importance.
 All the assumptions has to be explicitly acknowledged (O'Donoughue Jenkins et al., 2016)
 Any weaknesses and limitations caused by the limitations on time resources, accessibility of
the data should be described.
 A comprehensive explanation of how the findings were got has to be mentioned
 Persons or organizations, who have contributed towards the evaluation of the program should
receive feedbacks as per the needs, Any disclosure should follow the proprietary and legal
findings
 Officials who are certified to give the evaluation data has to be specified.
 The database and the necessary credentials has to be arranged in a way contained with the
availability procedures and the policies.
 Although the utilization of the findings cannot be guaranteed but the authorities should be
able to try this in the next possible manner such that in benefits the stakeholders (Ellen et al.,
2014). The evaluation results has to be provided to the appropriate users before taking
relevant decisions (APPG, 2013). It the duty of the evaluators to anticipate and prevent any
kinds of misinterpretations of the evaluative information. The evaluators should also try to
bring it to the focus of the decision makers and other relevant the audiences (O'Donoughue
Jenkins et al., 2016). An evaluation should be clearly be able to distinguish between the
conclusions of the evaluation and any sort of recommendations. An evaluation should be able
to consider the probable cost of the recommended course of actions (Lawrence & Weber,
2014).

10POLICY RESEARCH AND EVALUATION
XI. Discussion of the standards to be used to guide the design and conduct the evaluation
The design meant for an evaluation is not conceived out of nothing. It should be
mentioned that it is always influenced by ethical, logistical, administrative and fiscal
apprehensions. The design should take these in to account as a methodological requirement in to
the account. Schemes will differ in rigor and not all the dimensions are equally objective (Khan
& Rahman, 2017). Some of the standards that the evaluators should consider are- 1) The design
should be justifiable and suitable to the ways conclusions are to be drawn. 2) For the impact
studies, the problem for the evaluation design for estimation of the impact of the non-treatment
and selection of a particular method has to be properly justified and described (Khan &
Rahman, 2017). 3) In case the sample has to be used, the sampling method has to be detailed ,
choice of unit, modes of election and the timeframe has to be justified, on the basis of the explicit
scrutiny of the requirement for the evaluation, including the generalization of the results. 4) All
the methods and the measurements should be described and explained in a justifiable way (Khan
& Rahman, 2017). 5) All the necessary cooperation of the program staffs, the policy makers, the
associates of the community and those who are directly involved in the evaluation program has
to be planned from the very beginning.
XII. Description of the time line of the policy evaluation and budget
It is necessary the fix an amount of budget from the project resources. The official’s
should ensure that sufficient budget exists for the implementation of the relevant monitoring that
has been indicated in the original project proposal (Rutter, 2012). Funds has to be reserved for
setting up and monitoring the evaluation system including the internal and the self-evaluations. A
minimum of 5 % of the total budget has to be kept for the evaluation of the program (Lawrence

11POLICY RESEARCH AND EVALUATION
& Weber, 2014). The cost would mainly include conducting of the survey, printing of the
questionnaire, engaging the stakeholders, transportation cost. Cost has also been associated with
the recruitment of external evaluators.
A time frame of five months will be taken to evaluate the tobacco control policy that
would involve inviting and the engaging the stakeholders, conduction of a proper needs
assessment, finding out the prevalence of the issue- statistical data, data collection, data analysis,
interpretation of the findings, and dissemination of the findings and the implementation of the
findings (Lavallee et al., 2017).
Time frame
1st month 2nd month 3rd month 4th month 5th
Needs
assessment
Association of
the stakeholders
Formulation
and negotiation
Preparation and
collection of
data
Analysis and
interpretation
Dissemination

12POLICY RESEARCH AND EVALUATION

13POLICY RESEARCH AND EVALUATION
References
APPG, (2013). All Party Parliamentary Group on Smoking and Health.
Caffrey, L., & Munro, E. (2017). A systems approach to policy evaluation. Evaluation, 23(4),
463-478.
Dunn, W. N. (2015). Public policy analysis. Routledge.
Ellen, M. E., Léon, G., Bouchard, G., Ouimet, M., Grimshaw, J. M., & Lavis, J. N. (2014).
Barriers, facilitators and views about next steps to implementing supports for evidence-
informed decision-making in health systems: a qualitative study. Implementation Science,
9(1), 179.
Fischer, F., & Miller, G. J. (Eds.). (2017). Handbook of public policy analysis: theory,
politics, and methods. Routledge.
Fischer, F., & Miller, G. J. (Eds.). (2017). Handbook of public policy analysis: theory, politics,
and methods. Routledge.
International Agency for Research on Cancer, & World Health Organization. (2008). Methods
for evaluating tobacco control policies. Methods for evaluating tobacco control policies.
Khan, A. R., & Rahman, M. M. (2017). THE ROLE OF EVALUATION AT THE STAGES OF
POLICY FORMULATION, IMPLEMENTATION, AND IMPACT ASSESSMENT.
Agathos, 8(1), 173.

14POLICY RESEARCH AND EVALUATION
Lavallee, D. C., Williams, C. J., Tambor, E. S., & Deverka, P. A. (2012). Stakeholder
engagement in comparative effectiveness research: how will we measure success?.
Journal of comparative effectiveness research, 1(5), 397-407.
Lawrence, A. T., & Weber, J. (2014). Business and society: Stakeholders, ethics, public policy.
Tata McGraw-Hill Education.
Nabatchi, T., Gastil, J., Leighninger, M., & Weiksner, G. M. (Eds.). (2012). Democracy in
motion: Evaluating the practice and impact of deliberative civic engagement. Oxford
University Press.
O'Donoughue Jenkins, L., Kelly, P. M., Cherbuin, N., & Anstey, K. J. (2016). Evaluating and
Using Observational Evidence: The Contrasting Views of Policy Makers and
Epidemiologists. Frontiers in public health, 4, 267. doi:10.3389/fpubh.2016.00267
Rutter, J. (2012). Evidence and evaluation in policy making. London: Institute for Government.
Vedung, E. (2017). Public policy and program evaluation. Routledge.
Weimer, D. L., & Vining, A. R. (2017). Policy analysis: Concepts and practice. Routledge.
World Health Organization. (2013). WHO Evaluation practice handbook. World Health
Organization.
Zukoski, A., & Luluquisen, M. (2012). Participatory evaluation. What is it? Why do it? What are
the challenges?. Community-based public health policy & practice, (5), 1-6.