Critical Analysis of an Evaluation Report: Australia-Myanmar Aid

This report provides a critical analysis of an evaluation report concerning Australia's humanitarian assistance to Myanmar. The evaluation, conducted by a team including members from the Humanitarian Advisory Group and DFAT, aimed to assess the aid's effectiveness, relevance, and efficiency. The report's introduction, while stating the purpose, lacked a coherent flow. The methodology employed qualitative methods, including desk reviews, interviews, and focus groups, but suffered from flaws such as purposive sampling bias and missing data. The report's findings were inconsistent and lacked sufficient data to support the conclusions and recommendations. The report analyzes the involvement of NGOs and other organizations and evaluates the actions and achievements of the humanitarian aid. The analysis suggests the report's organization was flawed, with data mixed within the findings section, disrupting the logical flow of information. The report evaluates the actions of the humanitarian aid by focusing on the thematic areas and the range of the sectors funded.

Running head: A CRITICAL ANALYSIS OF AN EVALUATION REPORT 1
A Critical Analysis of Australia’s Humanitarian Assistance to Myanmar Evaluation Report
Student’s Name
Institution

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 2
A critical analysis of an evaluation report
Purposes of and rationale for the evaluation report
This report is about an evaluation of Australia’s humanitarian support to Myanmar. The
evaluation was done by a team that comprises individuals from the Humanitarian Advisory
Group, Australian Department of Foreign Affairs and Trade (DFAT), DFAT’s gender and social
development strategist from South East Asia, and NGO’s and Partnerships Division. The purpose
of the evaluation report was to determine whether the humanitarian aid given by the Australian
government to Myanmar is effective, relevant, and efficient in the framework of DFAT’s
priorities and strategies. The rationale for the evaluation was to provide comprehensive findings
and recommendations to Australia’s Department of Foreign Affairs and Trade.
Indeed, the title of this report precisely stated the subject of the paper. The authors
endeavored to explicitly evaluate Australia’s humanitarian assistance to Myanmar throughout the
paper by use of different effective methodologies. The introduction of the report is very explicit
and gives adequate details on the purposes of the evaluation, gives a background information on
why the evaluation was to be done, and explains why the evaluation was conducted specifically
to assist towards protracted areas and not to areas affected by other natural disasters such as
flooding (Thomas, Boucher & Pulliam, 2015).
On the other hand, in the course of writing the introduction of the paper, the authors did
not write the statements in a sequential manner. In other words, the information in the
introductory part of the report did not lead coherently to the objective of the study. Instead of
placing the research done within a context, the authors decided to report on a research that was
not done. They reported on South East and Thai-Myanmar boarder which was not part of the

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 3
evaluation. Introductory part of a report should be very precise and only give specific details on
the purpose of the evaluation, the reason why the research was conducted in that specific way,
and the importance of the research. (Cals & Kotz, 2013).
Methodology used in the evaluation
Due to the fieldwork nature of the report, the research methods employed in the research
were qualitative. The qualitative methods used were desk reviews, key informant interviews and
focus group discussions. According to Donovan (2018), this kind of research method was very
important in this evaluation in that the feelings and the emotions of the respondents were able to
be expressed, first hand data was obtained and analysis of all relevant documents including
reports and proposals were done.
During the compilation of the report, the authors summarized a well and detailed process
undertaken in conducting the research. An explanation of the materials used such as reports, what
procedures were done and how they were done, were well documented. The sample size of each
qualitative method of was equally documented properly. However, while reporting concerning
the focus groups, essential information was left out. The report did not state or name the six
communities that the people consulted came from assuming there may be more than six
communities since the total number of communities in both Kachin and Rakhine states was not
written in the report as well. Therefore, because of the above reasons, this particular research
methodology of focus groups cannot be duplicated and same results achieved (Kotz & Cals,
2013).
All the methods used as indicated in the report had flaws. The purposive sample selection
was very inadequate since it could lead to a bias. The report states that only selected partner sites
could be visited. What was the rationale for selecting the partner sites? What about the other

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 4
partner sites? What informed the decision undertaken of not visiting them? Could it be because
of safety reasons, inaccessibility reasons? These are some of the questions that could arise in
cases of perceived biasness. The report indicates that the purposive sampling strategy was
employed due to scarcity of resources such as time constrains without stating satisfactory reasons
why only three months was allocated for the evaluation. Why couldn’t proper planning with the
relevant authorities be done to prevent such avoidable limitations? (Wilson, 2016).
Underlying values and principles
With refence to humanitarian values and principles, the report only indicates that the
Australian government complies with the humanitarian principles. According to OCHA, the
humanitarian guiding principles are four and they include; neutrality, humanity, impatience and
independence. The principles ensure establishment and maintenance of access to affected people
whether in a complex emergency or a natural disaster (Gordon & Donini, 2015).
Inasmuch as the recommendations that the Australian government should increase its
engagements with relevant ministries within the government of Myanmar, caution should be
taken since this is direct involvement with the political affairs of the country which is against the
humanitarian principles of independence. According to Van Engeland (2016), the principle of
independence requires that humanitarian activities be independent from the political activities
that may be showing any interest to areas where the humanitarian aid is being implemented.
The findings of the report indicated that the government of Myanmar representatives who
were engaged in the evaluation process were not well conversant with the humanitarian
assistance of Australia. Additionally, the report indicates that the government of Australia shares
both spending and funding information to the government of Myanmar and that funding to
national organizations are done indirectly through Myanmar Humanitarian Fund. Essentially,

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 5
sharing the information with the government is enough and so it is upon the government to
decide which relevant ministries need to be engaged with the humanitarian funds.
Therefore, the recommendation that the Australian government should increase
engagement with relevant ministries in the government in relation to the humanitarian response
provided is against the humanitarian principles and values (Büthe, Major & Souza, 2012).
Engagements with relevant ministries within the government of Myanmar should be left entirely
to the decision of Myanmar.
Data collected and its appropriateness for the evaluation
Cuschieri (2018), explains that in report writing, a result or data section should be
included in the research paper. In this particular evaluation report, results have not been given
specific focus. After outlining all the methods used in conducting the evaluation, there was need
to indicate the data collected (Cuschieri, 2018). The report does not give the data collected
during the desk reviews, the key informant interviews and the focus group discussions. What was
the essence of conducting the evaluation if data collected is not well documented? Additionally,
no statistical test was performed in the evaluation as there is no evidence of it indicated in the
report.
My attention was particularly drawn to some data contained in the finding section of the
report. This particular data provided very scanty information and was inappropriately placed.
How did the researchers come to the conclusion that protection and education were the sectors
that could form the focus of the Australian aid to the Myanmar? Where is the quantitative data to
support the conclusion? How many respondents supported protection and education to be the
main focus of Australian aid and how many were of the contrary opinion? What was the
percentage of both opinions? The information on the data collected was mixed up with the

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 6
findings section. Therefore, anybody who needs to read only the data collected section would be
disadvantaged (Pushparajah & Qureshi, 2013).
Due to the inconsistencies indicated above, there was little or no data collected and so no
evaluation can be done. The validity of the findings and the recommendations given in the report
may as well be questioned since no evidence of the same is documented (Rhodes, 2012).
NGOs and other organizations involved
According to the report, NGOs a and other organizations were involved in the evaluation.
Australian humanitarian aid works with implementing partners who are the NGO’s to achieve
their strategic goals. The report indicates that nine implementing partners work with Australia
and these include the World Food Program (WFP), International Community of the Red Cross
(ICRC), UN Refugee Council(UNHRC), Save the Children, Myanmar Humanitarian Fund
(managed by UNOCHA), United Nations Populations Fund (UNFPA), Norwegian Refugee
Council (NRC), Danish Refugee Council (DRC), and the Oxfam. Under the subsection of key
informant interviews, the report indicates that 21 representatives were interviewed. This was
probably a good number to enable the evaluation conducted to achieve substantial results.
However, the data or the responses of the implementing partners were not documented.
The data only appears insufficiently under the findings section. This report did not indicate any
responses by the partners in a logical manner, but the report only discussed in the findings
section what the partners had recommended. How do we understand the recommendation
without first understanding the challenges posed by the partners? Furthermore, the data does not
give responses obtained from the interviews of all the implementing partners assuming that all
the partners were involved in the key informant interviews (Hogue, 2012). Incase not all the

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 7
partners were interviewed, then it is only appropriate that the information be written down with
viable reasons to avoid any cases of bias misconception.
With reference to the above, it is possible to conclude that not all partners of the
humanitarian assistance provided by Australia were involved in the evaluation since no data is
available to substantiate it. The report would have also documented the fact that not all the
partners were involved in the evaluation in order to avoid any conclusions and recommendations
that may appear biased (Cook, Hart, Kinney & Oler, 2014).
The criteria used to evaluate actions and achievements
The report evaluated the actions of the humanitarian aid by focusing on the thematic
areas which involve the strategic focus, policy priority, and national and local leadership while
an overview of the range of sectors that are supported by the humanitarian program was used to
evaluate the achievements. The report showed all the achievements gained by the humanitarian
support across a variety of sectors which included; WASH, protection, food security, shelter,
education, livelihoods, and health. Numerical values and percentages were also used to
determine the exact proportion of individuals reached by the Australian Humanitarian Assistance
program (Knottnerus & Tugwell, 2013).
However, this data given may not be validated since there are no evidences to show it.
How was the conclusion arrived at? Where was the investigations done? Who or what provided
the values? The report does not indicate or discuss any available data to show that there were any
achievements to be documented. This leads us to question ourselves whether evaluation of the
achievements was really done.
With regards to the actions by the humanitarian assistance, the report clearly states the
actions in reference to each thematic area. This was a very well organization which was simple to

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 8
follow through. Inasmuch as the report was discussing the findings in each theme, there was a
mix up of information. Instead of the report focusing on discussing the data collected, the report
also included the data collected in this section. This interrupted the logical flow of the
discussions and was confusing. In other words, the information on the data collected needs to be
documented separately and appropriately, and discussions of the results documented coherently
without interruption of the flow (Saper, 2014).
With reference to the above, it is possible to conclude that the criteria for evaluating the
report was in line with the Australia’s Humanitarian Assistance themes and the range of the
sectors funded. However, the report was not well organized because the data collected only
appeared in the findings and discussion section and interrupted the logical flow of information.
The data collected required to be documented separately after the methodology section (Sen,
2014).
How evaluation approach differed from that suggested by the International Federation of
Red Cross and Red Crescent Societies framework and any concerns
According to the International Federation of Red Cross and Red Crescent Societies
framework, the evaluation reporting should follow the procedure outlined in the framework. The
reporting need to logically flow and be structured coherently. The data ought to be presented and
analyzed and interpreted systematically with evidence supporting the recommendations and
conclusions. The framework also explains that the evaluation reporting should explain the
methods used in the evaluation as well as the techniques used for data collection, analysis and
management. Any assumptions made in the course of the evaluation should also be included in
the report (Vande veegaete, Borra, De Buck & Vandekerckhove, 2016).

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 9
Some aspects in the approach undertaken to report the evaluation differed with the
International Federation of Red Cross and Red Crescent Societies framework. The presentation
of the data collected was not logically shown and no or minimum analysis was done. The data
presentation was mixed up with the findings sections and hence disrupted the logical flow of the
discussion. The report evaluation did not indicate any evidence used to support the conclusions
and the recommendations hence the validity and adequacy of the statements is questionable. The
approach also differed in that the requirement to explain all techniques used in data collection
and data analysis was not documented ("International Federation of Red Cross and Red Crescent
Societies", 2010).
Due to the differences in the approaches contained in the International Federation of Red
Cross and Red Crescent Societies framework, serious concerns are unavoidable. Of much
concern is the validity of the evaluation because there are not evidences and sources documented
to support the conclusions and the recommendations. The absence of the data collection
techniques also poses serious concerns as to whether the research was really undertaken.
Therefore, absence of evidences and resources, and data collection techniques may form a basis
to reject the paper as inadequate and invalid.
Evaluation of health concepts, issues and programs
According to the report, health is not a focus to Australia’s Humanitarian Assistance to
Myanmar. The report only indicates that Australia has supported 15 hospitals and has contributed
to reaching more than 180,000 individuals through mobile clinics. This was an applaudable
achievement by the government of Australia in response to protracted crisis. However, the report
does not indicate where the information was retrieved. There cannot be assumption that the
audience knows the source of the information if it has not been documented. Writing down a

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 10
concluding statement in a research with authenticity but without proper evidence is considered
null and void. The report also indicates that 12% of the funding was directed to the health sector.
How true is the figure? Where is the evidence? Where is the data collected concerning the
amount of funding channeled to the health sector? The report ought to have supported the figures
with proper evidence (Lee, Johnson, Newhouse & Warren, 2013).
Throughout the report, the issue of health has not been mentioned while the achievements
indicate a significant number of people reached by the mobile clinics and the hospitals supported
by the humanitarian aid. This in turn rises many questions and concerns and confusions. If
achievements of the humanitarian help were part of the evaluation, then proper documentation
and evidence of the same needed to be included in the report (Salmond, 2013).
The appropriateness of the findings based on evidence
Evidence is very crucial in writing a research or evaluation report. Evidence gives the
report authenticity and prevents any doubts concerning the conclusions drawn. In this report,
almost all the findings were not based on evidence. Right from the introduction section, the
report gives a background information of the humanitarian assistance, and the background of the
areas being evaluated. However, there is no evidence attached to both the background
information (Beech, 2014). How is the audience to be sure and verify whether that all the
information is valid?
Moving on to the findings, the information given was not based on any evidence. There
was no mention of data collected after carrying out the desk reviews, the interviews and the
focus group discussions. The report indicates the findings by showing the achievements of the
humanitarian funding without any supporting evidence. How does the audience verify the
validity of the achievements? Which documents were used to arrive at such conclusions? Is there

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 11
any written source collected to validate the relevance of the achievements? In such instances
where evidence to support a claim is absent, the report or the evaluation may be considered as
not true.
In the findings concerning the policy themes of strategic focus, geographical and sectoral
focus, and the national and local leadership focus, the report gives information too without any
evidence. The only evidence that report refers to is the interviews. How was the recommendation
arrived at? Has that recommendation ever been tested elsewhere and has it worked? Is there any
written document to support the findings? From the entire evaluation report, there is no evidence
noted down to support the arguments. The report has relied entirely on the desk reviews, key
informant interviews, and focus groups interviews. The evaluation needed to have validated all
their findings with scholarly materials as evidence as this would put more weight in the
recommendations. To conclude, the findings of the report are not appropriate since there is no
evidence shown to support the findings (Florczak, 2016).

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 12
References
Beech, B. (2014). Answering the Call. Family & Community Health, 37(2), 102-103. doi:
10.1097/fch.0000000000000023
Büthe, T., Major, S., & Souza, A. (2012). The Politics of Private Foreign Aid: Humanitarian
Principles, Economic Development Objectives, and Organizational Interests in NGO Private
Aid Allocation. International Organization, 66(04), 571-607. doi:
10.1017/s0020818312000252
Cals, J., & Kotz, D. (2013). Effective writing and publishing scientific papers, part III:
introduction. Journal Of Clinical Epidemiology, 66(7), 702. doi:
10.1016/j.jclinepi.2013.01.004
Cook, K., Hart, M., Kinney, M., & Oler, D. (2014). How to Discuss a Paper. SSRN Electronic
Journal. doi: 10.2139/ssrn.2388922
Cuschieri, S. (2018). WASP (Write a Scientific Paper): Understanding research metrics. Early
Human Development, 118, 67-71. doi: 10.1016/j.earlhumdev.2018.01.015
Donovan, S. (2018). Reflective Practice: Eight Stages of Publishing a Scientific Research
Paper. Publications, 6(1), 8. doi: 10.3390/publications6010008
Florczak, K. (2016). Evidence-Based Practice. Nursing Science Quarterly, 29(2), 108-112. doi:
10.1177/0894318416630096

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 13
Gordon, S., & Donini, A. (2015). Romancing principles and human rights: Are humanitarian
principles salvageable?. International Review Of The Red Cross, 97(897-898), 77-109. doi:
10.1017/s1816383115000727
Hogue, C. (2012). Results. How To Write A Paper, 22-28. doi: 10.1002/9781118488713.ch4
International Federation of Red Cross and Red Crescent Societies. (2010). Choice Reviews
Online, 47(11), 47-6330-47-6330. doi: 10.5860/choice.47-6330
Knottnerus, J., & Tugwell, P. (2013). How to write a research paper. Journal Of Clinical
Epidemiology, 66(4), 353-354. doi: 10.1016/j.jclinepi.2013.01.007
Kotz, D., & Cals, J. (2013). Effective writing and publishing scientific papers, part IV:
methods. Journal Of Clinical Epidemiology, 66(8), 817. doi: 10.1016/j.jclinepi.2013.01.003
Lee, M., Johnson, K., Newhouse, R., & Warren, J. (2013). Evidence-Based Practice Process
Quality Assessment: EPQA Guidelines. Worldviews On Evidence-Based Nursing, 10(3),
140-149. doi: 10.1111/j.1741-6787.2012.00264.x
Pushparajah, K., & Qureshi, S. (2013). How to write a good scientific research paper and get it
published. Indian Journal Of Thoracic And Cardiovascular Surgery, 29(1), 10-13. doi:
10.1007/s12055-013-0194-4
Rhodes, M. (2012). How to undertake a research project and write a scientific paper. The Annals
Of The Royal College Of Surgeons Of England, 94(5), 297-299. doi:
10.1308/003588412x13171221590331
Salmond, S. (2013). Finding the Evidence to Support Evidence-Based Practice. Orthopaedic
Nursing, 32(1), 16-22. doi: 10.1097/nor.0b013e31827d960b

A CRITICAL ANALYSIS OF AN EVALUATION REPORT 14
Saper, C. (2014). Academic publishing, part III: How to write a research paper (so that it will be
accepted) in a high-quality journal. Annals Of Neurology, 77(1), 8-12. doi:
10.1002/ana.24317
Sen, G. (2014). Clues for writing a good research paper. Journal Of Oral And Maxillofacial
Radiology, 2(2), 39. doi: 10.4103/2321-3841.138629
Thomas, A., Boucher, M., & Pulliam, C. (2015). Qualitative to Quantitative and Spectrum to
Report: An Instrument-Focused Research Methods Course for First-Year Students. Journal
Of Chemical Education, 92(3), 439-443. doi: 10.1021/ed5007019
Van Engeland, A. (2016). Contextualisation of Humanitarian Assistance and its Shortcomings in
International Human Rights Law. Israel Law Review, 49(02), 169-195. doi:
10.1017/s0021223716000054
Vande veegaete, A., Borra, V., De Buck, E., & Vandekerckhove, P. (2016). Guidelines of the
International Federation of Red Cross and Red Crescent Societies: an overview and quality
appraisal using AGREE II. BMJ Open, 6(9), e011744. doi: 10.1136/bmjopen-2016-011744
Wilson, V. (2016). Research Methods: Scoping Studies. Evidence Based Library And
Information Practice, 11(1(S), 75. doi: 10.18438/b8tk83