Marram-Ngala Ganbu Project Evaluation Plan

The Marram-Ngala-Ganbu Program aims to protect and develop Victoria's Aboriginal Children populace from threats of family violence, harm and removal from families with special attention to the Koori Family. The evaluation plan for the program is based on an Impact-Evaluation Plan, specifically the Experimental Design or Randomised-Control-Trial (RCT) Design. The evaluation operations will be on the basis of Theory-based Evaluation Approach.

Marram-Ngala Ganbu Project-
Evaluation Plan

Table of Contents
Marram-Ngala Ganbu Project........................................................................................................1
Introduction: Marram-Ngala-Ganbu Evaluation Project...........................................................3
Goals and objectives...............................................................................................................3
Program strategies..................................................................................................................4
Literature Review.......................................................................................................................4
The Evaluation strategy..............................................................................................................5
Evaluation Objectives.............................................................................................................5
Evaluation Design – Description............................................................................................6
Rationale of Evaluation Design..............................................................................................6
Evaluation Approach..............................................................................................................7
Method of Data Collection.........................................................................................................8
Time-line of Evaluation Strategy.............................................................................................10
The entire Evaluation Plan has been devised on a 6-months’ time period with some important
dates:........................................................................................................................................10
Timeline...........................................................................................................................11
References................................................................................................................................13

Introduction: Marram-Ngala-Ganbu Evaluation Project
Explanation of the Program
The Marram-Ngala-Ganbu (we-are-one) Program was established on protecting and
developing Victoria’s Aboriginal Children populace (birth - 17 years) from threats of family
violence, harm and removal from families with special attention to the Koori Family. After
the initial identification of the problem in March 2009 AJF23 Forum, the Koori Hearing Day
was established by Protecting-Victoria’s-Vulnerable-Children-Inquiry, 2012, drawn on the
basis of data suggesting 59% increment in Aboriginal Victorian children populace at out-of-
home-care units, 2013-2015, with 40% cases having been separated from siblings and 60%
placed under non-Aboriginal care ("Marram Ngala Ganbu program | Jenny Mikakos", 2019).
Thus the Program carried out through Government support, aims to improvise the future
outcomes of Koori Children via a culturally apt procedure with skilled decision making,
adhering to Aboriginal-Child-Placement-Principle, of Children’s-Youth-and-Families-Act
2015. During the 12 month pilot program, 300 aboriginal families have been provided access,
and each Tuesday the program operates at the Children’s Court, Broadmeadows. In 2016, it
was identified by Royal-Commission-into-Family-Violence that Torres Strait Islander and
Aboriginal children are 8 times more subject to child-protection reports that non-Aboriginal
child populace and at present an independent evaluation plan is mandatory to justify the
outcomes and functioning of the 2-year operational Marram-Ngala-Ganbu Program, as have
been conducted in the present scenario.
Goals and objectives
The Marram-Ngala-Ganbu Program has the following objectives ("Marram-Ngala Ganbu:
We are one | NAJ2017", 2019):
 To Improvise Koori children outcomes, in child-protection proceeding
 To enhance Koori family and community participation
 To enable cultural-understand based decision making
 To assist culturally appropriate decision-making reflecting the Koori child’s
community
 To adhere to Placement Principles set out in Children’s-Youth-and-Families-Act 2015

 To provide cultural support to Koori children in consistent to out-of-home-caregiving
services, to develop and maintain their identity and culture-oriented community
connection.
 To enable court processes of Koori parents, children aiding in their full participation
in proceedings via respect to individual cultural needs and identity with the s522 of
Act, to deliver court procedures promoting cooperative relation according to s215B,
of Act.
Program strategies
The Victoria, Children’s Court seeks to engage an experienced, appropriately qualified and
independently functional Aboriginal-led team of Evaluation which will undertake the
Marram-Ngala-Ganbu Program’s evaluation, serving as evidence base to support expansion
and future refinement. The Project evaluation will be governed by the Marram-Ngala-Ganbu-
Evaluation-Steering-Committee and will be managed according to Project-Management-
Office (PMO) of Victoria’s Children Court, upon the engagement of a Project Manager
functioning as a secretariat of Steering Committee ("Marram-Ngala Ganbu – Koori Hearing
Days | Aboriginal Justice", 2019).
Literature Review
(Explanation of the Program – on which Evaluation Plan is based on)
The Victoria Children’s Court has 2 operational divisions, the Family Division (protection
and care of young people and children till 17 years from harm and family violence), and
Criminal Division (dealing with young people and children aged 10-17 years). The AJF 23 in
2009, reported concerns on child protection of Aboriginal children leading to development of
Hearing Day of Koori Family, in 2012 according to Protecting-Victoria’s-vulnerable-
children-inquiry (ALWAYS WAS, ALWAYS WILL BE KOORI CHILDREN Systemic
inquiry into services provided to Aboriginal children and young people in out-of-home care
in Victoria, 2019). Thus a pilot program, commenced in 2016, at Broadmeadows Court
named as the Marram-Ngala-Ganbu Program (we-are-one in Wurundjeri-Woiwurrung-
language). During the following year, Royal-Commission-into-Family-Violence stated that
the Torres Strait Islander and Aboriginal children are 8 times more subject to child-protection
reports that non-Aboriginal child populace. In 2018, a state-wise expansion was called for
Koori list in family division of Victoria Children’s Court, by the Youth Council of Koori.

During the 12 month pilot program, 300 aboriginal families have been provided access, and
each Tuesday the program operates at the Children’s Court, Broadmeadows. The AJA4, 2018
further suggested future considerations for an increased number in Koori Convenor and
Children Courts which provide the Program ("Victorian Aboriginal Affairs Framework 2018-
2023", 2019). At present an independent evaluation plan is mandatory for justifying the
Program outcomes requiring justified evaluation of the same consisted of experienced and
Aboriginal team members.
Which Technique used and Why?
In the present context, an Impact-Evaluation Plan (method used) is necessary for providing
impact information of the program, be positive or negative, direct or indirect in order to
establish the causal attribution of the impacts ("Impact evaluation | Better Evaluation", 2019).
Thus the Impact Evaluation Plan can reorient and improve upon the Marram-Ngala-Ganbu
Program’s operation for providing formative purpose, informing decisions on whether certain
operative procedures need to be continued, replicated, discontinued or scaled up on
(summative purpose). The formative evaluation process delves on procedures while the
impact evaluation used for summative procedures focuses on evaluation results and
outcomes. The impact evaluation Program, as selected in the present scenario can be used for
improving the program’s implementation for next participant intake by identification of
critical elements for proper management and monitoring of the Program. The Impact
Evaluation provides not only findings on what works the best, but also discusses on the
necessary requirements to make the Marram-Ngala-Ganbu Program a success taking in
consideration different group population in respect to different settings.

The Evaluation strategy
(Evaluation of Marram-Ngala-Ganbu Program)
Evaluation Objectives
The Evaluation Plan for the Marram-Ngala-Ganbu Program aims to test and evaluate
whether the aforementioned objectives of the Program is justifiably met, revolving on the
following questions which may provide the Evaluation Plan the correct direction.
Evaluation Questions (as asked to be included)
 What the outcomes of the Program? Are there any impacts of the Program which have
been unintended or unprecedented?
 What is present experience of Aboriginal families participating in the Program or the
Lawyers which represent them?
 What is present experience of the DHHS- Child-protection Workers, Judicial officers,
Conference Convenors and Staffs overseeing and managing the Program?
 What is the extent to which the Program improvised positive contact development
between court and Aboriginal family?
 What is the protection-placement-principle compliance rate of the Aboriginal
children?
 What are the major differential factors between outcomes of mainstream family
hearing and Marram-Ngala Ganbu hearing?
 What are the improvising opportunities that will help in expansion of the Program?
 How effective was the Intervention Program?
 What are the breach rate differences between people participating in mainstream
process and people involved in Marram-Ngala-Ganbu Program?
 Are the Marram-Ngala-Ganbu’s staff position description consistent with their
performances in the Program?

Evaluation Design – Description
Impact Evaluation Plan is of several types and in the present scenario the Experimental
Design or Randomised-Control-Trial (RCT) Design (method used) will be used. In this
Evaluation Design random assignment of units is done in the Program, leading to unbiased
estimation of the Program’s impact, being based on statistical design (Barahona, 2010). In the
RCT Evaluation Design, the eligible Aboriginal individuals/children/families are assigned
randomly to the Marram-Ngala-Ganbu Program for them to receive the necessary services.
This Design confirms equal chances of participation for each of the Koori Aboriginal Family,
for receiving the Program’s services and protection of the Koori children. The RCT Design is
typically conducted via random numbers generated by computers, public lottery system, or
natural randomness to generate transparent and credible evaluation sampling, also using
birth-dates, vehicular number-plates or veteran status, and permitting stratification.
Rationale of Evaluation Design
Why is the Design used?
The Randomised-Control-Trial (RCT) Design aids to ensure the fact that there exists no
systematic differences in between the beneficiaries and control groups during the Evaluation
Plan Initiation. Also the RCT confirms that no initial character influencing the interest
outcomes are correlated with Koori family/individual participation. In addition the RCT
removes biased sources which creates control difficulties in non-randomised evaluation
designs (White, 2013). But rather the RCT is based on self-selection biases where an
individual decides whether they want to participate in the evaluation plan or not. There is
another process followed in RCT in the form of Agency-Selection Bias, where a uniform and
transparent policy is maintained by implementing agency for selecting participatory units.
The RCT is also the chosen Evaluation Design as this is a scenario where the eligible
individuals are larger in number than the resources or selected services available and
therefore a faire transparent manner is used for assigning the required benefits to the large
target population of the Aboriginal families. This evaluation design thus provides an equality
of chance for every child, Koori Family or Aboriginal individual present in the sample
population.

Evaluation Approach
(Approach taken)
The evaluation operations will be on the basis of Theory-based Evaluation Approach also
known as the conceptual-analytical model which is an efficient manner of carrying out
analysis or for structuring the evaluation plan (Coryn, Noakes, Westine, & Schröter, 2011).
These approaches complement and can be combined in usage with most of evaluation designs
and techniques of data collection via a logic-of-enquiry.
The Theory-based evaluation approach is illustrated in the Figure below.
Evaluation Approach
Input
M NG plans, policies
M NG infrastructure
Resources/Expenditure
M NG families
Area staff
Support staff
Conference Convenors
Lawyers
DHHS- Child-protection
Workers,
Judicial officers,
Other service providers
Program Staffs
Production
Process
M NG management and planning
M NG coordination, service
delivery
M NG partnership
M NG arrangements
M NG service level
M NG agreements
M NG Facilitators and
M NG barriers to change
Output
Amount and Type of Support
Amount and Type of Services
Amount and Type of Information
provided
Access to service
Program satisfaction
Operators Support
Staff Support
Outcomes for
Koori families
Improved child protection
Stability
Community participation
Independence
Quality of life
Reduced family hostilities
Improved cultural connectivity with care
providing services
Improved social networks

The present approach differentiates between 4 distinct yet close-linked stage in the Plan –
input, process, output and outcomes, and is essential in understanding complexity in
individual, community, governmental body’s interaction over time. The approach helps in
drawing mannerisms through which the Program operations and how it is implemented, how
it impacts the services delivered and what are the eventual outcomes of the M NG Program.
Strengths of Approach (Cojocaru, 2009):
 Often used in circumstances where other approaches are not used
 Provide proper contribution effects evaluated by the managers or evaluators
 Deliver findings on why certain intervention measures are not delivering
 Provide conclusions on cause-effect program elements
 Leverage existential data to a great extent focusing on new sampling arenas with
significant gaps, causing efficient usage of evaluation resources.
Weakness
 The approach does not provide quantitative measure of contribution size that the M
NG program is making.
 It is difficult to develop theory-of-change due to larger stakeholder involvement and
synthesis of large number of information sources
 Theory based approaches are time taking and requires huge data sources
 More than one theory may emerge which needs testing to seek out the best theory that
reflects reality. Evaluators needs to focus on where the theories may differ and
explore the necessary reasons.
Method of Data Collection
Evaluation Method
(Which Method is Used)

The Evaluation Plan is based on quantitative and qualitative methods utilising both. A
consultative and participatory approach is requisite for the users of the M NG Program. A
Theory based evaluation approach is used for exploring how the M NG Program and why has
some outcomes been generated or not examining the causal factors bringing about the
necessary effects of the Program ("Marram Ngala Ganbu program | Jenny Mikakos", 2019).
The Victoria, Children’s Court is interested in recognising the most suited methodical
approach for enhancing the depth and variety of Aboriginal family research presently
available for the program’s evaluation and operation.
Data Collection Methods
(How is the Method Used?)
Fresh collected qualitative primary data will be gathered from Koori family, Aboriginal
families and individual Aboriginal children aged below 17 years and other the stakeholders
(Government and Community), for implementing and using the Marram-Ngala-Ganbu
Program. Focus group interviews will be conducted based on child population, aboriginal
children bereft of families, carer service providers, Koori children family, and Aboriginal
Torres Strait children respectively. The Aboriginal Children have to be aged from birth till 17
years of age, younger than other adult demographics. The random sample method will be
selected for sampling. Methods like field notes, observations, participant demographics,
audio records, discussions, key opinions summarisations, and interviews needs to be utilised
in impact evaluation procedure (Khandker et al., 2009). The data thus gathered, will be
encrypted and protected by saving in softcopy-encrypted format in appropriate locations such
that participant confidentiality is thoroughly maintained throughout the evaluation period of
the program.
Data analysis and interpretation
Analysis
The impact evaluation Analysis will be based on inferential data analysis and statistical
means for drawing inferences from the content. This is because the data collection is primary
qualitative data as collected from the target group interviews.
Standards and Indicators

The elements helping to understand performance success includes performance achievement
and stakeholder satisfaction. Some measurable indicators to understand the positive outcomes
of M NG Program can be in improving the care services offered to the Aboriginal children
populace by culturally orienting them or by including carers belonging to the same culture.
Standards for success analysis will be focused on several benchmarks of children lifestyle
and wellbeing which helps to generate positive outcome-based evidence after a thorough
Evaluation Plan has been conducted.
Some relevant Questions that needs to be assessed during the interviews are:
Q What the outcomes of the Program?
Q Are there any unprecedented impacts of the Program?
Q What is present experience of Aboriginal families participating in the Program?
Q What are the major differential factors between outcomes of mainstream family hearing
and Marram-Ngala-Ganbu hearing?
Q What is present experience of Lawyers representing Aboriginal families participating in the
Program?
Q Is the job role of the Staffs involved in direct proportion with their performance?
Time-line of Evaluation Strategy
(Time Taken)
The entire Evaluation Plan has been devised on a 6-months’ time period with some
important dates:
1. Ethics application to Justice-Human-Research-Ethics-Committee – 28.6.2019
2. Draft Evaluation Report of Marram-Ngala Ganbu Evaluation-Steering-Committee –
11. 10. 2019
3. Final Evaluation Report as endorsed by the Marram-Ngala-Ganbu-Evaluation-
Steering-Committee – 31.10.2019

Timeline
Year 2019
Activity May 2019 June 2019 July 2019 August
2019
September
2019
October
2019
1. planning
and design
evaluation
2. data
collection
3. gathering
clearances
4. training data
collector
5. Ethics
application to
Justice-
Human-
Research-
Ethics-
Committee
6. collecting
data and
monitoring
7. data
management
and analysing
data
8.
communicating

findings
9. Draft and
Final
Evaluation
Report as
endorsed by
the Marram-
Ngala-Ganbu-
Evaluation-
Steering-
Committee
10. Action
Planning

References
Barahona, C. (2010). Randomised Control Trials for the impact evaluation of development
initiatives: a statistician’s point of view.
Cojocaru, S. (2009). Clarifying the theory-based evaluation. Revista de Cercetare şi
Intervenţie Socială, (26), 76-86.
Coryn, C. L., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review
of theory-driven evaluation practice from 1990 to 2009. American journal of
Evaluation, 32(2), 199-226.
Impact evaluation | Better Evaluation. (2019). Retrieved 25 September 2019, from
https://www.betterevaluation.org/en/themes/impact_evaluation#ImpactEvaluation_2
Marram Ngala Ganbu program | Jenny Mikakos. (2019). Retrieved 25 September 2019, from
https://www.jennymikakos.com.au/parliament/marram-ngala-ganbu-program/
Marram-Ngala Ganbu – Koori Hearing Days | Aboriginal Justice. (2019). Retrieved 25
September 2019, from https://www.aboriginaljustice.vic.gov.au/the-
agreementaboriginal-justice-outcomes-frameworkgoal-11-aboriginal-families-are-
strong-and/marram
Marram-Ngala Ganbu: We are one | NAJ2017. (2019). Retrieved 25 September 2019, from
http://www.naj2017.com/1846
Victorian Aboriginal Affairs Framework 2018-2023. (2019). Retrieved 25 September 2019,
from https://w.www.vic.gov.au/aboriginalvictoria/policy/victorian-aboriginal-affairs-
framework/victorian-aboriginal-affairs-framework-2018-2023.html
Victorian Government Printer (2019). ALWAYS WAS, ALWAYS WILL BE KOORI
CHILDREN Systemic inquiry into services provided to Aboriginal children and
young people in out-of-home care in Victoria [Ebook]. Melbourne, Victoria, 3000.
ISBN 978-0-9945296-2-6.Retrieved from https://ccyp.vic.gov.au/assets/Publications-
inquiries/always-was-always-will-be-koori-children-inquiry-report-oct16.pdf
White, H. (2013). An introduction to the use of randomised control trials to evaluate
development interventions. Journal of Development Effectiveness, 5(1), 30-49.