06 Oct 2020

Mastering the Delivery and Impact of Community Projects

Capstone Project
Mastering the Delivery and Impact of Community Projects

Imagine you were asked to report on your contribution to community happiness, well being and cultural connection. 

Now imagine you had to measure these outcomes not for a year, but for decades. The measurement of complex long-term outcomes such as these is one of the most important and most difficult problems in project/program/portfolio management.

But the solution has application in all industries – even traditional hard industries like construction are now accountable for ‘placemaking’, which focuses on the management of public space and how that space is given character. 

  • How do you measure that?

  • How do you measure your impact on mental health, bushfire resilience, sustainability and culture?

This was the task set before a team of students from the University of Sydney in a ‘capstone project’ that marked the completion of their Master of Project & Program Management.

The capstone project brings Masters students together with the private and public sector to deliver real projects. From the students’ point of view, it’s an opportunity to learn from the experience of working with different industries, putting into practice the models, tools and skills acquired over the course of their studies. From the client’s point of view, the value comes through the introduction of novel perspectives and capabilities.

 

THE CHALLENGE

One such capstone project, delivered in the first half of 2020, involved Masters students Livia Ranieri and Samuel Bakara collaborating with the NSW Environmental Trust. The objective of this capstone project was the development of a prototype Monitoring, Evaluation and Reporting (MER) Framework for the Protecting our Places (POP) program – a contestable grants program administered by the Environmental Trust focussing on Aboriginal cultural preservation.

The POP program facilitates grants for Aboriginal organisations to undertake environmental and cultural projects on Country. MER Frameworks – like Benefits Realisation Frameworks – ensure that programs are assessed based on their outcomes, rather than their intentions, and support informed decision making, shared learnings and program improvements. Their value cannot be underestimated, especially on complex initiatives with long-term outcomes.

Prior to the capstone project, the POP program needed to evolve its MER Framework to embrace the measurement of impact on Aboriginal culture within grant funded environmental projects. Historically, value was captured anecdotally and acknowledged largely through the ongoing relationships between grantees and POP administrators. As a result, the value of POP projects with respect to cultural benefits was effectively invisible.

As Maggie Bushel, Aboriginal Programs Officer from the Environmental Trust, explained:

 

“It is the environmental and cultural benefits offered by POP projects that holds the greatest significance to the Aboriginal community. Aboriginal communities that participate in cultural activities have better mental and physical health. Therefore, the cultural impact of the POP program has a correlation with the happiness and wellbeing of the community. But these benefits weren’t being measured.”

 

COMPLICATION


Despite the Trust’s in-principle support for an enhanced approach to measuring cultural benefits, the key challenge for the capstone team was the development of measures for complex outcomes such as Aboriginal cultural protection and the associated health and community outcomes resulting from Indigenous peoples’ connection to Country.

These measures are particularly important when, as Ms Bushel indicated, “assessment purely using traditional market techniques will undermine the true and more significant benefits that the underlying projects aim to achieve.”
 

SOLUTION

The capstone team began where all good students begin – with an extensive literature review to ground the POP MER Framework in best practice. This review encompassed a wide range of publications, guidelines, standards, and case studies relating to Benefits Realisation/MER frameworks, with a focus on those that had been applied to Indigenous community programs.

The literature review not only justified the review of the MER framework for the POP program; it also strongly emphasised the need for consistency and alignment with established MER practices used in other contestable grants programs at the Environmental Trust.

This was essential to enable consistency in outcome reporting as well as comparative evaluation of grant programs in the context of the Trust’s strategic investment decisions. Indeed, it is hard to do a benefit-cost analysis at the portfolio level when some benefits are effectively invisible.

The capstone team integrated insights from the literature review with advice from a Steering Committee who support POP grantees through capacity building workshops on behalf of the Environmental Trust. The result was a prototype MER framework for the POP Program that:

  • identifies a range of qualitative and quantitative measures based on data
  • effectively assesses target outcomes
  • aligns with existing MER Frameworks within the Environmental Trust and the Program Logic for the POP program

The prototype MER Framework for the POP Program included quantitative data consistent with other environmental programs administered by the Trust, such as:

  1. hectares of land regenerated
  2. number of Aboriginal people working on Country
  3. people attending training and awareness raising events
  4. number and type of trees, shrubs and grasses planted
  5. number of and hours contributed by volunteers
  6. grant projects progressing to the Implementation Phase
  7. engagements with Aboriginal Land Councils and/or other key Aboriginal groups

The team then looked at opportunities to add to these measures by drawing on lessons learned from other Indigenous programs examined in the literature review, such as:

  1. Improved mental and physical health within Aboriginal communities resulting from participation in cultural activities
  2. Creation and performance of art, which is correlated with feelings of happiness within Aboriginal communities
  3. Anthropological evaluation techniques to obtain culturally relevant qualitative data
  4. Short form surveys used to measure the health and wellbeing of Aboriginal communities
  5. Qualitative questions such as “Do you identify with a tribal mob? Do you identify with a language group or clan? Do you recognise an area as your Country?” where positive answers indicate an increase in connection to country.

In addition to these additional measures for cultural outcomes, the capstone team proposed a change to reporting practices. They recommended the use of Performance Story Reports, that include a blend of quantitative and qualitative data, to report to both government and the community to demonstrate the connection between the outcomes of the program and how these outcomes were reached.


CONCLUSION 

This capstone project centred on the idea that the capture of meaningful data and its comprehensive and holistic reporting to both government and community supports sensemaking on complex projects and the measurement of long-term outcomes.

This is the core value that an effective MER Framework offers:

  • the ability to step back and build a big-picture view of something intangible and complex over a long period of time;
  • to capture knowledge held by many people who may never meet;
  • ​to test assumptions and theories about how to improve at a system level;
  • to understand the impact of past interventions as an input to selecting the next one. And above all, creating sustainable and beneficial outcomes that are measurable and visible to key stakeholders.

The prototype MER Framework delivered by the capstone project was just one step in a long journey which the Environmental Trust continues to take; and there is no easy answer, no ‘best’ MER Framework that can make the complex, simple.

The outcomes of complex programs will always be hard to measure due to the multitude of interacting systems (health, employment, housing, politics, climate change, etc) that drive emergence throughout the life of the program. Their long gestation period is also challenging, with projects typically running for 3-5 years, and outcomes often not measurable for decades. Add to this the increasing occurrence of ‘black swan’ events such as bushfire, flood and drought (all of which have impacted POP projects in the past 12 months), and the causal relationship between projects and their long-term outcomes becomes even more difficult to validate.

Nevertheless, this should not be a deterrent when considering the implementation or enhancement of MER frameworks. Rather, maturing in MER/Benefits Realisation should be embraced as an opportunity to improve our approach to important yet complex initiatives. Fortunately, our future generation of project leaders are already working on it.

 

Kestrel Stone is the CEO of Elemental Projects, an award-winning training and consulting business specialising in developing project management and leadership capability. As an adjunct lecturer, Kestrel also delivers the Capstone Unit in the Master of Project and Program Management at the University of Sydney’s School of Project Management.



CTA-org-partners-(3).png