8 Evaluating impact
Earlier, we mapped out our intended outcomes and required activities, and talked about measurement plans to measure both activities and outcomes. We also talked about principles to help us craft effective messages, as well as ways to ensure they are seen.
Our activity measures might include whether we produced creative content, whether people saw it, whether they liked it or engaged with it. Our impact measures might include whether their knowledge increased, or their perception changed, or their behaviour changed. It might also be whether these changes were visible to decision-makers, and whether those decision-makers made a change as a result. It can be useful to map these indicators into a linear form of engagement logic model, as below.
| Indicator 0 | Indicator 1 | Indicator 2 | Indicator 3 | Indicator 4 |
| Reach or visibility | Interest | Passive support | Active support | Action |
| Number of views, impressions etc | Click-through rate, video completion rate etc | Likes, comments, follows etc | Shares, endorsements etc | Petition signs, emails to politicians etc |
Once we have a piece of creative content (a video or a poster, for example), the first activity metric that we need to measure is reach or visibility of that message. If you’re using social media, you can easily understand reach through social media tools, but if we’re putting posters in shops or leaflets through doors, how do we best understand reach? We can do this by counting how many people are looking at our poster, or even just measuring how many leaflets we delivered, and how many were taken away.
Impact indicator 0 is simply the visibility of our message, since visibility is vital to the successful impact of any communications campaign: at the very least, did people see our messages (e.g. impressions, reach etc)?
Various studies have been conducted over the years which tell us that we need to see messages between 7 and 12 times before we start to pay attention to them. When thinking about placing our messages, it’s worth sharing them as broadly as possible so that audiences start to notice them, and hopefully hear or understand what they are trying to say.
Social media is a great tool for communicating, but so are other out-of-home methods such as posters and billboards. The key to message delivery is to think about the various ways we can reach people throughout their day. Perhaps on their commute to work, perhaps at the supermarket, perhaps at a sports centre, theatre, or other places where we might visit for our hobbies. Finding unusual partnerships is a great way to be creative with reach, and many organisations will be keen to partner to support environmental action as well as other public interest causes.
For social media advertising, you can find out whether Google will help subsidise your placement costs: https://www.google.com/nonprofits/offerings/google-ad-grants/
The next indicator (impact indicator 1) is interest: are there ways you can measure this, and are there any suggestions in your data that people were interested in your message?
Think about e.g. click-throughs, video completion rates, if you’re using social media. Perhaps there is a way to measure interest in your offline communication activities. Perhaps we can measure interest in person by seeing how long someone continues to read our poster. Perhaps we can only measure interest through asking about it in focus groups or interviews when we’re testing content – not a ‘real world’ indicator, but a useful learning nonetheless.
Both of these indicators are fairly passive indicators that don’t tell us much about our impact, but they help us understand whether our content has been seen, and whether our audiences find it interesting or compelling enough to begin their listening journey. These are the basic needs of an impactful communication campaign, so worth measuring to help us learn, and to identify the incremental ways through which we achieve impact. (What if no-one saw it? What if no-one was interested? What could we do differently?)
Next to reach and interest, we have support.
Similar to interest, here we are looking to measure a more active indication of support among our audiences. Did they share our social media post, or comment on it favourably for others to see? Did our offline audiences take a leaflet, or pass them to a friend, or perhaps visit a website we’d mentioned in our poster, or some other active indication that they were curious enough to take some small action in support of our efforts. What can we also learn, perhaps, from unfavourable comments, or what if no-one shared it? Is there a way we can make our post more shareable for audiences, perhaps allowing some element of co-creation?
And finally action.
Here we are looking for metrics which suggest our audience has taken active measures to contribute to our issue. Perhaps they’ve signed a petition, or emailed their elected representatives (both of which we can measure digitally). What measurement activities can we put in place to measure actions taken for offline interventions, based on your calls to action?
Another way to measure the impact of our campaign is to test it using public opinion research. This can be very expensive, so isn’t essential, but you can test videos and posters in a survey setting to see how people react, or in focus groups. You might even test the creative among friends, or people you know who resemble the audiences you’re trying to reach. You can also conduct A/B testing in digital formats to see the impact of the communication intervention on those who saw it online versus those who didn’t.
