Methods for measuring social impact

Once you have your outputs and outcomes set you can begin to think about how you will measure against them.

In order to do this effectively it is important to build a robust approach that is tailored to your particular project using a selection of the tools, detailed in this section.

However, it is also useful to think about what you don’t need to measure so you are concentrating your resources on measuring what is useful. You will also need to consider whether evaluation needs to be undertaken in-house or by an external evaluator, so your budget can reflect this. In-house evaluation is sometimes assumed to be the cheaper option, and external evaluation as being more robust: neither of which is necessarily true.

Evaluation Support Scotland has produced a useful guide if you are going down the route of external evaluation that can help you ensure you get the most from this process. See the further reading section of the guide.

Before choosing an approach, you should first consider if there is an existing approach being used by local partners that can be adapted. This can have the benefit of aiding the sharing of information across different sectors. Doncaster Museum is aiming to engage with local cultural not for profit organisations (NPOs), about a shared impact evaluation for their Esmeé Fairbairn Collections Fund project, building skills and raising aspirations in Doncaster’s ex-mining communities.

Carolyn Dalton, heritage service manager at Doncaster Museums explains “this is so that we can easily compare and use to advocate for the value of culture on health and wellbeing and community resilience.”

Critically, this process is being planned from the outset of the project ensuring baseline measurements can be assessed.

In museum evaluation we often think of examples of quantitative evaluation that are not necessarily relevant for measuring social impact, for example, visitor numbers.

However there have been attempts to develop quantitative methods for measuring social impact. Daniel Fujiwara of the London School of Economics developed such a method for the Happy Museum Project to look at the social value of museums in improving health and wellbeing of visitors and giving these a financial value to demonstrate their worth.

National Museums Liverpool used this wellbeing value to demonstrate its social impact in financial terms giving its venues a wellbeing value of £130 million. This approach is harder to undertake on a project level but can be a useful tool for a whole museum service to demonstrate its social value at a higher level. On a smaller scale there may be quantitative tools that can build up a picture of the social impact of a piece of work, especially if used in conjunction with qualitative tools.

For example, the University of Leicester’s “Encountering the Unexpected” project which used response cards to capture a wide variety of information.

Quantitative evaluation tools

These can include:

Closed questions: responses to interviews and questionnaires using closed questions can be used to ascertain simple attitudinal or knowledge level information about your audiences, for example, “Have you visited the museum before?” This can be very useful for establishing a baseline.

Personal data: personal data about users such as age, ethnic identity or postcode. However, there should always be an opportunity for participants to opt out of providing this data, such as a “prefer not to say” option.

Range statements: these can be a useful way to capture more abstract data such as feelings or desires. When using range statements, it is important to keep the statements simple and straightforward so it won’t adversely affect the work you are trying to do.

Visual aids: visual aids such as blob trees are a creative way of capturing data on a participant’s feelings towards a piece of work. It that can be measured at different stages of a project to reflect individual attitudinal change.

Qualitative evaluation

There are various different methods that can be used to gather qualitative data to demonstrate social impact. Qualitative data can provide more substance to your case for social value and will tell you more about the ways in which your work is impacting participants lives.

Interviews and questionnaires

Interviewing participants and stakeholders is one method of looking at the social value of your work. This may take place at various stages to see how respondents’ answers may vary. There are several factors you may wish to consider when planning for this type of data capture.

Conducting interviews can be a formal process with a set list of questions which the interviewer may not deviate from or less formally where the interviewee may just be asked generally about a specific topic or theme and questions adapted as the process unfolds. With the former the results can be easier to quantify among multiple participants, but the latter can offer more flexibility and therefore provide a wider plethora of information.

If undertaking formal interviews, the questions asked require careful consideration. Closed questions which require a yes or no response are easier to analyse but have little scope for anecdotal evidence unless there is a follow up question asking the respondent to expand.

The Arts Council has produced an excellent resource to help you formulate the right questions for your particular needs. The location of your interview can be just as important as what you are asking. Consider what will make participants comfortable, for example, will you interview people individually and will you use a recording device?

The same principles apply to questionnaires as interviews although you’ll need to consider whether the questionnaire is a self-led activity or if it will be facilitated.

The needs of those being questioned should be considered again; think about the use and tone of the language and make sure it is not going to be intimidating or off-putting, otherwise your response rate will be lower.

Web apps such as Survey Monkey are becoming increasingly common to capture data and this has the benefit of being accessible virtually, ensuring the anonymity of participants. As with written questionnaires the needs of the participants should be the main decider of the approach chosen as the use of technology can be a barrier to participation.

Observations

These are more one way with the observer recording interactions or activities and the responses from participants. Like interviews and questionnaires, a structured approach is necessary for this type of measurement. A structured form should be used and completed by the observer to ensure a level of continuity between instances of observation.

Observers can be part of the project team or it may be done by people who work with the participants in another way. The Happy Museum project points to dementia care mapping as a method of observational recording that is undertaken by care practitioners.

Testimonials

A popular and effective form of measuring social impact is to capture testimony from participants. This is often done towards the end of a project once a relationship and trust has been built up with individuals. It can be a powerful way to convey the difference streams of work can make to individual lives.

There are various different ways to capture and record this – it could be first-hand accounts from participants or it could come via partner organisations such as healthcare providers or community groups.

Gawthorpe Textiles Collections captured individual testimonials from members of their “Sew Social” programme – part of their Esmee Fairbairn Collections Fund project. The testimonials were captured as part of a film production and demonstrate very clearly the social impact of this work.

Testimonials can also be captured digitally. The social media app, Indeemo, was used by the Museums Association Transformers scheme as a digital method of capturing testimonials of the scheme’s participants.

Participants were asked to upload videos throughout the programme with an emphasis on these fitting in with their day so impact on their time was minimised.

“The uploads allowed the personality of individuals to come through and allowed them to formulate a more honest response in an unpressured or rushed environment” says Claire Renard of the Museums Association. “We found real value in this format of feedback because of the immediate, powerful and honest responses that wouldn’t necessarily easily translate on a normal paperbased evaluation”.

Creative methods

Creative methods for data capture are often well suited to qualitative measures and can work very well in a project setting. There are advantages in using creative ways to gather data as participants might be more relaxed and involved than in traditional methods such as questionnaires, although more thought might have to be given to quantifying the results.

The University of Leicester ‘s Encountering the Unexpected used journals as a method of capturing responses from participants. This provided an extra benefit to participants as it was something they could take away with them to reflect upon their experiences.

“We encouraged projects to give all participants Journals, we had envisaged that people would reflect on the process” says Jocelyn Dodd, director of the Research Centre for Museums and Galleries at the University. “The journals were especially interesting, helping to focus, gather momentum and for reflection too.”

National Museums Liverpool used visual minutes in their Esmée Fairbairn Collections Fund project, Sankofa, to record workshops and meetings with community groups in a fun and interesting way.

This enabled sessions to be kept on track and key outcomes recorded in a less formal way which didn’t alienate participants.

Off-the-shelf tools

There are a variety of tools that have already been developed and are available for use, meaning you don’t have to design your own. University College London (UCL) has designed and developed a toolkit for measuring wellbeing that has been used effectively in Pontypridd Museum to assess the social value of volunteering.

“I asked the volunteers to score each emotion on the [wellbeing] umbrella when they started their session in the morning, and then to score each one again at the end of the day. To keep it quite unobtrusive I just ask them to complete them once every two months or so. It is kept anonymous with the option of writing additional comments on the back.” says Morwenna Lewis, curator of Amgueddfa Pontypridd Museum.

The Happy Museum project has produced a number of resources and guides that are useful when developing an approach to evaluation. They have developed a number of tools and techniques for measuring wellbeing such as the Happy Tracker which encourages participants to demonstrate their wellbeing by lifting their hands, the higher the more positive they are feeling.

Further reading

Evaluation Support Guide, July 2009, Evaluation Support Scotland

Blob Tree resources

Inspiring Learning for All Question Bank, Arts Council England

Observational evaluation, Hilary Jennings, the Happy Museum

UCL Museum Wellbeing Measures

Evaluation toolkit for museum practitioners, Renaissance East of England

Links and downloads

Measuring socially engaged practice: a toolkit for museums (pdf)

Measuring socially engaged practice – large print version (word)