Impact through Engagement Co-production of administrative data research and the approach of the Administrative Data Research Centre Northern Ireland.

Abstract

Background
The Administrative Data Research Centre Northern Ireland (ADRC NI) is a research partnership between Queen’s University Belfast and Ulster University to facilitate access to linked administrative data for research purposes for public benefit and for evidence-based policy development. This requires a social licence extended by publics which is maintained by a robust approach to engagement and involvement.

Approach
Public engagement is central to the ADRC NI’s approach to research. Research impact is pursued and secured through robust engagement and co-production of research with publics and key stakeholders. This is done by focusing on data subjects (the cohort of people whose lives make up the datasets, placing value on experts by experience outside of academic knowledge, and working with public(s) as key data advocates, through project steering committees and targeted events with stakeholders. The work is led by a dedicated Public Engagement, Communications and Impact Manager.

Discussion
While there are strengths and weaknesses to the ADRC NI approach, examples of successful partnerships and clear pathways to impact demonstrate its utility and ability to amplify the positive impact of administrative data research. Working with publics as data use becomes more ubiquitous in a post-COVID-19 world will become more critical. ADRC NI’s model is a potential way forward.

Background/introduction

This paper describes the Administrative Data Research Centre Northern Ireland (ADRC NI), a research partnership between Queen’s University Belfast and Ulster University.

The ADRC NI is funded by the Economic and Social Research Council (ESRC), part of UK Research and Innovation (UKRI), with additional support from the Research and Development Division of Health and Social Care Northern Ireland (HSCR&D) and both academic institutions.

Alongside equivalent research centres in Scotland and Wales, ADRC NI is part of a UK-wide partnership called Administrative Data Research UK (ADR UK), coordinated by a Strategic Hub. ADRC NI along with the Northern Ireland Research and Statistics Agency (NISRA) are collectively referred to as Administrative Data Research Northern Ireland.

ADR UK’s ethos is to facilitate safe and secure access to linked administrative (government) datasets for research of public benefit and which may inform policy decisions and service provision. This mission is shared by ADRC NI, whose objective is to develop policy-led research providing evidence to both government and communities in an ethically responsible manner.

Following the initial ESRC investment in administrative data infrastructure in 2013 there has been an increasing recognition by stakeholders that public engagement is central to both developing and maintaining public acceptability in the use of administrative data for research. Several recent data ‘scandals’ (care.data¹ [1], which led to the 2016 Caldicott review [2] and the Digital Economy Act [3], and Cambridge Analytica, implicated in data misuse surrounding the Brexit referendum [4]) demonstrate the importance of partnering with communities on the research journey, and the requirement for robust public engagement and targeted involvement to invoke a social licence for the use of administrative data for research.

The aim of this paper is to illustrate the unique approach taken by ADRC NI to public engagement and moving towards involvement, its successes and challenges, and key learning for practitioners.

Context

Defining public engagement and involvement

While the two terms are often used interchangeably, it is important to distinguish between public engagement and public involvement. Within the ADRC NI context we refer to public engagement as a two-way dialogue process with various publics and/or their representatives in the research process, while involvement refers to a process whereby the research ‘subjects’ are part of the research process including question identification, project design and delivery [5, 6]. Our model focuses on voluntary, community and social enterprise (VCSE) sector organisations as experts in certain issues and representatives of the communities most affected by those issues by involving them in project steering committees and knowledge exchange activities, while using these relationships to begin to directly involve those communities for whom VCSEs advocate, whose lives make up our datasets.

In this paper we refer to public involvement and engagement (PI&E) collectively while exploring our approach to engagement and how we are expanding this work and the model built around it into more direct involvement as mentioned above. We understand PI&E as a spectrum of activities as elucidated by Arnstein’s ladder of citizen participation [7]. This model has been adapted many times, as we have done within our work and has been done successfully by other ADR UK partners [8].

Defining co-production

In choosing to use this term, we do not consider that our model as currently deployed is the only or most desirable method of co-production; indeed it is a model that deliberately moves towards a more strict application of co-production, working to directly involve data subjects and end users in the design, development and delivery of research. This is what will be described below.

Co-production can be an often-contentious concept within research. With the ‘heterogeneity’ of the concept it “runs the risk of being used for many different purposes: is it about policy-making or policy implementing, is it about individuals or organisations that co-produce?” [9].

Co-production may be defined as “...the mix of activities that both public service agents and citizens contribute to the provision of public services. The former are involved as professionals, or ‘regular producers’, while ‘citizen production’ is based on voluntary efforts by individuals and groups to enhance the quality and/or quantity of the services they use” [10]. A more classical interpretation of co-production is “the involvement of individual citizens and groups in public service delivery” [10]. Holmes offers a reference to the most common definition of co-production: “...collaboration in governance, priority-setting, conducting research and/or knowledge translation” which “involves researchers and others with a stake in the project: citizens, patients, health care providers, and/or health care decision and policymakers” [11]. Both assert that citizens and ‘public service agents’ can be considered legitimate co-producers.

In place of the specificity of ‘public service delivery’ we understand the general public, VCSE groups, policymakers and data owners all as participants in a model of co-production of research that enables not only end users of services or those most impacted by policy decisions to be seen as ‘co-producers’ but all stakeholders with the wider researcher environment. This approach is also informed by the Payback Framework as developed by the Brunel Health Economics Research Group, which demonstrates the non-linear nature of research and elucidates how different stakeholders feed into and derive impact from the research at different points (feedback loops) [12].

Approach

One of the strategic pillars of ADRC NI is to embed tailored public engagement at all stages of the research process, with a principal focus on data subjects, the cohort of people or communities whose lives are documented in the datasets we use. This strategic approach explicitly places value on the lives of data subjects as ‘experts by experience’ and on working with public(s) as key advocates for data research. Within each programme of research this is principally achieved by establishing project steering or advisory committees at project initiation stage to ensure representation and contribution to the research agenda and protocol, as well as dissemination and impact planning. Steering committees are instrumental in facilitating targeted knowledge exchange events in partnership with key stakeholders from across government, voluntary, community and social enterprise (VCSE) and service providers and conducting outreach with VCSE groups. This programme is led by a specialist Public Engagement, Communications and Impact Manager, whose expertise and affiliations across sectors underpin its success.

An ADRC NI model of co-production, which integrates with the wider ADR UK model of delivering thematic Strategic Impact Programmes (SIPs) of research acts as a blueprint for engagement and impact in all projects listed within the ADRC NI portfolio (Figure 1).

Figure 1: The ADRC NI model of co-production, which centres engagement within the research process.

Data subjects

Large scale linked administrative datasets are made up of potentially millions of data points. The purpose of our research is to address hypotheses by identifying and examining patterns across populations whilst preserving as far as possible the anonymity of the individuals recorded within the data. However, to the people whose life experiences make up that data set, the data subjects, each of those data points is an intimate part of who they are: their birth, education, jobs, illnesses, family, and even their death. ‘Bringing the public along with you’ in administrative data research requires the researcher to continually acknowledge that data is personal to someone. In their Consensus Statement on Public Involvement and Engagement in Data-Intensive Health Research, Aitken and colleagues discuss how PI&E plays a critical role in “bridging a gap” between researchers and the people to whom their data relates, as well as between people and data that is held about them [13]. This is crucial for making the case for PI&E in data-focused research generally, and underpins the ADRC NI approach as well.

In best practice, the people to whom the data pertain should be included through representation at all stages of the research cycle to increase the likelihood of the generated knowledge benefiting themselves and their communities. We are working towards a model that builds public involvement (with the general public as well as their representatives in the VCSE sector) as well as engagement into the cycle of our work.

Experts by experience

An important component of ADRC NI’s approach is the recognition of wider sites of knowledge with equal validity to the academic. This is not only knowledge created by NGOs and community organisations², but by people representative of the research cohort, whether it be demographic or characteristic. They are known as Experts by Experience (EBEs) and can be defined as people with lived experience of the service being evaluated or the topic or issue being researched³. Though a typically under-utilised resource within academia, particularly in quantitative research, inclusion of EBEs in data-focused research helps ‘get under the skin’ of data. EBE perspectives enrich the research by providing an enhanced and experiential understanding of the ‘problem’ being investigated, the articulation of the data analysis and its interpretation into policy responses.

ADRC NI’s approach acknowledges that those with lived experience of the issues being investigated, for example care-experienced young when examining the mental health and mortality of looked after children, are the ‘pinnacle’ of involvement in data research. We also place value on working with VCSE and NGO groups as they provide advocacy, support and campaigning work on behalf of and with these marginalised groups. It is often through them that essential contacts are made with communities.

Without the support and expertise of VCSEs/NGOs, who are trusted and respected as intermediaries, academics or research engagement professionals would likely find it difficult to engage in a meaningful way with these diverse communities and the path to their direct involvement would be much hampered.

New advocates for data research

The value and relevance of administrative data research becomes evident when VCSE groups have been included within research teams to co-produce research questions and interpret results, their shared ownership over the process, including ‘access’ to data, and producing evidence more relevant to their organisational needs (and those of the populations on whose behalf they work). This can cascade into advocacy with other VCSEs, and, crucially, to government departments and elected representatives. Constituents of these VCSE groups may also begin to advocate for further data use, as it becomes clearer the utility of opening up data access to researchers to illuminate issues that people may already be anecdotally aware or convinced of in their daily lives, but for which there is a lack the empirical evidence. This culture shift towards demanding an evidence-from-data policy agenda is critical to enabling access by researchers, creating robust pathways to impact from knowledge generated to contribute to improving people’s lives.

Project Steering Committees

The ADRC NI co-production model focuses on representation and participation by diverse stakeholders on Steering Committees to influence research priorities from project initiation through to the interpretation and contextualisation of research. This approach recognises multiplicity of knowledge beyond the academic and its contribution as an adjunct to the understanding of administrative data. Steering Committee representatives are drawn from government departments (as data custodians and policymakers), practitioners or service providers, VCSE organisations (as domain experts), and representation from Northern Ireland Commissioners’ offices⁴.

Targeted events for knowledge exchange

This model of co-production, or moving towards co-production, has allowed our researchers to work closely with community organisations as well as policymakers to develop and design impactful research and knowledge exchange events. These targeted events bring together NGOs, service providers, policymakers and elected representatives, as well as members of the public, to provide a forum for discussion of research and impact beyond the academic context. The format, in which presentations of research findings are followed by a community, policy or service response, help to articulate how the evidence has been used and the impact the knowledge generated by ADRC NI research has had on the work of its partners. ADRC NI acts as the facilitator to the dissemination of the research, which is put into practice by key stakeholders.

This approach demonstrates the value of involving and engaging key constituencies early in the research process, as it builds in pathways to impacts and allows partners ownership over the narrative of the research findings. It also supports the idea of stakeholders being key advocates for data research and helping to cascade the benefits of data-focused research to their own stakeholders and partners. During the period when Northern Ireland was without a devolved government (January 2017 – January 2020) these were particularly useful to connect researchers and their evidence with decision makers and communities.

Data Workshop Series

One of the earliest initiatives ADRC NI put in place to engage with key stakeholders was the Data Workshop Series. The workshops were developed and delivered in partnership with Detail Data, a project run by the Northern Ireland Council for Voluntary Action (NICVA) and The Detail, an investigative news outlet. Workshops were thematic; themes were chosen by identifying commonalities in research from ADRC NI with issues and campaigns from NICVA’s membership. The themes were: carers and caring, mental health, and reproductive justice. Each workshop consisted of several presentations from both researchers with interest in the area and the local VCSE groups who were working on the issue, followed by facilitated group discussions to tease out where the gaps in knowledge still might be following the research proposed by academics, what issues the VCSE groups (and often their constituent public members who attended as well) saw as most critical to serving their communities, and how VCSEs could either access some of the data being used or could advocate for the data to become more readily accessible.

One of our most successful and impactful projects, on factors affecting maternal employment, grew out of engagement at the reproductive health and justice workshop in the Data Workshop Series (discussed below).

Public Engagement, Communications and Knowledge Exchange Manager

Underpinning the public engagement and involvement aspects of ADRC NI’s research programme is the role of the Public Engagement, Communications and Impact Manager. As steward of public engagement at ADRC NI, the PECI Manager’s remit is to incorporate public engagement and involvement at every stage of the research and impact processes. This clearly defined role and remit demonstrates institutional commitment to public engagement which is key to securing buy-in from researchers as well [9].

Discussion

There are strengths and limitations to the ADRC NI approach to public involvement and engagement. If we examine our approach against the principles laid out for PI&E in data-intensive health research in the Consensus Statement on Public Involvement and Engagement in Data-Intensive Health Research [11] we find that ADRC NI’s approach fulfils most, namely institutional buy-in, clarity of purpose, transparency, involving two-way communication, being ongoing, and being designed to produce impact. Where we have room to expand and improve our activities is under accessibility to broad publics and evaluation.

As noted in earlier sections of this paper, we rely heavily on representatives of the public, namely VCSE organisations, on specific issues. An increasingly participatory approach would also offer space to publics, specifically those experts by experience whose input is invaluable (for example care-experienced young people themselves rather than VCSE groups working on the issue for our research into looked after children), as well as a forum for generally-interested publics wanting to engage with administrative data research generally. While more work is needed to move from engagement with VCSEs, as representative of publics, to these publics themselves, where the public or segment of the public whose experiences are being explored through data-focused research are vulnerable or marginalised, a two-step approach of engaging with VCSE organisations who work with the vulnerable and marginalised populations takes cognisance of the ethical considerations needed when seeking to involve these communities directly in research. Working with VCSE groups is necessary to gain trust and pave the way to full participation. This is particularly true of vulnerable groups where it would be not only fruitless in trying to involve them without prior relationship-building but would arguably bring up ethical implications as well [14].

A more formal evaluation process would also be beneficial both from the point of improving and enhancing practice but also to increase transparency and provide opportunities for publics to input into the direction and delivery of engagement activities.

Engaging with public representatives provides a potential to enrich research programmes, building in a wide range of perspectives, supporting data access, and amplifying reach of research beyond academic and government silos. It provides more opportunities to engage directly with the general public through trusted VCSEs and NGOs, but does not guarantee it.

A commonly perceived wisdom is that to achieve policy impact requires explicit government buy-in and support. Some government departments have public representative groups with whom they conduct engagement and consultation exercises, to the exclusion of others. In turn, some VCSE groups may be hesitant to, or formally reject, close association with government departments as a challenge to their autonomy and representativeness in challenging policy that affects their constituent communities.

In reflection of this, attaining true representation within research steering committees whilst preserving the independence of the research can present challenges in maintaining departmental approval for data access, whilst sustaining true engagement with NGOs, public service providers and publics as drivers for diversity of impact and dissemination opportunities. Our research Steering Committees at present lack ‘general’ public involvement, though as previously mentioned it is a trajectory of the ADRC NI’s approach to incorporate this into the model – whilst providing evidence of the benefit of engagement through targeted relationship building with representative groups.

All project steering committees for ADRC NI have at least one non-governmental representative, demonstrating that a balance can be achieved. Through a model of co-production the research process can be purposed to generate targeted outputs which demonstrate the value of participation for each stakeholder. These are typically diverse, but must be viable, reflecting the capacity and resources available to the project, ensuring participation expectations are met.

Governmental representatives are often focused on receiving evidence in the form of reports or policy briefings, VCSE groups are keen to augment anecdotal experiences encountered by constituent communities with statistical evidence. In addition to academic publications, these knowledge resources enable a multidimensional influence and outreach providing substantial return for research funders, many of which may be publicly funded. This again emphasizes the need to renew commitment to involvement of the public, both directly and through NGO representation, to provide some accountability for the use of public funds and public data and enhance transparency.

Examples of success

Example 1: Outcomes for looked after children in Northern Ireland

This project is the UK’s first historical, population-wide cohort of people known to social services as children. Covering a 30-year period, the linked dataset from this project is allowing researchers to better understand the implications of being in care on health and mortality, with ambitions to investigate educational, social and economic outcomes in the future.

The project Steering Committee was constituted so as to ensure that the questions being asked of this complex and sensitive dataset would create evidence to assist policymakers and service providers to improve the lives of young people in care. Officials from the Northern Ireland Department of Health, who have policy responsibility for looked after children, and Barnardo’s NI, an NGO that provides residential care services and works directly with care-experienced young people, were invited to sit on the project Committee. By working closely with departmental officials, the lead researcher, Dr Aideen Maguire of the Centre for Public Health at Queen’s University Belfast, is able to gain a better understanding of the dataset and any irregularities or quirks inherent in it, and is also able to add new research questions or data analyses that the Department needs in order to inform robust policy making. Similarly, involving Barnardo’s means that the issues they encounter in grassroots work with young people can potentially be explored through the data, providing evidence for the experiences young people are having, and giving them a more solid basis on which to make representations to decisionmakers on changes they would like to see occur that would directly impact the lives of young people in care.

Further work directly involving care experienced young people is planned. This workshop with care experienced young people and their representative organisations including Voices of Young People in Care (VOYPIC), Include Youth, Barnardo’s NI and the Children’s Law Centre, aims to help the attendees understand the results of the project and data research are more generally. A key function of the event is to facilitate discussion, supporting care-experienced young people to identify what future research may be a priority for the ADRC NI.

Example 2: Research into factors affecting maternal employment

During the Administrative Data Research Network (ADRN) investment ADRC NI conducted research into factors affecting maternal employment, led by Dr Corina Miller of the Centre for Public Health at Queen’s University Belfast. Public engagement, in the form of conversations with community organisations and Dr Miller’s participation in the Data Workshop Series thematic event on reproductive health and justice (as discussed above), played a crucial early role in the development and direction of the project, making it more responsive to the core issues identified by the Childcare for All Campaign, a coalition of VCSEs, childcare providers and individuals accessing (or wanting to access) childcare. Through this engagement, Dr Miller was able to access data curated by Employers for Childcare (an NGO which advocates for universal childcare and which also runs a childcare voucher scheme) from their annual survey for parents and childcare providers on experiences and views of childcare provision in Northern Ireland.

The research was launched at a Childcare Policy Conference developed and delivered in partnership with Childcare for All in March 2019. This was well-attended by community groups, childcare providers, health and social care and education organisations and departmental officials, and elected representatives. As many of the women present were childcare users themselves, the authors consider that this event had a level involvement of the data subjects of the research, something which we intend to expand and formalise in future work on this topic, employing the learning we have gained in engagement activities to apply to involvement activities as well.

Following this conference, an All-Party Group on Childcare and Early Years was established at the newly re-convened devolved Stormont Assembly, and Dr Miller and the ADRC NI PECI Manager regularly attend these meetings. Dr Miller will be presenting her findings to MLAs and departmental officials responsible for drafting the next Childcare Strategy in November 2020, which will inform the Childcare Strategy and its accompanying legislation.

Without the early engagement work done on this project, it is unlikely that the project team would have built up the trust and developed the relationships needed to craft research so directly responsive to grassroots needs, as well as access such a high-impact forum such as the APG where membership is open only to MLAs and groups working in the field, and put ADRC NI research in a position to positively impact a crucial forthcoming piece of strategy and legislation that will impact the lives of those whose experiences directly made up the research dataset.

Future plans

The successes of ADRC NI’s approach to public engagement (and movement towards explicit involvement) demonstrate how PI&E can act as a salve on publics’ concerns around data usage, precisely at the time when data usage and analysis by both government and private companies is becoming ubiquitous. Nowhere is the power of data and its potential privacy implications going to be more pressing than during and after the ongoing COVID-19 pandemic. In several devolved UK regions with mature data infrastructures, such as the Secure Anonymised Information Linkage (SAIL) Databank in Wales, decision makers are already being equipped with the latest information needed to analyse sensitive issues like the R rate, mortality and spread of the deadly virus, as well as potential economic and social impacts on vulnerable and marginalised populations. Others without such structures are left playing catch up, and subject to public frustration over the lack of publicly scrutinisible data and concerns over how equipped government is to make crucial decisions.

Sharing and use of health and other government data to support decisionmakers at this critical time has demonstrated its utility to publics, but the battle ahead will be between privacy and public benefit. Robust, grassroots public involvement and engagement will be crucial to maintain the social licence to link and use administrative data for research purposes. Without public trust in the organisations and people doing this work, public acceptance of widespread use of their personal, sensitive data may start to be withdrawn, jeopardising our ability to track and understand patterns within populations, just at a time when they are most needed.

ADRC NI, like most other data research centres, sees a large amount of COVID-19 work in its future. The entire world needs answers, and in the days, weeks, months and years to come, the questions will become even more pressing as the world slides from global pandemic into global economic recession, with expected wide-ranging impacts on vulnerable and oppressed populations. In order to provide policymakers with evidence to support and protect the hardest hit, researchers will have to integrate PI&E work into their research. ADRC NI’s work demonstrates how this can be done at all levels and intends to develop even deeper relationships with its stakeholders as COVID-19 work is further developed, affirming and acknowledging the personal tragedies behind each individual data point. This is the delicacy of the situation of data-focussed research at present and moving forward.

Conclusion

By focusing on engaging and involving data subjects, incorporating and valuing experts by experience, and working with publics as key advocates for data-focused research, ADRC NI’s model of coproduction pursues outcomes that are valued by each participant so that when these outcomes are produced, impact is more easily attained and long-lasting, and the benefits from the work are shared across the spheres of academia, government, community and service provision. This integrative approach will be vital to the post-COVID-19 world, where privacy implications and public health and economic concerns are destined to clash without robust public engagement and involvement.

Acknowledgements

The authors acknowledge the support of the staff of the Administrative Data Research Centre Northern Ireland (ADRC NI), without whom this work would not be possible.

The authors also acknowledge and thank the ESRC as funders of the ADRC NI and its wider partnership, ADR UK.

Footnotes

1. 1

   1 Care.data was an initiative to develop a joint database between NHS hospitals and GP practices in England. Led by NHS England and the former Health and Social Care Information Centre (HSCIC), now NHS Digital, its purpose was to collect, de-identify and link datasets across the NHS. However, public objections to the ability of commercial enterprises such as insurance companies to access the data, as well as what was seen as poor public engagement, and ethical concerns from some quarters, led to the project being dropped in July 2016.

2. 2

   2 NGOs and VCSEs regularly produce policy and research reports in their area of expertise, which are used to lobby government and raise awareness of issues facing their constituent communities. These can be created by the NGO alone or in partnership with research and community groups and can also take the form of consultation responses. Some examples:Nelson E, McBride R-S, O’Riordan O, & Smyth P (2010). Beyond the margins: Building trust in policing and young people. Belfast, UK: Achieve Enterprises and Institute for Conflict Research.Northern Ireland Commissioner for Children and Young People (NICCY) (2018). ‘Still Waiting’: A Rights-based review of mental health services for children and young people in Northern Ireland.’Women’s Ad Hoc Policy Group (2020). Hate Crime Legislation in Northern Ireland Independent Review – Consultation Response.

3. 3

   3 This is a broad definition of EBEs, which are used in various different contexts including but not limited to service design and evaluation, research, peer support, and inspections.

4. 4

   4 As in England, Scotland and Wales, Northern Ireland has many Commissioners Offices that provide independent oversight in critical and sensitive areas; for example the NI Children’s Commissioner (NICCY), the Commissioner for Older People NI (COPNI), the Equality Commission (with a Chief Commissioner and other Commissioners), the Northern Ireland Human Rights Commission (also with a Chief Commissioner and supporting Commissioners), the NI Victims Commissioner, etc. ADRC NI has worked directly NICCY on research into looked after children, and the Equality Commission on migrants and health inequalities. A representative from COPNI sits on our Steering Committee for the SIP on ageing and access to healthcare.

References

1. Care.data: How did it go so wrong? https://www.bbc.co.uk/news/health-26259101 accessed 2021.5.20 and 14.10.20

2. National Data Guardian for Health and Social Care, Review of Data Security, Consent and Opt-outs, 2016

3. Digital Economy Act 2017, http://www.legislation.gov.uk/ukpga/2017/30/contents/enacted

4. The Guardian, Cambridge Analytica Files: https://www.theguardian.com/news/series/cambridge-analytica-files

5. National Institute for Health Research School for Primary Care Research (2020) What is Patient and Public Involvement and Public Engagement? https://www.spcr.nihr.ac.uk/PPI/what-is-patient-and-public-involvement-and-engagement

6. INVOLVE https://www.invo.org.uk/find-out-more/what-is-public-involvement-in-research-2/

7. Arnstein, SR (1969) A Ladder Of Citizen Participation, Journal of the American Planning Association, 35: 4, 216-224 doi: 10.1080/01944366908977225

   https://doi.org/10.1080/01944366908977225

8. Jones, KH, Heys S, Thompson R et al (2020) Public involvement and engagement in the work of a data safe haven: a case study of the SAIL Databank. IJPDS 5:3 10.23889/ijpds.v5i3.1371

   https://doi.org/10.23889/ijpds.v5i3.1371

9. Verschure B, Brandsen T, Peskoff V (2012). Co-production: the state of the art in research and the future agenda. Voluntas 23:1083–1101 10.1007/s11266-012-9307-8

   https://doi.org/10.1007/s11266-012-9307-8

10. Parks, R. B., et al. (1981). Consumers as co-producers of public services: Some economic and institutional considerations. Policy Studies Journal, 9(7), 1001–1011.

11. Holmes, B (2017). On the co-production of research: why we should say what we mean, mean what we say, and learn as we go. London School of Economics Blog 21 September 2017. https://blogs.lse.ac.uk/impactofsocialsciences/2017/09/21/on-the-co-production-of-research-why-we-should-say-what-we-mean-mean-what-we-say-and-learn-as-we-go/

12. Donovan C and Hanney S (2011). The ‘Payback Framework’ explained. Research Evaluation, 20(3), September 2011, pages 181–183 10.3152/095820211X13118583635756; http://www.ingentaconnect.com/content/beech/rev

    https://doi.org/10.3152/095820211X13118583635756

13. Aitken M, Tully, MP, Porteous C, et al (2019) Consensus Statement on Public Involvement and Engagement with Data-Intensive Health Research. IJPDS, 4:1:06 doi: 10.23889/ijpds.v4i1.586

    https://doi.org/10.23889/ijpds.v4i1.586
14. Oxford and New York: Routledge Press; 2019.