Tag Archives: evaluation

Shining a spotlight on Evaluations Online: Scotland’s essential economic development resource

Posted on by

Image: Marcus Winkler (via Unsplash CC)

The UK is currently at the beginning of what is expected to be the deepest recession in living memory. From a policy point of view, governments around the world are facing the daunting task of navigating a route through uncharted territory. As the recently launched cross-institutional Economics Observatory noted last month, “sound and non-partisan advice is needed to inform decision-makers across all parts of society, about the choices they face in dealing with the crisis and the recovery”.

Key role of economic development and sustainability in the Covid-19 recovery

As statistical analysis suggests that Scotland’s GDP fell 18.9% during the month of April, and that in May output remains 22.1% below the level in February, the need for a recovery approach that is based on empowering regions, cities and local communities is clear.

The independent Advisory Group established by the Scottish Government to advise on Scotland’s economic recovery in the wake of the COVID-19 pandemic, published its report at the end of June. This identified interventions to support Scotland’s economic recovery within the context of the strategic goal of shifting to a greener, fairer and more inclusive economy with wellbeing at its heart.

New economic development initiatives and programmes in response to the pandemic have already been launched in Scotland. Some are focusing on helping specific sectors such as tourism and the creative industries. There is also a recognition that it is important during the recovery to build on current strengths, such as inward investment and low-carbon technologies.

What works in economic development

Here at the Knowledge Exchange, we’re committed to supporting the use of evidence to inform policy development and practice. So in the light of the current importance of economic development, we thought we’d highlight a useful resource which makes available the results of evaluation work and research in order to enhance decision-making and investment in the future.

Evaluations Online is a public portal providing access to a collection of evaluation and economic development research reports commissioned by Scottish Enterprise, Scotland’s main economic development agency.

Ensuring that public investment generates economic and social benefits, and long-term inclusive growth for Scotland is core to Scottish Enterprise’s remit. Making evaluation and research reports publicly available supports this aim, as well as ensuring transparency.

Established for over a decade, the site now contains over 750 research and evaluation reports dealing with different aspects of economic development activity, such as business support, investment, sector growth and improving skills. All of the reports are publicly accessible and free to access.

Learning lessons from previous programmes

Developing the economic response to Covid-19 is happening at a much faster pace than usual policy-making cycles. It is important, though, that spending and investment is focused on areas that will have most impact, and will also contribute to the overall goals of supporting jobs, protecting and progressing education and skills, and tackling inequality. Considering lessons from previous interventions when commissioning new projects or allocating funding, is one way to address effectiveness.

It’s worth repeating that repositories of evidence can help bring about better policy in a number of ways:

  • improving accountability by making it easier for people to scrutinise the activities and spending of public sector organisations;
  • improving the visibility and therefore the impact of evidence;
  • helping identify gaps in evidence by making it easier to compare research findings; and
  • increasing our understanding of what works (‘good practice’), not only in the activities covered, but also in evaluation and research methods.

Evaluations Online offers resources in key areas such as entrepreneurship, regeneration, social enterprise, economic inclusion, skills development, financing, inward investment and commercialisation, as well as by sector. In recent years, questions about inclusive growth and generating social value have also become more important policy issues.

Some of the most popular recent reports added to the site have focused on:

It’s clear that there are huge sectoral and regional challenges within the economy which will need faced immediately and in the longer term, as a result of Covid-19. Business practices have changed, as have all our lives. But we believe that the use of evidence and research will be fundamental in successful recovery and the transition towards a greener, net-zero and wellbeing economy.

The Knowledge Exchange work with Scottish Enterprise to manage the Evaluations Online portal.

Evaluations Online is a publicly accessible collection of evaluation and research reports from Scottish Enterprise. The reports cover all aspects of Scottish Enterprise’s economic development activities and are available for download at no cost.

Focus on: Evaluations Online

Posted on by

 

Evaluations Online is a public portal providing access to a collection of evaluation and economic development research reports commissioned by Scottish Enterprise, Scotland’s main economic development agency.

Ensuring that public investment generates economic and social benefits, and long-term inclusive growth for Scotland is core to Scottish Enterprise’s remit. Making evaluation and research reports publicly available, supports this aim as well as ensuring transparency.

Some of the most popular recent reports added to the site have focused on:

Working in partnership

Since 2007, Idox has been working with Scottish Enterprise to deliver Evaluations Online using a publishing platform designed specifically to deal with research material. Users can easily navigate to and assess the relevance of material thanks to specially-written abstracts and structured search functions based on a bespoke classification and record structure.

The site now contains over 600 evaluation and research reports commissioned by Scottish Enterprise, dealing with different aspects of economic development activity such as business support, investment, sector growth and improving skills. All of the reports are publicly accessible and free to access.

Since the site launched we have continued to refresh and improve the site, ensuring it better meets the needs of key user groups, including economic development policy-makers and practitioners across Scotland. In the last quarter of 2016, the reports hosted on the site were accessed over 30,000 times.

The importance of evaluation

We’ve highlighted the importance of evidence and evaluation and assessment of information quality on the blog several times before. It’s worth repeating that repositories of evidence can help bring about better policy in a number of ways:

  • improving accountability by making it easier for people to scrutinise the activities and spending of public sector organisations – this helps organisations meet Freedom of Information responsibilities;
  • improving the visibility and therefore the impact of evidence;
  • helping identify gaps in evidence by making it easier to compare research findings; and
  • increasing our understanding of what works (‘good practice’), not only in the activities covered, but also in evaluation and research methods.

We’re proud to support Scottish Enterprise in the dissemination of their evaluation and research output, through a portal which they believe increases the return on these activities.

You can find out more about the projects The Knowledge Exchange team has been involved in, and the consultancy services we offer, here.

Evaluations Online: evaluating economic development activity in Scotland

Posted on by

by Stacey Dingwall

Recently we profiled Research Online, one of the two research portals managed by the Knowledge Exchange team. In this blog, we focus on Evaluations Online.

Economic development activity in Scotland

Evaluations Online is a public portal providing access to a collection of evaluation and economic development research reports commissioned by Scottish Enterprise. Scottish Enterprise is Scotland’s main economic development agency and a non-departmental public body of the Scottish Government.

Idox won the contract to deliver Evaluations Online in 2007. The team developed a site which utilises a publishing platform designed specifically to deal with research material. Users can easily navigate to and assess the relevance of material thanks to specially-written abstracts and structured search functions based on a bespoke classification and record structure.

The site now contains over 500 evaluation and research reports commissioned by Scottish Enterprise, dealing with different aspects of economic development activity such as business support, investment, sector growth and improving skills. All of the reports are publicly accessible and free to access.

In 2011, the team won a further contract to refresh and improve the site, focusing on how the site could be refined to better meet the needs of key user groups including economic development policy-makers and practitioners across Scotland. In the last quarter of 2016, the reports hosted on the site were accessed over 30,000 times.

The importance of evaluation

One of the key reports hosted on Evaluation Online is the annual review of the risk capital market in Scotland. Scottish Enterprise commissions the report annually in order to consider the scale of new investment flows. The findings are also used to inform the nature of Scottish Enterprise interventions in the Scottish early stage risk capital market, such as the Scottish Co-Investment Fund and Scottish Venture Fund.

Scottish Enterprise commissions evaluations of projects and programmes each year in order to identify their contribution towards economic growth in Scotland, and particularly in terms of their impact on gross value added (GVA) and employment. As the findings of the evaluations inform decisions about public spending, it’s important that all of the appraisal and evaluation work is of a high technical standard.

We’ve highlighted the importance of evidence and evaluation on the blog several times before. It’s worth repeating that repositories of evidence can help bring about better policy in a number of ways:

  • improve accountability by making it easier for people to scrutinise the activities and spending of public sector organisations – this helps organisations meet Freedom of Information responsibilities;
  • improve the visibility and therefore the impact of evidence;
  • help identify gaps in evidence by making it easier to compare research findings; and
  • increase our understanding of what works (‘good practice’), not only in the activities covered, but also in evaluation and research methods.

We’re proud to support Scottish Enterprise in the dissemination of their evaluation and research output, through a portal which they believe increases the return on these activities.

You can find out more about the projects The Knowledge Exchange team has been involved in, and the consultancy services we offer, here.

 

The Men’s Sheds revolution spreading around the world

Posted on by

img_5079

by Stacey Dingwall

Last week I attended ‘Men’s Sheds: the movement in Scotland and the big picture internationally’, an event, organised by the Centre for Research & Development in Adult and Lifelong Learning (CR&DALL) at the University of Glasgow.

Our blog on the Men’s Sheds movement was one of our most popular last year. The movement originated in Victoria, Australia in the 1990s, as a place for men to socialise and take part in practical activities. 23 years later, there are now close to 1,000 such spaces in Australia. Sheds have also proven popular in Ireland (350 Sheds and counting) and Scotland (at least 38 up and running, with 30 in the start-up phase).

Research has indicated that loneliness and isolation are a particular issue for certain groups of men, which is reflected in higher suicide rates. Evaluations of Men’s Sheds have found participation to have a range of positive effects for these groups of men, predominantly in terms of their mental health and wellbeing.

The movement in Scotland …

The first speaker of the day was Willie Whitelaw, Secretary of the Scottish Men’s Sheds Association (SMSA). Willie highlighted two key points, which were themes throughout the rest of the afternoon:

  • The importance of Sheds not being regulated by outside agencies, e.g. government – this was something that those involved in Sheds felt particularly strong about. As noted by Professor Mike Osborne, the Director of CR&DALL, at the start of the afternoon, the reduction in government support for adult education has created a need for people to organise themselves in order to access lifelong learning opportunities. Thus, those who attend Sheds feel strongly about preserving the independence of the space, as well as its democratic dynamic.
  • How to ensure the sustainability of Sheds, and community projects in general – Willie described how the SMSA can support Sheds across Scotland by offering advice on applying for funding, how to keep things like rental costs low, and using mechanisms such as the Community Empowerment Bill and Community Asset Transfers to their advantage. Noting the difficulty that many community projects face in sustaining themselves long-term, Willie highlighted the Clydebank Independent Resource Centre (CIRC), which has been running for over 40 years, as a rare but good example of how sustainability can be achieved.

img_5077

…and the big picture internationally

The second speaker of the day was Professor Barry Golding from Federation University Australia. Barry is the most prolific researcher in the area of Men’s Sheds, and published The Men’s Shed Movement: The Company of Men last year. Barry described the origins of the movement in Australia, and suggested it took off due to its provision of the three key things that men need: somewhere to go, something to do, and someone to talk to.

Barry also emphasised the importance of not formalising Men’s Sheds, and particularly not promoting the spaces as somewhere where men with health issues go (not a very attractive prospect to an outsider!) This point was also picked up by David Helmers, CEO of the Australian Men’s Shed’s Association. David described the experience of one Australian Shed who had a busload of patients arrive after being referred by health services. The point of the Shed is to create a third space for men (other than home or work) where they can relax and socialise with their peers. Any learning or health improvements that arise from this is coincidental and not forced.

Barry and David were followed by John Evoy of the International Men’s Shed Organisation (IMSO). John focused on the experience of Sheds in Ireland, noting the impact of the recession as a particular reason why the movement has taken off in Ireland. The IMSO’s aim is to support a million men through Sheds by 2022.

img_5080

Strengthening the movement and using evidence

To finish the afternoon, two panels comprising Shed members and researchers considered the questions of how to strengthen and sustain the Men’s Sheds movement, and how research might be beneficial to this.

Shed members on the panel and in the audience suggested that changing the stereotype of Sheds as spaces for older men with health (particularly mental) issues is important. In fact, men of any age are welcome to attend their local Shed, and current members are particularly keen to encourage this in order to support the intergenerational transmission of practical skills that are otherwise at risk of being lost.

In terms of available evidence, it was noted that research on Men’s Sheds is still scarce, and focused on the Australian experience. Catherine Lido, a lecturer in psychology in the university’s School of Education, discussed the pros and cons of carrying out a systematic evaluation of the movement in the UK. Again, the importance of the democratic nature of Sheds was raised – allowing outside agencies, particularly government, to come in and carry out research would involve the loss of some control. Any research conducted would have to be participatory, in order that Shed members did not feel like they were the subject of an ‘experiment’. Barry Golding highlighted, however, that there is currently almost no data on UK Sheds available; rectifying this could strengthen Sheds’ chances of being successful in applications for funding to support their running costs.

If you enjoyed reading this, you may also be interested in our previous blog on ‘makerspaces‘, which have drawn comparisons with the Men’s Sheds movement.

Follow us on Twitter to see what developments in public and social policy are interesting our research team.

In support of qualitative research: the value of qualitative insight for policy formation

Posted on by

For many people, working in the tangible, measurable area of hard figures provides an element of certainty to decision making processes. It is perhaps for this reason that quantitative research was, for many years, the largely uncontested preference for decision makers and people looking for research to evidence their decisions.research

Some criticisms that are often made of qualitative research are that it:

  • can be limited in scope and size and have a longer turnaround time than quantitative studies;
  • can be difficult to replicate and scale to achieve multiple results across multiple test sites
  • can be too reliant on researcher interpretation, perception and experience, and therefore too exposed to bias and unreliability.

In contrast, quantitative research is stereotypically presented as producing results that are consistent and replicable; and therefore ‘higher quality’ and ‘more valid’.

A question of quality

Qualitative research has always suffered from a reputatio of being less rigorous. Instead of dealing with empirical data, it deals with the more human side of research and the effects of a programme. It questions the reasoning of understanding, and the emotional implications of an intervention. As a result, qualitative researchers approach their subject from an entirely different epistemological standpoint (i.e. they have a different view of what ‘knowledge’ is, what should be judged as evidence, and what should not).

This challenges the understanding of what is meant by “research standards”. While quantitative researchers base their understanding on demonstrable results which can be proven and replicated to the same standard, qualitative research brings to the fore questions of researcher subjectivity, the concepts of validity and reliability of results and questions of ethics. It also stresses the importance not only of measuring information to gain results, but gaining results through examining and interpreting experiences and social contexts. It considers these social factors on policy outcomes rather than by categorical measurement using a predefined scale.

A mixed methods approach

While this traditional dichotomy between qualitative and quantitative research methods is convenient, in practice many organisations value the social dimension especially when looking at understanding the impact of policy interventions.

In fact many local authorities and public sector bodies, including the Greater London Authority (GLA), are increasingly looking to qualitative researchers to form part of their wider research teams. In a policy context, it is clear that qualitative research has its place alongside quantitative research as part of a mixed methods approach to evidence based policy making.

Some of the things that qualitative research brings to policy research are: a flexible research method (the methods of collection and analysis can easily be changed as the research is being conducted and the data emerges); and very rich data (if done correctly, one study could provide research data for a number of research tasks).

It allows for a deeper understanding of what lies behind results – not just that something has had an impact, but why. It also allows researchers to understand social phenomena from an individual perspective and consider the specific contexts and conditions which have contributed to it (for example, the experience of stigma or discrimination).group-discussion_unsplash

Giving a voice to marginalised groups

Qualitative research is also finding a role in evaluative research teams, looking at the meanings and constructions which made an intervention effective or ineffective and potential steps to make it more successful in the future.

One example is in engaging with hard-to-reach or marginalised people. While quantitative research would tell you that certain groups of people engage less frequently in community groups for example, qualitative research would help to explore what motivates people to engage, and therefore tease out potential methods to increase engagement. Qualitative research also allows for bespoke research questions on niche topics to be created and explored thoroughly. This can be useful for local authorities who wish to explore issues at a local level with specific communities which would not necessarily be distinct within wider national quantitative data sets and statistics.

Supporting the personalisation agenda with bespoke research

Qualitative research is also becoming more popular as a supplementary option to hard data and raw statistics because of the increased importance which is being placed on individual experience and personalisation in public services, particularly, but not exclusively within health and social care. Qualitative data allows researchers to get an in-depth view of how people experience services.

While it will never be a replacement for the empirical data produced by quantitative data, qualitative data brings its own benefits and enhances understanding around policies in a way that hard data often cannot. It encourages professionals to think beyond figures as a benchmark for outcomes. It also allows them to gain rich data on the experiences of marginalised groups in society, who often go unrepresented in large national quantitative data sets.

Follow us on Twitter to see what developments in public and social policy are interesting our research team.

 

Evidence for the world we want – International Year of Evaluation

Posted on by

evaluation cycle 2015 has been declared the International Year of Evaluation, by a global movement of international partners seeking to enhance the capacity of Civil Society Organisations to influence policymakers and public opinion, and ensure public policies are based on evidence.

Our latest In Focus briefing considers the role of evaluation; the role of evaluation in programme planning; value for money; social return on investment; international experience; UK approaches; and the ethics of evaluation.

The primary purpose of evaluation is offering a way of determining whether a programme, project or initiative has been a worthwhile investment. It can help to shape and improve current initiatives as a means of reflection, correcting problems and finding what works.

However, there are many challenges to be overcome in carrying out evaluation such as:

  • engagement
  • bias
  • prejudice
  • setting realistic expectations
  • clear purpose and audience
  • lack of information and evidence

To find out more download the briefing here.

Find out more about the 2015 International Year of Evaluation here.

Read our recent blog on why using UK-sourced evidence when making policy or practice decisions is important.

Become a member of the Idox Information Service now, to access a wealth of further information on evaluation including government guidelines, best practice and examples of evaluations. Contact us for more details.

 

Enterprise Zones … did they work then, will they work now?

Posted on by

Modern office building

by Laura Dobie

In this article we look at past and present incarnations of Enterprise Zones, their potential to create jobs and growth, and issues to address in policy approaches.

Past experiences and lessons learned

The Enterprise Zone concept emerged in the UK in the 1970s. They were conceived by the planning academic Peter Hall, with the idea that removing all obstacles faced by businesses, such as regulation and bureaucracy, would allow enterprise to prosper, prompting a surge in the number of companies, employment levels and incomes in areas which had been ravaged by industrial decline and restructuring.

Enterprise Zones were created in the UK between 1981 and 1996 and were mostly concentrated in areas of post-industrial decline on the outskirts of towns and cities. The policy, as it was implemented, departed somewhat from the original, free-for-all vision: the zones concentrated on built environment challenges and the use of capital-based grants and rebates to drive growth.

The incentives offered included:

  • 100 percent tax allowances for capital expenditure on constructing, improving or extending commercial or industrial buildings;
  • Exemption from Business Rates for industrial and commercial premises;
  • Simplified planning procedures;
  • Exemption from industrial training levies;
  • Faster processing of applications for firms requiring warehousing free of Customs duties;
  • A reduction in government requests for statistical information.

The Department for Environment’s final evaluation found that approximately 126,000 jobs were created, of which up to 58,000 were additional, and that additionality was highest for manufacturing and lowest for retailing and distribution activity. It estimated that the cost per additional job created was around £17,000 (£26,000 at current prices), assuming a ten year job life, and that over than £2 billion (1994/95 prices) of private capital was invested in property on the Enterprise Zones, a public to private leverage ratio of about 1 to 2.3.

However, Enterprise Zones have been subject to much criticism, and it has been argued that, overall, they did not produce a lasting recovery in investment and employment. (Danson, 2013, p17). Critics have highlighted the displacement effects of Enterprise Zones, i.e. that many jobs created in Enterprise Zones were displaced from other areas (Sissons and Brown, 2011), and the expensiveness of the policy (Larkin and Wilcox, 2011).

Continue reading →

Improving attainment … what recent evaluations tell us

Posted on by
evaluation

Image from Flickr user Beth Kanter, licensed for reuse under a Creative Commons License

by Stacey Dingwall

The Education Endowment Foundation (EEF) is an independent charity that works to break the link between family income and educational achievement, ensuring that children from all backgrounds can fulfil their potential and make the most of their talents. Funded by the Sutton Trust and Department of Education, the EEF aims to raise the attainment of children facing disadvantage by:

  • Identifying and funding promising educational innovations that address the needs of disadvantaged children in primary and secondary schools in England.
  • Evaluating these innovations to extend and secure the evidence on what works and can be made to work at scale.
  • Encouraging schools, government, charities, and others to apply evidence and adopt innovations found to be effective.

At the beginning of October, the EEF published reports outlining findings from seven of their projects. Writing on the EEF blog, Dr Kevan Collins explained that “we investigate the strategies schools use to improve their students’ attainment. We use independent evaluators to assess whether these methods really make a difference, with a focus on their impact on pupils eligible for free school meals”. On the National Foundation for Education Research’s (NFER) blog, Marian Sainsbury praised the work of the EEF in the area of evaluation, particularly their use of the randomised control trial (RCT) method of evaluation, a method which she argued is not used often enough in education.

One of the EEF’s evaluation reports received particular attention in the media: an evaluation of the ‘Increasing Pupil Motivation’ initiative, which aimed to improve attainment at GCSE level by providing incentives to Year 11 pupils in England. The initiative was largely targeted at relatively deprived schools, and offered pupils two incentives: one financial, in which the amount was reduced if they did not maintain standards in four measures of effort (attendance, behaviour, classwork and homework); and the other offered the incentive of a trip or event, under which pupils were allocated a certain number of tickets, to be reduced if they did not meet the four measures of effort.

Despite previous research indicating that the provision of incentives directly for test scores had a positive effect, the EEF evaluation of the provision of incentives specifically for effort found no significant positive impact of either type of incentive on GCSE attainment in English, maths or science. The findings did, however, suggest that the financial incentive had a significant positive impact on classwork effort in each of the three subjects, while both incentives had a positive impact on GCSE maths for pupils with low levels of prior attainment.

Findings from evaluations of another six initiatives were also published:

Continue reading →

Troubled families approach expands … but is the evidence there?

Posted on by

paper family on handBy Dorothy Laing

Nobody likes the idea of experiencing antisocial behaviour on their doorsteps so further government action to help ‘troubled families’ will almost certainly be welcomed by neighbours and local communities alike.

On 19th August, Communities Secretary Eric Pickles announced an extension of the Troubled Families Programme, as findings from independent research carried out for the Department for Communities and Local Government (DCLG) – National Evaluation of the Troubled Families Programme interim report family monitoring data: a report by Ecorys UK and Understanding Troubled Families – revealed the current state of progress. Continue reading →

How collecting evidence can help improve public policy: Scottish Enterprise’s approach to economic development evaluation

Posted on by

By Stephen Lochore

EO homeThe Knowledge Exchange has helped a range of public sector organisations to collate, analyse and disseminate their evaluation and research material. One example is the Evaluations Online portal, a publicly accessible collection of evaluation and research reports from Scottish Enterprise, one of Scotland’s two regional development agencies. We’ve recently made some changes to the portal, and though these were mainly to design and layout, it got me (as Project Manager) thinking about wider issues of accountability and evidence-based policy. Continue reading →