12 februari 2013

Can we measure foresight impact?

106351691How can we meausure foresight impact? When it comes to policy-oriented foresight, like many other creative processes, proving or measuring it’s impact on policy does not come easily. Foresight professionals might recognize some of these experiences I’ve had in the past:

  •  the impact of the foresight endeavour is limited to the partitioner’s group and is hard to explain to ‘outsiders’
  • the products like scenarios and trend analyses, might be easy to communicate, but besides inspiration, how do they actually ‘change’ decisionmaking?
  • it can take a few months or even years for changes or policy decisions to be made and often the foresight exercise was not the exclusive contributor tot his occurance

My research in the scientific community reassured me that I’m not alone in my questions on how to actually prove or measure the impact of foresight. (Please not that in this post I focus on the literature on foresight for policymaking, nog foresight for commercial organisations.) The journal Foresight even devoted an entire issue on this matter in 2012.

The scientific foresight community seems to agree on the fact that proving the worth and importance of futures studies comes down to proving the impact that is made on policy making (Calof and Smith, 2012). There appears to be a void and a need for a structured and integrated framework for measuring the impact of futures studies on policy and for comparing futures studies. Since there are many different ways to conduct futures studies, it is not surprising there are various ways in which scientists have evaluated them. Many lists of benefits and functions of futures studies are presented. Strong emphasis is placed on the idea that not only the end-products (like scenarios or trend analysis), but also the process itself is valuable in futures studies.

Calof et al (2012), emphasize impact on decision making as the most important benefit of foresight: “In our collective experience, if there is not a clear link between foresight studies/work and policy/decisions, there is a low likelihood that the foresight initiative will be sustainable. While other benefits such as the development of new networks and new knowledge are valuable, in our experience and opinion the primary success measurement of foresight is decision impact. We hear this from the governments and businesses that fund foresight initiatives, we are told this by foresight practitioners and it is also consistent with the definitions provided above.”160612807

However, proving the impact of foresight on policy is very difficult. According to Calof and Smith (2012) one of the factors making measuring impact difficult is the time it often takes for results of futures studies to become evident: “Measuring impact has been identified in the foresight literature as being difficult to do. Georghiou (1998) identified several problems with measuring impact, including a major measurement issue of when impact actually does occur it can take many years for project effects to become evident. Others have noted that evaluating impact based solely on impact on policy is that the research suggests that for the most part impact has been low (Riedy, 2009).”

Another difficulty in evaluating foresight impact may be caused by the fact that most evaluations focus on a singular futures study. The Dutch Scientific Council for Government Policy (WRR), instead, evaluated a series of futures studies over time (WRR, 2010): “The meaning of one single futures study in policymaking processes is difficult to trace. Therefore, we started from the policy perspective. In various policy dossiers, we examined whether and how futures studies were used. This analysis led us to conclude that futures studies do have meaning in policymaking processes. A series of futures studies produced over a period of time by different actors may have a cumulative impact on policy. This phenomenon we call ‘beat’.”

Marvin (2000) also examined multiple futures studies, focusing on future visions for the city of Manchester, when considering the impact of foresight on ‘the city’. He also stresses the need to compare studies rather than focus on one study“What we need is a social process through which we stand these visions alongside one another and build a better understanding of the type of city they are trying to create. This could have several benefits. It creates a context for a wider debate about the multiplicity of paths to what could be quite different urban futures produced by currently disconnected groups. What we lack is an appraisal framework for comparing the social, economic, environmental, technical and spatial assumptions built into the different visions. Such a process could develop a package of techniques for urban futures thinking.”

Evidently, there is no easy answer or recipe available to measure the impact of foresight on policy (yet). When trying to measure this impact, we should keep in mind that this is not a simple endeavour. This fact is not exclusive to foresight exercies, but can be applied to the use of scientific knowledge in policymaking in general. A framework does not yet exist. However,  we should keep in mind that it often takes a series of foresight exercises, or future stories, to infleunce policymakers (or society in general for that matter) and it often takes several years to identify the true impact of the exercise(s). So a lot of work yet remains to be done.

What are your experiences with measuring the impact of foresight exercises on policy? Please share your thoughts in the comments section or email me: