Whose bright idea was that? How think tanks measure their effectiveness and Impact

by Sarah Bressan und Wade Hoxtell

This article is a re-published introduction to the study “Whose Bright Idea Was That?”.

What are think tanks for? What are those who work for them trying to achieve? What approaches do they use to be more effective and have impact in their priority area(s)? And how do they actually know whether they are making a difference? In an age that has many postulate a growing polarization around political questions or express worry over attacks on the legitimacy of science and expertise, these questions about the role and influence of think tanks are especially timely.

However, across the world, organizations that sit at the intersection of research and policy advice still struggle to find answers to these and other related questions. This is hardly surprising: achieving – as well as assessing and confidently claiming – effectiveness and impact are complicated endeavors. In addition, think tanks often lack the financial resources, expertise and time needed to design and implement well-conceived measurement systems. As a result, most organizations use measurement models and indicators that are not particularly useful for determining their effectiveness or impact. And unfortunately, the little guidance that does exist tends to be too broad or not fully applicable to the diverse range of think tank models out there or to the different contexts in which they operate.

To help fill this gap, this study proposes a framework that can help think tanks systematically measure and achieve impact and effectiveness. By illustrating important trends as well as challenges these organizations face, and by outlining key success factors for measuring the effectiveness and impact of the different types of work that think tanks do, it also aims to provide guidance for those who want to develop more thoughtful and tailored monitoring and evaluation systems. At the same time, we aimed to take a more nuanced look at the sector – by differentiating between think tanks based on their operating or funding models as well as their geographic, political, economic, or social contexts. Similarly, we took into account that different organizations pursue different objectives via various types of activities. Recognizing the effect of contemporary trends such as polarized political climates, a declining trust in experts or organizational development challenges, among others, we deliberately set out to not only highlight good practices in how think tanks develop effectiveness and impact indicators and how they monitor and evaluate the corresponding data, but also to provide an honest look at the challenges, both external and internal, that think tanks face as they pursue relevance in the 21st century.

A difficult endeavor

Through background research as well as interviews with think tank representatives from around the world, we found that there is a stronger focus on quantifying and qualifying the work of think tanks – and by and large this is seen as a positive development. Most think tanks find it important to consider how they can be more effective and impactful as well as how to measure this for a number of reasons, including that such information serves an accountability function, helps with learning and improving a think tank’s work and outreach, and can make an organization more competitive in a challenging influencing environment, among other benefits. 

At the same time, this emphasis on measuring effectiveness and impact – which also springs from related demands made by funders – is drawing much-needed attention to the fact that think tanks generally struggle to do this well. While most think tanks have a clearly defined theory of change and try to approximate effectiveness and impact as best as they can with the resources they have, they more often than not use indicators that, while helpful in illustrating some aspects of organizational effectiveness, say little about their actual impact. In fact, none of the think tank representatives we spoke to claimed to have a system for this that they find useful across the board, and most questioned the value of at least some of the metrics their organizations are monitoring. 

The fact is: it is difficult to measure effectiveness and impact, for a number of reasons. Causal chains and influence timelines are often long and it is tricky if not impossible to isolate the influence of one output or actor among many (the “attribution problem”). Further, organizations frequently lack the resources for monitoring and evaluation – an area considered to be outside their “core work” – and struggle to find funders who are willing to put up the money or other resources for them to do so in a more structured and professional manner. In addition, and in part because funders frequently require it, think tanks often use overly simplified linear models for conceptualizing how change happens. They are focusing on the outputs and outcomes of various activities instead of drawing up more complex and arguably more realistic non-linear models that try to also account for, for example, the role of other actors as well as medium- to long-term (intermediate) outcomes. 

No one-size-fits-all approach

There is no consensus regarding the best approach to all of this and even think tanks with a strong discussion culture around impact struggle to effectively and sustainably implement a formalized or semi-formalized measurement system. Still, some good practices have emerged. These include: 

• Building a culture of open reflection on these issues; 
• Building and maintaining strong relationships with stakeholders and clients to regularly solicit feedback on a think tank’s activities and impact; 
• Taking time to analyze successes and failures; and
• Crafting (and ideally sharing) impact stories.

In addition, even a simple – let alone a more complex and non-linear – theory of change model can serve as a useful tool for think tankers to design the appropriate activities that will help them achieve specific objectives, and to reflect on the effectiveness and impact of these activities at more strategic level using a tailored mix of quantitative and qualitative indicators.

That said, think tanks and their funders also need to be realistic regarding both the added value that an extensive measurement approach can offer and the extent to which think tanks can feasibly implement an “ideal” measurement system in light of internal or external constraints and other challenges. While well-designed and tailored monitoring systems and indicators can have a number of benefits, attempts to qualify or quantify all or even only certain aspects of a think tank’s work are no substitute for the real success factors in achieving impact, such as: 

• Producing high-quality outputs;
• Attracting and developing talented staff;
• Building a credible and sustainable organization; and again
• Fostering an organizational culture that encourages reflection and learning.

There is no one-size-fits-all approach to making an impact or measuring success. The authors of the study hope that it enriches the discussion on these issues by providing some practice-oriented guidance and actionable tools to help different types of think tanks operating in various contexts – and with different internal and external limitations – to develop theories of change, translate these into targeted activities, and better measure and evaluate the effectiveness and impact of their work. 

This study was originally published by the Global Public Policy Institute (GPPi). It was funded by the German Federal Ministry for Economic Cooperation and Development (BMZ) and supported by the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ).
 

Sarah is a research fellow at the Global Public Policy Institute (GPPi). Her work focuses on conflict prevention, evaluation, forecasting and foresight methods in foreign policy. Sarah has conducted evaluations for the World Food Program and the German Federal Foreign Office, is researching ways to improve the evaluation of extremism prevention and heads GPPi's work on strategic foresight, for which she develops methods to forecast authoritarian foreign influencing. She co-founded Better Think Tanking, a newsletter on think tank innovation and management and is an alumna of the Think Tank Lab's diversity challenge. Sarah studied political science and international security in Berlin, Paris, Mannheim and Seoul.

In addition to researching think tanks’ effectiveness and impact as well as technology-related topics, Wade Hoxtell serves as head of operations and finance at the Global Public Policy Institute (GPPi) in Berlin. His current projects include a study on combating disinformation within Europe as well as the Horizon Europe-funded research project ENSURED that examines how the EU and its members can transform and defend global governance to make it more robust, effective and democratic. Wade holds a master’s degree in European studies from Jagiellonian University in Krakow and undergraduate degrees in international relations, German language and culture, European studies, and global cultures from the University of Wisconsin-Madison.