The Indicators Paradox

Next week I wander up to very scenic, but very cold Jackson Hole, Wyoming to work with US Forest Service colleagues on monitoring and evaluation measures for a forest plan, related to an agency strategic plan, related to governmental obligations writ large for the US as part of the global community. Forest plan indicators must therefore, somehow, link to broader Criteria and Indicators for Sustainable Development.

I find the work paradoxical. Too often we come up with 'indicators' that don't indicate, but rather abstract too much or focus too narrowly. We find ourselves either overwhelmed by too many indicators, or tend to develop summary indices that conceal more than they reveal. Either way we lose. But maybe we, along with others are learning.

My continual prodding is that we need to address indicators in a functioning management system, where people both within government organizations and beyond can "own" (can resonate with) parts of the system, and relate parts to wholes in meaningful management settings. (Here, by the way is a little framing thing I'm working on titled Effective Organizations, linked to a "post" in my Adaptive Forest Management blog titled Rebuilding Public Trust.)

Years ago, I attended the first conference of the International Society for Ecological Economics, (Washington, DC, 1990) and listened to promises for greening up national income accounts, among other metrics to help policy-makers do better, and do more for "sustainability" re: ecology, economics and environment. I've been tracking "indicator" and "story line" progress and retrogress since. I even participate in minor ways, no doubt in both arenas, depending on who you talk to.

Yesterday, while prepping for my forest-related travels I decided to scan the Ecological Economics literature to see what I could find on recent developments. I was pleasantly surprised to find Accounting Technologies and Sustainability Assessment Models, In Press, Corrected Proof, Available online 22 December 2006, by Jan Bebbington, Judy Brown, and Bob Frame:

Abstract: Within ecological economics there is recognition of the need for new approaches to decision-making to support sustainable development initiatives. There is an increasing acknowledgement of the limitations of cost–benefit analysis approaches as a measure of the (un)sustainability of organizational activities. These are viewed as particularly inappropriate within the participatory settings that sustainable development proponents seek to foster. They also fail to deal with the highly contested nature of sustainable development discourse in contemporary pluralist democracies. While advances have been made in the field of multi-criteria decision-making, there is still a relative dearth of versatile models that accommodate monetization in a way that recognizes the limits of calculative technologies. This article introduces readers to developments within the accounting discipline designed to support sustainable development decision-making and evaluation. In particular, it proposes sustainability assessment models as a viable alternative to cost–benefit analysis. Sustainability assessment models are based on an inter-disciplinary approach that recognizes the need for “accountings” that facilitate more participatory forms of decision-making and accountability. As such, they address many of the weaknesses in current approaches to cost–benefit analysis. The authors’ first experiences with sustainability assessment models were with BP and the United Kingdom oil and gas sector, where models were developed as a means of making previously external costs more central to organizational decision-making. Later work has included exploration of a range of decision-making situations in private and public sector organizations in both the United Kingdom and New Zealand. This has involved more explicit attention to plural values and issues of participation, dialogue and democracy.

I was delighted to see this update on what I'll call "nails in the coffin" re: cost-benefit analysis. I have already been mining the "references" to update my own Critiques of Cost Benefit Analysis (embedded in this post along with 13 suggested books dealing with "CBA not!" and much more.)

I was even more impressed to see the authors carry the argument forward re: summary indices in the "social and environmental accounting" arena. I wish I could daylight the article, but I'm afraid it is in the "pay for view" category. I got to it via my 'Science Direct' subscription.

Ultimately, the authors endorse a social accounting medium that allows for practitioners, decision-makers, and the public to simultaneous test (in real life settings) various approaches to environmental/social accounting and evaluation, The vetting of various methods is immediate and ongoing, as interactive dialogue. Although I'm not yet familiar with the particulars I hope they somehow incorporate blogs (or blog-like feedback mechanisms) in the systems.

Let's hope that now more people will change focus toward value pluralism and civic discovery, i.e. "participation, dialogue, and democracy." Permit me one snip:

… In New Zealand, we have sought to explicitly address the issue of competing ideological orientations. By comparing how different stakeholders explain and illustrate their perspectives, we have encouraged actors to articulate their ideological standpoints on SD (e.g. their perspectives on monetization, the use of market valuation methodologies and views on the materiality of particular dimensions of SD). We see this as crucial in facilitating dialogic learning and interaction in polyvocal environments. We also envisage groups with different ideological orientations constructing their own SAMs. Separate SAMs rather than synthesis into a "unified" account leaves stakeholders with the ability to exchange SAMs as a way of explaining and justifying different courses of action and allows them to interrogate each others’ ways of knowing. We see this as important to protect against monologism and to provide a challenge to eco-modernist "business as usual" approaches.…

For the record, I find myself intrigued with summary indices like Ecological Footprint. Maybe it's the "star power." Take a look at Who's Who for the Global Economic Footprint Network. Alternatively, Amartya Sen's name is found among the proponents and developers of the Human Development Index. See, e.g. this [PDF] and this.

No doubt efforts at furthering development of more comprehensive indices have merit, as public and political discourse and more. And some of us will continue to follow the chatter (pro and con) that swirls around them. Some of us will also follow other efforts, some related to the more comprehensive measures, some not, where people try to make sense (political, social, and problem-solving) of various sets of measures, situationally defined and refined to fit whatever context is judged to be appropriate.

Criticism and commentary will continue: Consider Eric Neumayer's "Sustainability and Well-being Indicators" [PDF] March 2004. Neumayer concludes

…fully integrated indicators of well-being and sustainability encounter a fundamental conceptual problem: What affects current well-being and sustainability need not affect sustainability al all or not in the same way, and vice versa. Fully integrated indicators therefore tend to conflate the measurement of two items that should be kept conceptually different. Whereas well-being refers to the current use of the available capital stock in terms of preference satisfaction, sustainability refers tot sustaining the value of the total capital stock for the future. The inclusion of sustainability in a measure of current well-being can be justified if one assumes that the current generations' welfare fully takes the welfare of future generations into account. However, no similar justification exists for a measure of sustainability, which should be free of items that affect only current well-being as future generations cannot care for current welfare. …

Since the assumption that "the current generations' welfare fully takes the welfare of future generations into account" is itself highly contestable, the battle lines are clearly drawn.

Another commentary is Adnan A. Herzi and Stephen R. Dovers' "Sustainability Indicators, policy and governance: Issues for ecological economics" Ecological Economics 60 (2006) 86-99.

Abstract: …The role of sustainability indicators as an evaluation method for sustainability within the emerging context of governance merits further analysis. Focusing on policy processes surrounding the production of sustainability indicators, this paper addresses two questions: what is the potential utility of indicators for policy; and in what ways can indicators influence governance? The former is addressed by exploring three disciplines with long histories of indicator application: public administration studies, urban studies and environmental sciences. The latter is addressed by distilling key perspectives from public policy literature on knowledge utilisation and policy learning, which become the foundation for clarifying the notion of policy-resonant indicators. …

Here's a snip:

…Performance indicator systems serve the function of 'government' with the guiding logic of efficiency and legitimacy. With over two decades' evolution as a policy instrument, they offer unrivalled understanding of actual uses and abuses of indicators as manipulable policy variables.

Content-wise, with the exception of public health, these measures are however not informative to a more science-intensive policy area such as sustainability. For instance, no significant knowledge is incorporated from environmental sciences, in informing the theory and practice of performance indicators. In essence, public administration studies have strong links to government mechanisms but are not particularly science-relevant, nor tied closely to substantive issues.

Scholarship on community indicators, with an emphasis on policy procedures and processes, marks a 'critical turn' for the broader indicator literature. Underpinned by the logic of deliberative communication, scholars are exploring the connection to other non-state actors. As the field is profoundly influenced by ideas from the social sciences, the theory and practice of sustainability indicators now have an assortment of frameworks by which to appraise the social aspects surrounding sustainability measurement at local scale. The social constructivist perspective, in particular, has offered stronger explanatory power in understanding the links between sustainability, social justice and policy action.

Indicator systems from the environmental sciences are also contributing to the substantive understanding of sustainability. The guiding logic is predominantly technical appraisal under conditions of scientific uncertainty. The popularity of environmental indicators is growing at all scales of governance, and like community indicators, are approaching the scale of a movement comparable to the social indicators movement which held sway from the 1960s until the early 1980s (see for example Andrews, 1989). However, there is an urgent need for environmental sciences to expand their current focus on prescribing ideal type indicators based on the different frameworks and concepts. Currently, the level of reflexivity among indicator theorists and practitioners in the environmental sciences, and in EE and sustainability more generally, is insufficient to effect significant penetration into policy processes. For instance, there is a tendency to treat the environment as discrete, with little relevance to other established policy areas (Dovers, 2005 and Vogler and Jordan, 2003). Cognate policy areas such as public health or disaster management, and the traditional public policy literature, may offer valuable opportunities for policy learning. … [citations in original]

Comments

[Rob Metcalfe]

Nails in the CBA coffin, hardly Dave, come on now.

I was wondering whether you are against CBA full stop, or whether you are against CBA just for environmental appraisal?

I think what Neumayer argues is completely correct. The Index of Sustainable Economic Welfare is a classic example of where today's well-being is intertwinded with sustainable development. If you are to combine them both under the same indicator, you need both current and future well-being within the same function. But, that my friend, is hard conceptualise since what determines well-being can significantly change over time.

[Dave Iverson]

Rob,

As I have stated before, there are some very limited contexts where CBA can play a role, e.g. "damage claim assessments", "market cost accounting" (where markets actually exist). But even in these contexts, CBA must be appropriately vetted, cussed and discussed.

In the main, however, I agree with Sagoff (THE ECONOMY OF THE EARTH, 1988; "An aggregate measure of what?", Ecological Economics 60 (2006) 9-23 ) that, from Sagoff's book:

"...Utopian capitalism is dead; ... the concepts of resource and welfare economics, as a result, are largely obsolete and irrelevant; and that we must look to other concepts and cultural traditions to set priorities in solving environmental and social problems. (p.22).

"In this book, I shall argue against the use of the efficiency criterion in social regulation, and against the idea that workplace, consumer-product, and environmental problems exist largely because 'commodities' like environmental pollution, workplace safety, and product safety are not traded in markets. I shall argue, in contrast, that these problems are primarily moral, aesthetic, cultural, and political and that they must be addressed in those terms. The notion of allocatory efficiency and related concepts in the literature of resource economics, as I shall show, have become academic abstractions and serve today primarily to distract attention from the moral cultural, aesthetic, and political purposes on which social regulation is appropriately based. (p. 6)

"... I shall argue, however, that although many important virtues may underlie a free market -- freedom, autonomy, competitiveness, respect for property rights, and so on -- efficiency is not one of them. Free markets are rarely if ever efficient. Efficiency, I shall contend, functions not as a vindication of personal or property rights but primarily as a pretext for centralized governmental planning. (p. 6)

"... Discussions of the 'trade-off' between efficiency and equality have become a useless academic pastime to which this book seeks to write an epitaph. These discussions have little to contribute to the practical and political concerns of social regulation. (p. 60)"

Instead, both Sagoff and I argue for public deliberation and inquiry. In Sagoff's words:

"…[T]o solve policy problems we need to develop the ordinary virtues of inquiry and deliberation… [I]t means that we have to get along without certainty; we have to solve practical, not theoretical, problems; and we must adjust the ends we pursue to the means available to accomplish them. Otherwise, method becomes an obstacle to morality, dogma the foe of deliberation, and the ideal society we aspire to in theory will become a formidable enemy of the good society we can achieve in fact." (p. 14).

I believe that in the very political public process of image creation, social identity creation, etc. some aggregate indices of Sustainability will prove useful, although all have aggregation, interpretation biases. I like what Amartya Sen argues, that policy makers need to look at any and all aggregate indices for what they point to generally, then look deeper at more (and other) information. BTW, Sen is on record as thinking as little of CBA as I do. See, e.g. this:

"The Costs and Benefits of Cost-Benefit Analysis," Public Interest, Fall 2001 http://www.findarticles.com/p/articles/mi_m0377/is_2001_Fall/ai_78900303