This post originally appeared on Intelligent Utility.

How many millions of dollars does it take to change a state’s light bulbs?

This sounds like the start of a joke, but for the last seven years, it’s been anything but to California utilities and regulators. The crux of the dispute, which has had stakes in the hundreds of millions of dollars, has been an ambitious—but controversial—shareholder incentive designed to motivate California utilities toward greater energy efficiency.

The policy, called the Risk/Reward Incentive Mechanism, or RRIM, targeted California utilities. However, the concept of a shareholder incentive is one that 20 other states have adopted in recent years. It’s also under discussion at the federal level as part of President Obama’s proposed Race to the Top Energy Efficiency Initiative.

So what can utilities in other states learn from California’s experience? Climate Policy Initiative’s recent analysis, “Raising the Stakes for Energy Efficiency: California’s Risk/Reward Incentive Mechanism,” draws a few lessons that stand out.

1.) Shareholder incentives may be a step in the right direction, but the details matter.

The RRIM helped move energy efficiency into greater parity with new supply for utilities, and it successfully drew more utility management attention to energy efficiency because of the size of the potential rewards. It also introduced some novel elements in incentive design, including an attempt to pay for energy efficiency performance by basing incentive payments on program impacts as evaluated after implementation.

However, the RRIM’s accountability provisions proved difficult to enforce in practice. When evaluation results came in after programs were completed, the utilities’ incentive earnings changed by tens of millions of dollars—in one case, from reward to penalty. The resulting conflict consumed time and resources on the part of the utilities, the California Public Utilities Commission (CPUC), and other parties, ultimately leading CPUC to pause halfway through implementation and overhaul the incentive.

The takeaway? While the RRIM was a useful experiment and a valuable first step, it’s clear that the details matter, especially in the complex environment in which utilities make decisions.

2.) Warn regulators against sharp payment distinctions that do not reflect meaningful differences in program performance.

The RRIM structured incentives so that a small change in estimated energy savings could make the difference between reward and penalty. While the sharp cutoffs made it easy for utilities to set their program targets, this design did not reflect the reality of the energy savings estimation process, which involves some inherent uncertainty. As a result, the earnings calculation was too sensitive to small changes in energy savings estimates.

Because of the sharp earnings cutoffs, large differences in incentive payouts to utilities did not necessarily reflect meaningful differences in performance. To give a sense of the stakes, PG&E would have either been rewarded $180 million or penalized $75 million by the state for the energy efficiency efforts they’d already finished, depending on whose interpretation of program evaluations you used. In the end, they reached a compromise payment of $104 million.

3.) Expect that incentives will put pressure on evaluation processes and engender new disputes. Work with regulators and third parties to support a collaborative and impartial evaluation process.

Basing incentive payments on evaluated energy savings was a core principle of the RRIM and could have had a positive impact on ratepayer protection and utility performance. However, that provision put too much pressure on the evaluation process and amplified the conflict among parties, consuming resources that could otherwise have been put toward improving energy efficiency programs and planning for the future. In recent years, California has developed a more collaborative model for evaluation, and the conflict around evaluation has diminished.

Utilities in other states should support high-stakes incentives like the RRIM only if there are institutional arrangements for energy savings measurement and dispute resolution processes that are accepted by all parties. Utilities should not themselves control the evaluation process that determines incentive earnings—that would pose a conflict of interest. But they should expect a transparent view of the evaluation process and should be able to weigh in on evaluation methods.

4.) Use data to gain real-time feedback on how programs are performing—and work toward nimble programs that can keep pace with changing markets.

California regulators are committed to paying incentives based on evaluated performance, even if this means some retroactive adjustments to payments. Utilities have argued that these retroactive adjustments made the RRIM less valuable, because the utilities could not present incentives to their shareholders as a dependable earnings stream.

Can these positions be reconciled? The most promising path forward comes through the availability of real-time energy usage data. Going forward, utilities will be able to get a sense of how programs are performing in real time, by analyzing data on their customers’ energy usage. And as utilities learn to make use of these energy usage data, they should also prepare to change their energy efficiency programs as markets evolve. Good, timely data on program performance can make an incentive like the RRIM more workable, and the resulting improvements to energy efficiency programs will benefit utilities, customers—and the planet.