MIT Sloan professor Pless identifies key challenges and solutions for evaluating energy innovation policy

Cambridge, MA While substantial innovation is needed to enable the transition to clean energy systems, evidence is limited as to what policy mechanisms actually work to help drive private-sector innovation. In a recent paper, MIT Sloan School of Management professor Jacquelyn Pless identifies five key challenges to evaluating energy innovation policies and programs and recommends solutions for overcoming these challenges.

“It’s critical to ensure that the public resources spent on supporting innovation are not wasted. We need to design policies and support programs based on evidence of what mechanisms actually work,” said Pless.

“This requires identifying the barriers to clean energy innovation as well as the policies and interventions for overcoming them.”

Pless and her collaborators, Cameron Hepburn of the University of Oxford, and Niall Farrell of Queen’s University Belfast, developed a framework for energy innovation policy evaluation. In the framework, they highlight five reasons that generating evidence for which policies and programs are most effective is challenging. They also recommend ways to mitigate those challenges.

The primary challenge relates to quantifying how much innovation activity can be attributed directly to a policy or program as opposed to other factors. “Without a control group, you don’t know what would have occurred with funding versus without funding,” she says.

The second challenge involves the significant and uncertain time delays that occur in the energy sector between funding and technology development. “In some industries, like software or app development, technology can be developed in a few days or weeks. In the energy sector, there are long timelines – sometimes more than 10 years -- between idea generation and outcome. That makes it difficult to attribute results directly to a policy,” said Pless.

The third challenge is determining how to measure outcomes. She said, “What is the right measure of innovation for the energy sector? Inputs like spending on energy R&D can be measured, however capturing information on whether innovation actually occurs and subsequently impacts economic and environmental performance is less straightforward. Researchers often use measures like publications and patents, but new metrics are needed for the energy context.”

Distinguishing the direction of innovation when assessing these outputs is the fourth challenge. Did the innovation advance “clean” versus “dirty” technologies? This leads to the fifth challenge of disentangling policies and programs to determine the effects of each one and their interactions. That is essential in order to optimally design policy, she said.

Governments and agencies can address these challenges, according to Pless. “By tracking successful and non-successful applicants of grants, researchers can compare the innovative activity of firms that receive funds relative to similar firms that do not receive funds. This can help assess outcomes over the short- and long-term. It can also help measure whether innovation outcomes are environmentally-friendly, which is critical for assessing the direction of innovation.”

Another possible solution to these challenges is embedding some randomization into policy and program design, which can enable more rigorous analyses. For example, the requirements attached to funding could be randomized, such as collaboration with specific types of firms, national laboratories, or universities. Randomization combined with improved outcome measures can help determine whether collaborations affect commercialization, she said.

Pless points out that there are under-studied research areas that are critical for developing a better understanding of how to drive clean energy innovation, such as long-run changes to strategies, capabilities, and management practices that continue to enhance innovation.

“All of these solutions fall under the umbrella of multiple stakeholders working together, as access to data is critical for researchers to adequately answer these questions. Overcoming these challenges could lead to a significant increase in the evidence base of policies that can successfully drive innovation in the energy sector,” she said.

Pless and her colleagues are coauthors of “Bringing rigour to energy innovation policy evaluation,” which was published in Nature Energy. Pless also published a related blog in Nature Energy.