Every day it seems that generation from variable renewables hits new milestones. In March, California hit a new record in renewable energy production by supplying 56% of midday demand. And every day, prices for new projects seem to get cheaper. In February, an auction for a 750MW solar park in Madhya Pradesh attracted bids as low as $49/MWh, the lowest yet recorded in India. In the Middle East and Chile meanwhile, auction prices have fallen below $30/MWh.

These low prices have been driven by dramatic declines in technology costs, which in turn have driven a boom in deployment, creating a virtuous cycle of scale that drives down cost. Last year, 161GW of intermittent renewables were deployed around the world, bringing global cumulative capacity to 2,006GW.

A low-cost, low-carbon grid seems within our grasp.

In our report, Flexibility: the path to low-carbon, low-cost electricity grids, we find that a future grid powered mostly by the sun and wind will be cheaper than one fuelled by natural gas — even without a price on carbon. While a gas-based system may cost $73/MWh, a renewables-based system — including nearly $30/MWh of flexibility costs to integrate a high share of renewables — would be competitive at $70/MWh.

Our analysis was commissioned by the Energy Transitions Commission, which this week published its report, Better Energy, Greater Prosperity. Our key findings support the ETC’s challenge to business, governments and investors to seize this opportunity for a more prosperous economy while cutting annual carbon emissions from energy from 36GT today, to 20GT by 2040.

Decarbonization of our electricity is a cornerstone of the ETC’s strategy to transform the global energy system, which is responsible for 75% of total greenhouse gas emissions. Cleaning up our electrons is also one of the most achievable transition pathways identified by the ETC in its report and will account for almost half of those emissions reductions. However, this seemingly attractive future is by no means guaranteed.

Just last week, the US Energy Secretary Rick Perry ordered a study into impacts of state wind and solar subsidies on baseload coal and nuclear, with the assumption that these plants are a proxy for grid reliability.

Our view is that grid reliability depends not on large inflexible baseload generation, but increasingly on the ability of the grid to be flexible, adapt to changing conditions and a different set of risks. In fact we see flexibility, as a key enabler of a decarbonized electricity system.

A lack of flexible capacity is often cited as a constraint on the amount of variable renewable energy we can add to the grid. But in our report, we found that most systems already have enough latent flexibility to meet over 30% of their electricity demand from solar and wind. Moreover, technologies that exist today could support much higher shares of wind and solar; 80% or more.

Flexibility needs are well covered over the next 10 years in the four regions we looked at — California, Germany, Maharashtra and the Nordic region. But over time, higher penetrations of renewables will require more flexibility, especially for daily and seasonal balancing.

In California, ramping and daily balancing are expected to become pressing concerns as the state increasingly depends on solar to meet power needs. By some estimates, in 2040 utility- and residential-scale solar is expected to supply 36% of the state’s electricity, while wind at 30%, should constitute the second-largest resource.

California has ample flexibility in its electricity system; responsive demand, interconnections with neighbouring states, and significant existing hydro and gas capacity can all help reduce the cost of meeting the state’s flexibility needs. We have estimated the cost of providing one form of flexibility — daily shifting on peak days — could be reduced by 30%–50% by 2040 by making use of these existing flexibility resources.

So what does this mean for policymakers? We believe that they should be emboldened to raise their renewable energy ambitions, knowing that there are already many sources of latent flexibility in their electricity systems, and that the costs of renewable energy integration are likely to remain low.

But policymakers should also begin to make changes to planning, regulation, and market design— getting that last 20% of carbon out of the grid is likely to be the hardest step on the path a cost-effective low-carbon electricity systems.

In addition, a portfolio of approaches is needed. Shifting consumer demand or utilizing existing hydroelectric dams may be inexpensive, but they are limited in scale. Moreover, some technologies are great for shifting energy on a short time frame, but would be very expensive to shift energy across weeks, much less seasons. A combination of options will be needed to meet the full suite of flexibility needs demanded by a low-carbon grid.

Planning and market design of our electricity systems need to evolve to align investment decisions with the goal of decarbonization. In other words, if we make investments today that aren’t suited to meet the needs of the grid in 15 years, those investments could potentially be stranded down the line.

Finally, flexibility technologies are rapidly evolving. We need to make sure our markets and policies create long-term incentives for innovation, so that we can unlock low cost options to balance the low carbon grid of the future.

Read the full version of this blog with supporting analysis in the CPI Energy Finance Insights series on Medium.com. You can also watch the webinar that introduces our findings.