Why Data Is the Backbone of the Energy Transition
Ignition No. 42,
May 30, 2025
Australia’s energy sector stands at a pivotal juncture. The renewed mandate of the Albanese government underscores an intensified commitment to the energy transition, heralding ambitious targets and transformative changes, which we’ve discussed in this month’s Ignition.
The government is doubling down on its renewables push, coal is moving off the grid faster than ever, and pressure is intensifying across the NEM. There’s more variability, more complexity, and a bigger role for distributors, DERs and data.
At Ignite, we’ve spent the past decade working with utilities who are right in the thick of this shift, across generation, distribution and retail. Our collaborations have consistently highlighted a central truth: robust data strategies are no longer optional – they’re imperative. And now that point has been reinforced, as the path to a sustainable energy future is data-driven.
Whether it’s forecasting load, automating compliance, or just making faster decisions with confidence, data isn’t a nice-to-have. It’s the infrastructure behind the infrastructure.
The return of the Albanese government with an overwhelming majority signals a renewed push on the energy transition, with discussion already occurring around more ambitious targets.
The further penetration of renewables and the planned exit of coal generation from the NEM will see ongoing intermittency and variability in supply. Transitions are never easy, and Australia is about to hit the bumpiest part.
In preparation for its next iteration of the Integrated System Plan (ISP), AEMO recently published its draft 2025 Electricity Network Options Report. One look at the conceptual map is enough to grasp the challenge: more assets, more connections, more urgency, and more data requirements than ever before.
Survival Analysis is a statistical approach used to answer the question: “How long will something last?” That “something” could range from a patient’s lifespan to the durability of a machine component or the duration of a user’s subscription.
One of the most widely used tools in this area is the Kaplan-Meier estimator.
Born in the world of biology, Kaplan-Meier made its debut tracking life and death. But like any true celebrity algorithm, it didn’t stay in its lane. These days, it’s showing up in business dashboards, marketing teams, and churn analyses everywhere.
But here’s the catch: business isn’t biology. It’s messy, unpredictable, and full of plot twists. This is why there are a couple of issues that make our lives more difficult when we try to use survival analysis in the business world.
You may not be a survival analyst, but there’s a clever twist here worth noting. This guide reimagines survival analysis (using the Kaplan-Meier method) for scenarios where the ‘event’ isn’t death but a business scenario, like customer churn or product return. By shifting focus from time-to-event to value-at-risk, it opens up new avenues for applying these techniques in business contexts. Worth a read if you’re working on value retention or customer behaviour.
There are millions of articles online about how programmers should not write “clever” code, and instead write simple, maintainable code that everybody understands. Sometimes the example of “clever” code looks like this (src):
# Python p=n=1 exec("p*=n*n;n+=1;"*~-int(input())) print(p%n)
This is code-golfing, the sport of writing the most concise code possible. Obviously you shouldn’t run this in production for the same reason you shouldn’t eat dinner off a Rembrandt.
Other times the example looks like this:
def is_prime(x): if x == 1: return False return all([x%n != 0 for n in range(2, x)])
This is “clever” because it uses a single list comprehension, as opposed to a “simple” for loop. Yes, “list comprehensions are too clever” is something I’ve read in one of these articles.
I’ve also talked to people who think that datatypes besides lists and hashmaps are too clever to use, that most optimizations are too clever to bother with, and even that functions and classes are too clever and code should be a linear script.
Clever code is anything using features or domain concepts we don’t understand. Something that seems unbearably clever to me might be utterly mundane for you, and vice versa.
IN PRACTICE. You may not be in the habit of writing overly clever code, but Hillel Wayne makes a smart case for doing it, on purpose, and in private. His argument: clever code, when treated as practice not production, sharpens your skills and stretches your understanding of what’s possible. Just, please, don’t ship it.
In today’s rapidly evolving energy landscape, petroleum engineers face unprecedented challenges that extend far beyond traditional reservoir management.
While many engineers know precisely the type of analysis they want to conduct, they often lack the programming confidence to implement it effectively. Many experienced engineers lack programming knowledge and the vocabulary of data analysis, leading to what is termed “data-driven anxiety” within the workforce. This skills gap can hinder engineers’ ability to leverage big data for actionable insights, especially as their demanding jobs leave little time for upskilling.
Compounding this issue is the industry’s shortage of specialized data science talent, creating bottlenecks in decision-making processes. Databricks Assistant addresses these challenges by empowering engineers with AI-driven tools that simplify complex data tasks without requiring extensive coding expertise.
By leveraging natural language and automated code generation, engineers can independently perform advanced analyses, such as decline curve automation or anomaly detection, while focusing on domain-specific problem-solving.
You may not be a petroleum engineer, but there’s a valuable takeaway here for anyone working with data. Databricks Assistant lets users skip the code and ask complex questions in plain language, turning domain experts into data analysts overnight. If it can do that in oil and gas, there’s serious potential for unlocking insights in every corner of industry. Take utilities and Genie for example, dispatch price trends available at the drop of a whim, and plenty more applications.
Nuclear energy ranks among the world’s most regulated industries. AI and especially generative AI have created enough impact that thought leaders rank it among other transformative “general purpose technologies” such as electricity and the steam engine. Harnessing AI to reimagine nuclear operations across the industry means more carbon-free nuclear energy for electrical grids and data centers, which the International Energy Agency estimates demand to double by 2026. In September 2024, Westinghouse unveiled its HiVE™ AI system, powered by its fine-tuned bertha™ generative AI model, transforming how customers collaborate with Westinghouse.
Here’s a serious leap forward for a high-stakes industry. Westinghouse is using Databricks to power its HiVE system, turning decades of nuclear ops data into a scalable AI platform that speeds up inspections and licensing. And its not just about speed and cash, its detecting irregulaties with greater accuracy. It’s a pretty compelling example of how AI can modernise even the most regulated industries.
At the World Economic Forum annual meeting last month in Davos, Switzerland, MIT Sloan professor
joined a panel discussion about the opportunities and challenges of using AI to drive efficiency in the workplace. She talked about the need for businesses to define what they think success should look like, invest in data infrastructure, compensate their workers for lending their expertise, and much more.
Li discussed these topics with Aditya Bhasin, Bank of America’s chief technology and information officer; Jim Stratton, the chief technology officer at Workday; and Nitin Mittal, Deloitte’s global AI Leader. The panel was moderated by Andrew Hill, a senior business writer at the Financial Times.
Li, who studies how AI impacts the nature of work, offered four tips for companies experimenting with the technology.
Gen AI adoptions is increasingly becoming a matter of when not if. So for businesses looking to adopt – or even have adopted – gen AI tools, its worth taking note of these lessons from MIT Sloan. The advice is practical: set clear goals, get your data house in order, and make sure employees are part of the process, not just subject to it. And one extra important lesson that’ll ease the burden on analysts – show business users how it can improve their day-to-day and lead with good ol’ data democratisation.