The recent Databricks Energy and Utilities Forum brought together leaders from across the sector to explore how data and AI are driving transformation during one of the industry’s most dynamic periods. I had the privilege of joining Julien Debard, Director of Energy and Utilities at Databricks, who shared global insights, while I focused on the specific challenges and opportunities facing energy providers in Australia and New Zealand.
It was also an opportunity to stand alongside Alinta Energy and Energy Queensland, clients we have partnered with, to share real-world experiences of navigating the ongoing energy, technology, and workforce transitions.
The following is a summary of the key messages I shared during the forum, covering insights drawn from our work across the region and practical observations on what’s working, what’s not, and how organisations can position themselves for success.
What’s Top of Mind for Executives
A Sector in Transition
As we shared in last month’s Ignition, Australia’s energy industry is undergoing profound change. While recent political outcomes may bring some directional certainty, the pace of market, regulatory, and customer shifts has never been greater. For executive teams, this volatility presents both opportunity and risk.
In this context, data and AI systems play a critical role. Not only do they help identify opportunities, but they also ensure decisions are traceable and explainable. The need is clear: agile yet governed data systems that allow organisations to react to market dynamics while maintaining compliance and accountability.
The Technology Shift
Beyond the energy transition itself, boardrooms are grappling with an accelerating wave of technological change. AI, particularly generative AI, is being explored for its potential to reshape operations, enhance services, and unlock new business models. At the same time, cybersecurity concerns loom large, with executives acutely aware of the risks posed by data breaches, system vulnerabilities, and even threats to physical infrastructure.
At the centre of this tension is data management. Lean too far into innovation and the environment can quickly resemble the “Wild West” – unregulated, untraceable, and risky. Over-index on cybersecurity, and organisations risk becoming so tightly controlled that they suffocate innovation. Striking the right balance is no small feat, especially when data, AI, and cyber responsibilities often sit across multiple teams with conflicting priorities.
A Changing Workforce
At the same time, the workforce is evolving. As a client executive noted, “Every graduate who walks in the door can code.” While this opens new doors, it also makes enterprise-wide adoption of tools and standards more difficult. Supporting diverse skill sets while driving consistency is a growing challenge for CIOs and data leaders.
How We Are Using Databricks to Support the Industry
Agility
In a sector where timing can make or break opportunity, we’re using Databricks to help our clients to respond quickly to new market conditions – from identifying and developing new generation sites, to fast-tracking financial close, launching energy services, and automating customer operations. The platform combines auditability with the flexibility to explore, test, and scale ideas quickly.
Governance
As the technology landscape shifts, energy companies are looking for solutions that allow them to innovate without compromising on security. Databricks strikes that balance. With features like Unity Catalog, we help organisations manage their data estate confidently, while giving teams access to the tools needed to explore, build, and deploy AI solutions using the same governed data.
Engagement
Databricks has made smart investments to accommodate diverse data users. Whether it’s analysts using visual tools, engineers coding in multiple languages, or business users querying data in natural language, the platform supports all personas. With the upcoming release of Databricks One, we expect engagement to deepen further, as the interface becomes more accessible to non-technical users. This breadth of usability is essential in a workforce where skills and preferences are diverse.
What’s Working—and What Needs Rethinking?
In fairness, our key observations could be applied to most data investments and are not limited to just Databricks implementations.
In a dynamic industry, waiting too long to act on data insights can result in missed opportunities.
Larger incumbent players often find themselves constrained by rigid IT processes that are designed for stability and control rather than adaptability. Their systems are optimised for established business models, which makes it difficult for teams to move quickly on emerging opportunities, such as trading innovations, retail personalisation, or new energy ventures.
On the other hand, new entrants typically excel at experimentation and ideation but often get stuck in a perpetual state of prototyping. Without clear pathways to production, even their most promising ideas can remain trapped in pilot mode, unable to scale or deliver lasting value.
Modern data platforms like Databricks enable what we call the “Opportunity to Production” model, where teams are empowered to explore and test ideas quickly, with built-in mechanisms to productionise what works. This approach reduces risk, avoids shadow systems, and ensures innovation can be integrated into the core business, not just observed at the edge.
The most successful implementations find a balance that combines the creativity of agile teams with the rigour of IT. Done right, data becomes a strategic enabler, not a bottleneck.
As highlighted in the broader technology shift, many organisations find themselves caught between two competing forces: the Innovation Engine, which drives experimentation and growth, and the Risk Guardian, which protects against threats and ensures compliance. Nowhere is this tension more visible than in the approach to data governance.
Traditionally, data governance has leaned heavily toward control by prioritising restrictions, approval workflows, and access limitations in the name of security. This bias towards the Risk Guardian often slows innovation and creates barriers for teams trying to move at speed. Conversely, where governance is too relaxed, the Innovation Engine runs unchecked, leading to data chaos, shadow systems, and increased exposure.
Modern data governance needs to find the middle ground. At Ignite, we encourage a shift away from pure control toward discoverability and trust. Governance should empower, not restrict. Start by answering three key questions for your teams:
- What data is available?
- How good is it?
- Where can I find it?
Similarly, for AI: What models are available? When should they be used? How can they be applied responsibly?
Platforms like Databricks support this modern governance model. Unity Catalog enables visibility and control without bottlenecks. At Ignite, we complement this with our Data Quality Engine, which tracks data fitness-for-purpose. Combined with data usage tracking we’ve taken a more user-focussed and business-aligned approach to Data Governance.
“Build it and they will come” doesn’t work. Data strategies that prioritise platforms or practitioner tooling risk excluding the vast majority of employees. Data must be usable not just accessible.
Our recent work has focused on building data products tailored to frontline teams and investing in data literacy. We help teams understand what data exists, how to use it, and how to quickly prototype solutions that unlock business value.
Organisations taking a platform first approach without focus on end users are forgetting the first principle that data is everyone’s business.
Final Thoughts
As the energy sector navigates complex transitions – Market, Technology, and Workforce, related – the role of data has never been more vital. At Ignite, our mission is to simplify data for the energy transition, and I hope these insights provide a practical lens on what’s working, what’s not, and where to focus next.
