Notes from our August Brisbane industry breakfast with small group of energy and water industry, data-focused leaders who gathered to listen to Justin Kenny, Energy Queensland’s Advanced Analytics Platform Manager, share his and EQ’s journey to build a scalable data platform that is about to surpass 1.2 million NMIs.
Queensland is one of the leading states in the Energy Transition. What are some of the key things that you and your team are thinking about now?
From our point of view, it’s the demand for services and new products, particularly in relation to cross domain multimodal datasets. This is a growing conversation across the organisation.
Different business areas are seeing new opportunities that can add value as well as opportunities for simply doing existing tasks better.
The challenge is the volume of demand for new products and analytical services building from the enhanced capabilities of the analytics platform – opportunities like addressing how we forecast and price to optimise outcomes for our customers and the business.
In a new world where there’s increased interval metering, the analytics to inform what the tariffs are is really important. Outputs are used to support regulatory submissions with deep analysis, providing required insights and reports for the regulatory authorities, all underpinned by the platform’s capabilities.
Going back 10 years ago large business customers were the only sites with interval read metering in Queensland, Energex was only about 13,000 NMIs. We went from tens of thousands of premises with 30-minute feeds to now almost 1.2 million with five-minute data. That’s a fundamental step change in how you need to work with your data and analytics.
What sort of use cases are the team working on?
Pricing represents one-use case. There’s so many more that we’ve uncovered along the way. From grid planning through to working with the engineering teams on understanding where all the new technologies like electric vehicles are appearing on the network. And each have their own set of business complexities.
Currently, we get postcode level EV registration data from the Government, which is at a postcode level to maintain the sensitivity of the data. But we need to know where clusters of these EVs are actually charging, at local areas of the distribution network. Now our team has developed a model to identify patterns from the data whilst also respecting the sensitivity of the data.
After we worked through these modelling aspects, new questions get asked about what else we can do to detect things, like how many premises might have air conditioning and anything that is important for network planning engineers to know and improve decision making. There is a whole swathe of backlog use cases in that realm, naturally progressing to forecasting loads and energy usage on the network.
Another use case we are working on relates to how rooftop solar impacts the network in Queensland. We currently only have net interval metering data in Queensland, meaning we can’t see what premises are actually generating or consuming in gross. All we can see is that there is a net export back to the network at times when gross generation exceeds gross in-premises consumption. We don’t know how much latent demand there is being masked by solar generation.
And where that becomes important to understand is in events like when we had a storm in Southeast Queensland recently. On the day it came through, we actually had manageable demand on the network. It was a very hot day and all of the solar basically dropped off with 100% cloud cover on the storm front at exactly the same time that all the air conditioners were still running due to the heat, and demand just went straight up. We are now using the platform to work with the forecasting team, trying to plan for these types of events across the network.
How do you manage all these requests and use case ideas?
Like everyone, we are limited in our resources, but we work collectively across the different business teams. We made architectural decisions from the get-go to ensure that we could scale both to the volume of data that we knew was coming and the increase in requests from across the business for analytical services.
How did you get the support from the business to build the platform?
We got a lot of help from Ignite on that part, around the proof of value. We did need help to prove the value, to get investment to do this right.
Ignite did a lot of interviews around the business and helped pull that initial piece together. The users knew what types of things we were trying to do and felt ownership of that.
A lot of our teams within Energy Queensland weren’t familiar with what a contemporary analytics platform could do, and this is where the proof of value really added impact. We could see the big increase in volume of data that was coming, and we knew that it was just expensive and nearsighted to keep adding traditional servers.
We had to think about what we were solving for and what we would have to solve for in the future.
What was the solution you went with?
It was Databricks on AWS. That was a good story, and it was just making sure that we had some of the basics right around what was important to the business with respect to security, governance and growth. And what was really attractive was that you can scale up the compute power as needed and add further flexibility with that.
From an analytics perspective at EQ, this was our first foray into a cloud environment. We had not had any practical experience working in cloud environments or with products like Databricks before, so there wasn’t a great degree of understanding within the team. This is where the partnership with Ignite was invaluable.
They helped us design a good architecture, from good discussions upfront and making sure that the business needs were clearly identified across all the different stakeholders to feed into that initial architecture decision-making process.
This technology is very robust. The data is stored securely and there is a great deal of governance around that. Our platform needs to run for years, not just today. Ignite helped us to play off the use cases with stakeholders and define the architecture at the same time. We were building for what is possible.
What were the challenges around institutional architecture?
Our industry can be very restrictive and prescriptive about what they want going into the cloud. We worked on this with multiple business areas, particularly with the broader Digital team.
For EQ, it was about working closely with our security teams to make sure that we’re understanding and addressing their concerns while meeting the business’ operational needs.
Effectively, we were asking stakeholders to have confidence that the platform was secure. We opted for the customer-managed VPC construct, which means all of the compute and storage happens in our own AWS account. Data sovereignty, which was one of the key issues that we had to work through, was solved using AWS based in Sydney, which helped us a great deal.
Meter data analytics was the first use case or catalyst for what we now call the Advanced Analytics (previously Data Science) Platform at Energy Queensland. This was a new platform in the digital sense. We have a platform construct including our customer platform, our network platforms and now a data science platform.
We have the ability to bring on data, and since we went live and since the original proof of value, we’re now expanding into the full lake house within the enterprise data group. We are evolving into an enterprise intelligence platform usage as well.
And the great thing about Databricks as a common technology is that it provides a single solution for accessing data across the different capabilities. It’s a unified platform construct. A new Enterprise Data Group has been established now and there’s more and more data sets coming onto the platform that will enable us to further serve the corporate space.
It scales, does all the things we need it to do, and the order of magnitude of costs of implementing the system was just so much less than the alternatives.
We have elevated levels of controls around how we access the data. We work closely with our privacy officer to make sure that we are managing who and how data is accessed. We have established processes around ensuring they’re involved and aware of what’s happening in the analytics space.
Working together to understand what users are doing, with what data enables us to evolve our privacy policies in a dynamic, real-time and evolving manner.
Operational data vs data analytics – Don’t cross the streams.
We talk openly about not ‘crossing the streams’ at EQ.
There’s still a warehouse that’s needed to calculate billing determinants for billing purposes that for a whole bunch of reasons isn’t suitable for analytics. We consciously decided to have separate platform constructs and separate streams of data.
We are careful not to unduly influence or change anything that’s happening in the operational space, until we have a proven, robust product. And the data scientists also need a different capability set for analytics. We worked closely with our billing teams because it’s important to both understand each other’s processes.
We have two data sets as well. While it can be considered duplicating the data, we have two different capabilities that serve the business in different ways. We could do this because the Ignite analytics solution was implemented at a comparatively extremely low cost.
The data comes in the front door once and then a feed of that goes into the billing platform and a feed into the analytics one. We established validation rules and processes to support the way that we manage the data inflows.
We are bringing, governance and data management aspects together, making sure that we’ve got rigour around validation, and we are frequently collaborating with our colleagues in the billing space to stay on top of industry and regulatory changes.
Having a good relationship between the data analysts and the operational data teams is especially important. A lot of analysts, for example, are only used to looking at one thing at a time, so they don’t look at the whole construct. Sharing concepts and insights encourages good conversations and is where innovation happens. And it actually helps with people understanding the data sample they’re working with as well.
But we are ready to cross the streams when it works for the business.
Parts of calculating billing determinants can be done out of the analytics platform. We did actually prove that in the proof of value phase. We wanted to ensure that the platform could expand and serve the business in many ways. We had a separate process that took all of the interval data and calculated billing determinants.
We proved that we could do that with the same level of precision and replicate the exact same output that our billing system produced for those billing determinants from an analytics platform. So, we know that we can do it, if the business decides that is the direction that it wants to take in a future scenario.
Work with the Willing.
Work with the willing. Set up Working Groups with people who are passionate about the project. People who really care about the longevity of the platform, its potential and how it can help the business.
Be ready to repeat yourself.
You’ll have different individuals that come and go into different role. And sometimes we feel like, we’ve got to start the education process again. It’s an evolving piece and it is taken very seriously by Energy Queensland.
When we bring on a new use case, we work with stakeholders from the business teams that own the outcome working with us. That’s not negotiable for us and we use that as a way to help educate those not already familiar with the platform.
There are superusers and there are superusers.
There are considerations you have to work through around governance control. We have to enable super users on the platform. And we also have to realise that there will always be a business tension with data requests from people who need to do tasks quickly to deliver value for the organization.
And some of those people are stakeholders directly accountable to our executives and they’re expected to be able to provide quality answers in 48 hours. For data scientists, that’s a whole new use case that we haven’t even made a discovery on! We have to be practical.
Traditional thinking splits development and staging environments and then data sits in there. But we need to have the data in a place where the appropriate people with the right role can access it.
We still use role-based access control. It’s their role to do the analytics. They need to do it quickly. So, we built an incubation environment, where they can develop new analytical products or content.
We’re not developing the actual system itself. We are developing content. There’s only one level of control we’re using it to develop products. We think, let’s get on with it!
We need to enable the people with the deep market knowledge and IP to work with the people who can bring the data science and the thinking of how to embed the product into the business and support it at a later stage.
If we can get people with the right IP to develop to enterprise standards, we have found the Holy Grail.