Using analytics to gain an edge in the Australian Energy Market



Published: 21/03/2018

Every now and then a problem comes along for which there really is no existing knowledge article or precedent on how to solve. While this may be a very daunting concept to many, it is something that our data scientists relished tackling. We worked quickly to understand and wrangle large disparate data sets to prepare a series of supporting predictive models, in an environment that is effectively new to the Australian market. This provided our team an exciting challenge to brave untrodden ground and work closely with one of our partners, who are working to develop innovative solutions for their customers.

The challenge was to build a tool for our client to stress test their business model in a statistically rigorous and data-driven fashion. The main driver was to develop robust modelling techniques, under real time pressure to enable our client to pitch their business model, which seeks to create value from innovative and sustainable energy technologies.

Our approach in these scenarios involves working in close consultation with our client, since each engagement is always unique in data science, there is no standard process to follow.

This was especially the case here, as large scale Solar generation and storage has only recently become cost effective in Australia, and it is challenging to identify and source relevant data to represent the numerous events that could influence both the generation and storage of electricity.

To tackle a solution, we first had to understand the main problem that our client was trying to get to the root of.  This happened through several discussions between us and the client that created open and honest communication channels.  Our team was able to disseminate an initial course of action from a conversation, hand over a sketched out first draft and pivot quickly and effectively based on feedback. The ability of our team to provide quick turnaround of adjustments as their own priorities changed was called out as one of the highlights of the entire engagement.

In business terms, we created an Earnings at Risk Model that encompassed the wide range of outcomes that could potentially exist, and projected the effect of these onto base line earnings.

We created an interactive scenario model, represented an extensive dashboard that allowed the various levers of the model to be tweaked to mimic any number of potential outcomes in the market.

We created a plan of work that initially centralised and homogenised multiple datasets from differing sources such as usage predictions, unit pricing and generation – essentially performing extensive transformation – to create a workable dataset. These base datasets were incredibly large, totalling hundreds of gigabytes, and were extremely varied and misaligned. Our team carried out the necessary plumbing, from database to data science, with speed and versatility.

Using our clients deep expertise in the energy market, and our leading data science team we created a machine learning based, rich interactive dashboard as a deliverable to stress test their business model.

According to the client, “understanding the data structures, massaging and developing models that incorporate the range of outcomes experienced, and in turn the creation of a model that presents a spectrum of outcomes” was the one of the founding objectives of the project we met and then exceeded.

The provision of this custom dashboard enables any member of the organisation to run the model without the need for a Data Scientist each time. This in turn frees the Data Scientist to work on other issues and means that in consulting terms – the client is able to utilise the engagement to the fullest extent.