“A bit of a milestone, we hit 1M NMIs in Queensland with 5 min interval data over the weekend just gone.”
Justin Kenny,
Advanced Analytics Platform Manager, Energy Queensland
Overview
Energy Queensland is the group of electricity distribution, retail, and energy services businesses owned by the state of Queensland. They deliver electricity across the state through their ‘poles and wires’ businesses Energex and Ergon Energy Network. Their retailer, Ergon Energy Retail, sells this electricity to customers throughout regional Queensland.
These essential service activities are also supported by a range of innovative products and services delivered through their Yurika brand.
Developing a Deep Data Lake at EQ
Energy Queensland’s implementation of Data Lake and Data Science capability leveraging AWS and Databricks in early 2021 had three key deliverables:
- A Data Science platfom aligned with Energy Queensland’s Cloud project
- Meter Data Analytics Solution (MDAS) designed to support 5-minute intervals
- Operations Upskilling ensuring Energy Queensland’s team could own the platform
“There was a genuine desire from Ignite to build something with us that we would get value from. They continuously sought feedback, involving many business areas to work together to design and build the solution. This gave our teams the confidence that they could support the platform when it was time for Ignite to go.”
Justin Kenny,
Advanced Analytics Platform Manager, Energy Queensland
The system was built to enable:
- Access to 10 year data set including NMIs, tariff data, regions and time periods down to 30 minutes.
- Detection and interrogation of trends.
- Loading and processing of NEM12 files , meeting regulatory requirements.
- Standard, accurate and easy to generate reporting.
- Easy platform access.
- Growing and moving data easily into other systems.
- The development of sophisticated pricing queries and models.
Three Years on
With only minor maintenance, the Energy Queensland Data Science team has confirmed the platform is still going strong! The solution is now handling over 1 million NMI’s across Queensland at a 5 minute interval, processing over 600 million read values each day.
This has allowed them to tackle new and exciting use cases that they couldn’t have considered before, with more on the horizon!
Technologies
- databricks
- aws S3
