Our team recently worked with large Queensland energy provider, Energy Queensland to take a data project from Proof of Value Phase to Delivery Phase.
Here’s the perspective of the team on how to optimise and smooth the transition from Proof of Value to Delivery.
Proof of Value Discovery
Thanks to the successful Proof of Value Phase, the team had these key elements in place:
- A reasonably solid solution design, and a clear picture of what we had to build in the enterprise environment.
- A hit list of improvement items and new features, for example:
- The need to apply business-specific data validation rules (such as the removal of duplicate records), surfaced during the use of the Proof of Value solution, which otherwise might result in incorrect downstream reporting.
- The ability to cope with a new market data specification which went live during the project. This meant that 5-minute granular data was available from day one, and the data assets developed against the old specification were able to cope with the change seamlessly. This also meant that analytics teams had the tools and data available to see the impacts of the 5-minute settlement market change as it happened.
- Knowledge of key non-functional requirements that needed to be met, including around data governance, security and monitoring. For example, from a monitoring perspective, the Proof of Value phase helped to show what information operations teams would need to effectively manage the platform, including the ability to track resource usage.
- A very clear picture of the project’s success criteria.
“The stakeholders already had something in their hands that they saw would be useable. We knew what the goal was, so the use-case understanding was solidified, and we had that clearly in mind from the get-go.”
Dave Hamilton, Senior Data Engineer
During the build phase, Ignite begins implementing IT best practices, expanding on the Proof of Value solution. This includes addressing security requirements and data validation. The client’s IT Infrastructure team gave incredible support and were clear with networking rules and capabilities, which meant that Ignite was standing up solutions that met the needs of the broader enterprise.
Ignite Learnings From This Project And New Disciplines Developed
CI/CD
During this project, the Ignite team made some good inroads into developing a CI/CD (continuous integration continuous deployment) methodology. CI/CD is a way of streamlining code deployment, and managing the development lifecycle more robustly, with shorter cycle times. Approved code changes could be deployed in a matter of minutes, compared to several hours in an otherwise manual process. This can free up several workdays per month for a busy operations team. We are now using this for other clients, and it is already adding incredible value.
Collaboration
Working from a central code repository enabled everyone to see the build as it was progressing, add comments, raise bugs and suggest improvements. This enabled Ignite to collaborate openly and engage the client with a solution that they understand rather than a black box. A living, breathing code-base that they can pick up with and continue to use once Ignite had finished the immediate engagement. We used GitHub Enterprise as it was an existing service in the organisation which met governance requirements. We are rolling out similar patterns on new projects with other clients.
Modularity
During this project, we developed modular components to streamline the processing of industry-specific data formats. Ignite developed a software asset that can help other use cases in future. It is an extendible way of processing data and minimises duplication of effort by having a proven component in place for use by future project teams. This has the potential to eliminate days of effort in future initiatives which need to handle raw data.
Handover
At the end of the project, we had a period of handover to the operations team. Ignite prioritised the topics that the client team was most concerned about, with respect to ongoing maintenance. Ignite held a number of knowledge-sharing sessions to enable the client to progress their data journey in the new platform. They were able to pick up all the new concepts they had not previously worked with.
“We were so fortunate. We had seriously switched-on people in the client’s team – they had a future focus so we could build a platform to be future-ready. This means that they can sweat the new asset with machine learning and predictive technologies that will enable them to solve current business challenges and be ready for even trickier challenges on the horizon.”
Dave Hamilton, Senior Data Engineer