How to boost confidence in your Data Science investments
In 'How successful are your Data Science projects?' we acknowledged that the growing discipline of Data Science is benefiting from the recognition and associated investment being made by businesses. However, the failure rate of Data Science projects is alarmingly high, and we have to be careful this discipline is not overlooked as another hype cycle. We presented the case for organisations to adopt a delivery methodology for Data Science projects. However, upon exploring 'How complete are existing Data Science methodologies?', we found that whilst there are some options that get the broad brush strokes right, and some which excel in particular areas, there are still gaps where further work in this area is justified.
In this article, we outline how we have incorporated lessons learnt into a methodology that we have evolved to address the fundamental, yet not-so-obvious challenges facing Data Science projects.
- Data Science is experimental in nature – and this makes executives nervous when investing.
- Data Science is often executed as a primarily technical exercise, without enough focus on how much value the Data Science idea will deliver and whether that value is believable.
- Data Science is tackled inconsistently within many organisations, lacking the appropriate guidelines, templates and accelerators, leading to inconsistent results and prolonged time to outcomes – be that success or failure.
The importance of Proving the Value
We view the Data Science process in its entirety as following three high-level stages:
1. Ideation: Capture and prioritise potential opportunities
2. Proof of Value: Proving both the value and feasibility of an idea
3. Realisation: Implementation and business change to achieve Data Science @ Scale
Figure 1: end-to-end Data Science process, explicitly calling out the importance of a Proof of Value stage to test the feasibility of the idea
We have conceptualised the broader process in this way to emphasise the critical importance of identifying and quantifying business value and impact upfront.
By formally dividing the process to call out the Proof of Value stage, we can encourage due focus on these issues. Despite challenges that can occur during realisation, we believe in prioritising the right ideas through to realisation, and so in what follows we outline the Ignite Proof of Value methodology and demonstrate how this addresses gaps in existing offerings.
Closing the gaps with the Ignite Proof of Value Methodology
In 'How complete are existing Data Science methodologies?', we assessed existing Data Science methodologies against six key focus areas. Finding that none of the assessed offerings met our requirements, we endeavoured to put together a methodology to address these issues directly. These are outlined in the following sections.
Does the methodology recommend phases and activities that are appropriate for Data Science, allowing for the non-linear and iterative flow?
1 - Appropriate high-level process flow for Data Science
We have modelled the high-level process flow on the well-established CRISP-DM process, depicted in the figure below.
As we reviewed previously, the high-level process flow of CRISP-DM is a good representation of what is required in a typical Data Science project, including the need to iterate between phases. Its simplicity gives it appeal when demystifying the technical process of Data Science for stakeholders.
The key rule used with the Ignite Proof of Value methodology is to timebox the investment and avoid experimenting for the sake of experimenting. Generally speaking, if the value cannot be proven within six to eight weeks it is likely that the value is not currently feasible. Further investment outside of Data Science may be required.
Figure 2: high-level phases of the Ignite Proof of Value methodology
Does the methodology make it clear where the checkpoints are, and what questions should be asked at these gates to ensure we can proceed?
2 - Clear checkpoints and questions to help navigate the process
Like CRISP-DM and the other methodologies it has inspired (e.g. Microsoft’s TDSP and the Domino DSL), we acknowledge that a Data Science project will not in general proceed directly through these phases, but instead may need to iterate or loop to earlier phases based on findings. This is the inherently experimental and uncertain nature of Data Science in action.
The Ignite Proof of Value methodology helps to mitigate risk here with explicit checkpoints and confirmation questions at key junctures, ensuring that whilst we remain agile in the creative process, we continuously evaluate the desired business outcome on the latest findings.
With a detailed, step-by-step process guide, the Ignite Proof of Value approach helps empowers Data Scientists navigate through the phases, activities and artefacts that are required to successfully take a Data Science idea from opportunity definition through to evaluation of its feasibility. The methodology is underpinned by five key questions that we are trying to answer. Keeping these questions in mind throughout the project ensures that we are always working towards the right goals and know when to avoid running down rabbit holes.
Does the methodology encourage and guide us in establishing the size of the prize?
3 - Emphasis on quantifying potential return on investment
The core premise of the Ignite Proof of Value methodology is that we want to give stakeholders confidence in whether the idea can yield a return on investment. This premise drives the key questions presented above and is threaded throughout the process.
The Opportunity Definition and Evaluation phases have a clear focus on proving business value, with explicit tasks included that will help drive the right conversations with stakeholders.
The methodology comes ready with clear guidance on effective ways to consult with business stakeholders to understand the relevant value drivers and pull together a calculation of the potential size of the prize.
Is there any step in the methodology that helps to assess whether the value can feasibly be realised?
4 - Assessment of feasibility of realising value
We also emphasise assessing the feasibility of realising value, within the context of how the business operates.
The Ignite Proof of Value methodology guides the team through a process and value mapping exercise to be very clear about what the current state is and where the idea fits in; we need to understand where the impact is, and where the benefit is realised. There is nothing more disheartening than to run through a Data Science project only to realise that the business process can’t change to adopt the idea!
During Opportunity Definition we encourage working with the business to wireframe a prototype of the desired end product, so that it is clear to stakeholders what it should look and feel like. Whether it is a report, a standalone application, or a widget in an existing interface, having the wireframe will provide clarity and focus during the project to ensure the team is working towards something concrete and useful.
This step also informs stakeholders on potential investments required to scale the idea in the technical landscape as any significant investment will have an impact on the potential return identified previously.
Figure 3: rule of thumb when assessing feasibility
Is the methodology clear on how and when to evaluate results from a business perspective, as opposed to just a technical perspective?
5 - Thorough evaluation of results against business objectives
In the final evaluation, we thoroughly assess the results of the exercise against the business objectives and success criteria established upfront. We strongly emphasise a business evaluation, as opposed to just a technical one.
We have observed that too often, technical performance metrics are considered the measure of success for a Data Science project. In fact, what is needed to justify investment in pursuing an idea is to speak the language of the business; they will want to understand:
- what the results of the exercise mean for how they run their business
- how much value is at risk due to the nature of predictive models
- how much change will be required to implement it
Put clearly, the overarching goal with the methodology is to confidently answer the question: “Has the value been proven, and can the value be realised?”
Does the methodology come with accelerators that help achieve efficiency and consistent quality?
6 - Leverages accelerators to reduce time to outcome
The Ignite Proof of Value methodology also includes accelerators to improve time to outcome and consistency across the organisation.
Many tasks and artefacts are repeatable and can be standardised from project to project. We provide ready-to-go toolsets such as data profiling and experiment tracking capability, letting the Data Science team avoid rework and allowing focus on high-value activities.
The methodology provides templates and checklists that encourage the team to capture and present the right details. For the Opportunity Definition and Evaluation phases, accelerators are aimed at helping Data Science teams communicate back to the business in a language they will understand.
The methodology also provides standard project documentation to assist project managers who are also adapting to this growing discipline. The provision of templates from stakeholder engagement to project planning aim to accelerate mobilisation and project governance.
So, how successful are your Data Science projects?
Over the course of three articles we have discussed common issues, outside of the obvious technical and data challenges, facing Data Science initiatives.
By incorporating the lessons we have learnt, and continue to learn, into the Ignite Proof of Value methodology we look to tackle these issues and build upon best practices in Data Science. We place greater emphasis on giving confidence to the business that their investment is being well spent on the right ideas.
We encourage organisations to adopt a methodology – just like they would for the delivery of any project. The end goal is to see more Data Science projects make it into production and into the hands of users – to improve the organisations we work in and the communities we live in.
For a walkthrough of the Ignite Proof of Value methodology or to diagnose your Data Science projects contact us.