Our tech stack is a key component to our success when creating client solutions. Whilst we have preferred platforms at Ignite Data Solutions, we always take care to set the context broadly – the technology we deploy always depends on the client and what is right for them.
To determine the best solution for a client we take a considered look at their existing investment. What business value do they want to create? What are they already using? What have they already invested in? Not from just a financial perspective, but in terms of developing organisational skills and internal dependencies on systems and platforms. What ‘breaks’ if we move off certain environments?
We never recommend going down a path that costs a fortune and doesn’t fit with the organisation’s goals or their people’s skill sets. These are key to ensuring that any technical solution is adopted by a client, and stands the test of time.
But… we do have a preferred tech stack, one which we recommend to clients starting, or rethinking, their data investment. Our preference here leverages Cloud capability with Microsoft Azure, Databricks and PowerBI. Our key reason for this preference is that these platforms tend to slot in well with most organisations and provide the core capability in a typical data and analytics landscape. They are very easy to adopt, and that is key.
Microsoft Azure: Cloud Data Platform
Most organisations already use Microsoft, and these days, most are using O365. It is a very natural progression to move platforms into the cloud because they are already there from a start point with their O365 implementations. We can add on Azure services and link in security and active directory for access, quickly and seamlessly – from APIs to integrate with workflows or customer apps, to ingestion services allowing organisations to stream internal and external datasets. While everyone probably has their preference on cloud providers (be it Azure, AWS or GCP), they all offer great services. You will typically also find these services offer similar capabilities. We just find that Azure is a logical start point for many organisations.
PowerBI: Data Visualisation
PowerBI is just so easy for a business to turn on. It’s free for users to get started, and depending on their O365 licensing, they may even already have access to additional PowerBI Pro capability – making it super cost effective. Power BI is a great visualisation tool, deliberately aimed at a low barrier to entry, but with ability to build out more complex solutions when required. Because it’s easy to use we find it easy to engage end users, which means that they start to really care about the solution. In fact, the chances are that every client already has business users out there using PowerBI, so often, our task is around expanding access and users, while developing the platform and what it can do.
Databricks: Compute Powerhouse
Databricks is incredibly versatile, and our Data Scientists and Engineers just love working with it. It can run on any cloud platform – AWS, Azure or GCP – though technically, these are slightly different products. Databricks has customised elements of their build for each cloud provider. It is easy to deploy and use in the Azure environment – but we can and do work with AWS/Databricks.
We leverage Databricks essentially for two key functions: Advanced Analytics (e.g. Machine Learning) and for scalable data transformation and curation (ETL and ‘Delta Lake’ capability). With the simple notebook interface, our Data Scientists and Engineers can build out their code in stages, tracking output as they go. With the ability to code in multiple languages that can be switched between, such as Python, SQL, R, Scala – we can select the languages depending on the user. And underneath, it’s converting and leveraging SPARK.
Leveraging clusters we can scale any advanced analytics or transformation job by ‘throwing more grunt at it’ whilst managing the associated cost. We don’t have to change platforms when our client’s business, data or needs grow – we just make a larger cluster to crunch through more data to meet service level agreements. Being cloud based you only pay for what you use – if you only need to run for 10 minutes, we pay for 10 minutes. Then continue to scale as we grow.
Things always change – products are constantly coming out – there are a lot of changing services. We stay on top of the new solutions through monitoring new releases and encourage our Data Scientists and Engineers to ‘play’ with any new capability. We firmly believe in play as a form of learning. With a team Azure account, the team can spin something up – have a play – and create a prototype – this keeps the curiosity, creativity and excitement always burning.
