RESOURCES HOME
Algorithms are used across all industries to manage big data – from predicting buying trends (how much toilet paper a store should buy) to adding products to online product catalogs (matching logos or product images before they’re uploaded to an eCommerce website) to tallying reward payments (how much to pay individuals based on sales) – and much more.
One challenge with algorithms is that it takes highly skilled software developers numerous hours to develop them, and then ongoing hours to tweak them for new use cases or update them when they find obscure bugs.
Modern machine learning (ML) technology allows you to replace a coded algorithm with an ML model. The ML model does the same work as the coded algorithm, but costs less time and money to maintain. And an ML model can adapt more quickly to changing input data as well.
CoStrategix recently helped one of its clients in the fin-tech industry to reduce the cost of development with ML.
The Case for Replacing Algorithms with ML
In this case, our client was using a complex, monolithic algorithm to match noisy and sometimes incomplete sales data to a product catalog. This task is relatively simple for a human to perform, but with over one million transactions per day, a solution that involved human interaction was just not scalable.
When we replaced the algorithm with an ML model, our client was able to maintain its business process with a part-time data auditor at a fraction of the cost that it would have cost for a software developer. In this case, the data auditor was a trained data analyst. But in many cases, the auditor may be a subject matter expert from the business team, or even an entry-level employee with minimal training.
The cost of maintaining the business process was significantly reduced when implemented as an ML model, as compared to the original algorithm. In addition, the ML model was able to adapt to changing input data and data patterns through the simple data audit process, without the need for use case analysis, solutioning, software updates, code testing, and deployment. Even though there was an upfront cost to create and deploy the ML model, the company achieved a solid return on investment in the long term.
Identifying Good Use Cases for ML
Our case above was successful because the problems our client was solving didn’t have to have an exact solution. This business process itself could accept some level of inaccuracy in the data matches. (For example, matching “Coors Lt Bottle” against “Coors Liter Bottle” instead of “Coors Light.”) The data was used for analytics, not for billing or finance. So even with imperfect matches, we could still determine whether the Coors brand was trending up or down.
There are a number of use cases to reduce the cost of development with ML to replace algorithms. For example, ML is good at replacing complicated, rule-based software. We can train an ML model simply by providing examples of how the rules are applied, as opposed to trying to reverse engineer and then program the rules themselves. In general, ML can be useful for:
- Classification
- Anomaly detection / Duplicate detection
- Matching
- Prediction
- Pattern detection
ML can become more costly when there is a requirement for a high degree of accuracy, or when a wrong prediction is not acceptable.
The Transition to ML Doesn’t Have to Be Difficult
Training an ML model nowadays is surprisingly simple. You don’t have to write any complex code, you don’t need to run a compiler or a builder, and you don’t need to hire a data scientist. There are a variety of low-code and no-code options for implementing ML today, both commercial and open-source. Alternatively, your cloud provider probably offers APIs with pre-trained models for a variety of use cases and plenty of training and documentation on how to use them.
Initially, you will run the newly trained model in parallel with your production algorithm. Eventually, when your model starts to outperform your algorithm (in terms of accuracy), then you can retire the algorithm, and just run the production data through the ML model. Now your dependency on the expensive algorithm is removed, and you will have a solution that can be maintained solely by a data analyst, or an auditor with minimal training.
Our advice: don’t hesitate to jump on the ML bandwagon. You may have opportunities right now to replace costly and risky algorithms with ML models that can be maintained by a data analyst or an auditor. This would free up developers from tweaking and debugging legacy code – tasks at which developers are historically inefficient. In turn, this would allow them to work on more valuable tasks, especially tasks such as data engineering or MLOps, which can enable further use of ML in the future.
For more details, read our other blog for guidance on How to migrate an Algorithm to an ML model.
CoStrategix helps companies solve business problems with digital and data solutions. If you are struggling with how to apply data science to your business challenge, feel free to reach out to us.