Expert Models as an interim solution
We often get asked how we can provide a solution when no data is available. Expert models are nothing new and have been around for decades. They have been widely used and benefited from by many organisations, either opening a new market or product or those with limited volumes or struggling to extract historical data.
Before I address the question, let’s review the types of models there are:
Custom/Bespoke models: those built on historical data and hopefully validated using an out-of-time sample.
Expert/Generic models: built using information about the portfolio and the experience of the scorecard developer.
Semi-expert models: used when data is limited or biased and based on the experience of the scorecard developer using the data available as much as possible.
Whilst it is true that models developed using the full power of ML techniques and good-quality data are the most desirable option, this is not always possible. That should not stop organisations from applying the best tools available, in this case, the other two types of models, in order to assess the risk of any decision made, whether that is at the point of application, collections, limit increases, etc. This is the recommended course of action.
When and how to use expert models
Expert models should not be viewed as the optimal solution but rather as a stepping stone into good credit risk management practices. In my career, I have often seen this type of model outperform custom models. This is surprising. Unfortunately, too many models out there are based on biased data or are allowed to overfit the development sample, resulting in a model which is not robust enough to withstand real-life application into newer and real data. Sometimes, the market is just very dynamic!
Don’t take me wrong, I am not advocating for expert models to be implemented all the time, but they certainly have benefits in certain situations.
Please don’t think this applies only to new portfolios. Organisations with historical data sometimes haven’t kept the data or are unable to easily extract it. As an example, I have been waiting for close to a year for data to develop an application model. By now, expert models could have been implemented and optimised using the latest data, and I would be looking to build a semi-expert with early risk indicators to guide the patterns.
How they are built, and where can they be applied
The next question that may be on your mind is whether these models can be built to predict anything. Although this may be true in principle, I would only recommend them when the problem is well known, and the patterns by portfolio and region are clearly understood by the person creating them. Expert models should not be a copy/paste from organisation to organisation or from portfolio to portfolio. They should be constructed for that organisation/portfolio and reviewed until the real distributions of records by characteristic are known using the data available.
Unfortunately, not every scorecard developer has the knowledge to do this. You need an understanding of the predictive power of characteristics, their patterns and distributions, the pattern jumps within the variable and the correlations that exist between the predictors. If you don’t, you may end up with a model that lacks the score ranges necessary to implement a good decision strategy or worse, is unable to rank order.
Not long ago, I was asked if I had any experience in Egypt, as the client didn’t believe you could build an expert model, or even a custom one, if you don’t have the experience in the market. The position ‘We are different’ is often posed to scorecard developers, who then must provide justification on why they can address the needs of the organisation.
The truth is every portfolio, every country and every situation are indeed different. Best modelling practice, however, still applies, and so do a lot of the patterns we see worldwide. A good scorecard developer will know how to choose the characteristics that will be similar in nature and how to counteract the unknown distributions, jumps and distinct patterns. By doing so, the resulting expert model is great at extrapolating and ensuring that if one of the patterns is wrong, the model still rank orders overall.
As I mentioned before, it is still a temporary solution. Even if the model is good, it doesn’t mean it should not be validated using data as soon as possible. It is just a great starting point to get your portfolio assessment automated and on the way to the best models you can have.
Our solution
At Score4All, we see expert models as a starting point. We have already developed the initial scorecards for a number of portfolios and regions. This, together with our proprietary modelling pipeline allows our clients to start using the models early on, whilst ensuring that they get revised with any data that is available.
Our starting models get adapted to the new distributions as soon as records are processed. This is automated, and so the improvements are seen as early as one or two months of scoring. As performance data becomes available, early risk indicators are used as a performance definition. This has the benefit of being able to automatically build a custom model within 12-18 months, depending on volumes. And all this is completely automated using AI and ML techniques.
The end result is a full modelling solution which, not only provides the way forward for expert models but also ensures the custom models are updated as the portfolio shifts, rather than every 2-3 years.
How to use them
I haven’t yet touched on how to set cut-offs when you are using expert models given that the score-to-odds relationship is unknown. We tend to scale our models in a specific manner for each use case, so we have an idea of where the models stand. This relationship stays constant, no matter how many times we redevelop the scorecards. This applies to the expert models we build, but the relationship will not be exactly known for each portfolio until some data is processed. As this is quite a complex topic, it will be expanded in a separate blog.
So, in short, expert models have a time and a place and can help organisations start their modelling journey.
If you want to know more about these or any other types of models, or you just want to discuss your specific needs, please email us or book a 30 min online meeting with us.
Until next time…
Commenti