Choosing appropriate machine learning (ML) models in biomedical research typically depends on overall performance indicators. However, when these indicators yield very similar results across candidate models, determining the best option for practical clinical use becomes difficult. We developed an innovative combined approach that integrates visual examination of the Model Scoring Distribution Analysis (MSDA) with a newly introduced scoring measure (MSDscore). This technique was integrated into the Digital Phenomics platform via the MSDanalyser tool. It was evaluated by creating and assessing 27 predictive models for prognosis in breast, lung, and renal cancers. The method facilitated an in-depth review of the distributions of true positives, false positives, true negatives, and false negatives throughout the scoring range. This uncovered localized performance trends that standard global metrics often miss. While traditional metrics showed only slight differences among the models, the MSDA framework highlighted notable variations in behavior across specific score regions, enabling clearer differentiation. Combining this composite framework with conventional performance measures offers a valuable supplementary perspective, providing greater insight into model selection in clinical and biomedical contexts.
