Understanding Data Drift in Machine Learning Models for Diabetes Management
Prediction algorithms trained on historical data and deployed in dynamic environments are at risk from data drift. Machine learning models that rely on continuous measurements from sensors are particularly vulnerable to changes in both the devices themselves and their users. This can lead to drift, which has significant implications for the safety and security of these models. To maintain predictive performance, algorithms must be continually monitored and optimized to address fundamental changes in input data (covariate shift) and the relationship to the output (concept drift). This article explores how changes in user behavior, physiology, and sensors might impact model safety using automated sensor readings from continuous glucose monitors (CGM).
Methods and Analysis
In this article, we examine how data drifts in a machine learning model trained to predict short-term risks from glycemic control in individuals with type 1 diabetes. We simulate how changes in both user behavior and sensor accuracy can lead to covariate shifts and concept drift. For each scenario, we quantify changes to the input data using Jensen-Shannon divergence, assess the impact on model performance metrics, and evaluate the explainability of the model through shifts in feature importance.
Results
Our findings demonstrate that combining covariate shift detection with multiple performance metrics and feature importance analysis offers a robust approach for identifying various biases in sensor data. In the context of blood glucose management, our scenarios focused on user behavior, such as changes in blood glucose dynamics and CGM usage, along with device/sensor noise and variability. Relying on simpler methods for drift detection could result in misidentifying risks to model safety.
Conclusion
Machine learning and AI have the potential to enhance clinical decision-making, but they often lack the transparency necessary to ensure ongoing safety. By integrating complementary monitoring techniques, it becomes easier to identify changes in data or model behavior and to determine when retraining or intervention is required.
For further details, please refer to the original study Here.
“`
