Strategies for Handling Missing Values in Machine Learning
Written on
Introduction to Missing Values
In numerous predictive modeling scenarios, it's common to encounter missing feature values. For instance, in healthcare, patient records may lack critical diagnostic tests that aid in assessing diagnosis likelihood or predicting treatment success. Similarly, consumer data often omit certain attributes essential for understanding buying behavior.
Understanding the Context of Missing Data
It's vital to differentiate between two situations: missing features may occur during the training phase, or they might be absent in test cases when making predictions.
Removing Instances
One common strategy for handling missing values is simply to discard instances that contain them. This approach is often chosen by data engineers wanting to evaluate how well a learning method performs on a specific dataset. This method is suitable if the missing data is completely random. However, during prediction, it may be appropriate to exclude instances with missing values if it makes sense not to predict in certain cases. It's essential to weigh the costs of inaction against the costs of making incorrect predictions.
Acquiring Missing Values
In practice, obtaining a missing value may involve some expense. To optimize expected utility, one must assess the expected benefits of acquiring the missing value against the potential effectiveness of the alternatives. This decision necessitates a comprehensive understanding of available options and their performance comparisons.
Imputation Techniques
Imputation encompasses a range of methods aimed at estimating the missing value or its distribution to inform predictions from a model. Specifically, a missing value can either be substituted with an estimated value or the distribution of possible missing values can be estimated, leading to probabilistic model predictions.
There are various imputation methods applicable to training data that can also be utilized during prediction. Notably, multiple imputation—a Monte Carlo technique—generates several simulated versions of a dataset, analyzes each, and combines the results to draw inferences.
Predictive Value Imputation (PVI)
Through value imputation, missing feature values are filled with estimated values prior to model application. Imputation methods can vary significantly in complexity. A typical method involves replacing a missing value with the mean or mode of the attribute derived from the training data. Alternatively, imputation can be done using the average of the values of other attributes in the test case.
Distribution-based Imputation (DBI)
When one has an estimated distribution for an attribute's values, they can forecast the expected distribution for the target variable by weighing the possible assignments of the missing values. This approach is widely used in AI research, particularly with classification trees, such as the C4.5 algorithm. In this method, when a missing value is encountered, the instance is divided into several pseudo-instances, each assigned a different value for the missing feature, with weights corresponding to the probabilities of these values based on their frequency in the training data.
Each pseudo-instance follows the appropriate tree branch based on its assigned value, and the class membership probability is determined by the frequency of the class within the training instances associated with the leaf node it reaches. The final estimated probability of class membership is calculated as the weighted average across all pseudo-instances. If multiple values are missing, this process recurses, with weights combining multiplicatively.
Unique-value Imputation
Instead of estimating an unknown feature value, it's possible to substitute each missing value with a distinct unique value. This method is particularly effective when two conditions are met: the absence of a value is influenced by the class variable, and this dependence exists in both training and test data.
Reduced-feature Models
Imputation is necessary when the model requires an attribute whose value is missing in the test instance. An alternative is to apply a different model that only includes attributes available for the test case. For instance, a new classification tree can be created after excluding the features corresponding to the missing test feature from the training data. This reduced-model approach could potentially involve different models for each test instance.
In some cases, model induction can be postponed until a prediction is needed, a method referred to as "lazy" classification tree induction. Additionally, reduced-feature modeling can involve maintaining multiple models to accommodate different patterns of known and unknown test features.
Final Thoughts on Missing Values
Addressing missing values is a crucial aspect of data preprocessing in machine learning. While it may be tempting to discard instances, this can be costly, especially with large datasets. Thus, it is advisable to consider more sophisticated methods as discussed above.
References
- Friedman, R. Kohavi, and Y. Yun. Lazy decision trees. In Howard Shrobe and Ted Senator, editors, Proceedings of the Thirteenth National Conference on Artificial Intelligence and the Eighth Innovative Applications of Artificial Intelligence Conference, pages 717–724, Menlo Park, California, 1996. AAAI Press.
The first video discusses handling missing values in datasets, emphasizing imputation techniques and their importance in machine learning.
The second video covers strategies for dealing with missing data, including model optimization and parameter tuning for improved results.