Bias and Variance in Machine Learning: A Complete Guide with Examples
Machine learning has now emerged as one of the key technologies which enable recommendation engines, fraud detection, healthcare predictions, image recognition, and customer analytics. However, development of a machine learning algorithm involves much more than obtaining high levels of accuracy in the training set. For an effective machine learning algorithm, accuracy needs to be obtained even on unseen datasets.
Among other things, bias and variance are two key elements which explain the poor performance of certain machine learning algorithms and very good performance of others in training phase.
In this blog, we will cover what bias and variance are, their connection with underfitting and overfitting, bias-variance trade-off, examples, and ways to develop effective machine learning algorithms.
What Is Bias in Machine Learning?
Bias can be defined as the error caused due to simplifying assumptions made by machine learning algorithms regarding the relationship between the input variables and output variable.
In other words, high bias indicates a situation where the algorithm is too simple to capture patterns in the data.
Take for example the task of predicting house prices. There are many factors affecting house prices such as the location, size of the house, the number of bedrooms, facilities, age of the house and so forth. If a simple linear model is used that depends only on the size of the house then the model will fail to capture the important relationships between all these variables and the selling price.
Consequently, the model would give wrong predictions always.
Example of High Bias
Let’s assume that we are trying to predict students' examination marks from their study hours. In reality, there is a nonlinear relation between study hours and marks since every student improves at a certain rate depending on his/her preparation.
Therefore, using a simple linear model here leads to high bias.
Features of High Bias
The model makes too many simplifying assumptions.
The important patterns in the training data are ignored by the model.
The training error is high.
The validation error and testing error are high.
The model usually experiences underfitting.
How to Reduce Bias?
Here are some methods to reduce high bias:
1. Increase model complexity: Go for an algorithm that learns complex relationships such as Polynomial Regression or Decision Tree.
2. Add more features: Go for useful variables that can explain the target variable.
3. Enhance feature engineering: Develop meaningful interaction terms and features if necessary.
4. Less regularization: Heavy regularization can make it difficult for the model to learn important patterns.
5. Algorithm selection: Go for a model that fits the complexity of the dataset.
This doesn’t mean that you have to go for the most complex models.
What Is Variance in Machine Learning?
Variance refers to the extent to which the model changes its predictions based on the training on different samples from the same distribution.
High variance in layman's terms means that the model is too responsive to training data.
A model with high variance might discover important patterns, but it will also learn noise, anomalies and non-existent relations between the variables present in the training data.
For instance, a very large decision tree might learn specific observations and be highly accurate at training time while being inaccurate at making predictions for new customers.
Such behavior is typical for overfitting.
High Variance Example
Let us assume that we create a customer-purchase predictor based on the historic data of customers.
Our model predicts the purchase based on age, salary, browsing history, and purchase history of the customer.
If our model has learned extremely specific rules from each observation then it will be very accurate for these customers.
New customers will most likely behave differently and the model will not be able to generalize due to learning some coincidences.
Common Features of High Variance
The model is highly responsive to variations in the training data set.
Training error is usually very low.
Validation/test error is much higher compared to training error.
Predictions will vary widely between training sets.
Overfitting of the model is common.
How Can Variance Be Reduced?
There are several methods that can be used to lower variance:
1. Model simplification: Lowering the number of decision tree nodes or adopting a simpler model.
2. Increasing data representation: More data will enable the model to discern between noise and real signals.
3. Regularization: L1 and L2 regularization will prevent the creation of a complicated model.
4. Cross Validation: Try out the model’s performance on several different data sets.
5. Ensemble Learning: The techniques of bagging and random forests help make the model stable.
6. Feature Selection: Get rid of those features that do not improve the model’s validation performance.
7. Early Stopping: Stop the training process when the performance on validation decreases.
Everything depends on the specific model, the data set and the cause of overfitting.
Understanding Underfitting and Overfitting
Underfitting and overfitting are two major problems faced in machine learning. It will help us in diagnosing whether the model is learning well or not.
Underfitting: When the Model is too Simple
In this case, the problem arises when the model fails to learn the underlying relationship in the data.
Suppose the actual relationship between the size of a house and its selling price is curvy. Using a linear relationship for such kind of data will result in missing important information.
The model facing underfitting issue will perform badly on both training and test data.
Some of the remedies to the problem can be:
- Choosing an appropriate algorithm.
- Adding relevant features.
- Improving feature engineering.
- Removing unnecessary regularization.
Overfitting: When a Model Learns Too Much
Overfitting refers to a situation when the model fits the training data too well by learning not only patterns, but also noise.
Consider, for instance, a polynomial regression model with a very high degree. Such a model will fit almost all the training points, although making senseless predictions about new observations.
In case of overfitting, the model will have a small training error but large validation or test error.
Some possible ways out of this situation can be:
- Decreasing model complexity.
- Using regularization.
- Collecting more representative training data.
- Cross-validation.
- Choosing proper features.
- Implementing early stopping.
The idea is to build a model that will learn general patterns rather than memorize the training set.
What Is the Bias–Variance Trade-Off?
The bias-variance trade-off refers to the relationship between the systematic error resulting from a model being too simplistic in its assumptions and the error resulting from fluctuations in training data.
In most cases, the more complicated a model gets in machine learning problems, the lower its bias becomes while the chances of variance increase. Simplifying a model lowers the variance while increasing the bias.
With regard to standard regression with squared error loss function, the theoretical error decomposition for prediction at a certain input becomes:
Expected Squared Error = Bias² + Variance + Irreducible Noise
In this case:
- Bias²: Systematic error because of the assumptions made by the model.
- Variance: Error resulting from fluctuations in training data.
- Irreducible noise: Random error that cannot be reduced by the model.
This formula represents a theoretical scenario of regression analysis using a squared error loss function.
A Practical Example of the Trade-Off
Let us consider three models which predict house prices.
Model A: Simple linear regression model
The model assumes that the relationships between the chosen attributes and the prices of the houses follow a linear pattern. If there is a non-linear pattern in reality, then the model suffers from high bias.
Model B: Very complicated decision tree
The model learns complicated rules based on individual training data points. The model works well for the training data but has a hard time predicting for the properties which were not seen during training because of high variance.
Model C: Well-tuned model
The model knows the required dependencies without being overly complex.
Practical Example: Linear Regression v/s Polynomial Regression
Think of a data set with an input and an output variable, say study hours and examination marks respectively.
Let's suppose that the true relation between study hours and marks is nonlinear.
Linear regression tries to find a linear relation. Polynomial regression makes use of the input features raised to some powers as additional features, thereby enabling itself to model non-linear relations.
Take a look at the following example:
The values above are only hypothetical and used just to illustrate the idea.
Training and test errors of the linear model are quite high, indicating underfitting. Degree-3 polynomial model works better on unseen data. The degree-15 model reaches very low training error but much higher test error, indicating overfitting.
In this case, we can see that training accuracy cannot help to evaluate the model’s adequacy.
How to Identify Bias and Variance in a Machine Learning Project
The data scientists rely on training and validation performance metrics for identifying possible underfitting and overfitting.
Let’s see a typical approach.
Step 1: Data Set Preparation
Perform data preprocessing, handle missing data, detect and process outliers, and perform feature selection.
Split data set into training, validation, and test data sets when required.
Step 2: Train the Baseline Algorithm
- Begin with simple algorithms, evaluate their performance.
It will serve as your baseline and it will be improved upon later on.
Step 3: Analyze the Training and Validation Errors
- Training and validation errors are high – underfitting.
- Training error is low while the validation error is high – overfitting.
- Low errors for both – this means good generalization capabilities if the setup of the analysis is correct.
Step 4: Hyperparameter Tuning
Adjust hyperparameters such as the depth of a decision tree, the degree of a polynomial, the regularization coefficient, or the architecture of a neural network.
Step 5: Cross-Validation
Train the model on different data splits and check whether it is able to give reliable results.
Step 6: Selection of the Model
After choosing the model, evaluate its performance using an independent test dataset. Do not try to tune the model on the test dataset.
For the problem of regression, useful metrics include MAE, MSE, and RMSE. For the problem of classification, useful metrics include accuracy, precision, recall, F1-score, and log loss.
The metric should be relevant to the given business problem.
The chosen metric must be relevant to the specific business task. For instance, recall is very important in fraud detection.
Techniques to Balance Bias and Variance
1. Cross-Validation
Cross-Validation considers models under several training and validation sets. This assists in selecting models that are stable as opposed to those that work well for one particular split.
2. Regularization
Regularization is a process whereby penalties are included into the objective function of a model. This discourages large coefficients.
3. Feature Selection
Eliminating features that are not significant makes the learning problem much simpler. However, eliminating important features results in increased bias.
4. Ensemble Methods
Ensemble Methods combine the predictions of multiple models.
Bagging: Build multiple models based on bootstrap sampling and then combine their predictions; results in decreased variance.
Random Forest: Build multiple decision trees and make better predictions than just with a single decision tree.
Boosting: Train models one after another, correcting the mistakes made by previous models. This method allows decreasing the bias, but bad choice of parameters can lead to overfitting.
5. More Representative Training Data
More high-quality data will allow the model to learn general patterns instead of memorizing particular observations. Of course, it all depends on how representative and relevant the additional data is.
6. Hyperparameters Optimization
Grid search, random search, and Bayesian optimization are techniques for choosing proper model settings.
Real-World Applications of Bias and Variance
Bias and variance are important concepts for many different industries.
Healthcare: The model that is too simple will be unable to detect important correlations between patient features, while an overfitted model will work poorly on other patients.
Finance: It is important for fraud detection models to be complex enough to detect fraudulent behaviour without overfitting transaction history.
E-commerce: Recommendation engines need to learn the preferences of the user from their behavior patterns, without putting too much weight on their interaction history.
Real estate: Models for predicting housing prices need to learn how the property size, location, and other factors affect the price of the property.
Computer vision: Image classifiers need to learn to identify visual patterns even on images they haven’t seen before.
In any case, evaluation and proper model selection become essential.
The important point to draw from this analysis is obvious: the optimal machine learning algorithm is not the one that memorized the training data, but the one that learned how to detect the pattern and predict.
Author:
Shivsharan Kunchalwar
Related Links:
Resume Tips For Software Developers
Do visit our channel to know more: SevenMentor
Shivsharan Kunchalwar
Expert trainer and consultant at SevenMentor with years of industry experience. Passionate about sharing knowledge and empowering the next generation of tech leaders.