ML | Underfitting and Overfitting
Last Updated : 27 Jan, 2025
Machine learning models aim to perform well on both training data and new, unseen data and is considered "good" if:
- It learns patterns effectively from the training data.
- It generalizes well to new, unseen data.
- It avoids memorizing the training data (overfitting) or failing to capture relevant patterns (underfitting).
To evaluate how well a model learns and generalizes, we monitor its performance on both the training data and a separate validation or test dataset which is often measured by its accuracy or prediction errors. However, achieving this balance can be challenging. Two common issues that affect a model's performance and generalization ability are overfitting and underfitting. These problems are major contributors to poor performance in machine learning models. Let's us understand what they are and how they contribute to ML models.
Bias and Variance in Machine Learning
Bias and variance are two key sources of error in machine learning models that directly impact their performance and generalization ability.
Bias: is the error that happens when a machine learning model is too simple and doesn't learn enough details from the data. It's like assuming all birds can only be small and fly, so the model fails to recognize big birds like ostriches or penguins that can't fly and get biased with predictions.
- These assumptions make the model easier to train but may prevent it from capturing the underlying complexities of the data.
- High bias typically leads to underfitting, where the model performs poorly on both training and testing data because it fails to learn enough from the data.
- Example: A linear regression model applied to a dataset with a non-linear relationship.
Variance: Error that happens when a machine learning model learns too much from the data, including random noise.
- A high-variance model learns not only the patterns but also the noise in the training data, which leads to poor generalization on unseen data.
- High variance typically leads to overfitting, where the model performs well on training data but poorly on testing data.
Overfitting and Underfitting: The Core Issues
1. Overfitting in Machine Learning
Overfitting happens when a model learns too much from the training data, including details that don’t matter (like noise or outliers).
- For example, imagine fitting a very complicated curve to a set of points. The curve will go through every point, but it won’t represent the actual pattern.
- As a result, the model works great on training data but fails when tested on new data.
Overfitting models are like students who memorize answers instead of understanding the topic. They do well in practice tests (training) but struggle in real exams (testing).
Reasons for Overfitting:
- High variance and low bias.
- The model is too complex.
- The size of the training data.
2. Underfitting in Machine Learning
Underfitting is the opposite of overfitting. It happens when a model is too simple to capture what’s going on in the data.
- For example, imagine drawing a straight line to fit points that actually follow a curve. The line misses most of the pattern.
- In this case, the model doesn’t work well on either the training or testing data.
Underfitting models are like students who don’t study enough. They don’t do well in practice tests or real exams. Note: The underfitting model has High bias and low variance.
Reasons for Underfitting:
- The model is too simple, So it may be not capable to represent the complexities in the data.
- The input features which is used to train the model is not the adequate representations of underlying factors influencing the target variable.
- The size of the training dataset used is not enough.
- Excessive regularization are used to prevent the overfitting, which constraint the model to capture the data well.
- Features are not scaled.
Let's visually understand the concept of underfitting, proper fitting, and overfitting.
: Bias and Variance- Underfitting : Straight line trying to fit a curved dataset but cannot capture the data's patterns, leading to poor performance on both training and test sets.
- Overfitting: A squiggly curve passing through all training points, failing to generalize performing well on training data but poorly on test data.
- Appropriate Fitting: Curve that follows the data trend without overcomplicating to capture the true patterns in the data.
Balance Between Bias and Variance
The relationship between bias and variance is often referred to as the bias-variance tradeoff, which highlights the need for balance:
- Increasing model complexity reduces bias but increases variance (risk of overfitting).
- Simplifying the model reduces variance but increases bias (risk of underfitting).
The goal is to find an optimal balance where both bias and variance are minimized, resulting in good generalization performance.
Imagine you're trying to predict the price of houses based on their size, and you decide to draw a line or curve that best fits the data points on a graph. How well this line captures the trend in the data depends on the complexity of the model you use.
Underfitting and Overfitting - When a model is too simple, like fitting a straight line to curved data, it has high bias and fails to capture the true relationship, leading to underfitting. For example, a linear model cannot represent a non-linear increase in house prices with size.
- However, if the model becomes too complex, like a fourth-degree polynomial that adjusts to every point, it develops high variance, overfits the training data, and struggles to generalize to new data. This is overfitting, where the model performs well on training but poorly on testing.
- An ideal model strikes a balance with low bias and low variance, capturing the overall pattern without overreacting to noise. For instance, a smooth second-degree polynomial fits the data well without being overly complex.
How to Address Overfitting and Underfitting?
Techniques to Reduce Underfitting
- Increase model complexity.
- Increase the number of features, performing feature engineering.
- Remove noise from the data.
- Increase the number of epochs or increase the duration of training to get better results.
Techniques to Reduce Overfitting
- Improving the quality of training data reduces overfitting by focusing on meaningful patterns, mitigate the risk of fitting the noise or irrelevant features.
- Increase the training data can improve the model's ability to generalize to unseen data and reduce the likelihood of overfitting.
- Reduce model complexity.
- Early stopping during the training phase (have an eye over the loss over the training period as soon as loss begins to increase stop training).
- Ridge Regularization and Lasso Regularization.
- Use dropout for neural networks to tackle overfitting.
Similar Reads
Overfitting and Regularization in ML The effectiveness of a machine learning model is measured by its ability to make accurate predictions and minimize prediction errors. An ideal or good machine learning model should be able to perform well with new input data, allowing us to make accurate predictions about future data that the model
14 min read
Understanding the Overfitting Detector in CatBoost CatBoost, a gradient boosting library developed by Yandex, is known for its efficient handling of categorical features and robust performance. One of its key features is the overfitting detector, which helps prevent the model from overfitting to the training data. Overfitting occurs when a model lea
12 min read
How does L1 and L2 regularization prevent overfitting? Overfitting is a recurring problem in machine learning that can harm a model's capacity to perform well and be generalized. Regularization is a useful tactic for addressing this problem since it keeps models from becoming too complicated and, thus, too customized to the training set. L1 and L2, two
4 min read
How to handle overfitting in TensorFlow models? Overfitting occurs when a machine learning model learns to perform well on the training data but fails to generalize to new, unseen data. In TensorFlow models, overfitting typically manifests as high accuracy on the training dataset but lower accuracy on the validation or test datasets. This phenome
10 min read
Tips to Reduce Over and Underfitting Of Forecast Models Forecast models are powerful tools used across various industries to predict future outcomes based on historical data. However, achieving accurate predictions can be challenging due to two common pitfalls: Overfitting and Underfitting. In this comprehensive guide, we'll delve into tips and strategie
5 min read