Lecture 5. Stacking and Blending

Date: 2023-05-18

1. Ensemble Methods: Overview

Here's a categorized list of popular ensemble techniques:

Basic Ensemble Techniques
- Voting:
  - Majority Voting: Used for classification problems. The class that gets the most votes from individual models is chosen as the final prediction.
  - Weighted Voting: Each model's vote is assigned a weight based on its performance or reliability.
- Averaging: Used for regression problems. The average prediction from all models becomes the final prediction.
- Weighted Averaging: Each model's prediction is weighted, often according to performance, before averaging.
Bootstrap Aggregating (Bagging) Methods (see Lecture 3)
- Random Forests: Constructs multiple decision trees during training and outputs the mode of the classes (classification) or mean prediction (regression) of the individual trees.
- Bagged Decision Trees: Uses bootstrapping to create multiple subsets of the original dataset and then trains a decision tree on each subset. The final prediction is an average or majority vote of the predictions from individual trees.
Boosting Methods (see Lecture 3)
- AdaBoost (Adaptive Boosting): Starts with a weak learner and iteratively adjusts the weights of misclassified instances, making them more likely to be correctly classified in the next iteration.
- Gradient Boosting Machines (GBM): Builds decision trees iteratively, where each tree corrects the errors of its predecessor.
- XGBoost (Extreme Gradient Boosting): An optimized gradient boosting library that's particularly efficient and popular.
- LightGBM: Gradient boosting framework that uses tree-based algorithms and is designed to be distributed and efficient.
- CatBoost: Boosting algorithm that can handle categorical variables without preprocessing.
Stacking
- Stacked Generalization: Base models are trained on a complete training set, then a meta-model is trained on the outputs (predictions) of the base models.
- Blending: Similar to stacking, but the dataset is split into a training set and a validation set. The base models are trained on the training set, and the meta-model is trained on the predictions of the base models on the validation set.
Others
- Bayesian Model Averaging (BMA): Uses Bayesian methods to average over multiple models.
- Bayesian Model Combination (BMC): Combines models instead of just averaging them.
- Bucket of Models: A strategy where multiple models are trained, and the best one is selected based on performance on a validation set.
- Cascade Generalization: Base models are trained on different representations of the training data. These models' predictions are then combined using another model or heuristic.

Category	Technique/Method	Brief Description
Basic Ensemble Techniques	Voting (Majority & Weighted)	Combines class predictions; the class with most votes is chosen.
	Averaging & Weighted Averaging	Average predictions from all models.
Bagging	Random Forests	Multiple decision trees; mode or mean of their outputs.
	Bagged Decision Trees	Bootstrapped subsets; average or majority vote of trees.
Boosting	AdaBoost	Iteratively adjusts weights of misclassified instances.
	Gradient Boosting Machines (GBM)	Trees built iteratively, each correcting its predecessor.
	XGBoost	Optimized gradient boosting.
	LightGBM	Gradient boosting with tree-based algorithms.
	CatBoost	Boosting for categorical variables.
Stacking	Stacked Generalization	Meta-model trained on outputs of base models.
	Blending	Like stacking but with data split strategy.
Others	Bayesian Model Averaging (BMA)	Bayesian methods to average models.
	Bayesian Model Combination (BMC)	Combines models using Bayesian methods.
	Bucket of Models	Multiple models; best one is selected based on validation.
	Cascade Generalization	Base models trained on different data representations.

2. Stacking

Ensemble methods, at their core, are about combining multiple models to improve overall predictive performance. Stacking (or stacked generalization) stands out as it doesn't just combine models – it introduces a meta-model to make the final predictions based on the individual models' outputs.

The Idea of Stacking

Definition:

Stacking involves training a new model (often referred to as a meta-model or blender) to combine the predictions of several base models.

Workflow:

Base Models: Train multiple models on your training dataset.
Predictions as Input: Use these models to make predictions on a validation set. These predictions become the inputs for the next step.
Meta-model Training: Train a meta-model on the predictions from step 2 to make a final prediction.

The magic of stacking is that the meta-model can learn which model (or models) tends to be most accurate for different types of data or in different parts of the input space.

Training Stacked Model

Steps:

Split the Training Data: Often, the training data is split into a training set and a "holdout" set.
Train Base Models: Base models are trained on the training set.
Generate Meta-Features: Base models predict on the holdout set. These predictions are the "meta-features" for the next step.
Train Meta-model: Use the meta-features from the previous step and the true target values of the holdout set to train the meta-model.

In a more advanced approach, K-fold cross-validation can be used to generate out-of-sample predictions for training the meta-model.

Multiple Stacked Models

For added complexity and potential performance gains, you can:

Stack Multiple Layers: Instead of just having one layer of base models and one meta-model, you could have multiple layers. Predictions from one layer become input features for the next layer.
Different Meta-models: Instead of using a single type of model as the meta-model, you could experiment with different types. For instance, if your base models are all tree-based, you might try a linear regression or neural network as your meta-model.

Example

Imagine we're working on a classification problem.

Base Models:

Decision Tree
Support Vector Machine
K-Nearest Neighbors

Training:

We train these models on our training dataset.
Next, we let these models predict on a validation set. So for each instance in our validation set, we'll have three predictions (one from each model).
These predictions become the input features for our meta-model. If our validation set has 1000 instances, our meta-model's input will be a 1000x3 matrix.

Meta-model:

For simplicity, let's use logistic regression as our meta-model.
This logistic regression model will learn, for instance, that for input types A, B, and C, it should weigh the decision tree's prediction more heavily, but for input types X, Y, and Z, the SVM's prediction is more trustworthy.

By doing this, the meta-model captures the best aspects of each base model and can deliver superior performance than any single model.

3. Blending

While stacking has been an immensely popular ensemble technique, blending emerged as a simplified alternative that in practice often performs just as well, if not better in certain scenarios. Both stacking and blending are about leveraging the strength of multiple models, but the process of aggregating their predictions differs.

The Idea of Blending

Definition:

Blending, much like stacking, involves using multiple base models. However, instead of using a meta-model to combine the base models' predictions, blending usually takes a simpler approach, such as weighted averages.

Workflow:

Base Models: Similar to stacking, train various models on your training dataset.
Holdout Set Predictions: Instead of using cross-validation like in stacking, blending typically employs a single validation (or holdout) set. The base models make predictions on this holdout set.
Combine Predictions: The predictions from the base models on the holdout set are combined using straightforward techniques like weighted averages or other custom logic.

Training Blended Model

Steps:

Train/Test Split: Divide your data into a training set and a test set (you might also hear this referred to as a holdout set in this context).
Train Base Models: Just like in stacking, you train your base models on the training set.
Generate Predictions: Use the base models to predict on the holdout/test set.
Blending Predictions: Rather than training a meta-model, take a simpler approach. This could be as basic as taking the mean of the predictions (for regression problems) or a vote (for classification). For a more nuanced approach, you could assign different weights to models based on their perceived accuracy or relevance.

Advantages Over Stacking

Simplicity: Blending is generally simpler and more straightforward than stacking. You're removing the need for a meta-model, which simplifies the training process.
Less Risk of Overfitting: Because blending typically uses a single validation set and doesn't require additional model training (like the meta-model in stacking), there's often less risk of overfitting.
Efficiency: Blending can be more computationally efficient, especially if the meta-model in stacking is complex.

Relating Back to Stacking

While both stacking and blending utilize multiple base models to improve prediction accuracy, their methods of combining these predictions are what set them apart. Stacking uses a more complex approach with a meta-model to refine predictions, while blending often opts for simpler aggregation methods. In essence, while stacking tries to learn the best way to combine models, blending provides a more direct way of combining them.

4. Q&A

1. What is the primary difference between stacking and blending?

Answer: The main difference lies in how they combine the predictions from base models. Stacking uses a meta-model to combine the predictions, while blending often relies on simpler aggregation methods, like weighted averages.

2. In the context of blending, what is a holdout set?

Answer: In blending, a holdout set is a portion of the dataset that the base models don't train on but use to make predictions. These predictions are then combined using a blending technique to form the final prediction.

3. Why might blending be considered less prone to overfitting than stacking?

Answer: Blending typically uses a single validation or holdout set and avoids additional model training (like the meta-model in stacking). This simpler approach often results in less risk of overfitting.

4. Which ensemble technique, stacking or blending, uses cross-validation for its base models?

Answer: Stacking commonly uses cross-validation for its base models to generate predictions for the meta-model.

5. How do the base models in stacking and blending differ in their training?

Answer: In stacking, base models are typically trained on the entire dataset using cross-validation, while in blending, they're trained only on a training set, excluding the holdout set.

6. Why might someone choose blending over stacking, even if stacking is potentially more accurate?

Answer: Blending is generally simpler, more straightforward, and computationally efficient compared to stacking. These advantages can sometimes make blending a preferred choice, especially in scenarios where simplicity and speed are prioritized.

7. How does the meta-model in stacking get trained?

Answer: The meta-model in stacking is trained on a "meta-dataset" where the features are the predictions of the base models, usually obtained via cross-validation on the training data.

8. In what situations might blending outperform stacking?

Answer: Blending might outperform stacking when the meta-model in stacking overfits the base model predictions or when computational efficiency is crucial, and the simpler approach of blending is favored.

9. Can stacking and blending be used together in a single ensemble method?

Answer: Yes, it's possible to use them in tandem. For instance, one could blend several stacked models together, taking advantage of both techniques' strengths.

10. Why is it essential to ensure that the meta-model in stacking remains simple?

Answer: A complex meta-model can overfit the predictions of the base models, which could diminish the ensemble's generalization to new, unseen data.

11. What's the primary purpose of using either stacking or blending?

Answer: Both stacking and blending aim to improve predictive performance by combining multiple models, leveraging their individual strengths and compensating for their weaknesses.

12. How does blending prevent "leakage" that might occur in stacking?

Answer: Blending uses a separate holdout set for generating predictions, which are then used to blend or combine the base models. Since the holdout set is never seen during the training of base models, there's a reduction in potential data leakage compared to stacking.

13. If you have limited computational resources, which would you lean towards: stacking or blending?

Answer: Blending would typically be preferred in such cases since it's computationally less intensive. Without the need for repeated cross-validation used in stacking, blending tends to be faster.

14. How can you ensure diversity among base models in stacking or blending?

Answer: Ensuring diversity can be achieved by using different algorithms as base models, using different subsets of data, varying hyperparameters, or even using different feature engineering techniques for each model.

15. Why is diversity among base models important in ensemble techniques like stacking or blending?

Answer: Diversity ensures that individual model weaknesses are compensated by others. If all models make similar errors, the ensemble might not offer any benefit over individual models.

16. What are the potential pitfalls of using too many base models in stacking?

Answer: Using too many base models can lead to increased computational complexity, longer training times, and risk of overfitting, especially if the meta-model becomes too reliant on nuanced patterns that aren't generalizable.

17. Can stacking use different algorithms for its base models?

Answer: Yes, stacking can utilize a mix of diverse algorithms for its base models, from linear regression to decision trees, enhancing the ensemble's robustness.

18. In blending, how do you decide the weights for each base model's prediction when combining them?

Answer: Weights can be assigned based on the performance of each model on the holdout set. Models with better holdout performance might be given higher weights. Alternatively, optimization techniques can be used to find the best combination of weights.

19. Is there a limit to how many layers you can have in multi-layer stacking?

Answer: Technically, no, but with each additional layer, the risk of overfitting increases, and the complexity and computation time also grow. Usually, one or two layers are sufficient to achieve improved performance.

20. How would you handle overfitting in a stacked model?

Answer: To prevent overfitting in stacking, you can:

Use simpler base models.
Keep the meta-model simple.
Regularize the meta-model.
Use fewer base models.
Rely on larger training datasets.
Ensure cross-validation is correctly implemented.