Machine Learning - Supervised Learning & Ensemble Methods Flashcards

Question 1

What is Supervised Learning?

Accepted Answer

A machine learning task where an algorithm learns from a dataset of labeled examples, meaning each input feature vector has a corresponding correct output value (label). The goal is to learn a mapping function from inputs to outputs to make predictions on unseen data.

Question 2

What is 'labeled data' in supervised learning?

Accepted Answer

Data where each input example (features) is paired with its correct output value or target variable (label). For instance, in an image classification task, an image (input) would be labeled with the object it contains (output, e.g., 'cat' or 'dog').

Question 3

Differentiate between Regression and Classification tasks in supervised learning.

Accepted Answer

1. **Regression**: Predicts a continuous output value (e.g., house price, temperature).
2. **Classification**: Predicts a categorical output value (e.g., spam/not spam, disease/no disease, digit recognition).

Question 4

Explain the Bias-Variance Tradeoff.

Accepted Answer

A fundamental concept describing the conflict in simultaneously minimizing two sources of error that prevent supervised learning algorithms from generalizing beyond their training data:
*   **Bias**: Error from erroneous assumptions in the learning algorithm (underfitting).
*   **Variance**: Error from sensitivity to small fluctuations in the training set (overfitting).
The tradeoff implies that reducing one typically increases the other.

Question 5

What is overfitting in machine learning?

Accepted Answer

Occurs when a model learns the training data too well, including its noise and specific patterns, leading to excellent performance on the training set but poor generalization to new, unseen data. High variance is a characteristic of overfitting.

Question 6

What is underfitting in machine learning?

Accepted Answer

Occurs when a model is too simple to capture the underlying patterns in the training data, resulting in poor performance on both the training set and new data. High bias is a characteristic of underfitting.

Question 7

What is K-Fold Cross-Validation and why is it used?

Accepted Answer

A technique to evaluate a model's performance and assess its generalization ability. The data is split into $K$ equally sized folds. The model is trained $K$ times, each time using $K-1$ folds for training and the remaining fold for validation. The results are averaged. It helps mitigate overfitting to the validation set and provides a more robust estimate of performance.

Question 8

Explain Linear Regression and its objective function.

Accepted Answer

A supervised learning algorithm used for regression tasks. It models the relationship between a dependent variable and one or more independent variables by fitting a linear equation to the observed data.
*   **Equation**: $y = \beta_0 + \beta_1 x_1 + \dots + \beta_n x_n + \epsilon$
*   **Objective (MSE)**: Minimize the Mean Squared Error (MSE), which is $\frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2$.

Question 9

Describe Logistic Regression and its use.

Accepted Answer

A supervised learning algorithm primarily used for binary classification tasks. Despite its name, it's a classification algorithm. It uses the logistic (sigmoid) function to output a probability score between 0 and 1, which is then thresholded to assign a class label.
*   **Sigmoid Function**: $P(y=1|x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_n x_n)}}$

Question 10

How do Decision Trees work for classification?

Accepted Answer

A non-parametric supervised learning method used for both classification and regression. It partitions the data into subsets based on feature values, forming a tree-like structure of decisions (internal nodes) and outcomes (leaf nodes). Splits are chosen to maximize information gain or minimize Gini impurity.

Question 11

Explain the core idea behind Support Vector Machines (SVMs).

Accepted Answer

A powerful supervised learning algorithm for classification and regression. For classification, SVMs aim to find the optimal hyperplane that maximally separates data points of different classes in a high-dimensional space. The 'support vectors' are the data points closest to the hyperplane, defining the margin.

Question 12

What is the 'Kernel Trick' in SVMs?

Accepted Answer

A technique used by SVMs to handle non-linearly separable data. It implicitly maps the input features into a higher-dimensional feature space where the data might become linearly separable, without explicitly calculating the coordinates in that space. Common kernels include polynomial, radial basis function (RBF), and sigmoid.

Question 13

Describe the K-Nearest Neighbors (KNN) algorithm.

Accepted Answer

A non-parametric, lazy supervised learning algorithm used for both classification and regression. For a new data point, it finds the $K$ closest data points (neighbors) in the training set based on a distance metric (e.g., Euclidean distance). For classification, it assigns the class most common among its $K$ neighbors; for regression, it averages their values.

Question 14

What are Ensemble Methods in machine learning?

Accepted Answer

Techniques that combine multiple individual machine learning models (often called 'weak learners' or 'base estimators') to achieve better predictive performance than any single model could achieve alone. The core idea is that a group of diverse models can collectively make more accurate and robust predictions.

Question 15

Why are Ensemble Methods generally preferred over single models?

Accepted Answer

They typically improve accuracy, reduce variance (bagging), reduce bias (boosting), and are more robust to noise and outliers. By combining diverse perspectives, they can capture more complex patterns and generalize better to unseen data.

Question 16

Explain Bagging (Bootstrap Aggregating).

Accepted Answer

An ensemble method that trains multiple base models (e.g., Decision Trees) on different bootstrap samples (random samples with replacement) of the original training data. The final prediction is made by averaging (regression) or majority voting (classification) the predictions of individual models. It primarily reduces variance.

Question 17

How do Random Forests improve upon standard Decision Trees?

Accepted Answer

An extension of Bagging that uses Decision Trees as base estimators. In addition to bootstrap sampling, Random Forests introduce an extra layer of randomness: when splitting a node, only a random subset of features is considered. This decorrelates the individual trees, further reducing variance and preventing overfitting.

Question 18

What is the general principle of Boosting?

Accepted Answer

An ensemble method that sequentially builds models, where each new model attempts to correct the errors of the previous ones. It focuses on misclassified or harder-to-predict instances, iteratively improving the overall model's performance. Boosting primarily reduces bias.

Question 19

Describe the AdaBoost (Adaptive Boosting) algorithm.

Accepted Answer

One of the first successful boosting algorithms. It trains a sequence of weak learners (typically shallow Decision Trees, called 'stumps'). In each iteration, it adjusts the weights of the training instances, giving more weight to misclassified examples, and then trains a new weak learner on the re-weighted data. Final prediction is a weighted sum of weak learners.

Question 20

Explain the core idea behind Gradient Boosting.

Accepted Answer

A powerful boosting technique where new models are trained to predict the *residuals* (errors) of the previous models. It uses gradient descent to minimize the loss function by iteratively adding weak learners that 'push' the predictions closer to the true values.

Question 21

What are the key advantages of XGBoost?

Accepted Answer

An optimized distributed gradient boosting library designed for speed and performance. Key advantages include:
*   **Regularization**: L1 and L2 regularization to prevent overfitting.
*   **Parallel Processing**: Can run in parallel across multiple cores.
*   **Handling Missing Values**: Built-in handling for missing data.
*   **Tree Pruning**: 'Greedy' algorithm that prunes trees after construction.
*   **Flexibility**: Supports various objective functions.

Question 22

Describe the Stacking (Stacked Generalization) ensemble method.

Accepted Answer

An ensemble method that combines multiple diverse models using a 'meta-learner' or 'blender.' The base models are trained on the original data, and their predictions are then used as input features for a second-level model (the meta-learner), which makes the final prediction. It aims to leverage the strengths of different models.

Question 23

Briefly compare Bagging and Boosting.

Accepted Answer

*   **Bagging**: Parallel, reduces variance, trains models independently, combines by averaging/voting. Example: Random Forest.
*   **Boosting**: Sequential, reduces bias, trains models iteratively to correct errors, combines by weighted sum. Example: AdaBoost, Gradient Boosting, XGBoost.

Machine Learning - Supervised Learning & Ensemble Methods

Cards in this deck

Related Flashcard Sets

Korean TOPIK I Level 2 - Core Vocabulary & Sentence Structure

Azure Administrator (AZ-104) - Core Infrastructure & Services

JLPT N4 - Essential Kanji & Grammar

MATH201 - Real Analysis Fundamentals