Stanford CS229 - Machine Learning: Supervised Learning & Loss Functions Flashcards

Question 1

Supervised Learning

Accepted Answer

A machine learning paradigm where the algorithm is provided with a training dataset consisting of labeled input-output pairs $\{(x^{(i)}, y^{(i)})\}_{i=1}^m$. The goal is to learn a mapping function $h: X \	o Y$ that accurately predicts outputs for unseen inputs.

Question 2

Hypothesis Function (Linear Regression)

Accepted Answer

The linear predictor function defined as $h_\	heta(x) = \	heta^T x = \sum_{j=0}^n \	heta_j x_j$, where $\	heta \in \mathbb{R}^{n+1}$ represents the parameter vector and $x_0 = 1$ is the intercept/bias term.

Question 3

Mean Squared Error (MSE) Cost Function

Accepted Answer

The standard loss function for linear regression measuring average squared prediction error:
$$J(\	heta) = \frac{1}{2m} \sum_{i=1}^m (h_\	heta(x^{(i)}) - y^{(i)})^2$$
The factor of $\frac{1}{2}$ simplifies the derivative during gradient optimization.

Question 4

Batch Gradient Descent

Accepted Answer

An optimization algorithm that updates parameters $\	heta$ by computing the gradient of the cost function $J(\	heta)$ over the **entire** dataset at every step:
$$\	heta_j := \	heta_j - \alpha \frac{\partial}{\partial \	heta_j} J(\	heta) = \	heta_j + \alpha \frac{1}{m} \sum_{i=1}^m (y^{(i)} - h_\	heta(x^{(i)})) x_j^{(i)}$$

Question 5

Stochastic Gradient Descent (SGD)

Accepted Answer

An optimization algorithm that updates parameters based on a **single training example** at a time:
$$\	heta_j := \	heta_j + \alpha (y^{(i)} - h_\	heta(x^{(i)})) x_j^{(i)}$$
It is significantly faster than Batch Gradient Descent for large datasets, though its updates are noisier.

Question 6

Normal Equation

Accepted Answer

An analytical, closed-form solution for linear regression parameter estimation:
$$\	heta = (X^T X)^{-1} X^T y$$
Where $X$ is the $m \	imes (n+1)$ design matrix and $y$ is the target vector. It eliminates the need for iterative gradient descent but requires $O(n^3)$ matrix inversion time.

Question 7

Locally Weighted Regression (LWR)

Accepted Answer

A non-parametric algorithm that fits a local regression line around a target point $x$ by minimizing:
$$\sum_{i=1}^m w^{(i)} (y^{(i)} - \	heta^T x^{(i)})^2$$
Where weights are typically exponential functions of distance: $w^{(i)} = \exp\left(-\frac{(x^{(i)} - x)^2}{2\	au^2}\ight)$ with bandwidth parameter $\	au$.

Question 8

Logistic Regression Hypothesis

Accepted Answer

A probabilistic classification model defined as $h_\	heta(x) = g(\	heta^T x)$, where $g(z)$ is the sigmoid/logistic function:
$$g(z) = \frac{1}{1 + e^{-z}}$$
Outputs represent estimated probabilities $P(y=1 | x; \	heta) \in (0, 1)$.

Question 9

Derivative of the Sigmoid Function

Accepted Answer

The derivative of $g(z) = \frac{1}{1 + e^{-z}}$ expressed in terms of itself:
$$g'(z) = g(z)(1 - g(z))$$
This property makes computing gradients in logistic regression mathematically clean and computationally efficient.

Question 10

Binary Cross-Entropy Loss (Log Loss)

Accepted Answer

The cost function for binary logistic regression derived from maximum likelihood estimation:
$$J(\	heta) = -\frac{1}{m} \sum_{i=1}^m \left[ y^{(i)} \log(h_\	heta(x^{(i)})) + (1 - y^{(i)}) \log(1 - h_\	heta(x^{(i)})) \ight]$$

Question 11

Likelihood Function ($L(\	heta)$)

Accepted Answer

A function expressing the probability of observing the given dataset under parameters $\	heta$. For independent training examples:
$$L(\	heta) = \prod_{i=1}^m p(y^{(i)} | x^{(i)}; \	heta)$$
In Maximum Likelihood Estimation (MLE), parameters are chosen to maximize $L(\	heta)$ or its log $\ell(\	heta) = \log L(\	heta)$.

Question 12

Newton's Method (Newton-Raphson)

Accepted Answer

A second-order optimization method for finding roots of functions. For parameter update:
$$\	heta := \	heta - H^{-1} \
abla_\	heta J(\	heta)$$
Where $H$ is the Hessian matrix ($H_{ij} = \frac{\partial^2 J}{\partial \	heta_i \partial \	heta_j}$). Converges quadratically (faster than gradient descent) but requires expensive matrix inversion.

Question 13

Generalized Linear Models (GLMs)

Accepted Answer

A flexible framework unifying linear and logistic regression. A model is a GLM if:
1. $y | x; \	heta \sim \	ext{ExponentialFamily}(\eta)$
2. Output target prediction is $h(x) = \mathbb{E}[y | x]$
3. Natural parameter $\eta = \	heta^T x$

Question 14

Exponential Family Distribution

Accepted Answer

A class of probability distributions expressible in the canonical form:
$$p(y; \eta) = b(y) \exp\left( \eta^T T(y) - a(\eta) \ight)$$
Where $\eta$ is the natural parameter, $T(y)$ is the sufficient statistic, and $a(\eta)$ is the log-partition function. Includes Gaussian, Bernoulli, Poisson, and Gamma distributions.

Question 15

Perceptron Learning Algorithm

Accepted Answer

A threshold-based binary classifier defined by $h_\	heta(x) = g(\	heta^T x)$ where $g(z) = 1$ if $z \ge 0$ else $0$.
Update rule when misclassifying sample $i$:
$$\	heta := \	heta + \alpha (y^{(i)} - h_\	heta(x^{(i)})) x^{(i)}$$

Question 16

Underfitting (High Bias)

Accepted Answer

Occurs when a model is too simple to capture the underlying structure of the data (e.g., fitting a linear model to quadratic data). Results in poor performance on both training and validation sets.

Question 17

Overfitting (High Variance)

Accepted Answer

Occurs when a model learns noise and specific details of the training set rather than general trends (e.g., high-degree polynomial). Results in low training error but high generalization/validation error.

Question 18

Bias-Variance Tradeoff

Accepted Answer

Decomposition of expected generalization error into three additive terms:
$$\	ext{Expected Error} = \	ext{Bias}^2 + \	ext{Variance} + \	ext{Irreducible Error}$$
Increasing model complexity decreases bias but increases variance.

Question 19

L2 Regularization (Ridge)

Accepted Answer

Penalizes large feature weights by adding the sum of squared parameters to the loss function:
$$J_R(\	heta) = J(\	heta) + \frac{\lambda}{2m} \sum_{j=1}^n \	heta_j^2$$
Shrinks weights uniformly toward zero, helping to prevent overfitting without forcing zero coefficients.

Question 20

L1 Regularization (Lasso)

Accepted Answer

Penalizes parameter magnitude using absolute values:
$$J_L(\	heta) = J(\	heta) + \frac{\lambda}{m} \sum_{j=1}^n |\	heta_j|$$
Encourages **sparsity** by driving non-essential feature weights strictly to zero, serving as an automatic feature selection technique.

Question 21

Learning Rate ($\alpha$)

Accepted Answer

A hyperparameter in gradient descent governing step size:
- If $\alpha$ is **too small**, convergence is extremely slow.
- If $\alpha$ is **too large**, optimization may overshoot the minimum, oscillate, or diverge completely.

Stanford CS229 - Machine Learning: Supervised Learning & Loss Functions

Cards in this deck

Related Flashcard Sets

TypeScript - Advanced Type Systems, Generics & Design Patterns

Harvard ECON1011 - Intermediate Microeconomics: Game Theory & Market Structures

MIT 18.03 - Differential Equations: Ordinary Differential Equations, Laplace & Fourier

CPA Exam FAR - Financial Accounting & Reporting: US GAAP & Revenue Recognition