Machine Learning - Deep Learning Architectures (CNNs, RNNs, Transformers) Flashcards

Question 1

What is a Convolutional Neural Network (CNN)?

Accepted Answer

A class of deep neural networks primarily used for analyzing visual imagery. They are characterized by convolutional layers, pooling layers, and fully connected layers, leveraging spatial hierarchies in data.

Question 2

Explain the function of a 'convolutional layer' in a CNN.

Accepted Answer

A convolutional layer applies a set of learnable filters (kernels) to the input data. Each filter slides across the input, performing a dot product and producing a feature map that highlights specific features like edges or textures. This process involves operations like stride and padding.

Question 3

What is a 'pooling layer' and its purpose in a CNN?

Accepted Answer

A pooling layer reduces the spatial dimensions (width and height) of the input volume. Its main purposes are to reduce the number of parameters and computation in the network, control overfitting, and make the network more robust to small shifts and distortions. Common types include `max pooling` and `average pooling`.

Question 4

Why are 'activation functions' crucial in CNNs, and name a common one.

Accepted Answer

Activation functions introduce non-linearity into the network, allowing it to learn complex patterns and relationships that linear models cannot. Without them, a deep network would simply be a series of linear transformations. A common activation function is ReLU (Rectified Linear Unit), defined as $f(x) = \max(0, x)$.

Question 5

Describe the key idea behind the VGG architecture.

Accepted Answer

VGG (Visual Geometry Group) is a CNN architecture known for its simplicity and depth. It primarily uses very small $3 	imes 3$ convolutional filters stacked in multiple layers, demonstrating that increasing depth with small filters can significantly improve performance in image recognition tasks.

Question 6

What problem does 'ResNet' (Residual Network) address, and how?

Accepted Answer

ResNet addresses the vanishing gradient problem and degradation (accuracy saturation then degradation) in very deep neural networks. It introduces 'residual connections' (or skip connections) that allow gradients to flow directly through the network, enabling the training of much deeper models by learning residual functions instead of unreferenced functions.

Question 7

List common use cases for Convolutional Neural Networks (CNNs).

Accepted Answer

CNNs are widely used for image classification, object detection, facial recognition, image segmentation, medical image analysis, and video analysis.

Question 8

What is a Recurrent Neural Network (RNN)?

Accepted Answer

An RNN is a class of neural networks designed to process sequential data, where the output from the previous step is fed as input to the current step. This 'memory' allows them to handle tasks involving sequences like text or time series.

Question 9

Explain the 'vanishing gradient problem' in RNNs.

Accepted Answer

The vanishing gradient problem occurs in RNNs when the gradients used to update network weights become extremely small as they propagate backward through many time steps. This makes it difficult for the network to learn long-term dependencies, as the influence of earlier inputs on the current state diminishes rapidly.

Question 10

How do 'LSTMs' (Long Short-Term Memory networks) address the vanishing gradient problem?

Accepted Answer

LSTMs address the vanishing gradient problem through a sophisticated 'cell state' and 'gates' (input, forget, output gates). These gates regulate the flow of information into and out of the cell state, allowing the network to selectively remember or forget information over long sequences, thus preserving long-term dependencies.

Question 11

What is a 'GRU' (Gated Recurrent Unit), and how does it compare to an LSTM?

Accepted Answer

A GRU is a simplified version of an LSTM. It combines the cell state and hidden state, and uses fewer gates (update gate and reset gate) than an LSTM. While often performing comparably to LSTMs, GRUs have fewer parameters, making them computationally less expensive and sometimes faster to train.

Question 12

List common use cases for Recurrent Neural Networks (RNNs), LSTMs, and GRUs.

Accepted Answer

RNNs (including LSTMs and GRUs) are commonly used in Natural Language Processing (NLP) for tasks like machine translation, speech recognition, text generation, sentiment analysis, and time series prediction.

Question 13

What is the fundamental concept behind the 'Transformer' architecture?

Accepted Answer

The Transformer architecture, introduced in 'Attention Is All You Need', relies entirely on the 'attention mechanism' to draw global dependencies between input and output. Unlike RNNs, it processes sequences in parallel, eliminating recurrence and enabling faster training and better long-range dependency modeling.

Question 14

Explain the 'attention mechanism' in Transformers.

Accepted Answer

The attention mechanism allows the model to weigh the importance of different parts of the input sequence when processing each element. It computes 'attention scores' by comparing a 'query' vector (current element) with 'key' vectors (all other elements), then uses these scores to create a weighted sum of 'value' vectors, representing the context.

Question 15

What is 'Multi-Head Attention' and why is it used?

Accepted Answer

Multi-Head Attention runs the attention mechanism multiple times in parallel, each with different learned linear projections of the queries, keys, and values. The outputs from these 'attention heads' are then concatenated and linearly transformed. This allows the model to jointly attend to information from different representation subspaces at different positions, enriching its understanding.

Question 16

Why is 'Positional Encoding' necessary in Transformers?

Accepted Answer

Transformers process input tokens in parallel without any inherent recurrence or convolution, meaning they lose information about the order of words in a sequence. Positional encoding adds information about the relative or absolute position of tokens in the sequence to their embeddings, allowing the model to understand word order.

Question 17

Describe the 'Encoder-Decoder structure' of a Transformer.

Accepted Answer

A Transformer consists of an encoder stack and a decoder stack. The encoder processes the input sequence, generating a sequence of contextualized representations. The decoder then uses these encoder outputs (via cross-attention) along with its own previous outputs to generate the output sequence, typically one token at a time.

Question 18

What is 'Self-Attention' and where is it used in a Transformer?

Accepted Answer

Self-attention is a specific type of attention mechanism where the query, key, and value vectors all come from the *same* sequence. It allows the model to integrate information from all other positions in the input sequence to compute a representation for each position. It's used in both the encoder and decoder blocks of a Transformer.

Question 19

List common use cases for Transformer models.

Accepted Answer

Transformers have revolutionized Natural Language Processing (NLP) and are widely used for machine translation, text generation (e.g., LLMs like GPT), text summarization, question answering, and even increasingly in computer vision (e.g., Vision Transformers).

Question 20

What is a key advantage of Transformers over traditional RNNs (LSTMs/GRUs)?

Accepted Answer

A key advantage is their ability to process sequences in parallel, significantly speeding up training compared to the sequential processing of RNNs. They also excel at capturing long-range dependencies more effectively due to the attention mechanism, without suffering from vanishing gradients caused by recurrence.

Question 21

How do CNNs handle spatial hierarchies?

Accepted Answer

CNNs handle spatial hierarchies by progressively extracting features at different levels of abstraction. Early layers detect simple features like edges and corners, while deeper layers combine these to recognize more complex patterns like textures, parts of objects, and eventually entire objects, building a hierarchical representation of the input.

Question 22

What is the primary difference in how CNNs and RNNs process data?

Accepted Answer

CNNs are primarily designed for spatial data (like images) and process data by applying filters across local regions, leveraging translation invariance. RNNs are designed for sequential data (like text or time series) and process data step-by-step, maintaining a hidden state that carries information from previous steps.

Machine Learning - Deep Learning Architectures (CNNs, RNNs, Transformers)

Card Preview

Related Flashcard Sets

Pharmacology - Renal & Endocrine Systems

Organic Chemistry - Spectroscopy (NMR, IR, Mass Spec)

Monash FIT2093 - Introduction to Programming (Java Fundamentals)

EE105 Midterm Formulas