Here are some brief descriptions of the network technologies referenced in this module:
Convolutional Neural Networks (CNNs)
CNNs are specially designed for processing grid-like data such as images.
- Convolutional Layer: The cornerstone of CNNs. It uses filters (small, usually three-by-three, weight matrices) that slide over the input data (like an image) to produce a feature map. This operation helps detect local features like edges, textures, or patterns.
- Pooling Layer: Typically used after convolutional layers, pooling layers reduce the spatial dimensions of the feature maps, keeping the essential information. A standard method is max pooling, which retains the maximum value from a section of the feature map.
- Fully Connected Layer: After several convolutional and pooling layers, high-level reasoning about the features occurs in fully connected layers. They connect every neuron in one layer to every neuron in the next layer, much like traditional neural networks.
Recurrent Neural Networks (RNNs)
RNNs are designed to process sequential data by keeping a “memory” of earlier inputs in their internal state.
- Memory: Unlike traditional neural networks, an RNN retains a form of memory. It achieves this by keeping a hidden state that gets updated at each time step of a sequence.
- Sequence Processing: Given a data sequence (like a sentence broken down into individual words or a time series), an RNN processes one element at a time while updating its hidden state. This allows it to carry forward information from earlier parts in the sequence.
- Vanishing Gradient Problem: A significant challenge with basic RNNs is the vanishing (or exploding) gradient problem. This means that during training, the gradients—which are used to update the network’s weights—can become extremely small or large, making it hard for the RNN to learn long-term dependencies.
Long Short-Term Memory (LSTM)
LSTMs are a special kind of RNN designed to address the vanishing gradient problem and better capture long-term dependencies in data.
- Cell State: The core idea behind LSTMs is the cell state, a kind of “conveyor belt” that runs through the LSTM unit. Information can be added to or removed from this cell state via gates.
- Gates: LSTMs use three gates:
- Forget Gate: Decides what information from the cell state should be thrown away or kept.
- Input Gate: Updates the cell state with new information.
- Output Gate: Decides what information based on the cell state should be output from the LSTM unit.
These gates enable LSTMs to selectively remember or forget information, making them better suited for tasks that require understanding over longer sequences.
Transformers
Transformers revolutionized the NLP field and are now finding applications in other domains.
- Attention Mechanism: At the heart of the transformer is the attention mechanism. It allows the model to focus on different parts of the input data with varying (numerical) intensity. The “self-attention” mechanism lets each word in an input sequence look at every other word to determine its context.
- Positional Encoding: Since transformers don’t have a built-in sense of sequence (like RNNs do), positional encoding is added to give the model information about the position of words in a sequence.
- Stacked Layers: A transformer holds multiple stacked layers of these attention mechanisms, allowing for complex patterns and relationships in the data to be captured.
Generative Adversarial Networks (GANs)
GANs are designed for generative tasks, like creating images or other forms of data.
- Two-Part Structure: A GAN consists of two neural networks—a generator and a discriminator—trained simultaneously through adversarial training.
- Generator: This network takes random noise as input and generates data (like an image).
- Discriminator: This network tries to distinguish between genuine data samples and fake samples produced by the generator.
- Adversarial Training: During training, the generator tries to produce data that the discriminator can’t distinguish from real data. In contrast, the discriminator tries to distinguish real data from fake better. It’s like a cat-and-mouse game where both parties evolve together.
The goal is to have a generator so good at its task that the discriminator can’t tell the difference between the real and generated data.
Conditional Generative Adversarial Networks (cGANs)
cGANs are an extension of traditional GANs, tailored for controlled data generation based on specific conditions.
- Conditioned Generation: Unlike standard GANs, cGANs incorporate a conditional input to guide the data generation process. This condition could be a label, an image, or any relevant information.
- Modified Structure:
- Generator: Receives random noise and conditional input, aiming to produce data that aligns with the provided condition.
- Discriminator: Evaluates the authenticity of data samples, considering both the sample and its associated condition.
Stable Diffusion
While GANs utilize a generator-discriminator setup for data generation, Stable Diffusion models take a different approach based on diffusing data.
- Diffusion Process: The idea is to view data generation as a reverse diffusion process. Given a data point (like an image), it’s gradually corrupted with noise over a series of steps until it’s transformed into pure noise.
- Reverse Process: The generative task involves reversing this process. Starting with noise, the model aims to remove the noise step-by-step, reconstructing the original data point.
- Stability: One of the advantages of Stable Diffusion over GANs is its stable training process. Whereas GANs can suffer from issues like mode collapse (where the generator produces limited varieties of samples), Stable Diffusion models provide a more stable and consistent way to generate diverse, high-quality data samples.
Again, these lists are subject to the rapid advancement of AI techniques and architectures. If you’re interested in the latest model designs or the kinds of tasks being performed with AI, a model zoo would be the place to look. Model zoos are repositories of pre-trained machine learning models—these are like cookbooks. Find and use a good recipe or adapt it to your needs! Instead of building and training a model from scratch, which can be time-consuming and resource-intensive, researchers and developers (like you!) can visit a model zoo to find a model that suits their needs. These pre-trained models can then be used directly or fine-tuned for specific tasks, offering a head start in various applications. Model zoos act as a bridge, facilitating the sharing and dissemination of knowledge within the machine-learning community. One such zoo is Hugging Face, where hundreds of thousands of pre-trained models are organized by tasks: Huggingface Tasks. We encourage you to check it out!
