Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

• Perceptron

• Logistic Regression

• Softmax Regression (Multinomial Logistic Regression)

• Softmax Regression with MLxtend's plot_decision_regions on Iris

Multilayer Perceptrons

• Multilayer Perceptron

• Multilayer Perceptron with Dropout

• Multilayer Perceptron with Batch Normalization

• Multilayer Perceptron with Backpropagation from Scratch

Convolutional Neural Networks

Basic

• Convolutional Neural Network

• Convolutional Neural Network with He Initialization

Concepts

• Replacing Fully-Connnected by Equivalent Convolutional Layers

---

AlexNet

• AlexNet Trained on CIFAR-10

• AlexNet with Grouped Convolutions Trained on CIFAR-10

DenseNet

• DenseNet-121 Digit Classifier Trained on MNIST

• DenseNet-121 Image Classifier Trained on CIFAR-10

Fully Convolutional

• "All Convolutionl Net" -- A Fully Convolutional Neural Network

LeNet

• LeNet-5 on MNIST

• LeNet-5 on CIFAR-10

• LeNet-5 on QuickDraw

MobileNet

• MobileNet-v2 on Cifar-10

• MobileNet-v3 small on Cifar-10

• MobileNet-v3 large on Cifar-10

Network in Network

• Network in Network Trained on CIFAR-10

VGG

• Convolutional Neural Network VGG-16 Trained on CIFAR-10

• VGG-16 Smile Classifier Trained on CelebA

• VGG-16 Dogs vs Cats Classifier

• Convolutional Neural Network VGG-19

ResNet

• ResNet and Residual Blocks

• ResNet-18 Digit Classifier Trained on MNIST

• ResNet-18 Gender Classifier Trained on CelebA

• ResNet-34 Digit Classifier Trained on MNIST

• ResNet-34 Object Classifier Trained on QuickDraw

• ResNet-34 Gender Classifier Trained on CelebA

• ResNet-50 Digit Classifier Trained on MNIST

• ResNet-50 Gender Classifier Trained on CelebA

• ResNet-101 Gender Classifier Trained on CelebA

• ResNet-101 Trained on CIFAR-10

• ResNet-152 Gender Classifier Trained on CelebA

---

Normalization Layers

• BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier

• Filter Response Normalization for Network-in-Network CIFAR-10 Classifier

Metric Learning

• Siamese Network with Multilayer Perceptrons

Autoencoders

Fully-connected Autoencoders

• Autoencoder (MNIST)

• Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier

Convolutional Autoencoders

• Convolutional Autoencoder with Deconvolutions / Transposed Convolutions

• Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance

• Convolutional Autoencoder with Deconvolutions (without pooling operations)

• Convolutional Autoencoder with Nearest-neighbor Interpolation

• Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA

• Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw

Variational Autoencoders

• Variational Autoencoder

• Convolutional Variational Autoencoder

Conditional Variational Autoencoders

• Conditional Variational Autoencoder (with labels in reconstruction loss)

• Conditional Variational Autoencoder (without labels in reconstruction loss)

• Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss)

• Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss)

Generative Adversarial Networks (GANs)

• Fully Connected GAN on MNIST

• Fully Connected Wasserstein GAN on MNIST

• Convolutional GAN on MNIST

• Convolutional GAN on MNIST with Label Smoothing

• Convolutional Wasserstein GAN on MNIST

• "Deep Convolutional GAN" (DCGAN) on Cats and Dogs Images

• "Deep Convolutional GAN" (DCGAN) on CelebA Face Images

Graph Neural Networks (GNNs)

• Most Basic Graph Neural Network with Gaussian Filter on MNIST

• Basic Graph Neural Network with Edge Prediction on MNIST

• Basic Graph Neural Network with Spectral Graph Convolution on MNIST

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

• A simple single-layer RNN (IMDB)

• A simple single-layer RNN with packed sequences to ignore padding characters (IMDB)

• RNN with LSTM cells (IMDB)

• RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors

• RNN with LSTM cells and Own Dataset in CSV Format (IMDB)

• RNN with GRU cells (IMDB)

• Multilayer bi-directional RNN (IMDB)

• Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News)

Many-to-Many / Sequence-to-Sequence

• A simple character RNN to generate new text (Charles Dickens)

Ordinal Regression

• Ordinal Regression CNN -- CORAL w. ResNet34 on AFAD-Lite

• Ordinal Regression CNN -- Niu et al. 2016 w. ResNet34 on AFAD-Lite

• Ordinal Regression CNN -- Beckham and Pal 2016 w. ResNet34 on AFAD-Lite

Tips and Tricks

• Cyclical Learning Rate

• Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet)

• Gradient Clipping (w. MLP on MNIST)

Transfer Learning

• Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10)

Visualization and Interpretation

• Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)

• Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)

PyTorch Workflows and Mechanics

PyTorch Lightning Examples

• MLP in Lightning with TensorBoard -- continue training the last model

• MLP in Lightning with TensorBoard -- checkpointing best model

Custom Datasets

• Custom Data Loader Example for PNG Files

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD)

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images

• Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST

Training and Preprocessing

• Generating Validation Set Splits

• Dataloading with Pinned Memory

• Standardizing Images

• Image Transformation Examples

• Char-RNN with Own Text File

• Sentiment Classification RNN with Own CSV File

Improving Memory Efficiency

• Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10)

Parallel Computing

• Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA

• Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example)

Other

• PyTorch with and without Deterministic Behavior -- Runtime Benchmark

• Sequential API and hooks

• Weight Sharing Within a Layer

• Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib

Autograd

• Getting Gradients of an Intermediate Variable in PyTorch

TensorFlow Workflows and Mechanics

Custom Datasets

• Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives

• Storing an Image Dataset for Minibatch Training using HDF5

• Using Input Pipelines to Read Data from TFRecords Files

• Using Queue Runners to Feed Images Directly from Disk

• Using TensorFlow's Dataset API

Training and Preprocessing

• Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives