文章目录

• 论文实现
• • Transformers
  • Eleuther GPT-NeoX
  • Diffusion models
  • Generative Adversarial Networks
  • Recurrent Highway Networks
  • LSTM
  • HyperNetworks – HyperLSTM
  • ResNet
  • ConvMixer
  • Capsule Networks
  • U-Net
  • Sketch RNN
  • Graph Neural Networks
  • Reinforcement Learning
  • Counterfactual Regret Minimization (CFR)
  • Optimizers
  • Normalization Layers
  • Distillation
  • Adaptive Computation
  • Uncertainty
  • Activations
  • Language Model Sampling Techniques
  • Scalable Training/Inference
• 查看实例

这个项目还有个网站，地址：https://nn.labml.ai/
这个项目将论文和pytorch代码结合起来，大大方便了大家的学习。

论文实现

Transformers

Multi-headed attention
Transformer building blocks
Transformer XL
Relative multi-headed attention
Rotary Positional Embeddings (RoPE)
Attention with Linear Biases (ALiBi)
RETRO
Compressive Transformer
GPT Architecture
GLU Variants
kNN-LM: Generalization through Memorization
Feedback Transformer
Switch Transformer
Fast Weights Transformer
FNet
Attention Free Transformer
Masked Language Model
MLP-Mixer: An all-MLP Architecture for Vision
Pay Attention to MLPs (gMLP)
Vision Transformer (ViT)
Primer EZ
Hourglass

Eleuther GPT-NeoX

Generate on a 48GB GPU
Finetune on two 48GB GPUs
LLM.int8()

Diffusion models

Denoising Diffusion Probabilistic Models (DDPM)
Denoising Diffusion Implicit Models (DDIM)
Latent Diffusion Models
Stable Diffusion

Generative Adversarial Networks

Original GAN
GAN with deep convolutional network
Cycle GAN
Wasserstein GAN
Wasserstein GAN with Gradient Penalty
StyleGAN 2

Recurrent Highway Networks

LSTM

HyperNetworks – HyperLSTM

ResNet

ConvMixer

Capsule Networks

U-Net

Sketch RNN

Graph Neural Networks

Graph Attention Networks (GAT)
Graph Attention Networks v2 (GATv2)

Reinforcement Learning

Proximal Policy Optimization with Generalized Advantage Estimation
Deep Q Networks with with Dueling Network, Prioritized Replay and Double Q Network.

Counterfactual Regret Minimization (CFR)

Solving games with incomplete information such as poker with CFR.

Kuhn Poker

Optimizers

Adam
AMSGrad
Adam Optimizer with warmup
Noam Optimizer
Rectified Adam Optimizer
AdaBelief Optimizer

Normalization Layers

Batch Normalization
Layer Normalization
Instance Normalization
Group Normalization
Weight Standardization
Batch-Channel Normalization
DeepNorm

Distillation

Adaptive Computation

PonderNet

Uncertainty

Evidential Deep Learning to Quantify Classification Uncertainty

Activations

Fuzzy Tiling Activations

Language Model Sampling Techniques

Greedy Sampling
Temperature Sampling
Top-k Sampling
Nucleus Sampling

Scalable Training/Inference

Zero3 memory optimizations

查看实例

我们一起看一下ResNet的例子，地址：https://nn.labml.ai/resnet/index.html

这是Block。

这是Block里面的内容。

这样的方式理解pytorch代码是不是简单了许多。