Velamuvi AIAcademy
Deep Learning

Transformers

The architecture behind ChatGPT, Claude and Gemini. Build attention and a small Transformer step by step.

  • Advanced
  • About 14 hours
  • 4 modules
  • 13 lessons
  • Free

This course is not released yet. We publish courses one at a time, each one complete. The full syllabus is below.

What you will learn

  • AttentionThe Problem With Recurrence, Attention Intuition, Self-Attention Step by Step and more
  • The ArchitecturePositional Encoding, The Encoder Block, The Decoder Block and more
  • VariantsEncoder-Only: BERT, Decoder-Only: GPT, Encoder-Decoder: T5
  • Build ItProject: A Tiny Transformer in PyTorch

Course syllabus

Not released yet · full syllabus below

  1. 1

    Attention

    5 lessons
    1. 1The Problem With RecurrenceSoon
    2. 2Attention IntuitionSoon
    3. 3Self-Attention Step by StepSoon
    4. 4Queries, Keys and ValuesSoon
    5. 5Multi-Head AttentionSoon
  2. 2

    The Architecture

    4 lessons
    1. 6Positional EncodingSoon
    2. 7The Encoder BlockSoon
    3. 8The Decoder BlockSoon
    4. 9Layer Norm and ResidualsSoon
  3. 3

    Variants

    3 lessons
    1. 10Encoder-Only: BERTSoon
    2. 11Decoder-Only: GPTSoon
    3. 12Encoder-Decoder: T5Soon
  4. 4

    Build It

    1 lessons
    1. 13Project: A Tiny Transformer in PyTorchSoon

Recommended next