CodeUtility
Making your developer's life easier!
Back to challenges

AI learning path

Build the pieces of a Transformer in Python

9 challenges on how large language models work inside: tokenizers and byte pair encoding, positional encoding, self-attention, causal and multi-head attention, layer normalization and sampling the next token. Solve them in order in Python; each one runs in your browser with instant tests and an AI review.

Start the first challenge
  1. 1

    Tokenization

    Turn text into tokens: vocabularies, one-hot vectors and BPE.

    0/3
    Learn the concept first Tokenization: Vocabulary, One-Hot and Byte Pair Encoding in Python Read the lesson →
    1. 1 Word Tokenizer and Vocabulary Easy Solve in Python
    2. 2 One-Hot Encoding Easy Solve in Python
    3. 3 Byte Pair Encoding (BPE) Merges Medium Solve in Python
  2. 2

    Attention

    Positional encoding and self-attention, causal and multi-head.

    0/4
    Learn the concept first Attention and Self-Attention in Python Read the lesson →
    1. 1 Sinusoidal Positional Encoding Medium Solve in Python
    2. 2 Scaled Dot-Product Attention Hard Solve in Python
    3. 3 Causal (Masked) Self-Attention Medium Solve in Python
    4. 4 Multi-Head Self-Attention Hard Solve in Python
  3. 3

    Normalization and generation

    Layer normalization and choosing the next token.

    0/2
    Learn the concept first How Language Models Generate Text: Temperature and Top-k in Python Read the lesson →
    1. 1 Layer Normalization Medium Solve in Python
    2. 2 Temperature and Top-K Sampling Medium Solve in Python