Problem
I wanted to understand how to build and train a language model, from converting text into tokens to updating model weights across GPUs.
Implementation
I implemented a GPT language model without pretrained weights, including the tokenizer, transformer architecture, and training loop. I used distributed PyTorch and CUDA to scale training across multiple GPUs.
Technical decisions
- Built the tokenizer and transformer architecture as part of the project.
- Implemented the training loop instead of relying on a pretrained model.
- Used distributed PyTorch and CUDA for distributed GPU training.
- Worked across sequence modeling, GPU compute, and training infrastructure.
Results
I implemented the tokenizer, transformer, training loop, and distributed training setup using PyTorch and CUDA.