This repository implements the Decoder part of the Attention is all you need in pure PyTorch. The encoding is done using OpenAI's tiktoken library.
You'll need pixi or, when you have all the dependencies installed, you can just use pip.
pixi install # super easy
pip install . --no-depsgpt train --iterations 5000 --text data/tiny-shakespeare.txtthis will create a model.pt and config.txt file.
You can use those to then generate text using
gpt prompt --model model.pt --config config.txtwhich simply generates a fixed amount of tokens given the prompt.