This repository is supposed to be a simplified repository for pretraining a large language model. It tries to leverage Deepspeed and Megatron-LM through Huggingface's Accelerate library for simplicity. It's still a work-in-progress. It currently implements a simplified version of dataset packing for improved efficiency during training.

I provided a simple accelerate config in configs/, edit it to your needs.

Afterwards, run the bash script in the examples/ directory to train a 1b parameter llama model after providing the directory in which jsonl files reside.

Deepspeed currently runs great! Check out the deepspeed configuration file in the configs/ directory for an example.

There's been some issues fully integrating Megatron-LM. Accelerate relies on Huggingface's Megatron-LM repository. The repository has long been abandoned, I am currently in the process of developing an updated fork that is compatible with Accelerate. In theory though, the training code here is designed to work with Megatron-LM.

Should work, but it's a bit finicky. Support not mature yet.