Best practices & guides on how to write distributed pytorch training code

🦾💻🌐 distributed training & serverless inference at scale on RunPod

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

Fast and easy distributed model training examples.

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

Dataloading for JAX

Framework, Model & Kernel Optimizations for Distributed Deep Learning - Data Hack Summit

Fully Sharded Data Parallel (FSDP) implementation of Transformer XL

Comprehensive exploration of LLMs, including cutting-edge techniques and tools such as parameter-efficient fine-tuning (PEFT), quantization, zero redundancy optimizers (ZeRO), fully sharded data parallelism (FSDP), DeepSpeed, and Huggingface accelerate.