# jingyaogong/minimind

MiniMind: Train a 64M LLM from Scratch in 2 Hours

- HackerByte page: <https://hackerbyte.net/en/projects/minimind-train-a-64m-llm-from-scratch-in-2-hours-834369920>
- GitHub: <https://github.com/jingyaogong/minimind>
- Programming language: Python

🧠 Train a 64M\-parameter LLM from scratch in just 2h\!

## AI-generated summary

MiniMind is an open\-source project that allows developers to train a 64 million parameter Large Language Model \(LLM\) from scratch with minimal cost and time. It provides a complete training pipeline, including code for data cleaning, pre\-training, fine\-tuning, and reinforcement learning techniques. The project aims to lower the barrier to entry for understanding and building LLMs.

## Key points

- Train a 64M parameter LLM from scratch in approximately 2 hours for around $3.
- Includes code for the full LLM training lifecycle: MoE, data cleaning, pretraining, SFT, LoRA, RLHF \(DPO\), RLAIF, Tool Use, Agentic RL, and model distillation.
- Core algorithm code is implemented natively in PyTorch, without relying on high\-level abstraction libraries.
- Models are compatible with popular frameworks like Transformers, TRL, PEFT, and inference engines like llama.cpp, vLLM, and Ollama.
- Offers extended models for visual and multimodal capabilities \(MiniMind\-V, MiniMind\-O\), diffusion language models, and linear models.
