The Core Foundation & ML Physics
Data Pipelines, Pre-Training, Fine-Tuning & Model Alignment
Master the data engineering pipeline, transformer pre-training physics, LoRA/QLoRA adapter fine-tuning math, and modern alignment techniques (SFT, RLHF, DPO).
1. Production Data Engineering & MinHash Deduplication
Scrape, Clean, and Deduplicate Web-Scale Pre-Training Corpora
High-quality data beats raw compute. Training models on duplicate web pages (like boilerplate headers, cookie notices, or spam) wastes GPU clusters and degrades downstream reasoning accuracy.
Under the Hood Mechanics
Before feeding billions of tokens into transformer models, data passes through multi-stage scrubbing pipelines: 1) Quality filtering using FastText language classification and perplexity scores. 2) Near-duplicate document detection using MinHash Locality-Sensitive Hashing (LSH). MinHash generates a fixed-size signature for documents by hashing n-grams (shingles) and picking the minimum hash value per permutation function. Banding technique divides signatures into bands to identify candidate matching pairs in O(1) time.
At 100+ Terabyte scale, storing raw shingles is prohibitive. Distributed MinHash LSH requires Ray or Spark clusters.
2. Pre-Training Physics & LoRA / QLoRA Math
Compute-Optimal Scaling Laws & Rank Decomposition Adapters
Pre-training multi-billion parameter foundation models costs millions of dollars. Fine-tuning full model weights for downstream enterprise applications is computationally impractical. Parameter-Efficient Fine-Tuning (PEFT) like LoRA reduces VRAM footprint by 90%.
Under the Hood Mechanics
According to Chinchilla scaling laws, compute optimal training requires scaling tokens and parameters equally (~20 tokens per parameter). During fine-tuning, Low-Rank Adaptation (LoRA) freezes pretrained weight matrix W (dim d x k) and injects trainable rank decomposition matrices A (d x r) and B (r x k) where r << min(d, k). Forward pass becomes: h = Wx + (BA)x * (alpha/r). QLoRA pushes this further by quantizing base weights W to 4-bit NormalFloat (NF4) with Double Quantization and Paged Optimizers.
LoRA adapter rank 'r' must be chosen carefully: r=8 or r=16 works well for style modification, while r=64+ is needed for introducing new complex domain knowledge.
3. Alignment: SFT, RLHF & Direct Preference Optimization (DPO)
Steering Raw Next-Token Generators to Helpful & Safe Assistants
Base models generate next-tokens; they do not answer questions or follow instructions reliably. Alignment shapes model behavior to match human intent while curbing toxicity and hallucinations.
Under the Hood Mechanics
1) Supervised Fine-Tuning (SFT) trains the model on curated (Prompt, Completion) pairs. 2) Traditional RLHF uses SFT outputs to train a separate Reward Model, then optimizes the policy model using Proximal Policy Optimization (PPO) with a KL-divergence penalty. 3) Direct Preference Optimization (DPO) eliminates the complex reward model and PPO instability by mathematically expressing the implicit reward function directly through the LLM policy's log probabilities over preferred (y_w) vs dispreferred (y_l) responses.
DPO can cause likelihood displacement if hyperparameter beta is too high. DPO requires clean pairwise preference datasets (Prompt, Chosen, Rejected).
Production Knowledge Assessment
Why is MinHash LSH used for pre-training dataset deduplication?