LLM Training & Alignment

Cooking up a modern AI assistant like ChatGPT or Gemini is not one step. It starts with chaotic web-scale data, then refines it through several training stages. The arc from raw math to useful assistant has three big milestones: Pre-training, Supervised Fine-Tuning (SFT), and Alignment.

Phase 1: Pre-training, aka the base model era

Every LLM starts with a pre-trained base model. At this stage, the model gets petabytes of raw text from books, articles, code repositories, and web pages. The training objective is simple: predict the next word (token) in a sentence.

For example, given the text:

"The cat sat on the..."

The model calculates probabilities across its vocabulary and predicts "mat" (or "sofa", "bed", etc.). Do that trillions of times on supercomputer clusters, and the model builds an internal map of language, grammar, reasoning patterns, and facts. But a base model is not an assistant; it is a text completer. If you ask it "Write a recipe for chocolate cake," it might continue the pattern with another prompt: "And write a recipe for apple pie," because it is mimicking recipe lists from the internet.

Phase 2: Supervised Fine-Tuning, aka making it follow directions

To turn a text completer into an assistant, engineers run Supervised Fine-Tuning (SFT). In this phase, the base model trains on curated prompt-response examples written by human experts.

A typical training sample looks like:

Prompt: Explain photosynthesis in one sentence.
Response: Photosynthesis is the process by which plants use sunlight, water, and carbon dioxide to create oxygen and energy in the form of sugar.

After tens of thousands of these examples, the model learns "instruct" behavior: recognize the user request, answer directly, and keep the tone conversational.

Phase 3: Alignment with RLHF and DPO

Even after SFT, a model can still produce toxic, biased, wrong, or useless output. SFT teaches imitation; alignment teaches preference. Engineers align models with human preferences using two main techniques:

1. Reinforcement Learning from Human Feedback (RLHF)

RLHF is basically a grading loop with three steps:

• Generate Options: The model generates several possible answers to a prompt.
• Train a Reward Model: Human evaluators rank those answers from best to worst. A separate neural network, the Reward Model, learns to predict what score a human would give.
• Reinforce: Using an RL algorithm, usually PPO, the LLM updates its parameters to maximize the Reward Model score. Human-approved answers get boosted; disliked answers get pushed down.

2. Direct Preference Optimization (DPO)

RLHF works, but it is famously unstable, expensive, and messy because you have to juggle the LLM, Reward Model, and reference models at the same time.

In 2023, researchers introduced Direct Preference Optimization (DPO). DPO skips the reward model entirely. It shows you can optimize the LLM directly from paired choices: a prompt, a preferred (chosen) response, and a disliked (rejected) response. DPO adjusts the weights so the chosen response becomes more likely than the rejected one. Same alignment goal, cleaner loop.

Choosing the Right Adaptation Method

Most product teams do not train foundation models from scratch. They choose among smaller adaptation levers:

• Prompting: Best for behavior that can be described in instructions and examples.
• RAG: Best when the answer depends on changing, private, or auditable knowledge.
• Fine-tuning: Best when the model needs a consistent style, format, domain vocabulary, or task habit that is hard to teach in every prompt.
• Preference tuning: Best when several answers are plausible but the product has a clear preference for one kind of response.
• Guardrails and evals: Necessary when mistakes are expensive, no matter which training method is used.

A useful rule: do not fine-tune just to add facts. Facts change and should usually live in retrieval, tools, or databases. Fine-tune when you want the model to behave differently even when the same facts are already present.