Domain 5 of 5 · Chapter 2 of 2

Advanced Fine-Tuning and Model Customization

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Operationalizing Machine Learning and Generative AI Solutions premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Fine-tune for behavior; use RAG for changing facts
  • SFT, DPO, and RFT: the three tuning methods
  • Training data: JSONL schemas and file rules
  • Synthetic data: the Simulator and distillation
  • Hyperparameters, metrics, and spotting overfitting
  • Deploying a fine-tuned model: tiers and cost
  • Exam pattern recognition

SFT vs DPO vs RFT: the three fine-tuning methods

DimensionSFT (supervised)DPO (preference)RFT (reinforcement)
Learns fromLabeled input/output pairsPreferred vs non-preferred answer pairsModel-based grader reward signals
Training file schemamessages array (chat format)input / preferred_output / non_preferred_outputmessages array plus grader fields
Best forA task with one correct outputTone, style, subjective qualityOpen-ended reasoning, no single answer
Reward model or graderNoneNone (binary preferences, no reward model)Grader required; it defines the reward
Typical modelsBroad GPT-4.1 and GPT-4o familygpt-4o, gpt-4.1, gpt-4.1-miniReasoning models (o4-mini, gpt-5)

Decision tree

Need fresh or changingfacts?RAG / groundingnot fine-tuningYesNoSingle correct outputper example?SFTmessages JSONLYesNoOptimize reasoningwith a grader?RFTgraders, o4-miniYesNoDPOpreference pairsAlways: prove gains on held-out dataDeveloper tier for eval, then Standard / Provisioned
Cheat sheet

Cheat sheet

  • Foundry offers supervised (SFT), preference (DPO), and reinforcement (RFT) fine-tuning
  • Fine-tune for behavior and format; use RAG for changing factual knowledge
  • Reinforcement fine-tuning uses graders and targets reasoning models
  • SFT training data is JSONL in the Chat Completions conversational format
  • Fine-tune files must be UTF-8 with BOM, under 512 MB, with at least 10 examples
  • DPO training files use input, preferred_output, and non_preferred_output
  • The Simulator class generates synthetic interaction data without production traffic
  • Distillation uses a large teacher model to generate data for a smaller model
  • Key fine-tuning hyperparameters are epochs, batch size, and learning-rate multiplier
  • Track training loss and token accuracy; diverging validation loss signals overfitting
  • Use a held-out validation set and post-training evaluation to confirm gains
  • A Standard fine-tuned deployment adds a per-hour hosting fee on top of token cost
  • Developer tier has no hourly hosting fee but no SLA and auto-deletes in 24 hours
  • Manage a fine-tuned model from candidate evaluation to production promotion

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Microsoft Foundry fine-tuning considerations - Microsoft Foundry
  2. RAG and Generative AI - Azure AI Search
  3. Customize a model with fine-tuning - Microsoft Foundry
  4. Direct preference optimization - Microsoft Foundry
  5. Reinforcement fine-tuning - Microsoft Foundry
  6. Generate Synthetic and Simulated Data for Evaluation (classic) - Microsoft Foundry (classic) portal
  7. Understanding deployment types in Microsoft Foundry Models - Microsoft Foundry
  8. Deploy a fine-tuned model - Microsoft Foundry
  9. Azure OpenAI Service - Pricing