Domain 4 of 4 · Chapter 5 of 5

Evaluating AI Recommendation Accuracy

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing Designing, Deploying and Managing Network Automation Systems premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Accuracy is a verdict against ground truth
  • Validate before you deploy: CML and pyATS
  • Detecting hallucinations in AI output
  • Why 'it ran' and 'it looked right' fail
  • Accuracy is continuous, not a one-time gate
  • Reading the stem: exam patterns and traps

What counts as adequate accuracy evaluation

Validation approachRuns before deploy?Checked against ground truth?Adequate on its own?
Deploy, then roll back on alarmsNo, it runs in productionNo, production is the testNo
Re-run on a second model, accept if they agreeYesNo, compared to another modelNo
Trust the model's confidence scoreYesNo, self-reported and uncalibratedNo
CML simulation + pyATS Diff + human reviewYesYes: device state, docs, YANG, source of truthYes

Decision tree

Cited APIs, commands, YANGpaths and citations exist?Passes CML simulationand config validate?pyATS Diff shows theintended state only?Human reviewapproved?Deploythen keep reviewingReject: send backhallucinationReject: send backsemantically wrongReject: send backunintended changeReject: send backnot signed offYesYesYesYesNoNoNoNo
Cheat sheet

Cheat sheet

  • Test AI-generated config in CML before production
  • Verify intended state with pyATS learn and Diff
  • Deploy-then-rollback is not accuracy evaluation
  • Syntactically valid code can be semantically wrong
  • Human-in-the-loop review is the primary accuracy control
  • Accuracy is measured against ground truth
  • Two models agreeing is not proof of correctness
  • Cited sources in an AI answer must be verified
  • Overreliance falls under OWASP LLM09:2025 Misinformation
  • LLM confidence is not a calibrated correctness signal
  • Detect hallucinations by checking existence and validating config
  • AI-output accuracy needs ongoing, not one-time, review

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. RFC 8040: RESTCONF Protocol Whitepaper
  2. RFC 7950: The YANG 1.1 Data Modeling Language Whitepaper
  3. OWASP LLM09:2025 Misinformation Whitepaper
  4. Cisco Modeling Labs
  5. Overview of CML 2.x - Cisco Modeling Labs
  6. pyATS on DevNet
  7. RFC 6241: Network Configuration Protocol (NETCONF) Whitepaper