Evaluate ML and AI system performance

• MLflow connects experiment results with the parameters and artifacts that produced them
• Managed MLflow separates service-hosted metadata from customer-owned artifact storage
• A prompt version preserves a tested configuration while the draft continues to change
• MLflow traces connect an AI response to the components that produced it
• Bedrock can evaluate saved responses without invoking the generator again
• Repeating an experiment requires preserving its inputs and execution environment
• Data-quality monitoring compares observations with baseline statistics and constraints
• A legitimate model-data update can require a new monitoring baseline
• Model-quality monitoring requires actual outcomes as well as predictions
• Feature-attribution drift tracks changes in model reliance on input features
• A model performance baseline records predictive quality and acceptable metric boundaries
• Shadow testing copies production requests without returning shadow responses to callers
• Shadow tests evaluate serving changes as well as model changes
• Prediction comparisons require retaining shadow outputs rather than discarding them
• Shadow testing is restricted by endpoint feature compatibility
• Local and global feature attributions answer different explanation questions
• Removing one feature at a time can miss importance shared by correlated features
• Training convergence diagnostics need training metrics and tensors rather than production drift checks
• NaN or infinite training tensors indicate numerical instability
• Poor weight initialization can prevent effective learning from the first steps
• Persistently inactive ReLU units are a distinct training diagnostic
• Precision is important when false positive predictions are costly
• Recall is important when missing positive cases is costly
• F1 summarizes the joint precision and recall tradeoff
• ROC AUC compares binary ranking performance across thresholds
• Balanced accuracy reduces the dominance of a majority class in evaluation
• Log loss evaluates prediction probabilities and penalizes confident mistakes
• RMSE emphasizes large regression errors more than MAE
• Macro F1 gives each class equal influence in the final score
• K-fold validation assesses performance across multiple held-out partitions
• BLEU evaluates reference overlap through candidate n-gram precision
• ROUGE measures lexical overlap useful for summary evaluation
• ROUGE-L compares the longest shared word subsequence
• BERTScore compares contextual representations rather than only exact words
• Exact-match evaluation is appropriate only when exact answer equality is meaningful
• An LLM judge evaluates generated responses against selected criteria
• Human evaluation can assess subjective output quality with explicit instructions
• The configured worker count determines how many ratings each prompt receives
• Human evaluation can compare Bedrock outputs with supplied outputs from another model
• An evaluation task label does not replace instructions to the model
• A correct answer can still be incomplete or fail the requested format
• Safety evaluation separates harmful content, stereotyping, and refusal behavior
• Semantic robustness tests stability under meaning-preserving input changes
• An evaluation score must be interpreted alongside missing model responses
• Human-in-the-loop evaluation connects flagged outputs to review and corrective feedback
• Bias detection requires comparing model behavior across relevant groups
• Adversarial robustness evaluation tests deliberate attempts to mislead a model
• Retrieve-only evaluation isolates retrieval quality from answer generation
• Relevant retrieved passages may still omit information needed for the answer
• Faithfulness checks whether generated claims are supported by retrieved evidence
• Correct citations and complete citation support are separate quality requirements
• RAG diagnosis needs the query, retrieved context, and generated answer together
• A grounding judge must evaluate against the supplied evidence rather than the generator's prior conversation

Unlock with Premium — includes all practice exams and the complete study guide.