Domain 3 of 5 · Chapter 2 of 3

Multimodal understanding and visual analysis workflows

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AI-103 premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Three engines read the same pixels
  • Passing an image to a vision-enabled chat model
  • What each Image Analysis feature returns
  • Content Understanding analyzers for visual characteristics
  • Video analysis: content extraction, then field extraction
  • Question answering grounded in visual evidence
  • Exam-pattern recognition

Choosing between the three engines that read pixels

RequirementVision-enabled chat modelImage AnalysisContent Understanding analyzer
Output shapeFree-form prose in the chat replyA fixed result object per requested visual featureJSON matching your fieldSchema, plus Markdown
Pixel coordinatesNone; the model returns no bounding boxesBounding boxes for Objects and denseCaptionsNot a localization API; returnDetails adds metadata
Video inputStill images onlyStill images onlyYes, video and audio in standard mode
Scale per requestUp to ten images per chat call, 20 MB eachOne image per Analyze request, under 20 MBOne analyzer applied across a whole corpus
How you shape the outputThe prompt, and nothing elsePick from the built-in feature setDeclare fields with generate, classify, or extract
Typical jobReasoning and explanation about a sceneCaptions, alt text, object and region locationStructured metadata over an image or video library

Decision tree

Same declared fieldsfor every file?YesNoContent Understandingstandard-mode analyzerreads video and audio tooAnswer needs pixelpositions?NoYesVision-enabled chat modelprose in the replyno bounding boxesEach region needsits own sentence?YesNoImage Analysis 4.0denseCaptionsup to ten regions with boxesImage Analysis 4.0Objectsname, confidence, boxVideo or audio inputonly Content Understanding standard modeLogos and brandsuse Brands, a version 3.2 feature
Cheat sheet

Cheat sheet

  • Images reach a vision-enabled chat model as an image_url content part in the user message
  • The detail setting decides whether the model tiles the image at high resolution
  • One chat call can carry several images, up to a cap of ten per call
  • Input images to vision chat models are capped at 20 MB
  • Only standard mode accepts image, video, and audio input; pro mode is document-only
  • Each field in a fieldSchema declares a method of generate, classify, or extract
  • Image analyzers are not optimized for pictures whose meaning is mostly printed text
  • Face description fields are limited access and require face blurring to be disabled
  • The prebuilt video analyzer emits RAG-ready Markdown with a WEBVTT transcript and key frames
  • enableSegment chooses between one whole-video segment and model-authored custom segments
  • Video analysis samples about one frame per second at 512x512 and transcribes only speech
  • Turning on segmentation invokes the generative model and bills tokens even with no fields defined
  • Caption gives one sentence for the whole image; Dense Captions describes up to ten regions
  • gender-neutral-caption replaces gendered terms with 'person' in generated captions
  • Objects localizes things but never names brands; Brands is the separate logo mode
  • A multimodal chat model describes an image in prose and returns no coordinates
  • Object detection misses small and tightly clustered objects
  • Grounded visual QA indexes analyzer output rather than re-sending media at query time
  • Sentence-level timestamps and camera shot times appear only when returnDetails is true
  • Diarization attributes speech to speakers, and auto language enables per-phrase multilingual transcripts
  • Face identification and grouping in video are preview-only capabilities

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. How to use vision-enabled chat models
  2. What is Image Analysis?
  3. Azure Content Understanding: what is an analyzer? Configuration and reference
  4. Image captions in Image Analysis 4.0
  5. Object detection using Image Analysis 4.0
  6. Brand detection in Azure Vision
  7. Azure Content Understanding standard and pro modes
  8. Azure Content Understanding image overview
  9. Azure Content Understanding video overview
  10. What is Azure Content Understanding?
  11. Multimodal search concepts and guidance in Azure AI Search