Domain 4 of 5 · Chapter 2 of 2

Speech solutions for agentic and analytics workloads

Unlock the complete study guide + 1,040 practice questions across 16 full exams.

Bundled into the existing AI-103 premium course — no separate purchase.

14-day money-back guarantee — no questions asked.

Included in this chapter:

  • Which speech surface a requirement belongs to
  • Picking a speech to text mode
  • Customizing what the recognizer hears
  • Measuring a custom model with word error rate
  • What expiry does to a deployed model
  • Choosing a text to speech voice
  • Controlling synthesis with SSML
  • Delivering synthesized audio: real-time and batch
  • Voice Live: a voice agent over one connection
  • Translating speech into another language
  • Reasoning over a recording without a transcript stage
  • Exam-pattern recognition

Choosing a speech to text mode from the shape of the audio

RequirementReal-time speech to textFast transcriptionBatch transcription
Audio it acceptsA live stream or microphone inputOne completed file, under 5 hours and 500 MBIndividual files or a whole Blob Storage container
How you call itA streaming WebSocket connectionA synchronous HTTP requestAn asynchronous job you submit and poll
When results arriveInterim results while speaking, then a final result per utteranceIn the same response, faster than real timeUp to 30 minutes to start and up to 24 hours to finish at peak
Output formInterim and final results, lexical and display formsDisplay form onlyResult files written to a storage container
Custom speech modelRequires the model deployed to a custom endpointNot supported; phrase lists areNamed directly in the job, no endpoint needed
Typical fitLive captions, agent assist, and voice agentsA finished recording that must return a transcript in one callBack catalogues and scheduled analytics runs

Decision tree

A two-way voice-agentconversation?Voice Liveone WebSocket, audio in and outbarge-in, echo cancellation, end-of-turnvoice agentnot a live agentSynthesizing long-formaudio (over ~10 min)?Batch synthesis APIsubmit, poll, downloadnarration over ~10 minuteslong-form audionot long-formSynthesizing short,interactive speech?Real-time synthesisaudio returned in the responseshort replyaudio in, not outAnalysis about a recording,not a verbatim transcript?Content Understanding analyzeror audio-capable Foundry modelkeeps speaker, timing, deliveryanalyze audioneed a transcriptTranscript needed inanother language?Speech translationtarget-language text or audioup to 2 target languagesanother languagesame languageA live stream ormicrophone to transcribe?Real-time speech to textinterim results, then finallive stream over WebSocketlive streamnot a live streamOne finished file, transcriptin the same call?Fast transcriptiontranscript in the same responseone file < 5 h and 500 MBone finished filearchive in storageBatch transcriptionasync job over stored audioarchive in Blob Storage
Cheat sheet

Cheat sheet

  • Real-time speech to text streams interim results before the speaker stops
  • Batch transcription is an asynchronous job over audio already in storage
  • Fast transcription returns a pre-recorded file's transcript synchronously in the same response
  • Diarization attributes phrases to speakers but only on single-channel audio
  • A phrase list boosts specific terms at request time with no training and no hosted endpoint
  • Standard neural voices ship ready to use; custom voice is limited-access and must be trained
  • SSML is the synthesis control surface, but tag support differs across voice types
  • Audio longer than about ten minutes uses the asynchronous batch synthesis API
  • Viseme events drive facial animation and are supported only for en-US neural voices
  • The Voice Live API collapses recognition, reasoning, and synthesis into one managed WebSocket interface
  • Voice Live adds noise suppression, echo cancellation, interruption detection, and end-of-turn detection
  • Voice Live is event-compatible with the Azure OpenAI Realtime API and its extras are additive
  • Voice Live's generative models are fully managed, with no deployment or capacity planning
  • Voice Live supports function calling plus custom speech input and custom voice output
  • Related text improves vocabulary; audio with human-labeled transcripts adapts to acoustic conditions
  • A custom model needs a deployed endpoint for real-time use, but batch transcription can name it directly
  • An expired custom model silently falls back to the base model on the real-time endpoint but 4xx-errors in batch
  • A model deployed to a custom endpoint is pinned until you explicitly redeploy
  • Proving a custom speech model helps means running an accuracy test against human-labeled transcripts and comparing word error rate with the base model
  • Speech translation renders the speaker's own words into another language, as text or synthesized audio
  • One speech translation call covers two target languages; more requires a multi-service Foundry resource
  • Multi-lingual speech translation needs no declared input language but returns no source-language transcript
  • Content Understanding audio analyzers return a speaker-separated WEBVTT transcript plus generative fields
  • Audio-native analysis preserves speaker structure that a transcribe-then-prompt pipeline discards

Unlock with Premium — includes all practice exams and the complete study guide.

References

  1. Speech to text overview (Azure AI Speech)
  2. Content Understanding audio analyzers overview
  3. Text to speech overview (Azure AI Speech)
  4. Voice Live API overview
  5. Speech translation overview
  6. Use the fast transcription API
  7. Batch transcription overview
  8. Create a batch transcription
  9. Improve recognition accuracy with phrase list
  10. Training and testing datasets for custom speech
  11. Test accuracy of a custom speech model
  12. Custom speech model and endpoint lifecycle
  13. Deploy a custom speech model
  14. High definition (HD) voices
  15. Get facial position with viseme
  16. SSML document structure and events
  17. Batch synthesis API for text to speech
  18. Customize Voice Live input and output