All episodes

Episode 06 · Identify AI concepts

Vision, images, and extraction

The last concept episode: vision versus image generation, then the idea of information extraction across modalities — the setup for Content Understanding.

  • Computer vision capabilities
  • Image-generation models
  • Extract from text, images, audio, video
Microsoft Learn
0:00—

Transcript

On the mics

Owen Park

Episode six. We close the concepts floor. Computer vision, image-generation models, and techniques to extract information from text, images, audio, and videos. Three bullets, lots of mix-ups.

Mira Cole

Computer vision reads the world. Caption this photo. Tag 'forklift,' 'pallet,' 'spill.' Detect objects with boxes. O.C.R., optical character recognition, reads text that lives in pixels, a license plate, a whiteboard. Faces, when the scenario is allowed. Spatial understanding, what is next to what.

Owen Park

Image generation writes the world. Text or a reference image in, new pixels out. Marketing mockups, product shots, variations. The model is not finding the forklift. It is painting one that never sat in your warehouse.

Mira Cole

A multimodal chat model can do a bit of vision inside a prompt: here is a photo, what is wrong with this machine. That is interpreting visual input, which the Foundry half of the exam will make you implement. Conceptually it is still vision, not generation.

Owen Park

Information extraction is the other arrow: not 'what is in this picture' as a caption, but 'fill in these fields.' Invoice total. Patient name on a form. The goal time in a sports clip. A schema is the contract. You name the fields, the service fills them.

Mira Cole

The skills list is multimodal on purpose. Extract from text, images, audio, and videos. A P.D.F. is not the only document. A voicemail can yield a callback number. A shelf video can yield product counts. Same idea, different file type.

Owen Park

Azure Content Understanding in Foundry Tools is the named service on the implementation side. Today, just the idea. Later, analyzers, schemas, grounding, confidence. If the exam says 'structured fields from a form,' do not answer 'image generation.'

Mira Cole

Grounding, while we are here: when an extractor points at the bounding box or the timestamp that justified a field, that is how you audit it. Confidence scores tell you when a human should look. Those details smell like reliability and safety from episode two.

Mira Cole

Question one. A designer types 'a red city bike on wet cobblestones at dusk' and wants a campaign still. Which capability?

Owen Park

Image generation. New pixels from a prompt. Vision would be tagging a photo they already took.

Mira Cole

Question two. A parking app reads license plates from a camera. Which capability?

Owen Park

Computer vision, specifically O.C.R. on the plate. You could then extract the plate as a field, but the unique first step is reading text in an image.

Mira Cole

Question three. A technician uploads a photo of a broken pump and asks 'what part is leaking?' in the same prompt. Which model style?

Owen Park

A multimodal model interpreting visual input with a user prompt. Not image generation. Not a text-only model.

Mira Cole

Question four. You want speaker, action item, and due date from a recorded meeting. Workload?

Owen Park

Information extraction from audio. You might transcribe first, but the business output is fields, not a transcript sitting in a folder.

Mira Cole

Question five. Why would an extraction service return a bounding box next to each field?

Owen Park

Grounding. So a human, or an agent, can see where the value came from. That is transparency plus safety, wearing a product feature.

Mira Cole

Concepts floor complete. Episode seven we walk through the Foundry portal: system prompts, deploying a model, and a lightweight chat client with the Foundry S.D.K.

Owen Park

Python will appear. I promise it is a small, friendly amount of Python.