Azure AI Apps and Agents Core Notes

Choose the Foundry capability

Start from the task, not from a product you remember from AI-102. A large language model handles open-ended language. A small language model fits a narrow, cheaper step such as intent routing. A code model is for source code. A multimodal model is required when one request contains an image, audio, or video plus text. Foundry Tools covers prebuilt language, speech, vision, and understanding capabilities when you are not hosting the model yourself.

For an indexed RAG architecture, grounding needs a retrieval method and an index. Other grounding sources can include authorized API results or directly supplied documents. An agent that must remember a fact, call an API, or read a knowledge store needs memory, tool, and knowledge integrations. A bare prompt does none of those things. Say the missing capability out loud before you name a portal blade.

Model selection is an evaluated tradeoff

Start with a small set of representative requests and explicit acceptance criteria. Measure answer quality, unsupported claims, tool-call reliability, latency, and cost per completed task. A small model is a candidate for a constrained classifier, not an automatic winner for every short prompt. A large model can still choose an unauthorized tool or miss a document. Confirm supported input modalities, context limits, output features, and deployment availability before selecting an SDK example.

Separate knowledge from behavior. RAG supplies retrieved evidence at request time; it does not retrain the model. Fine-tuning can adapt behavior or style but does not automatically keep facts current or enforce per-user document access. Conversation memory stores selected state across turns, while a search index serves a corpus. Decide which information must persist, for whom, and for how long.

Set up the solution

Infrastructure questions ask where the project lives and how it is reached: region, network boundary, and identity. Deployment choices also include model capabilities, regional availability, capacity, data-processing requirements, and the API features the application needs. A repeatable release connects the Foundry project to CI/CD so an agent definition is promoted with the application, not only by clicking in the portal.

The application then connects to that project and calls the deployment with the Foundry SDK. A screenshot of a chat, or a key pasted into a mobile binary, is not a project connection.

Keep endpoints, deployment names, and credentials distinct

A Foundry project endpoint identifies the project; a model deployment name identifies the deployed model used for inference. They are not interchangeable with a model catalog name or a portal URL. Match the SDK generation and API family to the project. Older Foundry classic examples and current project-client examples can use different methods, so record the package versions used by the application.

Use a developer identity for a local exercise and an explicitly configured workload identity in Azure. DefaultAzureCredential tries supported credential sources; importing it does not create a managed identity or grant a role. Confirm the effective principal and data-plane permission on the intended project, plus separate access to Search, Storage, or other connected resources. A private endpoint does not supply those permissions.

Promote the application, agent instructions, tool schemas, model deployment configuration, and evaluation dataset through review. Retain their version identifiers so a regression can be reproduced. Validate a candidate against the same baseline set and preserve a rollback route; a successful infrastructure deployment alone does not prove useful agent behavior.

Reference: Foundry SDKs and project endpoints.

Monitor, cost, and security

For HTTP 429, inspect the response details, retry guidance, deployment capacity, and request/token limits. Use bounded retries with backoff and jitter, honor Retry-After when provided, and control concurrency. Long inputs, excessive output-token budgets, bursts, and repeated agent loops can consume capacity; buying more capacity is not always the first fix. Measure quality and cost before routing simpler work to a smaller deployment.

Wrong citations are a grounding and relevance signal. A change in tone or safety incidents can signal drift, a changed input mix, or a release regression; compare versions and representative evaluation data before assigning a cause. Files that never appear, or chunks that stay after a rebuild, are ingestion quality and index health. Those are different dashboards. Do not “fix” a stale index by only editing the system prompt.

Diagnose the failing layer before changing the model

Track cost per successful user task rather than tokens alone. Include retrieval, model calls, retries, reflection passes, extraction, and tool use where they incur charges. Compare p50 and p95 latency to distinguish ordinary performance from the slow tail. Set a maximum number of agent steps and a total execution budget, and make timeout or budget exhaustion visible to the caller.

References: Foundry private networking and throttling and retry design.

Responsible AI for generative and agent systems

Safety filters and moderation classify disallowed input and output. They are not replaced by a longer context window or a polite prompt. Evaluators and safety evaluations produce evidence before you promote a change. Explanation tooling matters when reviewers must see why a completion was allowed, not only that a blocklist exists.

Audit means traces, provenance for the passage that supported an answer, and an approval record for a high-risk tool. Oversight modes and tool constraints govern what an agent may do. If the agent may draft a refund or an email but must not submit it, the design is an approval gate, not a higher temperature.

Keep retrieved evidence separate from instructions

A search passage, image, transcript, or tool response can contain hostile instructions. Treat it as untrusted task data even when it arrived through an approved connector. Combine detection with constrained tools, least-privilege identities, argument validation, and approval enforcement. A content-safety score addresses its configured risk categories; it does not certify that a caller may view a document or execute a payment.

Log the request identifier, selected source identifiers, tool name, sanitized arguments, outcome, and approval reference needed for investigation. Restrict access and retention; avoid storing credentials or entire sensitive documents simply to obtain a trace. Explain decisions using verifiable evidence and policy results. Reproduce failures with synthetic or appropriately redacted samples rather than copying production secrets into a test prompt.

Reference: agent safety and tool approval controls.

Build generative applications

Retrieval-augmented generation fetches relevant passages and conditions the answer on them. It does not make an answer true by itself, so evaluation still looks for fabrications (citations that are not in the sources) and for relevance (a fluent answer about the wrong product). A tool-augmented flow calls a function and continues from the result. A multistep pipeline keeps stages such as eligibility and calculation in order.

For machine-readable results, use supported structured outputs with a JSON Schema, then validate the parsed data and business rules. JSON mode targets valid JSON but does not enforce a particular schema. A temperature setting does not guarantee either schema compliance or factual accuracy, and some models do not support that setting. Keep durable instructions in the appropriate agent configuration and enforce consequential rules in application code.

A response contract has three separate checks

1. Completion state: determine whether the request completed, was refused, or stopped before producing a usable answer.
2. Shape: parse the result and validate required keys, types, and supported schema constraints. Reject malformed or unexpected arguments rather than forwarding them.
3. Meaning: check identifiers, amounts, dates, source support, and authorization in application code. Schema-valid content can still be wrong.

For a draft action, a schema might include an order identifier and an integer amount in minor currency units. Keep the caller identity and approval record outside model control. A field named approved supplied by the model must never become proof that a human approved the action. Handle missing evidence with a clarification or an explicit inability to answer instead of manufacturing required values.

References: structured outputs and JSON mode and model-specific parameter support.

Build agents

Define the role, the goal, how conversation state is tracked, and the tool schema. Function calling returns a request that your code executes. The model must not invent the API result. Conversation memory keeps a fact such as a chosen size across turns. Search and Content Understanding are tools when the agent must query an index or read a form at question time.

Use an orchestrated multi-agent solution when specialist roles hand work to each other. One agent with one tool is enough when there is no second role. Semiautonomous workflows prepare an action and wait for approval before the side effect. If a tool fails and the reply still says the action succeeded, inspect the trace and do an error analysis. Hiding the error is not a fix.

Validate and authorize every tool call

The model proposes a tool call; the application or configured service executes it. A tool schema describes inputs but does not establish the user's identity, validate ownership of an order, or grant permission. Check required arguments and business limits, derive the caller from the authenticated session, and allow only the intended tools. An unavailable required argument should trigger clarification or a rejected call, not a fabricated customer identifier.

For a side effect, first produce a reviewable draft. Bind approval to the exact recipient, resource, amount, and action. If those details change, require a new approval. Enforce this in the execution path so a direct tool call cannot bypass the UI. Use an idempotency key and durable operation status where the target supports them. If a request times out after the remote service might have committed it, reconcile that status before retrying.

Scope memory to the authenticated user or tenant and the appropriate conversation. Establish retention and deletion behavior, and separate preferences from authoritative account data. An order ID remembered from another session does not prove access. For multiple agents, define handoff inputs, output contracts, step limits, and which component owns the final decision; more agents also create more failure and leakage paths.

Optimize and operate

A reflection or self-critique loop checks a draft against the retrieved passages. The study guide names chain-of-thought evaluation. In an application, evaluate the observable workflow: tool choices, arguments, results, citations, and explicit intermediate artifacts. Do not assume a provider exposes private internal reasoning, and do not treat a generated explanation as a faithful execution log. Observability for this exam is tracing, token analytics, safety signals, and a latency breakdown that separates retrieval from the model call. A marketing-site page view does not locate that delay.

Orchestrate more than one model when the steps differ. A rules engine should own an exact numeric threshold. The language model can explain the result. Leaving the number only to the model allows it to drift from the published rule. Do not expand autonomy while safety signals are still firing. Evaluate first.

Evaluate retrieval, answers, and actions separately

Build a versioned test set that includes ordinary requests, ambiguous questions, unsupported questions, denied documents, hostile retrieved instructions, and failed tools. Label expected evidence or allowed behavior. Retrieval evaluation asks whether relevant authorized passages were available; answer evaluation asks whether the response used them faithfully and answered the request. Tool evaluation asks whether the right operation ran with valid arguments and an accurately reported result.

Record failure categories rather than just one average score. A strong fluency score can hide unauthorized retrieval, and a useful answer can still cite an unrelated passage. Automated evaluators help compare releases, but inspect representative disagreements and consequential cases manually. Keep an independent held-out set; improving only the examples embedded in prompts can conceal poor generalization.

Bound critique loops with an iteration count and token budget. A second model can repeat the first model's error, so verify explicit source references and deterministic calculations rather than treating agreement as proof. Make a release decision from defined quality and safety thresholds together with latency and cost, and retain the baseline for future regression tests.

Computer vision

Generation creates pixels or frames. Understanding reads media that already exists. Image generation from a prompt and reference media is not a caption. Video generation is not alt text. Inpainting uses a mask to target a region; exact preservation outside the mask depends on the model and workflow and should be checked. Editing a generated video replaces a segment inside the clip. Use the generation and editing controls the platform provides rather than inventing an unsupported renderer.

Captions can be concise or detailed, for one image or many. Visual question answering must be grounded in the image, not in a slogan reused for every question. Alt text and extended descriptions follow accessibility guidance. Content Understanding can extract visual characteristics and identify objects, components, or regions. A focused analyzer can fit a single understanding task. The study guide also names pro mode; read the lifecycle note below before attempting an older tutorial. Video analysis interprets segments, not only the file name.

Classify and filter unsafe visual content. Text baked into an image that says to ignore instructions is indirect prompt injection. Detect it and do not promote that text to a system instruction. Visual policy can also require watermarks, blocked symbols, or brand checks. Those are safety controls, not a reason to skip inspection and publish the file.

Exam terminology and current Content Understanding workflows

Lifecycle note, reviewed October 29, 2026: the April 16 exam guide still names pro mode. Microsoft's current migration guidance says that pro mode belonged to the retired 2025-05-01-preview API. It directs new migration assessment to agentic document analysis in 2026-06-01-preview, which is a preview feature and not a drop-in replacement. Its initial scope is document analyzers with one input file per request. Check current limits and supported field methods before adapting an older multi-file example.

For visual tasks, select the modality and API actually supported by the chosen model or analyzer. Preserve image resolution sufficient for the target text or objects, and test cropped or low-quality inputs. A low-resolution photo may support a general caption but not a reliable serial number. For video, keep time offsets with extracted events so a reviewer can inspect the corresponding segment.

Reference: agentic mode and migration from pro mode.

Text analysis and speech

Language tasks on this exam include entities, topics, summaries, and structured JSON from generative prompting or Foundry Tools. Sentiment and tone are different from a safety or sensitive-content flag. Domain customization shapes a compliance summary or a field list. It is not “drop the source and improvise.”

Use Azure Translator in Foundry Tools when the stem asks for a dedicated translation service. An LLM translation flow fits when translation is part of a generative step such as a summary. Do not answer a translation stem with image captioning.

Speech to text listens. Text to speech speaks. For a few frequently changing names, first assess a supported phrase list. Custom Speech is appropriate when broader domain adaptation and suitable training/evaluation data are needed; measure recognition error before and after the change. Multimodal reasoning over audio pulls decisions and owners from a meeting, which is more than a raw transcript. Spoken translation changes the caller’s language. It is not an image task.

Keep extraction, translation, and audio quality measurable

A sentiment label describes expressed attitude; it is not a sensitive-data detector or an authorization decision. A summary should preserve the source's qualifications, dates, and negations. For entity extraction, retain a source reference and distinguish an absent field from a guessed value. Validate identifiers and dates after extraction, especially where locale changes the interpretation.

Test speech with realistic microphones, background noise, speaker accents, and domain vocabulary. Use word error rate alongside errors in important names or numbers; a small average improvement can still leave critical product names wrong. Evaluate phrase lists before undertaking model customization when the change is just a small vocabulary. For spoken translation, verify the language pair, recognition quality, translated text, and synthesized output separately.

When extracting meeting decisions, retain speaker labels or time references where available and distinguish a proposal from an agreed action. A fluent summary must not assign an owner who never accepted the task. Use representative recordings with appropriate consent and keep sensitive audio out of public exercise datasets.

Reference: phrase lists and recognition accuracy.

Information extraction

For an indexed retrieval workflow, ingest documents and derive searchable content from images, audio, and video using appropriate extraction, transcription, or multimodal representations. Scans with no text layer need OCR in that ingestion path. Layout enrichment keeps tables and reading order. Evaluate a suitable built-in skill against representative samples before adding a custom extractor. A custom skill calls an extractor you already trust when the built-in skill misreads a proprietary form.

Content Understanding produces a clean, grounded representation for RAG and agents. Analyzers should emit structured fields or markdown when the next step needs that shape. A multimodal invoice path combines OCR, layout analysis, and field extraction. Keyword search on the file name does not return the amount.

Retrieval stages are not interchangeable

Keyword retrieval is useful for exact terms; vector retrieval compares embeddings for conceptual similarity. Hybrid retrieval runs text and vector retrieval and merges the ranked results. Semantic ranker then improves the ordering of candidate text. In a basic reranking-only setup, it cannot recover a relevant passage that never entered the candidate set. Evaluate the retrieval stage before changing the answer prompt.

Use compatible embedding models and dimensions for indexed content and queries. Chunk documents at meaningful boundaries, retain headings and source/page identifiers, and test whether tables remain understandable. An exact identifier may need a filterable field or suitable analyzer; hybrid retrieval does not guarantee a specific product will rank first. Track document versions, failed items, and deletions so old policy text does not survive a source update.

Preserve document permissions through retrieval. Apply supported document-level authorization or carefully implemented security trimming before content enters the model context. Construct identity filters from trusted authentication information, not from a user-supplied tenant ID or a model argument. Keep permission changes, cached answers, conversation state, and indexed copies consistent. Telling the model to hide unauthorized chunks is too late once they have been retrieved into its context.

Extracted fields need evidence and acceptance rules

A Content Understanding analyzer defines the content and field representation needed downstream. Markdown can preserve readable structure for RAG; typed fields support an application contract. Use available document confidence and source grounding to locate uncertain fields, but do not interpret a confidence score as a guarantee. Validate totals, currency, date formats, and required values; send missing, conflicting, or low-confidence consequential fields for review.

References: hybrid retrieval, semantic ranking, document-level access, and confidence and grounding.

Worked scenario

A claims assistant must answer from policy PDFs, read photographed receipts, and draft a payment that a reviewer approves. Walk the controls in order.

1. Ingest the PDFs. Run OCR on scanned pages. Index them, and use hybrid search if both policy numbers and paraphrases matter.
2. Add Content Understanding so receipt photos become fields, not a file name.
3. Give the agent a search tool and a content-understanding tool. Store conversation memory for the claim number.
4. Keep the payout threshold in a rules engine. Let the model explain the result.
5. Require approval before the payment tool runs. Trace the passage and the approver.
6. Call the project with managed identity over a private endpoint. Evaluate fabrications and safety signals while retaining every approval required by the business policy.

If you can name the control that is missing from a shorter version of this story, you are studying the outline rather than a product list.

Extra What changed from AI-102

AI-102 retired on June 30, 2026. AI-103 is a different certification: Azure AI Apps and Agents Developer Associate. The current outline is organized around Foundry, agents, generation, and extraction. LUIS, QnA Maker, and Custom Vision training projects are historical context. Do not spend the timed practice on them until the five domains are familiar. Confirm any later skills-measured update on the Microsoft study guide before you book.

Five guided labs with observable outcomes

Use a disposable project and synthetic data. Before a cloud exercise, check model availability, role assignments, and potential charges. Preserve useful test evidence and remove temporary resources afterward. If cloud access is unavailable, work through the input, expected result, and failure case as a tabletop exercise.

1. Connect a Python client to the intended project

Use Python 3.10 or later and a compatible current Foundry SDK. The Microsoft SDK overview reviewed on October 29, 2026 lists azure-ai-projects>=2.3.0 with openai>=3.0.0 for the 2.x examples. Install the required packages, including azure-identity, in an isolated environment and lock the versions you validate. Set AZURE_AI_PROJECT_ENDPOINT to your project's endpoint and AZURE_AI_MODEL_DEPLOYMENT to a ready deployment that supports Responses. Authenticate with an identity authorized for that project.

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
deployment = os.environ["AZURE_AI_MODEL_DEPLOYMENT"]

with DefaultAzureCredential() as credential:
    with AIProjectClient(endpoint=endpoint, credential=credential) as project:
        with project.get_openai_client() as client:
            response = client.responses.create(
                model=deployment,
                input="Explain retrieval-augmented generation in one sentence.",
            )
            if response.status != "completed" or not response.output_text:
                raise RuntimeError("No completed text response; inspect the result status.")
            print(response.output_text)

Expected result: a completed text response from the intended deployment. As negative cases, try a nonexistent deployment name and a test identity without access. Diagnose the resulting error without adding a secret to source code. This example demonstrates connection and inference; it does not implement RAG or create an agent.

2. Compare retrieval and enforce a document boundary

Create a tiny corpus containing a product code, a paraphrased product description, and two tenant-specific policies. Record document IDs and versions. Compare keyword, vector, hybrid, and hybrid-plus-reranking retrieval using the same questions. Inspect the candidate passages before evaluating the final answer.

Expected result: the appropriate product and source passage are retrievable, while a user from tenant A never receives tenant B's passage in the tool result or model context. Update and then delete one policy and verify the index and any answer cache reflect that change. An unsupported question should produce an evidence-limited response.

3. Test the agent's action contract without sending anything

Use a stub order API that returns fixture data and a mock action recorder. The agent may draft an action but the execution service must validate arguments, order ownership, and an approval for those exact details. Supply a retrieved passage that asks the agent to bypass approval, omit a required argument, and simulate a timeout after the mock operation is recorded.

Expected result: no approval means zero executed actions; altered arguments invalidate the old approval; missing arguments are rejected; a retry does not duplicate a recorded operation. The assistant reports failure or uncertainty accurately instead of claiming success. Keep this exercise on mocks rather than a real payment or email service.

4. Extract a receipt and review uncertainty

Use a synthetic receipt with a total, currency, and date. Compare a clear image with a blurred copy and one missing the currency. Retain extracted fields, available confidence, and source references. Validate the amount and date in code, then route uncertain results to a review state.

Expected result: the clear sample matches the reference fields. Missing or conflicting evidence does not silently become an invented value, and a reviewer can locate the relevant source region. Record analyzer and API versions so a later model change can be compared.

5. Build a small multimodal regression set

Prepare a product photo, a photo containing an irrelevant hostile instruction, a short consented or synthetic speech sample with a product name, and a harmless content-policy boundary case. Record the allowed output for each. Compare a caption with a grounded visual answer; compare base speech recognition with a supported phrase list or evaluated custom model.

Expected result: the workflow uses visual or audio evidence for the requested task, preserves instruction boundaries, and records recognition errors and safety decisions. A failed tool or missing input must not be counted as a correct answer. Keep the same set for the next prompt, model, or tool-schema release.