AI-102 Implement an agentic solution

Implement an agentic solution Detailed Explanation

1. What “agentic” means in AI-102 (and what it does not mean)

1.1 Definition in the exam context

In AI-102, an agentic solution is a way of building an AI application where the model does more than “chat.” The model can work toward a goal by:

• Planning steps (for example: “First I will look up the policy, then I will check the user’s account status, then I will create a ticket.”)

• Calling tools (APIs / functions) to get reliable information or perform actions

• Keeping state so it can continue multi-step work (remember what it already did, what it still needs, and what the result was)

• Iterating when needed (retrying a tool call, using another tool, or asking the user for missing details)

In the Microsoft Azure ecosystem, the study guide explicitly references creating agents with Microsoft Foundry Agent Service and building more complex agents with Microsoft Agent Framework, plus orchestrating complex workflows such as multi-agent solutions.

What a beginner should picture

Think of a simple chatbot as a person who only talks.
Think of an agent as a person who can talk and also do work by using “helpers” (tools), like:

• searching a company knowledge base,

• checking a database,

• filing a support ticket,

• scheduling an appointment,

• sending an email draft for approval (depending on your system design).

Key idea

An agentic solution is goal-driven and action-capable, not just conversational.

1.2 What the exam usually emphasizes

AI-102 typically tests agentic solutions in a practical, engineering-focused way. The exam expects you to understand and implement the following themes.

Engineering choices: when to use an agent vs. a simple chat/RAG flow

You should be able to decide:

• When a single response is enough (simple chat)

• When you need retrieval grounding (RAG)

• When you need tool use + multi-step execution (agent)

A beginner-friendly guideline:

• Use chat when the user wants explanations, summaries, or creative content.

• Use RAG when the user needs answers grounded in your documents (“What does our handbook say?”).

• Use an agent when the user’s request requires steps and actions (“Open a ticket, check my status, then recommend the next step.”).

Tooling integration: calling functions, retrieving data, executing workflows

You should be able to design:

• A tool that retrieves authoritative info (search, database query)

• A tool that performs an action (create a ticket, update a record)

• A workflow that chains tools in the correct order

Reliability: guardrails, monitoring, evaluation, and safe tool execution

Agents can fail in more ways than chatbots because they:

• call external systems that may be unavailable,

• can make incorrect tool choices,

• can repeat actions unless controlled.

So reliability includes:

• Guardrails to keep behavior safe and predictable

• Monitoring to detect errors and quality issues

• Evaluation to measure success and catch regressions

• Safe execution patterns (validation, retries, approval gates for high-impact actions)

Multi-step orchestration: state, retries, idempotency, and error handling

You should understand how to:

• Keep track of what happened so far (state)

• Retry safely when a tool fails (retries)

• Avoid performing the same action twice (idempotency)

• Communicate clearly to the user when something cannot be completed (error handling)

2. Core building blocks of an agentic solution

2.1 Model (reasoning core)

The model is the “brain” that decides:

• what the user wants,

• whether it needs a tool,

• which tool to call,

• how to use the tool results,

• how to respond.

A beginner-friendly mental model:

• The model is like a smart coordinator.

• Tools are like specialized employees.

• The orchestrator is the manager who ensures work happens in the right order.

Choosing a suitable model

When selecting an LLM for agentic work, you care about more than “is it smart?”

Tool use capability (function calling / structured outputs)

Agents often need to produce structured tool calls, not just text.

• Example: calling a CreateTicket tool with fields like title, priority, category, description.

• A model suited for tool use will reliably produce valid structured inputs.

Why this matters:

• If the tool call structure is wrong, your API call fails.

• If fields are missing, the agent may do the wrong action.

Context length

Context length is “how much information the model can consider at once,” including:

• user conversation history,

• retrieved documents,

• tool results,

• system instructions.

Beginner rule:

• Longer context can help, but it also increases cost and can add noise.

• Good systems avoid dumping everything into context.

Latency and cost constraints

Agentic solutions can become expensive if they:

• take many turns (plan → tool → plan → tool → answer),

• retrieve too much text,

• call tools repeatedly.

So you should learn to:

• keep tool outputs concise,

• limit retries,

• terminate loops safely,

• cache stable results when appropriate.

Designing prompts and system messages (agent instructions)

In an agentic solution, prompts are not just “write nicely.” They are behavior contracts.

Define role, boundaries, and safety constraints

Your system instructions should clearly state:

• what the agent is allowed to do,

• what tools it can use,

• what it must never do (for example: “Never execute payments without user approval.”),

• how to treat sensitive information.

Beginner tip:

• If you do not write boundaries, the model will attempt to be helpful in unsafe ways.

Specify tool-use policy

A tool-use policy answers questions like:

• When should the agent call a tool vs. answer directly?

• When should the agent ask the user for missing information?

• When should the agent stop and return a partial result?

A simple tool-use policy pattern:

• If the user asks for a fact that must be accurate and current → call a retrieval tool.

• If the user asks for an action with real-world impact → confirm required fields and use approval gates for high-risk actions.

• If a tool fails repeatedly → stop, explain the failure, and provide next steps.

2.2 Tools (actions the agent can take)

A tool is any function/API the agent can call to:

• fetch reliable information, or

• perform an external action.

Tools are the difference between:

• “I think your order shipped yesterday” (guessing)
  and

• “Your order shipped on Jan 10, tracking number X…” (verified via system API).

Typical tool types (with beginner examples)

Retrieval tools

Purpose: get authoritative information.
Examples:

• Search a policy knowledge base (Azure AI Search)

• Query a product catalog database

• Fetch a document by ID

Beginner example:

• User: “What is the password reset policy?”

• Agent: calls SearchPolicyDocuments(query="password reset policy")

Action tools

Purpose: do something in an external system.
Examples:

• Create a support ticket

• Update a CRM record

• Schedule a meeting

Beginner example:

• User: “Create a high priority ticket for my VPN problem.”

• Agent: calls CreateTicket(title="VPN not connecting", priority="High", ...)

Computation tools

Purpose: perform calculations or validations reliably.
Examples:

• Compute totals, tax, discounts

• Validate a form input (dates, IDs)

• Transform data formats

Beginner example:

• User: “What’s the total cost if I buy 17 units with 8.25% tax?”

• Agent: calls CalculateTotal(quantity=17, price=..., taxRate=0.0825)

Workflow tools

Purpose: trigger multi-step processes that your system runs.
Examples:

• Start an Azure Logic Apps workflow

• Invoke an Azure Function

• Trigger a durable workflow for long-running tasks

Beginner example:

• User: “Onboard a new employee.”

• Agent: calls StartOnboardingWorkflow(employeeId=...)

Key tool design principles (exam-relevant, beginner-friendly)

Least privilege

Give each tool only the permissions it needs.

• A “read policy” tool should not be able to modify payroll.

• An “update CRM” tool should not access HR files.

Why it matters:

• Limits damage if the agent is tricked (prompt injection) or misbehaves.

Stable interfaces

Define clear input/output structure:

• required fields,

• allowed values,

• versioning if you change it over time.

Beginner tip:

• The model is more reliable when your tool schema is strict and clear.

Idempotency

Idempotency means: repeating the same request does not cause duplicate side effects.

• If a ticket creation times out and you retry, you do not want two tickets.

Common patterns:

• Use a client-generated requestId

• Have the server reject duplicates for the same requestId

Observability

Your system should log:

• which tool was called,

• when it was called,

• whether it succeeded,

• key error codes,

• timing.

But also:

• avoid logging sensitive content (PII, secrets),

• sanitize/redact where necessary.

2.3 Memory and state (what the agent “remembers”)

In real systems, “memory” is often not magic inside the model. It is usually stored outside the model and retrieved when needed.

AI-102 coverage commonly includes agent concepts like continuity across workflows and sessions, and Microsoft documentation describes long-term memory capabilities in Foundry Agent Service as a managed solution.

Three practical categories of memory/state

Short-term state

What the agent needs right now to finish the current task:

• conversation history (recent turns),

• the current plan,

• intermediate tool results.

Example:

• “We already verified the user’s identity.”

• “We already fetched the policy document.”

Long-term memory

What the system may keep across sessions:

• user preferences (“prefers email updates”),

• stable profile information (role, department) if allowed,

• recurring choices (“always use my SF office address”).

Important:

• Long-term memory should be carefully controlled for privacy and compliance.

Episodic traces

Structured records of “what happened” for:

• debugging,

• audits,

• evaluation,

• incident response.

Example trace:

• Tool called: SearchPolicyDocuments

• Input: query string

• Output: document IDs

• Next action: summarize and cite

Common storage choices (conceptual)

• Vector store: helps semantic recall (embeddings) to retrieve “similar” content.

• Document store: stores transcripts, documents, or artifacts.

• Key-value store: fast storage of simple session/user attributes.

Memory policies you should know (beginner essentials)

What is safe to store (PII and compliance)

• Do not store secrets (passwords, tokens).

• Be cautious with personal data (addresses, IDs).

• Follow your organization’s retention and consent rules.

Expiration / retention

• Short-term session state might expire quickly (minutes/hours).

• Long-term memory might have explicit retention windows (days/months).

• Logs often require retention policies and access controls.

User consent and access control

• Users may need to opt in to personalization.

• Only authorized systems/users should access stored memory.

2.4 Orchestration (the loop that runs the agent)

Orchestration is the “control loop” that turns a model + tools into a working agent.

The AI-102 study guide explicitly mentions implementing complex workflows, including orchestration for multi-agent solutions, multiple users, and autonomous capabilities.

The common agent loop (beginner explanation)

1. Interpret goal

   • Understand what the user wants and what constraints exist.

2. Plan

   • Decide the steps and which tools to use.

3. Act

   • Call tools and perform actions.

4. Observe

   • Read tool results and detect errors.

5. Reflect/iterate

   • Adjust the plan, retry if safe, or ask for missing information.

6. Respond

   • Provide the final outcome clearly, with grounding/citations if needed.

Orchestration options in Azure (high-level)

• Hosted in an agent runtime (Microsoft Foundry Agent Service) which can provide server-side orchestration and retry of tool calls with structured logging.

• Implemented via an agent framework (Microsoft Agent Framework is referenced in the AI-102 study guide as the path to implement complex agents).

• Built custom in your application (you write the orchestration logic yourself using SDKs and your own state machine)

3. Implementing agents with Azure AI Foundry Agents

3.1 What the hosted agent service provides

A hosted agent service means Azure runs and manages the agent runtime for you. In the context of AI-102, this is commonly described as Azure AI Foundry Agent Service (also referred to as Foundry Agents in Microsoft learning materials).

At a high level, the service provides:

• A managed execution environment for agents

• Built-in support for tool calling and multi-step reasoning

• Infrastructure features such as scaling, reliability, and logging

For a beginner, the key idea is:

You focus on what the agent should do, not on building and maintaining the execution engine.

What “managed” means in practice

You do not need to:

• Host your own orchestration server

• Manually manage retries for every tool call

• Build logging pipelines from scratch

• Handle concurrency and scaling logic yourself

Instead, you:

• Define the agent configuration

• Define tools and policies

• Call the agent from your application

What the exam expects you to understand

The exam does not test UI details or portal screenshots.
It tests whether you understand:

• What the service is responsible for

• What your application is responsible for

• When a hosted agent service is the correct architectural choice

3.2 Typical implementation steps (what you should be able to do)

Below is a conceptual step-by-step flow. This is exactly the level of understanding AI-102 expects.

Step 1: Provision or choose an agent runtime resource

You begin by selecting an agent-capable Azure AI resource that:

• Supports your chosen LLM

• Supports tool invocation

• Meets your region, compliance, and cost requirements

Beginner mindset:

• Treat this as “creating the place where my agent lives.”

Step 2: Define an agent

Defining an agent means configuring behavior, not writing application code.

Base model selection

You select:

• A model that supports tool calling

• A model with enough context length

• A model whose latency and cost fit your use case

Exam tip:

• If the scenario mentions function calling or structured tool invocation, you must choose a model that supports it.

System instructions

System instructions define:

• The role of the agent (for example: “You are an IT support assistant.”)

• Behavioral boundaries (“Do not perform destructive actions without confirmation.”)

• Safety expectations (“If unsure, ask for clarification.”)

These instructions guide all decisions the agent makes.

Tool definitions (function schemas)

You define:

• Tool name

• Input schema (required fields, data types)

• Output schema (what the tool returns)

Key beginner concept:

• The model does not “guess” how to call your API.

• It follows the schema you provide.

Grounding / retrieval configuration (when used)

If your agent uses retrieval:

• You connect it to a search index or data source

• You define how and when retrieval is used

This is common when combining agents + RAG.

Step 3: Run the agent

Your application:

• Sends the user message

• Optionally sends context (user ID, session data)

• Receives:

  • the final response,

  • and often a trace of tool calls (for debugging and monitoring)

Important:

• The agent runtime handles the internal loop.

• Your app simply consumes the result.

Step 4: Operationalize the solution

This is where many beginners forget exam-relevant details.

You should consider:

• Logging and telemetry: what actions did the agent take?

• Rate limits: how often can users invoke the agent?

• Cost controls: token limits, tool usage limits

• Environment separation:

  • Development

  • Testing

  • Production

Exam pattern:

• Case study questions often ask: “What should you do next before going to production?”

4. Function calling and tool invocation (high-frequency exam concept)

Function calling is one of the most tested concepts in agentic solutions.

4.1 Tool schema design

A tool schema is a contract between the model and your system.

Structured schemas

A good schema:

• Clearly defines required vs optional fields

• Uses precise data types (string, number, enum)

• Restricts allowed values when possible

Beginner example:

• Instead of priority: string

• Use priority: enum { Low, Medium, High }

This reduces errors and improves reliability.

Input validation (server-side)

Even if the model usually behaves well:

• Never trust inputs blindly

• Validate inputs before executing real-world actions

Why this matters:

• Protects against malformed requests

• Protects against prompt injection

• Prevents accidental destructive actions

Structured outputs

Tool outputs should also be structured:

• Clear success/failure indicators

• Error codes and messages

• Minimal but sufficient data

The model can then:

• Reason about what happened

• Decide the next step correctly

4.2 Tool routing strategy

Tool routing is how the agent decides which tool to call and in what order.

Single-tool strategy

The agent:

• Decides whether to call one specific tool or answer directly

Example:

• Tool: SearchKnowledgeBase

• If confidence is low → call tool

• If question is general → answer directly

This is the simplest agent pattern.

Multi-tool strategy

The agent:

• Chooses among several tools based on intent

Example:

• Search tool

• Ticket creation tool

• User profile tool

The agent must:

• Identify intent

• Select the correct tool

• Populate the correct schema

Chained tools

The output of one tool becomes input to another.

Example chain:

1. Search documents

2. Summarize results

3. Create a support ticket with the summary

Beginner caution:

• Chaining increases power, but also increases failure points.

• Always include clear stopping conditions.

4.3 Failure handling

Agents must expect failure as a normal condition.

Common failure types

• Tool timeouts

• Partial or empty results

• External systems returning inconsistent data

• Schema mismatches

• Permission or authorization errors

Mitigation patterns (exam-relevant)

• Retries with backoff: retry a few times for transient failures

• Fallback tools: try another data source if the primary one fails

• Ask the user: request missing or ambiguous information

• Safe partial completion:

  • Explain what succeeded

  • Explain what failed

  • Provide next steps

Exam insight:

• A “good” agent does not hide failures.

• It communicates clearly and safely.

5. Multi-agent solutions (coordination patterns)

5.1 When multi-agent makes sense

Multi-agent solutions are advanced, but the exam expects you to understand when they are appropriate.

Use multiple agents when:

• The task is complex and naturally decomposed

• Different steps require different permissions

• Different expertise is needed

Beginner example:

• One agent plans the solution

• Another agent executes actions

• A third agent validates compliance

5.2 Common orchestration topologies

Manager–worker pattern

• A manager agent:

  • Breaks down the task

  • Assigns work to worker agents

• Worker agents:

  • Execute specialized tasks

  • Return results

This pattern improves clarity and control.

Pipeline pattern

Agents work in sequence:

1. Draft

2. Verify

3. Execute

This is common in:

• Content generation with compliance checks

• Data extraction followed by validation

Debate / consensus pattern

• Multiple agents propose solutions

• A final agent evaluates and selects the best one

Exam note:

• This pattern is powerful but expensive.

• Cost and latency must be justified.

5.3 Key operational concerns

Multi-agent systems introduce new risks.

You must consider:

• Shared vs isolated state: what do agents see?

• Cost explosion: many agents × many turns

• Conflict resolution: agents disagree

• Deterministic termination: avoid infinite loops

AI-102 scenarios often ask:

“How do you prevent uncontrolled execution?”

6. Grounding and retrieval inside agentic systems

6.1 Agentic RAG

Agentic RAG extends traditional RAG by allowing the agent to:

• Retrieve information multiple times

• Refine queries

• Validate answers

Compared to classic RAG:

• Classic RAG: retrieve once, answer once

• Agentic RAG: retrieve → reason → retrieve again → answer

This is useful when:

• Data is complex

• Precision matters

• Permissions vary by user

6.2 Key design decisions

Retrieval source

Common choices:

• Azure AI Search index

• Internal APIs

• Databases

Query strategy

• Keyword search for exact matches

• Semantic search for meaning

• Vector search for similarity

• Hybrid for best coverage

Chunking strategy

How documents are split:

• By headings

• By semantic boundaries

• With overlap to preserve context

Poor chunking leads to poor answers.

Citation strategy

Agents should:

• Track document IDs

• Reference sources in the final response

• Avoid inventing citations

This is especially important in enterprise scenarios.

7. Safety, security, and governance for agents

7.1 Prompt injection and tool abuse defenses

Beginner principle:

Treat all retrieved content as untrusted.

Defenses include:

• Strict tool schemas

• Server-side validation

• Approval steps for sensitive actions

• Allowlists for tool usage

7.2 Access control

Agents must respect:

• User identity

• User permissions

• Data-level security

Examples:

• Row-level security in databases

• Document-level ACLs in search

• Identity-based authentication for APIs

Never:

• Embed secrets in prompts

• Hardcode credentials in tool definitions

7.3 Content safety and policy enforcement

You may need to:

• Filter harmful content

• Redact sensitive information

• Refuse disallowed requests politely

Exam insight:

• Safety is not optional.

• It is part of the system design.

8. Evaluation, testing, and monitoring (the exam’s practical angle)

8.1 What to evaluate

You should measure:

• Task success

• Correct tool usage

• Grounding accuracy

• Safety compliance

• Latency and cost

8.2 How to test agentic workflows

Testing should include:

• Unit tests for tools

• Integration tests with mocked systems

• Scenario tests (normal and adversarial)

• Regression tests to catch behavior drift

8.3 Observability basics

A production-ready agent:

• Logs tool calls

• Tracks errors

• Correlates sessions

• Alerts on unusual behavior

This ensures:

• Faster debugging

• Better reliability

• Safer deployments

Implement an agentic solution (Additional Content)

1. Agent configuration checklist for “Create custom agents”

This section explains how to correctly define an agent before it ever runs. In AI-102, many mistakes come from poorly defined agents rather than weak models. Think of this as “agent design hygiene”.

1.1 Agent identity and scope

An agent must have a clear identity, just like a human employee in an organization.

Defining the agent’s role

You should explicitly define:

• Who the agent is (for example: IT support assistant, HR policy assistant)

• What domain it operates in

• What type of help it provides

A clear role prevents the model from attempting tasks outside its expertise.

Example:

• Role: “Internal IT support assistant”

• Domain: “Company IT systems and policies”

• Audience: “Authenticated employees only”

Defining primary tasks

Primary tasks are what the agent is expected to do regularly.
Examples:

• Answer IT policy questions

• Check system status using tools

• Create support tickets when appropriate

These tasks guide tool selection and workflow design.

Defining out-of-scope tasks

Out-of-scope tasks are equally important.
Examples:

• Legal advice

• Financial decision-making

• Actions affecting payroll or contracts

Explicitly stating what the agent must not do reduces unsafe behavior.

Defining allowed outputs

You should define constraints on outputs, such as:

• Informational guidance only

• No professional or legal advice

• No commitments or guarantees

This is critical for compliance and risk management.

Defining termination rules

Termination rules define when the agent should stop.
Examples:

• Stop after providing a final answer

• Ask a follow-up question if required data is missing

• Stop and escalate if the request is disallowed

Without termination rules, agents may loop or overreach.

1.2 Instruction hierarchy and policy design

Agents follow instructions from different sources. Understanding priority is essential.

Instruction hierarchy

A standard hierarchy is:

1. System instructions – highest priority, non-negotiable

2. Developer instructions – workflow and behavior rules

3. User instructions – goals and requests

System instructions always override everything else.

System instructions

These define:

• Safety rules

• Compliance constraints

• Absolute prohibitions

Example:

• “Never perform irreversible actions without explicit confirmation.”

Developer instructions

These define:

• How the agent should work

• Preferred workflows

• Tool usage patterns

Example:

• “Use the search tool before answering policy-related questions.”

User requests

User requests express intent, but:

• They can be incomplete

• They can be unsafe

• They can conflict with policies

The agent must interpret them within system and developer constraints.

Explicit policy rules

You should clearly define policies for:

• When to use tools (for accuracy, freshness, authority)

• When to ask for missing details (IDs, dates, permissions)

• When to refuse or escalate (disallowed or risky requests)

This reduces ambiguity and improves predictable behavior.

1.3 Tool registration and permissions

Tools are powerful and dangerous if misconfigured.

Tool allowlist

An agent should only know about explicitly approved tools.

• No dynamic or implicit tool access

• Each tool has a clear purpose

This limits accidental or malicious actions.

Least-privilege permissions

Each tool should have:

• Only the permissions it absolutely needs

• No access to unrelated systems or data

Example:

• A “read policy” tool must not modify records.

Tool-specific constraints

For each tool, define:

• Allowed parameter ranges (for example, date limits)

• Restricted tenants, subscriptions, or resources

• Rate limits to prevent abuse or runaway loops

Constraints protect systems even if the agent makes mistakes.

1.4 Memory configuration (what is stored, where, and why)

Memory is a system design decision, not a model feature.

Short-term session state

Used for:

• Current conversation

• Current plan

• Recent tool outputs

This state is usually temporary and session-scoped.

Long-term memory

Used for:

• User preferences

• Stable facts that persist across sessions

Long-term memory must be:

• Explicitly justified

• Carefully governed

• Often optional

Episodic traces

These are structured logs of:

• Tool calls

• Decisions

• Errors

They support debugging, audits, and evaluations.

Retention and deletion policies

You must define:

• How long data is stored

• When it is deleted

• Who can access it

This is important for compliance and trust.

PII handling

You must decide:

• What personal data is allowed

• How it is redacted

• Whether consent is required

• How data is encrypted and protected

Agents should never freely store sensitive personal data.

1.5 Knowledge and grounding connections

Agents must know what data they are allowed to use.

Allowed data sources

Examples:

• Search indexes

• Knowledge bases

• Databases

Each agent should have an explicit list of allowed sources.

Scope boundaries

Define:

• Which indexes or collections are accessible

• Whether cross-domain access is allowed

This prevents data leakage.

Handling missing sources

If data is unavailable:

• The agent should say so

• It should not guess or hallucinate

Handling low-confidence retrieval

When retrieved data is weak:

• The agent should express uncertainty

• It may ask follow-up questions

• It may provide partial answers

Citation requirements

If citations are required:

• The agent must track source IDs

• The agent must not invent sources

This is critical in enterprise and compliance contexts.

2. Implementing complex agents with frameworks (conceptual, exam-aligned)

Frameworks exist to manage complexity.

2.1 When a framework is needed

A framework becomes necessary when:

• Workflows have many steps

• Tool usage branches based on conditions

• Retry and fallback logic is complex

• Tools evolve over time

• Multiple agents must coordinate

Simple agents do not require frameworks. Complex ones usually do.

2.2 Framework concepts to know

You are expected to understand concepts, not APIs.

Plugins, skills, or tools

Reusable capabilities that:

• Encapsulate business logic

• Can be added or removed independently

Planners

Components that:

• Break a goal into steps

• Decide execution order

State management abstractions

Mechanisms to:

• Store session data

• Store memory

• Track progress

Connectors

Standardized ways to connect to:

• Search services

• Storage

• Databases

• Enterprise systems

Workflow orchestration

Combines:

• Deterministic steps (fixed order)

• Model-driven decisions (conditional logic)

2.3 Why frameworks matter

Frameworks improve:

• Separation of concerns

• Testability

• Maintainability

• Observability

From an exam perspective, frameworks are justified when manual orchestration becomes error-prone.

3. Multi-user support and autonomy boundaries

Enterprise agents must safely support many users.

3.1 Multi-user session isolation

Each user must have:

• Separate session state

• Separate memory

• Separate permissions

Session identifiers and per-user storage prevent data leakage.

3.2 Permission propagation and security trimming

Tools must authenticate using:

• Delegated user identity, or

• Application identity

Regardless of method:

• Tools must enforce row-level and document-level security

• Search results must reflect user permissions

3.3 Human-in-the-loop (approval gates)

Some actions are too risky to automate.

High-risk actions include:

• Financial transactions

• Deletions

• Permission changes

• External communications

A common pattern is:

• Suggest the action

• Ask for confirmation

• Execute only after approval

3.4 Levels of autonomy

Agents can operate at different autonomy levels:

• Advisory only

• Action with confirmation

• Fully autonomous (low-risk only)

Choosing the right level is a design decision.

4. Test, optimize, and deploy an agent

Production readiness is a core AI-102 theme.

4.1 Versioning and change control

You should version:

• Prompts

• Tool schemas

• Agent configuration

• Retrieval settings

Changes should move through:

• Development

• Testing

• Production

4.2 Regression testing for agent behavior

Tests should cover:

• Normal scenarios

• Edge cases

• Adversarial inputs

• Tool failures

You must verify that the agent:

• Chooses correct tools

• Uses correct parameters

• Stops appropriately

• Refuses unsafe requests

4.3 Evaluation metrics

Consistently measure:

• Task success

• Tool accuracy

• Grounding accuracy

• Safety outcomes

• Latency and cost

Metrics turn subjective behavior into measurable quality.

4.4 Deployment safeguards

Safeguards include:

• Token and tool limits

• Circuit breakers

• Rollback strategies

• Monitoring alerts

These prevent runaway failures.

4.5 Operational playbooks

You should define responses for:

• Tool outages

• Retrieval degradation

• Model behavior drift

• Security incidents

Operational readiness is part of system design, not an afterthought.