AB-620 Test and manage agents

Test and manage agents Detailed Explanation

Evaluate agent performance with test sets, evaluation methods, and result review

Exam Radar

Core Priority: This topic tests evidence-based quality control. The question usually asks how to prove answer quality, task completion, escalation behavior, or regression status before release.

High Frequency: Expect test sets, evaluation method selection, result review, transcript analysis, task success checks, failure categories, and regression comparison.

Confusion Alert: An evaluation question may describe a failed answer, but the trap is fixing the first visible response without classifying whether the owner is routing, grounding, tool execution, handoff, formatting, escalation, or deployment.

Scenario Logic: A team needs to prove that a production agent handles policy questions, tool actions, and escalation cases before publishing a new version. The best first move is the one that preserves identity, data boundary, execution contract, or evidence collection before optimizing the conversation wording.

Version Delta: Testing scope is not just manual chat review: test sets, evaluation method selection, and result review are explicit management tasks.

Failure Trigger: Failure appears as unrepresented scenarios in the test set, no expected outcome, no failure category, no transcript-to-tool evidence, or no baseline to detect regression.

Operational Dependency: Agent evaluation must use representative scenarios and failure categorization because answer quality, grounding, tool completion, and escalation behavior fail in different ways.

How the Exam Asks It: A question may describe a business scenario, a failing Copilot Studio configuration, or a deployment constraint and ask for the best design, first troubleshooting step, or required component.

How Distractors Are Designed: Distractors often solve a visible symptom while ignoring the owning object. They may suggest prompt tuning for a permission problem, a connector for a retrieval problem, a channel change for a flow exception, or manual deployment for an ALM dependency.

Why the Correct Answer Works: The correct option works because it turns agent quality into repeatable evidence through test cases, evaluation methods, failed-result categories, and regression comparison.

Practice Question: A team needs to prove that a production agent handles policy questions, tool actions, and escalation cases before publishing a new version. Which evaluation approach should you use?

A. Publish first and use live user complaints as the evaluation method.
B. Create a representative test set, choose evaluation methods for answer quality and task completion, then review failed results by category.
C. Evaluate only one successful conversation because generative behavior is deterministic after the first pass.
D. Remove tool calls from tests so failures focus only on language quality.

Correct Answer: B

Explanation: A is wrong because live complaints are late production signals, not a controlled pre-publish evaluation method. B is correct: Agent evaluation must use representative scenarios and failure categorization because answer quality, grounding, tool completion, and escalation behavior fail in different ways. C is wrong because one successful conversation cannot prove generative, retrieval, tool, and escalation behavior across scenarios. D is wrong because removing tool calls from tests hides the action path that production users depend on.

Troubleshooting Practice Question: A release candidate passes five happy-path chats but later fails escalation and tool-action cases in review. What should be fixed in the evaluation process?

A. Use only one longer chat transcript.
B. Remove tool cases from the test set.
C. Add representative escalation, negative, and tool-action cases with expected outcomes and failure categories.
D. Change the agent icon before retesting.

Correct Answer: C

Explanation: The failed review shows coverage gaps. Evaluation needs representative cases and categories, not more happy-path chat length.

Exam Takeaway: For quality questions, prefer representative test sets, explicit evaluation methods, failure categories, and regression comparison over one successful manual chat.

Best Choice Rules: If the stem asks for readiness evidence, choose representative test sets and evaluation methods. If the stem mentions a failed answer, classify the failure before changing prompts. If the stem mentions release safety, compare against a baseline or previous evaluation run.

Atomic Deconstruction - Operational Level

An evaluation is a controlled measurement of agent behavior. The test set supplies representative questions and expected outcomes, the evaluation method decides how answers or tasks are judged, and result review turns failures into remediation categories. A single successful chat is not evidence of production readiness.

Operationally, evaluation begins with test-case design. Each case needs an expected outcome and an owning failure category. When a case fails, classify it as routing, grounding, tool execution, handoff, formatting, escalation, or deployment before selecting a fix.

Component Specifications

Object	Attribute	Value Range	Default State	Dependency	Failure State
Test set	Scenario coverage	Questions, expected outputs, edge cases	Empty	Blueprint domains and production intents	Evaluation misses high-risk scenarios
Evaluation method	Scoring approach	Manual review, automated evaluation, task success checks	Ad hoc review	Expected answer and tool evidence	Results are subjective or not repeatable
Result category	Failure classification	Grounding, tool, topic routing, escalation, formatting	Unclassified	Transcript and telemetry	Team fixes the wrong layer
Regression baseline	Comparison point	Previous run, published version, expected threshold	No baseline	Versioned test set	Quality drift is not detected

Step-by-Step Execution Path

1. Create a test set that covers each agent domain: planning assumptions, knowledge retrieval, tool execution, multi-agent handoff, and ALM-sensitive behavior. Coverage must match real user tasks, not only happy paths.
2. Select evaluation methods that fit each test case. Grounded answers need reference comparison, tool actions need completion evidence, and escalation needs route verification.
3. Run tests before publishing and record failed items with category, transcript, expected result, actual result, and likely owning object.
4. Review results by failure category before changing prompts. Topic routing, retrieval, tool schema, and identity failures require different fixes.

Evidence note: evaluation checks rely on test set coverage, evaluation-result details, transcripts, task-completion evidence, and comparison with a previous baseline.

Technical Chain

A test set triggers representative conversations. Each case produces a transcript, tool or flow evidence, and evaluation output. The reviewer compares expected and actual behavior, classifies the failure, and decides whether the owning object is topic routing, grounding, tool execution, handoff, formatting, escalation, or ALM. Without classification, teams change prompts for connector failures or rebuild tools for retrieval failures.

Operational Skills Matrix

Task	Precise Command or Path	Verification Standard
Inspect test coverage	Copilot Studio evaluation/test set view	Test set includes normal, edge, negative, and escalation cases across exam domains
Review failed result	Evaluation result detail and conversation transcript	Failure is classified by route, grounding, tool, handoff, formatting, or escalation cause
Validate task completion	Transcript plus tool/flow run evidence	Expected external action completed or produced a handled failure response
Compare regression outcome	Current evaluation run vs previous run or baseline	No critical scenario regressed without an approved remediation plan

Implement ALM for Copilot Studio agents with solutions, environment variables, and Power Platform Pipelines

Exam Radar

Core Priority: This topic is repeatable lifecycle management. AB-620 can ask how to move agents and dependencies across environments without hardcoded URLs, personal connections, or manual edits.

High Frequency: Expect solutions, adding existing agents to solutions, environment variables, connection references, Power Platform Pipelines, deployment history, and post-deployment smoke tests.

Confusion Alert: An ALM question may look like an import failure, but the clue often points to a hardcoded endpoint, missing environment-variable current value, unbound connection reference, or unmanaged dependency.

Scenario Logic: A developer must move an agent from development to test and production without hardcoding API URLs, connection values, or environment-specific settings. The best first move is the one that preserves identity, data boundary, execution contract, or evidence collection before optimizing the conversation wording.

Version Delta: ALM scope includes solutions, adding existing agents to solutions, environment variables, and Power Platform Pipelines, so deployment answers must be environment-aware.

Failure Trigger: Failure appears as a solution missing dependent flows or connectors, environment variables without target values, unbound connection references, failed pipeline stage, or a post-deploy smoke test calling a development endpoint.

Operational Dependency: Copilot Studio ALM depends on solution packaging, environment variables, connection references, and pipeline movement so environment-specific configuration is injected rather than hardcoded.

How the Exam Asks It: A question may describe a business scenario, a failing Copilot Studio configuration, or a deployment constraint and ask for the best design, first troubleshooting step, or required component.

How Distractors Are Designed: Distractors often solve a visible symptom while ignoring the owning object. They may suggest prompt tuning for a permission problem, a connector for a retrieval problem, a channel change for a flow exception, or manual deployment for an ALM dependency.

Why the Correct Answer Works: The correct option works because solutions, environment variables, connection references, and pipelines preserve dependencies while allowing target environments to supply their own values and connections.

Practice Question: A developer must move an agent from development to test and production without hardcoding API URLs, connection values, or environment-specific settings. Which ALM configuration should you implement?

A. Export the agent manually from the browser each time and edit URLs after import.
B. Build separate unrelated agents in each environment to avoid deployment dependencies.
C. Store environment-specific values in topic text so makers can find them easily.
D. Add the agent and dependencies to a solution, use environment variables and connection references, and deploy through Power Platform Pipelines.

Correct Answer: D

Explanation: A is wrong because manual export and post-import edits are not repeatable and can leave production values inconsistent. B is wrong because separate unrelated agents create drift and do not prove deployment dependencies. C is wrong because topic text is not a configuration store for environment-specific values. D is correct: Copilot Studio ALM depends on solution packaging, environment variables, connection references, and pipeline movement so environment-specific configuration is injected rather than hardcoded.

Troubleshooting Practice Question: After deployment to test, the agent still calls a development API endpoint even though the solution import succeeded. What should you inspect first?

A. Environment-variable current values and any hardcoded URLs in flows or topic actions.
B. The number of trigger phrases.
C. The public course description.
D. The user avatar in Teams.

Correct Answer: A

Explanation: The symptom is environment-specific configuration drift. Target current values and hardcoded endpoint references should be checked before conversation design.

Exam Takeaway: For deployment questions, choose solution packaging, environment variables, connection references, and Power Platform Pipelines when the stem mentions target-environment differences.

Best Choice Rules: If the stem mentions different URLs, IDs, or settings per environment, choose environment variables. If the stem mentions connector authentication after import, inspect connection references. If the stem mentions controlled promotion, choose solutions and Power Platform Pipelines over manual export/import.

Atomic Deconstruction - Operational Level

ALM for Copilot Studio agents treats the agent, flows, connectors, environment variables, connection references, and dependent components as one deployable unit. Environment variables carry target-specific values; connection references bind connectors to target-environment connections; pipelines provide controlled movement and deployment evidence.

Operationally, ALM starts in the solution. The agent, flows, connectors, environment variables, and connection references must be included together. During deployment, target environments provide current values and connections, and pipeline history becomes the evidence that promotion followed the intended path.

Component Specifications

Object	Attribute	Value Range	Default State	Dependency	Failure State
Solution	Package boundary	Agent, flows, connectors, variables, dependencies	Unmanaged local assets	Power Platform environment	Missing dependency during import or publish
Environment variable	Configurable value	URL, IDs, feature flags, service endpoints	Unset in target	Solution import and target environment values	Agent calls development endpoint in production
Connection reference	Connector binding	Per-environment connection	Maker connection	Connector auth and DLP policy	Flow/tool cannot authenticate after deployment
Power Platform Pipeline	Deployment path	Dev, test, production stages	No pipeline	Managed environments and permissions	Uncontrolled deployment and inconsistent versions

Step-by-Step Execution Path

1. In Power Apps solution view, create or inspect the solution that contains the Copilot Studio agent and its dependent flows, custom connectors, connection references, and environment variables.
2. Replace hardcoded endpoints, IDs, and tenant-specific values with environment variables before deployment. This allows the same managed package to resolve different target values.
3. Validate connection references for every connector-backed flow or tool. The target environment must supply a valid connection that satisfies DLP and authentication requirements.
4. Use Power Platform Pipelines to move the solution through approved stages. Pipeline evidence gives a repeatable deployment record instead of a manual export/import trail.
5. After import, run a smoke test for topic routing, tool calls, knowledge access, and environment-specific endpoints before enabling the production channel.

Evidence note: ALM checks rely on solution contents, environment-variable current values, connection references, pipeline deployment history, and post-deploy smoke-test transcripts.

Technical Chain

A maker adds the agent and dependencies to a solution. During deployment, environment variables receive target-specific current values and connection references bind to valid target connections. Power Platform Pipelines moves the package through approved stages and records deployment history. If an endpoint or connection is hardcoded in a topic or flow, the imported agent may still call development resources or fail authentication in test or production.

Operational Skills Matrix

Task	Precise Command or Path	Verification Standard
Inspect solution contents	Power Apps > Solutions > selected solution > Objects	Agent and all dependent flows, connectors, variables, and connection references are included
Verify environment variable values	Solution > Environment variables in target environment	Each required variable has a target-specific current value
Check connection references	Solution > Connection references	Every reference is bound to a valid connection in the target environment
Review pipeline deployment	Power Platform Pipelines deployment history	Deployment completed for intended stage with version and approver evidence
Run post-deploy smoke test	Copilot Studio test pane in target environment	Core route, tool call, and endpoint-specific response succeed or fail with handled diagnostics